Dev (old posts, page 22)

ElasticUtils sprint at PyCon US 2013

, | Tweet this

What is it?

ElasticUtils is a Python library for building and executing ElasticSearch searches.

PyCon US 2013 sprint

I was only at the sprints for a single day. Rob and I spent some time working on elasticutils. Several good things came out of that:

  1. Rob wrote up an elasticutils Django middleware which throws a 501 or 503 page if an unhandled pyelasticsearch or requests exception is raised
  2. I fixed the Django tasks, added a test, and updated the documentation
  3. I cleaned up the Django ElasticSearchTestCase class
  4. I spent a bunch of time thinking about queries, syntax and functionality

Someone on IRC asked whe the next version of elasticutils will go out. I have no schedule right now, but I think it's important to let the code get used by projects that don't mind being bleeding edge and bake for a bit. The code in master tip right now is and the big change since 0.6 is that we switched from pyes to pyelasticsearch. That's a big change---the more baking it does, the better.

Having said that, a release depends mostly on how much free time I have in the near future. I'm about to lose all free time for a bit, so my guess is that we won't see a 0.7 release until this summer unless there's a compelling reason to push one out.

In the meantime, I'm actively maintaining the v0.5 and v0.6 branches. I'd like to stop maintaining the v0.5 branch, but need to get Mozillians and AMO off of it first.

If you have any questions, let us know! We hang out on #elasticutils on

Adding Persona authentication to richard

, | Tweet this


This is a post covering my first time experience with integrating Persona authentication into my Django project named richard. I briefly cover why I did it, what I used, and list the commits I did the work in as an example of how it can be done. I hope this helps others implement it on their sites..


A month ago, I added Persona authentication support to richard. This allowed me to use Persona authentication for I did this for several reasons:

  1. I wanted to try it out and see how well it worked on a small Django site (tl;dr works great---I'll use this on all my sites)
  2. I wanted people to authenticate with an email-based identity rather than a social network based identity
  3. I wanted to allow people to create accounts on, but didn't want to deal with the responsibility of protecting things like passwords

So that's where I'm coming from.


I used django-browserid which gives you some JavaScript and a few template tags that make it easy to incorporate Persona authentication into a Django app.

It took about 15 minutes to get it working. I've made some minor edits to the code since then and updated to v0.8 of django-browserid. All told, I think I've spent a couple of hours on Persona implementation.

In the process of doing that work, I hit a few minor issues, created some pull requests, helped with other pull requests and became one of the maintainers. Yay!

Here are the commits I did the work in. I figured the diffs might help you implement similar things on your sites:

That last commit updates to django-browserid master tip to pick up a fix to login failures if BROWSERID_CREATE_USER is False. That fix will be released in v0.8.1 soon.

further reading

The Mozilla Persona site helps understand why it exists and has a Developer FAQ.

The django-browserid docs are pretty good and walk through setting it up, advanced usage, and troubleshooting. I encourage you to read through them in full---it'll give you a better understanding of the pieces.

Dan Callahan did a talk at PyCon US 2013 on Persona. That's worth watching. It covers why Mozilla built it, how it works, and why it's important that it works that way. He also demos integrating it into sites and talks about using Persona authentication alongside other authentication methods.

If you're interested in adding Persona authentication to your Django site and need help, let me know.

Django Eadred v0.2 released! Django app for generating sample data.

, | Tweet this

Django Eadred gives you some scaffolding for generating sample data to make it easier for new contributors to get up and running quickly, bootstrapping required database data, and generating large amounts of random data for testing graphs and things like that.

For v0.2, I added some helper methods for generating names, email addresses, sentences and paragraphs. It's definitely the case that these helpers won't handle all use cases, but I think they'll help specific ones.

There are no backwards-compatability problems with v0.1.

To update, do:

pip install -U eadred

Django Eadred v0.1 released! Django app for generating sample data.

, | Tweet this

I work on a few projects that had a need for generating sample data to make it easier for new contributors to get up and running quickly with little effort. These projects are fairly data-driven---they're kind of useless without data.

To satisfy that need, we wrote an app in richard to generate sample data across all the other apps in the project. Then I rewrote it for input.

Then we had a hankering for it in SUMO, plus I thought it made sense to turn it into its own app. So I spun it out into its own project.

Thus django-eadred was born.

Generally, it allows you to define a module with a generate_sampledata function that takes command line options to generate sample data for any app you want to generate sample data for.

You can use it to define different ways of generating sample data specified by the command line.

You can use it to generate random data, non-random data, initial data, data for contributors, sample data for large data sets, fixture data, etc.

Check out for use cases, documentation and project details.

ElasticUtils v0.4 released!

, | Tweet this

What is it?

ElasticUtils is a Python library for building and executing ElasticSearch searches.

v0.4 released!

I released v0.4 a couple of days ago. This release adds new functionality, fixes some issues, adds more tests, and includes improved documentation.

On top of that, we removed the requirement for Django and moved the Django-aiding components into elasticutils.contrib.django. I personally like this because it makes it much easier to write test scripts to see how things react.

For the complete list of what's new, What's new in Version 0.4.

If you have any questions, let us know! We hang out on #elasticutils on

Pyblosxom has moved

, | Tweet this

I'm in the process of passing off maintenance to a few people who expressed interest on the pyblosxom-devel mailing list. Towards that goal, I've moved pyblosxom to a new github repository under the pyblosxom organization. Thus it's now at:

I then forked it, so you can continue looking at my fork if you so desire, but I probably won't be doing anything with it going forward.

Me and Pyblosxom

, | Tweet this

I've been working on Pyblosxom since January 9th, 2003. The initial commit from Wari is on November 24th, 2002. I discovered that (according to the commits), I was the second person to commit to the codebase. I didn't know that.

In January of 2012, I started a hiatus from Pyblosxom. We had done a ton of overhaul work for 1.4 and 1.5 and I was tired and wanted to work on other things.

I've decided it's time to end my reign as maintainer of Pyblosxom. I sent an email to the pyblosxom-devel list as such. Further, I contended that maybe it's time for the project to end altogether.

Yes, I suggested maybe it's time to end the project. The reasons are two-fold:

  1. The code has a ton of technical debt. There are a lot of plugins that need a lot of help. There's a lot of squirrely code. I've done a poor job of fixing the "Where can I find plugins?" and "Where can I find pretty flavours?" problems. Those are big problems and potentially require a lot of maintenance.
  2. It's really hard to fix architectural problems with Pyblosxom without changing the scope of the project. I contend it's way easier at this point to just start a new one.

Anyhow, I'm a little sad and I'll have to figure out what to do with my blog, but I think it's been a long time coming and it feels good to put it to rest.

So, two things:

  1. If you're interested in maintaining Pyblosxom, hop on the pyblosxom-devel mailing list and say hi!
  2. I cut a lot of my teeth on this project. Most of the current problems are my fault. If you're a current user of Pyblosxom, thank you for using something I spent a ton of time on and cared very much about and thank you for your patience as I figured a lot of things out.

On to new horizons!

elasticutils status -- May 18th, 2012

, | Tweet this

A few months ago, I "took over" maintenance of elasticutils. We use it in SUMO as the API for building search queries with elasticsearch.

One of the first things I did was spend some time figuring out whether we should keep working on elasticutils at all. django-haystack also provides a django-ish API for working with elasticsearch. Why have two libraries that at a high level do the same thing?

The thing is that they're not exactly the same. django-haystack is really great and supports a variety of backends for search, elasticsearch being one of them. Right now, it only has support for elasticsearch in 2.0 which is in either an alpha or beta state now (their web-site could use some updates). However, because it supports a bunch of backends, it only supports functionality that works across all of them.

elasticutils, on the other hand, is elasticsearch-specific. As elasticsearch adds functionality, we can, too. That's the compelling reason to keep working on this library. However, django-haystack has some awesome ideas that we'd like to implement in elasticutils, too. This will fix some sharp edges in elasticutils, but also make it much easier for projects to switch from one to the other.

Currently, elasticutils only handles the query side of things. django-haystack handles that, but also has an API for defining mappings, indexing, and all the other things you need with a search system.

Thus, Rob Hudson and I are going to embrace and extend elasticutils to:

  1. fix the current situation where it seems every elasticutils user is actually using their own branch with additional functionality in it (ew!)
  2. implement the rest of the things you need with a search system
  3. document the things we've learned while working with elasticutils because at a minimum, it seems most of the Mozilla projects that use elasticutils bumped into, spent time on, and solved the same problems---that's a huge waste of time and a failure on my part

One of the things users of a library need is for the library to be a mature project with releases, tagged version, documentation, tests, stability, reliability, reproduceability, communication, community and all that. Thus, I'm also going to spend some time to turn this into a real project. Towards that end, I created #elasticutils on where we'll talk dirty elasticutils stuff. If we end up with more people pitching in, we'll create a mailing list. But for now, IRC will do.

My next step is to spend a little time cleaning up what's in the master branch, then tag and release a baseline version.

After that, I'm going to spend time identifying, thinking about and merging in the divergent functionality in the various branches while Rob works on continuing his imperative mapping work.

I think in a couple of months, we'll be in a better place and that'll make it easier for Mozilla projects and anyone else who wants to use elasticutils to use and contribute to it.

If you're a user of elasticutils, please come hang out with us! Let us know how we can better help you.

Fix for vagrant keys permission issue

, | Tweet this

Tim threw together a vagrant setup for Kitsune based on what Alex did in November. I went to give it a try but I'm using vagrant 0.8.10 and it died.

Important part of the error message was:

`read': Permission denied - /var/lib/gems/1.8/gems/vagrant-0.8.10/keys/vagrant

Issue is documented at

My fix was to do this:

saturn /var/lib/gems/1.8/gems/vagrant-0.8.10/keys> ls -al
total 24
drwxr-xr-x  2 root root 4096 Jan 11 20:00 .
drwxr-xr-x 10 root root 4096 Jan 11 20:00 ..
-rw-r--r--  1 root root  821 Jan 11 20:00
-rw-------  1 root root 1675 Jan 11 20:00 vagrant
-rw-r--r--  1 root root 1464 Jan 11 20:00 vagrant.ppk
-rw-r--r--  1 root root  409 Jan 11 20:00
saturn /var/lib/gems/1.8/gems/vagrant-0.8.10/keys> sudo chmod 644 vagrant
saturn /var/lib/gems/1.8/gems/vagrant-0.8.10/keys>

That fixed it for me. Figured I'd document it so that I could find it again when I bumped into it again.

Pyblosxom 1.5 released!

, | Tweet this

While I was writing up the release blog post, I figured out that I've probably been working on Pyblosxom 1.5 on and off for the last three years or so. That's pretty intense---I knew I'd been working on it for a while, but I didn't realize how long it actually was.

The last year or so of development has been fun. There's a pretty consistent contingent of friendly folks who hang out on #pyblosxom. I really appreciate their humoring me with my period, "Pyblosxom 1.5 is almost done! I just have to ..." announcements.

Details of the release are in the release blog post. If you've been waiting to upgrade, this is a good version to upgrade to.

Going forward, I'm going to take a hiatus from Pyblosxom development. I've got a bunch of other projects that I've been ignoring for a while that really need some attention. I'll continue to review patches, help out on #pyblosxom and the mailing lists, and I'll push out future releases, but I won't be doing any major development for a while.