A while back, I wrote a post about poxx.py which talked about a script I based on Ned Batchelder's poxx.py script and overhauled to provide a faux "Swedish Chef" translation of Miro strings allowing me to test localizations of the application.

The transform from English to "Swedish Chef" had the following four impotant properties:

1. the output is vaguely readable
2. the output is longer than the input which helps us find ui issues
3. the output is clearly distinguished from English which helps us find strings that aren't getting translated
4. the output is mildly amusing which is sometimes important in dark times

Back in August, I made some changes and pulled it into fjord. This helped us suss out localization issues on a new site. However, I wasn't really happy with it. Amongst other things, I always wondered if "Swedish Chef" was kind of culturally insensitive.

A couple of weeks ago, I overhauled poxx.py again. This time, PIRATE! It continues to have the four properties I think are important for a test locale.

We're using it now for SUMO development. It's the grog to your Jolly Roger:

SUMO -- In Pirate!

We're using this script on both SUMO and Fjord now. You can use it for your site, too! The code is at https://github.com/mozilla/kitsune/tree/master/scripts/.

If you see any problems with it, toss me a message in a bottle.

This was my first full year at Mozilla and it was intense. I essentially worked on four projects: SUMO, Input, ElasticUtils and Gaia. This blog post talks about the first two which are worked on by the James' Rifles SUMINPUT Megalosaur team.

We accomplished a lot on SUMO this year. I spent a couple of hours last week throwing together a rough "year in review" script that looked at Bugzilla and git and crunched some numbers:

Twas the year: 2012
===================


Bugzilla
========

Bugs created: 938

              rrosario : 201
               a.topal : 188
                willkg : 108
           scoobidiver : 51
               igarcia : 41
                mverdi : 36
      swarnavasengupta : 30
                 james : 29
                  bram : 19
            tobbi.bugs : 17

Bugs resolved: 1025

              rrosario : 335
                       : WORKSFORME 18
                       :    INVALID 16
                       :  DUPLICATE 23
                       :    WONTFIX 7
                       :      FIXED 263
                       : INCOMPLETE 8
               a.topal : 182
                       : WORKSFORME 36
                       :    INVALID 41
                       :  DUPLICATE 11
                       :    WONTFIX 70
                       :      FIXED 21
                       : INCOMPLETE 3
                willkg : 131
                       :  DUPLICATE 6
                       :      FIXED 110
                       : WORKSFORME 2
                       :    WONTFIX 11
                       :    INVALID 2
                rdalal : 84
                       :      FIXED 84
                 james : 51
                       : WORKSFORME 6
                       :    INVALID 5
                       :  DUPLICATE 3
                       :    WONTFIX 15
                       :      FIXED 14
                       : INCOMPLETE 8
               mcooper : 37
                       :      FIXED 36
                       :    INVALID 1
            tobbi.bugs : 29
                       :      FIXED 29
             tgavankar : 28
                       :    WONTFIX 1
                       : WORKSFORME 1
                       :      FIXED 26
           scoobidiver : 28
                       :      FIXED 4
                       :  DUPLICATE 4
                       : WORKSFORME 11
                       :    WONTFIX 3
                       :    INVALID 6
               bmo2010 : 13
                       :      FIXED 1
                       :  DUPLICATE 3
                       : WORKSFORME 3
                       :    INVALID 6

            INCOMPLETE : 21
             DUPLICATE : 61
            WORKSFORME : 82
               INVALID : 91
               WONTFIX : 117
                 FIXED : 653

git
===

Total commits: 916

         Ricky Rosario : 430
      Will Kahn-Greene : 192
           Rehan Dalal : 98
           Mike Cooper : 44
             Erik Rose : 34
                 Tobbi : 29
        Tanay Gavankar : 23
           Kadir Topal : 11
             Tim Watts : 10
         Berker Peksag : 9
           James Socol : 7
            Victor Neo : 6
      Cesar Carruitero : 5
           David Lilly : 4
                  Ibai : 3
        Isac Lagerblad : 2
                 icaaq : 1
           TylerDowner : 1
              browning : 1
         ricky rosario : 1
    Anatoli Papirovski : 1
     Clauber Stipkovic : 1
          Jason Thomas : 1
                atopal : 1
      Florin Strugariu : 1

There are some interesting bits in there:

1. Ricky does a lot of work! Holy cow!

2. There were 23 people who contributed code to Kitsune (the SUMO codebase) this year. Of those, about half are volunteer contributors.

   Compare with 2011, we had 19 people who contributed to the code base and less than half were volunteer contributors.

3. We resolved more bugs than we created in 2012. We did that in 2011 as well, so that's two years in a row. I've never seen that happen before on a project I work on.

The codebase is pretty different now than it was at the beginning of the year. I helped with the following semi-massive overhauls:

1. The push for more metrics and dashboards to view the numbers.
2. The switch from Sphinx to ElasticSearch.
3. The new Information Architecture which affected browsing and searching across the site.
4. The site redesign which covered both the desktop and mobile versions of the site.
5. The upgrade to Django 1.4.
6. The switch from arecibo to sentry.
7. The push to switch from fixtures to model makers for all our tests.
8. The switch from weekly deployments on Tuesdays to deploying whenever we want. Continuous deployment is fantastic.
9. Started switching the whole site from Webtrends to Google Analytics. I saw Ricky write up a bunch of bugs to finish up that work, so I'll say it's in progress.
10. During the redesign, Rehan redid all the CSS and switched us to use LESS.
11. I spun off some code I wrote for richard, then ported to Fjord, then improved into a project called django-eadred. That makes it a lot easier to generate sample data for a variety of purposes like new contributors, bootstrapping, and large random data sets.

On top of that, we did a lot of work on the documentation and making it easier to get to a working Kitsune development environment. We switched to a sprint-based work flow using Scrumbugz. We also nixed our daily checkin conference call for an IRC-based checkin system that we wrote called Standup.

It's been a big year.

For Input, it was a bigger year. We decided to abandon the old Input codebase (omfg yay) in favor of rewriting it from the ground up. The rewrite took a couple of months and then has sort of been sitting around waiting for a security review. In the meantime, we (actually, Mike did) fixed a bunch of issues with the old site code because that's what's currently in production.

Rewriting Input wouldn't have taken so long except that we did a lot of work fixing bugs in external libraries and updating Playdoh. That work definitely cut into our schedule, but it benefitted a bunch of other groups/people/sites, so that's good.

That's the gist of the year: it was a lot of work, but we accomplished a ton.

w00t for 2012!

For the next few months, I'm switching projects to help work on Gaia. I essentially started yesterday, but I'm still missing a bunch of pieces, so I haven't actually done any work. What I have done is spent time immersing myself in the project and trying to get my bearings.

Thus this blog post covers how I got my bearings so far.

Gaia is a project in heavy flux and moving fast. The state and stability of things changes day to day. There are things that aren't documented. There are things that are documented that are out of date. There are dozens of etherpads, lists of bugs, wiki pages, and tips and tricks scattered around. This is the way it is currently. Even this will probably change.

However it's not all chaos and entropy. While a lot of things are in flux, some things stay the same. That's why I decided to write this blog post of things I think help get you up and running faster.

On Monday

Read through these hacking wiki pages:

• Gaia/Hacking wiki page
• B2G/Hacking wiki page

Join the #gaia and #b2g IRC channels on irc.mozilla.org.

Join the dev-gaia mailing list.

Fork and clone the Gaia github repository.

That seems like a short list, but take the time to catalog in your head all the things that are there.

On Tuesday

Go to the Gaia weekly meeting.

Mute and facemute yourself. Make sure to follow along in the Etherpad notes they link to. The meeting I went to used this Etherpad: https://etherpad.mozilla.org/gaia-meeting-notes

About 1/3 of the way down that pad, there's a list of components, who's working on them, their status, etc---that's current as of the time of this writing.

In going to the meeting and reading through the notes, you'll get a sense of who's who, who's working on what, what the current sprint priorities are, and you might also get an indication of where you can help out.

After that, work on getting Gaia working in the B2G desktop nightly build.

The B2G desktop periodically has stability issues. If you run into problems, ask on #gaia on IRC.

On Wednesday

Assuming you have everything working so far, now's the time to start looking for bugs to work on and/or testing the existing apps.

As of the time of this writing, the B2G/Triage wiki page has a variety of lists of bugs in various states. There's the P1 and P2 lists in the Gaia section.

Also, I've accumulated these lists, but they may not be valid anymore:

• unowned polish bugs
• very short list of bugs marked with the mentored tag

I think the workflow is something like this:

1. find a bug you can work on that's not assigned to anyone
2. assign that bug to yourself
3. work on it
4. produce a patch --- must include tests!
5. create a pull request on github
6. find a reviewer to look at it --- probably want someone who works on that component; ask on #gaia on IRC
7. go through review until it's good
8. get someone to land it --- I'm fuzzy on this step, but the person needs commit access to the repository on github; ask on #gaia on IRC

Thursday, Friday, etc

Rinse, repeat.

Conclusion

Hope this helps someone else! I think the important thing is to go to a Gaia weekly meeting.

Addendums

Random thoughts that didn't fit anywhere else in this hastily written post:

1. If you bump into incorrect information in the Gaia/Hacking wiki page, please update it or ask someone on #gaia to verify it's incorrect.
2. If you ask a question and no one replies to you on IRC, wait a bit, then ask again. Folks are busy and in different time zones, but they are paying attention.
3. If you see anything in this blog post that's incorrect, find me on IRC. I'm willkg.
4. e.me stands for "everything.me"
5. FTU stands for "first-time-usage"

Also, I overheard this on IRC and it helped:

<fzzzy> here's something important to understand about ffos:
there's b2g, and there's gaia

<fzzzy> b2g is the big compiled blob of c++ and some js modules

<fzzzy> gaia is all js, but it is preprocessed into a profile
directory

<fzzzy> if you double-click B2G.app, you get a gaia profile
that is inside of the app

<fzzzy> if you run b2g-bin from the command line, you can pass
the -profile /path/to/profile flag, and b2g will use that gaia

<fzzzy> it just depends if you want to just kick the tires, or
actually hack on gaia itself

A few months ago, I "took over" maintenance of elasticutils. We use it in SUMO as the API for building search queries with elasticsearch.

One of the first things I did was spend some time figuring out whether we should keep working on elasticutils at all. django-haystack also provides a django-ish API for working with elasticsearch. Why have two libraries that at a high level do the same thing?

The thing is that they're not exactly the same. django-haystack is really great and supports a variety of backends for search, elasticsearch being one of them. Right now, it only has support for elasticsearch in 2.0 which is in either an alpha or beta state now (their web-site could use some updates). However, because it supports a bunch of backends, it only supports functionality that works across all of them.

elasticutils, on the other hand, is elasticsearch-specific. As elasticsearch adds functionality, we can, too. That's the compelling reason to keep working on this library. However, django-haystack has some awesome ideas that we'd like to implement in elasticutils, too. This will fix some sharp edges in elasticutils, but also make it much easier for projects to switch from one to the other.

Currently, elasticutils only handles the query side of things. django-haystack handles that, but also has an API for defining mappings, indexing, and all the other things you need with a search system.

Thus, Rob Hudson and I are going to embrace and extend elasticutils to:

1. fix the current situation where it seems every elasticutils user is actually using their own branch with additional functionality in it (ew!)
2. implement the rest of the things you need with a search system
3. document the things we've learned while working with elasticutils because at a minimum, it seems most of the Mozilla projects that use elasticutils bumped into, spent time on, and solved the same problems---that's a huge waste of time and a failure on my part

One of the things users of a library need is for the library to be a mature project with releases, tagged version, documentation, tests, stability, reliability, reproduceability, communication, community and all that. Thus, I'm also going to spend some time to turn this into a real project. Towards that end, I created #elasticutils on irc.mozilla.org where we'll talk dirty elasticutils stuff. If we end up with more people pitching in, we'll create a mailing list. But for now, IRC will do.

My next step is to spend a little time cleaning up what's in the master branch, then tag and release a baseline version.

After that, I'm going to spend time identifying, thinking about and merging in the divergent functionality in the various branches while Rob works on continuing his imperative mapping work.

I think in a couple of months, we'll be in a better place and that'll make it easier for Mozilla projects and anyone else who wants to use elasticutils to use and contribute to it.

If you're a user of elasticutils, please come hang out with us! Let us know how we can better help you.

I'm at the webdev work week in Santa Cruz, CA, USA this week. It's great to meet people I've been talking to for the last 6 months. It's also kind of nice to take a break from the SUMO sprints. I've been spending the time lifting my head and seeing what's been happening while I wasn't paying attention.

List of three things on my mind:

1. I wish I had more time. There are so many people doing exciting things that I'd love to work on, too, but I just don't have enough time. It's hard to suppress the, "Wow---that looks awesome! I'll help you on that project!" reflex. I just don't have time. It's kind of a bummer. On the flip side, if I do discover myself with free time, I have a long (long) list of things I'm interested in.
2. It's nice to talk about technical things. It's nice to make a (bad) joke about mime types and get a response. (An HTTP response! ha ha!)
3. Everyone is really super and friendly. I don't think I've been in a group this big that's lacking a token jerk. This has caused me to idly wonder if maybe I'm the token jerk. If so, I hope someone leaves me a note so I can adjust accordingly.

Things I'm taking away (so far):

1. I should dig up my compilers book and re-read it because I really liked studying compilers way back when.
2. I screwed up with spreading elasticsearch knowledge. I keep having conversations about problems we've seen and solved in SUMO that other projects are having. Plus it seems like everyone has their own branch of elasticutils with differing features. I'm going to try to spend some time working on making the situation better both by documenting things and also improving elasticutils so we're not all maintaining our own versions.
3. Need to spend more time learning things.

I haven't had time to blog much in the last few months. At work, I've been spending all my time with elasticsearch, elasticutils, and SUMO bug fixing. I've been working on the conversion from Sphinx search to elasticsearch for SUMO since I started at Mozilla, but I've only recently felt like I'm really getting the hang of it. There are a bunch of elasticutils-related things I want to blog about, but those will come in fugure entries.

In my spare time, I've been working on richard. This project has nothing to do with Richard of air mozilla fame, but rather is a video indexing web application. It's the software that runs pyvideo.org.

pyvideo.org has the distinction of being the first Django application I've built from the ground up. That distinction is both a virtue (yay for first apps!) and a vice (boo for silly things I did when doing it!).

The one thing I did that I'm really proud of is that when building the software, I knew I needed help if it was to succeed and thus I worked to make it easy and inviting for contributors to get involved:

• I wrote documentation: license file, README, documentation covering how to install it for hacking, how to contribute, where to find me, ...
• I parked the code on GitHub to make it easier for people to access. [1]
• I made sure there were a series of issues in the issue tracker that showed the next round of things that needed to be done.
• I made sure I had an IRC channel and that people knew where to find me to ask questions.
• I quickly got the documentation built on ReadTheDocs.
• I had a test infrastructure set up. [2]
• I respond to everyone who sends an email, creates a pull request, writes an issue, says hi on IRC, ...

The end result of that is that there are 4 contributors to richard including myself and one of them is very active.

Asheesh did a talk at LibrePlanet 2012 that mentioned Mako's power law of contributions to open source projects. The gist of it is that most open source projects only ever have one contributor. [3]

Well, I've got 5 on my video index web application software that I "launched" a month ago. I'm feeling good about that.

Tim threw together a vagrant setup for Kitsune based on what Alex did in November. I went to give it a try but I'm using vagrant 0.8.10 and it died.

Important part of the error message was:

/var/lib/gems/1.8/gems/net-ssh-2.1.4/lib/net/ssh/key_factory.rb:38:in
`read': Permission denied - /var/lib/gems/1.8/gems/vagrant-0.8.10/keys/vagrant
(Errno::EACCES)

Issue is documented at https://github.com/mitchellh/vagrant/issues/235.

My fix was to do this:

saturn /var/lib/gems/1.8/gems/vagrant-0.8.10/keys> ls -al
total 24
drwxr-xr-x  2 root root 4096 Jan 11 20:00 .
drwxr-xr-x 10 root root 4096 Jan 11 20:00 ..
-rw-r--r--  1 root root  821 Jan 11 20:00 README.md
-rw-------  1 root root 1675 Jan 11 20:00 vagrant
-rw-r--r--  1 root root 1464 Jan 11 20:00 vagrant.ppk
-rw-r--r--  1 root root  409 Jan 11 20:00 vagrant.pub
saturn /var/lib/gems/1.8/gems/vagrant-0.8.10/keys> sudo chmod 644 vagrant
saturn /var/lib/gems/1.8/gems/vagrant-0.8.10/keys>

That fixed it for me. Figured I'd document it so that I could find it again when I bumped into it again.