SUMO: 2013 retrospective

, | Tweet this

It was a big year for SUMO. In 2012, we got a lot accomplished: new search, new information architecture,

One thing I didn't do was make my year-end script product better output.

Anyhow---on with stats!

Twas the year: 2013
===================


Bugzilla
========


Bugs created: 889

               a.topal : 156
              rrosario : 100
                willkg : 87
           scoobidiver : 85
                  ibai : 58
                mverdi : 54
               mcooper : 42
                feer56 : 35
          krystaiceman : 26
                rdalal : 19
            david.weir : 15
                shuhao : 13
      swarnavasengupta : 9
       andrei.hutusoru : 8
           me+bugzilla : 7
            tobbi.bugs : 6
                 mluna : 6
          joshua-smith : 6
               tdowner : 6
               leszekz : 6
        yoshi.yokotani : 5
        stephen.donner : 5
                 slurp : 5
                  mana : 5
             madperson : 4
              kbrosnan : 4
             tonnes.mb : 4
               rardila : 4
              pmcclard : 4
               dbialer : 4
             michaljev : 4
                   abc : 4
                  l10n : 4
               pcvrcek : 3
                 rdaub : 3
              fabricio : 2
             rmcguigan : 2
          sudheesh1995 : 2
          alex_mayorga : 2
          simone.lando : 2
            nishant_cs : 2
                  bram : 2
               smolejv : 2
        bob.silverberg : 2
              rtanglao : 2
        kdurant35rules : 2
            amit103065 : 2
         subedimahadev : 2
                lhenry : 2
          thomas.lendo : 2
             shawnsumo : 2
              mhammond : 1
        kdurant35rules : 1
                  djst : 1
               curtisk : 1
        chiorean.ioana : 1
                bermea : 1
               friedel : 1
        bputstudentweb : 1
     margaret.leibovic : 1
             rbillings : 1
       nikitan.dolmart : 1
       georgevidalakis : 1
            nsm.nikhil : 1
              satishb3 : 1
            bwbrowning : 1
              bugzilla : 1
                  coce : 1
              EddyCarr : 1
              gryllida : 1
        mohammed.samad : 1
                  6a68 : 1
         krupa.mozbugs : 1
           John99-bugs : 1
             wjohnston : 1
              barderne : 1
               jan0286 : 1
               fwenzel : 1
               rnewman : 1
           this4midhun : 1
              bjohnson : 1
  bugzilla-fromthedeep : 1
             iamjithin : 1
               bmo2010 : 1
    chrismore.bugzilla : 1
                 evold : 1
              jbertsch : 1
                yousef : 1
          pmjcreations : 1
               rhelmer : 1
              danishka : 1
                  mail : 1
             gphemsley : 1
              Rebeccah : 1
           ckreinbring : 1
               stephen : 1
         berker.peksag : 1
                jezdez : 1
                 nchen : 1
         iamjayakumars : 1
            netfuzzerr : 1
              benjamin : 1

Bugs resolved: 1116

              rrosario : 386 resolved, 273 fixed
                rdalal : 152 resolved, 150 fixed
               a.topal : 121 resolved, 41 fixed
               mcooper : 118 resolved, 105 fixed
                willkg : 72 resolved, 61 fixed
           scoobidiver : 31 resolved, 0 fixed
             michaljev : 20 resolved, 17 fixed
      swarnavasengupta : 18 resolved, 0 fixed
                shuhao : 16 resolved, 13 fixed
           me+bugzilla : 12 resolved, 5 fixed
         berker.peksag : 12 resolved, 12 fixed
                mverdi : 11 resolved, 5 fixed
                  erik : 9 resolved, 9 fixed
                lhenry : 8 resolved, 1 fixed
          krystaiceman : 7 resolved, 1 fixed
            tobbi.bugs : 7 resolved, 6 fixed
          joshua-smith : 6 resolved, 4 fixed
                 jfong : 5 resolved, 5 fixed
               tdowner : 5 resolved, 2 fixed
                feer56 : 5 resolved, 0 fixed
            david.weir : 5 resolved, 4 fixed
            bwbrowning : 5 resolved, 5 fixed
                  bram : 5 resolved, 4 fixed
                  ibai : 4 resolved, 1 fixed
   alastra.mariagrazia : 4 resolved, 2 fixed
                 laura : 4 resolved, 4 fixed
              williamr : 3 resolved, 1 fixed
         buchanae+bugs : 3 resolved, 3 fixed
           bharath_ves : 3 resolved, 3 fixed
              paul+moz : 3 resolved, 3 fixed
          chris.lonnen : 2 resolved, 2 fixed
               smolejv : 2 resolved, 1 fixed
             zcampbell : 2 resolved, 0 fixed
         iamjayakumars : 2 resolved, 0 fixed
               curtisk : 2 resolved, 0 fixed
               leszekz : 2 resolved, 0 fixed
                   abc : 2 resolved, 1 fixed
               pcvrcek : 2 resolved, 0 fixed
          taygunagiali : 2 resolved, 2 fixed
                  mail : 1 resolved, 0 fixed
               bmo2010 : 1 resolved, 1 fixed
               wymette : 1 resolved, 0 fixed
                beaotx : 1 resolved, 1 fixed
                nelson : 1 resolved, 1 fixed
             madperson : 1 resolved, 1 fixed
                  coce : 1 resolved, 0 fixed
              pmcclard : 1 resolved, 1 fixed
             tgavankar : 1 resolved, 1 fixed
  bugzilla-fromthedeep : 1 resolved, 0 fixed
              rtanglao : 1 resolved, 0 fixed
        stephen.donner : 1 resolved, 1 fixed
       guillermo.movia : 1 resolved, 1 fixed
              lorchard : 1 resolved, 1 fixed
              nukeador : 1 resolved, 0 fixed
             rtucker11 : 1 resolved, 0 fixed
            nishant_cs : 1 resolved, 0 fixed
                  stas : 1 resolved, 0 fixed
             mattbasta : 1 resolved, 1 fixed
              satishb3 : 1 resolved, 0 fixed
              ragsagar : 1 resolved, 1 fixed
             rmcguigan : 1 resolved, 1 fixed
                 nchen : 1 resolved, 0 fixed
                kudrom : 1 resolved, 1 fixed
       andrei.hutusoru : 1 resolved, 0 fixed
                  reed : 1 resolved, 0 fixed
           tiziana.sel : 1 resolved, 0 fixed
       chance.zibolski : 1 resolved, 1 fixed
             alice0775 : 1 resolved, 0 fixed
                  ravi : 1 resolved, 0 fixed
            nsm.nikhil : 1 resolved, 0 fixed
              gryllida : 1 resolved, 1 fixed
    deletesoftware+moz : 1 resolved, 0 fixed
          pmjcreations : 1 resolved, 0 fixed
                boerni : 1 resolved, 1 fixed
               rforbes : 1 resolved, 0 fixed
               dbialer : 1 resolved, 1 fixed
                jgross : 1 resolved, 1 fixed

            INCOMPLETE : 43
               WONTFIX : 51
             DUPLICATE : 64
               INVALID : 73
            WORKSFORME : 121
                 FIXED : 764

Research bugs: 15

761582: [research] Add feature: Articles that link to this article
788104: [research] [ux] Support multiple products in the support forum
815089: [research] Investigate telling apart Firefox for Desktop and Firefox for Mobile tweets
816970: [research] SurveyGizmo API to be used in automated exit survey
823060: [research] Use datetime instead of ints in ES mappings
823891: [research] Adding KB revisions feature
825621: [research] Store the templates, article links and images in each article
825624: [research] Investigate how to update to Twitter API v1.1
841412: [research] Bad localization strings shouldn't break the site.
845290: [research] URL bar should fade away on SUMO
854554: [research] Youtube embeds don't work with templates
889884: [research] Open Badges!
889890: [research][discuss] figure out how to improve our l10n situation with search
906992: [research] Add support for multiple ES indexes by doc type
937889: [research] Login users via a URL in email

Tracker bugs: 20

433161: [Tracker] Support for forums in other languages
625891: [tracker] HTML email
721462: [tracker] Taxonomy IA improvements
758598: [Tracker] Search UX suggestions
783262: [tracker] Add rate limiting to protect us from spammers
790785: [Tracker] L10n tools editing part
790786: [Tracker] L10n tools organization part
800962: [Tracker] Add activity history page for KB
815625: [Tracker] Segment dashboards and other contributor pages by product
817540: [tracker] AJAXify the refine+focus panel
825606: [tracker] Switch everything from Webtrends to Google Analytics
827640: [tracker] Localize Questions
838584: [Tracker] Getting ready for Firefox OS launch
845286: [tracker] Use as little bandwidth as possible on mobile version of SUMO
845773: [Tracker] move to an OS charting solution
848520: [tracker] Make all traffic HTTPS
851730: [tracker] Close threads pro-actively
871559: [tracker] update codebase to django 1.4 layout
897057: [tracker] Open Badges -- stage 1
920530: [tracker] support Webmaker on SUMO

git
===

Total commits: 1138

         Ricky Rosario : 492  (+16258, -16435, files 2972)
      Will Kahn-Greene : 178  (+8311, -3748, files 438)
           Rehan Dalal : 168  (+13016, -5554, files 680)
           Mike Cooper : 145  (+46955, -22136, files 582)
           Kadir Topal : 39  (+352, -110, files 61)
      Michał Frontczak : 19  (+229, -182, files 78)
         Berker Peksag : 15  (+570, -717, files 73)
             Shuhao Wu : 15  (+1523, -127, files 51)
       Jen Fong-Adwent : 9  (+138, -18, files 17)
                 Tobbi : 8  (+338, -204, files 13)
              browning : 5  (+140, -16, files 12)
             davd Weir : 4  (+15, -1, files 4)
          Joshua Smith : 4  (+94, -87, files 13)
         Tobias Markus : 3  (+8, -8, files 4)
                 Anush : 3  (+4, -1, files 3)
       Gaurav Dadhania : 3  (+3, -3, files 3)
  Bharath Thiruveedula : 3  (+15, -14, files 3)
                  ibai : 3  (+30, -30, files 4)
                kudrom : 2  (+9, -9, files 5)
             Nghi Tran : 2  (+2, -1, files 2)
          Tanner Filip : 2  (+4, -4, files 2)
                Börni : 2  (+30, -15, files 4)
             madperson : 2  (+5, -4, files 2)
         Taygun AGIALI : 2  (+7, -6, files 3)
           TylerDowner : 2  (+3, -3, files 2)
           James Socol : 2  (+37, -27, files 3)
               david-w : 1  (+1, -1, files 1)
              ragsagar : 1  (+16, -1, files 2)
       Guillermo Movia : 1  (+1, -0, files 1)
              rosanaar : 1  (+9, -0, files 1)
              Gryllida : 1  (+26, -6, files 3)
       Beatriz Nombela : 1  (+9, -9, files 6)

Total lines added:   88158
Total lines deleted: 49477
Total files changed: 5048
  1. Ricky does a lot of work! Holy cow!

  2. In 2011, we had 19 people who contributed code changes.

    In 2012, we had 23 people.

    In 2013, we had 32 people.

  3. Like 2011 and 2012, we resolved more bugs than we created in 2013. That's three years in a row! I've never seen that happen on a project I work on.

  4. There are a lot of people braving Bugzilla to write up bugs. Skimming the list, I see developers, non-developers, Support contributors, localizers, support team and a lot of people I don't recognize.

Here's some number comparisons:

name 2011 2012 2013
Bugs created: 1357 938 889
Bugs resolved: 1637 1025 1116
Total commits: 1137 916 1138
Code contributors: 19 23 32

I spent a good chunk of 2013 working on Input, but here's what I remember from SUMO development in 2013:

  1. We rearranged the codebase for better Django 1.4 layout. That was a project. Oy.
  2. We added support for non-English languages to the support forums!
  3. We switched email to be HTML formatted. We also reworked email to be localized.
  4. We switched to Google Analytics.
  5. We implemented Open Badges---though there's still a few important pieces to finish there.
  6. We switched to YouTube for videos.
  7. We added support for Webmaker and Firefox OS. Thunderbird support will be added to SUMO in 2014.
  8. Mike took a lantern, a crust of bread and a big sword and spelunked into the darkest dungeons filled with stinky, squelchy muck and rewrote the showfor code.
  9. We reworked our search code to handle multiple indexes, though we haven't taken advantage of that, yet.
  10. We switched deployment to use Dennis to lint all translated strings before pushing them to production. This has almost assuredly saved us from production fires. I hated those kinds of fires. Hooray for Dennis!
  11. We wrote and switched to Ernest for sprint planning and coordination.
  12. We overhauled everything to add support for Persona authentication, but had to push off deployment indefinitely because of problems with Persona which are being ironed out by the Persona team.
  13. We added an escalation system for questions that haven't received a response in x hours for some positive value of x that is still in flux.
  14. We ditched Highcharts.
  15. We wrote a command-line deployer which tells us exactly what's going out and tells New Relic, too. This gives us a much better idea of what we're deploying and how it affected the site afterwards. This command-line deployer is named chief-james in honor of James who has moved on to greener and well measured pastures.
  16. We added a bunch of new metrics, dashboards, history pages, activity pages, icons, bicons, landing pages, take-off pages, topics, subtopics, toe picks and all kinds of stuff.

That's the gist of the year: it was a lot of work, but we accomplished a ton.

w00t for 2013!

Input: 2013 retrospective

, | Tweet this

It was a big year for Input. In 2012, we spent the last half rewriting Input. In 2013, it went through secreview, had a bunch of things fixed and then we migrated to the new system.

Since then, we've been fixing bugs, reimplementing features that were lost and writing the scaffolding for the new set of User Advocacy dashboards and tools.

Let's look at some Bugzilla and git stats for the year:

Twas the year: 2013
===================


Bugzilla
========

Bugs created: 150

                willkg : 100
            cwwmozilla : 5
                fbraun : 4
               mgrimes : 4
               tdowner : 3
        stephen.donner : 3
           me+bugzilla : 2
        gasell+mozilla : 2
               mcooper : 2
                 glind : 2
             mozaakash : 1
        kdurant35rules : 1
            hitmanarky : 1
              kbrosnan : 1
        bob.silverberg : 1
              splewako : 1
              rrosario : 1
             mattbasta : 1
              educmale : 1
                feer56 : 1
                326374 : 1
               anthony : 1
        shopov.bogomil : 1
               peterbe : 1
                  l10n : 1
    chrismore.bugzilla : 1
                landis : 1
          dron.rathore : 1
                    rq : 1
             MattN+bmo : 1
          joshua-smith : 1
                cturra : 1
        swagat.kanungo : 1

Bugs resolved: 268

                willkg : 157
                       :    WONTFIX 50
                       :      FIXED 89
                       : WORKSFORME 8
                       :  DUPLICATE 9
                       :    INVALID 1
            cwwmozilla : 57
                       :      FIXED 1
                       :    WONTFIX 7
                       : WORKSFORME 29
                       :  DUPLICATE 1
                       :    INVALID 19
               mgrimes : 10
                       :      FIXED 1
                       :  DUPLICATE 1
                       : WORKSFORME 5
                       :    INVALID 3
        shopov.bogomil : 7
                       :    WONTFIX 1
                       : WORKSFORME 2
                       :    INVALID 1
                       :      FIXED 2
                       :  DUPLICATE 1
               mcooper : 6
                       :  DUPLICATE 1
                       :      FIXED 5
               mozilla : 5
                       :      FIXED 5
           me+bugzilla : 4
                       :    WONTFIX 1
                       :      FIXED 1
                       :  DUPLICATE 1
                       :    INVALID 1
             mozaakash : 2
                       : WORKSFORME 1
                       :    INVALID 1
        trifandreialin : 2
                       : WORKSFORME 2
              rrosario : 2
                       :      FIXED 2
          joshua-smith : 2
                       :      FIXED 1
                       :    INVALID 1
           aaron.train : 2
                       :    WONTFIX 1
                       :  DUPLICATE 1
        stephen.donner : 1
                       : INCOMPLETE 1
               emorley : 1
                       :      FIXED 1
               curtisk : 1
                       :    INVALID 1
               unghost : 1
                       : WORKSFORME 1
          rajul.iitkgp : 1
                       :      FIXED 1
             jruderman : 1
                       : INCOMPLETE 1
          chris.lonnen : 1
                       :      FIXED 1
             nigelbabu : 1
                       :      FIXED 1
              tofumatt : 1
                       :      FIXED 1
                cturra : 1
                       :      FIXED 1
               fwenzel : 1
                       :      FIXED 1
               mbrandt : 1
                       :      FIXED 1

            INCOMPLETE : 2
             DUPLICATE : 15
               INVALID : 28
            WORKSFORME : 48
               WONTFIX : 60
                 FIXED : 115


git
===

Total commits: 277

      Will Kahn-Greene :   251  (+51614, -16878, files 1132)
           Mike Cooper :    12  (+38545, -249, files 219)
        Brandon Burton :     5  (+21, -178, files 6)
         Ricky Rosario :     4  (+36, -19, files 6)
        Bob Silverberg :     2  (+11, -6, files 2)
                 Rajul :     1  (+3, -0, files 1)
          Joshua Smith :     1  (+10, -5, files 1)
               bogomil :     1  (+1, -1, files 1)


Total lines added:   90241
Total lines deleted: 17336
Total files changed: 1368

I want to highlight some interesting bits:

  1. We resolved more bugs than we created. That's partially due to us going through and closing out old bugs for the old Input that aren't relevant anymore.

  2. According to the Bugzilla and git data, there were 47 contributors to Input this year: 326374, Bob Silverberg, Brandon Burton, Joshua Smith, MattN+bmo, Mike Cooper, Rajul, Ricky Rosario, Will Kahn-Greene, aaron.train, anthony, bogomil, chris.lonnen, chrismore.bugzilla, cturra, curtisk, cwwmozilla, dron.rathore, educmale, emorley, fbraun, feer56, fwenzel, gasell+mozilla, glind, hitmanarky, jruderman, kbrosnan, kdurant35rules, l10n, landis, mattbasta, mbrandt, me+bugzilla, mgrimes, mozaakash, nigelbabu, peterbe, rajul.iitkgp, rq, splewako, stephen.donner, swagat.kanungo, tdowner, tofumatt, trifandreialin, and unghost.

    That doesn't include localizers who do a ton of work translating the strings in the Input ui.

    That includes some of the folks who work on the input-tests repository, but possibly misses some.

  3. Most of the 47 contributors are not "core developers". That's cool, but I could be doing a better job here making it easier for non-core developers.

    We maintain a Get Involved page and we hang out on #input on irc.mozilla.org. We have a input-dev mailing list. If you want to work on Input, this is where it's at!

Those are the stats.

At a high-level, we accomplished the following:

  1. stood up a new Input code base
  2. the beginnings of spam identification and removal
  3. Input API for feedback submission
  4. Firefox OS feedback form
  5. infrastructure for an Analysts group with special privileges
  6. the beginnings of an Occurrence Comparison report dashboard

One thing I discovered in 2013q4 was that it's really hard to be the mostly-solo dev on a project like this. I'm lucky that I'm part of a larger team, so peer reviews for work I've done is possible and timely. However, I find I'm switching contexts between the technical details of what I'm working on now and the high-level details of a bunch of possible future tasks/projects. That's really hard to do day-to-day and still maintain development momentum. I have some thoughts on how to serialize my work so that I'm doing less context switching and I can focus on individual things more deeply which should produce better outcomes.

My goals for Input for 2014 are these:

  1. clean up the code base: there's still a bunch of weird stuff in there from the rapid development work we did in 2012
  2. reduce barriers to entry for new contributors: better documentation, fewer steps to get up and running, more bugs marked for mentoring, more outreach, ...
  3. build infrastructure that we can use for better User Advocacy tools: watched alerts, email notifications, dashboards, ...
  4. flesh out tests: we're really light on smoketests and regression-catching tests
  5. work with Matt and Cheng to figure out where Input fits into the grand scheme of things; how can we make it a general-purpose feedback system? how can we handle non Firefox products and initiatives?

Yay for 2013!

Update 7:08pm

My script only showed top tens which misses tons of people who did work. I redid the data and that increases the number of contributors from 16 to 47. Oops!

Update April 21st, 2015

LGuruprasad found a bug in the script that caused commits-by-author information to be wrong. Fixed the script and updated the stats!

pyvideo status: November 24th, 2013

, | Tweet this

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

Lot of stuff has happened since the last status report, but there are four things of note:

  1. Sheila is now a co-admin of pyvideo.org. She has been for a couple of months. I need to update the site to reflect this.

    I'm really psyched about this. It's a ton of work and I'm just not managing it well. Splitting the work should make it more manageable.

  2. Back in July, Sheila poked me about a tweet Jesse wrote suggesting Rackspace was interested in sponsoring Open Source projects. She contacted Jesse and set everything up.

    I'm psyched that Rackspace agreed to sponsor pyvideo.org by providing free hosting. Several months later, I moved pyvideo.org from where it was before to a vm at Rackspace.

    I'm really excited about this! It makes a bunch of problems that I was trying to figure out what to do about go away.

    Thank you, Rackspace!

    I need to update the site to reflect this.

  3. Sheila discovered that blip.tv was expiring a bunch of accounts that held conference videos and that those videos would go away. She and I scrambled to download all the files from blip and move them to Rackspace cloudfiles. It's about 600 videos and around 250gb of data.

    In the process of doing that, we saved videos for DjangoCon EU 2010, DjangoCon EU 2011 and PyGotham 2012. I added these to pyvideo.org today. These videos have pages that are stubs with no metadata. I've got that in my queue of things to fix.

    Also, the thumbnails for all the videos on blip.tv are on my laptop which isn't very helpful. I need to move those and update the videos in pyvideo.org.

    As a side note, if we didn't have hosting from Rackspace, we'd have been totally screwed. Thank you, Jesse Noller and Rackspace!

  4. I've been working on the richard codebase fixing architectural problems, reducing the complexities and trying to clean it up so it's in a better state. That work is almost done. When it is, I'll update pyvideo.org with the new site. At this rate, I think I can finish the work this year, but that assumes there aren't any more emergencies.

  5. I've been thinking about how to build a better communication channel for pyvideo.org so people can more easily follow what's going on so they can act on things they're interested in.

    pyvideo.org has a "site news" section. It's a pain in the ass to use and it's not syndicated anywhere and it's likely no one sees it.

    Blogging status reports like this on my blog is better, but I don't think my blog is very widely read. Making my blog more widely-read seems like a lot of work and I'm not sure I can do it effectively anyhow.

    So I've decided to ditch the "site news" section of pyvideo.org and switch to Twitter. I started a @PyvideoOrg account.

    I'll tweet site updates, calls for help and newly posted conferences. I'm tossing around tweeting new videos when they get posted, but videos tend to get posted in huge batches and getting > 40 tweets all at once is a total drag. I'll have to think about that some more.

    Follow @PyvideoOrg if you're interested! Also, feel free to tweet at that account.

    I need to update the site to reflect this.

Also, in my life things are pretty crazy. I have a new kid and juggling everything was impossible for a while. I think that should easy up now and I can spend more time on pyvideo.org going forward.

That's the state of things!

Also, thank you thank you thank you thank you Rackspace!

Dennis v0.3.10 released! Fixes, status subcommand and Zombie!

, | Tweet this

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files
  • a statuser for seeing the high-level status of your .po files
  • a translator for strings .po files

v0.3.10 released!

v0.3.8 fixed mismatched errors in plural strings. Thanks Mike!

v0.3.9 fixed two false positives in error detection.

v0.3.10 adds the status subcommand and the Zombie transform which, like the dubstep transform, is silly but fun.

/images/sumo_zombie1.thumbnail.png

SUMO ... in Zombie!

45 out of 47 Djangonauts use the Zombie transform to make their site accessible to those who have departed. This could open up your app to millions of new users. Truth.

Dennis v0.3.7 released! Dubstep and Django!

, | Tweet this

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files
  • a translator for strings .po files

v0.3.7 released!

v0.3.6 fixed a goof where the linter was skipping errors. Oops.

v0.3.7 adds a dubstep translator (which is just plain silly, but awesome).

/images/sumo_dubstep1.thumbnail.png

SUMO ... in Dubstep!

Truth: 9 out of 10 experts agree SUMO is extra helpful in dubstep.

v0.3.7 also adds Django command shims to make it easier to use Dennis in your Django project.

Use these instructions to set up Dennis so you can use its commands with ./manage.py.

If you aren't using Dennis, yet, it's worth taking a look at. l10n tools are the best!

Dennis v0.3.5 released!

, | Tweet this

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a translator for strings .po files
  • a linter for finding problems in strings in .po files

v0.3.5 released!

0.3.4 fixed an issue with the linter so it skips fuzzy strings.

0.3.5 fixes the rules default for the linter so that it includes the malformed lint rules. It also adds detection of formatting tokens like {0] where it doesn't end in a curly brace. This kicks up a ValueError in Python:

>>> '{0]'.format(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: unmatched '{' in format
>>>

If you're using Dennis---especially to detect errors in .po files before you push them to production---you should upgrade.

Talk like a pirate day!

Tomorrow, September 19th, is Talk like a Pirate Day. Dennis can help you celebrate with its built-in Pirate translator which works on .po files, but also works on any input from command line arguments or stdin.

Translate your HTML pages:

(cat < "$1" | dennis-cmd translate --pipeline=html,pirate -) > "pirate_$1"

Translate all your git commit messages with this hooks/commit-msg:

#!/bin/bash

# Pipe the contents of the commit message file through dennis to
# a temp file, then copy it back.
(cat < $1 | dennis-cmd translate - > $1.tmp) && mv $1.tmp $1

# We always exit 0 even if the dennis-cmd fails. If the dennis-cmd
# fails, you get your original commit message. No one likes it when
# shenanigans break your stuff for realz.
exit 0;

If you forget about this blog post, these two recipes are in the recipes section of the documentation. If you have other recipes, I'd love to hear about them!

Also, the Pirate! translator can always be improved. If there are improvements you want to make, please submit a pull request!

ElasticUtils v0.8 and v0.8.1 released!

, | Tweet this

What is it?

ElasticUtils is a Python library for building and executing Elasticsearch searches.

v0.8 and v0.8.1 released!

I missed the announcement for v0.8, so I'll cover both v0.8 and v0.8.1 here.

Roughly:

  • ElasticUtils now requires at least pyelasticsearch 0.6
  • adds range query and filter
  • adds S.filter_raw
  • changes the Indexable.index arguments dropping force_insert and picking up overwrite_existing

For the complete list of what's new, What's new in Version 0.8.1

Many thanks to everyone who helped out: Jannis Leidel, Rob Hudson and Grégoire Vigneron.

If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.

Switching to South

, | Tweet this

tl;dr

We just landed the bits that switch us from Schematic---the migration system we were using---to South. This is my account of that journey in case it helps others.

Context

Kitsune is the Django project that runs Mozilla Support. The project was started many years ago. For as long as I've worked on Kitsune, we used a migration system called Schematic.

Schematic has the nicety of being very very raw. You can do anything: raw SQL, raw shell, raw Python, raw fish---whatevs. This was nice because we could write whatever we wanted.

Schematic is a total pain in the ass because it hasn't been touched in 2 years, doesn't work well with recent versions of MySQL or MariaDB and makes it really difficult to write migrations that continue to work over time. Further, there's no way to do backwards migrations in Schematic even if you wanted to. We were constantly getting bit by these issues.

The switch

Switching from Schematic to South turned out to be pretty easy. I did it Monday afternoon. I essentially followed these steps:

  1. Added South as a dependency for our Django project.

  2. Initialized South migrations for all the apps we use in Kitsune which was a whole bunch of:

    $ ./manage.py schemamigration <appname> --initial
    
  3. Wrote a last Schematic migration that adds all the South bookkeeping which entailed dumping the output of this to a file:

    $ mysqldump <database> south_migrationhistory
    

    and then editing that file by hand.

    That creates the south_migrationhistory table and populates it with the bookkeeping for the initial commits for the apps initialized in step 2.

  4. Added ./manage.py migrate to our deploy script.

  5. Do a happy dance!

The relevant commits for Kitsune are here:

That's pretty much it.

Update: September 11, 2013
This was with Django 1.4.7 and South 0.8.2. If you're using different versions, you may experience different things.
Update: September 13, 2013

In Schematic, one thing we would do after creating new models is add a content type and the permissions.

This walks through setting that up automatically with South and post_migrate:

http://devwithpassion.com/felipe/south-django-permissions/

Dennis v0.3.3 released!

, | Tweet this

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a translator for strings .po files
  • a linter for finding problems in strings in .po files

v0.3.3 released!

This is the first blog-post-announced release. I think Dennis is good enough for wider use. I've been using it for development work on both kitsune (which drives Support) and fjord (which drives Input with great success.

Why Dennis?

It fills two basic needs I had:

  1. translate .po files so I can find problems during development related to localized strings, layout issues, unicode support, etc
  2. lint translated .po files so that errors in translated strings don't make it to production where they cause fires, make users angry and make me very sad and tired

There's another project called Translate Toolkit that you could use for item 1, but it doesn't have Pirate! and I like my pipeline architecture since it's more "pluggable" (whatever that means). Plus it didn't have a linter that covered my specific issues nor does it return a non-zero exit status so I can't use it for selective compiling.

Therefore I decided to write my own tool to meet my needs.

The ultra-basics

Install

$ pip install dennis
$ pip install blessings  # Optional for prettier output

Linting

Lint a single .po file for problems including mismatched/malformed Python variables in translated strings:

$ dennis-cmd lint locale/fr/LC_MESSAGES/messages.po

Produces output like this:

(dennis) saturn ~/mozilla/fjord> dennis-cmd lint locale/fr/LC_MES
SAGES/messages.po
dennis-cmd version 0.3.4.dev
>>> Working on: /home/willkg/mozilla/fjord/locale/fr/LC_MESSAGES/
messages.po
Error: mismatched: invalid variables: {count}
msgid: Most Recent Message
msgstr[0]: Les {count} derniers messages

Error: mismatched: invalid variables: {count}
msgid: Most Recent Message
msgid_plural: Last %(count)s Messages
msgstr[1]: Les {count} derniers messages

Warning: mismatched: missing variables: %(count)s
msgid: Most Recent Message
msgid_plural: Last %(count)s Messages
msgstr[1]: Les {count} derniers messages

Error: mismatched: invalid variables: {count}
msgid: {0} similar messages
msgstr: Les {count} derniers messages

Warning: mismatched: missing variables: {0}
msgid: {0} similar messages
msgstr: Les {count} derniers messages

Totals
  Warnings:     2
  Errors:       3

If you have blessings installed, it'll colorize that output.

You can also lint a directory structure of .po files:

$ dennis-cmd lint --errorsonly locale/

I use this to compile only the error-free .po files to .mo files and tell us which .po files have problems so we can fix them.

Translating

You can translate a .po file in place into Pirate! to help find l10n issues in your code:

$ dennis-cmd translate --pipeline=html,pirate \
    locale/xx/LC_MESSAGES/messages.po

This takes into account that the strings have HTML in them that should be ignored when translating. It uses a pipeline architecture where the output of one transform is fed as input to the next, so you can string them along and get shouty extra-pirate with anglequotes:

$ dennis-cmd translate --pipeline=html,pirate,pirate,shouty,anglequote \
    locale/xx/LC_MESSAGES/messages.po

Summary

That's the gist of it. In the Dennis documentation is a list of Dennis recipes covering linting, translating, etc.

Yay for Dennis!

Switched Geeksphone to nightly channel

, | Tweet this

Got my Geeksphone Peak yesterday in the mail, but hadn't had a chance to open the package until today (crazy life stuff).

I'm running Debian Testing and had done some Gaia/FirefoxOS work in late 2012. I had my development environment still set up including adb and fastboot.

I wanted to switch the phone to the nightly channel which is what I was using (or thought I was using) with my Unagi. Mostly I followed Updating and Tweaking your Firefox OS Developer Preview phone/Geeksphone.

After doing that, I moved my SIM card over as well as the Micro SD card I had in my Unagi phone. The Micro SD card has all my media on it---not my suer data. I thought about moving my user data over, but I've had the Unagi so long that I figured my user data is probably in a funky state anyhow and better to start over.

Couple of thoughts:

  1. The nightly build is version 2.0.0-prerelease.
  2. The phone is a lot bigger than the Unagi---it'll take a bit of getting used to.
  3. Most things worked fine. I even got a marketing message from T-Mobile minutes after I inserted my SIM card and got on the network.

There are a few issues (the camera is not working at all). I need to spend some time with Bugzilla to file those with useful data. Regardless, the phone is working pretty well and I'm probably going to switch to it as my primary phone soon. That might give me some incentive to finish rewriting magic10ball.