Input: changed query syntax across the site

, | Tweet this

Better search syntax is here!

Yesterday I landed the changes for bug 986589 which affects all the search boxes and search feeds on Input. Now they use the Elasticsearch simple-query-search query instead of the hand-rolled query parser I wrote.

This was only made possible in the last month after we were updated from Elasticsearch 0.20.6 (or whatever it was) to 0.90.10.

Tell me more about this ... syntax.

I'm pretty psyched! It's pretty much the minimum required syntax for useful searching. It's kind of lame it took a year to get to this point, but so it goes.

To quote the Elasticsearch 0.90 documentation:

+  signifies AND operation
|  signifies OR operation
-  negates a single token
"  wraps a number of tokens to signify a phrase for searching
*  at the end of a term signifies a prefix query
( and )  signify precedence

Negation and prefix were the two operators my hand-rolled query parser didn't have.

What does this mean for you?

It means that you need to use the new syntax for searches on the dashboard and other parts of the site.

Further, this affects feeds, so if you're using the Atom feed, you'll probably need to update the search query there, too.

Also, we added a ? next to search boxes which links to a wiki page that documents the syntax with examples. It's a wiki page, so if the documentation is subpar or it's missing examples, feel free to let me know or fix it yourself.

Dennis v0.4 released! Tweaks to Python 3 support, overhauled linter, string-by-string lint rules ignoring

, | Tweet this

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files like invalid Python variable syntax which leads to exceptions
  • a statuser for seeing the high-level translation/error status of your .po files
  • a translator for strings in your .po files to make development easier

v0.4 released!

v0.4 sports an overhauled linter. Instead of two rules ("malformed" and whatever the other one was), it now has a bunch of much smaller and more specific rules! Also, I renamed the rules so they are all numbered!

See the table of error/warning rules and their numbers here: http://dennis.readthedocs.org/en/v0.4/linting.html#warnings-and-errors

Additionally, dennis hits false positives for a variety of reasons. If you're doing a "keep the errors out of production!" kind of thing, then false positives can prevent locale files from making it. That sucks!

To alleviate this, dennis now allows you to tell it what to ignore in the extracted comments. What's an extracted comment? It's a comment in the .po file that starts with #.. You can specify the extracted comments with "context" or similar mechanisms depending on how you're extracting strings. You can tell dennis to skip specific rules or skip all the rules on a string-by-string basis.

Ignore everything:

#. dennis-ignore: *
msgid "German makes up 10% of our visitor base"
msgstr "A német a látogatóbázisunk 10%-át teszi ki"

Ignore specific rules (comma-separated):

#. dennis-ignore: E101,E102,E103
msgid "German makes up 10% of our visitor base"
msgstr "A német a látogatóbázisunk 10%-át teszi ki"

Ignore everything, but note the beginning of the line is ignored by dennis so you can tell localizers to ignore the ignore thing:

#. localizers--ignore this comment. dennis-ignore: *
msgid "German makes up 10% of our visitor base"
msgstr "A német a látogatóbázisunk 10%-át teszi ki"

I also tweaked some of the Python 3 support code because it looked at me funny.

Also, universal wheel!

For more specifics on this release, see here: http://dennis.readthedocs.org/en/v0.4/changelog.html#version-0-4-may-1st-2014

Documentation and quickstart here: http://dennis.readthedocs.org/en/v0.4/

Source code and issue tracker here: https://github.com/willkg/dennis

2 out of 10 people saw the Pirate translation on The Web We Want (Mozilla). Arrr!

Dennis v0.3.11 released! Fixes and Python 3 support

, | Tweet this

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files like invalid Python variable syntax which leads to exceptions
  • a statuser for seeing the high-level translation/error status of your .po files
  • a translator for strings in your .po files to make development easier

v0.3.11 released!

v0.3.11 adds Python 3 support (there might be problems, but it's working for me) and adds error detection for the case where there's a } but no {.

Definitely worth updating!

8 out of 11 people who have heard of Dennis and continue to ignore its baby mews of wonderfulness also have a severe allergy to rainbows and kittens.

Django Eadred v0.3 released! Django app for generating sample data.

, | Tweet this

Django Eadred gives you some scaffolding for generating sample data to make it easier for new contributors to get up and running quickly, bootstrapping required database data, and generating large amounts of random data for testing graphs and things like that.

The v0.3 release is a small one, but good:

  • Added support for Python 3.3 and later (Thank you Trey Hunner!)
  • Fixed test infrastructure and added tox support.
  • Fixed documentation.

There are no backwards-compatability problems with previous versions.

To update, do:

pip install -U eadred

pyvideo status: April 9th, 2014

, | Tweet this

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

I fixed a few issues and finally (finally) pushed out major site updates. Some of them are implemented in the worst possible way (e.g. facet filters for the search page), but some of them are great (e.g. Amara subtitle support).

I'm still struggling with a lot of technical debt on the site and a lack of time to really focus on it. That's mostly what's been making fixing the issues, improving the site and adding conferences take so long.

Sheila and I will be at PyCon US and hanging around for sprint days. If anyone is interested in sprinting, we'll be there. Even if we don't get any coding done, figuring out how to solve some of the bigger problems and planning what should be done in the next year would be a huge accomplishment.

If you're at PyCon and see either of us, feel free to give us a piece of your mind in regards to how you use PyVideo and what could be better.

Site development using pagekite

, | Tweet this

Problem

I have this basic problem where I do a lot of web-site work and I need to show people what I've done so far so they can review it and help me make it better or make it suit their needs better. Screenshots aren't very helpful because the site is interactive. Further, the site needs to get tested on multiple devices/platforms/browsers. Also, I need to make sure that the site is only accessed via https.

What I've been doing up to now is failing miserably: I'd push work to our staging server for people to test out, but that sucks as an answer and affects my co-workers and makes a mess of our staging server. Plus iterating on things is difficult.

So, requirements:

  1. endpoint must be https-only
  2. must be easy to set up and take down
  3. must be easy to access so people can easily test things on my local machine

Solution

I looked around and this would be pretty easy to do if I didn't have the https-only requirement. That makes things difficult without a lot of work.

Then I found pagekite. They make it really easy.

Here's how you set it up:

  1. Download and install the pagekite software: http://pagekite.net/downloads/

  2. Run your website. In my case, I'm working on Django sites, so I launch like this:

    $ ./manage.py runserver
    

    That runs the Django project I'm working on on localhost:8000.

  3. Run pagekite:

    $ pagekite.py 8000 YOUR_NAME.pagekite.me:443
    

    That creates a tunnel from your machine to the pagekite.me server. When someone accesses https://YOUR_NAME.pagekite.me/, the request goes through the tunnel to your pagekite backend and that performs the request over http to your local webserver (in my case, the Django project) bound to localhost:8000.

    Access is https-only. If anyone tries to access http://YOUR_NAME.pagekite.me/, then they get a connection error.

    The https-only requirement is satisfied by restricting the kite to only listening to port 443--the https port. That's pretty key.

This lets me run my Django project locally on http without dealing with self-signed certificates, but still require https access so data isn't floating around in clear text.

The one problem with this is that my local server thinks it's running http and so redirects that include the protocol go to http rather than https.

If you don't already have an account, I'm pretty sure step 3 will walk you through setting one up. Free accounts are limited in what they can do.

Also, they hang out on #pagekite on Freenode. I had a problem, asked a question and got a super helpful reply. The code is Open Source, so it's possible to look through it and debug it.

I'll be using this going forward.

Why write this?

This is a common use case for web developers. I figured I'd write this up because the https-only part is pretty key and it was the part that I had to ask for help with.

ElasticUtils v0.9 released!

, | Tweet this

What is it?

ElasticUtils is a Python library for building and executing Elasticsearch searches.

See the Quickstart for more details.

v0.9 released!

This is a big release, but there are some compromises in it that I'm not wildly excited about. Things like Elasticsearch 1.0 support didn't make the cut. I'm really sorry about that---we're working on it.

This release has a lot of changes in it. Roughly:

  • dropped pyelasticsearch for elasticsearch-py (Thank you Honza!)
  • fixed S.all() so it does what Django does which should let you use an S in the place of a QuerySet in some cases
  • new FacetResult class (Thank you James!)
  • S.facet() can take a size keyword
  • cleaned up ESTestCase
  • SearchResults now has facet data in the facets property
  • etc.

For the complete list of what's new, What's new in Version 0.9.

Many thanks to everyone who helped out: Alexey Kotlyarov, David Lundgren, Honza Král, James Reynolds, Jannis Leidel, Juan Ignacio Catalano, Kevin Stone, Mathieu Pillard, Mihnea Dobrescu-Balaur, nearlyfreeapps, Ricky Cook, Rob Hudson, William Tisäter and Will Kahn-Greene.

We're going to be sprinting on ElasticUtils 0.10 at PyCon US in Montreal mid April. If you're interested, come find me!

If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.

pyvideo status: February 15th, 2014

, | Tweet this

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

Over the last year, a number of things have led to a tangled mess of tasks that need to be done that were blocked on other tasks that were complicated by the fact that I had half-done a bunch of things. I've been chipping away at various aspects of things, but most of them were blocked on me finishing infrastructure changes I started in November when we moved everything to Rackspace.

I finally got my local pyvideo environment working and a staging environment working. I finally sorted out my postgres issues, so I've got backups and restores working (yes--I test restores). I finally fixed all the problems with my deploy script so I can deploy when I want to and can do it reliably.

Now that I've got all that working, I pushed changes to the footer recognizing that Sheila and I are co-adminning (and have been for some time) and that Rackspace is graciously hosting pyvideo.

In the queue of things to do:

  • finish up some changes to richard and then update pyvideo to the latest richard
  • re-encode all the .flv files I have from blip.tv into something more HTML5-palatable (I could use help with this--my encoding-fu sucks)
  • fix other blip.tv metadata fallout--for example most of the PyGotham videos have terrible metadata (my fault)
  • continue working on process and tools to make pyvideo easier to contribute to

That about covers it for this status report.

Questions, comments, thoughts, etc--send me email or twart me at @PyvideoOrg or or @willcage.

Blog update: December 31st, 2013

, | Tweet this

A few weeks ago, a friend of mine started a site called Nethack-a-day. It's fantastic---a Nethack game with color commentary one turn at a time. If you like Nethack, but haven't seen it, you're missing out.

As he was setting it up, he was looking at various systems he could build it with. I sort of wanted to say, "Yo, just use Pyblosxom." because I was pretty familiar with it (I spent the better part of 9 years maintaining it) and I knew it did 80% of what he needed. But I hesitated because I've been on the fence about switching to something else for a while now.

Then I committed a critical mistake. I said, "You know, Pyblosxom would be great for this, but how about I fix a handful of things first that'll make it easier to deal with." A handful of things turned into a massive overhaul of Pyblosxom ripping out a lot of the technical debt that had been accruing for years, re-imagining some of the bits I was never happy with and tweaking some things just because it seemed like a good idea at the time.

Thus was born Douglas.

Douglas resembles many of the static site blog generator systems written in Python that exist. That suggests it was a dumb idea to go and write it, but it has three compelling aspects that I think made it worth my time:

  1. It's derived from a blog system I maintained and thought about for a long time, so it has all the sorts of things I would want in a static blog generator
  2. I can continue to say, "I've been using the same blog system since 2002." Sure, it's not exactly the same system, but it's not like I have to go rewrite/reformat entries I wrote in 2002.
  3. It has a nicer callback system that I think makes it more malleable when it doesn't do exactly what you want by default.

Right now it's in an alpha state: the test suite doesn't cover enough of the software; the docs are mediocre and in some cases are filled with outright lies; there are a handful of issues; and there's still a bunch of technical debt and some architectural decisions that sucked and are increasingly difficult to work around.

Regardless, about a month, 102 commits and 9980 insertions and 22828 deletions later, I'm now switching my blog over to my new system. And that's how I'm going to end 2013.

Me: 2013 retrospective

, | Tweet this

I had a pretty intense 2013, but overall I think it was a good year. I say that with full knowledge that I had 2013 goals, but I have no idea where I wrote them down. Oh, well.

First off, I started or helped start a lot of new projects this year:

  • dennis: lint .po files for strings that will set production on fire and translates .po files to aid development (awesome)
  • ernest: sprint coordination system (awesome)
  • douglas: my complete overhaul of Pyblosxom (in development)
  • tedmund: console based slide presentation system
  • james: cli for deployment with chief
  • eugene: comms system for Artemis simulation
  • victor: system for sussing out version information for dependencies (failed project)
  • fredrik: Flask template (obsolete)
  • harold: rough feedback/support system (prototype)
  • captain shove: the mighty new deployment system we'll switch to in 2014
  • hy: dialect of Lisp that runs in Python vm

Fredrik is obsolete. Victor was a failed project. Captain Shove and Hy are doing well and other than helping to start them off, I haven't touched either in some time. I work on Dennis, Douglas and Ernest regularly. I'm thinking about Harold, but probably won't do anything with it for a while.

I also spent a ton of time working on existing projects:

I'm probably missing a bunch there. It's hard to keep track of what one did on other peoples' projects.

I got a lot accomplished at work this year:

  • I joined the sec-champs group and now help manage and coordinate security releases for Django and other software we use on our websites. (Security)
  • I performed an audit on playdoh-lib and helped coordinate a bunch of updates including to Django 1.5. (Security)
  • I helped work on Captain Shove which is a new deployment system to replace Chief. (IT)
  • I worked on l10n issues, created dennis and rewrote the deployment scripts for Input and SUMO to eliminate production problems related to errors in translated strings. (L10N)
  • I worked on community building as a mentor for SUMO and Input projects (more Input than SUMO) helping guide new contributors towards their first contributions. I should be doing a lot more on this front. One thing I did with Miro, Pyblosxom and other projects I've worked on is to keep track of people who have contributed to help keep them around. That sounds goofy, but it helps build lasting communities. (Community building)
  • I helped fix some quality problems with the Kitsune and Fjord test suites and I'm working with Bob from QA on updating the Input tests. (QA)
  • Implemented a more flexible filter/query system in Elasticutils and maintained it for the year. (Maintainer)
  • Did a ton of work on SUMO and Input.

Challenges I worked on this year:

  • I keep getting dinged in performace reviews for being acerbic (that's probably the nicest word for it). I recognize that it sucks to work with acerbic people and regardless of how busy or stressed out they are.

    I finally started working on it in earnest probably around August. I hope I'm more pleasant to work with. We'll see what people think.

  • Everyone above me in the Mozilla org chart left.

    At the beginning of 2013, it was something like me -> James -> Mike -> Todd -> Gary.

    Now it's me -> Ricky -> Wil -> Rick -> Harvey -> Jay.

    I'm still working on SUMO and Input, but the Webdev group went away and the SUMO engineering team got shifted between groups. I'm feeling ok about it, but wonder if all that shifting makes it difficult for me to be noticed.

  • I'm working on too many things and thus many things are starving for attention. That's a simple sentence for a very complex situation which involves all sorts of thought-provoking questions like, "why?" and "why is it important for all things to be worked on?"

    I stopped maintaining Pyblosxom. I stepped down from my work on MediaGoblin. I'm being more careful about what I commit to and try to commit to things that I feel that I'm the best fit for. I'm working on reducing context-switch time so I'll work on certain projects for longer periods of time, then put them down for longer periods of time. I'm trying to ask for help more often and spread the workload around and I'm trying to make helping me easier.

  • I need to spend more time on pyvideo.

    This is an important project. It just keeps getting back-burnered and when I have time to work on it, crazy ass things come up which need to get fixed immediately.

    For example, blip.tv ditched a bunch of Python conference accounts. We scrambled to download those videos and now we're hosting them. That took a ton of time.

    Regardless, I started several code overhauls that I need to finish. That's blocking everything else and that sucks. I really need to get that done.

  • I have a new roommate and having two roommates is crazy. I don't even know what to do to alleviate this short of just get through the day one day at a time.

  • I need to take better care of myself.

    This is a hard one. It's easy to push off me-focused things when there's so many things that need doing. I haven't a good answer here, yet, but maybe being more aware of it is helpful.

  • I got shingles. That sucked.

In 2014, I want to:

  • Continue to get better at front-end development.
  • Spend time working on pyvideo, richard, dennis and ernest and get them to better places.
  • Figure out what to do with Elasticutils: Overhaul it to make it suck less? Pass it off to someone else to maintain? Figure out a list of 5 things to fix now and let it continue to be mediocre?
  • Overhaul my blog. Douglas is coming along nicely. After I finish getting that working, I'll re-theme it. This is lower priority than the other things here, but it'd be nice to finally do this year.
  • Work on empowering and enabling other people to do things rather than blocking them. This one is hard and tricky especially where my free time is unpredictable on account of roommates and obligations I have no control over.

It's been a busy year and there are things I should be doing better, but generally speaking, I think I did pretty ok.