To do this I had to implement a __call__ method. This works fine, except I wanted to store things into the class -- after all the whole point of my use of classes was to keep the code a bit more compact, and cleaner. So, why pass the api_version around everywhere? Unfortunately thread-safety comes to play, and you need a separate instance of your class for each request.

λ

Django's urlpatterns expects a callable object. So you can't give it an instance of AddonDetailView(). But you could give it a callable that creates an instance of AddonDetailView() and passes it *args and **kwargs. Luckily python has lambda functions. You can note how we solved that in our urlpatterns.

λ λ

But wrapping all your urls with lambda is tedious and remembering to pass *args and **kwargs is error prone.

So let's make a lambda function that returns... a lambda function that turns an instance of our class into a callable.

We can now return to coding and not think about thread safety.

λλλ

When dealing with legacy data, you'll run into all kinds of edge cases. Perhaps, an object might not display correctly unless it has the right parameters, or if it has null parameters it might not display at all. So when testing Django, it's nice to actually use non-dummy data.

Luckily Django has a way of pulling real data out of your database using django.core.serializers:

from addons.models import Addon
a = Addon.objects.get(id=3615)
from django.core.serializers import serialize
jsonize = lambda a: serialize("json", a, indent=4)
jsonize([a])

This solution runs well in a Django shell and can be lots of fun for the whole family... until things get complicated.

Serializing alone isn't enough.

Serializing a fixture with foreign keys means you'll have an un-loadable fixture unless you serialize the dependent fixtures. Even for one or two foreign keys, this can be a pain. For addons.mozilla.org, we have a spidery-web of dependencies: Files need a Version which needs an Addon which need Translations.

Thus begat the dump_object management command. Give it an app, model name and a pk and it will give you not only a serialized JSON of that object, but all the objects that it requires.

Example:

./manage.py dump_object files.file 64874 64876 > my_new_fixture.json

This looks for the File model in the files app and pulls out of the database Files instances with pks of 64874 and 64876. It then recursively searches for any required objects.

Too much serial

If you create a lot of fixtures, you'll eventually have overlapping serialized objects. In addons.mozilla.org we have Addons, Versions (which depend on Addons) and AddonCategorys (which depend on Addons and Categorys). If we wanted to get serialize a specific Addon, it's dependent Versions and AddonCategorys it makes sense to start with dump_objecting the related Version and then dump_objecting the AddonCategory. Both dump_object commands will fetch the Addon in question, resulting in duplicated data.

To combat this we can use merge_fixtures to dedupe our fixtures:

./manage.py dump_object versions.version 64874 > 1.json
./manage.py dump_object categories.addoncategory > 2.json
./manage.py merge_json 1.json 2.json > happy_fixture.json

This should make creating test data slightly less painful. So give it a try.

Before I packed it, I updated to WebOS 1.3.x (a few days later I updated to 1.3.5) and I was blown away. The horsepower was increased by utilizing the GPU. The following problems were fixed:

* The device was no longer hot all the time
* Shutdown and startup were long, but not nearly as long as before.
* Render times were quicker
* All the elements usually rendered quickly in an app
* Network was fairly steady
* Phone calls also seemed fairly drop-free.

All these improvements helped me get over * The tiny keyboard... not so bad in practice. * No soft keyboard - I missed it, but I could deal without it.

Overall the device was great, it was fast enough to use, and most of the errors were annoying, but things I could deal with. Cut and paste could be improved, and I wish the USB connector was the same as the one for HTC devices (I can't keep micro or mini USB types straight).

So I love the device, and Murphy's Law dictates if work gives you a phone you don't like you get to keep it... until you start liking it again. So I sent the phone back into rotation for other people at Mozilla to try. Have at it.

I went to Add-on-Con some weeks back to represent my employer, the Mozilla Corporation.

One of the goodies you get as a registrant was a jigsaw puzzle from the Google Chrome Extensions team.

Perfect, my wife and I love solving jigsaw puzzles. We finally finished a few days ago. Anybody who has started at all will realize the puzzle is of a QR-code. The QR-code is an extension that will eventually lead you to a prize. It was a bit of a mini-puzzle not nearly as difficult as finding the QR code.

Although finding a QR code scanner was a bit difficult, I had to borrow a HTC Magic from Justin Scott and installed a decent barcode scanner.

Wil was open to Django and knew that's what we in the dev team wanted. Jeff spawned our foray into a new AMO with Zamboni. I've been working on some grunt-work tasks inside and outside of Django.

One of those tasks is building a transparent layer in Django to keep users logged in from our PHP-based site. That kind of problem almost immediately forces you to ask one of the most fundamental questions you ask when using any framework:

How much do I change my app, in order to accommodate the framework?

More specifically:

Should I use the django.contrib.auth User module, and to what extent?

The more we looked into what features of Django we might want to use, django.contrib.auth was heavily tied into other things we wanted, so it made sense for us to use it. The next question is whether we try the inheritance approach or do we treat our legacy users table as a sort of User Profile and utilize the User module using the related table approach?

Using model-inheritance seems real nice, because we can pretend that our legacy user is the same thing as a djaango.contrib.auth User - but this isn't true:

Looking at our users table more closely:

mysql> explain users;
+-------------------------+---------------------+------+-----+---------------------+----------------+
| Field                   | Type                | Null | Key | Default             | Extra          |
+-------------------------+---------------------+------+-----+---------------------+----------------+
| id                      | int(11) unsigned    | NO   | PRI | NULL                | auto_increment |
| email                   | varchar(255)        | YES  | UNI | NULL                |                |
| password                | varchar(255)        | NO   |     |                     |                |
| firstname               | varchar(255)        | NO   |     |                     |                |
| lastname                | varchar(255)        | NO   |     |                     |                |
| nickname                | varchar(255)        | YES  | MUL | NULL                |                |
| bio                     | int(11) unsigned    | YES  | MUL | NULL                |                |
| emailhidden             | tinyint(1) unsigned | NO   |     | 0                   |                |
| sandboxshown            | tinyint(1) unsigned | NO   |     | 0                   |                |
| homepage                | varchar(255)        | YES  |     | NULL                |                |
| display_collections     | tinyint(1) unsigned | NO   |     | 0                   |                |
| display_collections_fav | tinyint(1) unsigned | NO   |     | 0                   |                |
| confirmationcode        | varchar(255)        | NO   |     |                     |                |
| resetcode               | varchar(255)        | NO   |     |                     |                |
| resetcode_expires       | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notifycompat            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| notifyevents            | tinyint(1) unsigned | NO   | MUL | 1                   |                |
| deleted                 | tinyint(1)          | YES  |     | 0                   |                |
| created                 | datetime            | NO   | MUL | 0000-00-00 00:00:00 |                |
| modified                | datetime            | NO   |     | 0000-00-00 00:00:00 |                |
| notes                   | text                | YES  |     | NULL                |                |
| location                | varchar(255)        | NO   |     |                     |                |
| occupation              | varchar(255)        | NO   |     |                     |                |
| picture_type            | varchar(25)         | NO   |     |                     |                |
| averagerating           | varchar(255)        | YES  |     | NULL                |                |
+-------------------------+---------------------+------+-----+---------------------+----------------+

You can very easily argue that this is a profile table, which happens to have credential information thrown in.

I can see overtime, I'll just struggle to keep our legacy User to act like a Django User, whereas a UserProfile is fairly standard.

Had I been writing this app from scratch, I would have chosen the UserProfile route. This is extra data which takes up a lot of space, and changes far more often than user credentials. Changing 4M+ rows sucks, by making users our UserProfile table, any changes to that table, don't tie up the table used for sign-ins.

I'm curious what other people who port their apps to Django have done.

Immediately upon using the Pre I figured out why. In short, it's a crappy phone. It makes a very good attempt to do a lot, but it does them with such piss-poor performance, that nothing good is noticed.

I am disappointed. It's not even in the same class as an iPhone - maybe a future generation of Palm devices will be, but not this one. I was hoping WebOS would be a good alternative to the iPhone. It looks like Google will be doing that, though their phones haven't impressed me much either. I am hoping that maybe this phone is just a dud.

Here's what I didn't like: * The Palm was always hot. * The first run experience is painfully slow. * The first run was an indicator of things to come, startup and shutdown are ridiculously slow. * Every application is slow to render. * Not all elements of an app render. * The keys are too small. Some people aren't migrating from a Treo and aren't used to mini keys. * No soft keyboard. * The palm website doesn't let you use plus-style addressing * Media Mode was not self explanatory - and forced the phone to not work. * Network would constantly drop out. Couldn't use a lot of the data features. * Phone calls didn't work so great. * Did I mention it was ass slow, even the dialing program was slow. * The battery dies quickly * I could only cut/paste when composing, but I couldn't cut a string of text from an email. * Felt too much like an old palm

Despite the sadness there were a few good things:

• When it did fetch email, and other notices, it displayed them nicely
• The unification of Facebook and Gmail was pretty cool - it also made me want to trim some of those friends from highschool off my facebook - I ain't ever gonna call em.
• The Icons were pretty.
• The card interface was interesting.
• The travel charger could be modified to work in non US chargers fairly easily.

All in all, I'm glad that I had a chance to try out this device. It showed me, that user interfaces above all need to be very fast and responsive. Furthermore, everything you try to do should be done exceptionally well. I'm hopeful that software updates can alleviate some of the problem, but I think the root of the problem is slow hardware.

First, I'd like to thank Harper and Threadless. It was a great location in the greatest city in the universe. Before and after the meetup, Harper was just an all-around great guy to hang with and the threadless headquarters was a nice hangout place for meeting people interested in addons.

Shortly after my talk, our Engineering Ops team deployed the new AMO 5.1 complete with a new Sphinx powered search engine.

So let's talk about search. Note: parts of this are a rehash of my talk, so feel free to skip around.

A bit about addons

Addons is a huge growing space. Arguably it's Mozilla's best kept secret. Sure readers of this blog probably know what Addons are, but ask people who aren't as web-savvy. Most people don't know what a browser is - and it's hard to explain it to people without getting technical.

We can just skip that step. Because Addons are small things that people can easily "get".

"It's an easy way to customize the internet when your surfing."

While perhaps not technically correct, its one way of explaining it to people. Maybe a better way is just showing people what they can do with addons.

On my flight out to Chicago, I talked to a person on the plane who didn't know what a browser was, but after showing her AMO she was really intrigued.

If everyday non-technical people can realize the potential of addons, it's only a matter of time before they start knocking down the doors to AMO.

So we better be prepared to handle them, and get them what they want.

The technical details of addons.mozilla.org

Everytime you open Firefox, it pings AMO to see if there's any updates to any of the addons that happen to be installed. Over a third of the people using Firefox have at least one addon, and Firefox is roughly 22% of the browser market. That means roughly 7% of people opening their browsers are pinging our servers for updates.

Needless to say it's a lot of traffic, and to support it we need a fair amount of hardware. AMO is clearly the largest site in the Mozilla universe in both respects.

Some stats:

• 1 mySQL master
• 4 mySQL slaves
• 2 memached servers
• 2 Sphinx indexer/search daemons
• 24 Web Frontend
• Multiple Zeus ZXTM clusters all

Most of this is standard, we'll talk about Sphinx later, but Zeus is amazing. I didn't know what Zeus was until earlier this year when I interviewed with Mozilla's VP of Engineering Operations. All our requests get cached so much of our hits actually hit our Zeus cluster and not our web servers.

To see just how amazing they are read our mrz's ops blog.

Why search matters

If you have any kind of custom content and unique meta data a custom search solution is a must. Browsing through a site isn't going to cut it. Browsing is dead. Search is how you find things on a web site. On AMO you may see an addon that's featured somewhere, or you might want to see what's out there, but the right search query will find you the right addon in two clicks.

Improve Search

So my first job on AMO was to improve addons search. It was a vague request and born out of frustration with what we had. It wasn't a problem that certain things were indexed, or unicode didn't work, or results weren't sorted. We may have had all those problems, but as a product search needed to be replaced.

To me it meant that we needed some framework that would allow developers to quickly debug and fix any future search calamities at a moments notice.

So here were the goals I made for myself:

• Do something that sucks less than what we’ve got
• Do something that makes it easier to suck less in the future
• Do something that’s easy to use for our operations team, web developers and most importantly, end-users
• Reduce strain on our databases, developers and operations teams

Complex Data

Our data set is small (we have 5,000 addons), but there's a lot of secondary meta data about the addons that we track:

• Addons work in 1 or more locales (e.g. en-US, fr, de, etc)
• Addons are optionally platform specific (Linux, OS X, etc)
• Addons work with one or more products (Firefox, Thunderbird, Seamonkey, Sunbird or Fennec)
• Addons come in multiple flavors (extensions, themes, dictionaries and more)

We want to index all this data. Unfortunately to get at much of this data it involves either numerous queries, or numerous joins which put a strain on mysql. How much strain?

At peak we get about 10 search queries per second. If we do something smarter this won't have to cause a lot of strain.

Using Sphinx

Sphinx is an open source search indexer and daemon. It's used by Craigslist, the Pirate Bay and Mozilla Support. It was very easy to use and despite a complicated set of data and business logic, Sphinx was up to the task.

The challenges

We needed to search for addons in several languages. So indexing just addons wouldn't work, we need to make sure we have every translation of every addon indexed. For those counting, we have 5,000 addons, but 18,000 translations of addons.

All the joining and filtering that needed to be done for our old search still needs to be done, but we can do this all in one shot by using a mysql view. This view is a flat list of each translated addon as well as all meta data associated with it. This then gets fed into the sphinx indexer.

Along the way we ran into some issues which used to be dealt with outside of mysql, such as comparing versions. It was gross and quite a hack, so we turned the variety of acceptable version strings into integers.

We also learned that stemming wasn't a good idea as we assumed it would be. Stemming was great for searching through lots of text, but a great deal of addon searches were really just searches for product names, so we opted for substring searches. We'll see how that fares. There is probably room for improvement.

Much of this, however involved knowing our data, and knowing how it will be used by our users. Once we got that down, we could hammer it all out using Sphinx.

Wins

So Sphinx gains us a bit architecturally. We have a complicated query, but it only gets run once every 5 minutes versus the 180,000 times it was run "on demand."

Indexing happens rather quickly, just over a minute.

The API was a breeze to work with, and was easy to drop into our own codebase.

Because of our relatively small data set, and quick indexing, we're able to scale this simply by cloning and load balancing. Meaning, we just need to scale for traffic, but addon growth (which is slower than traffic growth) we can safely not worry about for a while.

Our ops team can monitor the sphinx clusters and just deploy additional nodes as needed.

Building a platform

What we've done is built a foundation for search. Not all the problems are gone, but a lot of the problems that our QA team finds are able to be resolved quickly. We have a nice pile of unit tests as well that help us keep our results in check when we start tweaking dials.

We even have the groundwork for some nifty advanced search syntax, that hopefully we can inject into future releases of AMO.

Enjoy. And if you find anything, let me know.

So my Plan B was to get a list of common smushed named addons (I'd say camelCase, but camelCase is different from SmushedText). Naturally I used my exceptional skill at regular expressions to concoct this query:

mysql> SELECT name FROM translated_addons WHERE name REGEXP '[a-z][A-Z][a-z]' = 1 LIMIT 10;
+------------------------+
| name                   |
+------------------------+
| Orbit Grey             | 
| Phoenity               | 
| Pinball                | 
| Qute                   | 
| FirefoxModern          | 
| Adblock                | 
| Add Bookmark Here      | 
| All-in-One Gestures    | 
| Bookmarks Synchronizer | 
| Browser Uptime         | 
+------------------------+
10 rows in set (41.28 sec)

Wait... none of these match. I scratched my head for a bit and then thought, oh wait, mysql is case insenstivie maybe it's turning [a-z][A-Z][a-z] into [a-z][a-z][a-z] ― stupid, but consistent with mysql. Then I pulled my other regexp card out of my sleve, character classes:

mysql> SELECT name FROM translated_addons WHERE name REGEXP '[[:lower:]][[:upper:]][[:lower:]]' = 1 LIMIT 10;
+------------------------+
| name                   |
+------------------------+
| Orbit Grey             |
| Phoenity               |
| Pinball                |
| Qute                   |
| FirefoxModern          |
| Adblock                |
| Add Bookmark Here      |
| All-in-One Gestures    |
| Bookmarks Synchronizer |
| Browser Uptime         |
+------------------------+
10 rows in set (12.96 sec)

No difference. Time to pull out the mysql documentation:

REGEXP is not case sensitive, except when used with binary strings.

ORLY?

Case-insenstive regular expressions when looking for [[:upper:]] or [[:lower:]]? Fine... I'll add some syntax to make you work right:

mysql> SELECT DISTINCT name FROM translated_addons WHERE name REGEXP BINARY '[[:lower:]][[:upper:]][[:lower:]]' = 1 LIMIT 10;
+---------------------------+
| name                      |
+---------------------------+
| FirefoxModern             |
| ChatZilla                 |
| ChromEdit                 |
| CuteMenus                 |
| DownloadWith              |
| easyGestures              |
| JavaScript Console Status |
| LinkVisitor               |
| OpenBook                  |
| QuickNote                 |
+---------------------------+
10 rows in set (9.68 sec)

That's more like it!

Unfortunately there's about 2609 addons matching this query and since I can't automatically fix these in mysql, I'll need to do some work:

1.  Create a new table for additional indexable data.
2.  Upon creation of any new addons with names that have SmushedText - store the "un smushed text".
3.  Index this "extras" field in Sphinx.

Bug: 517699

[root@ml-db10 sun]# pv -cN source < addons_remora.2009.09.15.sql.gz | gunzip|pv -cN gunzip > addons_remora.2009.09.15.sql
   gunzip: 10.1GB 0:06:48 [25.5MB/s] [   <=>                                  ]
   source: 3.47GB 0:06:48 [8.72MB/s] [======================>] 100%

Here we are calling pipe viewer with an argument that says to title this progress meter as source, and feeding it the gzip'd file. Pipe viewer will output two things the progress, and the actual file. We pipe that file into gunzip to unzip it, and back into another instance of pipe viewer (again with a title, of gunzip) and the standard output gets redirected to our destination file.

Now a simpler example is checking the progress of loading a large sql file into mysql:

[root@ml-db10 sun]# pv -cN sql < addons_remora.2009.09.15.sql | mysql -uroot addons_remora -p$PWD
      sql: 2.55GB 0:18:19 [5.68MB/s] [=====>                  ] 25% ETA 0:54:30

We could have probably combined all this, however:

[root@ml-db10 sun]# pv -cN source < addons_remora.2009.09.15.sql.gz | gunzip|pv -cN gunzip | mysql -u root addons_remora -p$PWD

Armed with this knowledge you can determine whether to grab a soda, a sandwich or a 2-hour lunch.

Here's a collection of notes I collected through the conference.

Day one

Keynote - Avi Bryant

Frameworks lock us into RDBMS = bad

This keynote mentioned the limits of modern frameworks and modern web development. Essentially frameworks are great for getting started, but as a site grows, the framework gets replaced little by little. Sometimes it can get in the way - such as with limitation of database choices.

UR doing it wrong - James Bennet

James outlined a few key problems that many Django developers run into:

• learning python as you go

  • doesn't work unless you know some programming upfront
  • do the python tutorial
  • read python in a nutshell or dive into python

• Things you should know:

  • subclasses
  • super()
  • slides went too fast... hopefully they'll be posted

All in all RTFM for python and Django

Learn about other py packages... like twisted. If Twisted Matrix was implemented in Ruby it would be advertised as the second coming of Christ.

Bennet's Django App review smoketests:

• installable via pip, easy_install or setup.py
  • read distutils-guide
  • stay away from setuptools
• have a README
• INSTALL file list deps
• Write DOCUMENTATION
  • use sphinx.pocoo.org
  • store it in your package and upload package docs
• LICENSE (most Django apps use BSD)
• Write unit tests
• django-lint - to look over code (like pep8.py)

pro-django is a decent book, but not written by Bennet.

Testing - Eric Holscher

• Django 1.1 encourages you to test by auto-creating tests.py.
• Support for:
  • Unittests
  • Doctest
  • Tests done in a db transacation
• Test Driven Documentation (TDD + DDD)
• Doctest
  • easy
  • can't use PDB
  • Hides certain failures
• Unittests via Django TestCase
  • XUnit
  • setup/Teardown
  • adds db fixtures
  • assertions
  • mail testing/inbox testing
  • url testing
• TestCase
  • Browserless Request/Response testing
  • Similar to sfBrowser in symfony
• Google Summer of Code (for Django 1.2)
  • Coverage reports!
• I need to learn PDB

Deploying Django -

Run mod_wsgi in daemon mode.

Day 2

Keynote - Ian Bicking

GNU Manifest:

I consider that the golden rule requires that if I like a program I must share it with other people who like it. Software sellers want to divide the users and conquer them, making each user agree not to share with others. I refuse to break solidarity with other users in this way. I cannot in good conscience sign a nondisclosure agreement or a software license agreement. ...

So that I can continue to use computers without dishonor, I have decided to put together a sufficient body of free software so that I will be able to get along without any software that is not free.

• GNU manifesto was the idea of sharing software amongst friends
• GNU has purpose - BSD, etc is just a rule - free to share
• Free is not just the absense of copyright
• Free is not a reaction to existing rules, but a golden rule
• Not just a fight against MS
• Need to find morality (the why) within the practical (the law, or what you can do)
• Open sourcing closed source code isn't building open source
• This might apply to Mozilla... as webkit has taken off more than Gecko.
• Open source is person to person not company to company - despite sponsorship.

Using Django in Non-standard ways - Eric Florenzano

• Django loosely coupled
• Replace templating with Jinja 2
• Copy Django methods into djangoext to easily customize Django behavior
• Not using django.contrib.auth
  • reasons: writing a fb app - no auth needed
  • no shoehorning needed - saves time - less overhead
• skip the orm?
  • legacy dbs
  • non standard or db (or non-relational database)
  • no database
• wsgi middleware has some cool shit
  • repose.bitblt: autoscales images
  • repose.squeeze: will concat js/css on the fly based on statistical analysis
• non standard Django based apps
  • YARDBird - IRCBot framework
  • djng micro framework
  • Jngo- singlefile cms
• using admin in a nonstandard way is hard/impossible coupled with ORM and auth

Real-time web and other Buzzwords - Chris Wanstrath

• more than just getting your rss feeds faster
• push vs. pull
• 1 persisting connection vs polling
• comet/flash-xml/or html5 web socket
• orbitted - open source python comet server
• zeddicus - does the business logic
• orbitted has its own js libs - its a simple port/socket thing for your server code to deal with - not request/response.
• all connections are persisting browser/orbitted orbitted/zeddicus
• You can even use orbitted to connect straight to IRC and write a client in JS
• Jetty also is good for comet

Also: * see webhooks * see pubsubhubub

Pluggable, Reusable Django Apps: A Use Case and Proposed Solution - Shawn Rider and Nowell Strite

• PBS moved from perl to django - build a lot of reusable apps
• convincing your superiors
  • need a good story -
  • existing base of python helped
  • With Django easy to do things right without doing things slow
  • be really good...
• built a lot of apps to be very reusable, and pluggable based on requirements PBS had

Day 3

Keynote - Ted Leung - Sun

• Django jobs are a growing market
• Preferred by startups
• Bespin/wave - cool
• APIs are big... still
• Physically impossible to create purely server-side interactions that are usable enough - rely on rest/comet/ajax/etc to bridge gap

Scaling Django Mike Malone

• MM from Pownce (now sixapart)
• Slides started out as "Building Scalable Web Applications"
• Django didn't get in the way too much when it came to scaling
• Django had tons of caching support
• Cached objects by hand (memcached) and object ID lists
• Use memache for sessions too
• use signals to signal cache invalidation
• race conditions...
• Queue shit... gearman, rabbit mq, etc.
• Memecached incr/decr operators are awesome
• See gh/mmalone/django-caching
• See gh:.../django-multidb
• to combat slavelag use a memcache key to alternate between master or slave

Gearman - working later - Chris Heisel

• Gearman - a work later alt to rabbit mq
• Makes the most sense for something like cesium, with a bazillion worker bees foxes feeding off a single queue

Also at the con, I talked to someone about rebuilding large apps... and they took a PHP app and used URL rewriting to and a lot of PHP/Python glue code to build a seamless transitory app. The rule is, all new functionality was done up in python while the old app was in maintenance mode.

More talks here!