Oliver's Place » Django

High Performance Media Merging with Django, Nginx and Memcached

oliver — Mon, 16 Feb 2009 00:20:30 +0000

I admit it – I’m a performance junkie. I can’t stand code that just works but performs poorly. Being said, I recently fell in love with Django (a fantastic Python powered web application framework). In one of my current projects – xarmory.com – the number of requests for static resources issued during page load began to bother me. The project makes intensive use of jQuery and it’s my personal belief that Django + jQuery is a match made in heaven. When working with jQuery you will find yourself often in the situation to rely in cool little jQuery plugins – each distributed as a seperate javascript file of course. When a project grows, those files begin to add up. In the case of xarmory.com we now had eleven javascript references in the header section of the page. Although the individual files are were only 3-11k in Size, the request overhead for all those tiny files became unacceptably in my eyes.

To alleviate the problem I decided to resort to a simple solution and just merge the individual script files into a single file. A pretty common practice. Before reinventing the wheel I surfed the net for existing solutions and found two which I gave a shot. Short story, both solutions had their share of problems mostly stemming from the fact that they wanted to do it all and integrate their own Javascript Minifier in addition to the merging. During my short evaluation both Minifiers choked on fifty percent of my javascript files, even on the official jquery 1.2.6 script. A bit frustrated I decided to roll out my own Django Template Tag based on this code.

I can be used like this:

^?View Code PYTHON

{% load mediamerge %}
 
{% mediamerge merged/standard 1 scripts/jquery-1.2.6.min.js scripts/jquery.cookie.js scripts/superfish.js scripts/supersubs.js scripts/shared.js %}

The first line obviously makes the mediamerge tag available to the template. The second line needs more explanation. The first parameter has two meanings

First it is the relative path of the generated merged javascript file. The path is relativ to settings.MEDIA_ROOT (not used if the mediamerged is configured to use memcached – on to that later)
Second it defines an URI relative to settings.MEDIA_URL

The second parameter defines a numeric version of the merged scripts and needs to be bumped by you whenever one of the files changes.

After the version follows the list of resources to be merged. All specified relative to settings.MEDIA_ROOT.

Assuming that settings.MEDIA_URL = http://static.example.com/ the tag above would result in the following HTML:

^?View Code HTML

Great, we’ve cut the number of requests by a factor. But it gets even faster. The mediamerge tag can be instructed to not generate an output file but to put the merged content directly into memcached (I’m obsessed with memcached) for a webserver supporting memcached to pick it up there. Nginx does and it is lightning fast serving static content.

So we instruct the mediamerge tag to use memcached in settings.py:

^?View Code PYTHON

MEDIAMERGE_ENABLE_MEMCACHED = True
MEDIAMERGE_MEMCACHED_URI_PREFIX = "mmerge_"
MEMCACHED_SERVERS = [ 'localhost:11211' ]

Using these settings, mediamerge will store the merged scripts in memcached under the key ‘mmerge_ http://static.example.com/merged/standard.js?1′ (settings.MEDIAMERGE_MEMCACHED_URI_PREFIX + settings.MEDIA_URL + )

The final thing to make it all work is to instruct Nginx to pick up request for resources handled by the mediamerge tag directly from memcached.

^?View Code APACHE

server
{
	listen 80;
	server_name static.example.com;
	expires 24h;
 
	# mediamerge
	location /merged/
	{
		set $memcached_key mmerge_$scheme://$server_name$request_uri;
		memcached_pass 127.0.0.1:11211;
	}
 
	location /
	{
		root /var/www/example_static;
	}
}

Bingo, Nginx will now serve all requests for resources in http://static.example.com /merged/ by looking up the contents in the specified memcached server (cluster). Nginx is already very fast for traditionally served static files but this is just ridiculously fast.

UPDATE: I’ve updated the source below since the article was originally posted. Changes:

A new option is now supported when using the disk based method: MEDIAMERGE_DISABLE_MERGE_UPDATES. Setting this option to True in settings.py will disable any checks for updated merge source files and is intended for production servers where your input files should not change in the background unless the web server is restarted. This should improve the rendering time of the tag in production considerably.
The files are now opened and written in binary mode

Here’s the source code for the mediamerge template tag:

^?View Code PYTHON

import os, re
import logging
from os import path
import cStringIO as StringIO
from django.template import Library, Node, TemplateSyntaxError
from django.conf import settings
 
logger = logging.getLogger('mediamerge')
mc = None
register = Library()
 
class JSMergeNode(Node):
	def __init__(self, js_name, js_ver, js_files):
		self.js_name = '%s.js' % js_name
		self.js_ver = js_ver
		self.js_files = js_files
		self.merge_filename = path.normpath(path.join(settings.MEDIA_ROOT, self.js_name))
 
		# make sure that the target directory exists
		if not settings.MEDIAMERGE_ENABLE_MEMCACHED and not path.exists(path.dirname(self.merge_filename)):
			os.makedirs(path.dirname(self.merge_filename))
 
	def merge_files(self, merge_file):
		# join all input files into a single output file
		for js in self.js_files:
			jspath = path.normpath(path.join(settings.MEDIA_ROOT, js))
			if not path.isfile(jspath):
				continue
			self.merge_js(js, jspath, merge_file)
 
	def render(self, context):
		'''Implementation of the render method of the tag'''
		if settings.MEDIAMERGE_ENABLE_MEMCACHED:
			return self.render_memcached(context)
		else:
			return self.render_file(context)
 
	def render_passthrough(self, context):
		result = ""
		for js in self.js_files:
			if settings.MEDIA_URL[-1] == '/':
				result += '\n' % (settings.MEDIA_URL, js)
			else:
				result += '\n' % (settings.MEDIA_URL, js)
		return result
 
	def render_memcached(self, context):
		'''Memcached based render method - merged output directly stored in memcache - where nginx will pick it up based on the rquest uri) without using a temp file'''
		import memcache
		key = settings.MEDIAMERGE_MEMCACHED_URI_PREFIX + str(self.compute_media_url())
 
		global mc
		if not mc:
			mc = memcache.Client(settings.MEMCACHED_SERVERS, debug=0)
 
		if not mc.get(key):
			merge_file = StringIO.StringIO()
			self.merge_files(merge_file)
			contents = merge_file.getvalue()
			mc.set(key, contents, 3600)
			logger.info('Setting memcached value for for resource %s - key %s' % (self.js_name, key))
 
		else:
			logger.debug('Merge key %s already exists in memcached for resource %s' % (key, self.js_name))
 
		return self.js_tag()
 
	def render_file(self, context):
		'''File based render method - merged output directly stored in the file specified with the tag - relative to settings.MEDIA_ROOT'''
		if not path.exists(self.merge_filename) or not self.is_merge_updated():
			logger.debug('Trying to open output file %s for resource %s' % (self.merge_filename, self.js_name))
			merge_file = open(self.merge_filename, 'wb')
			self.merge_files(merge_file)
			merge_file.close()
		else:
			logger.debug('Merge file %s already exists for resource %s' % (self.merge_filename, self.js_name))
 
		return self.js_tag()
 
	def is_merge_updated(self):
		if settings.MEDIAMERGE_DISABLE_MERGE_UPDATES:
			return True
 
		""" compares modification time of all js with merged js """
		last_mtime = 0 # last modification time of a js file
		for js in self.js_files:
			jspath = path.normpath(path.join(settings.MEDIA_ROOT, js))
			jsstat = os.stat(jspath)
			mtime = jsstat[-2]
			if last_mtime < mtime:
				last_mtime = mtime
		merge_mtime = os.stat(self.merge_filename)[-2]
		return merge_mtime > last_mtime
 
	def merge_js(self, jsname, jspath, fd):
		""" do merging and compressing of javascript """
		logger.debug('Merging file %s for resource %s' % (jsname, self.js_name))
 
		jsfile = open(jspath, "rb")
		jscontent = jsfile.read()
		fd.write('/* -- %s -- */\n\n' % jsname)
		fd.write(jscontent)
		fd.write("\n\n\n")
 
	def compute_media_url(self):
		# detect if MEDIA_URL ends with a slash
		if settings.MEDIA_URL[-1] == '/':
			js_url = '%s%s' % (settings.MEDIA_URL, self.js_name, self.js_ver)
		else:
			js_url = '%s/%s?%d' % (settings.MEDIA_URL, self.js_name, self.js_ver)
 
		return js_url
 
	def js_tag(self):
		""" write js tag for merged file inclusion """
 
		return '' % self.compute_media_url()
 
 
def do_jsmerge(parser, token):
	"""
	This will merge javascript files in only one compressed javascript.
 
	Usage::
 
		{% load mediamerge %}
		{% mediamerge   [jsfile1] [jsfile2] .. %}
 
	Example::
 
		{% load mediamerge %}
		{% mediamerge mediamergefile 1  js/file1.js js/file2.js js/file3.js %}
 
	This will create (if not exists) a /media/mediamergefile.js with three files merged. The HTML output for this will be::
 
		
	"""
	tokens = token.contents.split()
	if len(tokens) < 3:
		raise TemplateSyntaxError(u"'%r' tag requires at least 1 arguments." % tokens[0])
	return JSMergeNode(tokens[1], int(tokens[2]), tokens[3:])
 
 
register.tag('mediamerge', do_jsmerge)

django, nginx, memcached – the dynamic trio

oliver — Fri, 12 Sep 2008 21:01:22 +0000

Inspired by this article I decided to find out if the same technique can be exploited in my current project which is developed in django.

My first problem was to come up with a viable cache key scheme since simply using the full request URI as suggested in the article wouldn’t work for me because my site renders a different version of a navigation menu depending on the authentication state of current the user. After weighing in the advantages and disadvantages between the super clean variant of factoring the session cookie and all other cookies into the memcached key and a less heavy weight method that would only append a server supplied abstract “page version” field to the request URI, I went for the latter. My resulting nginx virtual host config was looking like this:

^?View Code APACHE

# define application servers
upstream backend
{
  server 127.0.0.1:8080 weight=1;
}
 
server
{
  listen 80;
  server_name domain.com;
  access_log /var/log/nginx/domain.com.log;
 
  location /
  {
    # we never cache post requests
    if ($request_method = POST)
    {
      proxy_pass http://backend;
      break;
    }
 
    # extract cache key args and compute cache key
    if ($http_cookie ~* "pv=([^;]+)(?:;|$)")
    {
      set $page_version $1;
    }
    set $memcached_key $request_uri&pv=$page_version;
 
    # Check if local memcached server can answer this request
    default_type text/html;
    memcached_pass 127.0.0.1:11211;
 
    # Send to app. server if Memcached could not answer the request
    error_page 404 = @cache_miss;
  }
 
  location @cache_miss
  {
    proxy_pass http://backend;
  }
}

The important lines are line 23-27. Line 23 tests the cookies header of the current request for the presence of a cookie named ‘pv’ using a regular expression and line 25 stores the extracted value of this cookie in a temporary variable. Later, in line 27 we combine the value of that variable with the request URI to form the final memcached key. For example if the request uri would be /foo and the headers would contain a cookie pv=acme123, then the cache key would be /foo&pv=acme123.

Line 31 is where the actual memcached lookup happens. If the computed key is present in the cache, the cached page is returned immediately – completely bypassing the backend. If the page (or the specific version of the page) is not present in the cache, then the alternate branch at line 37 is taken which ultimately forwards the request to our load balanced cluster of application servers in line 39. The actual servers making up the cluster is defined at line 2. There’s currently only one lonely application server defined there and that’s a apache 2.2 listening on localhost port 8080 running our django application through mod_wsgi.

So far I have explained how the front line request processing in nginx works but how is invalidation handled? To tackle that problem I wrote a django middleware that can be applied to arbitrary django views using a decorator:

^?View Code PYTHON

@cache_page_nginx(3600*6, compute_common_page_version)
def index(request):
  return render_to_response('index.html')
 
def compute_common_page_version(request):
  '''This method is called by the nginx_cache decorator
  The method is supposed to return some value that can be used to distinguish
  different cached versions of the same page. For now it is sufficient to differentiate
  between authenticated and anonymous users'''
  if request.user and request.user.is_authenticated():
    return 1
  return 0

Some readers will notice that the decorator looks very similar to Django’s built-in cache_page decorator. And it actually works similar but with several important differences:

It totally relies on nginx doing the actual cache lookup. The decorator is only responsible for storing the rendered content of page in the cache.
It bypasses the Django cache framework and talks to memcached directly using cmemcached or python-memcached. This was done because there’s reallly no point in supporting an additional level of abstraction if the consumer of the cached content won’t look for it anywhere but in memcached.

A word about the compute_common_page_version function. What does it do? The purpose of this method is to ensure that the cache will contain at maximum two distinct versions of the home page. Once for authenticated users and one for anonymous users. For more complex scenarios, the complexity of this method would also increase, resulting on more distinct versions of the same page. Obviously the difficulty lies in maintaining data integrity for every user while not producing to many cached versions of a page, thus increasing memory pressure on the server and lowering cache hit ratio.

The decorator:

^?View Code PYTHON

from django.utils.decorators import decorator_from_middleware
from django.conf import settings
from django.core.cache import cache
from django.utils.encoding import smart_unicode, smart_str
from utils import kdebug
 
try:
    import cmemcache as memcache
except ImportError:
    try:
        import memcache
    except:
        raise InvalidCacheBackendError("Memcached cache backend requires either the 'memcache' or 'cmemcache' library")
 
class UpdateCacheMiddleware(object):
  """
  Updates memcached with the response of the request. It is of _paramount_
  importance that the generated cache_key matches exactly the key generated
  by your web to lookup the page from the cache.
 
  This class talks to memcached, bypassing Djangos cache backend because
  it is only meant to talk to memcached and nothing else.
 
  Must the first piece of middleware in settings.MIDDLEWARE_CLASSES so
  that it'll get called last during the response phase.
  """
  def __init__(self, cache_timeout, page_version_method):
    '''timeout - is the timeout in seconds after which the cached response expires in memcached
    page_version_method - is called during response processing and must return an arbitrary value
    that can be used to distinguish different cached versions of the same page '''
    self.cache_timeout = cache_timeout
    self.page_version_method = page_version_method
    self.cache = memcache.Client(settings.NGINX_MEMCACHED_SERVERS)
    self.cookie_name = 'pv'
 
  def process_response(self, request, response):
    """Sets the cache, if needed."""
    if not settings.NGINX_MEMCACHED_ENABLE or request.method != 'GET' or not response.status_code == 200:
      # because of interactions between this middleware and the
      # HTTPMiddleware, which throws the body of a HEAD-request
      # away before this middleware gets a chance to cache it.
      return response
 
    self.cache_response(request, response)
    return response
 
  def cache_response(self, request, response):
    '''Manually insertion of cached version of page into the cache'''
    # retrieve page version
    pv = self.page_version_method(request)
 
    # compute key that's follows the same naming convention as specified in nginx configuration:
    # set $memcached_key $request_uri&sid=$session_id;
    cache_key = self.compute_cache_key_from_request(request, pv)
 
    # update cache
    self.cache.set(cache_key, response._get_content(), self.cache_timeout)
 
    # communicate the page version to the browser using cookie
    response.set_cookie(self.cookie_name, pv)
 
  def compute_cache_key_from_request(self, request, page_version):
    '''Computes the cache key for the specified page and version of the page'''
    return self.compute_cache_key(request.get_full_path(), page_version)
 
  def compute_cache_key(self, page, page_version):
    '''Computes the cache key for the specified page and version of the page'''
    return smart_str("%s&%s=%s" % (page, self.cookie_name, page_version))
 
  def invalidate_page_version_from_request(self, request, page_version):
    '''Removes the specified version of the page requested by 'request' from the cache'''
    cache_key = self.compute_cache_key_from_request(request, page_version)
    self.cache.delete(cache_key)
 
  def invalidate_page_version(self, page, page_version):
    '''Removes the specified version of the page requested by 'request' from the cache'''
    cache_key = self.compute_cache_key(request, page_version)
    self.cache.delete(cache_key)
 
# decorator
cache_page_nginx = decorator_from_middleware(UpdateCacheMiddleware)

Using IntelliJ for Django Development

oliver — Fri, 11 Jul 2008 10:08:56 +0000

About a year ago I’ve switched over to IntelliJ as my primary Java IDE. When I began to dabble a bit in Django a few days ago, I realized that IntelliJ had me spoiled when it comes to Editor Features – especially when working with Javascript and HTML files. Notepad++ – although a great text editor on its own right – simply didn’t cut it for me for web development.

When I tried opening my Django files in IntelliJ I had to realize that IntelliJ needs a project context for opening a file. Even for a simple html file. Fortunately the solution was pretty straight forward. Navigate to your Django project root directory and create a new file .project. Some of you have guessed it: we are pretending to be Eclipse. Open the file in a text editor and paste the following snipped into it:

myproject

Edit the tag value and change it to your liking.

Now it’s time to launch IntelliJ. Select File -> New Project. Select Import project from external model and click Next. Select Eclipse (should be the default) and click Next. Now enter the path to your Django project root folder in the Select Eclipse projects directory field. Click Next. You should now see the name of your pseudo eclipse project in the list. Click Next. You may now change the project name or just click Finish. Open the project view and change to Project mode. Congratulations, you have imported your Django project into IntelliJ.

I’d suggest that you install the Pythonid plugin for IntelliJ in order to get a better Python code editing experience and that you add .pyc and .pyo to the Ignored files and folders in IntelliJ’s General Settings as final steps.