Scalable Logs with ZeroMQ and Zend Framework

Do you want a scalable, high performance PHP application that logs everything in an intelligible manner; in a manner that you can use now, tomorrow and three weeks from now? It’s so possible! In a series of posts, I am going to show you how to put it together using Zend Framework, ZeroMQ and Hadoop.

In my recent discussion and rants, some would say, I’ve been really getting stuck in to what makes a good developer, what makes good code and generally, what is a professional approach to software development and maintenance.

What is High Performance?

Well, I started looking at logging specifically recently. I was talking with a former colleague who pointed me to this excellent article on high scalability application development. It specifically discussed just how much logging you can and should do in an application, yet still be able to have a high performance metric – even in an embedded system.

The points that stood out for me the most were these:

  • Store log messages in a circular queue to limit resource usage.

  • Write log messages to disk in big sequential blocks for efficiency.

  • Every object in your system should be dumpable to a log message.

  • Make the log message directly queueable to the log task so queuing doesn’t take more memory allocations.

  • Log to a separate task and let the task push out log data when it can.

  • Use a preallocated buffer pool for log messages so memory allocation is just pop and push.

It really got me to thinking, because all to often, logging is an after thought, or at the very least, not very well thought out from a longer-term perspective. So I thought, what if you could log in an intelligible manner, massive amounts of information, yet still be able to do it in a high-performance manner. How would you do it?

How Will it Work? Zend / ZeroMQ / Hadoop

Given my propensity for Zend Framework, I tended to it straight-away as the core base (ok, it’s not, out of the box, the fastest, but this is a proof of concept), then combining it with ZeroMQ through PHP’s extension to it, excellently written by Mikko Koppanen. If you’re not familiar with ZeroMQ, check out the superb talk given by my friend Ian Barber at this years PHP London.


With ZeroMQ, we could setup a publish/subscriber configuration, where objects, appropriately written, could be published to the pipe with the relevant information, such as:

  • event timestamp
  • retrieved data (from a SOAP/Rest request)
  • configuration/settings information
  • event/class/function information

The process would be published asynchronously, so as to not slow down the core PHP app. That way, we could, theoretically, have both a high performance application, with comprehensive logging, managed by another PHP process, or a process written in a different language entirely, logging to anything from the filesystem, a NoSQL database, Amazon S3 or a database – as your imagination, time and budget allowed.

So in the second series, we’re going to look at integrating a combination of Zend Framework, ZeroMQ and the Hadoop NoSQL database to store the log information. We’re going to cover:

  • Designing objects to be dumpable quickly and effectively
  • A publisher/subscriber configuration of ZeroMQ that’s contactable from ZF
  • A PHP process to subscribe to the ZeroMQ server, retrieve the dumped objects and store them in the Hadoop database

Now that’s a pretty lofty set of goals, as you can imagine. So we’re not going to cover security or MapReduce. Maybe later, but not for the time being. However, if you’re keen to contribute, I’m keen to hear from you.

You might also be interested in these tutorials too...

Tue, Nov 15, 2011

Rename uploaded files with Zend Framework

Recently I was asked how to rename a file with the Zend Framework that used a Zend Form and Zend File element. They key requirement was that it should not be a hack or a kludged solution. So I thought I’d write a quick post to provide a simple example on how it was achieved.

Wed, Nov 9, 2011

The Zend Framework Bootstrap made simple (Part 3)

Ok, we’ve established that with the Zend Framework, we need to do a bit more work than some of the other frameworks to get up to speed - but that’s not necessarily a bad thing - right?! But it can be a bit tedious and it’s something as professional developers, we want to automate away. So we’ve been addressing in this series how to do just that with a custom, extendable bootstrap class.

In the first part of the series we laid the foundation of our custom bootstrap class by creating a custom class directory structure, adding its namespace to the application ini and modifying the default application bootstrap so that it extends from it and had a look at the first component - caching.

Then, in the second part of the series, we built on the foundation laid in part one by creating plugin resources for the routing table, application navigation and the database connections - these being some of the most common tasks, usually, associated with a web-based application.

In this, the third and final part of the series, I’m going to finish up by looking at application placeholders and surprise, no not logging as originally promised, but pagination. As an added extra, we’re going to be using a key feature of Zend Application to make it a breeze.

Wed, Nov 2, 2011

The Zend Framework Bootstrap made simple (Part 2)

In the first part of the series, you’ll remember that we laid the foundation of our custom bootstrap class by creating a custom class directory structure, adding its namespace to the application ini and creating our custom bootstrap file that our application bootstrap will extend from.

After we did that, we put in the first but arguably the most important plugin resource – caching and stored it in the application registry. In this post we’re going to be building on that work and adding in three new plugin resources: routing, navigation and databases.

Want more tutorials like this?

If so, enter your email address in the field below and click subscribe.

You can unsubscribe at any time by clicking the link in the footer of the emails you'll receive. Here's my privacy policy, if you'd like to know more. I use Mailchimp to send emails. You can learn more about their privacy practices here.

Join the discussion

comments powered by Disqus