Measuring performance of distributed notifications

Thursday, May 10, 2007 at 4:16 PM

Posted by Mike Morton, Software Engineer

On April 4th, we announced Google Updater. It uses several pieces of software that run in the background, but we knew early on it was important for the one user-visible application to be able to show you what those background pieces were up to.

We looked briefly at using Apple's Distributed Objects technology, but it can take a fair amount of work to make DO work well. We also need multiple copies of the user interface to listen to a single background process, so the logical way to implement it was NSNotificationCenter, a class in Apple's Foundation framework that broadcasts notifications across processes. It can even broadcast to applications run by other users logged into the same computer.

We did have one worry, though. Apple’s doc warns that “Posting a distributed notification is an expensive operation. The notification gets sent to a system-wide server that distributes it to all the tasks that have objects registered for distributed notifications…”. This left me wondering two things:

(1) Just how expensive is it? How many notifications can you broadcast per second? As with all Google client products, we want to be good citizens and not bog down the client machine.

(2) Does subscribing to any distributed notification really mean that the server sends you every notification, and not just the ones you asked to receive? Could we assume that the server keeps track of the “names” you ask to hear about, and filters before distributing?

One of my favorite riddles is “How many empiricists does it take to change a light bulb?”. So I wrote a command-line tool that broadcasts NSDistributedNotifications and another that listens. I ran ten copies of the listener and one broadcaster on a 2GHz Intel Core Duo.

The first interesting result was that the broadcaster can consistently push out a notification about every 2.64 milliseconds, no matter whether the listening processes are listening for the same name-and-object.

The second finding was that nothing burns much CPU except Apple’s distnoted process, which is the bottleneck through which all notifications get sent. When the listeners did listen for the name the broadcaster was sending, distnoted used about 30% of the CPU. Shark showed distnoted was spending time in memory management involving simple data structures.

So we can conclude two things:

• We can easily send several messages per second to a dozen or so listeners without impairing a client machine.

• Registering to get only the name/object you want can help performance.

We designed our background processes to limit the number of notifications they broadcast, and distributed notifications have been a great base on which to build this part of the Updater system.

P.S. The answer to the riddle: “I don’t know, either. Want to find out?” ;-)