Bearded Magnum

The lack of high level ZooKeeper library exposes your code to the potential network or ZooKeeper node failures. So you have to code defensively and be prepared to handle disconnections. ZooKeeper client offers two differents ways to know about failures: exceptions and events aka watches. Exceptions are used for the synchronous operations, so they are usually pretty straightforward to handle. However the asynchronous nature of watches might make things more difficult to fully grasp. I did some tests lately to make sure I had understood everything correctly. I observed a few behaviors that were not clear to me based on the sole documentation. I though they might be worth sharing here.
Here are my findings.

1. Session events i.e. Type-NONE events are sent to all outstanding watch handlers. Node events are not.
   Let’s say you invoked getChildren(“/foo”, ze_watcher), if later the client gets disconnected, both the default watcher and ze_watcher will receive a watch event NONE/Disconnected. But only ze_watcher will receive a NodeChildrenChanged event.
2. Watches are one-time triggers, but session events do NOT remove a watcher.
   In other words, still with our getChildren example, if  a disconnection occurs, ze_watcher will eventually receive 3 events: Disconnected, SyncConnected and finally NodeChildrenChanged. This will happen without having to set any new watcher.
3. If the invocation of a (synchronous or asynchronous) method fails, the watcher is not set.
   For instance if getChildren(“/foo”, ze_watcher) failed because the client is disconnected, ze_watcher won’t be notified of futur events.

Patrick Hunt confirmed this on the ZooKeeper mailing list and added further details.