Jul 11

Classloaders for Apache Flume plugins

Apache Flume is a tool for moving large amounts of data from various sources to a centralized data store. It provides an extensible framework to expand its applicability to various sources and stores.

The plugin framework currently lacks the ability to provide an isolated class loading to the plugins. Today, developers have the responsibility of dealing with situations wherein the plugin needs a certain version of a dependency but Flume itself has a different and incompatible version of that dependency. This is commonly known as the jar hell.

Cognitree forked the Flume source code to provide a solution to the much awaited enhancement. With this fork, each plugin gets loaded in its own class loader along with its dependencies. The plugin developers now don’t have to worry about conflicts between the plugin’s dependencies with those of Flume’s. They just need to place their jars into the directory structure laid out by flume. The class loader behavior is turned off by default to keep it backward compatible but can also be turned on using the flag “–use-plugins-classloader”.

Classloaders for flume plugins

With this enhancement, we have been able to easily write and deploy plugins without having to deal with hassles of dependency versions. The binary distribution can be downloaded here.

Leave a reply

Your email address will not be published.