High speed "cron" daemon in C++ with dynamic library loading.

I am going to present how to make a cron-daemon like program in C++. But a difference is that it can run more frequently than what "cron" can do. For example, if you want to have a cron task that should run every seconds, you cannot do that, because the minimum time resolution of cron is "minute"; not "second".

I also tried "dynamic library" feature in Linux to load C++ binary file at runtime dynamically. This means the base program doesn't need to be recompiled with modified CPP files. Instead, it can load/unload a Position Independent Code, or PIC, at runtime. You can easily find more information from the manual page: "man dlopen".

I made five files for the base program:

pi@fileserver 20:57:15 cron_native$ ls
cron_native.cpp  dynamic_library.hpp  plugins
daemon.hpp       Makefile             stdafx.h
pi@fileserver 20:57:17 cron_native$

I named it "cron_native" but I am not sure if it is good enough. "plugins" is a directory that contains plugins. And the "plugin" programs are Position Independent Code that can be loaded/unloaded dynamically; as I will cover later, they need special compiler options to be PIC.

The main logic of the program is in the file, "cron_native.cpp":

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

// cron_native.cpp written by JaeHyukKwak #include "stdafx.h" #include "dynamic_library.hpp" #include "daemon.hpp" static bool s_reload_dl_requested = false; void request_reload_dl_cb( int ) { s_reload_dl_requested = true; } int main( int argc, const char **argv ) { const char *plugin_folder = ( argc >= 2 ? argv[1] : "./plugins" ); daemon_init( argv, request_reload_dl_cb ); dynamic_library_t dl( plugin_folder ); while ( true ) { dl.call( "dl_init" ); while ( true ) { dl.call( "dl_main" ); sleep( 1 ); if ( s_reload_dl_requested ) break; } s_reload_dl_requested = false; dl.call( "dl_shutdown" ); dl.reload(); } // infinite loop return 0; }

The function, "main()", calls "daemon_init()" to setup the process as a daemon. It looks for plugin files under the given or default directory. It executes "dl_init()" functions on each plugin and repeatedly executes "dl_main" function on each plugin. I put 1 second sleep on each loop to cool it down little; otherwise it may consume CPU too much. As a daemon process, it can reload the plugins when a signal, "SIGHUP", occurs. During the reloading step, it calls "dl_shutdown" function on each plugin to give a chance to properly shutdown the program.

Note that it is a good practice to have explicit "init/shutdown" calls on any code that can be dynamically loaded and shared. I am not going to explain the details of the aspect but you can easily find topics like "DLL hell" or something.

The Makefile looks like this:

1 2 3 4 5 6 7 8 9 10 11 12

# Makefile written by Jae Hyuk Kwak CC=g++-4.7 CFLAGS=-Wall -g -std=c++11 cron_native: daemon.hpp dynamic_library.hpp cron_native.cpp stdafx.h.gch $(CC) $(CFLAGS) -ldl -o cron_native cron_native.cpp stdafx.h.gch : stdafx.h $(CC) $(CFLAGS) -c stdafx.h clean: rm stdafx.h.gch cron_native

Note that it includes "dl" library.

The precompile header file, "stdafx.h", looks like this:

1 2 3 4 5 6 7 8 9 10 11 12 13

// stdafx.h written by JaeHyukKwak #include <dlfcn.h> #include <sys/types.h> #include <sys/stat.h> #include <syslog.h> #include <stdarg.h> #include <signal.h> #include <dirent.h> #include <list> #include <memory> // shared_ptr #include <unistd.h> // XXX_FILENO #include <stdexcept> // exception using namespace std;

The file, daemon.hpp, is to setup the process as a daemon process. The characteristics of daemon process is well described here.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

// daemon.hpp written by JaeHyukKwak typedef void (*signal_handler_fptr_type)( int ); void daemon_init ( const char **argv , signal_handler_fptr_type handler ) { umask( 0 ); openlog( argv[0], LOG_NOWAIT | LOG_PID, LOG_USER ); syslog( LOG_NOTICE, "Successfully started daemon\n" ); close( STDIN_FILENO ); close( STDOUT_FILENO ); close( STDERR_FILENO ); signal( 1, handler ); }

One of the characteristics of daemon is that it assumes stdout is not available. All outputs go to syslog which then gets stored in /var/log/syslog. You can monitor the message output with a command like "tail -f /var/log/syslog".

The long program, "dynamic_library.hpp", looks like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69

// dynamic_library.hpp written by JaeHyukKwak class dynamic_library_t { const char *folder_path_; typedef shared_ptr< void > dl_handle_sptr_type; list< dl_handle_sptr_type > dl_handle_list_; public: dynamic_library_t( const char *folder_path ) : folder_path_( folder_path ) { load(); } void load() { list< dl_handle_sptr_type > l; dlerror(); // clear the pre-existing error messages. for ( string file : get_so_file_list_( folder_path_ ) ) { if ( void *dl_handle = dlopen( file.c_str(), RTLD_LAZY ) ) { l.push_back( dl_handle_sptr_type( dl_handle, dlclose ) ); syslog( LOG_NOTICE, ( file + " added." ).c_str() ); } else syslog( LOG_ERR, ( file + ": " + dlerror() ).c_str() ); } dl_handle_list_ = l; } void unload() { dl_handle_list_.clear(); } void reload() { unload(); load(); } void call( const char *name ) const { typedef void (*fptr_type)( uint32_t ); for ( auto sptr : dl_handle_list_ ) { if ( auto fptr = get_fptr_< fptr_type >( sptr.get(), name ) ) { try { (*fptr)( time( NULL ) ); } catch ( exception e ) { syslog( LOG_ERR, e.what() ); } } else syslog( LOG_ERR , ( string( dlerror() ) + ": " + name ).c_str() ); } } private: static list< string > get_so_file_list_( const char *folder_path ) { list< string > l; DIR *dir = opendir( folder_path ); for ( dirent *de = readdir( dir ); de; de = readdir( dir ) ) { if ( de->d_type != DT_REG ) continue; string fname = de->d_name; if ( fname.find( ".so" ) != fname.size() - 3 ) continue; string path = string( folder_path ) + "/" + fname; l.push_back( path ); } closedir( dir ); syslog( LOG_NOTICE, ( string( "folder " ) + folder_path + " has " + to_string( l.size() ) + " plugins" ).c_str() ); return l; } template< typename FptrType > static FptrType get_fptr_( void *dl_handle, const char *name ) { union { void *void_ptr; FptrType dl_fptr; } dl_fptr_union = { dlsym( dl_handle, name ) }; return dl_fptr_union.dl_fptr; } };

The "Dynamic library" mainly consists of three functions: "dlopen", "dlclose" and "dlsym". As the name implies, "dlopen" opens a PIC program and increase the reference counter by one. "dlclose" reduces the reference count by one and when the reference counter hits zero, it will unload the program. In other words, if the reference counter is still above zero, it will not unload nor load a new PIC. "dlsym" searches the PIC with the name of function we want to use and returns a function pointer. You can find more explanation from Wiki pages: DynamicLoading and  PIC.

For the function, dynamic_library_t::call(), I was thinking of using thread. But I couldn't find a good use of it. The latest RaspberryPi 2 has four cores so I would like to utilize threads as much as possible. But I don't think the added complexity can be justified in this case. Non-thread style is much more straight-forward and less error-prone.

I noticed that when one of PIC crashes or causes "Segment fault" error, the base program crashes as well, which makes the cron unstable as a daemon process. One of ways to prevent it is to "fork" a child process and isolate the problem in the process. But then it may affect the performance and the source code will suffer from complexity. Another way is to have another monitoring process or cron job that checks if the program is still running; if it doesn't it can simply restart a new one. I think when a crash happens, it should look obvious rather than make it automatically recover without noticing.

As a simple example of PIC, I have a "plugin.hello.cpp". In order to compile the plugin, I made three files:

pi@fileserver 21:53:49 plugins$ ls
Makefile  plugin.hello.cpp  stdafx.h
pi@fileserver 21:53:49 plugins$

On PIC side, the program must be compiled with a compiler option, "-fPIC". I am not sure on "-shared"; probably both are needed. And the functions that will be found by "dlsym" must have a keyword "extern "C"" at the beginning of the function.

The file, "plugin.hello.cpp", looks like this:

1 2 3 4 5 6 7 8 9 10 11 12 13

// plugin.hello.cpp written by JaeHyukKwak #include "stdafx.h" static int s_max_count; extern "C" void dl_init( uint32_t ) { s_max_count = 10; } extern "C" void dl_shutdown( uint32_t ) {} extern "C" void dl_main( uint32_t ) { if ( s_max_count <= 0 ) return; s_max_count--; syslog( LOG_NOTICE, "Hello" ); }

The program simply prints out syslog message, "Hello". You will see the message printed every second in the file, "/var/log/syslog". Note that it stops after 10 times; it is to show how to control the frequency or longevity of each plugin. As I mentioned earlier, three functions have keyword, "extern "C"". This is a common way to export the mangled function name.

The Makefile shows how the program is compiled.

1 2 3 4 5 6 7 8 9 10 11 12 13 14

# Makefile written by Jae Hyuk Kwak CC=g++-4.7 CFLAGS=-Wall -g -std=c++11 all: plugin.hello.so plugin.hello.so: plugin.hello.cpp stdafx.h.gch -$(CC) $(CFLAGS) -shared -fPIC -o plugin.hello.so plugin.hello.cpp stdafx.h.gch : stdafx.h $(CC) $(CFLAGS) -c stdafx.h clean: rm stdafx.h.gch plugin.*.so

Note the option, "-shared" and "-fPIC".

Not important but for completeness, I like to show the file, "stdafx.h", as well:

1 2 3 4

// stdafx.h written by JaeHyukKwak #include <syslog.h> #include <cstdint> // uint32_t using namespace std;

Enjoy the high resolution cron.