In the last days, the inner workings of TIP changed too much. In fact, as soon as I plugged in the new Spamtrap module, I realized that the core engine was far from perfect. In particular, it was designed when I had no precise idea of the work load it had to face and this forced me to rethink about it from scratch. First of all the new implementation is based on the Twisted Application Framework. Using this infrastructure freed me from having to write a large amount of boilerplate code by hooking the application into existing tools that manage daemonization, logging, choosing a reactor and more. Moreover, TIP is moving towards a component-based architecture by using the interfaces and adapters created by the Zope3 team for developing the submodules. The current implementation scales much better than the previous one because every time a module is scheduled, it runs inside its own subprocess controlled by the twistd master process. This design allows to avoid any kind of memory leakage issue which is exactly the reason why I moved towards a new scheduler design. Each subprocess is independent from the others and the main aim of the master process is to synchronize the subprocesses and free resources when they complete their tasks. Another important change which is worth mentioning is about the Fast Flux tracking module which is now handled as a two-pass subprocess in such a way to free resources as soon as it completes the domain fluxiness classification. Right now the first tests are running. Stay tuned!
-
Recent Posts
Categories
Archives
Links