Latest Entries »

Thug 0.4.0 was released on June, 8th 2012 and a huge number of really important features were added since then. During the last two years I had a lot of fun thinking and designing the future of the project and I’m really proud of what Thug is now. I have to thank a lot of persons who contributed with their suggestions, ideas, bug reports and sometimes patches. You know who you are. Really thanks!

But I have decided that it’s time to start a new branch. Thug 0.5.0 will be hopefully released before the end of July and the branch 0.5 will be more focused on performance, scalability and efficiency optimizations.

Moreover I decided to start writing a Know Your Tools (KYT) paper about Thug. Please take a look at the KYE/KYT papers published by the Honeynet Project at But we were thinking about using a different approach this time. Historically there exists an Honeynet Project KYE/KYT committee which takes care of the quality of the paper through a strong review process. And this is good obviously. But the paper is not public until it is published. Obviously.

But if I take a look at Thug I realize there are a lot of persons out there which are using it daily sometimes in unexpected ways. And their feedback could be useful as well. So the idea is starting writing the paper and update it in the same GitHub tree ( This could allow everyone to easily contribute to the paper through GitHub pull requests. The KYE/KYT committee will still guarantee the high quality of the paper through its review job but this is the first experiment of a collaborative paper. So if you are a Thug user and want to share some of your experiences, tips and tricks you are welcome to contribute!

Two years are passed from the first commit and taking a look at the number of committed patches I realized that right now the patch number 1000 was committed. Let me say it’s really impressive realizing it. In the last two years I had a lot of fun thinking and designing the future of this project and I’m really proud of what Thug turned to be. I have to thank a lot of persons who contributed with their suggestions, ideas, bug reports and sometimes patches. You know who you are. Really thanks! Stay tuned because I’m already moving towards patch number 2000 with a lot of new features and ideas like the Thug Honeyclient Distributed Task Queing.

Thug provides a DOM implementation which is (almost) compliant with W3C DOM Core, HTML, Events, Views and Style specifications (Level 1, 2 and partially 3). It makes use of the Google V8 Javascript engine wrapped through PyV8 in order to analyze malicious Javascript code and of the Libemu library wrapped through Pylibemu in order to detect and emulate shellcodes.

Currently 8 Internet Explorer (Windows XP, Windows 2000, Windows 7), 6 Chrome (Windows XP, Windows 7, MacOS X, Android 4.0.3, Linux), 3 Firefox (Windows XP, Windows 7, Linux) and 3 Safari (Windows XP, Windows 7, MacOS X) personalities are emulated and about 90 vulnerability modules (ActiveX controls, core browser functionalities, browser plugins) are provided.

The source code is available here.

Have fun!

In the last months I spent a lot of efforts in Thug development. During these months a few interesting features and improvements were introduced but right now I want to spend some time for taking a look at the new plugin framework introduced in the version 0.3.0. If you ever thought about extending Thug with additional features but didn’t know how to do it you should really keep on reading. Let’s start by taking a look a the code.

Taking a look at src/ we can now read these lines of code


216 if p:
217     ThugPlugins(PRE_ANALYSIS_PLUGINS, self)()
218     p(args[0])
219     ThugPlugins(POST_ANALYSIS_PLUGINS, self)()


Please note that every operation done by Thug is started by line 218 but now you can see that two hooks exist in order to execute plugins in a pre and post-analysis stage. Let’s keep exploring the source code and let’s take a look at src/Plugins/


34 class ThugPlugins:
35     phases = {
36                         PRE_ANALYSIS_PLUGINS : ‘ThugPluginsPre’,
37                         POST_ANALYSIS_PLUGINS : ‘ThugPluginsPost’
38                       }
40     def __init__(self, phase, thug):
41         self.phase = phase
42         self.thug = thug
43         self.__init_config()
45     def __init_config(self):
46         self.plugins = set()
47         config = ConfigParser.ConfigParser()
49         conf_file = os.path.join(os.path.dirname(os.path.abspath(__file__)), ‘plugins.conf’)
52         plugins = config.get(self.phases[self.phase], ‘plugins’)
53         for plugin in plugins.split(‘,’):
54             self.plugins.add(plugin.strip())
56     def __call__(self):
59     def run(self):
60         for source in self.plugins:
61             module = __import__(source)
62             components = source.split(‘.’)
63             for component in components[1:]:
64                 module = getattr(module, component)
66             handler = getattr(module, “Handler”, None)
67             if handler:
68                 p = handler()
69             try:
70                 verifyObject(IPlugin, p)
71       , log)
72             except BrokenImplementation as e:
73                 log.warning(“[%s] %s” % (source, e, ))


and src/Plugins/plugins.conf


1 [ThugPluginsPre]
2 plugins: Plugins.TestPlugin
4 [ThugPluginsPost]
5 plugins: Plugins.TestPlugin


The configuration file plugins.conf defines which plugins are to be loaded in pre and post-analysis stage (you can specify many plugins by simply comma separating them). The plugins should contain a class named Handler which should be conform to this interface


21 class IPlugin(zope.interface.Interface):
22     def run(thug, log):
23     “””
24     This method is called when the plugin is invoked
26     Parameters:
27     @thug: Thug class main instance
28     @log: Thug root logger
29     “””


If the interface is correctly implemented the `run’ method is called with two parameters: the Thug class main instance and the Thug root logger. Let’s see a really simple example of plugin


20 import zope.interface
21 from .IPlugin import IPlugin
23 class Handler:
24     zope.interface.implements(IPlugin)
26     def run(self, thug, log):
27         log.debug(thug)
28         log.debug(log)


This plugin just logs the parameters but you can do whatever you want. Do you want to pre-check if the URL domain is within a blacklist? Just do it with a pre-analysis plugin. Do you want to extract and/or correlate information from the MAEC log files? Just do it with a post-analysis plugin. Simply staten… have fun!

I’m glad to announce I publicly released a brand new low-interaction honeyclient I’m working on from a few months now. The project name is Thug and it was publicly presented during the Honeynet Project Security Workshop in Facebook HQ in Menlo Park. Please take a look at the presentation for details about Thug.

Just a few highlights about Thug:

  • DOM (almost) compliant with W3C DOM Core and HTML specifications (Level 1, 2 and partially 3) and partially compliant with W3C DOM Events and Style specifications
  • Google V8 Javascript engine wrapped through PyV8
  • Vulnerability modules (ActiveX controls, core browser functionalities, browser plugins)
  • Currently 6 IE personalities supported
  • Hybrid static/dynamic analysis
  • MITRE MAEC native logging format
  • HPFeeds and MongoDB logging

The source code is available here.

Feedback and comments welcome.

Have fun!

Another challenge is ready to be tackled by forensic analysts, students, hackers and alike. This time, we present you with an attack vector that has become quite successful: malicious PDF files!

For challenge 6 of our series (provided by Mahmud Ab Rahman and Ahmad Azizan Idris from the Malaysia Honeynet Project Chapter) we present you with a pcap file that contains network traffic generated by the following scenario: An unsuspecting user opens a compromised web page, which redirects the user’s web browser to a URL of a malicious PDF file. As the PDF plug-in of the browser opens the PDF, the unpatched version of Adobe Acrobat Reader is exploited and, as a result, downloads and silently installs malware on the user’s machine.

We prepared a set of questions that requires you to dive deep into the portable document format. Submit your solution by November 30th 2010. The top three submissions will receive small prizes.


Christian Seifert
Chief Communications Officer
The Honeynet Project

Forensic Challenge 5 – Log Mysteries is now live! This challenge takes you into the world of virtual systems and confusing log data. Figure out what happened to a virtual server using all the logs from a possibly compromised server.

Challenge 5 has been created by Raffael Marty from the Bay Area Chapter, Anton Chuvakin from the Hawaiian Chapter, and Sebastien Tricaud from the French Chapter. It is a bit more open ended than the last challenges, so I am really looking forward to some creative answers!

Submission deadline is September 30th and we will be announcing winners around October 21st. We have a few small prizes for the top three submission.


A new improvement in PHoneyC DOM emulation code was committed in SVN r1624. The idea is to better emulate the DOM behaviour depending on the selected browser personality. Let’s take a look at the code starting from the personalities definition in

39 UserAgents = [
40     (1,
41      “Internet Explorer 6.0 (Windows 2000)”,
42      “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
43      “Mozilla”,
44      “Microsoft Internet Explorer”,
45      “4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
46      “ie60″,
47     ),
48     (2,
49      “Internet Explorer 6.1 (Windows XP)”,
50      “Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
51      “Mozilla”,
52      “Microsoft Internet Explorer”,
53      “4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
54      “ie61″,
55     ),
56     (3,
57      “Internet Explorer 7.0 (Windows XP)”,
58      “Mozilla/4.0 (Windows; MSIE 7.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)”,
59      “Mozilla”,
60      “Microsoft Internet Explorer”,
61      “4.0 (Windows; MSIE 7.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)”,
62      “ie70″,
63     ),
64     (4,
65      “Internet Explorer 8.0 (Windows XP)”,
66      “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; (R1 1.5); .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
67      “Mozilla”,
68      “Microsoft Internet Explorer”,
69      “4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; (R1 1.5); .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
70      “ie80″,
71     ),
72 ]

It’s important to realize that each personality was added a tag (i.e. ie80). Taking a look at DOM/ the following code can be seen.

229     def __init_methods(self):
230         for attr in dir(self):
231             prefix = “_Window__window_%s_” % (config.browserTag, )
232             if attr.startswith(prefix):
233                 p = attr.split(prefix)[1]
234                 self.__dict__[‘__cx’].add_global(p, getattr(self, attr))
235                 self.__dict__[‘__cx’].execute(“window.%s = %s;” % (p, p, ))

Let’s consider an example and assume the Internet Explorer 8.0 personality was selected. It’s easy to realize that the prefix would assume the value _Window__window_ie80_. A few simple wrappers were created, one per personality, to each method as shown in the following code.

340     def __window_back(self):
341         “””
342         Returns the window to the previous item in the history.
343         Syntax
345         window.back()
347         Parameters
349         None.
350         “””
351         pass
353     def __window_ie60_back(self):
354         self.__window_back()
356     def __window_ie61_back(self):
357         self.__window_back()
359     def __window_ie70_back(self):
360         self.__window_back()
362     def __window_ie80_back(self):
363         self.__window_back()
365     def __window_firefox_back(self):
366         self.__window_back()

This is a quite simple situation but what if you want to define addEventListener method just for Firefox personalities and attachEvent just for Internet Explorer ones? Really simple to do!

1191     def __window_attachEvent(self, sEvent, fpNotify):
1192         if dataetc.isevent(sEvent, ‘window’):
1193             self.__dict__[sEvent] = fpNotify
1195     def __window_ie60_attachEvent(self, sEvent, fpNotify):
1196         self.__window_attachEvent(sEvent, fpNotify)
1198     def __window_ie61_attachEvent(self, sEvent, fpNotify):
1199         self.__window_attachEvent(sEvent, fpNotify)
1201     def __window_ie70_attachEvent(self, sEvent, fpNotify):
1202         self.__window_attachEvent(sEvent, fpNotify)
1204     def __window_ie80_attachEvent(self, sEvent, fpNotify):
1205         self.__window_attachEvent(sEvent, fpNotify)
1208     def __window_detachEvent(self, sEvent, fpNotify):
1209         if sEvent in self.__dict__ and self.__dict__[sEvent] == fpNotify:
1210             del self.__dict__[sEvent]
1212     def __window_ie60_detachEvent(self, sEvent, fpNotify):
1213         self.__window_detachEvent(sEvent, fpNotify)
1215     def __window_ie61_detachEvent(self, sEvent, fpNotify):
1216         self.__window_detachEvent(sEvent, fpNotify)
1218     def __window_ie70_detachEvent(self, sEvent, fpNotify):
1219         self.__window_detachEvent(sEvent, fpNotify)
1221     def __window_ie80_detachEvent(self, sEvent, fpNotify):
1222         self.__window_detachEvent(sEvent, fpNotify)
1225     def __window_addEventListener(self, type, listener, useCapture = False):
1226         if dataetc.isevent(type, ‘window’):
1227             self.__dict__[type] = listener
1229     def __window_firefox_addEventListener(self, type, listener, useCapture = False):
1230         self.__window_addEventListener(type, listener, useCapture = False)
1233     def __window_removeEventListener(self, type, listener, useCapture = False):
1234         if type in self.__dict__ and self.__dict__[type] == listener:
1235             del self.__dict__[type]
1237     def __window_firefox_removeEventListener(self, type, listener, useCapture = False):
1238         self.__window_removeEventListener(type, listener, useCapture = False)

Moreover this approach could allow to insert specific code within the wrappers if needed while implementing the method functionalities in the higher level __window_<method_name> wrapper.

“Dionaea is meant to be a Nepenthes successor, embedding Python as scripting language, using libemu to detect shellcodes, supporting IPv6 and TLS” (taken from Dionaea homepage). Besides being the most interesting project for trapping malware exploiting vulnerabilities, Dionaea supports a really cool feature which allows it to log to XMPP services as described here. TIP now exploits this feature receiving and storing such logs (really thanks to Markus Koetter for his help and support). Just an example of what happened today…

2010-08-11 10:44:21+0200 [XmlStream,client] [Malware Sample] MD5: e4736922939a028384522b17e9406474
2010-08-11 10:44:21+0200 [XmlStream,client] [Malware Sample] SHA-1: 920b67cb250abdb593b1104a9922e2468b0fe252
2010-08-11 10:44:21+0200 [XmlStream,client] [Malware Sample] PEHash: 40891becb5ec8780f1c5e51f3971c9fb2cc17dab

Another great step forward was taken. Stay tuned!

A few weeks ago I started reviewing the PHoneyC DOM emulation code and realized it was turning to be hard to maintain and debug due to a huge amount of undocumented (and sometimes awful) hacks. For this reason I decided it was time to patch (and sometimes rewrite from scratch) such code. These posts will describe how the new DOM emulation code will work. The patch is not available right now since I’m testing the code but plans exists to commit it in the PHoneyC SVN in the next days.

In this first post we will take a look at the Window object in DOM/ During object inizialization, the following code is executed.

156     def __init_context(self):
157         “””
158             Spidermonkey Context initialization.
159         “””
160         document = Document(self)
161         self.__dict__[‘__cx’] = self.__dict__[‘__rt’].new_context(alertlist = [])
162         self.__dict__[‘__sl’] = []
163         self.__dict__[‘__fl’] = [document]
165         self.__init_properties(document)
166         self.__init_methods()
167         self.__finalize_context()

Let’s go into further details. First of all Window object properties are initialized through the __init_properties method.

181     def __init_properties(self, document):
182         self.__dict__[‘__cx’].add_global(‘window’, self)
183         self.__dict__[‘__cx’].add_global(‘self’  , self)
184         self.__dict__[‘__cx’].execute(“window.window = window;”)
186         self.__dict__[‘__cx’].add_global(‘document’, document)
187         self.__dict__[‘__cx’].execute(“window.document = document;”)
189         self.__dict__[‘__cx’].add_global(‘location’, document.location)
190         self.__dict__[‘__cx’].execute(“window.location = location;”)
192         self.__dict__[‘__cx’].add_global(“ActiveXObject”, ActiveXObject)
194         self.__dict__[‘__cx’].add_global(“navigator”, Navigator())
195         self.__dict__[‘__cx’].execute(“window.navigator = navigator;”)
197         self.__dict__[‘__cx’].add_global(“screen”, unknown())
198         self.__dict__[‘__cx’].execute(“window.screen = screen;”)
200         if ‘top_window’ in self.__dict__[‘__root’].__dict__:
201             if self.__dict__[‘__referrer’]:
202                 top = self.__dict__[‘__referrer’]
203             else:
204                 top = self.__dict__[‘__root’].top_window
205         else:
206             top = self
208         self.__dict__[‘__cx’].add_global(“top”, top)
209         self.__dict__[‘__cx’].execute(“ = top;”)
211         self.__dict__[‘__cx’].add_global(“parent”, top)
212         self.__dict__[‘__cx’].execute(“window.parent = parent;”)
214         self.__dict__[‘__cx’].add_global(“history”, History(document))
215         self.__dict__[‘__cx’].execute(“window.history = history;”)
217         self.__dict__[‘__cx’].execute(“window.innerWidth = 400;”)
218         self.__dict__[‘__cx’].execute(“window.innerHeight = 200;”)
220         self.__init_undefined_properties()
222     def __init_undefined_properties(self):
223         properties = (‘external’, ‘opera’, )
225         for prop in properties:
226             self.__dict__[‘__cx’].execute(“window.%s = undefined;” % (prop, ))

The code should be straightforward to understand. The idea beyond it is really simple. Simply stated this code allows Python objects’ variables and methods to be accessible from JS. Let’s move to most interesting stuff. Following the __init_methods method is called.

228     def __init_methods(self):
229         for attr in dir(self):
230             if attr.startswith(‘_Window__window’):
231                 p = attr.split(‘_Window__window_’)[1]
232                 self.__dict__[‘__cx’].add_global(p, getattr(self, attr))
233                 self.__dict__[‘__cx’].execute(“window.%s = %s;” % (p, p, ))

Not so easy to understand? Let’s take a look to the definition of a method.

322     def __window_back(self):
323         “””
324         Returns the window to the previous item in the history.
325         Syntax
327         window.back()
329         Parameters
331         None.
332         “””
333         pass

This is a private class method since its name starts with __. “If you try to call a private method, Python will raise a slightly misleading exception, saying that the method does not exist. Of course it does exist, but it’s private, so it’s not accessible outside the class. Strictly speaking, private methods are accessible outside their class, just not easily accessible. Nothing in Python is truly private; internally, the names of private methods and attributes are mangled and unmangled on the fly to make them seem inaccessible by their given names.” (taken from Dive Into Python). We can access the __window_back method of the Window class by the name _Window__window_back. This is the black magic __init_methods use for initializing methods. It’s quite easy to realize that adding a new method is really easy. All you need is to simply define a method named __window_<window_method_name> and match the signature of such method. How to emulate such method it’s up to you but a simple pass could do the trick.

The last step happens in __finalize_context method.

169     def __finalize_context(self):
170         self.__dict__[‘__cx’].execute(“Event = function(){}”)
171         self.__dict__[‘__cx’].execute(“function CollectGarbage() {};”)
172         self.__dict__[‘__cx’].execute(“function quit() {};”)
173         self.__dict__[‘__cx’].execute(“function prompt() {};”)
175         for clsname in dataetc.classlist:
176             inits = {‘window’ : self,
177                      ‘tagName': dataetc.classtotag(clsname),
178                      ‘parser’ : None}
179             self.__dict__[‘__cx’].add_global(clsname, DOMObjectFactory(clsname, inits))

The most interesting code is in lines 175-179. First of all let’s take a look at the DOMObjectFactory code (DOM/ which is a genuine Python hack.

3 class DynamicDOMObject(DOMObject):
4     def __init__(self):
5         self.__dict__.update(self.inits)
6         DOMObject.__init__(self, self.window, self.tagName, self.parser)
8 def DOMObjectFactory(name, initializers):
9     return type(name, (DynamicDOMObject,), {‘inits’ : initializers})

After reading Python documentation it should be easy to understand how this code works and how it’s able to dynamically add new DOM objects to the context.

type(name, bases, dict)

Return a new type object. This is essentially a dynamic form of the class statement. The name string is the class name and becomes the __name__ attribute; the bases tuple itemizes the base classes and becomes the __bases__ attribute; and the dict dictionary is the namespace containing definitions for class body and becomes the __dict__ attribute. For example, the following two statements create identical type objects:

>>> class X(object):
… a = 1

>>> X = type(‘X’, (object,), dict(a=1))

What about the Window event handlers? They are handled with a different mechanism which can be fully understood just by analyzing how the new DOM emulation code preparses the pages deferring code execution until the last possible moment. I’ll analyze such feature in a future post in greater detail. Right now what you have to know is that if the handler for the event <event> is set, the Window attribute on<event> is set and contains the handler code. Once you understand it, the following code in DOM/ used for event handling should be easy to understand.

171     def get_event_func(self, name, f):
172         begin = str(f).index(‘{‘) + 1
173         s = str(f)[begin:].split(‘}’)
174         script = ‘}’.join(s[:-1]) + s[-1]
175         return script
177     def event_handler(self, window, name, f):
178         if name in window.__dict__:
179             try:
180                 script = self.get_event_func(name, f)
181                 window.__dict__[‘__cx’].execute(script)
182             except:
183                 #print str(f)
184                 traceback.print_exc()
185                 pass
187     def handle_events(self, window):
188         window.__dict__[‘__warning’] = False
189         self.event_handler(window, ‘onabort’         , window.onabort)
190         self.event_handler(window, ‘onbeforeunload’  , window.onbeforeunload)
191         self.event_handler(window, ‘onblur’          , window.onblur)
192         self.event_handler(window, ‘onchange’        , window.onchange)
193         self.event_handler(window, ‘onclick’         , window.onclick)
194         self.event_handler(window, ‘onclose’         , window.onclose)
195         self.event_handler(window, ‘oncontextmenu’   , window.oncontextmenu)
196         self.event_handler(window, ‘ondragdrop’      , window.ondragdrop)
197         self.event_handler(window, ‘onerror’         , window.onerror)
198         self.event_handler(window, ‘onfocus’         , window.onfocus)
199         self.event_handler(window, ‘onhashchange’    , window.hashchange)
200         self.event_handler(window, ‘onkeydown’       , window.onkeydown)
201         self.event_handler(window, ‘onkeypress’      , window.onkeypress)
202         self.event_handler(window, ‘onkeyup’         , window.onkeyup)
203         self.event_handler(window, ‘onload’          , window.onload)
204         self.event_handler(window, ‘onmousedown’     , window.onmousedown)
205         self.event_handler(window, ‘onmousemove’     , window.onmousemove)
206         self.event_handler(window, ‘onmouseout’      , window.onmouseout)
207         self.event_handler(window, ‘onmouseover’     , window.onmouseover)
208         self.event_handler(window, ‘onmouseup’       , window.onmouseup)
209         self.event_handler(window, ‘onmozorientation’, window.onmozorientation)
210         self.event_handler(window, ‘onpaint’         , window.onpaint)
211         self.event_handler(window, ‘onpopstate’      , window.onpopstate)
212         self.event_handler(window, ‘onreset’         , window.onreset)
213         self.event_handler(window, ‘onresize’        , window.onresize)
214         self.event_handler(window, ‘onscroll’        , window.onscroll)
215         self.event_handler(window, ‘onselect’        , window.onselect)
216         self.event_handler(window, ‘onsubmit’        , window.onsubmit)
217         self.event_handler(window, ‘onunload’        , window.onunload)
218         self.event_handler(window, ‘onpageshow’      , window.onpageshow)
219         self.event_handler(window, ‘onpagehide’      , window.onpagehide)
220         window.__dict__[‘__warning’] = True

Today I was in need for fun and so I started adding a new API call which allows to check if a domain is malicious or not. The check avoids to hit the database at all but just queries the search index. The results I got are quite surprising. Take a look at it considering  that code 409 means ‘Object already exists’ while code 410 means ‘Object does not exist’.

Let’s start with a benign domain not tracked by TIP.

buffer@alnitak ~ $ time wget http://xxxx.xxxx.xx/api/check/domain/test@@it/
HTTP request sent, awaiting response… 410 GONE
2010-07-22 17:46:58 ERROR 410: GONE.

real    0m0.017s
user    0m0.001s
sys    0m0.001s

Now let’s move to a malicious domain tracked by TIP.

buffer@alnitak ~ $ time wget http://xxxx.xxxx.xx/api/check/domain/hazelpay@@ru/
HTTP request sent, awaiting response… 409 CONFLICT
2010-07-22 17:47:07 ERROR 409: CONFLICT.

real    0m0.022s
user    0m0.000s
sys    0m0.002s

Bad Behavior has blocked 15 access attempts in the last 7 days.