But I have decided that it’s time to start a new branch. Thug 0.5.0 will be hopefully released before the end of July and the branch 0.5 will be more focused on performance, scalability and efficiency optimizations.

Moreover I decided to start writing a Know Your Tools (KYT) paper about Thug. Please take a look at the KYE/KYT papers published by the Honeynet Project at https://www.honeynet.org/papers). But we were thinking about using a different approach this time. Historically there exists an Honeynet Project KYE/KYT committee which takes care of the quality of the paper through a strong review process. And this is good obviously. But the paper is not public until it is published. Obviously.

But if I take a look at Thug I realize there are a lot of persons out there which are using it daily sometimes in unexpected ways. And their feedback could be useful as well. So the idea is starting writing the paper and update it in the same GitHub tree (https://github.com/buffer/thug). This could allow everyone to easily contribute to the paper through GitHub pull requests. The KYE/KYT committee will still guarantee the high quality of the paper through its review job but this is the first experiment of a collaborative paper. So if you are a Thug user and want to share some of your experiences, tips and tricks you are welcome to contribute!

Just a few highlights about Thug:

• DOM (almost) compliant with W3C DOM Core and HTML specifications (Level 1, 2 and partially 3) and partially compliant with W3C DOM Events and Style specifications
• Google V8 Javascript engine wrapped through PyV8
• Vulnerability modules (ActiveX controls, core browser functionalities, browser plugins)
• Currently 6 IE personalities supported
• Hybrid static/dynamic analysis
• MITRE MAEC native logging format
• HPFeeds and MongoDB logging

The source code is available here.

Feedback and comments welcome.

Have fun!

For challenge 6 of our series (provided by Mahmud Ab Rahman and Ahmad Azizan Idris from the Malaysia Honeynet Project Chapter) we present you with a pcap file that contains network traffic generated by the following scenario: An unsuspecting user opens a compromised web page, which redirects the user’s web browser to a URL of a malicious PDF file. As the PDF plug-in of the browser opens the PDF, the unpatched version of Adobe Acrobat Reader is exploited and, as a result, downloads and silently installs malware on the user’s machine.

We prepared a set of questions that requires you to dive deep into the portable document format. Submit your solution by November 30th 2010. The top three submissions will receive small prizes.

Enjoy!

Christian Seifert
Chief Communications Officer
The Honeynet Project

Forensic Challenge 5 – Log Mysteries is now live! This challenge takes you into the world of virtual systems and confusing log data. Figure out what happened to a virtual server using all the logs from a possibly compromised server.

Challenge 5 has been created by Raffael Marty from the Bay Area Chapter, Anton Chuvakin from the Hawaiian Chapter, and Sebastien Tricaud from the French Chapter. It is a bit more open ended than the last challenges, so I am really looking forward to some creative answers!

Submission deadline is September 30th and we will be announcing winners around October 21st. We have a few small prizes for the top three submission.

Enjoy!

39 UserAgents = [
40     (1,
41      “Internet Explorer 6.0 (Windows 2000)”,
42      “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
43      “Mozilla”,
44      “Microsoft Internet Explorer”,
45      “4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
46      “ie60″,
47     ),
48     (2,
49      “Internet Explorer 6.1 (Windows XP)”,
50      “Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
51      “Mozilla”,
52      “Microsoft Internet Explorer”,
53      “4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
54      “ie61″,
55     ),
56     (3,
57      “Internet Explorer 7.0 (Windows XP)”,
58      “Mozilla/4.0 (Windows; MSIE 7.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)”,
59      “Mozilla”,
60      “Microsoft Internet Explorer”,
61      “4.0 (Windows; MSIE 7.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)”,
62      “ie70″,
63     ),
64     (4,
65      “Internet Explorer 8.0 (Windows XP)”,
66      “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; (R1 1.5); .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
67      “Mozilla”,
68      “Microsoft Internet Explorer”,
69      “4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; (R1 1.5); .NET CLR 1.1.4322; .NET CLR 2.0.50727)”,
70      “ie80″,
71     ),
72 ]

It’s important to realize that each personality was added a tag (i.e. ie80). Taking a look at DOM/Window.py the following code can be seen.

229     def __init_methods(self):
230         for attr in dir(self):
231             prefix = “_Window__window_%s_” % (config.browserTag, )
232             if attr.startswith(prefix):
233                 p = attr.split(prefix)[1]
234                 self.__dict__[‘__cx’].add_global(p, getattr(self, attr))
235                 self.__dict__[‘__cx’].execute(“window.%s = %s;” % (p, p, ))

Let’s consider an example and assume the Internet Explorer 8.0 personality was selected. It’s easy to realize that the prefix would assume the value _Window__window_ie80_. A few simple wrappers were created, one per personality, to each method as shown in the following code.

340     def __window_back(self):
341         “””
342         Returns the window to the previous item in the history.
343         Syntax
344
345         window.back()
346
347         Parameters
348
349         None.
350         “””
351         pass
352
353     def __window_ie60_back(self):
354         self.__window_back()
355
356     def __window_ie61_back(self):
357         self.__window_back()
358
359     def __window_ie70_back(self):
360         self.__window_back()
361
362     def __window_ie80_back(self):
363         self.__window_back()
364
365     def __window_firefox_back(self):
366         self.__window_back()

This is a quite simple situation but what if you want to define addEventListener method just for Firefox personalities and attachEvent just for Internet Explorer ones? Really simple to do!

1191     def __window_attachEvent(self, sEvent, fpNotify):
1192         if dataetc.isevent(sEvent, ‘window’):
1193             self.__dict__[sEvent] = fpNotify
1194
1195     def __window_ie60_attachEvent(self, sEvent, fpNotify):
1196         self.__window_attachEvent(sEvent, fpNotify)
1197
1198     def __window_ie61_attachEvent(self, sEvent, fpNotify):
1199         self.__window_attachEvent(sEvent, fpNotify)
1200
1201     def __window_ie70_attachEvent(self, sEvent, fpNotify):
1202         self.__window_attachEvent(sEvent, fpNotify)
1203
1204     def __window_ie80_attachEvent(self, sEvent, fpNotify):
1205         self.__window_attachEvent(sEvent, fpNotify)
1206
1207
1208     def __window_detachEvent(self, sEvent, fpNotify):
1209         if sEvent in self.__dict__ and self.__dict__[sEvent] == fpNotify:
1210             del self.__dict__[sEvent]
1211
1212     def __window_ie60_detachEvent(self, sEvent, fpNotify):
1213         self.__window_detachEvent(sEvent, fpNotify)
1214
1215     def __window_ie61_detachEvent(self, sEvent, fpNotify):
1216         self.__window_detachEvent(sEvent, fpNotify)
1217
1218     def __window_ie70_detachEvent(self, sEvent, fpNotify):
1219         self.__window_detachEvent(sEvent, fpNotify)
1220
1221     def __window_ie80_detachEvent(self, sEvent, fpNotify):
1222         self.__window_detachEvent(sEvent, fpNotify)
1223
1224
1225     def __window_addEventListener(self, type, listener, useCapture = False):
1226         if dataetc.isevent(type, ‘window’):
1227             self.__dict__[type] = listener
1228
1229     def __window_firefox_addEventListener(self, type, listener, useCapture = False):
1230         self.__window_addEventListener(type, listener, useCapture = False)
1231
1232
1233     def __window_removeEventListener(self, type, listener, useCapture = False):
1234         if type in self.__dict__ and self.__dict__[type] == listener:
1235             del self.__dict__[type]
1236
1237     def __window_firefox_removeEventListener(self, type, listener, useCapture = False):
1238         self.__window_removeEventListener(type, listener, useCapture = False)

Moreover this approach could allow to insert specific code within the wrappers if needed while implementing the method functionalities in the higher level __window_<method_name> wrapper.

2010-08-11 10:44:21+0200 [XmlStream,client] [Malware Sample] MD5: e4736922939a028384522b17e9406474
2010-08-11 10:44:21+0200 [XmlStream,client] [Malware Sample] SHA-1: 920b67cb250abdb593b1104a9922e2468b0fe252
2010-08-11 10:44:21+0200 [XmlStream,client] [Malware Sample] PEHash: 40891becb5ec8780f1c5e51f3971c9fb2cc17dab

Another great step forward was taken. Stay tuned!

In this first post we will take a look at the Window object in DOM/Window.py. During object inizialization, the following code is executed.

156     def __init_context(self):
157         “””
158             Spidermonkey Context initialization.
159         “””
160         document = Document(self)
161         self.__dict__[‘__cx’] = self.__dict__[‘__rt’].new_context(alertlist = [])
162         self.__dict__[‘__sl’] = []
163         self.__dict__[‘__fl’] = [document]
164
165         self.__init_properties(document)
166         self.__init_methods()
167         self.__finalize_context()

Let’s go into further details. First of all Window object properties are initialized through the __init_properties method.

181     def __init_properties(self, document):
182         self.__dict__[‘__cx’].add_global(‘window’, self)
183         self.__dict__[‘__cx’].add_global(‘self’  , self)
184         self.__dict__[‘__cx’].execute(“window.window = window;”)
185
186         self.__dict__[‘__cx’].add_global(‘document’, document)
187         self.__dict__[‘__cx’].execute(“window.document = document;”)
188
189         self.__dict__[‘__cx’].add_global(‘location’, document.location)
190         self.__dict__[‘__cx’].execute(“window.location = location;”)
191
192         self.__dict__[‘__cx’].add_global(“ActiveXObject”, ActiveXObject)
193
194         self.__dict__[‘__cx’].add_global(“navigator”, Navigator())
195         self.__dict__[‘__cx’].execute(“window.navigator = navigator;”)
196
197         self.__dict__[‘__cx’].add_global(“screen”, unknown())
198         self.__dict__[‘__cx’].execute(“window.screen = screen;”)
199
200         if ‘top_window’ in self.__dict__[‘__root’].__dict__:
201             if self.__dict__[‘__referrer’]:
202                 top = self.__dict__[‘__referrer’]
203             else:
204                 top = self.__dict__[‘__root’].top_window
205         else:
206             top = self
207
208         self.__dict__[‘__cx’].add_global(“top”, top)
209         self.__dict__[‘__cx’].execute(“window.top = top;”)
210
211         self.__dict__[‘__cx’].add_global(“parent”, top)
212         self.__dict__[‘__cx’].execute(“window.parent = parent;”)
213
214         self.__dict__[‘__cx’].add_global(“history”, History(document))
215         self.__dict__[‘__cx’].execute(“window.history = history;”)
216
217         self.__dict__[‘__cx’].execute(“window.innerWidth = 400;”)
218         self.__dict__[‘__cx’].execute(“window.innerHeight = 200;”)
219
220         self.__init_undefined_properties()
221
222     def __init_undefined_properties(self):
223         properties = (‘external’, ‘opera’, )
224
225         for prop in properties:
226             self.__dict__[‘__cx’].execute(“window.%s = undefined;” % (prop, ))

The code should be straightforward to understand. The idea beyond it is really simple. Simply stated this code allows Python objects’ variables and methods to be accessible from JS. Let’s move to most interesting stuff. Following the __init_methods method is called.

228     def __init_methods(self):
229         for attr in dir(self):
230             if attr.startswith(‘_Window__window’):
231                 p = attr.split(‘_Window__window_’)[1]
232                 self.__dict__[‘__cx’].add_global(p, getattr(self, attr))
233                 self.__dict__[‘__cx’].execute(“window.%s = %s;” % (p, p, ))

Not so easy to understand? Let’s take a look to the definition of a method.

322     def __window_back(self):
323         “””
324         Returns the window to the previous item in the history.
325         Syntax
326
327         window.back()
328
329         Parameters
330
331         None.
332         “””
333         pass

This is a private class method since its name starts with __. “If you try to call a private method, Python will raise a slightly misleading exception, saying that the method does not exist. Of course it does exist, but it’s private, so it’s not accessible outside the class. Strictly speaking, private methods are accessible outside their class, just not easily accessible. Nothing in Python is truly private; internally, the names of private methods and attributes are mangled and unmangled on the fly to make them seem inaccessible by their given names.” (taken from Dive Into Python). We can access the __window_back method of the Window class by the name _Window__window_back. This is the black magic __init_methods use for initializing methods. It’s quite easy to realize that adding a new method is really easy. All you need is to simply define a method named __window_<window_method_name> and match the signature of such method. How to emulate such method it’s up to you but a simple pass could do the trick.

The last step happens in __finalize_context method.

169     def __finalize_context(self):
170         self.__dict__[‘__cx’].execute(“Event = function(){}”)
171         self.__dict__[‘__cx’].execute(“function CollectGarbage() {};”)
172         self.__dict__[‘__cx’].execute(“function quit() {};”)
173         self.__dict__[‘__cx’].execute(“function prompt() {};”)
174
175         for clsname in dataetc.classlist:
176             inits = {‘window’ : self,
177                      ‘tagName': dataetc.classtotag(clsname),
178                      ‘parser’ : None}
179             self.__dict__[‘__cx’].add_global(clsname, DOMObjectFactory(clsname, inits))

The most interesting code is in lines 175-179. First of all let’s take a look at the DOMObjectFactory code (DOM/ClassFactory.py) which is a genuine Python hack.

3 class DynamicDOMObject(DOMObject):
4     def __init__(self):
5         self.__dict__.update(self.inits)
6         DOMObject.__init__(self, self.window, self.tagName, self.parser)
7
8 def DOMObjectFactory(name, initializers):
9     return type(name, (DynamicDOMObject,), {‘inits’ : initializers})

After reading Python documentation it should be easy to understand how this code works and how it’s able to dynamically add new DOM objects to the context.

type(name, bases, dict)

Return a new type object. This is essentially a dynamic form of the class statement. The name string is the class name and becomes the __name__ attribute; the bases tuple itemizes the base classes and becomes the __bases__ attribute; and the dict dictionary is the namespace containing definitions for class body and becomes the __dict__ attribute. For example, the following two statements create identical type objects:

>>> class X(object):
… a = 1
…
>>> X = type(‘X’, (object,), dict(a=1))

What about the Window event handlers? They are handled with a different mechanism which can be fully understood just by analyzing how the new DOM emulation code preparses the pages deferring code execution until the last possible moment. I’ll analyze such feature in a future post in greater detail. Right now what you have to know is that if the handler for the event <event> is set, the Window attribute on<event> is set and contains the handler code. Once you understand it, the following code in DOM/DOM.py used for event handling should be easy to understand.

171     def get_event_func(self, name, f):
172         begin = str(f).index(‘{‘) + 1
173         s = str(f)[begin:].split(‘}’)
174         script = ‘}’.join(s[:-1]) + s[-1]
175         return script
176
177     def event_handler(self, window, name, f):
178         if name in window.__dict__:
179             try:
180                 script = self.get_event_func(name, f)
181                 window.__dict__[‘__cx’].execute(script)
182             except:
183                 #print str(f)
184                 traceback.print_exc()
185                 pass
186
187     def handle_events(self, window):
188         window.__dict__[‘__warning’] = False
189         self.event_handler(window, ‘onabort’         , window.onabort)
190         self.event_handler(window, ‘onbeforeunload’  , window.onbeforeunload)
191         self.event_handler(window, ‘onblur’          , window.onblur)
192         self.event_handler(window, ‘onchange’        , window.onchange)
193         self.event_handler(window, ‘onclick’         , window.onclick)
194         self.event_handler(window, ‘onclose’         , window.onclose)
195         self.event_handler(window, ‘oncontextmenu’   , window.oncontextmenu)
196         self.event_handler(window, ‘ondragdrop’      , window.ondragdrop)
197         self.event_handler(window, ‘onerror’         , window.onerror)
198         self.event_handler(window, ‘onfocus’         , window.onfocus)
199         self.event_handler(window, ‘onhashchange’    , window.hashchange)
200         self.event_handler(window, ‘onkeydown’       , window.onkeydown)
201         self.event_handler(window, ‘onkeypress’      , window.onkeypress)
202         self.event_handler(window, ‘onkeyup’         , window.onkeyup)
203         self.event_handler(window, ‘onload’          , window.onload)
204         self.event_handler(window, ‘onmousedown’     , window.onmousedown)
205         self.event_handler(window, ‘onmousemove’     , window.onmousemove)
206         self.event_handler(window, ‘onmouseout’      , window.onmouseout)
207         self.event_handler(window, ‘onmouseover’     , window.onmouseover)
208         self.event_handler(window, ‘onmouseup’       , window.onmouseup)
209         self.event_handler(window, ‘onmozorientation’, window.onmozorientation)
210         self.event_handler(window, ‘onpaint’         , window.onpaint)
211         self.event_handler(window, ‘onpopstate’      , window.onpopstate)
212         self.event_handler(window, ‘onreset’         , window.onreset)
213         self.event_handler(window, ‘onresize’        , window.onresize)
214         self.event_handler(window, ‘onscroll’        , window.onscroll)
215         self.event_handler(window, ‘onselect’        , window.onselect)
216         self.event_handler(window, ‘onsubmit’        , window.onsubmit)
217         self.event_handler(window, ‘onunload’        , window.onunload)
218         self.event_handler(window, ‘onpageshow’      , window.onpageshow)
219         self.event_handler(window, ‘onpagehide’      , window.onpagehide)
220         window.__dict__[‘__warning’] = True

Let’s start with a benign domain not tracked by TIP.

buffer@alnitak ~ $ time wget http://xxxx.xxxx.xx/api/check/domain/test@@it/
HTTP request sent, awaiting response… 410 GONE
2010-07-22 17:46:58 ERROR 410: GONE.

real    0m0.017s
user    0m0.001s
sys    0m0.001s

Now let’s move to a malicious domain tracked by TIP.

buffer@alnitak ~ $ time wget http://xxxx.xxxx.xx/api/check/domain/hazelpay@@ru/
HTTP request sent, awaiting response… 409 CONFLICT
2010-07-22 17:47:07 ERROR 409: CONFLICT.

real    0m0.022s
user    0m0.000s
sys    0m0.002s

First of all, a new TIP Collector module named Malware was integrated and currently it handles data coming from GLSandbox, a sandbox for automated malware analysis written by Guido Landi. Other than just analyzing malware samples behavior, the idea is to collect additional data coming from such analysis too. An example of such interesting data is related to C&C identification which can be automatically handled by a botnet monitoring tool for further analysis. Another example is related to information about domains which could lead to the identification of new fast-flux domains.GLSandbox code is currently not public but plans exist to release it in the next future. A search engine was integrated in TIP in the last version! The idea is to index the database in order to be able to search into it with great efficiency and performance. In order to implement it, Haystack was used. The first tests were done using Apache Solr (deployed as Apache Tomcat application) as backend and confirm it works like a charm!  A new REST API was designed and realized in order to be able to more easily search and share data with other users and/or applications. The API was realized using Django-Piston and supports OAuth authentication. Moreover the last version of TIP supports Django 1.2 and stops supporting previous versions (due to some incompatible changes between versions 1.1 and 1.2) and introduces support to migrations using South in order to more easily make changes to the database schema while developing.

A lot of new cool features, a lot of upcoming cool ones! Stay tuned!

Note that our Chinese speaking chapters (Julia Cheng from the Taiwanese Chapter, Jianwei Zhuge from the Chinese Chapter and Roland Cheung from the Hongkong Chapter) have taken great initiative and translated the challenge into Chinese, which is available from the simplified Chinese and traditional Chinese pages (will be posted by EOD today.)

With this challenge, we are getting on a firm 2 month cycle. You will have one month to submit (deadline is June 30th 2010) and results will be released approximately 3 weeks later. Small prizes will be awarded to the top three submissions.

Enjoy the challenge!