Out Of The Box

A new improvement in PHoneyC DOM emulation code was committed in SVN r1624. The idea is to better emulate the DOM behaviour depending on the selected browser personality. Let’s take a look at the code starting from the personalities definition in config.py.

39 UserAgents = [
40     (1,
41      "Internet Explorer 6.0 (Windows 2000)",
42      "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
43      "Mozilla",
44      "Microsoft Internet Explorer",
45      "4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
46      "ie60",
47     ),
48     (2,
49      "Internet Explorer 6.1 (Windows XP)",
50      "Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
51      "Mozilla",
52      "Microsoft Internet Explorer",
53      "4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
54      "ie61",
55     ),
56     (3,
57      "Internet Explorer 7.0 (Windows XP)",
58      "Mozilla/4.0 (Windows; MSIE 7.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)",
59      "Mozilla",
60      "Microsoft Internet Explorer",
61      "4.0 (Windows; MSIE 7.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)",
62      "ie70",
63     ),
64     (4,
65      "Internet Explorer 8.0 (Windows XP)",
66      "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; (R1 1.5); .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
67      "Mozilla",
68      "Microsoft Internet Explorer",
69      "4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; (R1 1.5); .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
70      "ie80",
71     ),
72 ]

It’s important to realize that each personality was added a tag (i.e. ie80). Taking a look at DOM/Window.py the following code can be seen.

229     def __init_methods(self):
230         for attr in dir(self):
231             prefix = “_Window__window_%s_” % (config.browserTag, )
232             if attr.startswith(prefix):
233                 p = attr.split(prefix)[1]
234                 self.__dict__['__cx'].add_global(p, getattr(self, attr))
235                 self.__dict__['__cx'].execute(“window.%s = %s;” % (p, p, ))

Let’s consider an example and assume the Internet Explorer 8.0 personality was selected. It’s easy to realize that the prefix would assume the value _Window__window_ie80_. A few simple wrappers were created, one per personality, to each method as shown in the following code.

340     def __window_back(self):
341         “”"
342         Returns the window to the previous item in the history.
343         Syntax
344
345         window.back()
346
347         Parameters
348
349         None.
350         “”"
351         pass
352
353     def __window_ie60_back(self):
354         self.__window_back()
355
356     def __window_ie61_back(self):
357         self.__window_back()
358
359     def __window_ie70_back(self):
360         self.__window_back()
361
362     def __window_ie80_back(self):
363         self.__window_back()
364
365     def __window_firefox_back(self):
366         self.__window_back()

This is a quite simple situation but what if you want to define addEventListener method just for Firefox personalities and attachEvent just for Internet Explorer ones? Really simple to do!

1191     def __window_attachEvent(self, sEvent, fpNotify):
1192         if dataetc.isevent(sEvent, ‘window’):
1193             self.__dict__[sEvent] = fpNotify
1194
1195     def __window_ie60_attachEvent(self, sEvent, fpNotify):
1196         self.__window_attachEvent(sEvent, fpNotify)
1197
1198     def __window_ie61_attachEvent(self, sEvent, fpNotify):
1199         self.__window_attachEvent(sEvent, fpNotify)
1200
1201     def __window_ie70_attachEvent(self, sEvent, fpNotify):
1202         self.__window_attachEvent(sEvent, fpNotify)
1203
1204     def __window_ie80_attachEvent(self, sEvent, fpNotify):
1205         self.__window_attachEvent(sEvent, fpNotify)
1206
1207
1208     def __window_detachEvent(self, sEvent, fpNotify):
1209         if sEvent in self.__dict__ and self.__dict__[sEvent] == fpNotify:
1210             del self.__dict__[sEvent]
1211
1212     def __window_ie60_detachEvent(self, sEvent, fpNotify):
1213         self.__window_detachEvent(sEvent, fpNotify)
1214
1215     def __window_ie61_detachEvent(self, sEvent, fpNotify):
1216         self.__window_detachEvent(sEvent, fpNotify)
1217
1218     def __window_ie70_detachEvent(self, sEvent, fpNotify):
1219         self.__window_detachEvent(sEvent, fpNotify)
1220
1221     def __window_ie80_detachEvent(self, sEvent, fpNotify):
1222         self.__window_detachEvent(sEvent, fpNotify)
1223
1224
1225     def __window_addEventListener(self, type, listener, useCapture = False):
1226         if dataetc.isevent(type, ‘window’):
1227             self.__dict__[type] = listener
1228
1229     def __window_firefox_addEventListener(self, type, listener, useCapture = False):
1230         self.__window_addEventListener(type, listener, useCapture = False)
1231
1232
1233     def __window_removeEventListener(self, type, listener, useCapture = False):
1234         if type in self.__dict__ and self.__dict__[type] == listener:
1235             del self.__dict__[type]
1236
1237     def __window_firefox_removeEventListener(self, type, listener, useCapture = False):
1238         self.__window_removeEventListener(type, listener, useCapture = False)

Moreover this approach could allow to insert specific code within the wrappers if needed while implementing the method functionalities in the higher level __window_<method_name> wrapper.

“Dionaea is meant to be a Nepenthes successor, embedding Python as scripting language, using libemu to detect shellcodes, supporting IPv6 and TLS” (taken from Dionaea homepage). Besides being the most interesting project for trapping malware exploiting vulnerabilities, Dionaea supports a really cool feature which allows it to log to XMPP services as described here. TIP now exploits this feature receiving and storing such logs (really thanks to Markus Koetter for his help and support). Just an example of what happened today…

2010-08-11 10:44:21+0200 [XmlStream,client] [Malware Sample] MD5: e4736922939a028384522b17e9406474
2010-08-11 10:44:21+0200 [XmlStream,client] [Malware Sample] SHA-1: 920b67cb250abdb593b1104a9922e2468b0fe252
2010-08-11 10:44:21+0200 [XmlStream,client] [Malware Sample] PEHash: 40891becb5ec8780f1c5e51f3971c9fb2cc17dab

Another great step forward was taken. Stay tuned!

A few weeks ago I started reviewing the PHoneyC DOM emulation code and realized it was turning to be hard to maintain and debug due to a huge amount of undocumented (and sometimes awful) hacks. For this reason I decided it was time to patch (and sometimes rewrite from scratch) such code. These posts will describe how the new DOM emulation code will work. The patch is not available right now since I’m testing the code but plans exists to commit it in the PHoneyC SVN in the next days.

In this first post we will take a look at the Window object in DOM/Window.py. During object inizialization, the following code is executed.

156     def __init_context(self):
157         “”"
158             Spidermonkey Context initialization.
159         “”"
160         document = Document(self)
161         self.__dict__['__cx'] = self.__dict__['__rt'].new_context(alertlist = [])
162         self.__dict__['__sl'] = []
163         self.__dict__['__fl'] = [document]
164
165         self.__init_properties(document)
166         self.__init_methods()
167         self.__finalize_context()

Let’s go into further details. First of all Window object properties are initialized through the __init_properties method.

181     def __init_properties(self, document):
182         self.__dict__['__cx'].add_global(‘window’, self)
183         self.__dict__['__cx'].add_global(‘self’  , self)
184         self.__dict__['__cx'].execute(“window.window = window;”)
185
186         self.__dict__['__cx'].add_global(‘document’, document)
187         self.__dict__['__cx'].execute(“window.document = document;”)
188
189         self.__dict__['__cx'].add_global(‘location’, document.location)
190         self.__dict__['__cx'].execute(“window.location = location;”)
191
192         self.__dict__['__cx'].add_global(“ActiveXObject”, ActiveXObject)
193
194         self.__dict__['__cx'].add_global(“navigator”, Navigator())
195         self.__dict__['__cx'].execute(“window.navigator = navigator;”)
196
197         self.__dict__['__cx'].add_global(“screen”, unknown())
198         self.__dict__['__cx'].execute(“window.screen = screen;”)
199
200         if ‘top_window’ in self.__dict__['__root'].__dict__:
201             if self.__dict__['__referrer']:
202                 top = self.__dict__['__referrer']
203             else:
204                 top = self.__dict__['__root'].top_window
205         else:
206             top = self
207
208         self.__dict__['__cx'].add_global(“top”, top)
209         self.__dict__['__cx'].execute(“window.top = top;”)
210
211         self.__dict__['__cx'].add_global(“parent”, top)
212         self.__dict__['__cx'].execute(“window.parent = parent;”)
213
214         self.__dict__['__cx'].add_global(“history”, History(document))
215         self.__dict__['__cx'].execute(“window.history = history;”)
216
217         self.__dict__['__cx'].execute(“window.innerWidth = 400;”)
218         self.__dict__['__cx'].execute(“window.innerHeight = 200;”)
219
220         self.__init_undefined_properties()
221
222     def __init_undefined_properties(self):
223         properties = (‘external’, ‘opera’, )
224
225         for prop in properties:
226             self.__dict__['__cx'].execute(“window.%s = undefined;” % (prop, ))

The code should be straightforward to understand. The idea beyond it is really simple. Simply stated this code allows Python objects’ variables and methods to be accessible from JS. Let’s move to most interesting stuff. Following the __init_methods method is called.

228     def __init_methods(self):
229         for attr in dir(self):
230             if attr.startswith(‘_Window__window’):
231                 p = attr.split(‘_Window__window_’)[1]
232                 self.__dict__['__cx'].add_global(p, getattr(self, attr))
233                 self.__dict__['__cx'].execute(“window.%s = %s;” % (p, p, ))

Not so easy to understand? Let’s take a look to the definition of a method.

322     def __window_back(self):
323         “”"
324         Returns the window to the previous item in the history.
325         Syntax
326
327         window.back()
328
329         Parameters
330
331         None.
332         “”"
333         pass

This is a private class method since its name starts with __. “If you try to call a private method, Python will raise a slightly misleading exception, saying that the method does not exist. Of course it does exist, but it’s private, so it’s not accessible outside the class. Strictly speaking, private methods are accessible outside their class, just not easily accessible. Nothing in Python is truly private; internally, the names of private methods and attributes are mangled and unmangled on the fly to make them seem inaccessible by their given names.” (taken from Dive Into Python). We can access the __window_back method of the Window class by the name _Window__window_back. This is the black magic __init_methods use for initializing methods. It’s quite easy to realize that adding a new method is really easy. All you need is to simply define a method named __window_<window_method_name> and match the signature of such method. How to emulate such method it’s up to you but a simple pass could do the trick.

The last step happens in __finalize_context method.

169     def __finalize_context(self):
170         self.__dict__['__cx'].execute(“Event = function(){}”)
171         self.__dict__['__cx'].execute(“function CollectGarbage() {};”)
172         self.__dict__['__cx'].execute(“function quit() {};”)
173         self.__dict__['__cx'].execute(“function prompt() {};”)
174
175         for clsname in dataetc.classlist:
176             inits = {‘window’ : self,
177                      ‘tagName’: dataetc.classtotag(clsname),
178                      ‘parser’ : None}
179             self.__dict__['__cx'].add_global(clsname, DOMObjectFactory(clsname, inits))

The most interesting code is in lines 175-179. First of all let’s take a look at the DOMObjectFactory code (DOM/ClassFactory.py) which is a genuine Python hack.

3 class DynamicDOMObject(DOMObject):
4     def __init__(self):
5         self.__dict__.update(self.inits)
6         DOMObject.__init__(self, self.window, self.tagName, self.parser)
7
8 def DOMObjectFactory(name, initializers):
9     return type(name, (DynamicDOMObject,), {‘inits’ : initializers})

After reading Python documentation it should be easy to understand how this code works and how it’s able to dynamically add new DOM objects to the context.

type(name, bases, dict)

Return a new type object. This is essentially a dynamic form of the class statement. The name string is the class name and becomes the __name__ attribute; the bases tuple itemizes the base classes and becomes the __bases__ attribute; and the dict dictionary is the namespace containing definitions for class body and becomes the __dict__ attribute. For example, the following two statements create identical type objects:

>>> class X(object):
… a = 1
…
>>> X = type(‘X’, (object,), dict(a=1))

What about the Window event handlers? They are handled with a different mechanism which can be fully understood just by analyzing how the new DOM emulation code preparses the pages deferring code execution until the last possible moment. I’ll analyze such feature in a future post in greater detail. Right now what you have to know is that if the handler for the event <event> is set, the Window attribute on<event> is set and contains the handler code. Once you understand it, the following code in DOM/DOM.py used for event handling should be easy to understand.

171     def get_event_func(self, name, f):
172         begin = str(f).index(‘{‘) + 1
173         s = str(f)[begin:].split(‘}’)
174         script = ‘}’.join(s[:-1]) + s[-1]
175         return script
176
177     def event_handler(self, window, name, f):
178         if name in window.__dict__:
179             try:
180                 script = self.get_event_func(name, f)
181                 window.__dict__['__cx'].execute(script)
182             except:
183                 #print str(f)
184                 traceback.print_exc()
185                 pass
186
187     def handle_events(self, window):
188         window.__dict__['__warning'] = False
189         self.event_handler(window, ‘onabort’         , window.onabort)
190         self.event_handler(window, ‘onbeforeunload’  , window.onbeforeunload)
191         self.event_handler(window, ‘onblur’          , window.onblur)
192         self.event_handler(window, ‘onchange’        , window.onchange)
193         self.event_handler(window, ‘onclick’         , window.onclick)
194         self.event_handler(window, ‘onclose’         , window.onclose)
195         self.event_handler(window, ‘oncontextmenu’   , window.oncontextmenu)
196         self.event_handler(window, ‘ondragdrop’      , window.ondragdrop)
197         self.event_handler(window, ‘onerror’         , window.onerror)
198         self.event_handler(window, ‘onfocus’         , window.onfocus)
199         self.event_handler(window, ‘onhashchange’    , window.hashchange)
200         self.event_handler(window, ‘onkeydown’       , window.onkeydown)
201         self.event_handler(window, ‘onkeypress’      , window.onkeypress)
202         self.event_handler(window, ‘onkeyup’         , window.onkeyup)
203         self.event_handler(window, ‘onload’          , window.onload)
204         self.event_handler(window, ‘onmousedown’     , window.onmousedown)
205         self.event_handler(window, ‘onmousemove’     , window.onmousemove)
206         self.event_handler(window, ‘onmouseout’      , window.onmouseout)
207         self.event_handler(window, ‘onmouseover’     , window.onmouseover)
208         self.event_handler(window, ‘onmouseup’       , window.onmouseup)
209         self.event_handler(window, ‘onmozorientation’, window.onmozorientation)
210         self.event_handler(window, ‘onpaint’         , window.onpaint)
211         self.event_handler(window, ‘onpopstate’      , window.onpopstate)
212         self.event_handler(window, ‘onreset’         , window.onreset)
213         self.event_handler(window, ‘onresize’        , window.onresize)
214         self.event_handler(window, ‘onscroll’        , window.onscroll)
215         self.event_handler(window, ‘onselect’        , window.onselect)
216         self.event_handler(window, ‘onsubmit’        , window.onsubmit)
217         self.event_handler(window, ‘onunload’        , window.onunload)
218         self.event_handler(window, ‘onpageshow’      , window.onpageshow)
219         self.event_handler(window, ‘onpagehide’      , window.onpagehide)
220         window.__dict__['__warning'] = True

Today I was in need for fun and so I started adding a new API call which allows to check if a domain is malicious or not. The check avoids to hit the database at all but just queries the search index. The results I got are quite surprising. Take a look at it considering  that code 409 means ‘Object already exists’ while code 410 means ‘Object does not exist’.

Let’s start with a benign domain not tracked by TIP.

buffer@alnitak ~ $ time wget http://xxxx.xxxx.xx/api/check/domain/test@@it/
HTTP request sent, awaiting response… 410 GONE
2010-07-22 17:46:58 ERROR 410: GONE.

real    0m0.017s
user    0m0.001s
sys    0m0.001s

Now let’s move to a malicious domain tracked by TIP.

buffer@alnitak ~ $ time wget http://xxxx.xxxx.xx/api/check/domain/hazelpay@@ru/
HTTP request sent, awaiting response… 409 CONFLICT
2010-07-22 17:47:07 ERROR 409: CONFLICT.

real    0m0.022s
user    0m0.000s
sys    0m0.002s

It’s really a long time I do not post about TIP. The good news is that TIP is starting growing really fast and this is mainly due to its modular design which allows to plug different kind of tracking modules with minimum effort. In this post I’ll provide a brief overview of the new still integrated features and the upcoming ones.

First of all, a new TIP Collector module named Malware was integrated and currently it handles data coming from GLSandbox, a sandbox for automated malware analysis written by Guido Landi. Other than just analyzing malware samples behavior, the idea is to collect additional data coming from such analysis too. An example of such interesting data is related to C&C identification which can be automatically handled by a botnet monitoring tool for further analysis. Another example is related to information about domains which could lead to the identification of new fast-flux domains.GLSandbox code is currently not public but plans exist to release it in the next future. A search engine was integrated in TIP in the last version! The idea is to index the database in order to be able to search into it with great efficiency and performance. In order to implement it, Haystack was used. The first tests were done using Apache Solr (deployed as Apache Tomcat application) as backend and confirm it works like a charm!  A new REST API was designed and realized in order to be able to more easily search and share data with other users and/or applications. The API was realized using Django-Piston and supports OAuth authentication. Moreover the last version of TIP supports Django 1.2 and stops supporting previous versions (due to some incompatible changes between versions 1.1 and 1.2) and introduces support to migrations using South in order to more easily make changes to the database schema while developing.

A lot of new cool features, a lot of upcoming cool ones! Stay tuned!

Challenge 4 of the Honeynet Project Forensic Challenge – titled “VoIP” – is now live. This challenge 4 – provided by Ben Reardon from the Australian and Sjur Eivind Usken from Norwegian Chapter – takes you into the realm of voice communications on the Internet. VoIP with SIP is becoming the de-facto standard. As this technology becomes more common, malicious parties have more opportunities and stronger motives to take control of these systems to conduct nefarious activities. This Challenge is designed to examine and explore some of attributes of the SIP and RTP protocols.

Note that our Chinese speaking chapters (Julia Cheng from the Taiwanese Chapter, Jianwei Zhuge from the Chinese Chapter and Roland Cheung from the Hongkong Chapter) have taken great initiative and translated the challenge into Chinese, which is available from the simplified Chinese and traditional Chinese pages (will be posted by EOD today.)

With this challenge, we are getting on a firm 2 month cycle. You will have one month to submit (deadline is June 30th 2010) and results will be released approximately 3 weeks later. Small prizes will be awarded to the top three submissions.

Enjoy the challenge!

Honeynet Project Challenge 2010/3 – “Banking Troubles” has just been posted and is to investigate a memory image of an infected virtual machine. The challenge has been provided by Josh Smith and Matt Cote from The Rochester Institute of Technology Chapter, Angelo Dell’Aera from the Italian Chapter and Nicolas Collery from the Singapore Chapter.

Submit your solution at http://www.honeynet.org/challenge2010/ by 17:00 EST, Sunday, April 18th 2010. Results will be released on Wednesday, May 5th 2010. Small prizes will be awarded to the top three submissions.

Skill Level: Difficult

UPDATE: Submission deadline extended to Monday, 26th of April 2010

Challenge 2 of the Honeynet Project Forensic Challenge has just been posted. The challenge has been provided by Nicolas Collery from the Singapore Chapter and Guillaume Arcas from the French Chapter and is titled browsers under attack.

Submission deadline is March 1st and results will be released on Monday, March 15th 2010. Small prizes will be awarded to the top three submissions.

Have fun!

UPDATE: Submission deadline extended to Monday, 8th of March 2010

About two months ago I started contributing PhoneyC, a pure Python honeyclient implementation originally developed by Jose Nazario. The perception is that our development efforts are moving on the right track. The code can be downloaded here. If you’re interested take a look at the different development branches and give us your feedback. Moreover if you’re interested in technical details about PhoneyC please read this paper by Jose Nazario.

After several years without any Honeynet Project Challenges, there will finally be new Forensic Challenges starting next Monday (January 18th, 2010). Here is the official announcement.

I am very happy to announce the Honeynet Project Forensic Challenge 2010. The purpose of the Forensic Challenges is to take learning one step farther. Instead of having the Honeynet Project analyze attacks and share their findings, Forensic Challenges give the security community the opportunity to analyze attacks and share their findings. In the end, individuals and organizations not only learn about threats, but also learn how to analyze them. Even better, individuals can access the write-ups from other individuals, and learn about new tools and techniques for analyzing attacks. Best of all, the attacks of the Forensic Challenge are attacks encountered in the wild, real hacks, provided by our members.
It has been several years since we provided Forensic Challenges and with the Forensic Challenge 2010, we will provide desperately needed upgrades. The Forensic Challenge 2010 will include a mixture of server-side attacks on the latest operating systems and services, attacks on client-side attacks that emerged in the past few years, attacks on VoiP systems, web applications, etc. At the end of challenge, we will provide a sample solution created by our members using the state-of-the-art tools that are publicly available, such as libemu and dionaea.
The first challenge (of several for 2010) will be posted on our Forensic Challenges web site on Monday, January 18th 2010. We will be open to submissions for about two weeks and announce the winners by February 15th 2010. This year, we will also award the top three submissions with prizes! Please check the web site on Monday, January 18th 2010 for further details…

Christian Seifert