I spent a good deal of time adding bots and phones until even myself could not tell what that agent string meant. What is left is a pile of unintelligible mess.

With that said, this patch supports every bot and feature phone that provided some useful info and occured more than a few times in a day worth of logs on a busy website.

One possible improvement would bo to assign types to browsers, so I can easily discern the bots from the phones from the desktops.

Is there interest in updating support IE Compatibilty modes? Currently all compatibilty mode user agent strings return as showing Internet Explorer 7.0

Sample user agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET4.0C; .NET4.0E; InfoPath.3; MS-RTC LM 8)

Current detection: {'os': {'version': '7', 'name': 'Windows'}, 'browser': {'version': '7.0', 'name': 'Microsoft Internet Explorer'}}

Proposed detection: {'os': {'version': '7', 'name': 'Windows'}, 'browser': {'version': '8.0 Compatibility View', 'name': 'Microsoft Internet Explorer'}}

Description

This attempts to address situations where globals() changed during iteration.

    import httpagentparser
  File "/.virtualenv/lib/python3.6/site-packages/httpagentparser/__init__.py", line 647, in <module>
    detectorshub = DetectorsHub()
  File "/.virtualenv/lib/python3.6/site-packages/httpagentparser/__init__.py", line 21, in __init__
    self.registerDetectors()
  File "/.virtualenv/lib/python3.6/site-packages/httpagentparser/__init__.py", line 34, in registerDetectors
    detectors = [v() for v in globals().values() if DetectorBase in getattr(v, '__mro__', [])]
  File "/.virtualenv/lib/python3.6/site-packages/httpagentparser/__init__.py", line 34, in <listcomp>
    detectors = [v() for v in globals().values() if DetectorBase in getattr(v, '__mro__', [])]
RuntimeError: dictionary changed size during iteration

It occurred during start up of a multi-threaded Flask app.

Approach

Iterate a list copy of globals().values() instead.

It will be nice to add a new type of "browser" for bot/spiders like this:

"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

Added support for Opera Mobile on Android devices. Had to fix the incorrect assumption that if the browser is not "Mobile Safari", it is a tablet. For alternative browsers, it will be Opera, Dolphin, Firefox, ... but no conclusion can be drawn about the fact that it is a phone or a tablet.

pip install/easy_install called in a virtualenv -p python3

I checked http://guide.python-distribute.org/creation.html your setup is OK I checked your MANIFEST.in OK I checked the presence of README.rst it is OK I checked the doc, you play by the rules.

I am clueless. Removing l7 in setup.py and doing python /home/jul/src/parse3/build/httpagentparser/setup.py install works however.

Downloading/unpacking httpagentparser Real name of requirement httpagentparser is httpagentparser Downloading httpagentparser-1.1.0.tar.gz Running setup.py egg_info for package httpagentparser Traceback (most recent call last): File "", line 14, in File "/home/jul/src/parse3/build/httpagentparser/setup.py", line 7, in long_description=file("README.rst").read(), NameError: name 'file' is not defined Complete output from command python setup.py egg_info: Traceback (most recent call last):

File "", line 14, in

File "/home/jul/src/parse3/build/httpagentparser/setup.py", line 7, in

long_description=file("README.rst").read(),

NameError: name 'file' is not defined

For example, this UA spotted in the wild:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618; .NET4.0C)

Detects as:

{'os': {'version': 'XP', 'name': 'Windows'}, 'browser': {'version': '6.0', 'name': 'Microsoft Internet Explorer'}}

When it's actually IE8.

I traced the problem to the default getVersion implementation, which favours the last occurrence of MSIE in the version string.

I write a new one on the MSIE class, which favours the first one instead:

class MSIE(Browser):
    look_for = "MSIE" 
    skip_if_found = ["Opera"]
    name = "Microsoft Internet Explorer"
    version_splitters = [" ", ";"]
    def getVersion(self, agent):
        parts = agent.split(self.look_for + self.version_splitters[0])
        return parts[min(len(parts) - 1, 1)].split(self.version_splitters[1])[0].strip()

My one grabs the FIRST version detected rather than the last one.

I haven't provided this as a pull request, because the tests for MSIE are non-existent currently, so I have no idea if my change breaks a bunch of UAs in the wild.

I added a test case for this UA at least:

    ('Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618; .NET4.0C)',
    ('Windows XP', 'Microsoft Internet Explorer 8.0'),
    {'os': {'version': 'XP', 'name': 'Windows'}, 'browser': {'version': '8.0', 'name': 'Microsoft Internet Explorer'}},),

I'd suggest adding a bunch of MSIE test cases first.

looks like for the Opera useragent Opera/9.80 (Windows NT 6.1; U; Edition Next; en) Presto/2.10.229 Version/12.00 we're getting Opera v9.80 reported, not Version 12... not sure if it's an Opera or httpagentparser issue...

These user agents parse fine: Opera/9.20 (Windows NT 6.0; U; en), Opera 9, Windows Vista Opera/9.00 (Windows NT 5.1; U; en), Opera 9, Windows XP

These fail to parse the version correctly, returning something like "Opera Mozilla/4.0" for browser: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.50 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.0 Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Opera 7.02 [en]

This one fails to detect the browser altogether: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127 Netscape/8.1

Hei Shon,

First of all, thanks for sharing this! I was looking for an agent parser, and found out your lib in python.org.

I don't know if you are aware of this or not, but I was running some tests and found some inconsistent parsing on the following user-agent:

import httpagentparser s = "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24" print httpagentparser.detect(s) {'os': {'version': 'NT 5.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24', 'name': 'Windows'}, 'browser': {'version': '11.0.696.60', 'name': 'Chrome'}}

My computer is running Windows XP SP3, using Chrome 11.0.696.60. All other tests (Linux and Mac) went ok, just so you know.

I took the liberty to write a patch for this, so I pulling it to your code later on.

Thanks,

Added detection for iPad and Android phone/tablet.

• iPad: Simply copy-pasted the iPhone detection and changed the string. Looks like it works.
• Android: Added "Mobile Safari" string check, as described here: http://stackoverflow.com/questions/5341637/how-do-detect-android-tablets-in-general-useragent

Hi,

When I try it with the user agent Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; Microsoft Outlook 16.0.5257; ms-office; MSOffice 16)

Instead of the correct output MS Office 16 (user agent when Outlook load a external image) , I get the result: Microsoft Internet Explorer 7.0

Do you know why ?

It would be very helpful if you could tag releases as well. This would enable distributions who want to run your tests to fetch the package from GitHub instead of Pypi, where the tests are excluded.

Thanks

I manually looked through some logs and feed it into httpagendparser and found several wrongly identified user agents, including high profiles google bot:

>>> httpagentparser.detect("Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)")
{'platform': {'name': None, 'version': None}}

>>> httpagentparser.detect("Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")
{'platform': {'name': 'Android', 'version': '6.0.1'}, 'os': {'name': 'Linux'}, 'bot': False, 'dist': {'name': 'Android', 'version': '6.0.1'}, 'browser': {'name': 'Chrome', 'version': '41.0.2272.96'}}

>>> httpagentparser.detect("Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0")
{'platform': {'name': 'Linux', 'version': None}, 'os': {'name': 'Linux'}, 'bot': False, 'browser': {'name': 'Firefox', 'version': '38.0'}}

>>> httpagentparser.detect("crawler_eb_germany")
{'platform': {'name': None, 'version': None}}

>>> httpagentparser.detect("Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)")
{'platform': {'name': None, 'version': None}}

>>> httpagentparser.detect("Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)")
{'platform': {'name': None, 'version': None}}

>>> httpagentparser.detect("Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)")
{'platform': {'name': None, 'version': None}}

>>> httpagentparser.detect("ZoominfoBot (zoominfobot at zoominfo dot com)")
{'platform': {'name': None, 'version': None}}

Trying to build a regex, by adding all "look_for", Example:-

if we add all "look_for" of "os"(known_type)

"Windows Phone OS|Windows Phone|Symbian|SymbianOS|CrOS|PlayStation|PLAYSTATION|Windows|Series40|iPhone|iPad|BlackBerry|Macintosh|Linux"

Then with this regex we can jump to respective "detector" class on which we have to run detect as appose to iterating.

some profiling:- With changes

0.00596690177917 // import httpagentparser 

0.000109910964966 // httpagentparser.detect(ua)
         13214 function calls (13037 primitive calls) in 0.006 seconds

without changes

0.00134301185608  // import httpagentparser 

0.000181913375854  // httpagentparser.detect(ua)
         693 function calls in 0.002 seconds

So i am actually making it slower overall

But if u run on in a loop of 100000 with changes

0.00691199302673 // import 
2.00802302361 // loop on httpagentparser.detect(ua)
         3513180 function calls (3513003 primitive calls) in 2.015 seconds

without changes

0.00165581703186 // import 
6.87254285812  //loop on httpagentparser.detect(ua)
         22500469 function calls in 6.874 seconds

There is difference in this case

This idea, i wanted to try out, code is messy but i can clean it a bit if u see potential in it. But agree, regex is always a little tricky and error prone.

Here is the user agent:

Mozilla/5.0 (Windows Phone 10.0; Android 4.2.1; NOKIA; Lumia 920) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Mobile Safari/537.36 Edge/13.10586