Replace the non pythonic method "inspect_module" in manager/utils with metaclasses in order to read the models and crawlers modules written by users. :-)

Now we're doing the http requests without any delay. It can be a problem when sending a thousands of requests to the same server.

The solution is make delayed http requests when we're are overloading a external server (Consider the algorithm to decide this).

Put the delay time constant in a config file.

We've a simple dsl designed and we're able to compile it into a scraper classes. Now we can finish the integration of the run-time generated scrapers with the crawlers.

Write more tests and more complex dsl templates.

Evaluate the possibility of use difflib in order to recognize similar html pages.

http://docs.python.org/library/difflib.html

Write some tests to check if it works properly and relatively fast. Then we can write a "SmartCrawler" class wich crawls the web searching for similar pages.

I'm using PyQuery, and I get wrong encode detection for this page:

http://www1.abracom.org.br/cms/opencms/abracom/pt/associados/resultado_busca.html?nomeArq=0148.html

The problem is that the html has this meta tag:

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

But the page is actually utf-8

I get this info from the response headers:

Connection:close
Content-Length:29187
Content-Type:text/html;charset=UTF-8
Date:Fri, 11 Jul 2014 23:21:04 GMT
Last-Modified:Fri, 11 Jul 2014 23:21:05 GMT
Server:OpenCms/7.5.4

That's how the browser (chrome) is able to guess the right encoding and display the page with the right encoding. I work in a place that have to deal with a lot of different kinds of pages, and I can tell this is far from a rare case (especially in brazilian portuguese websites), so it would be nice to fix this in crawley.

So far I saw two solutions as proposed in this answer in SO, using chardet module or UnicodeDammit (from BeautifulSoup).

I've develop, locally, these two alternatives and tested them with PyQuery, seems to fix the problem.

I would like to hear your opinion on this issue and if you want, I can submit one of those solutions.

BTW, good work in building crawley, I'm having a very nice time using it! Hope I can contribute somehow.

https://docs.python.org/2/library/urlparse.html#urlparse.urljoin provides a robust way to make a relative url into a absolute one.

This fixes some issues like this one:

When accessing this url: http://www1.abracom.org.br/cms/opencms/abracom/pt/associados/

We find relative links like this: resultado_busca.html?letra=a

The browser (chrome) build the absolute url like this: http://www1.abracom.org.br/cms/opencms/abracom/pt/associados/resultado_busca.html?letra=a

But crawley build the url like this: http://www1.abracom.org.br/resultado_busca.html?letra=a

urljoin fixes the issue, keeping the right behavior for /relativeurl:

In a hypothetical page http://mydomain.com/my/web/page.html:

'/relativeurl.html' link should become 'http://mydomain.com/relativeurl.html'

and

'relativeurl.html' link should become 'http://mydomain.com/my/web/relativeurl.html'

I tryed to use the shell command to test my xpaths, but it does'nt work.

$ crawley shell http://somewebsite.com/index.html Traceback (most recent call last): File "/home/maik/.virtualenvs/crawley/bin/crawley", line 4, in manage() File "/home/maik/.virtualenvs/crawley/local/lib/python2.7/site-packages/crawley/manager/init.py", line 25, in manage run_cmd(sys.argv) File "/home/maik/.virtualenvs/crawley/local/lib/python2.7/site-packages/crawley/manager/init.py", line 18, in run_cmd cmd.checked_execute() File "/home/maik/.virtualenvs/crawley/local/lib/python2.7/site-packages/crawley/manager/commands/command.py", line 50, in checked_execute self.execute() File "/home/maik/.virtualenvs/crawley/local/lib/python2.7/site-packages/crawley/manager/commands/shell.py", line 30, in execute response = crawler._get_data(url) AttributeError: 'BaseCrawler' object has no attribute '_get_data'