Installing robobrowser with pip 6.0+ was failing due to: TypeError: parse_requirements() missing 1 required keyword argument: 'session'

Fixed setup.py to include a session.

Hi,

i try to use a file upload form, but unfortunately there is a field which gets added by JS before submitting. Is there a way to simply add fields, or disable the checks that raise the KeyError?

This hard version requirements can cause a lot of woes in ArchLinux packaging. Would you please consider the change to allow greater versions of the dependencies? Thank you!

If I have the following HTML:

<form name="input" action="demo_form_action.asp" method="get">
Username: <input type="text" name="user">
<input type="submit" value="Action1" name="action1_name">
<input type="submit" value="Action2" name="action2_name">
</form> 

When I click the Action1 button in my browser (Firefox or Chrome), the following URL gets sent http://localhost:8001/demo_form_action.asp?user=asfdasd&action1_name=Action1. And a different URL gets sent for Action2: http://localhost:8001/demo_form_action.asp?user=asfdasd&action2_name=Action2. Using the submit_form method in robobrowser, puts the actions for both buttons in the request: http://localhost:8001/demo_form_action.asp?user=asdsafd&action2_name=Action2&action1_name=Action1.

I guess I can model the browser button pressing behaviour by deleting the buttons I do not want pressed from the robobrowser form object. However, it would be nice if the API for submit_form could be extended to include the button being pressed to submit the form.

>>> url = 'https://bitbucket.org/repo/import'
>>> browser.open(url)
>>> import_form = browser.get_form(id='import-form')
>>> import_form
<RoboForm source_scm=, source=source-git, goog_project_name=, goog_scm=svn, sourceforge_project_name=, sourceforge_mount_point=, sourceforge_scm=svn, codeplex_project_name=, codeplex_scm=svn, url=, auth=[], username=, password=, owner=2039394, name=, description=, is_private=[None], forking=no_public_forks, no_forks=, no_public_forks=True, scm=git, has_issues=[], has_wiki=[], language=, csrfmiddlewaretoken=icwpCBLZdWAPht1rmnACawMHcYwtorNA>
>>> type(import_form['auth'])
<class 'robobrowser.forms.fields.Checkbox'>

The 'auth' is a checkbox. How do I set it to true? or selected? I couldn't find required info in documentation, so asking here. Thank you!

I'm getting the following error when I try to install with pip:

$ pip install robobrowser
Downloading/unpacking robobrowser
  Running setup.py egg_info for package robobrowser
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/home/tsc/.virtualenvs/vm_export_tool/build/robobrowser/setup.py", line 19, in <module>
        for requirement in parse_requirements('requirements.txt')
      File "/home/tsc/.virtualenvs/vm_export_tool/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/req.py", line 1240, in parse_requirements
        skip_regex = options.skip_requirements_regex
    AttributeError: 'NoneType' object has no attribute 'skip_requirements_regex'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/home/tsc/.virtualenvs/vm_export_tool/build/robobrowser/setup.py", line 19, in <module>

    for requirement in parse_requirements('requirements.txt')

  File "/home/tsc/.virtualenvs/vm_export_tool/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/req.py", line 1240, in parse_requirements

    skip_regex = options.skip_requirements_regex

AttributeError: 'NoneType' object has no attribute 'skip_requirements_regex'

----------------------------------------
Command python setup.py egg_info failed with error code 1 in /home/tsc/.virtualenvs/vm_export_tool/build/robobrowser
Storing complete log in /home/tsc/.pip/pip.log

python 2.7 pip 1.1 ubuntu 12.10

OS: 16.04 Tried pip and pip easyinstall pip install robobrowser sudo -H pip install robobrowser ....

Even tried wiping pip && requests and reinstall - same result.

Package installs, tried sudo ldconfig - however, not getting this to work. Iam a python newb, please excuse if theres a usefail from my side.

python3 robobrowser.py
Traceback (most recent call last):
  File "robobrowser.py", line 4, in <module>
    from robobrowser import RoboBrowser
  File "/home/robotux/robobrowser.py", line 4, in <module>
    from robobrowser import RoboBrowser
ImportError: cannot import name 'RoboBrowser'

I have a form like this:

<form method="post">
    <input type="hidden" name="action" value="del_file">
    <table class="table table-bordered table-striped">
        <tbody>
            <tr>
                <td class="item-title"> test1.txt</td>
                <td class="item-action">
                    <button type="submit" class="btn" name="file_id" value="24247579">
                        Delete
                    </button>
                </td>
            </tr>
            <tr>
                <td class="item-title"> test2.txt</td>
                <td class="item-action">
                    <button type="submit" class="btn" name="file_id" value="22608379">
                        Delete
                    </button>
                </td>
            </tr>
            <tr>
                <td class="item-title"> test3.txt</td>
                <td class="item-action">
                    <button type="submit" class="btn" name="file_id" value="22608377">
                        Delete
                    </button>
                </td>
            </tr>
        </tbody>
    </table>
</form>

How can I click a delete button ? The delete request is like this: {post} http://host {Content-Type: application/x-www-form-urlencoded} data is action=del_file&file_id=22608377

please tell me what can I do for this form.

I am running into an issue with forms that have text areas. When grabbing the form using browser.get_form all blank text areas return a \r and when submitted now have a new line.

Also, all non-blank text areas are prefixed with a \r\n to the front of it when pulling the form.

This fixes #8 by adding an error message that references the field and the non-existent option being selected. I did not list some of the valid options as it seemed to make for an overly lengthy error message, but they would be easy to add in.

Hi Joshua,

RoboBrowser failed to parse forms where there were empty select boxes. I have added the tests as well as the fix for the same.

The use cases of empty select boxes are pretty common on ASP website where forms require multiple submits. Before the initial submit the select boxes are kept empty. On initial submit, the select boxes get a few options based on other values.

Do let me know if anything is required to be modified in the pull request. Pratyush

Since Robobrowser seems to be unmaintained since 2015, here is a list of alternatives that I've found which all have had commits with the list year (ie, during the year 2021). I've not yet started the process of portin any of my scripts over from Robobrowser to any of these, so I can't vouch for how similar any are.

• Scrapy currently 43k stars on github
• Mechanical Soup currently 3.9k stars on github
• PSpider currently 1.3k stars on github
• Feedparser currently 1.3k stars on github
• Feedparser is specifically designed for RSS, ATOM, and other feeds, rather than a generic scraper
• Spidy currently 273 stars on github
  • Spidy is more of a commandline tool than a framework for automation, but there's probably a non-zero number of users who end up looking at Robobrowser but would be happier with a commandline tool.

I included the stars because that helps gauge how popular a project is. Popular projects tend be more less likely to disappear if a maintainer loses interest in the project. RoboBrowser is currently at 3.6k stars. But all of the above projects appear to be currently maintained.

In start its work for me then later when i run command python3 rank.py https://www.uselessthingstobuy.com/ desktop its show data empty with my file, please help

Werkzeug was upgraded to 1.0.0 and introduced this error: ImportError: cannot import name 'cached_property' from 'werkzeug' (/usr/local/lib/python3.8/site-packages/werkzeug/__init__.py)

Suggesting this change which has been tested locally and discussed here: https://github.com/jmcarp/robobrowser/issues/93

.. to simulate an upload dialog where a user selects multiple files.

upform = forms[0] upform['files'].value = [open("a.gz",'rb'), open("b.gz",'rb')]

This gives me: ValueError('Value must be a file object or file path')

needed to update as it throws error cannot import name 'cached_property' from 'werkzeug' (/app/.heroku/python/lib/python3.7/site-packages/werkzeug/init.py) for the new updated werkzeug package. just changed a import line **_ #from werkzeug import cached_propert_**y to from werkzeug.utils import cached_property