A Python library for automating interaction with websites.

This is in regards to issue #250

For the tests, I followed @moy 's train of thought :

• they are basically a copy+paste without the creation of a temp file
• assert value["doc"] == "" checks that the response contains an empty file

Thought a different test definition was necessary, was I right to assume so ?

In browser.py, I changed the continue around line 179 to something similar to what has been done in test__request_file here

There are 2 Add no file input submit test commits : the second one is simply a clean up of some commented code. Will avoid it next time !

I was unable to run test_browser.py due to some weird Import module error on modules that are installed, so I'm kind of Pull Requesting blindly. Does it matter that I say I'm confident in the changes though ?

In the Roadmap, some artwork is requested. I asked an artistic friend to try to interpret this request, and this is what they came up with. I would love to use this as our logo (in both the README, as per the roadmap, and perhaps also as our organization icon). Before I make a PR, I just wanted to see if this was what you were going for.

Every couple of Travis builds, I see one of the sub-builds hang. It happens frequently enough that I feel like I have to babysit Travis, which is not a good situation to be in. From what I can tell, this occurs under two conditions:

1. httpbin.org is under heavy load (this occurs infrequently, but can occur for extended periods of time)
2. flake8 hangs for some unknown reason (seems arbitrary, and rerunning almost always fixes it)

I really want to understand 2), because for 1) we could simply rely on httpbin.org a bit less if necessary.

I ran into a site with forms including buttons of type "button" with name attributes. Because Form.choose_submit() was only removing name from buttons of type "submit", the values for the "button" buttons were being erroneously sent on POST, thereby breaking my submission. This patch fixes the issue, even when a submit button isn't explicitly chosen.

Note that all buttons that aren't of type "button" or "reset" function as "submit" in all major browsers and should therefore be choosable.

https://www.w3.org/TR/html52/sec-forms.html#element-attrdef-disabledformelements-disabled

The disabled attribute is used to make the control non-interactive and to prevent its value from being submitted.

MechanicalSoup ignores disabled attributes which should be fixed.

Some additional notes: (from https://www.wufoo.com/html5/disabled-attribute/)

• If the disabled attribute is set on a <fieldset>, the descendent form controls are disabled.
• A disabled field can’t be modified, tabbed to, highlighted, or have its contents copied. Its value is also ignored when the form goes thru constraint validation.
• The disabled value is Boolean, and therefore doesn’t need a value. But, if you must, you can include disabled="disabled".
• Setting the value of the disabled attribute to null does not remove the effects of the attribute. Instead use removeAttribute('disabled').
• You can target elements that are disabled with the :disabled pseudo-class. Or, if you want to specifically target the presence of the attribute, you can use input[disabled]. Similarly, you can use :enabled and input:not([disabled]) to target elements that are not disabled.
• You do not need to include aria-disabled="true" when including the disabled attribute because disabled is already well supported. However, if you are programmatically disabling an element that is not a form control and therefore the disabled attribute does not apply, include aria-disabled="true".
• The disabled attribute is valid for all form controls including all <input> types, <textarea>, <button>, <select>, <fieldset>, and <keygen>.

As noted elsewhere, I've recently been debugging behind an SSL proxy, which requires telling requests to not verify SSL certificates. Generally I've done that with

    kwargs = { "verify": False }
    # ...
    r = br.submit_selected(**kwargs)

which is fine. But it's not so fine when I need to follow a link, because browser.follow_link() uses its **kwargs for BS4's tag finding, but not for actually following the link.

So instead of

    r = br.follow_link(text='Link anchor', **kwargs)

I end up with

    link = br.find_link(text='Link anchor')
    r = br.open_relative(link['href'], **kwargs)

I am not sure how to fix this. Some thoughts:

1. If nothing changes, add some more clarity to browser.follow_link()'s documentation explaining how to work around this situation.
2. Add kwargs-ish params to browser.follow_link(), one for BS4 and one for Requests. Of course, only one gets to be **kwargs, but at least one might be able to call browser.follow_link(text='Link anchor', requests_args=kwargs) or something.
3. Send the same **kwargs parameter to both

Maybe there's a better way. I guess in my case I could set this state in requests' Session object, ~which I think would be browser.session.merge_environment_settings(...)~ no, that's not right, I'm not sure how to accomplish it actually.

The pytest-httpbin module provides pytest support for the httpbin module (which is the code that runs the remote server http://httpbin.org). This locally spins up an internal webserver when tests are run.

With this change, MechanicalSoup tests can be run without an internet connection. As a result, the tests run much faster.

You may need the python{,3}-dev package on your system to pip install the pytest-httpbin module.

/usr/local/lib/python3.4/dist-packages/bs4/init.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

To get rid of this warning, change this:

BeautifulSoup([your markup])

to this:

BeautifulSoup([your markup], "lxml")

markup_type=markup_type))

Need to use add_soup method or what?

When we use mechanicalsoup, we sometimes want to verify a request before submitting it.

If you merge this pull request, the package will be able to provide a way for the package's users to review the request.

This is my first pull request for this project. Please let me know if I'm missing anything.

LGTM.com finds one issue in our code, and it seems legitimate to me (although I'm guilty for introducing it):

https://lgtm.com/projects/g/hickford/MechanicalSoup/

We should fix this, and configure lgtm so that it checks pull-requests.

Dear Dan and Matthieu,

first things first: Thanks for conceiving and maintaining this great library. We also switched one of our implementations over from "mechanize" as per https://github.com/ip-tools/ip-navigator/commit/a26c3a8a and it worked really well.

When doing so, we encountered a minor problem when trying to call the follow_link method with the url_regex keyword argument like

response = self.browser.follow_link(url_regex='register/PAT_.*VIEW=pdf', headers={'Referer': result.url})

This raises the exception

TypeError: links() got multiple values for keyword argument 'url_regex'

I am are currently a bit short on time, otherwise i would have submitted a pull request without further ado. Thanks a bunch for looking into this issue.

With kind regards, Andreas.

I was writing a fuzzer for a cybersecurity assignment, and it crashed when it tried to find the links on a PDF file. I think it would make more sense to return that there are no links, if the page fails to parse. This seems relatively straightforward to implement.

We already have basic static analysis with flake8 (and the underlying pyflakes), but using typing annotations and a static typechecker may 1) find more bugs, 2) help our users by providing completion and other smart features in their IDE.

mypy is the historical typechecker, pyright is a more recent one which in my (very limited) experience works better (it's also the tool behind the new Python mode of VSCode). So I'd suggest pyright if we don't have arguments to choose mypy.

For now, neither tool can typecheck the project without error, so a first step would be to add the necessary annotations to get an error-free pyright check.

MechanicalSoup is a really nice package i have used for, but it still requires C Compiler to compile the lxml on *nix systems.

It may be a problem to port to some platforms without C Compiler, such as Android or some minified Linux.

Currently i used a script to build MechanicalSoup without lxml:

#!/bin/sh

# Remove lxml in requirements.txt
sed -i '/lxml/d' requirements.txt

# Use `html.parser` instead `lxml`
sed -i "s@{'features': 'lxml'}@{'features': 'html.parser'}@g" mechanicalsoup/*.py

# Fix examples and tests
sed -i "s@\\(BeautifulSoup(.\\{1,\\}\\)'lxml'\\(.*)\\)@\1'html.parser'\2@g" examples/*.py tests/*.py

It works well, so i think it is not a big problem...

I'm trying to get a form but it only has a class attribute and I'm continuously getting a "LinkNotFoundError". I've inspected the page and I know that I have the correct class name but it doesn't work at all and I don't see any real reference to this type of issue in the docs. I would try to get the form with BS4 but then there wouldn't be a way to select the form.

I can attempt to get the form with BS4 then maybe add an id attribute to it then try selecting it with an id attribute?

I'd really appreciate any help, thank you!

Don't use Python's in operator to match Content-Types, since that is a simple substring match.

It's obviously not correct since a Content-Type string can be relatively complicated, like

Content-Type: application/xhtml+xml; charset="utf-8"; boundary="This is not text/html"

Although that's rather contrived, the prior test "text/html" in response.headers.get("Content-Type", "") would return True here, incorrectly.

Also, the existance of subtypes with +'s means that using the prior test for "application/xhtml" would match the above example when it probably shouldn't.

Instead, leverage requests's code, which comes from the Python Standard Library's cgi.py.

Clarify that we don't implement MIME sniffing, nor X-Content-Type-Options: nosniff instead we do our own thing.

I was looking at this code because of #373.

I've marked this as a draft, because I'm not quite sure this is the way to go, both because of the long discursive comment, the use of a _ function from requests (versus cgi.py's parse_header()).

Also, I'm kind of perplexed what's going on here:

            http_encoding = (
                response.encoding
                if 'charset' in parameters
                else None
            )

Like…why does the presence of charset=utf-8 in the Content-Type header mean that we should trust requests's encoding field? Oh, I see, it's because sometimes requests does some sniffing-ish-stuff and sometimes it doesn't (in which case it parses the Content-Type) and we need to know which, and we're backing out a conclusion about its heuristics? Probably seems like maybe we should parse it ourselves if so. idk.

Maybe we should be doing more formal mime sniffing. And maybe we should be honoring X-Content-Type-Options: nosniff. And… … …

I'm also not sure what kind of test coverage is really appropriate here, if anything additional. Seems like the answer shouldn't be "zero," so…