Hi all,

I am running Splash in a docker container on Ubuntu 12.04.5 LTS and am having trouble getting proxy-profiles to work.

I have this in my /etc/splash/proxy-profiles/crawlera.ini file:

[proxy]
host=<mydomain>.crawlera.com
port=8010

; optional, default is no auth
username=<user>
password=<pass>

and I start the docker container mapping that volume to its equivalent -v /etc/splash/proxy-profiles/:/etc/splash/proxy-profiles/. It appears that by default Splash is launched in the docker container with a flag that tells it where to look for proxy profiles.

And, when I pass the &proxy=crawlera paramter into the typical splash:8050/?render.html?ur... url, it does not throw an error (if I pass a nonexistent proxy-profile it shows "proxy profile not found) - so I am confident it is finding the profile.

In the logs, I am actually seeing:

2015-06-18 17:33:46.661570 [stats] {"maxrss": 148776, "load": [0.0, 0.01, 0.05], "fds": 50, "qsize": 0, "rendertime": 1.3779900074005127, "active": 0, "path": "/execute", "args": {"lua_source": ["function main(splash)\r\n  local url = splash.args.url\r\n  splash.images_enabled = false\r\n  assert(splash:go(url))\r\n  assert(splash:wait(0.5))\r\n  return {\r\n    html = splash:html(),\r\n    png = splash:png(),\r\n    har = splash:har(),\r\n  }\r\nend"], "url": ["http://www.whatismyip.com"], "proxy": ["crawlera"], "images": ["1"], "expand": ["1"], "wait": ["0.5"]}, "_id": 91663608}

So the proxy parameter is definitely there and recognized... but it doesn't do anything. Visiting http://www.whatismyip.com yields the same IP whether or not I have the proxy parameter on.

Any ideas? Or thoughts on how to better diagnose the issue?

QT5 starts printing "device not open" messages in an infinite loop when request is aborted in the middle of download.

I've opened an issue in qt bug tracker: https://bugreports.qt.io/browse/QTBUG-47654. A short script to reproduce it: https://gist.github.com/kmike/ff287998e02fa953b4a2. This issue makes tests hang on Travis (https://github.com/scrapinghub/splash/pull/260).

It looks like a blocker for qt5 branch merging; I'm not sure resource_timeout is the only condition which can trigger this bug (what about a regular timeout?).

Hi,

Thank you very much for this library. I must say you made tremendous update on it since last time i tried to use it (we stopped using it one year ago because it was too troublesome to use with proxies). It seems like splash does not yet support proxies that use the https or connect method. Is that something you can easily add or not ? Are you open to a PR for something like this ?

Best,

Hi,

I've been having trouble getting a lua script that calls something like:

splash:runjs("$(':submit')[0].click()")

to run. That is, splash doesn't seem to actually render the result of the page after submission - as if the submit button had been clicked in the first place.

Any idea how to work around this or what I may be doing wrong?

Thanks, Alex

I use the following command to deamonize the process 'docker run -d -p 8050:8050 --restart=always -v /etc/splash/proxy-profiles:/etc/splash/proxy-profiles scrapinghub/splash:1.5 --maxrss 500' but whenever the splash process crash (it gets killed by the system because it takes too much memory even if i have 1GB on the server) docker fails to restart it with the following error:

Traceback (most recent call last):
  File "/app/bin/splash", line 3, in <module>
    from splash.server import main
  File "/usr/local/lib/python2.7/dist-packages/splash/server.py", line 12, in <module>
    from splash.qtutils import init_qt_app
  File "/usr/local/lib/python2.7/dist-packages/splash/qtutils.py", line 12, in <module>
    from PyQt4.QtCore import (QAbstractEventDispatcher, QDateTime, QObject,
ImportError: /usr/lib/python2.7/dist-packages/PyQt4/QtCore.so: cannot read file data: Input/output error

The current running parameters i used, only gives me the page ones it was rendered.

( curl -s -X POST -H 'content-type: application/json' -d "{"render_all": 1, "timeout": ${timeout}, "wait": ${wait}, "images": 0, "url": "${url}", "html": 1, http://${ip}:${port}/render.json )

My problem is that there are a lot of javascript that the page contain that i am interested in that in the final version of the page are removed.

Is there a way to get the network traffic while the page is loaded? like in chrome "network" tab for example?

Hello there, There is a dependency problem in the last splash-jupyter commit. Because of the update of nbconvert, tornado version 6 has been changed. I think we should downgrade nbcovert to a version where tornado version 5 is used. I'm gonna try with that and let you know. Kind regards, Ahmed.

I have added some examples and UI to load the examples as part of issue #286.

Feel free to give feedback about the UI, the examples and suggestions for other examples.

POST requests do not seem to be going through the HTTP proxy. They do appear in the splash:on_request(callback) and splash:on_response_headers(callback) hooks, but are not seen through the proxy.

Anyone has quick thoughts on where this issue may be in the code?

Related to #239 in that it deals with adding the ability to use splash:http_get(...) with POST requests, but I'm not sure if it's adding support for POST proxying as well.

I'm trying to scrape javascript-heavy page using Scrapy+Splash, but Splash cant render it and just shows few links and top menu. There are template tags in HTML code it returns, my guess is that not all javascript being executed. Even in web-interface (i tried setting wait timer for up to 30 seconds). I tried PhantomJS and selenium - they both work fine, but they are slow compared to Splash. Here is an example page i'm trying to scrape: http://profile.majorleaguegaming.com/crosswrecks/forums

Any idea about what can cause this? I checked docs, tried changing a few options, but with zero effect. Thanks.

I'm interested in building a scraper that goes through a site and finds what cookies are being set both 1st party and 3rd party. Scrapy's built in cookie middleware only deals with cookies set in http headers (set-cookies). I'm interested in cookies set through other methods, for example javascript. Therefore I wonder, how does Splash/scrapyjs handle cookies set through other means than http headers? Does it set javascript cookies or handle them in any way? Or are they just ignored?

I'm trying to download HTML for https://factory.jcrew.com/p/womens/categories/clothing/dresses/ruffle-tiered-mini-dress/BM068?display=standard&fit=Classic&color_name=black-light-khaki&colorProductCode=BM068 but the response has missing text (like price) even after I wait for enough time (5 seconds). Is there something I might be missing here?

I see this error when visiting https://bananarepublic.gap.com/browse/product.do?pid=407431#pdp-page-content via Splash. I tried to increase Wait duration as well but it didn't solve the problem. I'm not seeing this error on other webistes so not sure what might be missing here.

Problem

Splash seems to throw the error: "ReferenceError: Can't find variable: IntersectionObserver" when loading certain websites. From what I can tell this error occurs in older browsers, like prior versions of Safari and I guess could be related to the version of WebKit Splash uses under the hood. Some Stackoverflow posts have stated that even the most recent version of Safari (2019 post) can still throw this error since the functionality was deemed experimental and older devices disable such features. I don't know if there is a way to tweak the Webkit configuration Splash uses? I've seen this on multiple high traffic sites so it seems like core functionality that other browsers have supported for a while now. I raised this issue with Zyte and their suggestion was to use Playwright or Puppeteer instead. I'm quite invested in a system built around Splash and don't have the time it would take to port everything over.

Steps to Reproduce

This is the only code that I'm running in a fresh notebook, from the Splash Jupyter notebook docker image that Zyte provides, set up successfully on OSX with XQuartz for the QT Webkit browser and inspection tool. To setup the notebook with splash:

brew install --cask xquartz
IP=$(/usr/sbin/ipconfig getifaddr en0) 
echo $IP 
/opt/X11/bin/xhost + "$IP"
docker run   -e QT_DEBUG_PLUGINS=1 \
             -e DISPLAY="$IP":0 \
             -v /tmp/.X11-unix:/tmp/.X11-unix \
             -v $XAUTHORITY:$XAUTHORITY \
             -e XAUTHORITY=$XAUTHORITY \
             -p 8888:8888 \
             -it scrapinghub/splash-jupyter --disable-xvfb

Then from a new Splash notebook instance:

splash:on_request(function (request)
      request:set_header('X-Crawlera-Cookies', 'disable')
      request:set_header('X-Crawlera-Profile', 'desktop')
      request:set_header('X-Crawlera-Timeout', '5000')
      request:set_proxy{
          host = "<proxy endpoint>",
          port = "8010",
          username = "<password>",
          password = ""
      }
end)

splash.private_mode_enabled = false
assert(splash.private_mode_enabled == false)

splash:go("https://byjus.com/question-answer/why-is-air-called-breath-of-life-enumerate-functions-of-air-or-atmosphere/")

After running this, parts of the page don't render and using the browser inspection tool provided for this splash browser I can see the InspectionObserver error being thrown in the console with cascading errors following. I've observed this on multiple sites now

My issue is related to querying the URL below using splash in version 3.5:

• https://leismunicipais.com.br/a/sp/s/sorocaba/decreto/2022/2738/27375/decreto-n-27375-2022-dispoe-sobre-a-revogacao-do-decreto-n-23551-de-13-de-marco-de-2018-que-dispoe-sobre-permissao-de-uso-a-titulo-precario-de-bem-publico-municipal-e-da-outras-providencias

Even after many hours, it doesn't show any results. Here are some screenshots of the execution.

When using chromium engine in a spider to render a page that cannot be rendered with webkit engine with scrapy crawl example and spider:

import scrapy
from scrapy_splash import SplashRequest


class ExampleSpider(scrapy.Spider):
    name = 'example'
    url = 'http://example.com/'

    def start_requests(self):
        yield SplashRequest(
            url=self.url,
            callback=self.parse,
            endpoint='render.html',  # have gone with and without specifying
            headers=None,  # have gone with and without specifying
            args={
                'wait': 0.5,
                'engine': 'chromium',
                # 'headers': None,  # also tried this
            },
        )

    def parse(self, response, **kwargs):
        yield None

However, I get:

2022-11-02 19:56:00 [scrapy.core.scraper] ERROR: Error downloading <GET http://example.com/ via http://localhost:8050/render.html>
Traceback (most recent call last):
  File "/path/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1696, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "/path/venv/lib/python3.10/site-packages/scrapy/core/downloader/middleware.py", line 60, in process_response
    response = yield deferred_from_coro(method(request=request, response=response, spider=spider))
  File "/path/venv/lib/python3.10/site-packages/scrapy_splash/middleware.py", line 412, in process_response
    response = self._change_response_class(request, response)
  File "/path/venv/lib/python3.10/site-packages/scrapy_splash/middleware.py", line 433, in _change_response_class
    response = response.replace(cls=respcls, request=request)
  File "/path/venv/lib/python3.10/site-packages/scrapy/http/response/__init__.py", line 117, in replace
    return cls(*args, **kwargs)
  File "/path/venv/lib/python3.10/site-packages/scrapy_splash/response.py", line 119, in __init__
    self._load_from_json()
  File "/path/venv/lib/python3.10/site-packages/scrapy_splash/response.py", line 165, in _load_from_json
    error = self.data['info']['error']
TypeError: string indices must be integers

The root of this issue is found looking through the Scrapy logs:

2022-11-02 23:56:00.000802 [-] Unhandled error in Deferred:
2022-11-02 23:56:00.000931 [-] Unhandled Error
        Traceback (most recent call last):
          File "/app/splash/pool.py", line 47, in render
            self.queue.put(slot)
          File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 1872, in put
            self.waiting.pop(0).callback(obj)
          File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 460, in callback
            self._startRunCallbacks(result)
          File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/app/splash/pool.py", line 76, in _start_render
            render.start(**slot_args.kwargs)
          File "/app/splash/engines/chromium/render_scripts.py", line 59, in start
            raise BadOption("headers is not implemented")
        splash.errors.BadOption: headers is not implemented

This doesn't seem to be a problem when using Scrapy in browser: or with curl:

versions

Python 3.10.7 Scrapy 2.7.0 scrapy-splash 0.8.0 Splash 3.5.0

My issue happens on splash 3.0 and 3.5 but NOT on 2.3.3. i am currently running prod on 2.3.3 as a workaround and would like a permanent solution to run 3.x

i have been running splash + HAProxy set up by aquarium for years before experiencing this issue, including successfully rendering the sites in question without issue prior to the day before yesterday

here is a url that consistently produces the issue, even simply using render.html from [host]:8050 https://www.schooljobs.com/careers/kirkwoodcc/jobs/3776251/adjunct-dental-hygiene

happens with aquarium default configuration

this happens in both dev (mac OS 15+) and prod (ubuntu) environments, and i did try wiping all my containers and starting over with aquarium. splash works fine for other urls but the above and some others kills it. every time, it locks up the entire docker container (immediately) and the HAPROXY stats shows a level 7 timeout (splash 3.5) or Level 4 timeout (3.0).

i cannot attach to a splash docker instance that hangs in this way - if i try, my terminal hangs.

thanks to docker-compose with aquarium i can watch splash output live. on 3.5 i often don't even get to see output of the request starting. sometimes i just see the request and then no more output as the instance hangs

on 3.0 only i get the following info

i have googled the network issue and found a bunch of issues right here in this repo with no clear answers about what is going on.

happy to be very responsive. please let me know if more info is needed. I want to get back to splash 3.x