urlwatch is intended to help you watch changes in webpages and get notified of any changes.

I run urlwatch --edit and have the url tag and the filter tag in it only if I change the url it changes the page that gets sent to my address, tried multiple filter examples from the readme, no change in the email output, it always sends the html not plain text. Tried out the html2text command in the shell and it works it converts it to plain text. Email is set to send unchanged too.

tried these filter: html2text,grep:Current.*version,strip or filter: element-by-tag:body

same html output

In previous versions I was able to use double quotes to grep. When using the grep case insensitive flag (?i) my jobs fail if the search terms are in single or double quotes.

When searching for two terms using the pipe character how can I specify one term as being case insensitive and the next term as case sensitive? Like this... grep: (?i)"National Age Group Record"|(?-i)"NAG"

When I don't quote NAG or specify it as case sensitive I end up with matches I don't want i.e. Snags

Hi, how do i configure urlwatch to send e-mails (my smtp provider requires TLS) without having to interactively type a password? Currently i get a popup saying "Please enter password for encrypted keyring:" which means i can't run the command in crontab/automatically.

thanks.

The following exception happens, when the return value from a shell script contains umlaute (ä, ö, ü):

Traceback (most recent call last):
 File "/usr/bin/urlwatch", line 376, in <module>
   main(parser.parse_args())
 File "/usr/bin/urlwatch", line 343, in main
   report.finish()
 File "/usr/lib/python3.5/site-packages/urlwatch/handler.py", line 128, in finish
   ReporterBase.submit_all(self, self.job_states, duration)
 File "/usr/lib/python3.5/site-packages/urlwatch/reporters.py", line 81, in submit_all
   cls(report, cfg, job_states, duration).submit()
 File "/usr/lib/python3.5/site-packages/urlwatch/reporters.py", line 306, in submit
   print(line)
UnicodeEncodeError: 'ascii' codec can't encode character '\xfc' in position 6: ordinal not in range(128)

Hi! I get the following exception:

Traceback (most recent call last):
  File "/usr/bin/urlwatch", line 376, in <module>
    main(parser.parse_args())
  File "/usr/bin/urlwatch", line 343, in main
    report.finish()
  File "/usr/lib/python3.5/site-packages/urlwatch/handler.py", line 128, in finish
    ReporterBase.submit_all(self, self.job_states, duration)
  File "/usr/lib/python3.5/site-packages/urlwatch/reporters.py", line 81, in submit_all
    cls(report, cfg, job_states, duration).submit()
  File "/usr/lib/python3.5/site-packages/urlwatch/reporters.py", line 298, in submit
    print(self._red(line))
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 63: ordinal not in range(256)

I traced it back to the following content of the website:

European Union’s Please note the ' between n and s causing the hickup. Maybe all string operations must be moved to utf8?

Thanks!

Copied from the XMPP patch, support for Pushover API to send notifications about website changes. Pushover config file needs to have App Key & User Key (created with a pushover account)

Command ./urlwatch --urls sonstiges.url --cache sonstigesWatch.db

Filesize: sonstigesWatch.db about 12 MB

On Prometheuts stats you see disk write marked as 1

Now I use another db which has about 35 MB. Then you see in the graph the disk write marked with number 2.

Job with 12 MB runs every 5 minutes. Thats 8760 hours per year * 60 minutes / 5 minutes * 12 MB = 1,261,440 MB or 1,2 TB written per year. Job with now 35 MB DB runs every 30 minutes, thats about 600 GB written per year.

Thats a lot of workload per year, just because of urlwatch and with an SD-Card of raspberry PI this is a really huge workload.

DB is even rewritten when there is no change detected, because timestamp of last run is updated.

Since this is basically SQLite, I can't imagine that there is no other way than rewriting the whole file.

Update: This also affects urlwatch performance with large databases. Since each time database is rewritten it slows down urlwatch process.

I'm not really happy with the email notification and how to see changes.

Is there a way to trigger a simple external script with some parameters like the name-of-urls.yaml, name and url? my intention is to pipe all in a text/html/xml file and commit them to a personal git/svn/hg repository. this way i have a nice history and see the changes on different devices in a easy way.

i would use the name of the urls.yaml as a initial folder name (i want to use this to group them), the name for the filename and the the url in the header of the text/html/xml file for a quickly open the site again. maybe complete entry from the urls.yaml.

Something like this is maybe helpful to #53 as well.

Hi Thomas, thanks for writing urlwatch, it's exactly what I needed.

I have it watching ~100 URLs and sporadically get an SQLite exception on a subset of URLs for that run. It doesn't happen with any URL(s) in particular, but it does happen regularly (perhaps not every single run, but most of the time) and usually happens for a few URLs in a run.

This is with urlwatch 2.5, minidb 2.0.1, SQLite 3.8.7.1, and Python 3.4.2 (the latter two are from Debian jessie packaging).

I put some debugging output in the affected exception handler. The query and args are:

  SELECT data, timestamp FROM CacheEntry WHERE guid = ? ORDER BY timestamp DESC LIMIT ?
  ['2483ad9118822dd4099841a766fc86b65625d20e', 1]

It's puzzling since I'd think the GUID would always be a string.

  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/urlwatch/handler.py", line 65, in process
      self.load()
    File "/usr/lib/python3/dist-packages/urlwatch/handler.py", line 57, in load
      self.old_data, self.timestamp = self.cache_storage.load(self.job, self.job.get_guid())
    File "/usr/lib/python3/dist-packages/urlwatch/storage.py", line 322, in load
      where=CacheEntry.c.guid == guid, limit=1):
    File "/usr/local/lib/python3.4/dist-packages/minidb.py", line 932, in query
      return db.query(cls, select=select, where=where, order_by=order_by, group_by=group_by,
  limit=limit)
    File "/usr/local/lib/python3.4/dist-packages/minidb.py", line 401, in query
      result = self._execute(sql, args)
    File "/usr/local/lib/python3.4/dist-packages/minidb.py", line 158, in _execute
      return self.db.execute(sql, args)
  sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.

The urlwatch 2.x has changed their configuration and cache storing system, but the migration from 1.X fails with:

urlwatch --edit
No jobs file found

        Migrating cache: /home/user/.urlwatch/cache -> /home/user/.urlwatch/cache.db

Removing: 0141a464b24bbc03f2bbba2bf91fe94179572238
Removing: 35ecdde4fad96b22d3fd11cee70f2bb141296e7a
Removing: 6d501928989a55a0325b63f450309da105bc4be8
Removing: a6d62130ca1dae859c462d0b9325bee60e449a21
Removing: 7f8dc223248751c43f5fff47399ad808332bba88
Removing: cdc6c20b4ed496b8b185d37c7966f4e31a6dabc5
Removing: e382ca109a55e69d45da802b58a0d37d37dd5024
Removing: f0f6be08f546f22bbd3b22c9b1650ea426666ea5
Removing: cc3acca41c7ee65dadbd2693d969eb18729472b6
Removing: d952b8a505165d60824ccd73a34a15e7035a4b2e
Removing: b06d40c441961c272fc4bb13f9a0494d7c64be9e
Removing: 86c5f26c01db07316e61516f95626d246d7735c8
Removing: cce3f7a7af3039f374199a849fb7e24d306695d9
Removing: 4242a29b794b46739bcb7144e599182827e5b596
Removing: b3eaf7d9df6af694d067475dc5c3bffd845a5c42
Removing: 56a05b6aa9c51fc2e29bfed42bc33c72ca83aeb2
Removing: 85bf86ea5490220da6be0418c53348b6a2b8b110
Removing: 22c201209094bfa9b508182022933236b5039b3b
Removing: 1921dd46f0c55c2e45ff5f68a9cc483971bee7dd
Removing: 06f09294cf250d4ffa3ebf9371129f678eb3e173
Parsing failed:
======
[Errno 2] No such file or directory: '/usr/lib/python-exec/python3.4/share/urlwatch/examples/urls.yaml.example'
======

The file /home/slavko/.urlwatch/urls.yaml was NOT updated.
Your changes have been saved in /home/user/.urlwatch/urls.edit.yaml

It seems, that the urls.yaml.example file is installed into /usr/share/urlwatch/examples/urls.yaml.example, but the urlwatch expects it in the /usr/lib/python-exec/python3.4...:

Simple simlink /usr/lib/python-exec/python3.4/share/urlwatch -> /usr/share/urlwatch solves the problem...

It is reported by me on Funtoo bugs too, you can see it at https://bugs.funtoo.org/browse/FL-3128

If it's using OpenSSL, I need to force it using older version of OpenSSL on Windows10! My service is using an old crypto that is implemented in openssl with version<v1.1.0.

With the urlwatch-default config file (urlwatch.yaml), it seems that a KeyError is thrown since commit 5385365

➜  urlwatch git:(master) urlwatch --urls ~/urls.yaml
Traceback (most recent call last):
  File "/Users/trevorshannon/projects/urlwatch/urlwatch", line 9, in <module>
    main()
  File "/Users/trevorshannon/projects/urlwatch/lib/urlwatch/cli.py", line 112, in main
    urlwatch_command.run()
  File "/Users/trevorshannon/projects/urlwatch/lib/urlwatch/command.py", line 433, in run
    self.urlwatcher.close()
  File "/Users/trevorshannon/projects/urlwatch/lib/urlwatch/main.py", line 97, in close
    self.report.finish()
  File "/Users/trevorshannon/projects/urlwatch/lib/urlwatch/handler.py", line 217, in finish
    ReporterBase.submit_all(self, self.job_states, duration)
  File "/Users/trevorshannon/projects/urlwatch/lib/urlwatch/reporters.py", line 138, in submit_all
    if cfg['enabled']:
KeyError: 'enabled'

I think this has something to do with adding a __kind__ to MarkdownReporter, TextReporter, and HtmlReporter

This can be worked around by explicitly adding enabled: false to the text, html, and markdown reporter configuration settings.

FYI @ryneeverett

When using a redis cache, the --gc-cache option successfully removes old entries, but does not delete old keys (despite claiming to do so). This seems to be due to a string/bytes type issue somewhere around here.

When deleting unused keys, the system attempts to remove e.g. "guid:b'27687e4f702f8547973bc60451b4f1539ee3d428'" instead of "guid:27687e4f702f8547973bc60451b4f1539ee3d428"

This PR adds a new command line option --change-location to change the location (i.e. url, command, or navigate) of a job while preserving history for that job. The tool ensures that the new location string is unique within the list of jobs prior to changing anything.

Note that in the (admittedly contrived) scenario of having abandoned entries in a redis db that use the user-provided new location (perhaps from old runs that were never garbage-collected) then those abandoned entries will be clobbered by the --change-location option. I think that should be ok, but I'm sure I could find a way to preserve those entries if needed.

I did some manual testing to verify functionality.

Save the starting version of a simple urls YAML:

➜  urlwatch git:(feat/update-url)  cp ~/urls.yaml ~/urls_init.yaml
➜  urlwatch git:(feat/update-url) cat ~/urls_init.yaml 
command: date
kind: shell
---
kind: url
url: https://www.example.org
---
kind: url
url: https://www.wikipedia.org
user_visible_url: https://en.wikipedia.org
---
kind: browser
navigate: https://www.google.com

Change location of all 4 jobs:

➜  urlwatch git:(feat/update-url)  urlwatch --urls ~/urls.yaml --change-location 1 "date -v1m"                                
Moving location of <shell command='date'> to "date -v1m"
Saving updated list to '/Users/trevorshannon/urls.yaml'
➜  urlwatch git:(feat/update-url)  urlwatch --urls ~/urls.yaml --change-location https://www.example.org https://www.example.net
Moving location of <url url='https://www.example.org'> to "https://www.example.net"
Saving updated list to '/Users/trevorshannon/urls.yaml'
➜  urlwatch git:(feat/update-url)  urlwatch --urls ~/urls.yaml --change-location 3 https://es.wikipedia.org                     
Moving location of <url url='https://www.wikipedia.org' user_visible_url='https://en.wikipedia.org'> to "https://es.wikipedia.org"
Saving updated list to '/Users/trevorshannon/urls.yaml'
➜  urlwatch git:(feat/update-url)  urlwatch --urls ~/urls.yaml --change-location 4 https://www.google.co.uk
Moving location of <browser navigate='https://www.google.com'> to "https://www.google.co.uk"
Saving updated list to '/Users/trevorshannon/urls.yaml'

Compare the updated urls YAML to the initial reference version. Appropriate key values are updated according to job type:

➜  urlwatch git:(feat/update-url)  diff ~/urls_init.yaml ~/urls.yaml   
1c1
< command: date
---
> command: date -v1m
5c5
< url: https://www.example.org
---
> url: https://www.example.net
9c9
< user_visible_url: https://en.wikipedia.org
---
> user_visible_url: https://es.wikipedia.org
12c12
< navigate: https://www.google.com
---
> navigate: https://www.google.co.uk

Existing history is moved with --change-location:

➜  urlwatch git:(feat/update-url) urlwatch --urls ~/urls.yaml --dump-history 1
==============================
2022-12-20 13:08
------------------------------
Tue Dec 20 13:08:59 EST 2022

============================== 

==============================
2022-12-20 13:18
------------------------------
Tue Dec 20 13:18:54 EST 2022

============================== 

2 historic snapshot(s) available
➜  urlwatch git:(feat/update-url) urlwatch --urls ~/urls.yaml --change-location date "date -v1m"
Moving location of <shell command='date'> to "date -v1m"
Moved 2 snapshots of "date" to "date -v1m"
Saving updated list to '/Users/trevorshannon/urls.yaml'
➜  urlwatch git:(feat/update-url) urlwatch --urls ~/urls.yaml                                   
===========================================================================
01. CHANGED: date -v1m
===========================================================================

---------------------------------------------------------------------------
CHANGED: date -v1m
---------------------------------------------------------------------------
--- @	Tue, 20 Dec 2022 13:18:54 -0500
+++ @	Tue, 20 Dec 2022 13:20:33 -0500
@@ -1 +1 @@
-Tue Dec 20 13:18:54 EST 2022
+Thu Jan 20 13:20:33 EST 2022
---------------------------------------------------------------------------

➜  urlwatch git:(feat/update-url) urlwatch --urls ~/urls.yaml --dump-history 1                  
==============================
2022-12-20 13:08
------------------------------
Tue Dec 20 13:08:59 EST 2022

============================== 

==============================
2022-12-20 13:18
------------------------------
Tue Dec 20 13:18:54 EST 2022

============================== 

==============================
2022-12-20 13:20
------------------------------
Thu Jan 20 13:20:33 EST 2022

============================== 

3 historic snapshot(s) available

I confirmed the same behavior with --cache redis://127.0.0.1:6379

Fixes #300

Hi Everyone,

i created my first browser job after isntalling pyppeeeer. Sadly running the job causes some kind of error. I couldn´t find any solution myself. So maybe someone here knows the problem and can make more sense of this error:/

`Exception while releasing resources for job: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 138, in test_filter raise job_state.exception File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/handler.py", line 68, in enter self.job.main_thread_enter() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/jobs.py", line 377, in main_thread_enter self.ctx = BrowserContext() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 90, in init BrowserContext._BROWSER_LOOP = BrowserLoop() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 45, in init self._browser = self._event_loop.run_until_complete(self._launch_browser()) File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 51, in _launch_browser browser = yield from pyppeteer.launch() File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 307, in launch return await Launcher(options, **kwargs).launch() File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 168, in launch self.browserWSEndpoint = get_ws_endpoint(self.url) File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 227, in get_ws_endpoint raise BrowserError('Browser closed unexpectedly:\n') pyppeteer.errors.BrowserError: Browser closed unexpectedly:

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/handler.py", line 78, in exit self.job.main_thread_exit() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/jobs.py", line 380, in main_thread_exit self.ctx.close() AttributeError: 'BrowserJob' object has no attribute 'ctx' Traceback (most recent call last): File "/usr/local/bin/urlwatch", line 11, in load_entry_point('urlwatch==2.24', 'console_scripts', 'urlwatch')() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/cli.py", line 112, in main File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 408, in run File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 210, in handle_actions File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 138, in test_filter File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/handler.py", line 68, in enter File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/jobs.py", line 377, in main_thread_enter File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 90, in init File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 45, in init File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 51, in _launch_browser File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 307, in launch return await Launcher(options, **kwargs).launch() File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 168, in launch self.browserWSEndpoint = get_ws_endpoint(self.url) File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 227, in get_ws_endpoint raise BrowserError('Browser closed unexpectedly:\n') pyppeteer.errors.BrowserError: Browser closed unexpectedly:`

I'm tracking a website that sometimes gives me an errormessage like:

('Connection broken: IncompleteRead(7450 bytes read, 646 more expected)', IncompleteRead(7450 bytes read, 646 more expected))

The configuration I'm using includes the following statements that seem to have no effect on this particular error:

ignore_connection_errors: true
ignore_http_error_codes: 1xx, 4xx, 5xx
timeout: 0

I've also tried "treating" it as a timeout, by setting a stricter timeout and ignoring timeout errors like so:

ignore_connection_errors: true
ignore_http_error_codes: 1xx, 4xx, 5xx
ignore_timeout_errors: true
timeout: 10

But it doesn't really help anything. What else can I try? This is urlwatch v2.25 on Linux