The next generation Canto RSS daemon

Overview

Canto Daemon

This is the RSS backend for Canto clients.

Canto-curses is the default client at:

http://github.com/themoken/canto-curses

Requirements

Debian / Ubuntu
- python3
- python3-feedparser

Arch (Py3 is default)
- python
- python-feedparser

Install

From this directory, run

    $ sudo python3 setup.py install

If you're a systemd user you can easily run it on startup

    $ systemctl --user enable canto-daemon

Or start it manually with

    $ systemctl --user start canto-daemon

By default, user sessions start on login, and end on logoff stopping daemons. This is good behavior, but if you don't want canto-daemon to stop when you log out (to keep gathering news) enable "lingering" for your account.

    $ loginctl enable-linger 

This will start one session for you on boot that will last until shutdown.

Comments
  • Old tag items are missing if restarting daemon session (former: default.keep_unread seems to be ignored with 0.9)

    Old tag items are missing if restarting daemon session (former: default.keep_unread seems to be ignored with 0.9)

    I currently notice that feeds are vanishing, even if keep_unread is set to True.

    Or is this option gone with 0.9?

    (On another note. Somewhere a place where all options can be seen? Autocompleting through all options isn't very effective :D. Maybe a skeleton file could be added (or example config) to have a start base)

    opened by Narrat 20
  • Daemon never deletes any of the items

    Daemon never deletes any of the items

    This is more a question than an issue.

    File .canto-ng/feeds grows and grows until it reaches the point when I couldn't download it over ssh and client breaks and do not work. Then I delete this file and it starts all over again.

    I guess this is because of the option I set: :keep_unread = [True|False]

    I am never reading news through the reader itself, I choose interesting topics and press 'g' (open through web-server). Is that why daemon never deletes all old stuff? Will it delete the old stuff if I set this to False? And will I miss anything if this will be set to False, but daemon will work 24/7 on my raspbery PI server?

    Btw. I am marking all the news as 'read' (they change color and dissapear from the client view configured with filter)

    opened by ewancoder 16
  • Corrupted feeds file

    Corrupted feeds file

    Got this problem two times now. Dunno how I achieved it. Trying to start canto-curses and all I get is a empty screen.

    Checking the daemon log provides the following:

    02:42:43 : SHELF -> Failed to JSON load, old shelf? 02:42:43 : SHELF -> Failed to migrate old shelf: db type could not be determined Traceback (most recent call last): File "/usr/lib/python3.5/site-packages/canto_next/storage.py", line 48, in open self.cache = json.load(fp) File "/usr/lib/python3.5/json/init.py", line 265, in load return loads(fp.read(), File "/usr/lib/python3.5/gzip.py", line 274, in read return self._buffer.read(size) File "/usr/lib/python3.5/gzip.py", line 461, in read if not self._read_gzip_header(): File "/usr/lib/python3.5/gzip.py", line 409, in _read_gzip_header raise OSError('Not a gzipped file (%r)' % magic) OSError: Not a gzipped file (b'f\xec')

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/usr/bin/canto-daemon", line 7, in c = CantoBackend() File "/usr/lib/python3.5/site-packages/canto_next/canto_backend.py", line 123, in init self.get_storage() File "/usr/lib/python3.5/site-packages/canto_next/canto_backend.py", line 796, in get_storage self.shelf = CantoShelf(self.feed_path) File "/usr/lib/python3.5/site-packages/canto_next/storage.py", line 27, in init self.open() File "/usr/lib/python3.5/site-packages/canto_next/feed.py", line 102, in _fl return fn(*args) File "/usr/lib/python3.5/site-packages/canto_next/storage.py", line 53, in open s = shelve.open(self.filename, "r") File "/usr/lib/python3.5/shelve.py", line 243, in open return DbfilenameShelf(filename, flag, protocol, writeback) File "/usr/lib/python3.5/shelve.py", line 227, in init Shelf.init(self, dbm.open(filename, flag), protocol, writeback) File "/usr/lib/python3.5/dbm/init.py", line 88, in open raise error[0]("db type could not be determined") dbm.error: db type could not be determined

    Removing feeds file makes canto responsive again. But still the question how it could end like this

    opened by Narrat 13
  • got decoding str is not supported

    got decoding str is not supported

    Hi, I just installed canto and I think this must be very simple to solve but I could not find any reference to it.

    This is what I get in my daemon-log:

    18:50:22 : CANTO-FETCH -> Empty feed, attempt to update. 18:50:22 : CANTO-FETCH -> Empty feed, attempt to update. 18:50:22 : CANTO-FETCH -> Empty feed, attempt to update. 18:50:22 : CANTO-FETCH -> Empty feed, attempt to update. 18:50:22 : CANTO-FETCH -> ERROR: try to parse http://rss.slashdot.org/slashdot/Slashdot, got decoding str is not supported 18:50:22 : CANTO-FETCH -> ERROR: try to parse http://reddit.com/.rss, got decoding str is not supported 18:50:22 : CANTO-FETCH -> ERROR: try to parse http://codezen.org/canto-ng/feed/, got decoding str is not supported

    I added one new URL and has the same problem.

    Thank you,

    EDIT:

    I have traced to problem to feedparser but I could not solve it.

    I first thought it was a problem of str vs bytes so I appended .encode() to the URL. Then the error became:

    17:24:06 : CANTO-FETCH -> Empty feed, attempt to update. 17:24:06 : CANTO-FETCH -> No content in http://codezen.org/canto-ng/feed/: :2:-1: Document is empty

    and like this with every possible feed.

    Then I tried to use feedparser directly following the examples in http://pythonhosted.org/feedparser/introduction.html but those did not work either:

    exemple 1:

    import feedparser d = feedparser.parse('http://feedparser.org/docs/examples/atom10.xml') Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python3.2/site-packages/feedparser-5.1.3-py3.2.egg/feedparser.py", line 3745, in parse saxparser.parse(source) File "/usr/lib64/python3.2/site-packages/drv_libxml2.py", line 189, in parse eltName = (_d(reader.NamespaceUri()),
    File "/usr/lib64/python3.2/site-packages/drv_libxml2.py", line 70, in _d return _decoder(s)[0] File "/usr/lib64/python3.2/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) TypeError: 'str' does not support the buffer interface

    example 2:

    import feedparser d = feedparser.parse('http://feedparser.org/docs/examples/atom10.xml'.encode()) d {'feed': {}, 'encoding': 'utf-8', 'bozo': 1, 'version': '', 'namespaces': {}, 'entries': [], 'bozo_exception': SAXParseException('Do cument is empty\n',)}

    See the difference when using bytes and str.

    I have canto-next and feedparser from their git repo.

    Can anyone else reproduce this problem?

    opened by focs 9
  • Very poor ARM performance

    Very poor ARM performance

    I'm getting severe delays and outright non-responsiveness when running canto-daemon on ARM. This manifests itself as the server eating huge resources due to spawning a large number of threads, and the client communication is slow to the point of unusability.

    opened by kozross 8
  • Yet another hook request (and question about ssh)

    Yet another hook request (and question about ssh)

    It would be great to have a hook which will work upon pressing 'g' (upon opening an article in a web-browser). Then one can form his own RSS feed based upon the items he reads (I am basically posting cool articles to my friends and it takes much time... it would be quite an optimisation to post them just an rss :D)

    And I have question about ssh: it takes too long to get contents and everything lagging (over LAN). Should it be like this? I did on server: canto-daemon -p 31000 I did on client: canto-curses -a 192.168.100.33 -p 31000

    (I am connecting just via LAN, not via ssh. The question is how to do it via ssh and will it be faster than lan and etc.)

    UPD: And canto-curses sometimes outputs errno 111 - connection refused

    opened by ewancoder 8
  • user password for Jira RSS feed

    user password for Jira RSS feed

    Configure user and password for url (Jira2) do not work for me. ":2:942: mismatched tag". User and password as args in url (Jira1) works fine

    https://confluence.atlassian.com/display/JIRA/Receiving+Search+Results+as+an+RSS+Feed

    {
        "CantoCurses": {
            "browser": {
                "path": "lynx"
            }
        },
        "defaults": {
            "global_transform": "None",
            "keep_time": 86400,
            "keep_unread": false,
            "rate": 10
        },
        "feeds": [
            {
                "name": "Jira1",
                "url": "https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic&os_username=benny&os_password=secret"
            },
            {
                "name": "Jira2",
                "password": "secret",
                "url": "https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic",
                "user": "benny"
            }
        ]
    }
    
    16:01:56 : CANTO-DAEMON -> canto-daemon 0.9.0
    16:01:56 : CANTO-DAEMON -> verbosity = 1
    16:01:56 : CANTO-DAEMON -> conf_dir = /home/kjellgrb/.canto-ng/
    16:01:56 : PLUGINS -> [plugin] pushbullet - DISABLED
    16:01:56 : SHELF -> Shelf opened: <shelve.DbfilenameShelf object at 0x7f68640307f0>
    16:01:56 : CONFIG -> Read /home/kjellgrb/.canto-ng//conf
    16:01:56 : CONFIG -> Parsed into: {'CantoCurses': {'browser': {'path': 'lynx'}}, 'feeds': [{'url': 'https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic&os_username=benny&os_password=secret', 'name': 'Jira1'}, {'url': 'https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic', 'password': 'secret', 'user': 'benny', 'name': 'Jira2'}], 'defaults': {'keep_time': 86400, 'keep_unread': False, 'global_transform': 'None', 'rate': 10}}
    16:01:56 : CONFIG -> Validated: {'CantoCurses': {'browser': {'path': 'lynx'}}, 'feeds': [{'url': 'https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic&os_username=benny&os_password=secret', 'name': 'Jira1'}, {'url': 'https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic', 'password': 'secret', 'user': 'benny', 'name': 'Jira2'}], 'defaults': {'keep_time': 86400, 'keep_unread': False, 'rate': 10, 'global_transform': 'None'}}
    16:01:56 : CANTO-DAEMON -> Listening on unix socket: /home/kjellgrb/.canto-ng//.canto_socket
    16:01:56 : SERVER -> Spawned connection monitor thread.
    16:01:56 : CANTO-FETCH -> Started thread for feed https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic&os_username=benny&os_password=secret
    16:01:56 : FEED -> Fetched previous content for https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic&os_username=benny&os_password=secret.
    16:01:56 : FEED -> Item not over keep_time (86400): urn:uuid:51ad59f9-f9c0-3ea4-9c70-517d8d45ec60
    16:01:56 : FEED -> Item not over keep_time (86400): urn:uuid:d525cf67-d314-31ab-b1a1-f072cb620920
    16:01:56 : FEED -> Item not over keep_time (86400): urn:uuid:e29efeeb-d79d-3532-ad08-3362eca2d4bb
    16:01:56 : FEED -> Item not over keep_time (86400): urn:uuid:f5099065-7a90-3478-8f05-59e8996dff6a
    16:01:56 : FEED -> Item not over keep_time (86400): urn:uuid:a56ad22a-03cc-3df8-974b-3c43acc68892
    16:01:56 : FEED -> Item not over keep_time (86400): urn:uuid:f8db34c4-bf17-37b3-8077-b38ec648a2f4
    16:01:56 : FEED -> Item not over keep_time (86400): urn:uuid:c6656fd5-c8e1-35cb-a08b-b903ab041a6d
    16:01:56 : FEED -> Item not over keep_time (86400): urn:uuid:a2a2d9c2-f3e1-3a44-b682-01da68c6973c
    16:01:56 : FEED -> Item not over keep_time (86400): urn:uuid:3b65cd43-53fd-3fe3-80a2-ccf21eee7ba9
    16:01:56 : FEED -> Item not over keep_time (86400): urn:uuid:c6a472c6-2e9d-3ac6-bfe5-23fd2a7570e7
    16:01:56 : CANTO-FETCH -> Started thread for feed https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic
    16:01:56 : CANTO-DAEMON -> Beginning to serve...
    16:01:56 : FEED -> Fetched previous content for https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic.
    16:01:56 : CANTO-FETCH -> Started thread for feed https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic&os_username=benny&os_password=secret
    16:01:57 : CANTO-FETCH -> Started thread for feed https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic
    16:01:57 : CANTO-FETCH -> No content in https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic: <unknown>:2:942: mismatched tag
    16:01:59 : CANTO-FETCH -> Parsed https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic&os_username=benny&os_password=secret
    16:01:59 : CANTO-FETCH -> Plugins complete.
    16:01:59 : FEED -> Fetched previous content for https://jira.3s.intern/activity?maxResults=10&streams=key+IS+UNIX&providers=issues+thirdparty&os_authType=basic&os_username=benny&os_password=secret.
    16:02:00 : SHELF -> Unloaded.
    16:02:00 : SHELF -> Synced.
    16:02:00 : SHELF -> Successfully trimmed db
    16:02:00 : SHELF -> Shelf opened: <shelve.DbfilenameShelf object at 0x7f686213b048>
    
    opened by kjellgren 5
  • Data format compatibility with other RSS-readers.

    Data format compatibility with other RSS-readers.

    So when I'm at another house with my friends which are using MS Windows, I need to install VirtualBox, Linux and canto-curses in order to just simply read the news, meanwhile a friend of mine simply evil-smiling and using android app to read his rss. So I want to use RSS on the go (with my smartphone/another OS), but it's impossible with canto (because it's only ncurses and linux). Is it possible to implement / or make plugin for compatibility with other RSS readers?

    opened by ewancoder 5
  • HI CPU usage of canto-daemon when started

    HI CPU usage of canto-daemon when started

    Using canto-daemon 0.8.2 on ArchLinux on laptop. When started it uses at least one cpu at 100% for at least 5 minutes and also when fetching feeds. Was not paying attension until now, when I use more time on battery. htop screenshot: http://gallery.zebulj.si/var/albums/Razno/screenshot-2013-07-18.jpg?m=1374167699 canto-daemon log: http://zebulj.si/ajaxp/data/public/180638.php

    opened by Zaplanincan 4
  • Keep a limited number of items in each feed.

    Keep a limited number of items in each feed.

    As in the title, I'd like to have something like default_keep in 0.7, to keep a limited number of items.

    I've seen the keep (and rate, but that's not related to this issue) attributes in CantoFeed, but I don't see right away where they are used. Setting them does not seem to make a difference. Is this not implemented yet?

    Also, are read items ever removed? Is there something akin to never_discard from the previous version?

    opened by langner 4
  • Crazy canto-next behavior: always overwrite my config file with default version configs.

    Crazy canto-next behavior: always overwrite my config file with default version configs.

    1: I have a config file like this: https://gist.github.com/3765491

    Every time I open canto-curses, it will overwrite my new config file with default config file. really crazy.

    2: And I feel the new style of adding (write) feeds is really bad style. looks tooooooo long (if I have many feeds). Can Canto separate the feeds into a separate file ?

    opened by stardiviner 4
  • json.decoder.JSONDecodeError: Expecting value: line 13 column 5 (char 296)

    json.decoder.JSONDecodeError: Expecting value: line 13 column 5 (char 296)

    heres my config, I probably screwed up something here. You should totally check for valid JSON

    {
        "defaults": {
            "global_transform": "filter_read",
            "keep_time": 86400,
            "keep_unread": false,
            "rate": 10
        },
        "feeds": [
            {
                "name": "ctftime running ctfs",
                "url": "https://ctftime.org/event/list/running/rss/"
            }
        ],
        "tags": {
          # here
        }
    }
    

    daemon log

    Traceback (most recent call last):
      File "/usr/bin/canto-daemon", line 7, in <module>
        c = CantoBackend()
      File "/usr/lib/python3.10/site-packages/canto_next/canto_backend.py", line 125, in __init__
        self.get_config()
      File "/usr/lib/python3.10/site-packages/canto_next/canto_backend.py", line 794, in get_config
        config.parse()
      File "/usr/lib/python3.10/site-packages/canto_next/config.py", line 135, in parse
        self.read_config()
      File "/usr/lib/python3.10/site-packages/canto_next/config.py", line 152, in read_config
        self.json = json.load(c)
      File "/usr/lib/python3.10/json/__init__.py", line 293, in load
        return loads(fp.read(),
      File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 13 column 5 (char 296)
    
    opened by RewardedIvan 0
  • Added string padding/justification option

    Added string padding/justification option

    Added two variables to allow the user to "even out" the entries in their Reddit RSS feed.

    The variables are passed as parameters to str.ljust() for entry["reddit-score"] and entry["subreddit"].

    opened by drizbit 1
Owner
Jack Miller
Jack Miller
Simple Wayland HotKey Daemon

swhkd Simple Wayland HotKey Daemon This project is still very new and I'm making new decisions everyday as to where I should drive this project. I'm u

Aakash Sen Sharma 407 Dec 30, 2022
Hopefully the the next-generation backend server of bgm.tv

Hopefully the the next-generation backend server of bgm.tv

Bangumi 475 Jan 1, 2023
This Python3 script will monitor Upwork RSS feed and then email you the results.

Upwork RSS Parser This Python3 script will monitor Upwork RSS feed and then email you the results. Table of Contents General Info Technologies Used Fe

Chris 5 Nov 29, 2021
A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

Josh Cardenzana 0 Dec 13, 2021
Konomi: Kind and Optimized Next brOadcast watching systeM Infrastructure

Konomi 備考・注意事項 現在 α 版で、まだ実験的なプロダクトです。通常利用には耐えないでしょうし、サポートもできません。 安定しているとは到底言いがたい品質ですが、それでも構わない方のみ導入してください。 使い方などの説明も用意できていないため、自力でトラブルに対処できるエンジニアの方以外に

tsukumi 243 Dec 30, 2022
An application to see if your Ethereum staking validator(s) are members of the current or next post-Altair sync committees.

eth_sync_committee.py Since the Altair upgrade, 512 validators are randomly chosen every 256 epochs (~27 hours) to form a sync committee. Validators i

null 4 Oct 27, 2022
Listen Surah, prepare for next and Endless life...

Al-Quran In this repository, I have linked up all Surah with Arabic-Bangla Audio From Youtube. So, you just need to choose and listen. and the ( surah

SpiderX 1 Sep 30, 2022
Magenta: Music and Art Generation with Machine Intelligence

Magenta is a research project exploring the role of machine learning in the process of creating art and music. Primarily this involves developing new

Magenta 18.1k Jan 5, 2023
Procedural 3D data generation pipeline for architecture

Synthetic Dataset Generator Authors: Stanislava Fedorova Alberto Tono Meher Shashwat Nigam Jiayao Zhang Amirhossein Ahmadnia Cecilia bolognesi Dominik

Computational Design Institute 49 Nov 25, 2022
python DroneCAN code generation, interface and utilities

UAVCAN v0 stack in Python Python implementation of the UAVCAN v0 protocol stack. UAVCAN is a lightweight protocol designed for reliable communication

DroneCAN 11 Dec 12, 2022
An Agora Python Flask token generation server

A Flask Starter Application with Login and Registration About A token generation Server using the factory pattern and Blueprints. A forked stripped do

Nii Ayi 1 Jan 21, 2022
Python client SDK designed to simplify integrations by automating key generation and certificate enrollment using Venafi machine identity services.

This open source project is community-supported. To report a problem or share an idea, use Issues; and if you have a suggestion for fixing the issue,

Venafi, Inc. 13 Sep 27, 2022
VCM EE1.2 P-layer feature map anchor generation 137th MPEG-VCM

VCM EE1.2 P-layer feature map anchor generation 137th MPEG-VCM

IPSL 6 Oct 18, 2022
Python bindings for Basler's VisualApplets TCL script generation

About visualapplets.py The Basler AG company provides a TCL scripting engine to automatize the creation of VisualApplets designs (a former Silicon Sof

Jürgen Hock 2 Dec 7, 2022
Automated rop chain generation

This is the accompanying code to the blog post talking about automated rop chain generation. Build the test file with: make Install the dependencies:

Christopher Roberts 14 Nov 22, 2022
Python wrapper for Xeno-canto API 2.0. Enables downloading bird data with one command line

Python wrapper for Xeno-canto API 2.0. Enables downloading bird data with one command line. Supports multithreading

_zza 9 Dec 10, 2022
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (CVPR2021)

NExT-QA We reproduce some SOTA VideoQA methods to provide benchmark results for our NExT-QA dataset accepted to CVPR2021 (with 1 'Strong Accept' and 2

Junbin Xiao 50 Nov 24, 2022
NExT-Ford-aula4 - NExT Ford aula4

Questão 1: vocês deveram fazer o passo a passo de como ficará as pilhas(Stack) e

Gerson 1 Jan 6, 2022
Daemon to ban hosts that cause multiple authentication errors

__ _ _ ___ _ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \

Fail2Ban 7.8k Jan 9, 2023