Utility for downloading fanfiction in bulk from the Archive of Our Own

Hello,nineyna! First and foremost, thank you so much for creating this wonderful program. I have 0 experience in programming and I really appreciate your instructions and this program in general. ( I sent an ask previously if this question is familiar to you.)

I have run into an issue while using it. I can't download individual stories from collections for example Big Bangs. If I include stories in a series, they are able to be downloaded, however the individual fics do not show up in the downloads folder.

I have tried deleting the settings, re-downloading the program itself. and I tried to see if the files were hidden but still nothing. I checked for this issue in several separate collections as well as tried to see if maybe my filters affected the download but the result is the same.

I have checked the logs and there is only the link of the collection listed but then no stories listed as being downloaded if I don't select to download from a series.

I don't have this issue if I am downloading from my favorites, from a creators page or from a works tag, just the collections.

I do not know if other people also ran into this issue or if it is just me. I understand if you do not have the time or if it is not possible to fix this issue but I wanted to bring this to your attention.

Thank you so much for making this awesome program! Me and my library are very grateful to you.

Let me preface by saying I don't know coding at all and am following the guide to the best of my ability. I run a Mac OS and I'm getting a return error in the initial set-up run command window following the command "pip install -r requirements.txt". I think what the error means is that I'm missing a key code package, but I figure it would be best to ask the developer to be sure. Code pasted below:

Collecting beautifulsoup4==4.9.3 Using cached beautifulsoup4-4.9.3-py3-none-any.whl (115 kB) Collecting certifi==2020.12.5 Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB) Collecting cffi==1.15.0 Using cached cffi-1.15.0-cp310-cp310-macosx_11_0_arm64.whl (173 kB) Collecting chardet==4.0.0 Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB) Collecting colorama==0.4.4 Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB) Collecting cryptography==36.0.1 Using cached cryptography-36.0.1-cp36-abi3-macosx_10_10_universal2.whl (4.8 MB) Collecting cssselect==1.1.0 Using cached cssselect-1.1.0-py2.py3-none-any.whl (16 kB) Collecting EbookLib==0.17.1 Using cached EbookLib-0.17.1.tar.gz (111 kB) Preparing metadata (setup.py) ... done Collecting idna==2.10 Using cached idna-2.10-py2.py3-none-any.whl (58 kB) Collecting loguru==0.5.3 Using cached loguru-0.5.3-py3-none-any.whl (57 kB) Collecting lxml==4.6.4 Using cached lxml-4.6.4.tar.gz (3.2 MB) Preparing metadata (setup.py) ... error error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [3 lines of output] Building lxml version 4.6.4. Building without Cython. Error: Please make sure the libxml2 and libxslt development packages are installed. [end of output]

Hello,

Thank you for making the the AO3 downloader. I have an issue. I decided to download the newest versions of the downloader and python but now have a NoModuleFound: No module named "requests" error. I think I downloaded everything correctly but I don't know what to do now.

Thank you for reading.

Regression caused by #43. Previously, the downloader only recognized internal/relative work links (of the form /works/12345). Now, work links of the form [...]/works/12345 are included also. This has the unintended side effect of including absolute work links (https://archiveofourown.org/works/12345) in the list. This can happen when an author manually links to another work on ao3 in their summary. Aside from not being intended behavior, the ao3 root url is still prepended to the absolute links (https://archiveofourown.orghttps://archiveofourown.org/works/12345) causing mayhem.

tl;dr amend the work and series patterns to recognize internal (starting with "/") links only.

Currently the program does its best to skate just under the ao3 rate limit, which makes it go as fast as possible but tends to foul up casual ao3 browsing while it's running. Add an option to halve the speed so you can browse in peace.

Nothing fancy here. Probably just gonna label them with the fic name and dump them in a subfolder in downloads (presuming I can get it to work in the first place, but theoretically it should just be a matter of parsing the soup for img tags)

The pinboard and redownload functions are a bit scary because it is expected that they might need to download very large numbers of fics - potentially taking many hours - and there is absolutely no way to pause them in the middle. There has GOT to be some way to save state on these guys.

• Make all custom exceptions inherit from a base class
• Don't bother logging stack trace for custom exceptions... I know where they came from
• Instead of "message" field, just pass the relevant message into the exception constructor (no idea why I didn't do this in the first place...)

Currently a log is only written at the download step, and that only happens if the fic is found to have new chapters. It would be better to have visibility into all incomplete fics, including the ones that haven't been updated.

Related to #10

Integrating a way to preserve workskins is challenging. As a stopgap, add a column to the log file to allow users to identify works which have custom skins, so that they can manually download them some other way. h/t @heartlessglasses for this idea.

Discussed in https://github.com/nianeyna/ao3downloader/discussions/41

Specifically:

• inside repo.py when alerting the user about status 429 events
• inside soup.py when alerting the user that a new page has started downloading

Reasoning: separation of concerns between the logic layer and the UI layer

Would want to do this in some persistent way that doesn't require running the program in order to change. Possibly an ini file. Should probably be turned off by default, rather than on like it currently is.