It would be interesting to be able to launch the crawler several times on the same site, detect new files and analyse only the accessibility of these new files.

The crawl.sh script seemed to work just fine. Was able to scrape a good list of other documents this way.

Trying to run the analysis wasn't so good:

% ./analyse.sh 
find: ./crawled_files/10x.gsa.gov/*.pdf: No such file or directory
./crawled_files/apprenticeship.gov/2021%20Apprenticeship%20Mailer.pdf
Traceback (most recent call last):
  File "/Users/mgifford/Documents/GitHub/simplA11yPDFCrawler/./pdfCheck.py", line 1, in <module>
    from pikepdf import Pdf, String, _qpdf
ModuleNotFoundError: No module named 'pikepdf'
./crawled_files/apprenticeship.gov/29_cf_30_regs_only.pdf
Traceback (most recent call last):
  File "/Users/mgifford/Documents/GitHub/simplA11yPDFCrawler/./pdfCheck.py", line 1, in <module>
    from pikepdf import Pdf, String, _qpdf
ModuleNotFoundError: No module named 'pikepdf'

I'm running on a Mac, but didn't think that would be a problem:

% pip3 install pikepdf
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Requirement already satisfied: pikepdf in /usr/local/lib/python3.9/site-packages (4.2.0)
Requirement already satisfied: lxml>=4.0 in /usr/local/lib/python3.9/site-packages (from pikepdf) (4.6.3)
Requirement already satisfied: Pillow>=6.0 in /usr/local/lib/python3.9/site-packages (from pikepdf) (8.4.0)
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621

I've tried installing pikepdf on it's own with pip & pip3.

I notice that pikepdf is in the requirements.txt

Not sure if this is a problem at my end or not.

I did cut short my crawl.sh as I seemed to be getting a lot more errors. Anyways, don't think that's the cause of this. It is finding lots of files in the directory.

When this code has been developed, we were not aware of the existence of the VeraPDF tool. It could be integrated for our accessibility tests (PDF-UA1).

In some countries not all PDF files should be compliant. All files published before a given date are exempt. Currently this date is hard-coded for Luxembourg, it should be a parameter.

In the crawler, remove GET parameters in the filename when saving. Find a solution for duplicates. The file extension is used for some statistics, this can lead to further issues.