What works:

• Starting the calibre GUI
• Calibre C extensions
• Adding a book

What doesn't work:

• Recipies
• Everything else
• Python 2 compatibility

Anyway, this pull request is just here to get some early feedback. There's a lot of work that is left to do, like manually testing as many code paths as possible, as well as porting recipies to use MechanicalSoup or something like that instead of mechanize.

My goal is to have this code work the same on python2 and on python3, with the goal of eventually dropping python2 support late 2019 or early 2020.

To bootstrap this version, you'll need to install the following pip packages:

apsw==3.9.2.post1
dukpy==0.3
Elements==0.2.0
feedparser==5.2.1
html2text==2018.1.9
html5-parser==0.4.5
netifaces==0.10.7

and delete all recipes except ajc.recipie

...metadata. Add a formatter function to use it.

@Kovid: I am not completely happy with this implementation. The changes to cache.py are required to avoid infinite recursion. The method user_categories_for_book calls field_for() which calls composite_for(). The composite_for method created a new ProxyMetadata object and then attempts to evaluate the composite column in question. If that column was the one with a template calling user_categories_for_book, it recurses because the new ProxyMetadata object doesn't see the original's cache.

As you can see, I put a stop to the recursion by passing in the instance of proxy_metadata for the field_for method to use. I thing that makes me nervous with this approach is that it changes a fundamental API. I considered trying to use more basic APIs in user_categories_for_book, but that has the drawbacks of code cloning and being more difficult.

If you have an alternate idea to avoid the recursion, I am all ears.

BTW: the fact that composite_for creates a new ProxyMetadata instance raises the question of whether the caches are as effective as they should/could be. It seems that the value will be cached twice and then discarded once.

Added a variety of functions to viewport to allow working based on block and inline direction rather than X and Y.

Changed paged mode and flow mode to be based on block and inline directions, making them agnostic to writing direction.

Added jump_to_cfi function to flow mode code, and use this in iframe. This fixes some issues with CFI jumping in flow mode.

Use self.jump_to_cfi in iframe so that it is based on the current mode.

Removed a redundant self.onscroll call that was causing CFIs to be queried twice on load, doubling the time they took.

Fixed some bugs related to scrolling in flow mode related to a decorator not working with keyword arguments, by removing the keyword arguments.

Fixed some bugs with flow mode visibility anchor function.

CFI changes: Renamed some functions and variables to clarify meaning.

Use getBoundingClientRect in range_has_point since it doesn't care about which particular rect matches.

Remove use of exceptions, since they were causing CFI calculation to run about six times more slowly in benchmarks. (Worst-case went from ~6 to ~1 second.)

Remove forward flag and allow text offset to be equal to node length, and correct for this by adding a flag to the decoded CFI to indicate this.

Added comments clarifying the results of decoding a CFI.

Fix bugs in the point (now decode_with_range) function and simplified it.

Added new decoded_with_range_to_document_position function, to replace two places where this was being done in slightly different ways.

scroll_to did not work in my tests, so I fixed the bugs and simplified the logic.

Rewrote at_current to do a much simplified scan of the viewport in the inline and then block direction.

This implements changes to page indicator to allow for chapter length and delta display. There is a new configuration option that controls the new behavior.

I find this helpful in determining when there is a natural stopping point, while reading.

There are small optimizations that can be made, and I know the current reader is being phased out. If there are positive comments on this feature, I will move on and add these changes, there.

Added an option to switch off saving and showing the SMTP password. Enabling the option induces a "Enter password" dialog when ever the user starts an email sending job. Drawback is, that automatic email sending must be prevented.

• Optional dithering down to the exact eInk color palette.

  Recent FW versions got better at this than they used to, but it's still not perfect. Note that Mk. 7 devices can and will enforce HW dithering, at the very least on sleep screens, using an ordered dither pattern, like we do.

  Speaking of, said ordered dithering pattern has been ported from ImageMagick.

  This has the added bonus of generating much smaller PNG files.

  Depends on B&W covers to avoid nonsensical results.

• Optional letterboxing of full-screen covers to avoid extra Nickel processing.

  Depends on Keep AR to avoid nonsensical results.

• Optional storage as PNG to avoid JPG wrecking the dithering

  Depends on B&W covers to avoid storing stupidly large color PNGs.

• Revamp thumbnail dimension calculations to match what's done with store-bought KePubs, fixing a few AR rounding issues in the process.

• Speaking of thumbnails, the target sizes have been unified, and the sleep cover dimensions have been updated for a few specific models (Aura, H2O, Forma) to match their effective screen resolution, instead of something less accurate.

If an author last name is preceded by von, van, di, la, le, da, de, etc. include that in the last name.

So Leonardo da Vinci -> da Vinci, Leonardo instead of Vinci, Leonardo da. This seems to generally be the more acceptable approach. A notable exception is Ludwid van Beethoven. However, he was not a popular author of literature, and most of the examples I've come across sorting my library do follow the rule.

I've also simplified / sped up the tokenizing code. There is no need to allocate a new list for every prefix and suffix that is removed. Simply looping over the allowed range and using the else clause of the for loop does the same thing, but simpler. In the same spirit, I removed the need to recount all the Counter values when stripping parentheses.

My one concern is that I'm not sure where to put unit tests for the changes.

As discussed in: https://www.mobileread.com/forums/showthread.php?t=333893 Open points:

• [x] disable updates
• [x] handle load failures
• [x] prevent removal
• [x] prevent replacement
• [x] mention in the UI that this is managed system-wide
• [ ] new translations
• [ ] documentation

This prevents general errors when trying to use types that no longer exist in python3.

Not all code has been updated to use this, some things like src/tinycss/ and src/css_selectors/ already have small shims for this which I did not migrate to import from polyglot.

BeautifulSoup is something I left alone for now, because it seems like it would be complex to take care of and I was wondering if maybe it would make more sense, in the long run, to investigate the feasibility of migrating to beautifulsoup4.

Before I do src/odf, I was wondering if it would be feasible to push changes upstream to https://github.com/eea/odfpy (which currently supports python3 and seems to be getting somewhat active development) and then drop it from the calibre tree.

We cannot do the metadata sources, store plugins, and recipes until a released version of calibre has polyglot.builtins.unicode_type, so I've begun work on that here: https://github.com/eli-schwartz/calibre/commit/642f454fcc0d587fc094ad04801462a2e3400bd9 and will submit a followup PR once a) there is a release, b) I finish up the recipes/ directory. This can probably be left alone for a number of release cycles to minimize the disruption...

Also fix not recomputing the search cache when vls are edited.

There is one backwards incompatibility that I found. If a user has created a grouped search term with the name "vl" then the new vl search location will hide that grouped term.

Hi,

I am not a python programmer (and this is the first time I was writing any python code) so please inspect my PR. I tried my best to write code as clean as possible, but I might not know some python quirks so I might make some mistakes there. Anyway, this PR should add the following feature (I tested it and it is working perfectly for me):

Added way to make custom labeled page break in APNX file using ARIA ( Accessible Rich Internet Applications) tag in html.

Using <any_html_element *** pagebreak *** aria-label="some_label" ***> will produce page break location. Possible usage

<span id="pg159" role="doc-pagebreak" aria-label="159"/>
<h1 id="pg_header1" role="doc-pagebreak" aria-label="Header 1">Header</h1>
<p role="doc-pagebreak" aria-label="§ 1 part 4 page 6 of 9">Text</p>

Hi!

Today I tried to setup Calibre for my mother to use at home, and it revealed to be very useful (good job!) but also a bit frustrating on the search panel when the application fetch many websites, and it takes ages to finish (we have a somehow slow connexion).

Hence quickly setup a dev environment (really easy to set up, good job again!) to try to figure out which online stores are slow to answer and should be disabled.

This resulted in this PR, that adds a bit of extra logging resulting in those lines:

While loading OpenSearch description, could not fetch URL http://assets0.feedbooks.net/opensearch.xml?t=1253087147
Overseer.fetch_url took 3.24s for URL https://www.amazon.com/s/?i=digital-text&k=georges+perec
Overseer.fetch_url took 8.23s for URL https://www.kobobooks.com/search/search.html?q=georges+perec
While loading OpenSearch description, could not fetch URL http://www.manybooks.net/opds/
Overseer.fetch_url took 9.65s for URL https://www.kobo.com/fr/en/ebook/the-art-and-craft-of-approaching-your-head-of-department-to-submit-a-request-for-a-raise
Overseer.fetch_url took 9.52s for URL https://www.kobo.com/fr/en/ebook/portrait-of-a-man-10
Overseer.fetch_url took 9.75s for URL https://www.kobo.com/fr/en/ebook/life-38
Overseer.fetch_url took 3.74s for URL https://www.kobo.com/fr/en/ebook/w-or-the-memory-of-childhood-by-georges-perec-book-analysis
Overseer.fetch_url took 4.37s for URL https://www.kobo.com/fr/en/ebook/georges-perec-a-life-in-words
Overseer.fetch_url took 4.75s for URL https://www.kobo.com/fr/en/ebook/georges-perec-is-my-hero
While loading OpenSearch description, could not fetch URL http://www.manybooks.net/opds/
While loading OpenSearch description, could not fetch URL http://assets0.feedbooks.net/opensearch.xml?t=1253087147
Overseer.fetch_url took 9.55s for URL https://www.kobobooks.com/search/search.html?q=georges+perec
Overseer.fetch_url took 5.44s for URL https://www.amazon.com/s/?i=digital-text&k=georges+perec
Overseer.fetch_url took 9.67s for URL https://www.kobo.com/fr/en/ebook/the-art-and-craft-of-approaching-your-head-of-department-to-submit-a-request-for-a-raise
Overseer.fetch_url took 9.86s for URL https://www.kobo.com/fr/en/ebook/portrait-of-a-man-10
Overseer.fetch_url took 9.88s for URL https://www.kobo.com/fr/en/ebook/i-remember-32
Overseer.fetch_url took 9.92s for URL https://www.kobo.com/fr/en/ebook/life-38
Overseer.fetch_url took 1.62s for URL https://www.kobo.com/fr/en/ebook/georges-perec-a-life-in-words
Overseer.fetch_url took 2.14s for URL https://www.kobo.com/fr/en/ebook/georges-perec-is-my-hero
Overseer.fetch_url took 2.78s for URL https://www.kobo.com/fr/en/ebook/the-hand-9
Overseer.fetch_url took 3.43s for URL https://www.kobo.com/fr/en/ebook/w-or-the-memory-of-childhood-by-georges-perec-book-analysis
Overseer.fetch_url took 2.40s for URL https://www.kobo.com/fr/en/ebook/diary-of-a-country-priest-2
Overseer.fetch_url took 5.07s for URL https://amazon.com/dp/B004ASOVSK
Overseer.fetch_url took 8.27s for URL https://amazon.com/dp/B085GMM4BK
Overseer.fetch_url took 13.14s for URL https://amazon.com/dp/B01A4APN84
Overseer.fetch_url took 8.85s for URL https://amazon.com/dp/B08T6JFSCJ
Overseer.fetch_url took 7.73s for URL https://amazon.com/dp/B0B9YNRNFH
Overseer.fetch_url took 14.65s for URL https://amazon.com/dp/B0711N6Y6C
Overseer.fetch_url took 5.33s for URL https://amazon.com/dp/B088FW2PDB
Overseer.fetch_url took 6.30s for URL https://amazon.com/dp/B00WNYBBTU
Overseer.fetch_url took 6.11s for URL https://amazon.com/dp/B081HJ5H8J
Overseer.fetch_url took 7.01s for URL https://amazon.com/dp/B00HRKONM2
Overseer.fetch_url took 5.64s for URL https://amazon.com/dp/B00SLVOLKU
Overseer.fetch_url took 5.45s for URL https://amazon.com/dp/B071P8C9HS
Overseer.fetch_url took 6.91s for URL https://amazon.com/dp/B010BEEWNU
Overseer.fetch_url took 6.83s for URL https://amazon.com/dp/B07F165YX3
Overseer.fetch_url took 6.19s for URL https://amazon.com/dp/B084YR8R66
Overseer.fetch_url took 7.07s for URL https://amazon.com/dp/B07147SK5W

I found it quite useful. Do you think this would make a nice addition to Calibre?

Regards

Add primary-writing-mode option for supporting right-to-left language ebooks such as Japanese Manga.

Feature request here:

https://bugs.launchpad.net/calibre/+bug/1919006