Read the Docs Conda Support

This will add the ability to generate documentation with conda environments on Read the Docs. This is mainly useful for libraries with large C dependencies, including many packages in the Scentific Python ecosystem.

Task List

• [x] Add Project option to use conda
• [x] Abstract the "environment path" in the builders to respect conda & virtualenv
• [x] Ensure cleanup of conda environment in wiping & other env delete scenarios
• [x] Ensure Python 2 & 3 support
• [x] Write tests for proper conda environment setup
• [x] Document conda support
• [ ] Write a topical guide for Choosing a Build Environment
• [ ] Optional: Investigate pruning

Abilities

You will be able to specify a conda environment.yml file, and Read the Docs will install these dependencies in your build environment.

Considerations

Read the Docs will keep seperate virtualenv & conda directories:

• /user_builds//env for pip
• /user_builds//conda for conda

Users will be able to define a way to install packages for a project:

• None
• Pip
• Conda

Read the Docs will need to change it's build code so that we don't hard-code virtualenv paths. We'll need to vary our environment creation, as well as bin path's for executables, based on the backend environment.

The other main thing is that we'll also need to install Sphinx & other build dependencies into the conda environment. We will continue to use pip for this, and it should be transparent, other than using the pip executable in the conda environment instead of the virtualenv.

It should also be noted that miniconda has a different install process from Python 2 and 3 -- also they recommend installing it from their bash scripts instead of pip. I hope that we will be able to use pip, as it will simplify our installation, and won't require an update to a bash script on version upgrades. We will have to see if we hit issues in testing.

Cleanup

Read the Docs will manage conda environment deletion on the removal of a project or version.

Documentation

We will need to add information about conda support to our documentation. We might want to add a topic guide around installing requirements, along with adding a specific reference for how to use & enable conda support.

Sponsorship

This work is being funded by Clinical Graphics -- many thanks for their support of Open Source.

It would be nice to be able to build docs for libraries managed with poetry.

maybe by checking for a [tool.poetry] section in pyproject.toml, having to keep a separate requirements.txt file in sync feels a bit fragile.

I was wondering if xelatex is supported for PDF generation? If so is the font Symobla.ttf installed as well? If not do you know when it will be supported.

Thank you

Details

• Project URL: https://readthedocs.org/projects/tethys-platform/
• Build URL (if applicable): https://readthedocs.org/projects/tethys-platform/builds/5043136/
• Read the Docs username (if applicable): swainn

Expected Result

I expect the docs for the dev branch of my project to build.

Actual Result

I encounter a failure during the creation of the conda environment (see full output below).

Things I have tried:

(1) Found this suggestion on the Conda issues page, so I attempted wiping the environment several times. (2) Reverted the back to commit with last successful docs build, but still failed with same error. (3) Created this Conda environment my dev machine successfully (using miniconda).

Here is the error:

conda env create --name dev --file /home/docs/checkouts/readthedocs.org/user_builds/tethys-platform/checkouts/dev/docs-conda_env.yml Fetching package metadata ......... Solving package specifications: .......... WARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13] An unexpected error has occurred. Please consider posting the following information to the conda GitHub issue tracker at:

https://github.com/conda/conda/issues

Current conda install:

           platform : linux-64
      conda version : 4.2.12
   conda is private : False
  conda-env version : 4.2.12
conda-build version : not installed
     python version : 2.7.12.final.0
   requests version : 2.11.1
   root environment : /usr/local/miniconda  (read only)
default environment : /home/docs/checkouts/readthedocs.org/user_builds/tethys-platform/conda/dev
   envs directories : /home/docs/checkouts/readthedocs.org/user_builds/tethys-platform/conda
                      /home/docs/.conda/envs
                      /usr/local/miniconda/envs
      package cache : /home/docs/checkouts/readthedocs.org/user_builds/tethys-platform/conda/.pkgs
                      /home/docs/.conda/envs/.pkgs
                      /usr/local/miniconda/pkgs
       channel URLs : https://repo.continuum.io/pkgs/free/linux-64
                      https://repo.continuum.io/pkgs/free/noarch
                      https://repo.continuum.io/pkgs/pro/linux-64
                      https://repo.continuum.io/pkgs/pro/noarch
        config file : None
       offline mode : False

$ /usr/local/miniconda/bin/conda-env create --name dev --file /home/docs/checkouts/readthedocs.org/user_builds/tethys-platform/checkouts/dev/docs-conda_env.yml

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/exceptions.py", line 479, in conda_exception_handler
    return_value = func(*args, **kwargs)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda_env/cli/main_create.py", line 111, in execute
    installer.install(prefix, pkg_specs, args, env)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda_env/installers/conda.py", line 38, in install
    raise CondaRuntimeError('RuntimeError: %s' % e)
CondaRuntimeError: Runtime error: RuntimeError: Runtime error: Could not open u'/usr/local/miniconda/pkgs/yaml-0.1.6-0.tar.bz2.part' for writing ([Errno 13] Permission denied: u'/usr/local/miniconda/pkgs/yaml-0.1.6-0.tar.bz2.part').

Fetching packages ... yaml-0.1.6-0.t 0% | | ETA: --:--:-- 0.00 B/s

See there:

http://borgbackup.readthedocs.org/en/stable/

Link at top right points to: https://github.com/borgbackup/borg/blob/55fef2457e483701254e5b2a7d4a1a60963971a8/docs/index.rst (which is 404)

Same for the view/edit links in the bottom left box, they also point to same 404 location.

OTOH, the docs shown for "stable" are (currently) correctly showing 0.28.2 release, which is the latest release.

I tried a "stable" rebuild, did not help.

Details

• Project URL: http://github.com/molpopgen/fwdpy
• Build URL (if applicable): https://readthedocs.org/projects/fwdpy/builds/5121049/
• Read the Docs username (if applicable): molpopgen

Expected Result

The project is a Cython extension module that depends on C++11 libraries. The relevant dependent libraries are installed via conda (docs/environment.yml) in the repo. In order for Sphinx to work, the module must be built or installed, at which point the docs should be built.

Actual Result

The header file from one of the dependencies installed via conda cannot be found (see build log). I've not taken the time to replicate this behavior via the docker image, but it seems like I'm doing something wrong. The gcc for the docker image is is g++5, which is more than good enough.

The problem seems to be that after installing the conda packages, attempting to install my project is done in an environment where the conda installs do not exist. I've not been able to find explicit docs on how to use dependent C/C++ libraries in a build beyond including them in the environment.yml file.

It'd be great if RTD supported HTTPS when using a custom domain. Once upon a time this would have been difficult and expensive, however nowadays with SNI and Lets Encrypt I think this is possible -- providers like SquareSpace and Wordpress have demonstrated this.

Here's how it could work:

• When a new custom domain is added, trigger an attempt to obtain a certificate, triggering the lets encrypt validation flow.
• Redirect all /.well-known/* from all RTD sites to a special validation server
• On the validation server, serve the correct value for the LE challenge for that domain
• Obtain the certificate, and store it somewhere.
• (Make sure to repeat every X days so certs don't expire!)
• When a request comes in with SNI=foo.com, look up that cert and server it if it's a RTD site

Hopefully that makes sense!

Details

• Project URL: http://channels.readthedocs.io
• Build URL (if applicable): http://channels.readthedocs.io/en/stable/
• Read the Docs username (if applicable): andrewgodwin

Expected Result

I expect the "stable" build version to only follow tags, not new branches, as described in http://docs.readthedocs.io/en/latest/versions.html

Actual Result

The "stable" build says it is following origin/2.0, which looks like a reference to a remote Git branch rather than the project's most recent tag, 1.1.8, which means the "stable" docs have ended up being the in-development version docs.

The "wipe" command does not appear to fix this, and I would like to avoid deleting and remaking the project just to see if it's a bad tag being stored somewhere.

• Project URL: https://qnet.readthedocs.io/en/latest/index.html#
• Build URL (if applicable): https://readthedocs.org/projects/qnet/builds/7008115/

I'm seeing a strange effect where the navigation sidebar is broken depending on the RTD configuration. The effect is that clicking on the "Read the Docs"/version at the very bottom of the sidebar does nothing (it should pop up the version selector). Also, on a small screen where the sidebar is hidden by default, clicking on the sidebar button (the symbol of three horizontal bars in the top left corner) has no effect (as opposed to showing the sidebar).

The broken behavior can be seen in the current build on the Project URL. When loading that page e.g. in Safari with an open Web Inspector, there is an error message that hints at the problem:

[Error] TypeError: undefined is not an object (evaluating 'SphinxRtdTheme.Navigation.enableSticky')
	(anonymous function) (index.html:365)
	f (jquery-2.0.3.min.js:3:2256)
	fireWith (jquery-2.0.3.min.js:3:3059)
	ready (jquery-2.0.3.min.js:1:12573)
	ge (jquery-2.0.3.min.js:1:10197)

The project uses a readthedocs.yml file and it's set up to use conda, and also to install the package for which the documentation is generated via pip (pip_install: true)

It turns out that this is the problem. If I switch from pip_install to setup_py_install (and adapt the environment.yml file accordingly to install all pip-prerequisites manually), it works fine.

The commit that makes everything work is https://github.com/mabuchilab/QNET/commit/cc55fe4d789c32ee25436af4749a2d7d9bc6c2ad

I've reverted that commit to have a "broken" build, which I'll leave up for a few days to give people a chance to look at it.

I wonder if the problem is that I have sphinx_rtd_theme listed in setup.py: maybe this overwrites the system-sphinx-rtd_theme during the pip_install of the package, which doesn't happen with setup_py_install?

Details

• Read the Docs project URL: n/a
• Build URL (if applicable): n/a
• Read the Docs username (if applicable): JoshIzaac

Expected Result

I am having trouble importing repositories from an organization (https://github.com/PennyLaneAI). I am listed as an owner of the organization, and 3rd party access has been provided to the organization:

Actual Result

Pressing the sync button on my ReadTheDocs 'Import a Project' page has no effect; the PennyLaneAI organization is not listed.

Details

• Read the Docs project URL:https://readthedocs.org/projects/matchzoo/
• Build URL (if applicable):https://readthedocs.org/projects/matchzoo/builds/7406071/ and https://readthedocs.org/projects/matchzoo/builds/77406100/
• Read the Docs username (if applicable):wqh17101

Expected Result

Get the right content which generated by sphinx-apidoc

Actual Result

The 77406100 is the one can build normally. But it can not display the content generated by sphinx-apidoc ,so i check the log i find an error , obviously, it caused the bug. So i fixed it and commit the file . it works in my local env but in the readthedoc's env ,it got the result of 7406071 i got a core dump And it seems that whatever i change (the right code in my env )and commit ,it would usually get a core dump.

I just learned of the management command format:

python manage.py sync_vcs_data --users saadmk11

From: https://github.com/readthedocs/readthedocs.org/pull/7803

This seems useful to fix/debug issues with user connected accounts, and something that we should surface in the admin dashboard. I do like having user and admin operations centralized in the admin dashboard, as management commands are not as easy to share knowledge on or even find.

Basically remove this: https://github.com/readthedocs/readthedocs.org/blob/8afcb48c1a637b50622fba7a6a210b7a043e6272/readthedocs/proxito/views/serve.py#L177-L186

We kept this turned off for performance, but now that we're on a CDN we don't hit this code nearly as frequently.

https://github.com/readthedocs/readthedocs.org/pull/9243

Changes:

• Rename the page from custom_domains.rst -> custom-domain.rst (I'll create a redirect when this is approved)
• Describe canonical urls in their own page (since they work even if you don't add a custom domain)
• The creation of a custom domain for .com and .org are almost the same, so no more tabs
• I have updated the FAQ about CAA records to mention that we already set the correct ones
• Describe default subdomains on the hosting.rst page.

ref https://github.com/readthedocs/readthedocs.org/issues/9258

When creating the container to build the documentation, we are mounting different paths depending on if we are running locally in the development instance or in production. This differentiation makes the local development instance to use root as the user to execute all the commands, while on production we are using docs. This has bitten us multiple times, in particular with errors like "Permission denied" on files and also when working with asdf.

Besides, as we are using the same path on the host to be shared, when running multiple Celery workes locally this breaks because they overrides the files that the other worker is using.

• Mount different paths depending on the environment: https://github.com/readthedocs/readthedocs.org/blob/3ff05c939bed20c6071b3aa16691d46ec48ab080/readthedocs/doc_builder/environments.py#L717-L742
• Run different code depending if we are root or docs for asdf: https://github.com/readthedocs/readthedocs.org/blob/3ff05c939bed20c6071b3aa16691d46ec48ab080/readthedocs/doc_builder/python_environments.py#L54-L68

I'd like to remove this conditional logic and make usage of Docker volumes instead:

1. create a temporal volume
2. spin the container and attach the volume under the same path we are currently using
3. build the project
4. delete the container
5. delete the temporal volume

By doing this, all the logic executed by CLEAN_AFTER_BUILD=True can be completely removed since we are deleting the temporal volume which will remove everything from the build and the new build will start fresh. So, this idea will also remove this complexity.

@humitos noted in chat:

The first step is to decide those top 5 features, which I think they are:

• versioning

• full-text search

• pull request preview

• build customization

• custom domains with instant SSL support

• for Business, add to that list as well:

  • CDN
  • SSO
  • auth auditing

That said, I think these are the pages we should focus on initially:

• https://docs.readthedocs.io/en/latest/features.html
  • improve that page mentioning those top 5 features that immediately catches readers' attention (using more marketing language)
  • currently, pull request preview is not even mentioned there :disappointed:
• https://docs.readthedocs.io/en/latest/versions.html
  • in-depth but simple explanation about how it works
  • explain how latest and stable work with more narrative content
  • remove deprecated VCS (svn, bzr) from the table --maybe remove the table completely after that since there is no differentiation to do
• https://docs.readthedocs.io/en/latest/server-side-search.html
  • we call the feature "full-text search" support, but the page is named "Server Side Search"; we should probably unify them
  • the content looks good to me, just some minor edits would be fine for now
• https://docs.readthedocs.io/en/latest/pull-requests.html
  • the content looks good, it could use a more marketing language to sell it more, but it's fine for now
  • maybe some small edits will help
• https://docs.readthedocs.io/en/stable/builds.html and https://docs.readthedocs.io/en/stable/build-customization.html
  • these page were updated some weeks ago
  • anthony made some notes of improvements he wanted to see here, so we can recover those notes and update it
• https://docs.readthedocs.io/en/stable/custom_domains.html
  • starts talking about subdomain support, why? the page is about custom domains. That seems distracting to me and could be moved to another page where we talk about hosting features or similar
  • the main section "Custom domain support" could have more narrative content
  • the content from this section could be moved to "Configuration" to keep consistency with other feature pages
  • mention SSL support bigger :)
  • canonical URLs content can be improved a little

Note that these pages should be listed at the top of the "Feature overview" title in the left sidebar instead of hidden between other less relevant features.

This is a good list to start from. I won't break out individual cards, but feel free to assign below:

• [ ] Features
• [ ] Versions
• [ ] Search
• [ ] Pull requests
• [ ] Builds
• [ ] Custom domains