Read the Docs Conda Support

This will add the ability to generate documentation with conda environments on Read the Docs. This is mainly useful for libraries with large C dependencies, including many packages in the Scentific Python ecosystem.

Task List

• [x] Add Project option to use conda
• [x] Abstract the "environment path" in the builders to respect conda & virtualenv
• [x] Ensure cleanup of conda environment in wiping & other env delete scenarios
• [x] Ensure Python 2 & 3 support
• [x] Write tests for proper conda environment setup
• [x] Document conda support
• [ ] Write a topical guide for Choosing a Build Environment
• [ ] Optional: Investigate pruning

Abilities

You will be able to specify a conda environment.yml file, and Read the Docs will install these dependencies in your build environment.

Considerations

Read the Docs will keep seperate virtualenv & conda directories:

• /user_builds//env for pip
• /user_builds//conda for conda

Users will be able to define a way to install packages for a project:

• None
• Pip
• Conda

Read the Docs will need to change it's build code so that we don't hard-code virtualenv paths. We'll need to vary our environment creation, as well as bin path's for executables, based on the backend environment.

The other main thing is that we'll also need to install Sphinx & other build dependencies into the conda environment. We will continue to use pip for this, and it should be transparent, other than using the pip executable in the conda environment instead of the virtualenv.

It should also be noted that miniconda has a different install process from Python 2 and 3 -- also they recommend installing it from their bash scripts instead of pip. I hope that we will be able to use pip, as it will simplify our installation, and won't require an update to a bash script on version upgrades. We will have to see if we hit issues in testing.

Cleanup

Read the Docs will manage conda environment deletion on the removal of a project or version.

Documentation

We will need to add information about conda support to our documentation. We might want to add a topic guide around installing requirements, along with adding a specific reference for how to use & enable conda support.

Sponsorship

This work is being funded by Clinical Graphics -- many thanks for their support of Open Source.

It would be nice to be able to build docs for libraries managed with poetry.

maybe by checking for a [tool.poetry] section in pyproject.toml, having to keep a separate requirements.txt file in sync feels a bit fragile.

I was wondering if xelatex is supported for PDF generation? If so is the font Symobla.ttf installed as well? If not do you know when it will be supported.

Thank you

Details

• Project URL: https://readthedocs.org/projects/tethys-platform/
• Build URL (if applicable): https://readthedocs.org/projects/tethys-platform/builds/5043136/
• Read the Docs username (if applicable): swainn

Expected Result

I expect the docs for the dev branch of my project to build.

Actual Result

I encounter a failure during the creation of the conda environment (see full output below).

Things I have tried:

(1) Found this suggestion on the Conda issues page, so I attempted wiping the environment several times. (2) Reverted the back to commit with last successful docs build, but still failed with same error. (3) Created this Conda environment my dev machine successfully (using miniconda).

Here is the error:

conda env create --name dev --file /home/docs/checkouts/readthedocs.org/user_builds/tethys-platform/checkouts/dev/docs-conda_env.yml Fetching package metadata ......... Solving package specifications: .......... WARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13] An unexpected error has occurred. Please consider posting the following information to the conda GitHub issue tracker at:

https://github.com/conda/conda/issues

Current conda install:

           platform : linux-64
      conda version : 4.2.12
   conda is private : False
  conda-env version : 4.2.12
conda-build version : not installed
     python version : 2.7.12.final.0
   requests version : 2.11.1
   root environment : /usr/local/miniconda  (read only)
default environment : /home/docs/checkouts/readthedocs.org/user_builds/tethys-platform/conda/dev
   envs directories : /home/docs/checkouts/readthedocs.org/user_builds/tethys-platform/conda
                      /home/docs/.conda/envs
                      /usr/local/miniconda/envs
      package cache : /home/docs/checkouts/readthedocs.org/user_builds/tethys-platform/conda/.pkgs
                      /home/docs/.conda/envs/.pkgs
                      /usr/local/miniconda/pkgs
       channel URLs : https://repo.continuum.io/pkgs/free/linux-64
                      https://repo.continuum.io/pkgs/free/noarch
                      https://repo.continuum.io/pkgs/pro/linux-64
                      https://repo.continuum.io/pkgs/pro/noarch
        config file : None
       offline mode : False

$ /usr/local/miniconda/bin/conda-env create --name dev --file /home/docs/checkouts/readthedocs.org/user_builds/tethys-platform/checkouts/dev/docs-conda_env.yml

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/exceptions.py", line 479, in conda_exception_handler
    return_value = func(*args, **kwargs)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda_env/cli/main_create.py", line 111, in execute
    installer.install(prefix, pkg_specs, args, env)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda_env/installers/conda.py", line 38, in install
    raise CondaRuntimeError('RuntimeError: %s' % e)
CondaRuntimeError: Runtime error: RuntimeError: Runtime error: Could not open u'/usr/local/miniconda/pkgs/yaml-0.1.6-0.tar.bz2.part' for writing ([Errno 13] Permission denied: u'/usr/local/miniconda/pkgs/yaml-0.1.6-0.tar.bz2.part').

Fetching packages ... yaml-0.1.6-0.t 0% | | ETA: --:--:-- 0.00 B/s

See there:

http://borgbackup.readthedocs.org/en/stable/

Link at top right points to: https://github.com/borgbackup/borg/blob/55fef2457e483701254e5b2a7d4a1a60963971a8/docs/index.rst (which is 404)

Same for the view/edit links in the bottom left box, they also point to same 404 location.

OTOH, the docs shown for "stable" are (currently) correctly showing 0.28.2 release, which is the latest release.

I tried a "stable" rebuild, did not help.

Details

• Project URL: http://github.com/molpopgen/fwdpy
• Build URL (if applicable): https://readthedocs.org/projects/fwdpy/builds/5121049/
• Read the Docs username (if applicable): molpopgen

Expected Result

The project is a Cython extension module that depends on C++11 libraries. The relevant dependent libraries are installed via conda (docs/environment.yml) in the repo. In order for Sphinx to work, the module must be built or installed, at which point the docs should be built.

Actual Result

The header file from one of the dependencies installed via conda cannot be found (see build log). I've not taken the time to replicate this behavior via the docker image, but it seems like I'm doing something wrong. The gcc for the docker image is is g++5, which is more than good enough.

The problem seems to be that after installing the conda packages, attempting to install my project is done in an environment where the conda installs do not exist. I've not been able to find explicit docs on how to use dependent C/C++ libraries in a build beyond including them in the environment.yml file.

It'd be great if RTD supported HTTPS when using a custom domain. Once upon a time this would have been difficult and expensive, however nowadays with SNI and Lets Encrypt I think this is possible -- providers like SquareSpace and Wordpress have demonstrated this.

Here's how it could work:

• When a new custom domain is added, trigger an attempt to obtain a certificate, triggering the lets encrypt validation flow.
• Redirect all /.well-known/* from all RTD sites to a special validation server
• On the validation server, serve the correct value for the LE challenge for that domain
• Obtain the certificate, and store it somewhere.
• (Make sure to repeat every X days so certs don't expire!)
• When a request comes in with SNI=foo.com, look up that cert and server it if it's a RTD site

Hopefully that makes sense!

As discussed here, OpenGraph support would allow Discourse forums and other popular systems to display inline previews of links to readthedocs pages, especially useful for forums about programming languages and tools.

It just takes a few <meta> tags injected into the rendered HTML, so I don't think it should be too burdensome. Plus Jeff Atwood himself is pushing for it :)

Details

• Project URL: http://channels.readthedocs.io
• Build URL (if applicable): http://channels.readthedocs.io/en/stable/
• Read the Docs username (if applicable): andrewgodwin

Expected Result

I expect the "stable" build version to only follow tags, not new branches, as described in http://docs.readthedocs.io/en/latest/versions.html

Actual Result

The "stable" build says it is following origin/2.0, which looks like a reference to a remote Git branch rather than the project's most recent tag, 1.1.8, which means the "stable" docs have ended up being the in-development version docs.

The "wipe" command does not appear to fix this, and I would like to avoid deleting and remaking the project just to see if it's a bad tag being stored somewhere.

• Project URL: https://qnet.readthedocs.io/en/latest/index.html#
• Build URL (if applicable): https://readthedocs.org/projects/qnet/builds/7008115/

I'm seeing a strange effect where the navigation sidebar is broken depending on the RTD configuration. The effect is that clicking on the "Read the Docs"/version at the very bottom of the sidebar does nothing (it should pop up the version selector). Also, on a small screen where the sidebar is hidden by default, clicking on the sidebar button (the symbol of three horizontal bars in the top left corner) has no effect (as opposed to showing the sidebar).

The broken behavior can be seen in the current build on the Project URL. When loading that page e.g. in Safari with an open Web Inspector, there is an error message that hints at the problem:

[Error] TypeError: undefined is not an object (evaluating 'SphinxRtdTheme.Navigation.enableSticky')
	(anonymous function) (index.html:365)
	f (jquery-2.0.3.min.js:3:2256)
	fireWith (jquery-2.0.3.min.js:3:3059)
	ready (jquery-2.0.3.min.js:1:12573)
	ge (jquery-2.0.3.min.js:1:10197)

The project uses a readthedocs.yml file and it's set up to use conda, and also to install the package for which the documentation is generated via pip (pip_install: true)

It turns out that this is the problem. If I switch from pip_install to setup_py_install (and adapt the environment.yml file accordingly to install all pip-prerequisites manually), it works fine.

The commit that makes everything work is https://github.com/mabuchilab/QNET/commit/cc55fe4d789c32ee25436af4749a2d7d9bc6c2ad

I've reverted that commit to have a "broken" build, which I'll leave up for a few days to give people a chance to look at it.

I wonder if the problem is that I have sphinx_rtd_theme listed in setup.py: maybe this overwrites the system-sphinx-rtd_theme during the pip_install of the package, which doesn't happen with setup_py_install?

Details

• Read the Docs project URL: n/a
• Build URL (if applicable): n/a
• Read the Docs username (if applicable): JoshIzaac

Expected Result

I am having trouble importing repositories from an organization (https://github.com/PennyLaneAI). I am listed as an owner of the organization, and 3rd party access has been provided to the organization:

Actual Result

Pressing the sync button on my ReadTheDocs 'Import a Project' page has no effect; the PennyLaneAI organization is not listed.

Details

• Read the Docs project URL: https://readthedocs.org/projects/raidzero-swerve-docs-zh-tw/
• Build URL (if applicable): https://readthedocs.org/projects/raidzero-swerve-docs-zh-tw/builds/19005126/
• Read the Docs username (if applicable): https://readthedocs.org/profiles/Yessir4253/

Expected Result

The text in the docs is supposed to be in Traditional Chinese.

Actual Result

The project is configured to be in Traditional Chinese (zh_TW) but many translations are in Simplified Chinese. For example, next page is this:

when it's actually supposed to be "下一頁"

Similarly, previous page is this: this is in simplified Chinese when it's actually supposed to be "下一頁" which is traditional chinese

Details

• Read the Docs project URL: https://readthedocs.org/projects/diofant/

Expected Result

When I click on the v:latest - the flyout menu does appear with links to versions, etc. Also, search should be working too.

Actual Result

No search results (shows "searching..." forever: try anything, e.g. "residue" to seach) and no flyout menu for the latest version. c.f.: https://diofant.readthedocs.io/en/stable/

I'm trying to install readthedocs on my local gitlab server and I'm following the guide in the link.

ReadThedocs Guide

Here's the python version and pip version I'm using;

Python3.9 Pip3.9

To run the virtual pip environment, I use the following command.

python3.9 -m venv tutorial-env # I completed the install process with this command.

source tutorial-env/bin/activate # with this command I access the virtual pip environment**

my steps in the virtual pip environment;

I run this from the readthedocs installation steps.

(tutorial-env) [redhat@gitlab tutorial-env]$ pip install -r requirements.txt And no error occurred.

(tutorial-env) [redhat@gitlab tutorial-env]$ python3.9 /home/redhat/readthedocs.org/manage.py migrate

Gave the following error message

[debug ] Using slumber v2. [readthedocs.api.v2.client] api_host=http://127.0.0.1:8000 username=test

Traceback (most recent call last):

File "/home/redhat/readthedocs.org/manage.py", line 11, in

execute_from_command_line(sys.argv)

File "/home/redhat/tutorial-env/lib64/python3.9/site-packages/django/core/management/init.py", line 419, in execute_from_command_line

utility.execute()

File "/home/redhat/tutorial-env/lib64/python3.9/site-packages/django/core/management/init.py", line 395, in execute

django.setup()

File "/home/redhat/tutorial-env/lib64/python3.9/site-packages/django/init.py", line 24, in setup

apps.populate(settings.INSTALLED_APPS)

File "/home/redhat/tutorial-env/lib64/python3.9/site-packages/django/apps/registry.py", line 122, in populate

app_config.ready()

File "/home/redhat/readthedocs.org/readthedocs/core/apps.py", line 15, in ready

import readthedocs.core.signals  # noqa

File "/home/redhat/readthedocs.org/readthedocs/core/signals.py", line 17, in

from readthedocs.core.unresolver import unresolve

File "/home/redhat/readthedocs.org/readthedocs/core/unresolver.py", line 16, in

@dataclass(slots=True)

TypeError: dataclass() got an unexpected keyword argument 'slots'

because of this problem I can not install local readthedocs. What should I pay attention to to overcome this problem? I need your support on this issue.

closes https://github.com/readthedocs/readthedocs.org/issues/9718

Output:

Note: The Remote Organisation API (/api/v2/remote/org/) does not return the Provider Name with correct case but the Account API Does. We can either change the Remote Organisation API to return the display name or update readthedocs/projects/static-src/projects/js/import.js to change the name depending on the value (e.g.: github -> GitHub)

:books: Documentation previews :books:

• User's documentation (docs): https://docs--9846.org.readthedocs.build/en/9846/

• Developer's documentation (dev): https://dev--9846.org.readthedocs.build/en/9846/

I was getting errors from Tox about base_path replacement not existing. This was using inv docker.test, so I don't think this is limited to me.

inv docker.test -a "-e py310 -- -k ProjectsEndpointTests"             
Traceback (most recent call last):
  File "/usr/local/bin/tox", line 8, in <module>
    sys.exit(cmdline())
  File "/usr/local/lib/python3.10/dist-packages/tox/session/__init__.py", line 44, in cmdline
    main(args)
  File "/usr/local/lib/python3.10/dist-packages/tox/session/__init__.py", line 65, in main
    config = load_config(args)
  File "/usr/local/lib/python3.10/dist-packages/tox/session/__init__.py", line 81, in load_config
    config = parseconfig(args)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 299, in parseconfig
    ParseIni(config, config_file, content)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 1323, in __init__
    raise tox.exception.ConfigError(
tox.exception.ConfigError: ConfigError: migrations failed with ConfigError: substitution key 'base_python' not found at Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 1299, in run
    results[name] = cur_self.make_envconfig(name, section, subs, config)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 1475, in make_envconfig
    res = meth(env_attr.name, env_attr.default, replace=replace)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 1772, in getargvlist
    return _ArgvlistReader.getargvlist(self, s, replace=replace, name=name)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 2036, in getargvlist
    commands.append(cls.processcommand(reader, current_command, replace, name=name))
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 2062, in processcommand
    new_word = reader._replace(word, name=name)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 1857, in _replace
    replaced = Replacer(self, crossonly=crossonly).do_replace(value)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 1893, in do_replace
    expanded = substitute_once(value)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 1891, in substitute_once
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 1964, in _replace_match
    return self._replace_substitution(sub_value)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 1999, in _replace_substitution
    val = self._substitute_from_other_section(sub_key)
  File "/usr/local/lib/python3.10/dist-packages/tox/config/__init__.py", line 1994, in _substitute_from_other_section
    raise tox.exception.ConfigError("substitution key {!r} not found".format(key))
tox.exception.ConfigError: ConfigError: substitution key 'base_python' not found