Sphinx-performance - CLI tool to measure the build time of different, free configurable Sphinx-Projects

When installing globally on Windows without environment

pip install sphinx-performance

-> on running sphinx-performance there is the following error ... File "c:\users\kuempel\appdata\local\programs\python\python39\lib\site-packages\sphinx_performance\projectenv.py", line 250, in install_dependencies subprocess.call(dep_command, stdout=subprocess.DEVNULL)

It seem like the correct paths are not calculated correctly.

Something like:

sphinx-performance --project projects/p1 --project projects/p2 --pages 100 --dummies 1 --keep

is currently not supported. Would be nice to have for great flexibility. This could also make the very specific --theme option redundant.

Running this

sphinx-performance --parallel 4 --pages 20 --dummies 200 --snakeviz --profile foo

leads to

Starting snakeviz servers

Traceback (most recent call last):
  File "/home/marco/ub/sphinx-performance/.venv/bin/sphinx-performance", line 2, in <module>
    from sphinx_performance.main import cli
  File "/home/marco/ub/sphinx-performance/sphinx_performance/main.py", line 143, in <module>
    cli()
  File "/home/marco/ub/sphinx-performance/.venv/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/marco/ub/sphinx-performance/.venv/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/marco/ub/sphinx-performance/.venv/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/marco/ub/sphinx-performance/.venv/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/marco/ub/sphinx-performance/.venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/marco/ub/sphinx-performance/sphinx_performance/main.py", line 133, in cli
    proc = subprocess.Popen(["snakeviz", f"profile/{p}.prof"])
  File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'snakeviz'

It would be great to have predefined reference projects, which can be easily selected and always provide the same configuration. This would be helpful to compare measurements quite easily and also to be able to have a name of a used config in discussions.

I think the "project" itself shall be able to provide such reference configs. They behave like the current default values, but can be selected by name.

Example: sphinx-performance --project needs --ref huge And huge is configured to have 100 needs and 5 needtables per page, with 20 pages, 10 folders and a depth of 3. So the reference config must be complete and can't be overwritten by the user.

It shall be also possible to use multiple reference configs in one run: sphinx-performance --project needs --ref huge --ref small

A command to show the config behind a reference would be also great: sphinx-performance --reflist to show all available reference configs.

Currently Sphinx-Performance can create a profile file for each built project and open Snakeviz to open it.

This is already nice, but some features are missing and it also has some drawbacks:

Drawback: sphinx-build gets executed via subprocess as a mostly isolated process.

This makes it hard to start a profiling for a complete build. Currently, we use code-internal flags to mark the needed section for profiling

New Feature: Sphinx specific analysis

We don't get a good insight into what is really happening inside Sphinx builds. It would be good to get extension-specific numbers for runtime and maybe memory consumption. Also, numbers for events should be somehow summarized.

New Feature: Memory consumption

We should use memray to get a memory-analysis of a complete sphinx-build.

Technical realization

We should not implement the deeper performance analysis into the general, parallel build execution. Mostly because it may add overhead and the current way of calling sphinx-build via subprocess it not often supported by analysis-tool, as they don't get access to it. Also, upcoming features may be analysis specific and should not get in conflict with the sphinx-performance call.

So, let's have a new command sphinx-analysis, which can create a single project by the same parameters as sphinx-performance does. It shall have options to activate the needed analysis methods. Multiple analysis methods can be executed on the same call or on different calls.

Analysis methods:

• memory, uses memray and creates a final HTML as result
• runtime, uses the Python profiles and created a cProfile file
• internals, logs sphinx-external numbers to stdout and a json-file

Other options

• snakeviz to open snakeviz for the created cProfile file from runtime

Ideas

• Maybe a sphinx-specific analysis of the cProfile-file makes sense. But this would be something for a new issue.

This looks like an awesome project! I noticed a few phrases that didn't sound quite right in English so I thought I'd submit a quick PR - I hope this is okay?

It would be nice to have an option to control where the output project gets created, not always in /tmp. For example, within a docker container, this would allow to let the project be generated in a path which is mounted and available to the native host system (to open in a browser and see results etc.)

Sphinx-Performance should also provide a Sphinx extension and maybe builder, so that we can get insights of the Sphinx-Build.

The first feature would be to collect the stats of the Garbage collector. This code can be used:

import gc
stats = gc.get_stats()

print('** Python Garbage Collector stats **')
for index, col in enumerate(stats):
    for key, value in col.items():
        print(f'{index}.{key}: {value}')
    print('')

The output would like like this at the end of the Sphinx build:

** Python Garbage Collector stats **
0.collections: 492
0.collected: 2738
0.uncollectable: 0

1.collections: 44
1.collected: 769
1.uncollectable: 0

2.collections: 4
2.collected: 32384
2.uncollectable: 0

Currently, when specifiying sphinx-performance --parallel auto (equivalent sphinx-build -j auto) we get the error: Error: Invalid value for '--parallel': 'auto' is not a valid integer.

Using sphinx-performance with option parallel seems to not report the read-time and write-time correctly. I think i remember that this was documented somewhere, but nonetheless, i'll open an issue also here.

When installed using pip install sphinx-performance, i get the follownig error on calling sphinx-performance on my build machine:

File "/home/docs-as-code-user/.local/lib/python3.8/site-packages/sphinx_performance/projectenv.py", line 250, in install_dependencies
    subprocess.call(dep_command, stdout=subprocess.DEVNULL)
  File "/usr/lib/python3.8/subprocess.py", line 3[40](https://github.boschdevcloud.com/docs-as-code/sphinx-performance-tests/runs/974184?check_suite_focus=true#step:6:40), in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/pip'

If i check for python and pip installation paths on my (provided) build machine i get:

which python -> /usr/bin/python
which pip       -> /usr/local/bin/pip

Great to see --temp parameter support. Would be nice if it creates the temp folder incase missing.

line 27, in __init__
    raise ProjectException(f'Given temp folder does not exist: {temp}')
sphinx_performance.projectenv.ProjectException: Given temp folder does not exist: ./sphinx-perf-temp