Reproducible nvim completion framework benchmarks.

Overview

Nvim.Bench

Reproducible nvim completion framework benchmarks.

Runs inside Docker. Fair and balanced


Methodology

Note: for all "randomness", they are generated from the same seed for each run, and therefore "fair".

Input

tmux is used to send keys to simulate ideal human typing.

The words typed are naive tokens from parsing current document into (alphanum + "_") delimited by whitespaces and symbols.

This tokenization should work fairly well for c family of languages, which are the industry standard.

A uniform distribution of whitespaces is also generated from the same buffer.

Measurement

n keystrokes of --samples is performed.

Speed

Using --avg-word-len, --wpm and --variance, a Normal Distribution is constructed of the desired delay between keystrokes.

Data

See ./fs/data/

Modularity

Some frameworks will have by default, very little sources enabled, if any.

Other ones will come with more out of the box.

For a fair comparison: All frameworks tested will have to following enabled, on top of whatever else they come enabled by default:

  • buffer

  • lsp

  • path

The reasoning is that: 1) Almost all authors will have written these sources firsthand, and 2) they seem to be the most useful sources.

No default sources will be disabled, because users don't tend to do that.


Cool, pictures

The plots are kernel density estimations, have no idea why they fitted more than 1 curve for some plots.

I usually use R, not used to python ploting. Anyways, they are an estimate of the true probability density function.

Q0, 50, 95, 100?

Mean min, median, 1 in 20, max, respectively.

Without assuming any statistical distribution:

Q50 is a more robust measure than avg, and Q95 is a decent measure of a common bad value.


Analysis

Please keep in mind that this is purely a synthetic benchmark, which definitely is one of those need context to interpret type of things.

There is no good way to measure real speed across frameworks, raw numbers here come with big caveats.

Study design limitations

Streaming completion

Streaming completion is very good for time to first result (TTFR), but it presents us with an issue:

While the fast sources will return right away, the slower ones might never make it before the next keystroke.

This has the funny effect of removing the influence of slower sources entirely, which is disastrous for study integrity.

The mitigation is actually to set typing speed unrealistically slow, enough so that we have confidence that the LSP servers can catch up.

This is obviously not ideal.

Fast on paper != fast IRL

The most responsive frameworks are not necessarily the fastest ones, because humans still have to choose the results.

For example the streaming completion approach actually has severe trade offs infavor of faster TTFR:

Ranking

Having suboptimal ranking is BAD, it pushes work from fast machines onto slow humans.

The streaming approach has to be additive, because its too disruptive to shift existing menu items around.

Therefore it is limited to sorting only within stream batches, and to make things worse, slower batches typically contain higher quality results.

That means better results will often end up at the bottom, necessitating more work for humans.

Limiting

This is a direct consequence of limited ranking optimizations.

Because the framework have no idea how much each source will send, it has the dilemma of either sending too many results or too little.

Sending too many results in early batches from likely inferior sources will waste the users time, and sending too little will obscure potentially useful completions.

Clarity on when / if results will come in

This is a HCI thing:

Having higher quality results come in slower is likely to inadvertently train users to wait for them. This is evidently bad for input speed.

Conclusion

There is never going to be a closed form solution to "what is the fastest framework", because of the trade offs detailed above.

A toy example of a degenerate framework that returns a single fixed 👌 emoji will probably beat anything out there in terms of raw speed, but it is utterly useless.

Before you reach your own conclusion, the results of this repo must be considered alongside inextricably human measure.

You might also like...
EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way.

EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way.

Results of Robot Framework 5.0 survey

Robot Framework 5.0 survey results We had a survey asking what features Robot Framework community members would like to see in the forthcoming Robot F

The dynamic code loading framework used in LocalStack

localstack-plugin-loader localstack-plugin-loader is the dynamic code loading framework used in LocalStack. Install pip install localstack-plugin-load

A new mini-batch framework for optimal transport in deep generative models, deep domain adaptation, approximate Bayesian computation, color transfer, and gradient flow.

BoMb-OT Python3 implementation of the papers On Transportation of Mini-batches: A Hierarchical Approach and Improving Mini-batch Optimal Transport via

Framework To Ease Operating with Quantum Computers

QType Framework To Ease Operating with Quantum Computers Concept # define an array of 15 cubits:

An execution framework for systematic strategies

WAGMI is an execution framework for systematic strategies. It is very much a work in progress, please don't expect it to work! Architecture The Django

Bionic is Python Framework for crafting beautiful, fast user experiences for web and is free and open source.
Bionic is Python Framework for crafting beautiful, fast user experiences for web and is free and open source.

Bionic is Python Framework for crafting beautiful, fast user experiences for web and is free and open source. Getting Started This is an example of ho

Flames Calculater App used to calculate flames status between two names created using python's Flask web framework.
Flames Calculater App used to calculate flames status between two names created using python's Flask web framework.

Flames Finder Web App Flames Calculater App used to calculate flames status between two names created using python's Flask web framework. First, App g

redun aims to be a more expressive and efficient workflow framework

redun yet another redundant workflow engine redun aims to be a more expressive and efficient workflow framework, built on top of the popular Python pr

Comments
  • run.sh error

    run.sh error

    returns this:

    ❯ ./run.sh
    [+] Building 4.4s (7/15)                                                                                                                                                                                                                                                        
     => [internal] load build definition from Dockerfile                                                                                                                                                                                                                       0.1s
     => => transferring dockerfile: 925B                                                                                                                                                                                                                                       0.0s
     => [internal] load .dockerignore                                                                                                                                                                                                                                          0.1s
     => => transferring context: 53B                                                                                                                                                                                                                                           0.0s
     => [internal] load metadata for docker.io/library/ubuntu:focal                                                                                                                                                                                                            1.6s
     => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                                                                                                              0.0s
     => [ 1/11] FROM docker.io/library/ubuntu:focal@sha256:9d6a8699fb5c9c39cf08a0871bd6219f0400981c570894cd8cbea30d3424a31f                                                                                                                                                    2.6s
     => => resolve docker.io/library/ubuntu:focal@sha256:9d6a8699fb5c9c39cf08a0871bd6219f0400981c570894cd8cbea30d3424a31f                                                                                                                                                      0.0s
     => => sha256:35807b77a593c1147d13dc926a91dcc3015616ff7307cc30442c5a8e07546283 28.57MB / 28.57MB                                                                                                                                                                           2.2s
     => => sha256:9d6a8699fb5c9c39cf08a0871bd6219f0400981c570894cd8cbea30d3424a31f 1.42kB / 1.42kB                                                                                                                                                                             0.0s
     => => sha256:10cbddb6cf8568f56584ccb6c866203e68ab8e621bb87038e254f6f27f955bbe 529B / 529B                                                                                                                                                                                 0.0s
     => => sha256:fb52e22af1b01869e23e75089c368a1130fa538946d0411d47f964f8b1076180 1.46kB / 1.46kB                                                                                                                                                                             0.0s
     => => extracting sha256:35807b77a593c1147d13dc926a91dcc3015616ff7307cc30442c5a8e07546283                                                                                                                                                                                  0.3s
     => [internal] load build context                                                                                                                                                                                                                                          0.0s
     => => transferring context: 312.63kB                                                                                                                                                                                                                                      0.0s
     => ERROR [ 2/11] RUN apt update &&     DEBIAN_FRONTEND=noninteractive apt install --yes --     software-properties-common     apt-transport-https     curl &&     curl --location -- 'https://deb.nodesource.com/gpgkey/nodesource.gpg.key' | apt-key add - &&     add-a  0.1s
    ------
     > [ 2/11] RUN apt update &&     DEBIAN_FRONTEND=noninteractive apt install --yes --     software-properties-common     apt-transport-https     curl &&     curl --location -- 'https://deb.nodesource.com/gpgkey/nodesource.gpg.key' | apt-key add - &&     add-apt-repository ppa:neovim-ppa/unstable:
    ------
    error: failed to solve: executor failed running [/bin/sh -c apt update &&     DEBIAN_FRONTEND=noninteractive apt install --yes --     software-properties-common     apt-transport-https     curl &&     curl --location -- 'https://deb.nodesource.com/gpgkey/nodesource.gpg.key' | apt-key add - &&     add-apt-repository ppa:neovim-ppa/unstable]: failed to create endpoint kwer7ncqkqkisr6mndmjojpq1 on network bridge: failed to add the host (veth9081174) <=> sandbox (veth1f1fa44) pair interfaces: operation not supported
    
    opened by marcelarie 3
  • Update benchmark?

    Update benchmark?

    Hey ms-jpq!

    Thanks for the repo, I DM'd you about this on reddit, but can you update your benchmark with results from neovim at least as new as this commit https://github.com/neovim/neovim/commit/912a6e5a9c58fce74134f9f8c2801373928e8289?

    Should improve the numbers for coq_nvim and nvim-cmp :)

    cc @hrsh7th

    opened by mjlbach 5
Owner
i love my dog
dogs are love dogs are life
i love my dog
The RAP community of practice includes all analysts and data scientists who are interested in adopting the working practices included in reproducible analytical pipelines (RAP) at NHS Digital.

The RAP community of practice includes all analysts and data scientists who are interested in adopting the working practices included in reproducible analytical pipelines (RAP) at NHS Digital.

NHS Digital 50 Dec 22, 2022
Sublime Text 2/3 style auto completion for ST4

Hippie Autocompletion Sublime Text 2/3 style auto completion for ST4: cycle through words, do not show popup. Simply hit Tab to insert completion, hit

Alexander Schepanovski 20 May 19, 2022
BlackMamba is a multi client C2/post exploitation framework

BlackMamba is a multi client C2/post exploitation framework with some spyware features. Powered by Python 3.8.6 and QT Framework.

Gustavo 873 Dec 29, 2022
Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Viewflow Viewflow is a framework built on the top of Airflow that enables data scientists to create materialized views. It allows data scientists to f

DataCamp 114 Oct 12, 2022
Package pyVHR is a comprehensive framework for studying methods of pulse rate estimation relying on remote photoplethysmography (rPPG)

Package pyVHR (short for Python framework for Virtual Heart Rate) is a comprehensive framework for studying methods of pulse rate estimation relying on remote photoplethysmography (rPPG)

PHUSE Lab 261 Jan 3, 2023
Simple dependency injection framework for Python

A simple, strictly typed dependency injection library.

BentoML 14 Jun 29, 2022
Backtest framework based on DAGs

MultitaskQueue It's a simple framework based on three composed concepts: Task: A task is the smaller unit of execution or simple a node in the DAG, ev

null 4 Dec 9, 2021
Framework for creating efficient data processing pipelines

Aqueduct Framework for creating efficient data processing pipelines. Contact Feel free to ask questions in telegram t.me/avito-ml Key Features Increas

avito.tech 137 Dec 29, 2022
A topology optimization framework written in Taichi programming language, which is embedded in Python.

Taichi TopOpt (Under Active Development) Intro A topology optimization framework written in Taichi programming language, which is embedded in Python.

Li Zhehao 41 Nov 17, 2022
Sabe is a python framework written for easy web server setup.

Sabe is a python framework written for easy web server setup. Sabe, kolay web sunucusu kurulumu için yazılmış bir python çerçevesidir. Öğrenmesi kola

null 2 Jan 1, 2022