Nvim.Bench
Reproducible nvim completion framework benchmarks.
Runs inside Docker. Fair and balanced
Methodology
Note: for all "randomness", they are generated from the same seed for each run, and therefore "fair".
Input
tmux is used to send keys to simulate ideal human typing.
The words typed are naive tokens from parsing current document into (alphanum + "_") delimited by whitespaces and symbols.
This tokenization should work fairly well for c family of languages, which are the industry standard.
A uniform distribution of whitespaces is also generated from the same buffer.
Measurement
n keystrokes of --samples is performed.
Speed
Using --avg-word-len, --wpm and --variance, a Normal Distribution is constructed of the desired delay between keystrokes.
Data
See ./fs/data/
Modularity
Some frameworks will have by default, very little sources enabled, if any.
Other ones will come with more out of the box.
For a fair comparison: All frameworks tested will have to following enabled, on top of whatever else they come enabled by default:
-
buffer
-
lsp
-
path
The reasoning is that: 1) Almost all authors will have written these sources firsthand, and 2) they seem to be the most useful sources.
No default sources will be disabled, because users don't tend to do that.
Cool, pictures
The plots are kernel density estimations, have no idea why they fitted more than 1 curve for some plots.
I usually use R, not used to python ploting. Anyways, they are an estimate of the true probability density function.
Q0, 50, 95, 100?
Mean min, median, 1 in 20, max, respectively.
Without assuming any statistical distribution:
Q50 is a more robust measure than avg, and Q95 is a decent measure of a common bad value.
Analysis
Please keep in mind that this is purely a synthetic benchmark, which definitely is one of those need context to interpret type of things.
There is no good way to measure real speed across frameworks, raw numbers here come with big caveats.
Study design limitations
Streaming completion
Streaming completion is very good for time to first result (TTFR), but it presents us with an issue:
While the fast sources will return right away, the slower ones might never make it before the next keystroke.
This has the funny effect of removing the influence of slower sources entirely, which is disastrous for study integrity.
The mitigation is actually to set typing speed unrealistically slow, enough so that we have confidence that the LSP servers can catch up.
This is obviously not ideal.
Fast on paper != fast IRL
The most responsive frameworks are not necessarily the fastest ones, because humans still have to choose the results.
For example the streaming completion approach actually has severe trade offs infavor of faster TTFR:
Ranking
Having suboptimal ranking is BAD, it pushes work from fast machines onto slow humans.
The streaming approach has to be additive, because its too disruptive to shift existing menu items around.
Therefore it is limited to sorting only within stream batches, and to make things worse, slower batches typically contain higher quality results.
That means better results will often end up at the bottom, necessitating more work for humans.
Limiting
This is a direct consequence of limited ranking optimizations.
Because the framework have no idea how much each source will send, it has the dilemma of either sending too many results or too little.
Sending too many results in early batches from likely inferior sources will waste the users time, and sending too little will obscure potentially useful completions.
Clarity on when / if results will come in
This is a HCI thing:
Having higher quality results come in slower is likely to inadvertently train users to wait for them. This is evidently bad for input speed.
Conclusion
There is never going to be a closed form solution to "what is the fastest framework", because of the trade offs detailed above.
A toy example of a degenerate framework that returns a single fixed 👌 emoji will probably beat anything out there in terms of raw speed, but it is utterly useless.
Before you reach your own conclusion, the results of this repo must be considered alongside inextricably human measure.
87 Dec 27, 2022
2 Oct 16, 2021
5 Oct 9, 2022
18 Nov 14, 2022
2 Jun 6, 2022
10 Mar 28, 2022
14 Apr 10, 2022
4 Jan 2, 2022
372 Jan 4, 2023
50 Dec 22, 2022
20 May 19, 2022
873 Dec 29, 2022
114 Oct 12, 2022
261 Jan 3, 2023
14 Jun 29, 2022
4 Dec 9, 2021
137 Dec 29, 2022
41 Nov 17, 2022
2 Jan 1, 2022