Thank you for making this amazing problem statement public, along with a very comprehensive dataset!

Can a relatively smaller size dataset ( a subset ) of it be made available for independent developers/researchers who might try running this on their personal machines ?

This will open up the problem for a larger audience and may bring in some innovative solutions!

Hi!

I've made a custom model and now I'm trying to submit it to the leaderboard. Here is the run: https://app.wandb.ai/github/codesearchnet/runs/lqqo1i4m

Above it says 'Awaiting review from codesearchnet benchmark' and that's probably the reason why I get an error when I try to 'Publish to GitHub'. Am I doing something wrong, or do I just have to wait?

Thanks.

Did I get it right, that for a submission on the challenge, I have to run those 99 queries against the whole code corpus, and not just the test set? Thanks in advance :)

The default ordering on the leaderboard is "Mean NDCG" over all results that are not None.

This means it is very easy to appear #1 by overfitting on a single language.

What about computing "Mean NDCG" over all 6 languages and replacing None by 0?

e.g. some short methods may contain a description in the return tag, but not a description of the method itself. (to avoid redundancy).

Doing this would extract more methods, but they may be of lower quality if incorrectly parsed or automatically generated. I'd expect a description such as @return bool STARTDESCRIPTION true if this is a float to be extracted (I'm not familiar with how the data representation works)

• Some libraries/applications are strict about always having a summary in their coding standards, others aren't.
• e.g. I've seen @return the description without a type for php
• Could try to annotate the fact that the fallback was used and this is a non-official summary

    /**
     * @return bool true if this is a float
     */
    public function isFloat()

It would be nice to account for code such as @return HasTemplate<string, stdClass>, etc. Making sure that <([{ in the first token are matched up may be useful as a basic approach (and give up if that fails). (There's no official standard and different static analyzers have their own extensions to the type syntax)

An example implementation for PHP is https://github.com/phan/phan/blob/2.2.12/src/Phan/Language/Element/MarkupDescription.php - examples of what it generates as descriptions are https://github.com/phan/phan/blob/2.2.12/tests/Phan/Language/Element/MarkupDescriptionTest.php#L132-L155

In CodeSearchNet/function_parser/function_parser/demo.ipynb . I kept everything same till thrid cell and then I did this processor.process_single_file(py_file_path) . Here py_file_path contains the complete path of .py file that I want to process.

After executing the above line I got the following error : unhashable type: 'tree_sitter.Node' in file function_parser/function_parser/parsers/language_parser.py.

Am I missing something?

I'm currently trying to submit a custom model. After entering the run, evaluating the ndcg score, and writing a brief note on how we approached the results, I get the following error

Now, when I click refresh, I will be redirected to the submission page again. When trying to re-submit, I get the following error Invalid CSV format. Please upload a well-formatted CSV file..

Hello,

first thanks for the challenge, the code and the dataset! Really cool stuff that you're doing and I want to work on this task. :)

I've read the Contribution Guidelines and know that you will not change any of the preprocessing code, but nevertheless I want to discuss the preprocessing of the docstrings here in case someone wants to produce a similar dataset (or maybe v3 ;) ) .

Current preprocessing of docstrings

I read your code and it seems that this is the way you preprocess the docstrings:

1. You extract the documentation from the method and strip c-style delimiters https://github.com/github/CodeSearchNet/blob/9356b3181eaa9d4a38df4d309018158fd23448cb/function_parser/function_parser/parsers/commentutils.py#L1
2. Then you extract the relevant part of the docstring (which acts as a summary) using the following heuristics: 2.1 If \n\n is found you take the part before 2.2 otherwise take the part before the first @ 2.3. otherwise take the full docstring https://github.com/github/CodeSearchNet/blob/9356b3181eaa9d4a38df4d309018158fd23448cb/function_parser/function_parser/parsers/commentutils.py#L18-L24
3. Finally you tokenize the extracted summary using the following regex: https://github.com/github/CodeSearchNet/blob/9356b3181eaa9d4a38df4d309018158fd23448cb/function_parser/function_parser/parsers/language_parser.py#L5

Problems with this pipeline

This way of preprocessing produces a couple of results that are probably not wanted and could be improved.

Extracting the summary

Compare the first 12 lines of the tokenized docstrings of the java train set to the raw ones

Bind indexed elements to the supplied collection .
Set {
Add {
Set servlet names that the filter will be registered against . This will replace any previously specified servlet names .
Add servlet names for the filter .
Set the URL patterns that the filter will be registered against . This will replace any previously specified URL patterns .
Add URL patterns as defined in the Servlet specification that the filter will be registered against .
Convenience method to {
Configure registration settings . Subclasses can override this method to perform additional configuration if required .
Create a nested {
Create a nested {
Create a nested {

As you can see 6/12 are basically not usable.

**Bind indexed elements to the supplied collection.** @param name the name of the property to bind @param target the target bindable @param elementBinder the binder to use for elements @param aggregateType the aggregate type, may be a collection or an array @param elementType the element type @param result the destination for results
**Set** {@link **ServletRegistrationBean**}**s that the filter will be registered against.** @param servletRegistrationBeans the Servlet registration beans
**Add** {@link **ServletRegistrationBean**}**s for the filter.** @param servletRegistrationBeans the servlet registration beans to add @see #setServletRegistrationBeans
**Set servlet names that the filter will be registered against. This will replace any previously specified servlet names.** @param servletNames the servlet names @see #setServletRegistrationBeans @see #setUrlPatterns
**Add servlet names for the filter.** @param servletNames the servlet names to add
**Set the URL patterns that the filter will be registered against. This will replace any previously specified URL patterns.** @param urlPatterns the URL patterns @see #setServletRegistrationBeans @see #setServletNames
**Add URL patterns, as defined in the Servlet specification, that the filter will be registered against.** @param urlPatterns the URL patterns
**Convenience method to** {@link **#setDispatcherTypes(EnumSet) set dispatcher types**} **using the specified elements.** @param first the first dispatcher type @param rest additional dispatcher types
**Configure registration settings. Subclasses can override this method to perform additional configuration if required.** @param registration the registration
**Create a nested** {@link **DependencyCustomizer**} **that only applies if any of the specified class names are not on the class path.** @param classNames the class names to test @return a nested {@link DependencyCustomizer}
**Create a nested** {@link **DependencyCustomizer**} **that only applies if all of the specified class names are not on the class path.** @param classNames the class names to test @return a nested {@link DependencyCustomizer}
**Create a nested** {@link **DependencyCustomizer**} **that only applies if the specified paths are on the class path.** @param paths the paths to test @return a nested {@link DependencyCustomizer}

However, the relevant information is in the raw docstrings (I added the ** to highlight relevant passages). Simply using the part before the first @ produces pretty bad results (at least in java) as its common practice to highlight code blocks or links with javadoc-tags. Possible solution: Stripping everything before the first param (or maybe @param) and afterwards removing javadoc-tags (maybe keeping the tokens inside).

Cleaning

The preprocessing does not include any cleaning. This manifests in docstrings that contain html-tags (which are commonly found in javadoc), as well as URLs (which afterwards get pretty verbose tokenized). See these two samples from java

Determine if a uri is in origin - form according to <a href = https : // tools . ietf . org / html / rfc7230#section - 5 . 3 > rfc7230 5 . 3< / a > .
Determine if a uri is in asterisk - form according to <a href = https : // tools . ietf . org / html / rfc7230#section - 5 . 3 > rfc7230 5 . 3< / a > .

Some stats

>>> wc -l java.train.comment 
454436 java.train.comment
>>> grep -E '<p >|<p >' java.train.comment | wc -l
42750

At least 10% of the tokenized java docstrings still contain html tags.

>>> grep '{ @' java.train.comment | wc -l
44500

Another 10% still contain javadoc.

>>> grep "{ @inheritDoc }" java.train.comment | wc -l
1685

2k consist only of a single javadoc-tag indication that the doc was inherited.

Many of the golang docstrings contain URLs, which are not very useful in the tokenized version the regex produces (see above in java).

>>> wc -l go.train.comment 
317822 go.train.comment
>>> grep -E 'http :|https :' go.train.comment | wc -l
19753

~6% contain URLs (starting with http :)

>>> grep -E "autogenerated|auto generated" go.train.comment | wc -l
4850

Around 5k auto generated methods

>>> grep "/ *" go.train.comment | wc -l
33620

10% still contain c-style comment delimiters.

Tokenization

Any specific reason you keep punctuation symbols like ., ,, -, /, :, <, >, *, =, @, (, ) as tokens? Is it to keep code in the docstrings?

Summary

I really think better cleaning and a language-dependent preprocessing would produce higher quality docstrings At least for java a removal of javadoc and html could be beneficial. As well as using everything before the first param as a summary (maybe in combination with the first paragraph \n\n heuristic).

This pull request represents a submission to the codesearchnet benchmark.

It is the main communication channel for you and the reviewers.

Wandb run results for review

• encoder classes are model class fields instead of constructor arguments – the constructor arguments are now sound wrt. subtyping;
• allow different encoder types for different code languages (not tested in practice);
• data, metadata loaders and losses are defined in separate modules.

Since the relevance_annotations.csv is not available, a script to upload model_predictions.csv and download a model statistics file (NDCG, MRR, for example) would be great.

https://s3.amazonaws.com/code-search-net/CodeSearchNet/v2/%7Bpython,java,go,php,ruby,javascript%7D.zip 👆This URL can not be opened and says AccessDenied. Could you please tell me where should I download the dataset now?

how do you divide the training/testing/validation set, I don't seem to find the basis for the division, only the ratio. If we want to perform cross-validation, can we combine the original training set, test set, and validation set and re-divided.

Hello, I tried cloning this repo. Got the following error: error: invalid path 'benchmarks/02-Feb-20-16:33_github_g2od7jac.json' fatal: unable to checkout working tree warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry with 'git restore --source=HEAD :/'

I tried git restore, but that resulted in a staged delete of all the code.

I was trying to access codesearch net dataset for my work - specifically java based data via the given link in codesearchnet repository : https://s3.amazonaws.com/code-search-net/CodeSearchNet/v2/java.zip

While accessing the data parameters and initial data exploration, I observed that a lot of JAVA code snippets, which were taken from various sources of public github repositories, a lot of those repositories have been either turned private or their respective github repository links have expired.

Due to the reason mentioned above, I am not able to access their original github repository.

Can you kindly take a look and let me know if there is any way possible to extract the entire github repository from which the java code snippets and their respective documentation has been obtained?

Actually my work requires cloning the entire project repository from which the codesearchnet java dataset has been extracted.

In the Dockerfile of function-parser, the script downloads tree-sitter==0.0.5, however, the language parser repositories are updated, which leads to conflicts, please add the git checkout command for each language parser.