GitPython is a python library used to interact with Git repositories.

Hello,

I was looking for any type hints for GitPython, but I found nothing neither in the code nor in independent stub repositories.

Is there any plan to introduce type hinting?

1. Use git to count the objects in a repo:

$ git rev-list --objects --all | wc -l
6990030
$ git rev-list --objects --all | sort | uniq | wc -l
6990030

2. Parse the output from git rev-list --objects --all, fetch each object with name_to_object, and count each type:

Commits: 909667, Tags: 2469, Trees: 4178263, Blobs: 1899631

3. Query what is ostensibly the same information using git-python:

import argparse
import pathlib

import git

def log(testname, a, b):
    print(testname, ':', a, b)

def main():

    parser = argparse.ArgumentParser(description='Git x ref.')
    parser.add_argument('repository', metavar='repository', type=pathlib.Path,
                        help='Path to Git repository.')

    args = parser.parse_args()

    repos = [
        git.Repo(str(args.repository), odbt=git.GitCmdObjectDB),
        git.Repo(str(args.repository), odbt=git.GitDB)
    ]

    log('size()', *[r.odb.size() for r in repos])
    log('len(sha_iter())', *[sum(1 for x in r.odb.sha_iter()) for r in repos])
    log('len(iter_trees())', *[sum(1 for x in r.iter_trees()) for r in repos])


if __name__ == '__main__':
    main()

Result:

size() : 3839 8268978
len(sha_iter()) : 3839 8268978
len(iter_trees()) : 568851 568851

So:

Git thinks there are 6,990,030 objects in the database. GitDB thinks there are 8,268,978. GitCmdObjectDB thinks there are 3,839.

Git thinks there are 4,178,263 trees in the database. Both GitDB and GitCmdObjectDB think there are 568,851.

I've implemented the Progress class to implement a nice progress bar:

• https://github.com/guyzmo/git-repo/blob/master/git_repo/services/service.py#L46-56

I'm then using it, for example here:

• https://github.com/guyzmo/git-repo/blob/master/git_repo/services/service.py#L196-202

And then, when I clone a very large repository (like mozilla's gecko):

% git hub clone mozilla/gecko-dev
Pulling from github |████████████████████████████████|
Successfully cloned `https://github.com/guyzmo/git-repo` into `./git-repo`!

the progress bar is "progressing" (i.e. incrementing from 0 to 100%), but it shows up only at the very end of the fetching, after hanging for a long while. I started to debug it, and it looks like it's related to subprocess stuff.

Though, I might be implementing it wrong, which is why I'm asking here while I'm still debugging that issue.

This appeared in the CVE feed today: https://security.snyk.io/vuln/SNYK-PYTHON-GITPYTHON-3113858

Not sure if this was reported to you before or not, reporting it here just in case.

I'm working on a script using GitPython to loop over lots of Git repos, pull remote changes, etc...

At some point, I noticed that doing repo.is_dirty() was opening some files which never got closed until the process exits, which in turns causes the tool to crash with "too many open files" after iterating over enough repos:

import os

import git

# Put here whatever Git repos you might have in the current folder
# Listing more than one makes the problem more visible
repos = ["ipset", "libhtp"]

raw_input("Check open files with `lsof -p %s | grep %s`" % (os.getpid(),
                                                            os.getcwd()))

for name in repos:
    repo = git.Repo(name)
    repo.is_dirty()

    del repo

raw_input("Check open files again")                 # files are still open

I tried digging deeper down the GitPython stack, but couldn't find the actual cause.

In case that's helpful, below is the same as the previous snippet, but using directly the lowest-level gitdb objects I could find to open the files:

import os

import git
from gitdb.util import hex_to_bin

# Put here whatever Git repos you might have in the current folder
# Listing more than one makes the problem more visible
repos = ["ipset", "libhtp"]

raw_input("Check open files with `lsof -p %s | grep %s`" % (os.getpid(),
                                                            os.getcwd()))

raw_input("Check open files again")                 # files are still open

for name in repos:
    repo = git.Repo(name)
    sha = hex_to_bin("71acab3ca115b9ec200e440188181f6878e26f08")

    for db in repo.odb._dbs:
        try:
            for item in db._entities:               # db._entities opens *.pack files
                item[2](sha)                        # opens *.idx files

        except AttributeError as e:
            # Not all DBs have this attribute
            pass

    del repo

raw_input("Check open files again")                 # files are still open

But that's just where the files are open (well, in fact they are open even lower down but I didn't manage to go deeper), not where they are supposed to be closed.

• Use Appveyor integration servers;
• Copy part of the travis logic, to speed-up test-time (no flake8, site & coverage), the only extra is wheel-packaging;
• Limit combination of Anaconda, CPython and MINGW/Cygwin in only 6 variations.
• Replaced poll/select buffer-assembly code with pump-threads (Win TCs faling 90-->20!) and rework unicode handling in many places.
• Popen() procs it with subprocess.CREATE_NEW_PROCESS_GROUP so that they can be killed.
• Invoke MINGW git-daemon (instead of git daemon); otherwise it detaches (unix-ism here) from parent git cmd, and then CANNOT DIE!
• Stop reading directly Popen().proc.stdout/stderr from the launching-thread, it freezes due to GIL/buffering.
  • Unslurp diff consumption code (but not raw-patch) to handle it line-by line to support arbitrary big output.
• Ensure read-only files do delete on Windows. +Rework GitCommandError.str() and add TCs on unicode handling.
• Fixed TCs on config-reader/writer:
  • Modify lock/read-config-file code to ensure files closed.
  • Use with GitConfigarser() more systematically in TCs.
  • Clean up any locks left hanging from prev TCs.
  • Util: mark lock-files as SHORT_LIVED; save some SSDs...
• Fixed TestRepo.test_submodule_update():
  • submod: del .git file prior overwrite; Windows denied otherwise!
• FIX TestRepo.test_untracked_files():
  • In the git add <file> case, PY2 fails when args are unicode. Must encode them with locale.getpreferredencoding() AND use SHELL.
• cmd: add shell into execute() kwds, for overriding USE_SHELL per command.
• repo: replace blocky communicate() in _clone() with thread-pumps (UNNECESSARY, see this later discussion).
• test_repo.py: unittestize (almost all) assertions.
• Fixed file-access problems by using with... construct (not complete).
  • Replace open --> with open for index (base and TC).
• Enabled some TCs, more remain dormant (ie the Hook TC never runs on Linux).
  • test_index.py: Enabled dormant fake-symlink assertion.
• Various encoding issues.
• Debug-log all Popen() calls, to collect cmd usage.

Apveyor results

• Py27: FAILED (SKIP=3, errors=1)
• Py34: FAILED (SKIP=3, errors=3)
• Py35 : FAILED (SKIP=3, errors=3)

(hopefully) closes #1339 This PR does as of e058c4c :

• renames timeout to kill_after_timeout and set default to None instead of 10 s
• Surfaces kill_after_timeout to user facing functions (remote. pull/push/fetch)
• refactor AutoInterrupt so we can ._terminate() the underlying process and grab status codes from terminated processes
• Add a class variable AutoInterrupt._status_code_if_terminate; if non-zero it overrides any status code from the process in _terminate. This can be used to prevent race conditions in tests, as described in this comment

Some history of the commits (while trying a different strategy):

aafb300 changes the default timeout to 60 s 1d26515 allows push, pull and fetch keyword timeout that will be propagated down febd4fe alters some tests to use the old 10 s timeout for fetch and pull 4113d01 also alters a test call to pull to add a timeout of 10s c55a8e3 fixes the test suite 7df33f3e sets the default timeout to None

if the current default is too lose, users can now give their own timeout

The destructor of the cmd class (actually AutoInterrupt) seems to throw some errors using python 3.5 on Arclinux:

Exception ignored in: <bound method Git.AutoInterrupt.__del__ of <git.cmd.Git.AutoInterrupt object at 0x7f7c2b101d68>>
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/git/cmd.py", line 294, in __del__
TypeError: 'NoneType' object is not callable
Exception ignored in: <bound method Git.AutoInterrupt.__del__ of <git.cmd.Git.AutoInterrupt object at 0x7f7c2b101cc0>>
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/git/cmd.py", line 294, in __del__
TypeError: 'NoneType' object is not callable

It seems this error is triggered by a call to sys.exit(0) at the end of my script, since it doesn't appear if I comment it out. However I couldn't reproduce this behavior with python 2.7, here sys.exit(0) seems to be working with the module correctly.

What I do is simply clone a repo, add some files commit and push them to a remote.

...
repo = Repo.clone_from(target_git_path, target_temp_dir)
...
repo.index.add([file])
...
repo.index.commit('Initial commit')
...
refspec='refs/heads/{0}:refs/heads/{0}'.format(repo.active_branch) # this will fix error if user has set "push.default = matching"
repo.remotes.origin.push(refspec=refspec)

Requirments file includes gitdb2 and, but "compat.py" file is importing "gitdb" instead and throwing missing "gitdb" module error when this library is fresh installed in a new enviroment with missing "gitdb" dependency library. Please fix the problem.

Stack trace:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 424, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/var/lib/abc/ramble/venom/fabfile.py", line 286, in deploy_celery
    prepare_deploy(settings, branch_name=branch_name)
  File "/var/lib/abc/ramble/venom/fabfile.py", line 195, in prepare_deploy
    repo_install_path=install_path)
  File "/var/lib/abc/ramble/ramble/utils/git_util.py", line 31, in set_deploy_context
    remote.fetch()
  File "/usr/local/lib/python2.7/site-packages/git/remote.py", line 613, in fetch
    res = self._get_fetch_info_from_stderr(proc, progress or RemoteProgress())
  File "/usr/local/lib/python2.7/site-packages/git/remote.py", line 552, in _get_fetch_info_from_stderr
    fetch_info_lines)
AssertionError: len(....) <= len (...) (elided for length)

I wrote a script to parse the two things that it is trying to compare, from the assert output. FETCH_HEAD right now has 813 lines. The output of git fetch, when I run it manually on the command line (I have not been able to figure out exactly what command it is running, but I'm assuming it's git fetch --all -v - is this correct?) has also 813 lines if you ignore the first two lines: Fetching origin and From github.com:org/repo

However, the very last 4 branches in FETCH_HEAD are missing in the GitPython fetch output, the last of which is this line:

a3a81646dd65dfa4b4412c0f3a5fd4183cbaa2e5 not-for-merge branch 'zendesk' of github.com:org/repo

It can't find the matching line in the fetch output (or for the previous 3):

= [up to date] zendesk -> origin/zendesk

Even though I see the above lines when I do git fetch --all -v, I don't see the last 4 lines in the GitPython assert output. Let me know if you need more info. Is there possibly some length limit? I'm using git 1.8.3.4.

BTW I was using GitPython==0.3.2.RC1 prior to this and it was still failing an assertion, although I'm not sure the assertion was exactly the same.

As we all know, today more than ever before, it is crucial to be able to trust our computing environments. One of the main difficulties that package maintainers of Linux distributions face, is the difficulty to verify the authenticity and the integrity of the source code.

The Arch Linux team would appreciate it if you would provide us GPG signatures in order to verify easily and quickly your source code releases.

Overview of the required tasks:

• Create and/or use a 4096-bit RSA keypair for the file signing.
• Keep your key secret, use a strong unique passphrase for the key.
• Upload the public key to a key server and publish the full fingerprint.
• Sign every new git commit and tag.
• Create signed compressed (xz --best) release archives
• Upload a strong message digest (sha512) of the archive
• Configure https for your download server

GPGit is meant to bring GPG to the masses. It is not only a shell script that automates the process of creating new signed git releases with GPG but also comes with this step-by-step readme guide for learning how to use GPG.

Additional Information:

• https://github.com/NicoHood/gpgit
• https://help.github.com/categories/gpg/
• https://wiki.archlinux.org/index.php/GnuPG
• https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
• https://www.qubes-os.org/doc/verifying-signatures/
• https://lkml.org/lkml/2016/8/15/445
• https://developers.google.com/web/fundamentals/security/encrypt-in-transit/why-https
• https://www.enigmail.net/index.php/en/

Thanks in advance.

PS: I plan to use this module in gpgit soon. However this and #611 is a blocking issue for this to happen. Would be really nice to see gpg signatures as well as this project part of gpgit.

Noticed that there is no verbose option when performing pull on a remote, and it is hardcoded to be verbose. Adding a keyword argument, similar to this change

This enhancement proposal is motivated by confusion in the behavior of subtrees. Issue #851 is one example. This stackoverflow question is another example.

As further example consider the following bash+python script behavior:

mkdir testrepo
cd testrepo
git init
mkdir 1
mkdir 2
echo hi > 1/hi.txt
cp 1/hi.txt 2/.
git add .
git commit -m 'test'

and then

import git

repo = git.Repo()
one = repo.head.commit.tree['1']
two = repo.head.commit.tree['2']
print("Same object?", one is two)
print("Same Python hash?", hash(one) == hash(two))
print("Same git hash?", one.hexsha == two.hexsha)
print("Equal?", one == two)
print("Same paths?", one.path == two.path)
print("hi.txt in one and two?", "hi.txt" in one, "hi.txt" in two)
print("1/hi.txt in one and two?", "1/hi.txt" in one, "1/hi.txt" in two)
print("2/hi.txt in one and two?", "2/hi.txt" in one, "2/hi.txt" in two)

which prints out:

Same object? False
Same Python hash? False
Same git hash? True
Equal? True
Same paths? False
hi.txt in one and two? False False
1/hi.txt in one and two? True False
2/hi.txt in one and two? False True

What I think most people would expect is that "hi.txt" is contained in both one and two and 1/hi.txt and 2/hi.txt are contained in neither one nor two. This is essentially the same issue noted in #851.

I think most people would also not expect these Tree objects to be ==-equal but then also have a property path that has different values.

It seems to me the fundamental problem is that the Tree class is trying to behave like two things that are different things:

• a git tree, in the git object database
• a subdirectory/subtree in a git index

In the sample above, one and two are the same git tree but they are not the same subdirectory added to a git index. A git tree does not have a path but a subdirectory added to a git index does. The current __contain__ logic is kind-of-OK if the objects are subdirectories added to a git index, but make no sense for a git tree.

Due to backwards compatibility, this seems a challenging issue to address. I don't know the code and backwards compatibility situation well enough to make any recommendations. But I'm happy to help. I can suggest some different approaches, but I figure it's best to seek your @Byron input on this first.

I can clone from a private repo by specifying the path to he ssh key, like this:

git_dir = "[email protected]:user_id/repo_name.git"
save_at = './'
Repo.clone_from(git_dir, save_at, env={"GIT_SSH_COMMAND": 'ssh -o StrictHostKeyChecking=no  -i ./my_path/mykey'})

However, I this code cannot pull from the same private repo

repo = Repo(save_at)
ssh_cmd =  'ssh -o StrictHostKeyChecking=no  -i ./my_path/mykey'}
with repo.git.custom_environment(GIT_SSH_COMMAND=ssh_cmd):
    repo.remotes.origin.pull('master')

and throws the error

File "/usr/local/lib/python3.10/site-packages/git/remote.py", line 910, in pull
    res = self._get_fetch_info_from_stderr(proc, progress,
  File "/usr/local/lib/python3.10/site-packages/git/remote.py", line 750, in _get_fetch_info_from_stderr
    proc.wait(stderr=stderr_text)
  File "/usr/local/lib/python3.10/site-packages/git/cmd.py", line 502, in wait
    raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(1)
  cmdline: git pull -v origin master
  stderr: 'fatal: Could not read from remote repository.'

What am I missing here?

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/git/cmd.py", line 114, in pump_stream
    handler(line)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/git/diff.py", line 570, in <lambda>
    handle_process_output(proc, lambda byt: cls._handle_diff_line(byt, repo, index),
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/git/diff.py", line 557, in _handle_diff_line
python-BaseException
    diff = Diff(repo, a_path, b_path, a_blob_id, b_blob_id, old_mode, new_mode,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/git/diff.py", line 303, in __init__
    self.a_mode = mode_str_to_int(a_mode) if a_mode else None
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/git/objects/util.py", line 80, in mode_str_to_int
    mode += int(char) << iteration * 3
ValueError: invalid literal for int() with base 10: 'x'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/git/cmd.py", line 120, in pump_stream
    raise CommandError([f'<{name}-pump>'] + remove_password_if_present(cmdline), ex) from ex
git.exc.CommandError: Cmd('<stdout-pump>') failed due to: ValueError('invalid literal for int() with base 10: 'x'')
  cmdline: <stdout-pump> git diff-tree 81a1f0c4f84aeb6ee97edea64251419515eff571 8c38dfe86157463d745c453d9a1d8c066cc9b7c5 -r --abbrev=40 --full-index -M --raw -z --no-color

This is my first time posting an issue, so please tell me what extra information is needed!

When I git init --object-format=sha256 I receive

Traceback (most recent call last):
  ...
  File "/usr/local/lib/python3.9/site-packages/git/repo/fun.py", line 163, in name_to_object
    raise BadName(name)
gitdb.exc.BadName: Ref 'HEAD' did not resolve to an object

because the hash size is longer

Just happen to notice that in GitPython == 3.1.27 the static typing pip mypy fails to recognize the **kwargs of Remote.fetch() in remote.py. This happens because the kwargs argument is typed as Any, which is plausible but may be narrowed down to e.g. Union[None, Dict[str, int, bool]]. For the correct list of possible types we'd need to check docs at git-fetch.

Use case is the following code snippet:

    def fetch_all(self, repository: git.Repo) -> List[git.FetchInfo]:
        fetch_kwargs = {'tags': True, 'force': True, 'prune': True, 'prune-tags': True}

        origin_server = repository.remotes.origin
        origin_server.fetch(**fetch_kwargs)  # Note: mypy does not recognize kwargs because GitPython does type it as Any
        fetch_info_list = origin_server.pull()

        return fetch_info_list

(Code above is my custom code and not part of this project!)

Edit: Using mypy==0.971 and mypy-extensions==0.4.3. Not sure if mypy should ignore this at least we can help it find a potential type.