Fixes issue 68: https://github.com/gaubert/gmvault/issues/68 .

The command to export is: gmvault export -d DB_DIR -t FORMAT OUTPUT_DIR

Currently valid formats are 'maildir' and 'mbox'. The maildir variant is 'Maildir++' as used by dovecote, and indeed the exported maildir can be used as-is as a dovecote backend. The original 'mboxo' variant of mbox is used, with '.sbd' as a suffix for directories as used by Thunderbird.

As mentioned in the issue, the maildir (or mbox) format duplicates emails, once per label. This is clearly not optimal, but there's no way to avoid it. Users should continue to use gmvault's internal format for everyday use, only exporting to maildir when they need it.

FYI, I just received this email. I've been using Gmvault for a few years assuming it was abandoned, since there have been no commits since 2016, but thought I would create this issue anyway, just in case...

Hello,

Although you don’t need to take any action, we wanted to let you know that the following third-party apps will no longer be able to access some data in your Google Account, including your Gmail content. This change will go into effect starting March 31, 2019.

Gmvault

We are making this change as part of ongoing efforts to make sure your data is protected and private. These apps haven’t yet complied with our updated data privacy requirements announced on October 8, 2018.

You can always view, manage and remove apps you’ve given access to your account by visiting your Google Account.

Thanks, The Google Accounts team

I started downloading Imap messages fine. Then about 700 messages in it just freezes.

After stopping the process, I removed the .info db and quarantine directories. And tried again. But I keep getting this error

Received an IMAP abort error. Wait 2 seconds and retrying.
Disconnecting from Gmail Server and sleeping ...
Reconnecting to the from Gmail Server.
Get XOAuth credential from /root/.gmvault/[email protected]

I've also deleted /root/.gmvault config and re-authed with Google, without luck. Any ideas?

I've tried to perform a complete backup, it works well but when using french language, it seems that Chat conversations are not backed up. When I change the language interface for english, it's working.

New user of v1.0-beta. Happy user. I downloaded all 20,000 emails from one Gmail account and restored them to another. Everything worked, even more easily than I hoped.

So, then I tried the same thing with another Gmail account. Downloaded all 2,000 emails as problem-free as before. But when I went to restore them to another Gmail account, only two of the emails were restored before I hit this error:

Error create failed: '[ALREADYEXISTS] Duplicate folder name (Failure)'. For more information see log file.

I can't find the log file. It's not in the .gmvault directory. It's not in the gm-vault.db directory. It's not in the directory where gmvault itself is installed. Rerunning with --debug doesn't seem to create a log file, either. In both cases, a log is dumped to the screen (which is where I pulled the error message above), but I can't find any more information about what this error means or how to get around it. It's repeatable.

Help, please.

I have set up a small Gmail account with 5000 messages. I already did an initial sync with gmvault. When I do a full sync in order to update my backup, it takes about 7m30 to complete. Even if I just did a sync moments before, it still takes gmvault seven minutes to figure out that it has nothing to do.

In comparison, when I do this using OfflineIMAP, it takes only 8s to complete. It's much faster even though OfflineIMAP isn't smart about things, and has to scan every label separately!

The situation becomes worse as the size of the Gmail account increases. On my main account with 185 000 emails, gmvault takes over 13 hours to sync. Again, OfflineIMAP is much quicker, taking only a few minutes. The obvious workaround is to do a 'quick sync', but quick syncs ignore changes to old messages, so I have to do full syncs at least once in awhile.

Profiling indicates that most of gmvault's time is spent executing this line:

new_data = self.src.fetch(the_id, imap_utils.GIMAPFetcher.GET_ALL_BUT_DATA )

This is run once per message, but it could be done for groups of messages instead. Here are some runtimes for different batch sizes on my test account:

1 message at a time - 7m30 10 messages - 1m08 30 - 36s 100 - 28s 1000 - 20s

Here's the change I made to test this out: https://github.com/vasi/gmvault/commit/e892fdbec81043dfc9d51c32251b7b97a7b7e274 . It really is a hack for now, I have no idea whether it interacts ok with things like quarantine and other error checking.

Would some sort of batching like this be an acceptable optimization? If I decide to work on a patch, any advice on how you'd like it to integrate with current code?

• What version of Gmvault do you use: v.1.7-beta
• On what Operating System (Linux flavour, MacOSX, Win7/XP/Vista) ? Win XP
• Type of requests (Bug, documentation issue, features) ? Bug
• Explain the request subject (issue details, feature details): I get the following error when restoring my email to a new Google Apps account:

Error when reaching Gmail server. Wait 4 seconds and retry up to 2 times. Error: UID command error: BAD ['Invalid Arguments: Label name is not allowed: Migrated'].

=== Exception traceback === Traceback (most recent call last): File "gmv\gmv_cmd.pyc", line 645, in run File "gmv\gmv_cmd.pyc", line 531, in _restore File "gmv\gmvault.pyc", line 1396, in restore File "gmv\gmvault.pyc", line 1747, in restore_emails error: UID command error: BAD ['Invalid Arguments: Label name is not allowed: Migrated']

• In case of bug or issue, please detail how we can reproduce the problem: I simply ran gmvault restore --label "dlp account" [email protected]
• Contact details (emails to recontact you if you didn't use the github interface):

Any specific tags to check out or should I just use the 1.8.2 branch? Feels counterintuative to test 1.9-beta on a 1.8.2 branch.

Give me a hint and I will work on updating the docker image and run my various backups for a test run.

After restoring the first 19050 emails to a new gmail address (migrating from an old one), I received the following exception (win7 x64, 6gb ram, gmvault 1.5-beta):

Restore email with id 1319881181591886388. Error: .

=== Exception traceback === Traceback (most recent call last): File "gmv_cmd.py", line 515, in run File "gmv_cmd.py", line 423, in _restore File "gmvault.pyc", line 1032, in restore MemoryError

=== End of Exception traceback ===

When retrying, it raised the same exception after restoring only 6550 mails. I already tried to see if it possible to adjust the amount of reserved memory for python application (as it is possible for Java for example), but apparently there is no such option as Python's memory manager takes care of that.

I haven't done any Python coding myself, but my first guess at a fix would be that some resources need to be freed after a successful iteration of the for-loop starting at line 960 of gmvault.py...

Hi,

I've been trying to restore recently backuped mailbox to the new one and got an error after processing a few hundreds emails:

Eugene-MacBook:bin ei$ ./gmvault restore *******@gmail.com --resume [....] Jun 25 23:02:02 Eugene-MacBook.local gmv_runner[30449] : Pushing email body with id 1407299977163162301. Jun 25 23:02:03 Eugene-MacBook.local gmv_runner[30449] : Pushing email body with id 1407316912607036964. Jun 25 23:02:03 Eugene-MacBook.local gmv_runner[30449] : Applying labels to the current batch of emails. Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : Error: %d format: a number is required, not set. Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : === Exception traceback === Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : Traceback (most recent call last): Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : File "gmv/gmv_cmd.pyc", line 745, in run Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : File "gmv/gmv_cmd.pyc", line 621, in _restore Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : File "gmv/gmvault.pyc", line 882, in restore Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : File "gmv/gmvault.pyc", line 1146, in restore_emails Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : TypeError: %d format: a number is required, not set Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : Jun 25 23:02:04 Eugene-MacBook.local gmv_runner[30449] : === End of Exception traceback ===

in debug mode: Eugene-MacBook:bin ei$ ./gmvault restore ********@gmail.com --resume -debug [....] Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : result_uid = 882 Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : label = \Inbox Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Pushing email body with id 1407316912607036964. Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Subject = Re: news. Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Before to Append email contents Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Appended data with flags [u'\Seen'] and internal time 2012-07-12 22:39:54. Operation time = 0.355708122253. Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : res = APPENDUID 11 883 Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : result_uid = 883 Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : label = \Inbox Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : label = \Important Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Labels creation tentative for emails with ids <itertools.ifilter object at 0x10622e390>. Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Labels to create: [set([u'\Draft', u'\Inbox', u'\Sent', u'\Important'])] Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Applying labels to the current batch of emails. Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Changing directory. Going into ALLMAIL Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Changed dir. Operation time = 0.190242052078 ms Jun 25 23:15:58 Eugene-MacBook.local gmv_runner[30528] : Applying labels [u'\Draft'] Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : After storing labels (\Draft). Operation time = 0.106867074966 s. Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : ret = NO Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : data=['System Error (Failure)'] Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : Problem when applying labels \Draft to the following ids: set([865, 874]) Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : Problem when applying labels. Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : Error: %d format: a number is required, not set. Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : === Exception traceback === Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : Traceback (most recent call last): Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : File "gmv/gmv_cmd.pyc", line 745, in run Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : File "gmv/gmv_cmd.pyc", line 621, in _restore Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : File "gmv/gmvault.pyc", line 882, in restore Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : File "gmv/gmvault.pyc", line 1146, in restore_emails Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : TypeError: %d format: a number is required, not set Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : Jun 25 23:15:59 Eugene-MacBook.local gmv_runner[30528] : === End of Exception traceback ===

Hi,

when I try to launch gmailvault, I've got the following :

~/Downloads/gmvault-v1.0-beta/bin $ ./gmvault 
dyld: Library not loaded: @executable_path/../.Python
  Referenced from: /Users/jp/Downloads/gmvault-v1.0-beta/bin/../lib/python-lib/bin/python
  Reason: image not found
./gmvault: line 43:   569 Trace/BPT trap          $PYTHON_BIN /tmp/rnpicker.bootstrap.$$ $ARGS

MacOSX 10.6.8

received this.. any ideas what we should change in order to keep gmvault working?

In .conf there is a line that refers to oob

# Hardcoded dummy redirect URI for non-web apps. redirect_uri=urn:ietf:wg:oauth:2.0:oob

email:

Our records indicate you have OAuth clients that used the OAuth OOB flow in the past.
Hello Google OAuth Developer,

We are writing to inform you that OAuth out-of-band (OOB) flow will be deprecated on October 3, 2022, to protect users from phishing and app impersonation attacks.

What do I need to know?
Starting October 3, 2022, we will block OOB requests to Google’s OAuth 2.0 authorization endpoint for existing clients. Apps using OOB in testing mode will not be affected. However, we strongly recommend you to migrate them to safer methods as these apps will be immediately blocked when switching to in production status.

Note: New OOB usage has already been disallowed since February 28, 2022.

Below are key dates for compliance

September 5, 2022: A user-facing warning message may be displayed to non-compliant OAuth requests
October 3, 2022: The OOB flow is blocked for all clients and users will see the error page.
Please check out our recent blog post about Making Google OAuth interactions safer for more information.

What do I need to do?
Migrate your app(s) to an appropriate alternative method by following these instructions:

Determine your app(s) client type from your Google Cloud project by following the client links below.
Migrate your app(s) to a more secure alternative method by following the instructions in the blog post above for your client type.
If necessary, you may request a one-time extension for migrating your app until January 31, 2023. Keep in mind that all OOB authorization requests will be blocked on February 1, 2023.

The following OAuth client(s) will be blocked on Oct 3, 2022.

OAuth client list:

Project ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Client: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thanks for choosing Google OAuth.

— The Google OAuth Developer Team

I'm sure this is a known issue but I couldn't find any trace of this error on github nor a workaround on the gmvault website.

After running gmvault sync [email protected] and pressing ENTER to open the browser I get the following error:

This app is blocked This app tried to access sensitive info in your Google Account. To keep your account safe, Google blocked this access.

Unfortunately I am having an issue restoring my backed up mails to a new account (Error: argument of type 'NoneType' is not iterable) and it seems this is a well known error that was addressed by the 1.9.2 beta or a patch. Unfortunately I am not very deep into programming (yes, the typical dumb windows user) and I am despairing of installing /upgrading to 1.9.2 beta. I couldn't find a howto describing the installation process. Is there anybody here who can explain me (ideally in a way an Windows user understands) what to do to get to the 1.9.2 beta

Thanks for your support!

There is a small typo in src/gmv/gmvault_utils.py.

Should read return rather than retrun.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

Hi there, I tried to install gmvault via pip, however the setup failed, I'm not familiar at all with python. Did I got something wrong ?

$ pip3 install gmvault
Collecting gmvault
  Downloading gmvault-1.9.1.tar.gz (97 kB)
     |████████████████████████████████| 97 kB 6.4 MB/s
Collecting argparse
  Downloading argparse-1.4.0-py2.py3-none-any.whl (23 kB)
Collecting Logbook==0.10.1
  Downloading Logbook-0.10.1.tar.gz (155 kB)
     |████████████████████████████████| 155 kB 18.2 MB/s
    ERROR: Command errored out with exit status 1:
     command: /usr/local/opt/[email protected]/bin/python3.8 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/3_/ff3xnwm57k33ctrbjtsxx2vr0000gn/T/pip-install-9j233xxr/logbook/setup.py'"'"'; __file__='"'"'/private/var/folders/3_/ff3xnwm57k33ctrbjtsxx2vr0000gn/T/pip-install-9j233xxr/logbook/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/3_/ff3xnwm57k33ctrbjtsxx2vr0000gn/T/pip-pip-egg-info-gokc7vup
         cwd: /private/var/folders/3_/ff3xnwm57k33ctrbjtsxx2vr0000gn/T/pip-install-9j233xxr/logbook/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/3_/ff3xnwm57k33ctrbjtsxx2vr0000gn/T/pip-install-9j233xxr/logbook/setup.py", line 57, in <module>
        from setuptools import setup, Extension, Feature
    ImportError: cannot import name 'Feature' from 'setuptools' (/usr/local/lib/python3.8/site-packages/setuptools/__init__.py)
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

$ pip3 --version
pip 20.2.4 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)
$ python3 --version
Python 3.8.6