First things first... Thanks so much @mandarons, this project is what I've looking for since years... I appreciate so much your efforts on this...

Describe the bug Some files are downloaded on every pass, even if the file already exists and no changes have occurred on it... Usually this issue is related to MacOS packages, which are downloaded as a file, instead of as a directory, but I have also noticed that it applies to many photos...

To Reproduce In my case, this behavior seems to be always the same, although I'm not sure you know which files it occurs with and which it doesn't.

Expected behavior I expect not to download any file if it is already downloaded and it has not changed...

Screenshots You can see some examples on "icloud.log" where unmodified files are downloaded several times (PHOTOS sync_interval is lower than DRIVE one):

/app # more icloud.log |grep LOGO-Contacto
2022-02-26 17:02:56,650 :: INFO :: root :: sync_drive.py :: 90 :: Downloading /app/icloud/drive/Pixelmator/LOGO-Contacto.pxm ...
2022-02-27 10:33:47,321 :: INFO :: root :: sync_drive.py :: 90 :: Downloading /app/icloud/drive/Pixelmator/LOGO-Contacto.pxm ...
2022-02-28 04:38:14,972 :: INFO :: root :: sync_drive.py :: 90 :: Downloading /app/icloud/drive/Pixelmator/LOGO-Contacto.pxm ...
/app # ls -la /app/icloud/drive/Pixelmator/LOGO-Contacto.pxm 
----------    1 root     root        899758 Sep 21  2018 /app/icloud/drive/Pixelmator/LOGO-Contacto.pxm

/app # more icloud.log |grep "Imagen 175.JPG"
2022-02-26 23:20:18,431 :: INFO :: root :: sync_photos.py :: 37 :: Downloading /app/icloud/photos/all/Imagen 175.JPG ...
2022-02-26 23:47:35,904 :: INFO :: root :: sync_photos.py :: 37 :: Downloading /app/icloud/photos/all/Imagen 175.JPG ...
2022-02-27 00:59:13,071 :: INFO :: root :: sync_photos.py :: 37 :: Downloading /app/icloud/photos/all/Imagen 175.JPG ...
2022-02-27 17:27:57,205 :: INFO :: root :: sync_photos.py :: 37 :: Downloading /app/icloud/photos/all/Imagen 175.JPG ...
2022-02-27 17:50:31,021 :: INFO :: root :: sync_photos.py :: 37 :: Downloading /app/icloud/photos/all/Imagen 175.JPG ...
2022-02-27 18:41:28,510 :: INFO :: root :: sync_photos.py :: 37 :: Downloading /app/icloud/photos/all/Imagen 175.JPG ...
2022-02-28 10:42:34,244 :: INFO :: root :: sync_photos.py :: 37 :: Downloading /app/icloud/photos/all/Imagen 175.JPG ...
2022-02-28 11:05:29,406 :: INFO :: root :: sync_photos.py :: 37 :: Downloading /app/icloud/photos/all/Imagen 175.JPG ...
2022-02-28 11:56:30,420 :: INFO :: root :: sync_photos.py :: 37 :: Downloading /app/icloud/photos/all/Imagen 175.JPG ...
/app # ls -la "/app/icloud/photos/all/Imagen 175.JPG"
----------    1 root     root        735560 Mar 13  2011 /app/icloud/photos/all/Imagen 175.JPG

Configuration My config YAML is:

app:
  logger:
    level: "info"
    filename: "icloud.log"
  credentials:
    username: "MY-ID"
    retry_login_interval: 600
  root: "icloud"
  smtp:
drive:
  destination: "drive"
  remove_obsolete: true
  sync_interval: 3000
  filters:
    folders:
    file_extensions:
photos:
  destination: "photos"
  remove_obsolete: true
  sync_interval: 300
  filters:
    albums:
    file_sizes:
      - "original"

Describe the bug

-v '/mnt/user/appdata/icloud-drive-sync/config.yaml':'/app/config.yaml':'rw' -v '/mnt/user/icloud-drive-sync/':'/app/icloud/':'rw' --restart always 'mandarons/icloud-drive'
63a2034e9070c6f14422ee29112f6ca48ea8d28146184a4c26d01d94866301da
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting "/mnt/user/appdata/icloud-drive-sync/config.yaml" to rootfs at "/app/config.yaml" caused: mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.

To Reproduce It appears on Unraid

Screenshots

I tried to use newest version published and successfully authenticated and started downloading all files in Documents folder, then suddenly icloud crashed on sync:

icloud-drive    | No changes detected. Skipping the file /app/drive/Documents/for_blog_post/Figure_1.png
icloud-drive    | No changes detected. Skipping the file /app/drive/Documents/Foundation Ontologies for Distributed Multi-Agent systems/Screenshot 2020-02-11 at 09.55.18.png
icloud-drive    | No changes detected. Skipping the file /app/drive/Documents/Foundation Ontologies for Distributed Multi-Agent systems/Screenshot 2020-02-11 at 10.00.59.png
icloud-drive    | No changes detected. Skipping the file /app/drive/Documents/Foundation Ontologies for Distributed Multi-Agent systems/Screenshot 2020-02-11 at 10.05.13.png
icloud-drive    | No changes detected. Skipping the file /app/drive/Documents/Foundation Ontologies for Distributed Multi-Agent systems/Screenshot 2020-02-11 at 10.10.45.png
icloud-drive    | No changes detected. Skipping the file /app/drive/Documents/Foundation Ontologies for Distributed Multi-Agent systems/Screenshot 2020-02-11 at 10.22.21.png
icloud-drive    | No changes detected. Skipping the file /app/drive/Documents/Foundation Ontologies for Distributed Multi-Agent systems/Screenshot 2020-02-11 at 10.26.28.png
icloud-drive    | No changes detected. Skipping the file /app/drive/Documents/Foundation Ontologies for Distributed Multi-Agent systems/Screenshot 2020-02-11 at 10.34.40.png
icloud-drive    | Downloading /app/drive/Documents/Foundation Ontologies for Distributed Multi-Agent systems.md ...
icloud-drive    | Downloading /app/drive/Documents/Foundation Ontologies for Distributed Multi-Agent systems.textbundle ...
icloud-drive    | Traceback (most recent call last):
icloud-drive    |   File "/app/./src/main.py", line 6, in <module>
icloud-drive    |     sync.sync_drive()
icloud-drive    |   File "/app/src/sync.py", line 159, in sync_drive
icloud-drive    |     sync_directory(drive=api.drive, destination_path=destination_path, root=destination_path,
icloud-drive    |   File "/app/src/sync.py", line 136, in sync_directory
icloud-drive    |     files.update(sync_directory(drive=item, destination_path=new_folder, items=item.dir(), root=root,
icloud-drive    |   File "/app/src/sync.py", line 141, in sync_directory
icloud-drive    |     process_file(item=item, destination_path=destination_path,
icloud-drive    |   File "/app/src/sync.py", line 102, in process_file
icloud-drive    |     download_file(item=item, local_file=local_file, verbose=verbose)
icloud-drive    |   File "/app/src/sync.py", line 82, in download_file
icloud-drive    |     with item.open(stream=True) as response:
icloud-drive    |   File "/usr/local/lib/python3.9/site-packages/pyicloud/services/drive.py", line 270, in open
icloud-drive    |     return self.connection.get_file(self.data["docwsid"], **kwargs)
icloud-drive    |   File "/usr/local/lib/python3.9/site-packages/pyicloud/services/drive.py", line 58, in get_file
icloud-drive    |     url = response.json()["data_token"]["url"]
icloud-drive    | KeyError: 'data_token'

Describe the bug After execting the login command given in README.md

docker exec -it icloud /bin/sh -c "icloud --username=xxxx --session-directory=/app/session_data"

The following error shows:

tian@dorm-synology:/volume1/docker/icloud-drive$ sudo docker exec -it icloud /bin/sh -c "icloud [email protected] --session-directory=/app/session_data"
Enter iCloud password for [email protected]:
Traceback (most recent call last):
  File "/app/venv/bin//icloud", line 8, in <module>
    sys.exit(main())
  File "/app/venv/lib/python3.9/site-packages/icloudpy/cmdline.py", line 207, in main
    api = ICloudPyService(
  File "/app/venv/lib/python3.9/site-packages/icloudpy/base.py", line 273, in __init__
    self.authenticate()
  File "/app/venv/lib/python3.9/site-packages/icloudpy/base.py", line 344, in authenticate
    self._webservices = self.data["webservices"]
KeyError: 'webservices'

To Reproduce Steps to reproduce the behavior:

1. Execute the command above.
2. See error

Configuration If applicable, please share the configuration details

docker-compose.yaml:

version: "3.4"
services:
  icloud:
    image: mandarons/icloud-drive
    environment:
      - PUID=1026
      - GUID=100
    env_file:
      - /volume1/docker/icloud-drive/.env.icloud
    container_name: icloud
    restart: unless-stopped
    volumes:
      - /volume1/docker/icloud-drive/config.yaml:/app/config.yaml
      - /volume2/Onedrive/icloud:/app/icloud/drive
      - /volume1/docker/icloud-drive/session_data:/app/session_data

config.yaml:

app:
  logger:
    # level - debug, info (default), warning or error
    level: "info"
    # log filename icloud.log (default)
    filename: "icloud.log"
  credentials:
    # iCloud drive username
    username: "[email protected]"
    # Retry login interval - default is 10 minutes
    retry_login_interval: 600
  # Drive destination
  root: "icloud"
  smtp:
    ...
drive:
  destination: "drive"
  remove_obsolete: false
  sync_interval: 300

Describe the bug The sync process consistently hangs during downloading large Drive objects. The log outputs "Downloading [file name]..." but never returns or prints other output, even after waiting hours. The Docker task continues to run, however. Small files preceding the large file download fine.

Configuration Latest commit from the Git repo. Docker version 20.10.12, build e91ed57.

I also tried cloning the repo and adding additional lines of verbose output to identify exactly where the download hangs. The code enters the copyfileobj call on line 100 but does not return from it.

Describe the bug Application folders do not sync, root folders are created but no sub folders or files are downloaded.

To Reproduce Steps to reproduce the behavior:

1. For instance, a note taking app like Obsidian creates an application folder on iCloud Drive to have all your notes synced across all devices.
2. In the configuration, comment out any and all include folder or include files so it syncs everything.
3. Folders get created but there are no contents, empty root folder Obsidian.

Expected behavior Should recursively download and sync all files, including application specific folders.

Screenshots Screenshot of an iCloud Application folder, for example Obsidian:

Screenshot of Docker iCloud data output folder:

See issue 98, 99 and 100.

It could be improved by adding hash to filename only when there's conflict.

I skip "All albums" folder because in my case all photos belongs to (at least) one folder.

This looks like a promising project that I would really like to get working. I installed per the readme. Getting the following error when attempting to sync:

Traceback (most recent call last): File "/app/./src/main.py", line 6, in sync.sync() File "/app/src/sync.py", line 28, in sync config=config, photos=api.photos, verbose=verbose File "/app/venv/lib/python3.9/site-packages/icloudpy/base.py", line 578, in photos self._photos = PhotosService(service_root, self.session, self.params) File "/app/venv/lib/python3.9/site-packages/icloudpy/services/photos.py", line 147, in init request = self.session.post( File "/app/venv/lib/python3.9/site-packages/requests/sessions.py", line 577, in post return self.request('POST', url, data=data, json=json, **kwargs) File "/app/venv/lib/python3.9/site-packages/icloudpy/base.py", line 158, in request self._raise_error(code, reason) File "/app/venv/lib/python3.9/site-packages/icloudpy/base.py", line 176, in _raise_error raise api_error icloudpy.exceptions.ICloudPyServiceNotActivatedException: Please log into https://icloud.com/ to manually finish setting up your iCloud service (ZONE_NOT_FOUND)

Any help would be greatly appreciated!

Describe the bug An error occurs during the process_photo routine. Other files seem to download fine.

2022-01-24T23:39:33.858349778Z Traceback (most recent call last):
2022-01-24T23:39:33.858359179Z   File "/app/./src/main.py", line 6, in <module>
2022-01-24T23:39:33.858424680Z     sync.sync()
2022-01-24T23:39:33.858433902Z   File "/app/src/sync.py", line 25, in sync
2022-01-24T23:39:33.858492048Z     sync_photos.sync_photos(
2022-01-24T23:39:33.858503176Z   File "/app/src/sync_photos.py", line 83, in sync_photos
2022-01-24T23:39:33.858555230Z     sync_album(
2022-01-24T23:39:33.858562917Z   File "/app/src/sync_photos.py", line 68, in sync_album
2022-01-24T23:39:33.858621337Z     process_photo(photo, file_size, destination_path, verbose)
2022-01-24T23:39:33.858634493Z   File "/app/src/sync_photos.py", line 56, in process_photo
2022-01-24T23:39:33.858673947Z     if photo_exists(photo, file_size, photo_path, verbose=verbose):
2022-01-24T23:39:33.858684174Z   File "/app/src/sync_photos.py", line 27, in photo_exists
2022-01-24T23:39:33.858743616Z     remote_size = int(photo.versions[file_size]["size"])
2022-01-24T23:39:33.858753439Z KeyError: 'medium'

Configuration Standard config, with file_sizes set to original & medium.

Use case I have some projects in my iCloud documents with .git and node_module folders that i don't really need to sync.

Describe the solution you'd like Add an ignore to the yml. Allow direct paths or asterisks eg,

ignore:
  - */node_modules # ignore node modules
  - /projects/examples/.git # ignore specific git folder

Use case As a user, I want to pass password through environment variable ICLOUD_PASSWORD so that I don't have to enter it manually.

Describe the solution you'd like Ability to pass password through environment variable in container

Describe alternatives you've considered Passing through config.yaml. Generally environment variables are considered as more secure than plain-text files.

Additional context None

Describe the bug Files from unpacked package are deleted at end of sync.

To Reproduce Do normal sync_drive with remove_obsolete: true

Expected behavior Files from package are not removed

Screenshots

Configuration remove_obsolete: true

Sync all folders

Use case I want to backup all photos but also preserve albums structure.

Describe the solution you'd like When not providing filter and providing extra flag docker should backup photos not to all folder but according to album structure. It would be nice if docker would support nested albums as nested folders. Otherwise it can create flat folder structure replacing / with -, eg. Folder-Subfolder/[photos inside] instead of Folder/Subfolder/[photos inside].

Describe alternatives you've considered I do not see any :(

Additional context Photo albums are available as filters so it should be doable. Maybe if providing filter as * it should copy all?

eg.

if filters["albums"]:
    if filters["albums"] == ["*"]:
        for album in photos.albums:
            sync_album(
                album=photos.albums[album],
                destination_path=os.path.join(destination_path, album),
                file_sizes=filters["file_sizes"],
            )
    else:
       old code path

Describe the bug Photo Library can contain multiple photos with same @property filename. Eg. default names IMG_xyza.HEIC when using multiple phones.

To Reproduce Add photos with same names to library and run docker image. You will see three things:

1. number of downloaded files is different than in iCloud
2. docker constantly downloads same files
3. number of downloaded files is constant

Expected behavior Docker downloads different files even if filenames are same. It should use @property id to differentiate photos.

Screenshots Configuration Backup all photos

Additional context None

Use case As a user, I want to download the entire content of my iCloud account to have a local backup. This means, as I own a 2TB account, I have to potentially download a lot of files and pictures. Unfortunately, this tool executes the file download sequentially which causes the sync job to last weeks.

Describe the solution you'd like I would like to enable parallel downloads of files. This would shorten the script run time and allow a more frequent sync.

Describe alternatives you've considered I could think of having different folders with different synchronisation times, to allow frequent updates for folders with frequent changes and sporadic updates for folders I use less.

But this solution is very cumbersome and forces me to decide beforehand which folders I want to keep synchronised more and less often.

I've tried the file extention filter on the photos part, but it doesn't seem to work. Could it be extended to that also as my wife takes a billion screenshot of "stuff" and I don't want to waste storage on these? Basically I want to filter out the png files.

Thanks so much for this project, it is awesome.

Cheers.