py-image-dedup is a tool to sort out or remove duplicates within a photo library

Overview

py-image-dedup Build Status Code Climate PyPI version

py-image-dedup is a tool to sort out or remove duplicates within a photo library. Unlike most other solutions, py-image-dedup intentionally uses an approximate image comparison to also detect duplicates of images that slightly differ in resolution, color or other minor details.

It is build upon Image-Match a very popular library to compute a pHash for an image and store the result in an ElasticSearch backend for very high scalability.

asciicast

How it works

Phase 1 - Database cleanup

In the first phase the elasticsearch backend is checked against the current filesystem state, cleaning up database entries of files that no longer exist. This will speed up queries made later on.

Phase 2 - Counting files

Although not necessary for the deduplication process it is very convenient to have some kind of progress indication while the deduplication process is at work. To be able to provide that, available files must be counted beforehand.

Phase 3 - Analysing files

In this phase every image file is analysed. This means generating a signature (pHash) to quickly compare it to other images and adding other metadata of the image to the elasticsearch backend that is used in the next phase.

This phase is quite CPU intensive and the first run take take quite some time. Using as much threads as feasible (using the -t parameter) is advised to get the best performance.

Since we might already have a previous version of this file in the database before analysing a given file the file modification time is compared to the given one. If the database content seems to be still correct the signature for this file will not be recalculated. Because of this, subsequent runs will be much faster. There still has to happen some file access though, so it is probably limited by that.

Phase 4 - Finding duplicates

Every file is now processed again - but only by means of querying the database backend for similar images (within the given max_dist). If there are images found that match the similarity criteria they are considered duplicate candidates. All candidates are then ordered by the following criteria (in this exact order):

  1. pixel count (more is better)
  2. EXIF data (more exif data is better)
  3. file size (bigger is better)
  4. file modification time (newer is better)
  5. distance (lower is better)
  6. filename contains "copy" (False is better)
  7. filename length (longer is better) - (for "edited" versions)
  8. parent folder path length (shorter is better)
  9. score (higher is better)

The first candidate in the resulting list is considered to be the best available version of all candidates.

Phase 5 - Moving/Deleting duplicates

All but the best version of duplicate candidates identified in the previous phase are now deleted from the file system (if you didn't specify --dry-run of course).

If duplicates_target_directory is set, the specified folder will be used as a root directory to move duplicates to, instead of deleting them, replicating their original folder structure.

Phase 6 - Removing empty folders (Optional)

In the last phase, folders that are empty due to the deduplication process are deleted, cleaning up the directory structure (if turned on in configuration).

How to use

Install

Install py-image-dedup using pip:

pip3 install py-image-dedup

Configuration

py-image-dedup uses container-app-conf to provide configuration via a YAML file as well as ENV variables which generates a reference config on startup. Have a look at the documentation about it

See py_image_dedup_reference.yaml for an example in this repo.

Setup elasticsearch backend

Since this library is based on Image-Match you need a running elasticsearch instance for efficient storing and querying of image signatures.

Elasticsearch version

This library requires elasticsearch version 5 or later. Sadly the Image-Match library still specifies version 2, so a fork of the original project is used instead. This fork is maintained by me, and any contributions are very much appreciated.

Set up the index

py-image-dedup uses a single index (called images by default). When configured, this index will be created automatically for you.

Command line usage

py-image-dedup can be used from the command line like this:

py-image-dedup deduplicate --help

Have a look at the help output to see how you can customize it.

Daemon

CAUTION! This feature is still very much a work in progress. Always have a backup of your data!

py-image-dedup has a built in daemon that allows you to continuously monitor your source directories and deduplicate them on the fly.

When running the daemon (and enabled in configuration) a prometheus reporter is used to allow you to gather some statistical insights.

py-image-dedup daemon

Dry run

To analyze images and get an overview of what images would be deleted be sure to make a dry run first.

py-image-dedup deduplicate --dry-run

FreeBSD

If you want to run this on a FreeBSD host make sure you have an up to date release that is able to install ports.

Since Image-Match does a lot of math it relies on numpy and scipy. To get those working on FreeBSD you have to install them as a port:

pkg install pkgconf
pkg install py38-numpy
pkg install py27-scipy

For .png support you also need to install

pkg install png

I still ran into issues after installing all these and just threw those two in the mix and it finally worked:

pkg install freetype
pkg install py27-matplotlib  # this has a LOT of dependencies

Encoding issues

When using the python library click on FreeBSD you might run into encoding issues. To mitigate this change your locale from ANSII to UTF-8 if possible.

This can be achieved f.ex. by creating a file ~/.login_conf with the following content:

me:\
	:charset=ISO-8859-1:\
	:lang=de_DE.UTF-8:

Docker

To run py-image-dedup using docker you can use the markusressel/py-image-dedup image from DockerHub:

sudo docker run -t \
    -p 8000:8000 \
    -v /where/the/original/photolibrary/is/located:/data/in \
    -v /where/duplicates/should/be/moved/to:/data/out \
    -e PY_IMAGE_DEDUP_DRY_RUN=False \
    -e PY_IMAGE_DEDUP_ANALYSIS_SOURCE_DIRECTORIES=/data/in/ \
    -e PY_IMAGE_DEDUP_ANALYSIS_RECURSIVE=True \
    -e PY_IMAGE_DEDUP_ANALYSIS_ACROSS_DIRS=True \
    -e PY_IMAGE_DEDUP_ANALYSIS_FILE_EXTENSIONS=.png,.jpg,.jpeg \
    -e PY_IMAGE_DEDUP_ANALYSIS_THREADS=8 \
    -e PY_IMAGE_DEDUP_ANALYSIS_USE_EXIF_DATA=True \
    -e PY_IMAGE_DEDUP_DEDUPLICATION_DUPLICATES_TARGET_DIRECTORY=/data/out/ \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_AUTO_CREATE_INDEX=True \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_HOST=elasticsearch \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_PORT=9200 \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_INDEX=images \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_AUTO_CREATE_INDEX=True \
    -e PY_IMAGE_DEDUP_ELASTICSEARCH_MAX_DISTANCE=0.1 \
    -e PY_IMAGE_DEDUP_REMOVE_EMPTY_FOLDERS=False \
    -e PY_IMAGE_DEDUP_STATS_ENABLED=True \
    -e PY_IMAGE_DEDUP_STATS_PORT=8000 \
    markusressel/py-image-dedup:latest

Since an elasticsearch instance is required too, you can also use the docker-compose.yml file included in this repo which will set up a single-node elasticsearch cluster too:

sudo docker-compose up

UID and GID

To run py-image-dedup inside the container using a specific user id and group id you can use the env variables PUID=1000 and PGID=1000.

Contributing

GitHub is for social coding: if you want to write code, I encourage contributions through pull requests from forks of this repository. Create GitHub tickets for bugs and new features and comment on the ones that you are interested in.

License

py-image-dedup by Markus Ressel
Copyright (C) 2018  Markus Ressel

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see 
   .
Comments
  • v2.0.0 not published on PyPi

    v2.0.0 not published on PyPi

    Hi Markus,

    unfortunately I have some trouble with ElasticSearch and can't get the code running properly. I receive this TypeError: init() missing 1 required positional argument: 'el_version'

    (virtual_env) M-Air:Documents michael$ py-image-dedup deduplicate -d "/Users/michael/Desktop/bilder_test/input" -dr -esh "127.0.0.1"
    Traceback (most recent call last):
      File "/Users/michael/Documents/python_workspace/virtual_env/bin/py-image-dedup", line 8, in <module>
        sys.exit(cli())
      File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/py_image_dedup/cli/Main.py", line 91, in c_deduplicate
        image_signature_store=ElasticSearchStoreBackend(
      File "/Users/michael/Documents/Python_Workspace/virtual_env/lib/python3.8/site-packages/py_image_dedup/persistence/ElasticSearchStoreBackend.py", line 41, in __init__
        self._store = SignatureES(
    TypeError: __init__() missing 1 required positional argument: 'el_version'
    

    What can I do to solve this issue?

    Thanks in advance, Michael

    bug 
    opened by stormriderDE 8
  • README overhaul

    README overhaul

    Some things about how py-image-dedup works changed since v1.0.0 and the README needs some guidance on how to use the docker-compose file. A big overhaul of the README is necessary.

    Specify

    • since a fork of image-match supporting elasticsearch v6 as well as v7 is now used the cumbersome package dependency section for it can be removed
    • how the daemon works
    • how and what statistics are exposed
    • how to use with docker-compose
    enhancement 
    opened by markusressel 7
  • Simplified usage with Docker

    Simplified usage with Docker

    Hi there.

    This is more kind of a feature request then it is an issue. It would be great to have that entire solution in a self-contained Docker image. Both DB and application. This would make the usage far easier.

    For instance one could directly use it on Synology-NASes.

    Thanks for considering. If I have time, I might try to setup something myself.

    enhancement 
    opened by qdrop17 7
  • TypeError: Unable to serialize xxx (type: <class 'PIL.TiffImagePlugin.IFDRational'>)

    TypeError: Unable to serialize xxx (type: )

    Using the Docker image markusressel/py-image-dedup:latest, I modified the two mountpoint paths in docker-compose.yml and ran docker-compose up. I got a bunch of errors TypeError: Unable to serialize xxx (type: <class 'PIL.TiffImagePlugin.IFDRational'>) like this:

    py-image-dedup_1  | ERROR:root:({'path': '/mnt/source/0.jpg', 'signature': [0, 0, 0, 0, 1, 0, 0, -2, 0, 0, 0, -1, 1, -1, -2, -2, 0, 0, 0, -1, -2, -2, -2, -2, 0, 0, 0, 2, -1, -1, -2, -2, 0, 0, 0, 1, -1, -1, -2, -1, 0, 0, 0, 1, 1, -1, -1, -1, 0, 0, 0, -1, 0, -1, -1, -1, 0, 0, 0, 0, 1, -1, -1, -2, 0, 0, 0, -1, 0, -2, -2, 0, 0, 0, 1, 0, -2, 0, -2, -2, 2, 2, 2, 2, -1, -2, -2, 2, 2, 2, 1, 1, -1, -1, 2, 2, 2, 2, 1, 1, -1, 2, 2, 2, 2, 2, 1, 1, 1, 2, 2, 1, 1, 1, 1, -1, -1, 2, -1, 1, 1, 1, 1, 1, 0, 1, 2, 2, 1, 1, 2, 0, -1, 2, 2, 1, 2, 2, 0, 1, 0, 2, 1, 0, 0, 2, 2, 0, -1, 0, -1, 2, 2, 2, 1, 1, 2, 1, 2, 2, -2, -2, -2, -2, -1, 1, 1, 1, -2, -2, -2, 1, -1, 1, 2, -2, -2, -2, -2, 1, -2, 2, -2, -2, -1, 1, -1, 2, 2, 1, 0, 1, -1, -2, -2, -2, 1, -2, -1, -1, -2, -2, -2, -1, -2, -2, -1, -1, -1, -1, 0, 2, 0, 1, 1, 0, 0, 1, -1, 0, 2, 0, 1, 2, -2, -2, -1, -2, 1, -2, 2, -1, -2, -1, -1, -1, 1, 1, -1, -2, -1, -2, -2, -1, -2, -1, -2, -2, 2, 2, -1, 2, -1, 2, 2, 1, 2, 0, 2, 1, 1, 2, 1, 1, -1, 1, 2, -1, 1, 1, 1, 1, 1, 1, -1, -1, 0, -1, 1, 2, 1, -1, 0, 0, 0, 1, 2, 0, 0, -1, 2, 0, 2, 0, 1, 2, -2, -2, -1, -2, -2, -2, -2, -1, 1, 1, 1, 2, -2, -1, 1, -2, 2, 2, -2, 2, -1, 2, 0, 2, 2, -2, -2, 1, -1, 1, 2, 2, -1, -1, -1, 1, -1, 2, 2, 1, -1, -1, 1, 1, 1, 2, 1, 2, -1, -1, -1, -1, 1, 1, 2, 2, -2, -2, 0, -1, 0, 1, 1, 0, 0, -1, 2, 0, 2, 0, -2, 2, -2, 2, 1, -2, 2, -2, 2, -1, 1, -1, -2, -2, -2, 1, -2, -1, 2, 0, -1, 2, 2, 1, 2, 2, -2, -2, -2, -2, -2, -2, -1, -1, -2, -2, -2, 2, -2, 1, 1, 1, -1, -1, -1, 2, 1, 2, 2, 1, -2, -2, -1, -1, 1, 2, -1, 2, -2, -1, 0, -1, 0, -1, 1, 0, 0, 2, 2, 0, 2, 0, 0, 2, -2, -2, -1, -2, -2, -2, -2, -2, 1, 2, -1, 2, 1, -1, -1, 2, 1, -2, 2, -1, 1, -2, 1, 1, -2, 1, -1, -1, -1, -1, 0, 1, 1, -1, -2, 1, -1, 1, 1, 1, -1, -2, -2, 1, -2, 1, 1, -2, -1, 1, 1, 2, 2, 2, -2, -2, -2, -1, 0, -2, 0, -2, -2, 0, 0, 0, 2, 0, 2, 0, 2, 1, -2, 2, 1, -2, 1, 2, -2, 2, 2, 1, 2, -1, 2, -2, 2, 2, -2, -1, 1, -2, 1, 2, 1, -1, -1, 0, -1, -1, 1, 1, -1, 1, -1, -1, -1, -1, -1, -2, 1, -1, -1, -1, -2, 1, -2, 1, 1, -2, 2, 2, 2, 2, -1, 2, 2, 2, 2, 2, 0, 1, 0, 2, 2, 0, 0, -2, -2, 0, -2, 0, 0, 0, -1, 2, 2, 2, 2, 0, 0, 0, -2, -2, -2, -2, -1, 0, 0, 0, -2, -1, -1, 1, -1, 0, 0, 0, 1, 1, 2, 1, 2, 0, 0, 0, -1, -1, -1, -2, -1, 0, 0, 0, 1, -1, -2, 1, -2, 0, 0, 0, 2, -2, -2, 2, 1, 0, 0, 0, -2, -2, 0, -1, 0, 0, 0, 0], 'metadata': {'path': '/mnt/source/0.jpg', 'py-image-dedup_datamodel-version': 5, 'filesize': 120968, 'file_modification_date': 1148934648.0, 'pixelcount': 272640, 'exif_data': {'ExifVersion': "b'0220'", 'ShutterSpeedValue': 5.64385986328125, 'ApertureValue': 4.33984375, 'DateTimeOriginal': '2006:05:17 19:53:58', 'DateTimeDigitized': '2006:05:17 19:53:58', 'ExposureBiasValue': 0.0, 'MaxApertureValue': 4.33984375, 'MeteringMode': 5, 'ColorSpace': 1, 'Flash': 0, 'FocalLength': 105.0, 'Make': 'Canon', 'Model': 'Canon EOS DIGITAL REBEL', 'Orientation': 1, 'YCbCrPositioning': 1, 'SensingMethod': 2, 'ExposureTime': 0.02, 'XResolution': 72.0, 'YResolution': 72.0, 'FNumber': 4.5, 'ISOSpeedRatings': 400, 'ResolutionUnit': 2, 'ExifOffset': 254, 'Software': 'QuickTime 7.1', 'DateTime': '2006:05:26 22:11:28', 'HostComputer': 'Mac OS X 10.4.6'}}, 'simple_word_0': 1149349, 'simple_word_1': 19141372, 'simple_word_2': 35611974, 'simple_word_3': 13522824, 'simple_word_4': 14554561, 'simple_word_5': 20775288, 'simple_word_6': 8741790, 'simple_word_7': 29222986, 'simple_word_8': 42574094, 'simple_word_9': 43033590, 'simple_word_10': 30292136, 'simple_word_11': 42869015, 'simple_word_12': 28651883, 'simple_word_13': 9327311, 'simple_word_14': 43044829, 'simple_word_15': 4606550, 'simple_word_16': 28859730, 'simple_word_17': 41811554, 'simple_word_18': 356231, 'simple_word_19': 14348913, 'simple_word_20': 34836237, 'simple_word_21': 10635091, 'simple_word_22': 472572, 'simple_word_23': 32949350, 'simple_word_24': 42867945, 'simple_word_25': 42924247, 'simple_word_26': 18976596, 'simple_word_27': 36961350, 'simple_word_28': 6283, 'simple_word_29': 39384954, 'simple_word_30': 30218445, 'simple_word_31': 31943833, 'simple_word_32': 14331348, 'simple_word_33': 15923789, 'simple_word_34': 36960569, 'simple_word_35': 10791109, 'simple_word_36': 24270779, 'simple_word_37': 529665, 'simple_word_38': 31886468, 'simple_word_39': 14291982, 'simple_word_40': 15569495, 'simple_word_41': 38010323, 'simple_word_42': 5560, 'simple_word_43': 12917880, 'simple_word_44': 3675836, 'simple_word_45': 32050547, 'simple_word_46': 4371750, 'simple_word_47': 14525399, 'simple_word_48': 37439498, 'simple_word_49': 33113092, 'simple_word_50': 4389140, 'simple_word_51': 1766441, 'simple_word_52': 45227, 'simple_word_53': 33067926, 'simple_word_54': 28658279, 'simple_word_55': 5075783, 'simple_word_56': 22304026, 'simple_word_57': 2303288, 'simple_word_58': 42775572, 'simple_word_59': 14379520, 'simple_word_60': 35730072, 'simple_word_61': 15169637, 'simple_word_62': 21522523, 'timestamp': datetime.datetime(2020, 11, 7, 11, 42, 43, 149502)}, TypeError("Unable to serialize 5.64385986328125 (type: <class 'PIL.TiffImagePlugin.IFDRational'>)"))
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.8/site-packages/elasticsearch/serializer.py", line 120, in dumps
    py-image-dedup_1  |     return json.dumps(
    py-image-dedup_1  |   File "/usr/local/lib/python3.8/json/__init__.py", line 234, in dumps
    py-image-dedup_1  |     return cls(
    py-image-dedup_1  |   File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
    py-image-dedup_1  |     chunks = self.iterencode(o, _one_shot=True)
    py-image-dedup_1  |   File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
    py-image-dedup_1  |     return _iterencode(o, 0)
    py-image-dedup_1  |   File "/usr/local/lib/python3.8/site-packages/elasticsearch/serializer.py", line 106, in default
    py-image-dedup_1  |     raise TypeError("Unable to serialize %r (type: %s)" % (data, type(data)))
    py-image-dedup_1  | TypeError: Unable to serialize 5.64385986328125 (type: <class 'PIL.TiffImagePlugin.IFDRational'>)
    py-image-dedup_1  | 
    
    % docker images
    REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
    markusressel/py-image-dedup                     latest              b5a69df15036        44 hours ago        1.3GB
    docker.elastic.co/elasticsearch/elasticsearch   7.9.2               caa7a21ca06e        6 weeks ago         763MB
    

    Dupe of #21?

    bug 
    opened by jrc 6
  • Issue with connecting to Elasticsearch

    Issue with connecting to Elasticsearch

    Describe the bug I'm having issue with connections to my Elasticsearch instance.

    The following is an example of the behaviour I'm experiences. I'm not sure how to handle it or if it is an issue with Elasticsearch or py-image-dedup.

    issue
    (venv) fer@SynHome:~/py-image-dedup$ docker-compose up
    Creating network "py-image-dedup_docker-elk" with driver "bridge"
    Creating py-image-dedup_elasticsearch_1 ... done
    Creating py-image-dedup_py-image-dedup_1 ... done
    Attaching to py-image-dedup_elasticsearch_1, py-image-dedup_py-image-dedup_1
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:26:45,833Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "unable to install syscall filter: ",
    elasticsearch_1   | "stacktrace": ["java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:342) ~[elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:617) ~[elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:260) [elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:116) [elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) [elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.9.2.jar:7.9.2]",
    elasticsearch_1   | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.9.2.jar:7.9.2]"] }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:26:47,794Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "version[7.9.2], pid[8], build[default/docker/d34da0ea4a966c4e49417f2da2f244e3e97b4e6e/2020-09-23T00:45:33.626720Z], OS[Linux/3.10.108/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15/15+36]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:26:47,824Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:26:47,827Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-16856439618775791360, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,767Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [aggs-matrix-stats]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,771Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [analysis-common]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,786Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [constant-keyword]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,790Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [flattened]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,791Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [frozen-indices]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,799Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [ingest-common]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,803Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [ingest-geoip]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,805Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [ingest-user-agent]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,826Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [kibana]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,836Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [lang-expression]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,837Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [lang-mustache]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,838Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [lang-painless]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,839Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [mapper-extras]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,840Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [parent-join]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,841Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [percolator]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,851Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [rank-eval]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,865Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [reindex]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,868Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [repository-url]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,869Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [search-business-rules]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,872Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [searchable-snapshots]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,874Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [spatial]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,877Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [tasks]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,885Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [transform]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,894Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [transport-netty4]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,895Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [vectors]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,896Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [wildcard]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,897Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-analytics]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,897Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-async]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,898Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-async-search]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,899Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-autoscaling]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,899Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-ccr]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,900Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-core]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,901Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-data-streams]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,902Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-deprecation]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,903Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-enrich]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,903Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-eql]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,904Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-graph]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,905Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-identity-provider]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,906Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-ilm]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,906Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-logstash]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,907Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-ml]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,908Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-monitoring]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,909Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-ql]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,910Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-rollup]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,910Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-security]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,912Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-sql]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,913Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-stack]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,913Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-voting-only-node]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,914Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "loaded module [x-pack-watcher]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:04,916Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "no plugins loaded" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:05,408Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/cachedev_1)]], net usable_space [54.8gb], net total_space [888.9gb], types [btrfs]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:05,417Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:06,025Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "node name [elasticsearch], node ID [Rnp6dtg9QsmDG0INGrREvg], cluster name [docker-cluster]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:35,273Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "[controller/221] [Main.cc@114] controller (64 bit): Version 7.9.2 (Build 6a60f0cf2dd5a5) Copyright (c) 2020 Elasticsearch BV" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:38,174Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:45,386Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:46,063Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:50,394Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:53,830Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "initialized" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:53,834Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "starting ..." }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:54,729Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "publish_address {192.168.16.2:9300}, bound_addresses {0.0.0.0:9300}" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:56,713Z", "level": "WARN", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:56,715Z", "level": "WARN", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:56,721Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "cluster UUID [VrXadE3iRXC2MvquiVfAtQ]" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:57,084Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "elected-as-master ([1] nodes joined)[{elasticsearch}{Rnp6dtg9QsmDG0INGrREvg}{j5fgjBsMS7aLU13iOZiOOw}{192.168.16.2}{192.168.16.2:9300}{dilmrt}{ml.machine_memory=8287412224, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 42, version: 210, delta: master node changed {previous [], current [{elasticsearch}{Rnp6dtg9QsmDG0INGrREvg}{j5fgjBsMS7aLU13iOZiOOw}{192.168.16.2}{192.168.16.2:9300}{dilmrt}{ml.machine_memory=8287412224, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:57,475Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "master node changed {previous [], current [{elasticsearch}{Rnp6dtg9QsmDG0INGrREvg}{j5fgjBsMS7aLU13iOZiOOw}{192.168.16.2}{192.168.16.2:9300}{dilmrt}{ml.machine_memory=8287412224, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 42, version: 210, reason: Publication{term=42, version=210}" }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:57,728Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "publish_address {192.168.16.2:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:57,743Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "started", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:59,605Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "license [8a370969-b86b-4e90-8722-29ba38b2975b] mode [basic] - valid", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:59,612Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:27:59,689Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "recovered [1] indices into cluster_state", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
    elasticsearch_1   | {"type": "server", "timestamp": "2022-07-23T18:28:03,268Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[images][0]]]).", "cluster.uuid": "VrXadE3iRXC2MvquiVfAtQ", "node.id": "Rnp6dtg9QsmDG0INGrREvg"  }
    py-image-dedup_1  | DEBUG:py_image_dedup.util:Starting daemon...
    py-image-dedup_1  | Starting daemon...
    py-image-dedup_1  | DEBUG:py_image_dedup.util:Starting prometheus reporter...
    py-image-dedup_1  | Starting prometheus reporter...
    py-image-dedup_1  | DEBUG:py_image_dedup.util:==> DRY RUN! No files or folders will actually be deleted! <==
    py-image-dedup_1  | ==> DRY RUN! No files or folders will actually be deleted! <==
    py-image-dedup_1  | DEBUG:py_image_dedup.util:Phase 1/6: Cleaning up database ...
    py-image-dedup_1  | Phase 1/6: Cleaning up database ...
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 0% (1/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 1% (23/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 2% (46/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 3% (69/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 4% (92/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 5% (115/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 6% (138/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 7% (161/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 8% (184/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 9% (207/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 10% (229/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 11% (252/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 12% (275/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 13% (298/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 14% (321/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 15% (344/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 16% (367/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 17% (390/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 18% (413/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 19% (435/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 20% (458/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 21% (481/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 22% (504/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 23% (527/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 24% (550/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 25% (573/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 26% (596/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 27% (619/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 28% (641/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 29% (664/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 30% (687/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 31% (710/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 32% (733/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 33% (756/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 34% (779/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 35% (802/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 36% (825/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 37% (847/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 38% (870/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 39% (893/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 40% (916/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 41% (939/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 42% (962/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 43% (985/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 44% (1008/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 45% (1031/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 46% (1053/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 47% (1076/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 48% (1099/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 49% (1122/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 50% (1145/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 51% (1168/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 52% (1191/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 53% (1214/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 54% (1237/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 55% (1259/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 56% (1282/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 57% (1305/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 58% (1328/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 59% (1351/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 60% (1374/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 61% (1397/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 62% (1420/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 63% (1443/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 64% (1465/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 65% (1488/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 66% (1511/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 67% (1534/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 68% (1557/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 69% (1580/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 70% (1603/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 71% (1626/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 72% (1649/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 73% (1671/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 74% (1694/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 75% (1717/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 76% (1740/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 77% (1763/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 78% (1786/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 79% (1809/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 80% (1832/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 81% (1855/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 82% (1877/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 83% (1900/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 84% (1923/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 85% (1946/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 86% (1969/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 87% (1992/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 88% (2015/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 89% (2038/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 90% (2061/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 91% (2083/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 92% (2106/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 93% (2129/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 94% (2152/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 95% (2175/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 96% (2198/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 97% (2221/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 98% (2244/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 99% (2267/2289)
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Cleanup database: 100% (2289/2289)
    py-image-dedup_1  | DEBUG:py_image_dedup.util:Phase 2/6: Counting files ...
    py-image-dedup_1  | Phase 2/6: Counting files ...
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Counting files: 100% (1/1)
    py-image-dedup_1  | DEBUG:py_image_dedup.util:Phase 3/6: Analyzing files
    py-image-dedup_1  | Phase 3/6: Analyzing files
    py-image-dedup_1  | INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 0% (1/179325)
    py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.021s]
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    py-image-dedup_1  |     conn = connection.create_connection(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    py-image-dedup_1  |     raise err
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    py-image-dedup_1  |     sock.connect(sa)
    py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
    py-image-dedup_1  |
    py-image-dedup_1  | During handling of the above exception, another exception occurred:
    py-image-dedup_1  |
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    py-image-dedup_1  |     response = self.pool.urlopen(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    py-image-dedup_1  |     retries = retries.increment(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
    py-image-dedup_1  |     raise six.reraise(type(error), error, _stacktrace)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    py-image-dedup_1  |     raise value
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    py-image-dedup_1  |     httplib_response = self._make_request(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    py-image-dedup_1  |     conn.request(method, url, **httplib_request_kw)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
    py-image-dedup_1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
    py-image-dedup_1  |     self._send_request(method, url, body, headers, encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
    py-image-dedup_1  |     self.endheaders(body, encode_chunked=encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
    py-image-dedup_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
    py-image-dedup_1  |     self.send(msg)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 975, in send
    py-image-dedup_1  |     self.connect()
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    py-image-dedup_1  |     conn = self._new_conn()
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    py-image-dedup_1  |     raise NewConnectionError(
    py-image-dedup_1  | urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
    py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.005s]
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    py-image-dedup_1  |     conn = connection.create_connection(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    py-image-dedup_1  |     raise err
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    py-image-dedup_1  |     sock.connect(sa)
    py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
    py-image-dedup_1  |
    py-image-dedup_1  | During handling of the above exception, another exception occurred:
    py-image-dedup_1  |
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    py-image-dedup_1  |     response = self.pool.urlopen(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    py-image-dedup_1  |     retries = retries.increment(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
    py-image-dedup_1  |     raise six.reraise(type(error), error, _stacktrace)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    py-image-dedup_1  |     raise value
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    py-image-dedup_1  |     httplib_response = self._make_request(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    py-image-dedup_1  |     conn.request(method, url, **httplib_request_kw)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
    py-image-dedup_1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
    py-image-dedup_1  |     self._send_request(method, url, body, headers, encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
    py-image-dedup_1  |     self.endheaders(body, encode_chunked=encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
    py-image-dedup_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
    py-image-dedup_1  |     self.send(msg)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 975, in send
    py-image-dedup_1  |     self.connect()
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    py-image-dedup_1  |     conn = self._new_conn()
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    py-image-dedup_1  |     raise NewConnectionError(
    py-image-dedup_1  | urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
    py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.015s]
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    py-image-dedup_1  |     conn = connection.create_connection(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    py-image-dedup_1  |     raise err
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    py-image-dedup_1  |     sock.connect(sa)
    py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
    py-image-dedup_1  |
    py-image-dedup_1  | During handling of the above exception, another exception occurred:
    py-image-dedup_1  |
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    py-image-dedup_1  |     response = self.pool.urlopen(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    py-image-dedup_1  |     retries = retries.increment(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
    py-image-dedup_1  |     raise six.reraise(type(error), error, _stacktrace)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    py-image-dedup_1  |     raise value
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    py-image-dedup_1  |     httplib_response = self._make_request(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    py-image-dedup_1  |     conn.request(method, url, **httplib_request_kw)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
    py-image-dedup_1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
    py-image-dedup_1  |     self._send_request(method, url, body, headers, encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
    py-image-dedup_1  |     self.endheaders(body, encode_chunked=encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
    py-image-dedup_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
    py-image-dedup_1  |     self.send(msg)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 975, in send
    py-image-dedup_1  |     self.connect()
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    py-image-dedup_1  |     conn = self._new_conn()
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    py-image-dedup_1  |     raise NewConnectionError(
    py-image-dedup_1  | urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
    py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.006s]
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    py-image-dedup_1  |     conn = connection.create_connection(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    py-image-dedup_1  |     raise err
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    py-image-dedup_1  |     sock.connect(sa)
    py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
    py-image-dedup_1  |
    py-image-dedup_1  | During handling of the above exception, another exception occurred:
    py-image-dedup_1  |
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    py-image-dedup_1  |     response = self.pool.urlopen(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    py-image-dedup_1  |     retries = retries.increment(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
    py-image-dedup_1  |     raise six.reraise(type(error), error, _stacktrace)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    py-image-dedup_1  |     raise value
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    py-image-dedup_1  |     httplib_response = self._make_request(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    py-image-dedup_1  |     conn.request(method, url, **httplib_request_kw)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
    py-image-dedup_1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
    py-image-dedup_1  |     self._send_request(method, url, body, headers, encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
    py-image-dedup_1  |     self.endheaders(body, encode_chunked=encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
    py-image-dedup_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
    py-image-dedup_1  |     self.send(msg)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 975, in send
    py-image-dedup_1  |     self.connect()
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    py-image-dedup_1  |     conn = self._new_conn()
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    py-image-dedup_1  |     raise NewConnectionError(
    py-image-dedup_1  | urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
    py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.004s]
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    py-image-dedup_1  |     conn = connection.create_connection(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    py-image-dedup_1  |     raise err
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    py-image-dedup_1  |     sock.connect(sa)
    py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
    py-image-dedup_1  |
    py-image-dedup_1  | During handling of the above exception, another exception occurred:
    py-image-dedup_1  |
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    py-image-dedup_1  |     response = self.pool.urlopen(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    py-image-dedup_1  |     retries = retries.increment(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 525, in increment
    py-image-dedup_1  |     raise six.reraise(type(error), error, _stacktrace)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    py-image-dedup_1  |     raise value
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    py-image-dedup_1  |     httplib_response = self._make_request(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    py-image-dedup_1  |     conn.request(method, url, **httplib_request_kw)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
    py-image-dedup_1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
    py-image-dedup_1  |     self._send_request(method, url, body, headers, encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
    py-image-dedup_1  |     self.endheaders(body, encode_chunked=encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
    py-image-dedup_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
    py-image-dedup_1  |     self.send(msg)
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/http/client.py", line 975, in send
    py-image-dedup_1  |     self.connect()
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    py-image-dedup_1  |     conn = self._new_conn()
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    py-image-dedup_1  |     raise NewConnectionError(
    py-image-dedup_1  | urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
    py-image-dedup_1  | WARNING:elasticsearch:POST http://elasticsearch:9200/images/_doc?refresh=false [status:N/A request:0.003s]
    py-image-dedup_1  | Traceback (most recent call last):
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    py-image-dedup_1  |     conn = connection.create_connection(
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    py-image-dedup_1  |     raise err
    py-image-dedup_1  |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    py-image-dedup_1  |     sock.connect(sa)
    py-image-dedup_1  | ConnectionRefusedError: [Errno 111] Connection refused
    py-image-dedup_1  |
    

    To Reproduce Running with the following

    docker-compose.yaml

    version: '3.7'
    
    services:
      elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
        ports:
          - "9200:9200"
          - "9300:9300"
        environment:
          - discovery.type=single-node
          - cluster.routing.allocation.disk.threshold_enabled=true 
          - cluster.routing.allocation.disk.watermark.flood_stage=200mb
          - cluster.routing.allocation.disk.watermark.low=500mb 
          - cluster.routing.allocation.disk.watermark.high=300mb
          - node.name=elasticsearch
          - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
        networks:
          - docker-elk
        restart: on-failure
        volumes:
          - ./esdata1:/usr/share/elasticsearch/data
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9200"]
          interval: 30s
          timeout: 30s
          retries: 10
      py-image-dedup:
        image: markusressel/py-image-dedup:latest
        environment:
          - PUID=0
          - PGID=100
          # change configuration to your liking
          - PY_IMAGE_DEDUP_DRY_RUN=True
          - PY_IMAGE_DEDUP_ANALYSIS_SOURCE_DIRECTORIES=/mnt/source/
          - PY_IMAGE_DEDUP_ANALYSIS_RECURSIVE=True
          - PY_IMAGE_DEDUP_ANALYSIS_ACROSS_DIRS=True
          - PY_IMAGE_DEDUP_ANALYSIS_FILE_EXTENSIONS=.png,.jpg,.jpeg
          - PY_IMAGE_DEDUP_ANALYSIS_THREADS=4
          - PY_IMAGE_DEDUP_ANALYSIS_USE_EXIF_DATA=True
          - PY_IMAGE_DEDUP_DEDUPLICATION_DUPLICATES_TARGET_DIRECTORY=/mnt/duplicates/
          - PY_IMAGE_DEDUP_ELASTICSEARCH_HOST=elasticsearch
          - PY_IMAGE_DEDUP_ELASTICSEARCH_PORT=9200
          - PY_IMAGE_DEDUP_ELASTICSEARCH_INDEX=images
          - PY_IMAGE_DEDUP_ELASTICSEARCH_AUTO_CREATE_INDEX=True
          - PY_IMAGE_DEDUP_ELASTICSEARCH_MAX_DISTANCE=0.1
          - PY_IMAGE_DEDUP_REMOVE_EMPTY_FOLDERS=False
          - PY_IMAGE_DEDUP_STATS_ENABLED=True
          - PY_IMAGE_DEDUP_STATS_PORT=8000
        volumes:
          # optionally mount a YAML configuration file
          # into /app/py_image_dedup.yaml instead of using environment:
          # - /mnt/data3/py_image_dedup.yaml:/app/py_image_dedup.yaml
          # change this to your local source directory:
          - /volume2/pictures/gto:/mnt/source:rw
          # change this to your local duplicates directory:
          - /volume2/Media/photo-dupes:/mnt/duplicates/
        links:
          - elasticsearch
        networks:
         - docker-elk
        ports:
          - "8000:8000"
        depends_on:
          elasticsearch:
              condition: service_healthy
        restart: on-failure
    networks:
      docker-elk:
        driver: bridge
    
    
    question 
    opened by fredrike 5
  • random crash

    random crash

    Describe the bug Tried to run the program on my image folder, but at a certain point, it consistently crashes, without any usable error log.

    To Reproduce Steps to reproduce the behavior:

    1. Set up elastic, py-image-dedup using docker-compose
    2. run on image folder
    3. see it analyze files, and at a certain percentage, always crash

    Expected behavior The program should finish analyzing my files, and report back the duplicates.

    Screenshots

    INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 63% (2037/3233)
    INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 64% (2070/3233)
    INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 65% (2102/3233)
    INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 66% (2134/3233)
    INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 67% (2167/3233)
    Killed
    <docker automatically restarts>
    DEBUG:py_image_dedup.util:Starting daemon...
    Starting daemon...
    DEBUG:py_image_dedup.util:Starting prometheus reporter...
    Starting prometheus reporter...
    DEBUG:py_image_dedup.util:==> DRY RUN! No files or folders will actually be deleted! <==
    DEBUG:py_image_dedup.util:Phase 1/6: Cleaning up database ...
    ==> DRY RUN! No files or folders will actually be deleted! <==
    Phase 1/6: Cleaning up database ...
    

    Desktop (please complete the following information):

    • OS: Ubuntu 20.04, docker
    bug 
    opened by mortee 5
  • Installing via setup.py does not use forked image-match version

    Installing via setup.py does not use forked image-match version

    Describe the bug Image-Match dependency on 'elasticsearch<2.4,>=2.3'

    To Reproduce Steps to reproduce the behavior:

    1. Clone Git
    2. Setup virtual environment
    3. Run python setup.py install
    4. pkg_resources.ContextualVersionConflict: (elasticsearch 7.10.0 (/Users/michael/Documents/Python_Workspace/py-image-dedup/myenv/lib/python3.8/site-packages/elasticsearch-7.10.0-py3.8.egg), Requirement.parse('elasticsearch<2.4,>=2.3'), {'image-match'})

    Expected behavior Conflict with Elasticsearch 7.10.0 on py-image-dedup-end and elasticsearch<2.4,>=2.3 on image-match-end

    Screenshots image

    Desktop (please complete the following information):

    • OS: Mac OS

    Additional context Add any other context about the problem here.

    bug 
    opened by stormriderDE 4
  • Bump tqdm from 4.49.0 to 4.50.0

    Bump tqdm from 4.49.0 to 4.50.0

    Bumps tqdm from 4.49.0 to 4.50.0.

    Release notes

    Sourced from tqdm's releases.

    tqdm v4.50.0 stable

    • add bar colour option (#1040 <- #450)
      • for notebook as well as std/CLI
      • plain text as well as hex (e.g. 'green' or '#00ff00')
    • notebook improvements (#1032 <- #1031)
      • split bar description into separate widget
      • improve reset() (handle width and colour)
    • fix exceptions when file is closed (#1041, #1036 <- #1033)
    • add & update tests
    • minor documentation updates
    • move and configure issue/pr templates
    Commits
    • 00dabd6 bump version, merge branch 'devel'
    • 53a84ea closed file: use miniters=inf rather than hacky disable=1
    • 098745d test and fix disable when writing to closed files
    • ea2367e Merge branch 'colour' into devel
    • 06fb6ec add tests
    • c89b0fe fix notebook
    • 8baa057 support hex colours
    • afe9ad8 support CLI colours
    • dd5739a notebook: add colour
    • e835860 Merge branch 'notebook-nototal' into devel
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
    • @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

    Additionally, you can set the following in your Dependabot dashboard:

    • Update frequency (including time of day and day of week)
    • Pull request limits (per update run and/or open at any time)
    • Out-of-range updates (receive only lockfile updates, if desired)
    • Security updates (receive only security updates, if desired)
    dependencies 
    opened by dependabot-preview[bot] 3
  • container_app_conf

    container_app_conf

    Running the example.py results in the following error: ModuleNotFoundError: No module named 'container_app_conf

    To replicate the error: create a python environment install py-image-dedup via pip install py-image-dedup python3 example.py

    Any idea why this could happen? pip list shows py-imaged-dedup as installed but the installation gave some error messages. I can post them here if needed

    opened by byagiz 2
  • Hitting lots of errors from lib folder

    Hitting lots of errors from lib folder

    everytime I run the image dedup it hits an error but they all seems to becoming from... ..\python\python36\lib[some package] some examples are: python\python36\lib\site-packages\urllib3\util\retry.py python\python36\lib\http\client.py python\python36\lib\site-packages\urllib3\connectionpool.py ...

    idk if you I need to copy and paste the massive errors that are occurring but idk what to do anymore. Either from deleting everything in the lib folder or what to do anymore.

    help wanted 
    opened by HosAkh 2
  • Bump elasticsearch from 6.8.2 to 7.17.7

    Bump elasticsearch from 6.8.2 to 7.17.7

    Bumps elasticsearch from 6.8.2 to 7.17.7.

    Release notes

    Sourced from elasticsearch's releases.

    7.17.7

    • Client is compatible with Elasticsearch 7.17.7

    7.17.6

    • Client is compatible with Elasticsearch 7.17.6

    7.17.5

    • Client is compatible with Elasticsearch 7.17.5

    7.17.4

    Client

    • Client is compatible with Elasticsearch 7.17.4

    7.17.3

    Client

    • Client is compatible with Elasticsearch 7.17.3

    7.17.2

    Client

    • Client is compatible with Elasticsearch 7.17.2

    7.17.1

    Helpers

    Transport

    7.17.0

    Machine Learning

    • Added the ml.get_model_snapshot_upgrade_stats API
    • Added the body parameter to the ml.forecast and ml.open_job APIs

    Transform

    • Added the timeout parameter to the transform.delete_transform, transform.preview_transform, transform.put_transform, transform.update_transform, and transform.upgrade_transform APIs

    7.16.3

    Client

    • API is compatible with Elasticsearch 7.16.3

    7.16.2

    Client

    ... (truncated)

    Commits
    • 9221e2d Bumps to version 7.17.7
    • 72f102d Update assemble task - SNAPSHOT
    • d411230 non-root user to build artifacts inside the container
    • 576fae9 Bumps to version 7.17.6
    • 7b5389d Add release notes for 7.173 and 7.17.4
    • a0ee925 Bumps to version 7.17.5
    • 2f5e8e2 Bumps to version 7.17.4
    • 12ab98a [7.17] Use major.minor-SNAPSHOT for STACK_VERSION
    • 16d57d2 [7.17] Implement '.ci/make.sh bump' to set version
    • d2432ef Pin click to version compatible with black v21.12b0
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • Bump elasticsearch from 6.8.2 to 7.17.8

    Bump elasticsearch from 6.8.2 to 7.17.8

    Bumps elasticsearch from 6.8.2 to 7.17.8.

    Release notes

    Sourced from elasticsearch's releases.

    7.17.8

    • Removed the work-around for a bug in aiohttp when sending HEAD requests. This change was backported from v8.x of elastic-transport.

    7.17.7

    • Client is compatible with Elasticsearch 7.17.7

    7.17.6

    • Client is compatible with Elasticsearch 7.17.6

    7.17.5

    • Client is compatible with Elasticsearch 7.17.5

    7.17.4

    Client

    • Client is compatible with Elasticsearch 7.17.4

    7.17.3

    Client

    • Client is compatible with Elasticsearch 7.17.3

    7.17.2

    Client

    • Client is compatible with Elasticsearch 7.17.2

    7.17.1

    Helpers

    Transport

    7.17.0

    Machine Learning

    • Added the ml.get_model_snapshot_upgrade_stats API
    • Added the body parameter to the ml.forecast and ml.open_job APIs

    Transform

    • Added the timeout parameter to the transform.delete_transform, transform.preview_transform, transform.put_transform, transform.update_transform, and transform.upgrade_transform APIs

    7.16.3

    Client

    • API is compatible with Elasticsearch 7.16.3

    ... (truncated)

    Commits
    • 9b0d43b Backport aiohttp conditional HEAD bug workaround
    • ebad38c Add release notes for 7.17.5, 7.17.6 and 7.17.7
    • 673a720 Bumps 7.17 to 7.17.8
    • 9221e2d Bumps to version 7.17.7
    • 72f102d Update assemble task - SNAPSHOT
    • d411230 non-root user to build artifacts inside the container
    • 576fae9 Bumps to version 7.17.6
    • 7b5389d Add release notes for 7.173 and 7.17.4
    • a0ee925 Bumps to version 7.17.5
    • 2f5e8e2 Bumps to version 7.17.4
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
  • ProcessPoolExecutor

    ProcessPoolExecutor

    Is your feature request related to a problem? Please describe. Due to the GIL, multithreading isn't really sufficient for fully utilizing a multicore CPU.

    Describe the solution you'd like Integrate ProcessPoolExecutor to work around the GIL limitation.

    enhancement 
    opened by markusressel 0
  • Watch file changes in newly created folders

    Watch file changes in newly created folders

    Is your feature request related to a problem? Please describe. Currently only folders that exist at the time of FileWatcher setup, therefore folders that are created while the FileWatcher is already running are not watched.

    Describe the solution you'd like Add new observers on the fly when a new folder has been created.

    enhancement 
    opened by markusressel 0
  • Logging & Console output

    Logging & Console output

    To run py-image-dedup as a daemon, logging should be used more extensively. Almost all output is currently done through click.echo() and even with logging the cli should support displaying the progress in an interactive way.

    enhancement 
    opened by markusressel 0
Releases(v2.0.1)
  • v2.0.1(Jul 10, 2022)

  • v2.0.0(Oct 7, 2020)

    This is a breaking release, since most of the configuration has been moved to a config file.

    • added "analyse" cli command
    • markusressel/#2_move_instead_of_delete
    • added preference to enable/disable auto index creation
    • Feature/#5 file config
      • moved most of the configuration parameters to the config
    • added option to disable removal of empty folders
    • #1 remove folders only if empty due to deduplication
    • better error handling while threading
    • better summary output
    • automatically detect elasticsearch version
    • support both v6 and v7 (needs patches in image-match too)
    • elasticsearch fix for new hit count format
    • Feature/#3 run as service
    • Feature/#4 docker
    • feature/#12_statistics
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Mar 8, 2019)

Owner
Markus Ressel
❤️ Open Source 📱 Mobile App Development 💡 Home Automation 🔐 Typed Languages 🏍 Motorcycling
Markus Ressel
A terminal utility to sort image files based on their characteristics.

About A terminal utility to sort image files based on their characteristics. Motivation This program was developed after I've realized that I had too

José Ferreira 1 Dec 10, 2022
Interactive Python interpreter for executing commands within Node.js

Python Interactive Interactive Python interpreter for executing commands within Node.js. This module provides a means of using the Python interactive

Louis Lefevre 2 Sep 21, 2022
Bad Apple printed out on the console with Python!

Bad Apple printed out on the console with Python!

CalvinLoke 186 Dec 1, 2022
Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Phil Wang 4.4k Jan 9, 2023
MasterDuel Image Recognition Translation Command Line Tool

MasterDuelTranslate(Use Ygo Card DataBase,belong win32 window shot & image match)

PatchouliTC 77 Dec 1, 2022
Shortcut-Maker - It is a tool that can be set to run any tool with a single command

Shortcut-Maker It is a tool that can be set to run any tool with a single command Coded by Dave Smith(Owner of Sl Cyber Warriors) Command list ?? pkg

Dave Smith 10 Sep 14, 2022
A simple CLI based any Download Tool, that find files and let you stream or download thorugh WebTorrent CLI or Aria or any command tool

Privateer A simple CLI based any Download Tool, that find files and let you stream or download thorugh WebTorrent CLI or Aria or any command tool How

Shreyash Chavan 2 Apr 4, 2022
A command line tool (and Python library) for archiving Twitter JSON

A command line tool (and Python library) for archiving Twitter JSON

Documenting the Now 1.3k Dec 28, 2022
Python package with library and CLI tool for analyzing SeaFlow data

Seaflowpy A Python package for SeaFlow flow cytometer data. Table of Contents Install Read EVT/OPP/VCT Files Command-line Interface Configuration Inte

SeaFlow@UW 3 Nov 3, 2021
A webmining CLI tool & library for python.

minet is a webmining command line tool & library for python (>= 3.6) that can be used to collect and extract data from a large variety of web sources

médialab Sciences Po 165 Dec 17, 2022
Python library and command line tool for interacting with Bugzilla

python-bugzilla This package provides two bits: bugzilla python module for talking to a Bugzilla instance over XMLRPC or REST /usr/bin/bugzilla comman

Python Bugzilla Project 112 Nov 5, 2022
Python library & console tool for controlling Xiaomi smart appliances

python-miio This library (and its accompanying cli tool) can be used to interface with devices using Xiaomi's miIO and MIoT protocols. Getting started

Teemu R. 2.4k Jan 2, 2023
cmdpxl: a totally practical command-line image editor

cmdpxl: a totally practical command-line image editor

Jieruei Chang 476 Jan 7, 2023
Limit your docker image size with a simple CLI command. Perfect to be used inside your CI process.

docker-image-size-limit Limit your docker image size with a simple CLI command. Perfect to be used inside your CI process. Read the announcing post. I

wemake.services 102 Dec 14, 2022
Unconventional ways to save an Image

Unexpected Image Saves Unconventional ways to save an image ?? Have you ever been bored by the same old .png, .jpg, .jpeg, .gif and all other image ex

Eric Mendes 15 Nov 6, 2022
Dead simple CLI tool to try Python packages - It's never been easier! :package:

try - It's never been easier to try Python packages try is an easy-to-use cli tool to try out Python packages. Features Install specific package versi

Timo Furrer 659 Dec 28, 2022
Python commandline tool for remembering linux/terminal commands

ehh Remember linux commands Commandline tool for remembering linux/terminal commands. It stores your favorite commands in ~/ehh.json in your homedir a

null 56 Nov 10, 2022
Command-line tool for looking up colors and palettes.

Colorpedia Colorpedia is a command-line tool for looking up colors, shades and palettes. Supported color models: HEX, RGB, HSL, HSV, CMYK. Requirement

Joohwan Oh 282 Dec 27, 2022