Describe the bug There is a file in the folder which should be pr

Please update to the new v3, where your runs fine: <div class="snippet-clip

Thanks for the hints so far <a class="user-mention notranslate" data-hovercard-type="u

The one liner version works fine <div class="snippet-clipboard-content notranslate

Skipped files / Silent Error / Lack of verbose run,about tfeldmann/organize

Comments (7)

kisst commented on August 26, 2024

Just to test I upgraded to v3.2.3 and the problem still existing there, I would be happy to try to debug it, but at this point unsure how, any pointers would be appreciated.

from organize.

tfeldmann commented on August 26, 2024

Please update to the new v3, where your script runs fine:

╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ SIMULATION                                                                                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Config: ".configs/issue375-python.yml"

⚙ Rule #0: New images ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  .venv/docx-template/docProps/thumbnail.jpeg
    - (echo) Based on Filesystem ctime .venv/docx-template/docProps/thumbnail.jpeg should go to 2024/02/16/
    - (move) Move to /main/pictures/2024/02/16/thumbnail.jpeg
  tests/resources/images-with-exif/1.jpg
    - (echo) Based on Exif.Image.DateTime tests/resources/images-with-exif/1.jpg should go to 2018/09/29/
    - (move) Move to /main/pictures/2018/09/29/1.jpg
  tests/resources/images-with-exif/2.jpg
    - (echo) Based on Exif.Image.DateTime tests/resources/images-with-exif/2.jpg should go to 2018/04/29/
    - (move) Move to /main/pictures/2018/04/29/2.jpg
  tests/resources/images-with-exif/3.jpg
    - (echo) Based on Exif.Image.DateTime tests/resources/images-with-exif/3.jpg should go to 2017/08/12/
    - (move) Move to /main/pictures/2017/08/12/3.jpg
  tests/resources/images-with-exif/4.jpg
    - (echo) Based on Exif.Image.DateTime tests/resources/images-with-exif/4.jpg should go to 2018/02/22/
    - (move) Move to /main/pictures/2018/02/22/4.jpg
  tests/resources/images-with-exif/5.jpg
    - (echo) Based on Exif.Image.DateTime tests/resources/images-with-exif/5.jpg should go to 2015/07/08/
    - (move) Move to /main/pictures/2015/07/08/5.jpg

success 6 / fail 0
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ SIMULATION                                                                                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

In v3 you will also see any exceptions raised in your custom python filter.

from organize.

tfeldmann commented on August 26, 2024

Please note the new and very much improved exif filter in v3 where you can use exiftool for parsing your images by setting an environment variable: ORGANIZE_EXIFTOOL_PATH="exiftool". Also in this version the dates and times are automatically parsed and available as datetime.datetime objects.

from organize.

kisst commented on August 26, 2024

I tested the new exif tool filter as an option but unfortunately I couldn't find a way to reproduce the desired logic.

As per the example config above I am trying to date the images , and the method for doing it so doesn't only depends on exif data, if it exist it is the primary but if it doesn't then I check the file name too and if neither of them give me a date just use file create date.

I am still considering the add more options for example OCR from the bottom corner the date, as old photos had the date burnt in for a few years between full manual and digital cameras. I scan quite a few of those too.

I considered to just package it up into a python library and maybe add video support too , however I am way more a script kiddie rather than a programmer and writing it just for the sake of it makes no sense , unless it is used somewhere other than my own organize config .

However back to the issue , I still see skipped files and I have no means to debug the issue , any suggestions ?

from organize.

kisst commented on August 26, 2024

Thanks for the hints so far @tfeldmann I narrowed down the error to the exif filter, here is my test setup

The folder has only a test file and a test config

/tmp/issue375$ ls -1
1ae75a1e661e44eda7e1f920d8e5e82c.jpg
test-config.yaml

The config is validated

/tmp/issue375$ organize check test-config.yaml 
No problems found in "test-config.yaml".

Setting up and running the test config

/tmp/issue375$ ORGANIZE_EXIFTOOL_PATH="exiftool"
/tmp/issue375$ organize run test-config.yaml 
Config: "test-config.yaml"

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 1 / fail 0

running exiftool manually on the file in Q

/tmp/issue375$ exiftool 1ae75a1e661e44eda7e1f920d8e5e82c.jpg 
ExifTool Version Number         : 12.40
File Name                       : 1ae75a1e661e44eda7e1f920d8e5e82c.jpg
Directory                       : .
File Size                       : 111 KiB
File Modification Date/Time     : 2024:04:13 19:11:18+02:00
File Access Date/Time           : 2024:04:13 19:24:28+02:00
File Inode Change Date/Time     : 2024:04:13 19:11:18+02:00
File Permissions                : -rw-r--r--
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
JFIF Version                    : 1.01
Resolution Unit                 : None
X Resolution                    : 1
Y Resolution                    : 1
Image Width                     : 1080
Image Height                    : 1080
Encoding Process                : Progressive DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:2:0 (2 2)
Image Size                      : 1080x1080
Megapixels                      : 1.2

Also tried with full path for exiftool, same result

/tmp/issue375$ command -v exiftool 
/usr/bin/exiftool
/tmp/issue375$ ORGANIZE_EXIFTOOL_PATH="/usr/bin/exiftool"
/tmp/issue375$ organize run test-config.yaml 
Config: "test-config.yaml"

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 1 / fail 0

I am on the latest release from organize and 12.40 on exiftool

/tmp/issue375$ organize --version
organize v3.2.3

/tmp/issue375$ exiftool -ver
12.40

And the config itself

---
rules:
  - name: "image with exif filter"
    subfolders: true
    locations:
      - "."
    filters:
      - extension:
          - jpg
      - exif
    actions:
      - echo: "This doesn't work"
  - name: "image without exif filter"
    subfolders: true
    locations:
      - "."
    filters:
      - extension:
          - jpg
    actions:
      - echo: "This works"

or in debug format

Config(
    rules=[
        Rule(
            name='image with exif filter',
            enabled=True,
            targets='files',
            locations=[
                Location(
                    path=[
                        '.'
                    ],
                    min_depth=0,
                    max_depth='inherit',
                    search='breadth',
                    exclude_files=set(),
                    exclude_dirs=set(),
                    system_exclude_files={
                        '.localized',
                        '~$*',
                        'desktop.ini',
                        '.DS_Store',
                        'thumbs.db'
                    },
                    system_exclude_dirs={
                        '.git',
                        '.svn'
                    },
                    filter=None,
                    filter_dirs=None,
                    ignore_errors=False
                )
            ],
            subfolders=True,
            tags=set(),
            filters=[
                Extension(
                    extensions={
                        'jpg'
                    }
                ),
                Exif(
                    filter_tags={},
                    lowercase_keys=True
                )
            ],
            filter_mode='all',
            actions=[
                Echo(
                    msg="This doesn't work"
                )
            ]
        ),
        Rule(
            name='image without exif filter',
            enabled=True,
            targets='files',
            locations=[
                Location(
                    path=[
                        '.'
                    ],
                    min_depth=0,
                    max_depth='inherit',
                    search='breadth',
                    exclude_files=set(),
                    exclude_dirs=set(),
                    system_exclude_files={
                        '.localized',
                        '~$*',
                        'desktop.ini',
                        '.DS_Store',
                        'thumbs.db'
                    },
                    system_exclude_dirs={
                        '.git',
                        '.svn'
                    },
                    filter=None,
                    filter_dirs=None,
                    ignore_errors=False
                )
            ],
            subfolders=True,
            tags=set(),
            filters=[
                Extension(
                    extensions={
                        'jpg'
                    }
                )
            ],
            filter_mode='all',
            actions=[
                Echo(
                    msg='This works'
                )
            ]
        )
    ],
    _config_path=PosixPath('test-config.yaml')
)

from organize.

tfeldmann commented on August 26, 2024

What is the output of echo $ORGANIZE_EXIFTOOL_PATH? I ask because your environment variable might not be set correctly. Can you try: ORGANIZE_EXIFTOOL_PATH='exiftool' organize run test-config.yaml (in a single command)?

from organize.

kisst commented on August 26, 2024

The one liner version works fine

/tmp/issue375$ ORGANIZE_EXIFTOOL_PATH='exiftool' organize run test-config.yaml
Config: "test-config.yaml"

⚙ Rule #0: image with exif filter ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This doesn't work

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 2 / fail 0

but simple variable set does not work

/tmp/issue375$ ORGANIZE_EXIFTOOL_PATH="/usr/bin/exiftool"
/tmp/issue375$ echo $ORGANIZE_EXIFTOOL_PATH
/usr/bin/exiftool
organize run test-config.yaml
Config: "test-config.yaml"

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 1 / fail 0

and

/tmp/issue375$ ORGANIZE_EXIFTOOL_PATH="exiftool"
/tmp/issue375$ echo $ORGANIZE_EXIFTOOL_PATH
exiftool
/tmp/issue375$ organize run test-config.yaml
Config: "test-config.yaml"

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 1 / fail 0

The good news is if I export it then it works

/tmp/issue375$ export ORGANIZE_EXIFTOOL_PATH="exiftool"
/tmp/issue375$ organize run test-config.yaml
Config: "test-config.yaml"

⚙ Rule #0: image with exif filter ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This doesn't work

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 2 / fail 0

Note, if it's just set as variable then it does not show up in env

So the "fix" can, a simple documentation update to replace

ORGANIZE_EXIFTOOL_PATH="exiftool"

export ORGANIZE_EXIFTOOL_PATH="exiftool"

or alternatively the method for detecting the path could be more try to run in as a shell something like command -v exiftool, and set it based on that.

As far as my issue goes I actually do not use the exif data from organize at all, ( as described above the complex fallback logic ) so removing the filter fixed it for me, but also added the variable to .bashrc just in case I will use it at some point.

I could not find the relevant doc string in the repo, if you point me at it happy to send a PR on that, or if you consider the detection logic, where the os.environ.get default fallback value is from command -v instead of nothing, happy to attempt that path too.

from organize.

Skipped files / Silent Error / Lack of verbose run about organize HOT 7 OPEN

Comments (7)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent