Describe the bug
Following the example in the documentation, I struggle with making an Index.
Hopefully it's an rookie mistake, but I have checked file permissions and everything seems to be ok (possible to manipulate and load files in other code)

To Reproduce
Steps to reproduce the behavior:
Make a new project, and have a folder inside with .jpg images.

Try to run:

import os

from pupyl.search import PupylImageSearch

# This will create pupyl database on the current directory, inside a
# directory called 'pupyl'
PUPYL = PupylImageSearch(data_dir=os.path.join(os.curdir, 'pupyl'))

PUPYL.index('Z:\\pokemon')

Expected behavior
That the images inside the folder are indexed.
The library are able to create the folder 'pupyl'

Desktop (please complete the following information):

• Windows 11 Python 3.11

--

Thanks a lot for any help :)

• Create methods for file and url discovery
• Must iterate over all levels of a directory, finding suitable and valid images to read
• The iteration must respond visually, letting the user know how much files will be read and how many files aren't read yet.
• User also can have a text file (compressed or not) read, with urls or local files inside it.
• Images reference file type should be recognized previously the reading process, hence a file type (or compression) discovery must be implemented.

Hi - Is it possible to export the original images as opposed to the shrunken copies inside the pupyl database?

" It would be great to be able to batch interrogate the database using a Bash script (within the virtual environment) and the CLI to:

eg input (eg -in flag) search image to CLI and copy matches (within a defined match value) to a (-o flag) named folder

Originally posted by @rleaver152 in #122
"

This bash code does it the hard way

reverse_search_images_my_pupyl_search_and_replace_with_originals() {
echo "start this above pupyl source files directory"
thisDirectory="$PWD"
export_baseDirectory="$PUPYL_EXPORT_DIRECTORY"

if [[ -z "$1" ]]; then
   echo "need directory  to search"
   else
        target_Directory="$(basename "$1")"
#       echo "target directory=$target_Directory"
for directoryFile in "$PUPYL_EXPORT_DIRECTORY"/"$target_Directory"/* ; do \
myVal=$(gfind "$directoryFile" -mindepth 1 -maxdepth 1 -type f  -exec echo "{}" \; )
mySearch=$(basename "$myVal" )
gfind "$target_Directory" -type f -name "$mySearch" -exec gcp "{}" "$directoryFile"/. \;
done
fi


}
export -f reverse_search_images_my_pupyl_search_and_replace_with_originals

Discussed in #198

The confidence between query and database images must be returned. As explained in the referred discussion #198, the unicity check is already implemented, but with a fixed (internal) threshold.

• When indexing new images, a configuration file should be created to help loading features used in the index process.
• The configuration file must be very simple, probably a json file with some simple keys.
• Better with it stay side by side with the index file (maybe in the same directory).

• Use annoy or faiss to perform a fast approximate nearest neighbours.
• Try new distance metrics (angular, hamming, etc)
• Think about better compressions (or use a lzma compressed index, for example)
• Avoid empty index elements (like numpy.zeros)

• Discover the minimum viable byte chunk to discover the file type
• Unify file subtypes before separated.

The majority of the error catching on pupyl throws an user error, but without clear explanation of the meanders that generated the error.

For instance

which should be something like that:

...
 raise FileIsNotImage("This file wasn't recognized as a valid image file. Please retry.")

Please, add descriptive error messages that will help user to debug their applications.

Resume indexing
pupyl must be more resilient during the indexation process, hence if any issue happens on the process, it should resume where it left off.

Describe the solution you'd like
The method PupylImageSearch.index() must verify an already created database, check the last indexed image on the sequence and continue from there.

• OpenCV is not needed anymore;
• All operations used can be made with numpy and tensorflow.

• Use https://pepy.tech/project/gensim to generate badge;
• Do this after publication on pypi.

• The current indexing process is not fast enough, taking hours to index few images;
• Investigate what's the bottleneck.

i was reading about pupyl and i noticed it said: Pupyl is a really fast image search library which you can index your own (millions of) images and find similar images in milliseconds. i have around 160 million 32x32 images and was wondering if it's possible to insert them in a very fast way. i tried using the normal method provided in the docs, but it was very slow. around 3 images/s.

• Indexer instances must behave like a python list, for instance:

len(Indexer)
Indexer[:12]
Indexer.append(value)
Indexer.remove(value)
Indexer.pop(index)
Indexer.insert(index, value)

still unable to get the 'export original id' function working. on invoking "--keep_ids" it still results in numerically indexed images with no original ids

Every time the main pupyl class is loaded with a feature extraction model, it takes seconds to respond

• This is due to the tensorflow strategy to load the model into memory (RAM ou VRAM);
• The module must consider the best (and fastest) way to load the model, to avoid excessive memory consumption or slower loading.

Describe the bug

Pupyl fails to index images. I have tried with different sets of images, but it always fails. After importing and scanning all images, a FileIsNotImage exception is raised.

To Reproduce

import os
from pathlib import Path
from pupyl.search import PupylImageSearch

p = PupylImageSearch(os.path.join(Path.home(), 'pupyl'))
p.index('https://github.com/policratus/pupyl/raw/main/samples/images.tar.xz')

Running the above code gives the following output:

...🕔 Scanning images: 594 items.
FileNotFoundError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "c:\Code\dragonvale\dragonvale_env\lib\site-packages\pupyl\search.py", line 285, in index
   futures = {
 File "c:\Code\dragonvale\dragonvale_env\lib\site-packages\pupyl\search.py", line 291, in <dictcomp>
   self.image_database.load_image_metadata(
 File "c:\Code\dragonvale\dragonvale_env\lib\site-packages\pupyl\storage\database.py", line 274, in load_image_metadata
   raise IndexError(
IndexError: Index not found: 106

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
   return _run_code(code, main_globals, None,
 File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
   exec(code, run_globals)
 File "c:\Users\amroo\.vscode\extensions\ms-python.python-2022.12.0\pythonFiles\lib\python\debugpy\__main__.py", line 39, in <module>
   cli.main()
 File "c:\Users\amroo\.vscode\extensions\ms-python.python-2022.12.0\pythonFiles\lib\python\debugpy/..\debugpy\server\cli.py", line 430, in main
   run()
 File "c:\Users\amroo\.vscode\extensions\ms-python.python-2022.12.0\pythonFiles\lib\python\debugpy/..\debugpy\server\cli.py", line 284, in run_file
   runpy.run_path(target, run_name="__main__")
 File "c:\Users\amroo\.vscode\extensions\ms-python.python-2022.12.0\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 321, in run_path
   return _run_module_code(code, init_globals, run_name,
 File "c:\Users\amroo\.vscode\extensions\ms-python.python-2022.12.0\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 135, in _run_module_code
   _run_code(code, mod_globals, init_globals,
 File "c:\Users\amroo\.vscode\extensions\ms-python.python-2022.12.0\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 124, in _run_code
   exec(code, run_globals)
 File "c:\Code\dragonvale\imagematch.py", line 6, in <module>
   p.index('https://github.com/policratus/pupyl/raw/main/samples/images.tar.xz')
 File "c:\Code\dragonvale\dragonvale_env\lib\site-packages\pupyl\search.py", line 307, in index
   raise FileIsNotImage('Please, check your input images.') \
pupyl.duplex.exceptions.FileIsNotImage: Please, check your input images.

Desktop (please complete the following information):

 * OS: Windows 11; Version 10.0.22622 Build 22622

 * Python: 3.10.5

 * Pupyl: 0.14.0

More metadata on web interface
As reported by @mastreips, would be nice to have more metadata information on the web interface.

Adding to web interface
Better image descriptors on web interface like local image file location (inside database), links to the actual image file, etc.

• Create a documentation for installation on Windows;
• Consider Visual Studio C++ auto-installation.

• Deal with open ports which collides with a configured one
• Retry recursively until find a free network port

• Create a command line interface for the library, so everyone can issue operations from the shell;
• It must expose the main features.

Is your feature request related to a problem? Please describe.
I need to index images one by one, since I have thousands of directories and I need to index only few images from each of them instead of all of them. I am following the discussion #138 and I noticed that the image_database.insert() accepts an int as index.

The issue for me is that the images might be deleted at some point and I would like to just delete them by their UUID that they have, rather than dropping whole index and re-indexing whole list again.

Describe the solution you'd like
Make the index parameter int or string or just string

Describe alternatives you've considered
I would need to make a clumsy wrapper around it and track the image indexes.

Additional context
Some of scanned images might be duplicated. There may be multiple directories containing almost the same image (maybe bigger in resolution, or different saturation) How can I make sure, that they won't be imported twice as I am scanning them one by one ?

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. up your server with:
   comment: inside of the directory aps I put 3 jpeg imgs.

from pupyl.search import PupylImageSearch
from pupyl.web import interface

SEARCH = PupylImageSearch()

SEARCH.index(
    'aps'
)

interface.serve()

2. use the search with some public img from the internet.
   suggestion: https://i5.walmartimages.com/asr/9cd67e8a-a6be-4544-9159-b8ef0a192510_2.3a4169a1e12daa021a808b64d70f27eb.jpeg
3. at this moment you will face some error like:

• pupyl.duplex.exceptions.FileIsNotImage, or
• urllib.error.HTTPError: HTTP Error 403: Forbidden

Expected behavior
The platform should return to the search screen, but it stops on a blank page.

Desktop :

• OS: MacOS
• Chrome
• 95.0

Additional context
Change urllib to python requests module, it will solve all your problems.
urllib could be more fast sometimes, but with concerns like that.

• Use pre-trained feature extraction architectures
• Abstracts complexity, size and speed selections (probably with enums)
• Help CPU features/GPU discover to tensorflow

• After indexing and internal image representations (if chosen), the user should run a web interface to visualize the results
• The visualization must show image with kept ratio
• Image representation must be non-uniform (mixing small and large images).

Is your feature request related to a problem? Please describe.
Some users reports that the same image is indexed if they ran the process several times on some source.

Describe the solution you'd like
A way to choose if the library should check for duplicates before index some image or not.

As reported by #138 and #147, indexation speed have been an issue when the input dataset is huge, and sometimes leading to a memory hog or even lasting days to finish.

Proposed solution
Currently, pupyl users can choose (indirectly) between three feature extractors, the first one a really fast but not so precise and the third one a massively precise but slow and resource demanding. Users must have the freedom to choose between one of them more easily, maybe using the same idea of zip trade-off (compression vs speed), like what's proposed next:

from pupyl.search import PupylImageSearch

INDEX = PupylImageSearch(precision: int)

, where precision in (0, 1, 2, 3, 4, 5, 6, 7, 8, 9), meaning precision=0 pupyl must use the fastest but inaccurate (leading to worst search results) feature extractor and precision=9 the most accurate but slow and GPU/CPU/RAM demanding.

Is your feature request related to a problem? Please describe.
I don't get as much utility from the web search results as I would like. Currently, I only have access to a static image and the associated filename.

Describe the solution you'd like
I would love to see more metadata with the Web Search results, including file location and a URI to the location so I can open the folder where the image is located. It would be nice to access individual images from the search results. It would be nice to be able to do some image collection management with the web service itself (delete, move, etc.)

Describe alternatives you've considered
I have looked at other image similarity solutions on github that do have the functionality I am seeking, such as https://github.com/zazaho/SimImg However, I would like to take advantage of the interesting deep learning (EfficiencyNet and DenseNet) functionality of this tool since it does a very nice of finding interesting pattern associations across image categories.

Additional context
N/A

Describe the bug
Pupyl fails to index images. I have tried with different sets of images, but it always fails. After importing and scanning all images, a FileIsNotImage exception is raised.

To Reproduce

import os
from pathlib import Path
from pupyl.search import PupylImageSearch

p = PupylImageSearch(os.path.join(Path.home(), 'pupyl'))
p.index('https://github.com/policratus/pupyl/raw/main/samples/images.tar.xz')

Running the above code gives the following output:

...
🕔 Scanning images: 594 items.
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
FileNotFoundError: 

The above exception was the direct cause of the following exception:

IndexError                                Traceback (most recent call last)
File ~\miniconda3\lib\site-packages\pupyl\search.py:195, in PupylImageSearch.index(self, uri, **kwargs)
    194 try:
--> 195     futures = {
    196         executor.submit(
    197             self.extractor.extract_save,
    198             self.image_database.mount_file_name(
    199                 rank, extension='npy'
    200             ),
    201             self.image_database.load_image_metadata(
    202                 rank, filtered=['internal_path']
    203             )['internal_path']
    204         ): rank
    205         for rank in ranks
    206     }
    208     _ = [
    209         *self.extractor.progress(
    210             concurrent.futures.as_completed(futures),
   (...)
    213         )
    214     ]

File ~\miniconda3\lib\site-packages\pupyl\search.py:201, in <dictcomp>(.0)
    194 try:
    195     futures = {
    196         executor.submit(
    197             self.extractor.extract_save,
    198             self.image_database.mount_file_name(
    199                 rank, extension='npy'
    200             ),
--> 201             self.image_database.load_image_metadata(
    202                 rank, filtered=['internal_path']
    203             )['internal_path']
    204         ): rank
    205         for rank in ranks
    206     }
    208     _ = [
    209         *self.extractor.progress(
    210             concurrent.futures.as_completed(futures),
   (...)
    213         )
    214     ]

File ~\miniconda3\lib\site-packages\pupyl\storage\database.py:274, in ImageDatabase.load_image_metadata(self, index, **kwargs)
    273 except FileNotFoundError:
--> 274     raise IndexError(
    275         f'Index not found: {index}'
    276     ) from FileNotFoundError

IndexError: Index not found: 227

The above exception was the direct cause of the following exception:

FileIsNotImage                            Traceback (most recent call last)
Input In [168], in <cell line: 10>()
      6 from pupyl.search import PupylImageSearch
      8 p = PupylImageSearch('/content/pupyl')
---> 10 p.index('https://github.com/policratus/pupyl/raw/main/samples/images.tar.xz')

File ~\miniconda3\lib\site-packages\pupyl\search.py:217, in PupylImageSearch.index(self, uri, **kwargs)
    208         _ = [
    209             *self.extractor.progress(
    210                 concurrent.futures.as_completed(futures),
   (...)
    213             )
    214         ]
    216     except IndexError as index_error:
--> 217         raise FileIsNotImage('Please, check your input images.') \
    218             from index_error
    220 for rank in self.extractor.progress(
    221     sorted(ranks),
    222     precise=True,
    223     message='Indexing features:'
    224 ):
    225     self.indexer.append(
    226         self.extractor.load(
    227             self.image_database.mount_file_name(
   (...)
    231         check_unique=check_unique
    232     )

FileIsNotImage: Please, check your input images.

Desktop (please complete the following information):

• OS: Windows 10
• Python: 3.9.7
• Pupyl: 0.13.5

Memory leak reading files bigger than memory
When doing a scan on a resource looking for images, pupyl tries to read every file to acknowledge their type and some other contents. But if it come across a huge file (say, 20 GB), pupyl starts to read it to memory, which can lead to a memory leak really fast because such file doesn't fit on memory.

Steps to reproduce the behavior:

1. Put files way greater than your machine memory into a directory;
2. Try to read your directory (for images) with pupyl.duplex.image.ImageIO.scan_images():
3. Wait for the process kill due to the memory leak.

Expected behavior
pupyl should identify if the current file has a size common to an image. There's a likelihood that (encoded) image files don't suppose to have more than 100 MB, for instance.

Desktop:

• OS: Ubuntu
• Version: 20.04.3

• Some parts of pupyl relays in directory separators with /;
• This may result on incompatibility with systems that uses alternative separators (like Windows, which uses \ instead);
• Analyze every part that uses fixed separators and change to something like os.path.join() path resolving.

• Publish it on pypi.org

Discussed in #122 and #129

• Expose search mechanism on CLI interface;
• Optionally give the option to export results image files to a folder.

Examples

# Returning only image ids
pupyl --data_dir /path/to/your/data/dir search /path/to/image/query.jpg
> 129
685
346
524

# Returning metadata
pupyl --data_dir /path/to/your/data/dir search /path/to/image/query.jpg --metadata
> {
       "id": 129,
       "internal_path": "/tmp/pupyl/0/129.jpg",
       "original_access_time": "2021-09-17T13:42:33",
       "original_file_name": "517791629_f05c1518bd.jpg",
       "original_file_size": "52K",
       "original_path": "/tmp/tmpn6duarq6"
   }
   {
       "id": 685,
       "internal_path": "/tmp/pupyl/0/685.jpg",
       "original_access_time": "2021-09-17T13:42:34",
       "original_file_name": "427954022_a072d55df9.jpg",
       "original_file_size": "65K",
       "original_path": "/tmp/tmpn6duarq6"
   }
   ...

# Copying results to folder
pupyl --data_dir /path/to/your/data/dir search /path/to/image/query.jpg --results /path/to/output/results

• Must be possible to export the current index, image database and metadata to an unique file and easily transport it to other machines;
• Compress with lzma and package it with tar;
• Easily import the compressed packaged database on another machine.

• GIFs are ubiquitous when speaking of images;
• It's represented by a tensor (f, h, w, c), which f references every frame;
• Import GIFs dealing with every frame;
• Should import every GIF frame or a sample?
• How to represent it on the database?

@policratus, does pupyl offer the ability to remove all the images in an index (flush)? What if I wanted to remove a specific image from the index, is that possible?

While testing pupyl, I find that these capabilities would be useful if one decides not to index a particular image for example.

Originally posted by @92amartins in #70

• Analyze if it's really necessary to use memoryview of tensors or convert it to numpy arrays: https://github.com/policratus/pupyl/blob/master/duplex/image.py#L156

• Fix the web interface pagination (currently without pagination), to show at startup only a few images, not the entire catalog.
• In the search bar, remove the example url and change it to a more informative text.
• Automatically open default browser after server start.

• Indexed images metadata should be saved on the image database. Ideal metadata:
  • Original file name (even for http images);
  • Original file path;
  • Identification (int) number in index file.
• All metadata must be saved as json files, having the name as its index number.
• Users must have an option to save the image itself in the database or not:
  • If not saving the image in the database, just save the json file with metadata;
  • Else, if save the image to the database is chosen, and if image size is greater than 800x600, should resize the image to the mentioned size, respecting the original image ratio.
  • In the latter case, should convert to a JPEG 80 quality and save the compressed (JPEG) representation of the tensor, being the file name its index number.

Describe the bug
When an index process runs, it tries to index every file in a source, not only the supported ones.

To Reproduce
Steps to reproduce the behavior:

1. Try to index a directory with images and simple plain text files;
2. It will try to index the plain text files too;
3. An FileIsNotImage error will raise.

Expected behavior
If the indexer finds a non-image file, the process should continue and non-image files must be ignored.

Desktop (please complete the following information):

• OS: Ubuntu (Linux)
• Version: 20.04

• To turn easy the usage of pupyl, create a main class which exposes the main features, letting the user configure more specific parameters
• Think about the inheritance of actual class, centralizing (minimally) inheritances to main class

Example

from pupyl import PupylImageSearch
pupyl_image_search = PupylImageSearch(data_dir='/my/data/dir')
print(*pupyl_image_search.search('/path/to/my/query/image.jpg'))
> 4 1 0 5 2 3
print(*pupyl_image_search.search('/path/to/my/query/image.jpg', return_images=True))
> data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGY2NjY2NjY2NjY
...

I can't get the web search engine to properly recognize local file uris (file://) searches. I can retrieve the local URI in the browser, but when I use that same URI in the Pupyl Web Interface Search Bar, I get a "File or Directory not found" error. What is the proper way of searching the index using local files (not http addresses)? Thanks!