iscc / iscc-core Goto Github PK
View Code? Open in Web Editor NEWISCC - Codec & Algorithms
Home Page: https://core.iscc.codes/
License: Apache License 2.0
ISCC - Codec & Algorithms
Home Page: https://core.iscc.codes/
License: Apache License 2.0
The wallet-id of the declaring party should be mixed
into the ISCC-ID (in a reversible manner) to support pre-assigments without race conditions.
A way to aggregate multiple ISCC-CODEs into a single ISCC-CODE.
Use Case:
see also: https://github.com/andhus/dirhash
Hello,
When I try to encode both audio and video codes into a single code, I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/mble/Code/repos/mbledkowski/mdrnshData/iscc-api/main.py", line 16, in iscc
return ic.gen_mixed_code(
^^^^^^^^^^^^^^^^^^
File "/home/mble/.cache/pypoetry/virtualenvs/iscc-api-5CADumn--py3.12/lib64/python3.12/site-packages/iscc_core/code_content_mixed.py", line 36, in gen_mixed_code
return gen_mixed_code_v0(codes, bits=bits)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mble/.cache/pypoetry/virtualenvs/iscc-api-5CADumn--py3.12/lib64/python3.12/site-packages/iscc_core/code_content_mixed.py", line 52, in gen_mixed_code_v0
digests = [ic.decode_base32(ic.iscc_clean(code)) for code in codes]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mble/.cache/pypoetry/virtualenvs/iscc-api-5CADumn--py3.12/lib64/python3.12/site-packages/iscc_core/codec.py", line 283, in decode_base32
return bytes(b32decode(code + "=" * pad_length, casefold=True))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/base64.py", line 254, in b32decode
return _b32decode(_b32alphabet, s, casefold, map01)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/base64.py", line 240, in _b32decode
raise binascii.Error('Incorrect padding')
binascii.Error: Incorrect padding
Executed code:
return ic.gen_mixed_code(
idk.code_audio(fileName).iscc, idk.code_video(fileName).iscc
)
It seems that the issue lies in the decode_base32 function in codec.py, which adds incorrect padding length.
Best regards
I'll have to work on this my end, although I'm not able to be admin of this particular machine. Logging for anyone who may also happen upon this issue:
Running: python -m pip install iscc-core
Collecting iscc-core
Using cached iscc-core-0.2.8.tar.gz (56 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting data-url<2.0,>=1.0
Using cached data_url-1.0.3-py3-none-any.whl
Collecting bitarray-hardbyte
Using cached bitarray_hardbyte-2.3.8-cp39-cp39-win_amd64.whl (103 kB)
Collecting blake3
Using cached blake3-0.3.1-cp39-none-win_amd64.whl (193 kB)
Collecting jcs<0.3.0,>=0.2.0
Using cached jcs-0.2.1-py3-none-any.whl (7.6 kB)
Collecting pybase64
Using cached pybase64-1.2.2-cp39-cp39-win_amd64.whl (38 kB)
Collecting pydantic[dotenv]
Using cached pydantic-1.9.1-cp39-cp39-win_amd64.whl (2.0 MB)
Collecting more-itertools
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Collecting xxhash
Using cached xxhash-3.0.0-cp39-cp39-win_amd64.whl (29 kB)
Collecting uvarint<2.0,>=1.2
Using cached uvarint-1.2.0-py3-none-any.whl (4.4 kB)
Collecting loguru<0.7,>=0.6
Using cached loguru-0.6.0-py3-none-any.whl (58 kB)
Collecting bases<0.3,>=0.2
Using cached bases-0.2.1-py3-none-any.whl (35 kB)
Collecting base58<3.0,>=2.1
Using cached base58-2.1.1-py3-none-any.whl (5.6 kB)
Collecting typing-validation
Using cached typing_validation-0.0.1.post7-py3-none-any.whl (9.5 kB)
Requirement already satisfied: typing-extensions in c:\users\spencer\apps\python\lib\site-packages (from bases<0.3,>=0.2->iscc-core) (4.1.1)
Requirement already satisfied: colorama>=0.3.4 in c:\users\spencer\apps\python\lib\site-packages (from loguru<0.7,>=0.6->iscc-core) (0.4.4)
Collecting win32-setctime>=1.0.0
Using cached win32_setctime-1.1.0-py3-none-any.whl (3.6 kB)
Collecting python-dotenv>=0.10.4
Using cached python_dotenv-0.20.0-py3-none-any.whl (17 kB)
Building wheels for collected packages: iscc-core
Building wheel for iscc-core (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for iscc-core (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [75 lines of output]
A setup.py file already exists. Using it.
C:\Users\Spencer\AppData\Local\Temp\pip-build-env-wrk8753n\overlay\Lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\iscc_core\cdc.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
C:\Users\Spencer\AppData\Local\Temp\pip-build-env-wrk8753n\overlay\Lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\iscc_core\minhash.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
C:\Users\Spencer\AppData\Local\Temp\pip-build-env-wrk8753n\overlay\Lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\iscc_core\simhash.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
Compiling iscc_core/cdc.py because it changed.
Compiling iscc_core/minhash.py because it changed.
Compiling iscc_core/simhash.py because it changed.
[1/3] Cythonizing iscc_core/cdc.py
[2/3] Cythonizing iscc_core/minhash.py
[3/3] Cythonizing iscc_core/simhash.py
************************************************************
Succeeded to compile Cython/C accelerator modules :) *
************************************************************
running build
running build_py
creating C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build
creating C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39
creating C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\cdc.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\check.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\codec.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\code_content_audio.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\code_content_image.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\code_content_mixed.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\code_content_text.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\code_content_video.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\code_data.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\code_flake.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\code_instance.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\code_meta.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\conformance.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\constants.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\dct.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\iscc_code.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\iscc_id.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\minhash.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\models.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\options.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\simhash.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\utils.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\wtahash.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\__init__.py -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\cdc.c -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\cdc.pxd -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\data.json -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\minhash.c -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\minhash.pxd -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\simhash.c -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
copying iscc_core\simhash.pxd -> C:\Users\Spencer\AppData\Local\Temp\pip-install-g8z84mcf\iscc-core_94f9d89354664a52b747d9b61b700b0a\build\lib.win-amd64-cpython-39\iscc_core
running build_ext
building 'iscc_core.cdc' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
Traceback (most recent call last):
File "C:\Users\Spencer\Apps\python\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 363, in <module>
main()
File "C:\Users\Spencer\Apps\python\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "C:\Users\Spencer\Apps\python\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 261, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "C:\Users\Spencer\AppData\Local\Temp\pip-build-env-wrk8753n\overlay\Lib\site-packages\poetry\core\masonry\api.py", line 68, in build_wheel
return unicode(WheelBuilder.make_in(poetry, Path(wheel_directory)))
File "C:\Users\Spencer\AppData\Local\Temp\pip-build-env-wrk8753n\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 78, in make_in
wb.build()
File "C:\Users\Spencer\AppData\Local\Temp\pip-build-env-wrk8753n\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 110, in build
self._build(zip_file)
File "C:\Users\Spencer\AppData\Local\Temp\pip-build-env-wrk8753n\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 162, in _build
self._run_build_command(setup)
File "C:\Users\Spencer\AppData\Local\Temp\pip-build-env-wrk8753n\overlay\Lib\site-packages\poetry\core\masonry\builders\wheel.py", line 190, in _run_build_command
subprocess.check_call(
File "C:\Users\Spencer\Apps\python\lib\subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:/Users/Spencer/Apps/python/python.exe', 'C:\\Users\\Spencer\\AppData\\Local\\Temp\\pip-install-g8z84mcf\\iscc-core_94f9d89354664a52b747d9b61b700b0a\\setup.py', 'build', '-b', 'C:\\Users\\Spencer\\AppData\\Local\\Temp\\pip-install-g8z84mcf\\iscc-core_94f9d89354664a52b747d9b61b700b0a\\build']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for iscc-core
Failed to build iscc-core
ERROR: Could not build wheels for iscc-core, which is required to install pyproject.toml-based projects
This is a question rather than an issue.
How would one compare two svg images to detect similarities? I understand that an svg object should be treated as text as far as its content goes. Supposing I have two svgs which differ only in a path
element with the first one colored using #97C024
and the second using #97C020
, if I create my iscc codes from instance, content(text), and data units and compare them I obtain a similarity of 74% calculated with iscc_similarity
. Would not such a value be an underestimation of their similarity? The analyzed svgs should be practically considered equal. How would I use the ISCC algorithms to spot such subtle differences? Thanks in advance.
Combine return types of gen_*
code generator functions into a single unified type.
It seems that the package is incompatible with recent changes in Pydantic. I just installed iscc-core
into a fresh Python 3.10 venv and then tried to import it. This was the result:
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import iscc_core
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/xxx/proj/iscc/venv/lib/python3.10/site-packages/iscc_core/__init__.py", line 2, in <module>
from iscc_core.options import core_opts, conformant_options
File "/home/xxx/proj/iscc/venv/lib/python3.10/site-packages/iscc_core/options.py", line 15, in <module>
from pydantic import BaseSettings, Field
File "/home/xxx/proj/iscc/venv/lib/python3.10/site-packages/pydantic/__init__.py", line 363, in __getattr__
return _getattr_migration(attr_name)
File "/home/xxx/proj/iscc/venv/lib/python3.10/site-packages/pydantic/_migration.py", line 296, in wrapper
raise PydanticImportError(
pydantic.errors.PydanticImportError: `BaseSettings` has been moved to the `pydantic-settings` package. See https://docs.pydantic.dev/2.5/migration/#basesettings-has-moved-to-pydantic-settings for more details.
For further information visit https://errors.pydantic.dev/2.5/u/import-error
There is currently no version restriction defined for the Pydantic dependency:
Line 50 in fb114bb
The packages that were automatically installed by pip are:
Package Version
----------------- -------
annotated-types 0.6.0
base58 2.1.1
bitarray 2.8.5
blake3 0.3.4
data-url 1.0.7
iscc-core 1.0.5
jcs 0.2.1
loguru 0.7.2
more-itertools 10.1.0
pip 22.0.2
pybase64 1.3.1
pydantic 2.5.2
pydantic_core 2.14.5
setuptools 59.6.0
typing_extensions 4.9.0
uvarint 1.2.0
xxhash 3.4.1
The solution for this and similar issues is to specify more specific version requirements for dependencies.
Define a canonical hashing scheme for ISCC metadata such that the resulting hash is a IPFS CIDv1 that can be used as token-id in ERC721 or ERC-1155 contracts. See also: https://forum.openzeppelin.com/t/how-to-erc-1155-id-substitution-for-token-uri/3312/14
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.