brazil-data-cube / bdc-catalog Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 9.0 10.39 MB

Brazil Data Cube Image Metadata Catalog

Home Page: https://bdc-catalog.readthedocs.io/en/latest/

License: GNU General Public License v3.0

Python 94.14% Shell 0.72% PLpgSQL 5.14%

catalog database geospatial image-database metadata python

bdc-catalog's People

Contributors

Stargazers

Watchers

Forkers

raphaelrpl scottwedge mzaglia m3nin0 fabianazioti betonr inpe-cdsr foxander harmonize-brazil

bdc-catalog's Issues

add cascade in the quicklook table

add cascade in FK (colection_id) -> table Quicklook

Review the TimestampMixin

We have add the fields updated and created in the TimestampMixin. However, it should be set as server_default instead default.

Adopt the Brazil Data Cube Sphinx template

Review jsonschema collection metadata - instrument key

The instrument should be a list instead a single value to support multiple data source processing (Surface Reflectance, Surface Temperature, etc)

Release version 0.6.0

Tasks:

Inform the right version in: bdc_catalog/version.py
Run all the tests.
Review the CHANGES.rst.
Create a branch named: b-0.6.
Review links in the documentation for the stable version (README.rst).
Create a tag named: v0.6.0.
In the master branch increase version in bdc_db/version.py to 0.8.0a.
Notify to the community the release of a new version.

Prepare the codebase for using coveralls.io

Add a script to record the LabISA CBERS-4 MUX collection

Add jsonschemas for JSONB columns

Create field on CollectionItem to store source data

We should store the raw data (.zip, .tar.gz) on collection item.

Add a field compressed_file as optional to the CollectionItem model.

Document package modules, classes and functions

Change to server_default in TimestampMixin

We should consider to change the created and updated fields in bdc_catalog.models.base_sql to server_default=func.now() instead default=datetime.utcnow since the last only validates in python SQLAlchemy ORM and with server_default, these values are handled by database itself.

If you need to insert in database directly without SQLAlchemy, you must give created and updated with function now()

Or we can just remove nullable argument.

Should we support external collections?

To support external collection, we should consider a prefix field in collections to be prepended in assets

Error in migrations - missing timeline

SQLALCHEMY_DATABASE_URI="postgresql://localhost:5434/catalog2" bdc-db alembic revision upgrade
/home/user/bdc-catalog/venv/lib/python3.7/site-packages/alembic/script/revision.py:154: UserWarning: Revision 149deb59082a referenced from 149deb59082a -> f3f27ad116d2 (head), timeline is not present
  % (downrev, rev)
Traceback (most recent call last):
  File "/home/user/bdc-catalog/venv/bin/bdc-db", line 8, in <module>
    sys.exit(cli())
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/flask_alembic/cli/click.py", line 114, in revision
    base.revision(message, empty, branch, parent, splice, depend, label, path)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/flask_alembic/cli/base.py", line 97, in revision
    get_alembic().revision(message, empty, branch, parent, splice, depend, label, path)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/flask_alembic/extension.py", line 391, in revision
    branch_exists = any(r for r in self.script_directory.revision_map.get_revisions(branch) if r is not None)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/alembic/script/revision.py", line 321, in get_revisions
    resolved_id, branch_label = self._resolve_revision_number(id_)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/alembic/script/revision.py", line 501, in _resolve_revision_number
    self._revision_map
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/alembic/util/langhelpers.py", line 230, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/home/user/bdc-catalog/venv/lib/python3.7/site-packages/alembic/script/revision.py", line 156, in _revision_map
    down_revision = map_[downrev]
KeyError: '149deb59082a'

Fix release of version 0.4.0

Add BDC-DB version 0.2.0 as main dependency in setup.py. Remove tag and recreate it.

Include definitions for each concept in the schema

Utility to create Grid geometry table

Create Grid geometry table as lowered case.

Add a table to track collection diretories

Change size of column id on collections

Currently, the column id has only 20 chars. In this way, we have faced issues while creating a data cube. Maybe 60 chars is enough?

Release version 0.4.0

Tasks:

Inform the right version in: bdc_db/version.py
Run all the tests.
Review the ER diagram and its associated PNG.
Review the CHANGES.rst.
Create a branch named: b-0.4.
Review links in the documentation for the stable version (README.rst).
From the branch b-0.4, create a tag named: v0.4.0.
In the master branch increase version in bdc_db/version.py to 0.6.0.
Notify to the community the release of a new version.

Add utility for check_sum generation

Review migrations for release - 0.4.0

Add triggers to the repository

The following triggers will be required:

Before inserting/updating on bdc.items:
- Check if tile_id belongs to the Collection's grid_ref_sys`.
- Check assets field.
After inserting/updating on bdc.items:
- update time_line table with the time instants.
- update the date range in table bdc.collections.
- update the extent in table bdc.collections.

create srid field in items table

Add command line interface

Integrate the catalog data model with LCCS data model

Add CLI utility

bug get_geom_table (grid_ref_sys)

is referencing grs_schemas instead of grid_ref_sys

Add time_line table model

Review bdc-db migrations intialization

We must review the initialization of bdc-db as branches in flask-migrate/flask-alembic.

The license field for collections or items should follow the SPDX License List

Check against STAC proposed validation pattern: ^[\w\-\.\+]+$.

Error while creating a new SRID using spatial_ref_sys model

We have defined a model SpatialRefSys to make a wrapper of PostGIS spatial_ref_sys table. However, the following error occurs when i'm trying to insert a new one.

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column spatial_ref_sys.created does not exist
LINE 1: ...ys.proj4text AS public_spatial_ref_sys_proj4text, public.spa...

This model must not inherit from BaseModel. It should be a simple model db.Model

Add access control to collections

Fix extension BDCCatalog loading

The ext BDCCatalog should initialize BrazilDataCubeDB (bdc-db) on init_app.

Add ER document

Include the Draw.io ER model under the folder doc/datamodel and a PDF with the diagram.

Review the collection-temporal-composition-schema.jon

The key intervals should be reviewed.

Review Sphinx documentation

Release version 0.2.0

Tasks:

Inform the right version in: bdc_db/version.py
Review the ER diagram and its associated PNG.
Review the CHANGES.rst.
Create a branch named: b-0.2.
From the branch b-0.2, create a tag named: v0.2.0.
In the master branch increase version in bdc_db/version.py to 0.4.0

Prepare the Python package support

Error while migrate bdc-db

The following error occurs while executing migration:

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column "chunk_size_t" of relation "assets" does not exist

[SQL: ALTER TABLE assets ALTER COLUMN chunk_size_t TYPE FLOAT(50) ]
(Background on this error at: http://sqlalche.me/e/f405)

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.ddl.postgresql] Detected sequence named 'assets_id_seq' as owned by integer column 'assets(id)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'assets.chunk_size_t'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'assets.chunk_size_x'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'assets.chunk_size_y'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'assets.raster_size_t'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'assets.raster_size_x'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'assets.raster_size_y'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'bands.resolution_x'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'bands.resolution_y'
INFO  [alembic.autogenerate.compare] Detected added unique constraint 'None' on '['id']'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'raster_size_schemas.chunk_size_t'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'raster_size_schemas.chunk_size_x'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'raster_size_schemas.chunk_size_y'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'raster_size_schemas.raster_size_t'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'raster_size_schemas.raster_size_x'
INFO  [alembic.autogenerate.compare] Detected type change from DOUBLE_PRECISION(precision=53) to Float(precision=50) on 'raster_size_schemas.raster_size_y'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiles_geom' on 'tiles'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiles_geom_wgs84' on 'tiles'
  Generating /home/raphael/devel/cube/bdc-db/migrations/versions/da976c811be3_.py ...  done

It always generate a new one migration, recreating both idx_tiles_geom and idx_tiles_geom_wgs84 spatial indexes. It also changes the Float columns to PostgreSQL Double precision.

It seems related with manually precision on these models assets, bands and raster_size_schemas. By default, SQLAlchemy detects PostgreSQL float columns with 53 digits precision.

Possibly solutions:

Set to Float precision to 53

Create spatial indices using sqlalchemy.Index

class Tile(BaseModel):
  __tablename__ = 'tiles'
  __table_args__ = (
      Index('idx_tiles_geom_wgs84', 'geom_wgs84', postgres_using='gist'),
      Index('idx_tiles_geom', 'geom', postgres_using='gist'),
  )
  ...

Remove useless unique=True directive from CompositeFunctionSchema

Steps to reproduce

git clone https://github.com/brazil-data-cube/bdc-db
cd bdc-db
python3 -m venv venv
source venv/bin/activate
pip3 install --upgrade pip
pip3 install wheel
pip3 install -e .[all]
export FLASK_APP=examples.app
flask db upgrade
flask db migrate

UniqueViolation while inserting a new Tile with different GrsSchema

We must remove the constraint Unique from Tile

sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "tiles_id_key"
DETAIL:  Key (id)=(086095) already exists.

fields: max_x and min_y

because we will use it to generate the automatic raster_size_schema

brazil-data-cube / bdc-catalog Goto Github PK

bdc-catalog's People

Contributors

Stargazers

Watchers

Forkers

bdc-catalog's Issues

Steps to reproduce

Recommend Projects

Recommend Topics

Recommend Org