gisaia / arlas-persistence Goto Github PK

License: Apache License 2.0

Java 84.96% HTML 0.21% Dockerfile 0.69% Shell 14.10% JavaScript 0.05%

arlas-persistence's Introduction

Gisaïa

Gisaïa develops ARLAS, an open source platform for exploring geo-analytically huge volumes of spatio-temporal data.

To begin with ARLAS Exploration, you can start the full software stack on your computer with the ARLAS Exploration stack project. Also, three tutorials are available for loading data in ARLAS. You can try a tutorial with bird tracking data, another one with vessel tracking data (also called AIS data) and finally a last one with polluant data.

In case you are interested in massively processing geotracked asset data, then our open source library ARLAS PROC ML can be very useful.

arlas-persistence's People

Contributors

Watchers

Forkers

sebbousquet bradh

arlas-persistence's Issues

Add check on group before POST and PUT

If a user try to POST or PUT data with reader/writers informations we need to check that the user belongs to the groups provided.
Think to deal with the anonymous mode

REST API Get by Id

Add REST endpoint GET /persist/resources/{id}

Add organisation and public status in each element of the resouces list

For /persist/resource/{zone} endpoint, enrich the response with

organisation name of the resource
whether the resource is public or not based on docwriters/readers

If a resource is public and the "write" right is activated, should any authenticated user be able to edit/share/duplicate/rename the resource ?

Scenario of the question

I am a user A authenticated with IAM and I belong to organisation O1
There is a resource belonging to an organisation called arlas that is 'public'.
I, as user A, don't belong to arlas organisation
This resource readers and writers include both the 'public' group
I, as user A, was expecting to be able to duplicate/share/edit/rename the dashboard but it's not the case
- the updatable attribute is false

Enable a user to have at least one default configuration per group

Since we need to have a default config.json ressource for each organization in city, we need to be able to publish several config named default, at least one per group (or organization).

Firestore supports up to 10 groups

INFO [2021-04-08 12:33:57,815] io.arlas.persistence.rest.PersistenceRestService: User='auth0|5f917f519b8822006edf50a8' / Org='' / Groups='[group/config.json/insdex/crimeproba, group/config.json/insdex/gisaia, group/config.json/insdex/heatwave, group/config.json/insdex/meteo, group/config.json/insdex/policedata, group/config.json/insdex/policyclaim, group/config.json/insdex/solar, group/config.json/insdex/technicalrating, group/config.json/insdex/wave, group/public]'
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: Error occurred io.arlas.server.exceptions.ArlasException: Error listing document: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: 'ARRAY_CONTAINS_ANY' supports up to 10 comparison values.
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: ! io.arlas.persistence.server.impl.GoogleFirestorePersistenceServiceImpl.list(GoogleFirestorePersistenceServiceImpl.java:175)
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: ! io.arlas.persistence.rest.PersistenceRestService.list(PersistenceRestService.java:134)
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: ! sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: ! sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: ! java.lang.reflect.Method.invoke(Method.java:498)
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: ! org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: ! org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: ! org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: ! org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
ERROR [2021-04-08 12:33:57,833] io.arlas.server.exceptions.ArlasExceptionMapper: ! org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)

Return readers and writers list in Data response

[Firestore] Make the collection name configurable

In order to be able to store several projects' collection in the same GCP project, we need to be able to configure the collection name (prefixing with the workspace name for instance).
Beware that the current collection name is also used in the Postgres implementation so we may need to use a different value for the PG and Firestore implementation.

Order by date is not implemented for file system implementation

Public dashboards are not available when logged in

In order to fix this issue, we decided to:

remove the endpoints that fetches documents by key
remove unicity constraint on docKey
return public dashboards as well as the current organisation's ones
add the organisation to the attributes of the returned document as well as its public status

Endpoint getGroupsByZone return DataWithLinks

Can't update a dashboard if not in all authorized writers groups

If a dashboard is writable by group A and B, and a user only belongs to group A, he cannot update a dashboard.

In file system impl, update doc key in body valu don't update the key in the filename

Documentation: gisaia/arlas-persistence:latest => gisaia/arlas-persistence-server:latest

gisaia/arlas-persistence:latest => gisaia/arlas-persistence-server:latest

See https://hub.docker.com/r/gisaia/arlas-persistence-server

Here:

ARLAS-persistence/docs/arlas-persistence-configuration.md

Line 24 in 4600f70

gisaia/arlas-persistence:latest

and later in the document.

FileSystemImpl with anonymous access error

ERROR [2023-12-06 14:09:01,307] io.arlas.commons.exceptions.ArlasExceptionMapper: Error occurred io.arlas.commons.exceptions.ArlasException: A unique organisation must be set in IdParam but received: []
ERROR [2023-12-06 14:09:01,308] io.arlas.commons.exceptions.ArlasExceptionMapper: ! io.arlas.persistence.server.impl.FileSystemPersistenceServiceImpl.create(FileSystemPersistenceServiceImpl.java:108)
ERROR [2023-12-06 14:09:01,309] io.arlas.commons.exceptions.ArlasExceptionMapper: ! io.arlas.persistence.rest.PersistenceRestService.create(PersistenceRestService.java:381)
ERROR [2023-12-06 14:09:01,310] io.arlas.commons.exceptions.ArlasExceptionMapper: ! java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
ERROR [2023-12-06 14:09:01,310] io.arlas.commons.exceptions.ArlasExceptionMapper: ! java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
ERROR [2023-12-06 14:09:01,310] io.arlas.commons.exceptions.ArlasExceptionMapper: ! java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
ERROR [2023-12-06 14:09:01,310] io.arlas.commons.exceptions.ArlasExceptionMapper: ! java.base/java.lang.reflect.Method.invoke(Method.java:568)
ERROR [2023-12-06 14:09:01,310] io.arlas.commons.exceptions.ArlasExceptionMapper: ! org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
ERROR [2023-12-06 14:09:01,310] io.arlas.commons.exceptions.ArlasExceptionMapper: ! org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
ERROR [2023-12-06 14:09:01,310] io.arlas.commons.exceptions.ArlasExceptionMapper: ! org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
ERROR [2023-12-06 14:09:01,310] io.arlas.commons.exceptions.ArlasExceptionMapper: ! org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)

Add an end-point to know if a ressource exist or not

Need to be discut:

/exist/{id} => return HTTP OK 200 with true or false
/exist/{zone}/{key} => return HTTP OK 200 with true or false

Usefull in front end to avoid 404 on delete i18n file with config.

@alainbodiguel Do you agree to take this task ?

A resource doesn't appear for its owner when a shared user removes the read and write rights to all groups

Storage : FileStore and File System

How to reproduce

Create a resource by a user A (the owner)
Share the resource with read AND write rights with a group G
Login with a user B who belongs to group G
User B has the right to share the resource as well
With user B remove the read AND write rights from all groups of this resource
The resource is fetched with GET list for user B, but user B should no longer see this resource.
Login now with the user A. The resource is not fetched with GET list but user A is its owner, so it should appear in the list

Upgrade to java 17 + dependencies update

Upgrade to java 17 +
Dependencies update:

log4j-core 2.13.2 -> 2.14.1
java-uuid-generator 3.1.5 -> 4.0.1
dropwizard 1.3.9 -> 2.0.25
postgresql-jdbc 42.2.10 -> 42.2.24
hibernate-types-52 2.9.5 -> 2.13.0
Google library bom 4.2.0 -> 23.1.0
jackson-databind 2.9.9.3 -> 2.13.0
junit 4.13.1 -> 4.13.2
hamcrest-core 1.3 -> 2.2
rest-assured 3.3.0 -> 4.4.0

Forbid setting the 'write' right to 'public' group

Property `Creation Date` always return current date instead of the stored one

Data in DB

Result of curl command

Convertion timestamp to human readable date

The implementation of update methods don't check the unicity of key/zone/orga

Delete endpoint cause Mapper Exception when connected to postgres db

This error doesn't occur in file mode.

Stack trace

[2020-09-25 14:40:39,272] io.dropwizard.jersey.jackson.JsonProcessingExceptionMapper: Unable to process JSON
! org.hibernate.LazyInitializationException: failed to lazily initialize a collection, could not initialize proxy - no Session
! at org.hibernate.collection.internal.AbstractPersistentCollection.throwLazyInitializationException(AbstractPersistentCollection.java:587)
! at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:202)
! at org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:146)
! at org.hibernate.collection.internal.PersistentBag.isEmpty(PersistentBag.java:266)
! at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.isEmpty(CollectionSerializer.java:80)
! at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.isEmpty(CollectionSerializer.java:25)
! at com.fasterxml.jackson.datatype.hibernate5.PersistentCollectionSerializer.isEmpty(PersistentCollectionSerializer.java:162)
! at com.fasterxml.jackson.module.afterburner.ser.ObjectMethodPropertyWriter.serializeAsField(ObjectMethodPropertyWriter.java:72)
! at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
! ... 71 common frames omitted
! Causing: com.fasterxml.jackson.databind.JsonMappingException: failed to lazily initialize a collection, could not initialize proxy - no Session (through reference chain: io.arlas.persistence.model.DataWithLinks["doc_writers"])
! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
! at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
! at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
! at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
! at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
! at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
! at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)
! at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913)
! at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625)
! at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
! at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
! at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
! at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
! at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
! at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86)
! at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
! at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)
! at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)
! at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
! at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
! at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329)
! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
! at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
! at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
! at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
! at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
! at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
! at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
! at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
! at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
! at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)
! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
! at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
! at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:45)
! at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39)
! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
! at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:311)
! at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:265)
! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
! at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
! at io.dropwizard.jetty.ContextRoutingHandler.handle(ContextRoutingHandler.java:38)
! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)
! at io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:67)
! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
! at org.eclipse.jetty.server.Server.handle(Server.java:502)
! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
! at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
! at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
! at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
! at java.lang.Thread.run(Thread.java:748)
`
``

List with no arlas-org-filter returns only public ressource

I have three organisations A,B,C.
I'm connected.

If I call list with arlas-org-filter = A, I get all the ressources for org A and only public ones of B and C. ==> It's OK

But

If I call list with no arlas-org-filter, I get only public ressources of A, B and C.

What I want : If no arlas-org-filter, retrieve all my org in the token and returns all ressources for A B and C, with isupdatable property correctly set (and return public ressources for another organisation)