dataswift / hat2.0 Goto Github PK
View Code? Open in Web Editor NEWThe HAT Personal Microserver
Home Page: https://hubofallthings.com
License: GNU Affero General Public License v3.0
The HAT Personal Microserver
Home Page: https://hubofallthings.com
License: GNU Affero General Public License v3.0
When running FacebookDataPlug, it tries to create a table model in HAT (see https://github.com/Hub-of-all-Things/DataPlugFacebook/blob/master/config/fbHatModels.js)
I had part of that fail because of primary key violations (which is fine).
But as a result, parts of the table model have been successfully created (the top-level table), while others have not (such as fields or sub-tables), making it difficult to clean up before the request could be run again.
Wherever feasible, modifications made by a REST API call should be applied all or nothing. This should be possible here with database transactions.
Having created the location and (event,thing,organisation):
location
{
"id": 191,
"name": "stairs"
}
thing
{
"id": 44,
"name": "pressure sensor"
}
I cannot create a relationship between them by posting
{
"relationshipType": "parent child"
}
to
location/191/thing/44?userAuth..
Returns 400 and seems like it tries to bind the first id to the thing rather than taking the second one as designed:
ERROR: insert or update on table "locations_locationthingcrossref" violates foreign key constraint "thing_id_refs_id_fk"
Detail: Key (thing_id)=(191) is not present in table "things_thing".
For consideration: to avoid simple sequencing issues and further minimise exposure of internal HAT state, all publicly visible (via APIs) and hence database IDs should be converted to UUID type instead of auto-incrementing integers
To reduce the code complexity and the load on the DS backend, can I propose parameterized combinators as a future enhancement?
e.g.
[
{
"endpoint": "rumpel/locations",
"filters": [
{
"field": "data.locations.timestamp",
"transformation": {
"transformation": "datetimeExtract",
"part": "hour"
},
"operator": {
"operator": "between",
"lower": "$lower",
"upper": "$upper"
}
}
]
}
]
The call to get the result would then be:
GET /api/v2.6/combinator/$COMBINATOR_NAME?lower=7&upper=9
Endpoint: PUT /api/v2.6/data
Description: Internal Server Error thrown when updating data to an empty array
Steps to replicate
Create the following data into an endpoint POST /api/v2.6/data
{ "id": "some-random-id", "symptoms": ["fever", "cough"], "timestamp": 1591802656398 }
Update the above data using PUT /api/v2.6/data with
{ "id": "some-random-id", "symptoms": [], "timestamp": 1591802656398 }
Internal Server Error thrown.
Expected behaviour: Data record should be updated with symptoms = []
Other information
There is no issues creating an initial record with an empty array. Only updating has issues.
Creating an entity and filling it with data (attaching types and properties) requires several API calls, with the associated overhead and loss of transactional capabilities.
It would be nice to be able to create an entity and specify its types and properties in a single call, similar to how it can be done for data records.
When calling into the API to create a new table (POST /data/table
) with invalid data (such as an empty JSON object {}
) you correctly get error code 400 (Bad Request "Object is missing required member 'name'" ) when you are authenticating with username and password, but when authenticating with an acccess token you get 401 (Unauthorized), even though the token is valid.
The correct error code (and the very useful detailed JSON validation or database error message that goes along with it) should be returned when a valid access token is presented as well.
Instagram API data format has changed significantly in a recent migration to the graph API. Currently, SHE feed mapper supports only the new Instagram data format. It should be improved to support both the old and the new data formats.
Property lookup endpoints are only enabled for the owner account
Managed to get my own server running (localhost proxied through ngrok for now), and I can access Rumple and update my details, but when trying to enable a data-plug, like social-plug.hubofallthings.com, I receive a 502 after entering my PHATA.
The ngrok proxy throws a few GET /data/table 404 Not Found
, but it seems to be rumpel querying missing data sources
[INFO ] [08/07/2016 13:33:22] API-Access - GET:http://a5b6c56f.ngrok.io/data/table?name=events&source=facebook:404 Not Found
[INFO ] [08/07/2016 13:33:22] API-Access - GET:http://a5b6c56f.ngrok.io/data/table?name=posts&source=facebook:404 Not Found
[INFO ] [08/07/2016 13:33:22] API-Access - GET:http://a5b6c56f.ngrok.io/data/table?name=events&source=ical:404 Not Found
[INFO ] [08/07/2016 13:33:23] API-Access - GET:http://a5b6c56f.ngrok.io/data/table?name=locations&source=iphone:404 Not Found
As part of the URL, query parameters are often logged along the way, putting a password in there seems risky.
Please also allow sending the credentials via the Basic-Auth mechanism in the request header.
When doing things like creating a new record with fields filled in you need to provide the API with the relevant field or table name in addition to the field or table id (which is already unique). If the name is missing the API call is rejected.
But it seems that the names provided do not have to be correct (i.e. match the field/table definition). You can post any dummy name, and the request will be processed.
If the purpose of including the name in the request is to provide for extra validation of intent / avoid mistakes, then this parameter should be checked and the request refused when invalid names are specified.
If there is no validation, there should be no need to pass in the name at all (which makes writing API client code easier, too).
Possible middle-ground / best-of-both-worlds: Make the name optional, but if specified, it must be correct.
I am following the install instructions from: https://github.com/Hub-of-all-Things/HAT2.0 specifically Running the project, HAT Setup.
Cloned the repository:
https://github.com/Hub-of-all-Things/HAT2.0
Steps 1-3, creating the postgreSQL db, compiling the project and local domain mapping went without error.
Step 4 sbt "project hat" -Dconfig.resource=dev.conf run give me this error:
don-u@don--u:~$ sbt compile
[info] Loading project definition from /home/don-w510-u/project
[info] Set current project to don-u (in build file:/home/don-u/)
[info] Executing in batch mode. For better performance use sbt's shell
[success] Total time: 0 s, completed 13-Nov-2017 8:28:01 AM
don-u@don-u:~$ sbt "project hat" -Dconfig.resource=dev.conf run
[info] Loading project definition from /home/don-u/project
[info] Set current project to don-w510-u (in build file:/home/don-u/)
[error] Not a valid project ID: hat
[error] project hat
[error] ^
Also tried:
don-u@don-u:~$ sbt "project hat2.0" -Dconfig.resource=dev.conf run
[info] Loading project definition from /home/don-u/project
[info] Set current project to don-u (in build file:/home/don-u/)
[error] Expected project ID
[error] project hat2.0
[error] ^
Please tell me what I am doing incorrectly. Thanks, Don
The feature would improve data submission efficiency and reduce client-side code complexity when large amount of records need to be processed at once.
Feature would improve the efficiency by reducing the number of required client-server round trips.
Current API endpoints do not include entity Type in the response. Please add this information in entity retrieval. Also, please add filtering by type.
Currently (a limitation of Slick) to reconstruct structures of deeply nested data, such as recovering the structures of nested data_table
s it is necessary to do one roundtrip to the database per each table, another roundtrip collect its related tables and repeat the same operation on them recursively, resulting in O(n) trips to the database where n is the number of tables in the given structure.
One way around it is to use SQL common table expressions (marked for 3.1.0 release of Slick as well), e.g.:
WITH RECURSIVE recursive_data_tables(id, date_created, last_updated, name, source_name) AS (
SELECT id, date_created, last_updated, name, source_name FROM data_table WHERE name = 'HyperDataBrowser'
UNION ALL
SELECT tab.id, tab.date_created, tab.last_updated, tab.name, tab.source_name
FROM recursive_data_tables r_tab, data_table tab
WHERE tab.id IN (SELECT table2 FROM data_tabletotablecrossref WHERE table1 = r_tab.id)
)
SELECT * FROM recursive_data_tables;
However need to work out parsing back into Slick data models and be careful with infinite recursion
Running the deploy.sh script throws a couple of missing file errors -
Setting up database user and database
dbuser: hat20test, database: hat20test
CREATE ROLE hat20test NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;
createuser: creation of new role failed: ERROR: role "hat20test" already exists
createdb: database creation failed: ERROR: database "hat20test" already exists
ALTER ROLE
Handling schemas
NOTICE: drop cascades to 2 other objects
DETAIL: drop cascades to extension uuid-ossp
drop cascades to extension pgcrypto
DROP SCHEMA
CREATE SCHEMA
ALTER SCHEMA
Setting up database
CREATE EXTENSION
CREATE EXTENSION
./deployment/deploy.sh: line 31: ./src/sql/HAT-V2.0.sql: No such file or directory
./src/sql/HAT-V2.0.sql
Setting up corresponding configuration
Setting up HAT access
./deployment/deploy.sh: line 51: ./src/sql/boilerplate/authentication.sql: No such file or directory
bash: applyEvolutions.sh: No such file or directory
Preparing the project to be executed
It seems like references to code that's present in https://github.com/Hub-of-all-Things/hat-database-schema - but this isn't a submodule of the HAT2.0 repo, and even then, the HAT-V2.0.sql file doesn't seem to be present in any repo?
Continued from "Trying to run the project, getting an error " #36
Hi AndriusA. I think the startup page I am receiving is an error page "Hat not available". The two buttons on the page link off to https://hubofallthings.com/main/what-is-the-hat/ (file not found) and https://hatters.hubofallthings.com/ , and do not allow me to create a hat on my local system.
Is this an actual error page or the default front page? Looking at the page source I see html such as
<div class="error-logo">, <h1 class="error-title">, <h3 class="error-subtitle">, <h3 class="error-subtitle-action">
I am getting some warnings that I don't know how to resolve, about private keys.
How can I create a hat? For sure I am missing something.
[info] Exporting web-assets:hat
[info] Compiling 16 Scala sources and 3 Java sources to /home/don-w510-u/HAT2.0/hat/target/scala-2.11/classes...
[WARN ] [11/13/2017 09:09:01] [o.h.h.r.a.HatServerActor] Error while trying to fetch HAT bobtheplumber.hat.org:9000 Server configuration: Private Key for bobtheplumber.hat.org:9000 not found
[WARN ] [11/13/2017 09:09:01] [o.h.h.r.HatServerProviderImpl] Error while retrieving HAT Server info: Private Key for bobtheplumber.hat.org:9000 not found
Thanks, Don
You can currently continue to use an expired data debit key (or one whose startDate has not yet been reached) to retrieve the bundle contents.
DataDebitAuthorization.scala is checking if the recipient is the right one and if the data debit has been enabled. It should also check if the current date is between the data debit startDate and endDate.
As an example, user might want to save their FB post record with one of the values being a list of likes. Currently storing such a record is cumbersome. It requires the creation of additional "likes" data source model and manually referencing each "like" record inside the "post" record.
The copyright year for the Claim your HAT email is not updated
Current message:
Copyright © 2019 Dataswift Ltd,
Expected message:
Copyright © 2020 Dataswift Ltd,
To simplify the underlying data model (not the conceptual one), consider merging all entity data tables (person, thing, event, location and organisation) into one, differentiated by kind
, simplifying api programming and reducing the number of tables
I just deployed HAT2.0 using the provided Docker scripts (thanks for those, by the way, this makes installation so much faster), and when I tried to create a new data table (using POST API), I got this error:
{
"message": "Error creating Table",
"cause": "ERROR: duplicate key value violates unique constraint \"data_table_pk\"\n
Detail: Key (id)=(4) already exists."
}
The id mentioned increases by one for each request.
I believe this is caused by using a sequence to generate the PK field, but the boilerplate data that is inserted upon deployment does not increment the sequence. That boilerplate should be changed to refer to nextval() instead in the INSERT statements.
Encountered an error when handling GET /users/access_token:
Unexpected exception[IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern
Reproduced with postman but not via Rumpel. Status code: 500
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.