Git Product home page Git Product logo

arche-localhost-utils's Introduction

arche-localhost-utils

A bunch of shell scripts to ease working with arche locally

i don't want to type sudo all the time

on default you can execute docker related commands only with sudo; to change this you have a look at https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user (TLDR, create a docker group and add you user to it; restart the system)

how to install

  • Clone the repo
  • One time: run ./init-repo.sh to create needed directories
    • answer VOLUMES_DIR env var is not set, please provide the installation directory: with .
  • To (re)start the continer run ./docker-start.sh
    • Be aware full repository initialization at the first run (particularly external vocabularies import) may take a lot of time. Anyway once # Running 40-updateVocabularies.php initialization stage is reached you may safely start using the repository without further waiting.
    • This repo ships with a database dump to skip avoid initial download of external vocabulaires
    • To stop the container run docker container stop acdh-repo
  • To remove all data run ./reset-repo.sh
  • To remove everything just delete the current folder

enter a running container

docker exec -it -u www-data acdh-repo bash (or run ./enter-repo.sh)

check initscript logs

  • inside the container (see above), look around ls
  • to inspect initscript logs run tail -f log/initScripts.log

the GUI

you should mainly focus on ARCHEs API, but you can find the ARCHE-GUI at http://localhost/browser/

local PHP setup

  • either install all needed things; see information about the needed PHP setup to run ingestions and file checks can be found here

  • or user docker

how to test

for the following steps make sure you are in the testing directory (cd testing)

ingest metadata

docker setup

  • first create and enter a php/arche container by running ./enter_php_container.sh
  • you are now in the mounted testing directory (which inside the docker is called data); if you look around (ls) you should see the same files as in the host`s testing repo.
  • run ./metadata_ingest.sh

composer setup

run filechecker

docker setup

  • first create and enter a php/arche container by running ./enter_php_container.sh
  • you are now in the mounted testing directory (which inside the docker is called data); if you look around (ls) you should see the same files as in the host`s testing repo.
  • run ./filechecker.sh
  • check the results in testing/fc_out/{datetime-of-last-run} (e.g. cd into the directory, start python dev server python -m http.server and open the printed URL)
  • spoileralert: 2/3 files did not pass the test!

composer setup

  • run ./filechecker.sh
  • check the results in testing/fc_out/{datetime-of-last-run} (e.g. cd into the directory, start python dev server python -m http.server and open the printed URL)
  • spoileralert: 2/3 files did not pass the test!

ingest binaries

docker setup

  • first create and enter a php/arche container by running ./enter_php_container.sh
  • you are now in the mounted testing directory (which inside the docker is called data); if you look around (ls) you should see the same files as in the host`s testing repo.
  • run ./binaries_import.sh

composer setup

  • (well you actually shouldn't do that, because they didn't pass the filechecker)
  • run ./binaries_import.sh
  • see repo-file-checker for some documentation

add new dump

  • remove everything and clone the repo again
  • comment # cp dump.sql -d ${VOLUMES_DIR}/data in init-repo.sh
  • enter container ./enter-repo.sh
  • change user su www-data
  • create dump pg_dumpall -f data/dump.sql
  • leave container
  • copy data.sql into repo root

arche-localhost-utils's People

Contributors

csae8092 avatar fsanzl avatar zozlak avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

Forkers

ofmgraz

arche-localhost-utils's Issues

Vocabularies not working

Problems with ingestion. This is the output of trying with the sample provided with default config on a fresh-cloned tree:

--------------------
--- Exception backtrace:
acdhOeaw\arche\lib\ingest\IndexerException Object
(
    [message:protected] => There was at least one error during the import:
.	https://id.acdh.oeaw.ac.at/wollmilchsau/Formats: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasLicense value https://vocabs.acdh.oeaw.ac.at/archelicenses/cc-by-4-0 is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_licenses/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/project-wollmilchsauzucht: Server error: `POST http://127.0.0.1/api/metadata` resulted in a `500 Internal Server Error` response:
SQLSTATE[23503]: Foreign key violation: 7 ERROR:  update or delete on table "resources" violates foreign key constraint  (truncated...)
(GuzzleHttp\Exception\ServerException)
	https://id.acdh.oeaw.ac.at/wollmilchsau: Server error: `POST http://127.0.0.1/api/metadata` resulted in a `500 Internal Server Error` response:
SQLSTATE[23503]: Foreign key violation: 7 ERROR:  update or delete on table "resources" violates foreign key constraint  (truncated...)
(GuzzleHttp\Exception\ServerException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasLicense value https://vocabs.acdh.oeaw.ac.at/archelicenses/cc-by-4-0 is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_licenses/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/Metadata: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasLicense value https://vocabs.acdh.oeaw.ac.at/archelicenses/cc-by-4-0 is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_licenses/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/ACDH-DepositionAgreement.docx: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/text is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/ACDH-DepositionAgreement_2.docx: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/text is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/Formats/FormatsACDH-layout.xlsx: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/dataset is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/Formats/FormatsACDH-layout.pdf: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/text is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/lifecycle.png: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/acdhUserFlow_v4.pdf: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/text is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/archiving-RL.pdf: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/text is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/Formats/formatsACDH.xml: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/text is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/65_2_1_shot1.jpg: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/65_2_1_shot1.xmp: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/oais.png: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/oais.tiff: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/Metadata/diagramOntology-cardinality.png: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/duck_sissi.jpg: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/sissiObjectLIDO.xml: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/EntenhausenPlanDetail.png: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/EntenhausenPlanDetail_annotation.ttl: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/Sebastian_Wallroth-Ente.wav: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/sound is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/kantharos.x3d: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/3dData is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/treeCookieCutter.ply: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/3dData is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/saint-graal.tiff: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/images/excalibur.svg: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/grossoelschraege.pdf: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/text is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/somemorespecialcharacters.pdf: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/text is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)
	https://id.acdh.oeaw.ac.at/wollmilchsau/title-image.jpg: 400 property https://vocabs.acdh.oeaw.ac.at/schema#hasCategory value https://vocabs.acdh.oeaw.ac.at/archecategory/image is not in the https://vocabs.acdh.oeaw.ac.at/rest/v1/arche_category/data vocabulary(GuzzleHttp\Exception\ClientException)

    [string:Exception:private] => 
    [code:protected] => 1
    [file:protected] => /home/fernando/Projects/testing/vendor/acdh-oeaw/arche-lib-ingest/src/acdhOeaw/arche/lib/ingest/MetadataCollection.php
    [line:protected] => 346
    [trace:Exception:private] => Array
        (
            [0] => Array
                (
                    [file] => /home/fernando/Projects/testing/vendor/acdh-oeaw/arche-ingest/import_metadata_sample.php
                    [line] => 69
                    [function] => import
                    [class] => acdhOeaw\arche\lib\ingest\MetadataCollection
                    [type] => ->
                )

        )

    [previous:Exception:private] => 
    [resources:acdhOeaw\arche\lib\ingest\IndexerException:private] => Array
        (
        )

)

I get similar errors with another ttl for hasLanguage.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.