As from the invenio-talk forum on release 2 - November

We got:

how to get access to the codes like html, python files and etc.

Shall we document some sort of "module architecture", i.e. which modules are involved in RDM and who does what?

Following the installation docs and at the part of attaching a file to a record, the record generated is having a link to "files": "https://localhost/api/records/<recid>/files". Following this link the server responds with 404.

Note
Accessing the file directly through api, e.g https://localhost/api/records/<recid>/files/my-image.jpg works as expected.

Automate pushing releases to PyPi

Ideas:

• Title
• Description
• Side part with permissions/access information
• Cool DOI highlighted top right (?)

Support the usage of different:

• plugins (e.g. Storage via invenio-s3, OCFL).
• extensions (e.g. invenio-github).

Note: via CLI or invenio.cfg (+pipenv) manual configuration.

Support customizing UI using some templating mechanism (e.g. Jinja), from the templates folder created by the init command of invenio-scripts (inveniosoftware/invenio-cli#2).

Monthly (or other) webinars with setting up InvenioRDM or going through the Bootcamp guides.

Show a preview of the file on the record page.

Part of inveniosoftware/invenio-rdm-records#42

We are already using the latest invenio version v3.2.0a9.

• We can already start testing files upload (REST API)
• Wait for the release of 3.2
• Pin to 3.2 (using only bundles)
• Investigate possible inconsistencies in pinned versions.

The invenio-app-rdm module should ensure that InvenioRDM specific configurations are set.

This is a non-trivial issue because:

• there is no guaranteed configuration loading order among invenio modules.
• invenio-rdm-records defines an InvenioRDM record in part via configuration, but it may be loaded after or before the configuration it is meant to change thus not working.
• invenio-rdm-records used to forcefully set its configuration which would suffer from same problem but with invenio-app-rdm (this is fixed in inveniosoftware/invenio-rdm-records#39)

The simple solution is: #37
We could write an ext.py and re-implement flask/config.py::from_object, but I don't see any real benefit.

Currently we need to be in line with invenio_search and invenio_db versions. It would be nice if we could just specify invenio with different extras. At the moment that seems to be ignored by pip/setuptools, it checks that the dependency is installed and ignores the fact that extras are different.

It would result in a setup.py file much more condensed:

invenio_version = '==3.2'

extras_require = {
   'elasticsearch6': [
       'Invenio[elasticsearch6]{}'.format(invenio_version)
   ],
   'elasticsearch7': [
       'Invenio[elasticsearch7]{}'.format(invenio_version)
   ],
   'mysql': [
       'Invenio[mysql]{}'.format(invenio_version)
   ],
   'postgresql': [
       'Invenio[postgresql]{}'.format(invenio_version)
   ],
   'sqlite': [
       'Invenio[sqlite]{}'.format(invenio_version)
   ]
}

install_requires = [
   'Invenio[base,auth,metadata,files]{}'.format(invenio_version),
   'invenio-rdm-records>=1.0.0a7',
   'invenio-records-permissions>=1.0.0a5',
]

Following the wiki instructions and getting to the pipenv run invenio rdm-records demo part, the
command fails with:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "invenio-app-rdm"

It fails because the SQLALCHEMY_DATABASE_URI used is the one defined
in invenio-app-rdm's config.py:

SQLALCHEMY_DATABASE_URI = \
    'postgresql+psycopg2://invenio-app-rdm:invenio-app-rdm@localhost/' \
    'invenio-app-rdm'

and not the one defined in docker-services.yml and used when the database
was created. Therefore we need a way to set SQLALCHEMY_DATABASE_URI with the
correct credentials AND the correct host:
@localhost when in development on host computer and @db when in container.

invenio.cfg would be a perfect contender to hold this information, except it
needs to be placed in <virtualenv>/var/instance/ to be read. And that folder is
only created after invenio-cli build is run which is run after the
cookiecutter instance is ran in invenio-cli init!

So it would be nice to have a better solution than the manual steps provided
in the installation, but that solution is not clear right now.

Document (tutorial) how to use (install and configure) different plugins and extensions. This documentation should be part of the general docs repository.

On record page, show list of files associated with the record.

Part of inveniosoftware/invenio-rdm-records#42

Aggregate issue for implementing "record as a collection" in invenioRDM.

Related identifiers/URLs can be used to achieve most backend needs I believe.

As a depositor I want to have a low overhead way of creating a grouping of records as I am depositing (or later editing) a record. This "collection-record" is a record (DOI and all), but has special UI to highlight the records it links to (perhaps? - I would be happy with just regular UI).

Split future workshops/meetings in an optional training/hands-on part (for the first couple of days) and the second part is about planning/discussions/presentations.

As a depositor, I only want those who can see the list of files to see the list of files associated with a record. And only those who can read files can preview (they are already checked for download) them. This way the access rules I decided upon depositing my record are respected.

This would require:

• Changing base docker file image in cookiecutter-invenio-rdm
• Adding support through the modules? or just invenio-cli

Ideas:

• RDM logo
• Search bar collapsed to the top
• Aggregations (e.g. access_rights)
• Files + Records
• Files preview (?)

See how to translate a module

(we have purposefully deferred this task, but we need to not forget it)

As a user navigating the site, I want to have a consistent header / footer to help orient myself.
As a developer, I want to have 1 place to set the header/footer and know it is applied on all pages, so I am more efficient and address the above.

By default, header/footer should be the same across pages, right now it isn't.

Now we have:

record page

search page

(frontpage is different but that's fine)

The footer doesn't stick to the bottom of the screen if the screen content is too small, so it looks weird:

The footer should stick to the bottom in that case and should flow with the content normally if the content of the page is large enough.

See https://css-tricks.com/couple-takes-sticky-footer/

When imports are fixed around Invenio:

• invenio-oauthclient
• flask-babelex (theoretically fixed in 0.9.4)

• Resource type is not mandatory
• Resource type fails to validate with "Dataset" as instructed in the Wiki.

PDFs and fonts can be previewed in --containers mode but NOT in --local because of violation of content-security policies. We would like to be able to preview them in both cases from the get-go.

See frontpage.html

<link href="https://fonts.googleapis.com/css?family=Open+Sans:300,300i,400,400i,600|Oswald|Source+Code+Pro:400&amp;display=swap" rel="stylesheet">

See invenio_app/config.py and invenio_app/ext.py

Move Wiki instruction to central RDM documentation repository.

• Find technology for it (mkdocs, docosaurus, jekyll, etc.)

• Move current Wiki.

Dovecot (IMAP Server) has a nice feature. When something goes wrong it attaches a URL to error message. This URL goes to a dedicated page in their Wiki.

This dedicated page contains possible causes & solutions to the problem. Instead of leaving users on their own, Dovecot provides a nice starting point for troubleshooting.

Lastly, Dovecots error reporting language makes whole process much less stressful (e.g. Instead of "X didin't work, terminating" the use "Something went wrong and X failed. Con continue, must exit (for more information see )

via Hakan

Made to work in debug mode. Steps to follow:

1. Install invenio-app-rdm.

export FLASK_ENV=development
invenio-cli install
invenio-cli update
invenio-cli services
invenio-cli demo

2. In one terminal run:

export FLASK_ENV=development
invenio-cli run

3. In another terminal run:

export FLASK_ENV=development
invenio webpack run start

Starting implementing the changes in the correct places.

NOTE 1: Seems that on macos if you don't increase the depth of the traversed subdirectories changes are not detected. I have opened a ticket webpack/watchpack#141 to understand if this is something is known and fixed in the latest beta version.

NOTE 2: followSymlinks was added on watchpack (see webpack/watchpack#114). However, it is not compatible with webpack 4 (it cannot be set from there) and as stated here webpack/webpack#6845 it will come with webpack 5...

Credits also to @slint and @ppanero :)

Support loading/adding custom CSS

Switch to the new versioning schema:

• Change setup.py to not be alpha
• Update version to 0.0.1

@zzacharo Gitter is down so I guess we can try to communicate via a thread here (this is so hacky >_<)

I haven't encountered a CSP problem that prevents the search page from working in local mode as far as I can tell. I get:

And I can click around and the UI elements work. So far so good.

Check compatibility and perform necessary changes to comply with the latest changes in invenio-records-permissions made by @fenekku.

Related to inveniosoftware/invenio-records-permissions#2

Support loading custom configuration from the invenio.cfg file created by the init command (inveniosoftware/invenio-cli#2).

Make gradient (blue/orange) configurable using RGB/CMYK to give implementers an easy way to make their InvenioRDM look like their organisation

Let the file list of a records page be sortable (I think sorting by name, size and filetype/extension would suffice)

Ideas:

• RDM logo
• Product page gradient
• Search bar
• Features
• Partners

Basically copy the RDM product page.

This will be changed not only by us, but by partners, so not much engineering is needed here.

October

• Invenio-Access:
  • Pending PR on explicit-permissions
• Invenio-Records-Permissions:
  • Merge Policy config PR (G)
  • Release v1.0.0a3 (G)
  • Enabling sharing permissions PR (P to review, G afterwards) moved to Nov board
  • Metadata to deal with explicit permissions (Check RFC) moved to Nov board
• Invenio-PIDstore:
  • Extract rand string into independent base32-identifier package (G)
  • Use independent package in Invenio-PIDstore (BLOCKED until package released but PR there)
  • Release Invenio-PIDstore (G: BLOCKED by above)
• Invenio-RDM-Records:
  • Integrate Policy config from Invenio-Records-Permissions (G)
  • Merge non-issue fields PR inveniosoftware/invenio-rdm-records#21 (P+G)
  • Release v1.0.0a2 (G)
  • Integrate new PID from random strings. (BLOCKED until base32-identifier package is released)
  • Release (G)
• Invenio-App-RDM:
  • Integrate Invenio-RDM-Records release #17
  • Merge PR to remove config #14 (P)
  • Release v1.0.0a2 (P)
  • Create demo CLI to populate DB with demo data records (G)
  • Release v1.0.0a3 (G)
• Cookiecutter-invenio-rdm:
  • Bump versions of the Invenio-App-RDM inveniosoftware/cookiecutter-invenio-rdm#6 (P)
  • Remove setup.py file inveniosoftware/cookiecutter-invenio-rdm#6 (P)
  • Remove scripts folder
  • Add static folder for logo inveniosoftware/cookiecutter-invenio-rdm#6 (P)
  • Release
• Invenio-CLI:
  • Integrate release of Cookiecutter-invenio-rdm inveniosoftware/invenio-cli#23 (P)
  • Fix foreground containers bug inveniosoftware/invenio-cli#23 (P)
  • Release
• Docs:
  • Document changing logo customization https://github.com/inveniosoftware/invenio-app-rdm/wiki#customization

Like what was done in cookiecutter-invenio-rdm, organize the configuration to make it usable as a reference and make sure things are well set and organized.

As a hoster, I want to have a single point of configuration for both backend and frontend, so configuring my instance is simple.

Because the mechanism for config.py is pretty extensive, we want it to also drive the javascript configuration variables (config.js ... or something like that).

Document the implementation of the mechanism to support different plugins and extensions.

Stil Needed: details about adding custom configuration code to instance.

Deleting a record from the admin panel will delete only the json field from table records_metadata, and the references and other fields are not deleted.

see screenshot 1 for the admin panel delete icon.
After navigating back to the search page I am able to see the exact record in the list, see screenshot 2
once I click on record i get this error see screenshot 3

• 1
  

• 2
  

• 3
  

Add ability to fields influence other fields in the deposit form. E.g .funding or subjects limit embargo duration (min/max limit or fixes the duration)

via Hakan

I have tried to upload a pdf file to a specific record. uploading worked fine 👍 , i can also download the file from the record.

BUT the issue is viewing the pdf file.

console error

[2020-01-09 15:28:44,376] WARNING in views: Preview failed for test.pdf, in recid:d46f9-vz793
raise BuildError(endpoint, values, method, self)
werkzeug.routing.BuildError: Could not build url for endpoint 'invenio_records_ui.recid_files' with values ['filename', 'pid_value']. Did you mean 'invenio_records_ui.recid_file' instead?

screenshot from postman:

screenshot from record page.

Wireframes for

• Search Page
• Deposit/Edit Page
• Curation(?) Page

so they can be discussed at the January meeting.

We need to integrate react in some of invenio-app-rdm UI templates. More specifically, the search page should be a react application that uses https://github.com/inveniosoftware/react-searchkit. In this fashion we need to provide some customisation in the level of:

• React templates (e.g search results)
• Jinja search templates (e.g customised search header template)

The solution to achieve the above involves the following PRs:

• inveniosoftware/invenio-assets#105 . Webpack config @templates alias to dynamically import react templates from an instance.
• inveniosoftware/react-searchkit#91 . External search triggering through a new introduced event listener. With this we can split the search page in a jinja template that can render to react applications:
  • SearchBar application which renders a search bar and is able to trigger a search through the event listener
  • Results page where the results are displayed along with facets, pagination, sorting, etc.
  • With this splitting the users of invenio-app-rdm can configure easily their search header
    layout and still having an out of the box search interface. SearchBar can be included in their
    custom template by just using a macro function.
• inveniosoftware/cookiecutter-invenio-rdm#42. Introducing 2 theme folders that users can
  override css for the main and search page accordingly.
• #62. This introduces:
  • New search UI module with the 2 before-mentioned react applications (SearchBar, Results)
  • Adding react templating support for overriding the react presentational components.
  • Adding overridable jinja templates for the search page.

We have (or will have at time of writing) merged search integration and theming code here, but that code would be helpful to wider community by being in invenio-search-ui. It would also reduce the amount of custom code in this repository.

As a depositor/curator, I want to group records under a community to

• manage permission access to the records
• provide OAI-PMH feed of records in the community

Can be seen as a more heavy-weight version of #2

Enable markdown commenting. Maybe something like Reddit's Fancy Parts/Markdown combo edit can be implemented.

via Hakan

Customization is a pretty big topic and this issue just lays it all out (what we extracted so far). Other issues will do further structuring and address the specifics.

Levels
We attempt to address customization at increasing level of flexibility at the cost of increasing level of expertise needed. You can gradually customize an instance by:

1. setting configuration values -- least effort, no customization beyond what developers have marked as customizable.
2. installing more modules (and setting their configuration values) -- need (very little) knowledge of command line
3. creating your own modules and installing them -- more effort and expertise, but more flexibility
4. adding code to your instance (do we even want to go there, or just tell people to fork at this point?) -- much more effort and expertise, but even more flexibility
5. forking (you do you!) -- most effort, expertise and maintenance cost for most flexibility

Customization points
Obviously this is an open source project, so one could theoretically customize anything by forking (see 5.) and going off. This is not recommended. We want to have clear configuration points that address clear configuration needs.

• config,py and invenio.cfg are used for tier-1 customizations mostly. invenio.cfg is needed/helpful for differing context: local development, production, staging as well as secrets/API keys, while config.py is helpful for settings common across contexts and non sensitive values. Using environment variables should be possible too (invenio-config does it all!). Configuration files for deployment also fall in this bucket.
• loaded data. Customization can be achieved by loading + using different controlled vocabularies for otherwise fields (or loading roles for instance). Usually the size of the controlled vocab explains why config.py is not used. CLI tool is needed out of the box with some means of letting caller configure how things should be loaded. (tier 2 if user is doing it, tier 1 if provided by choice of profile). Examples: resource types, terms, relation_types
• placing templates / static files in predefined locations. To customize look and layout, custom stylesheets and html templates can be placed in locations where InvenioRDM will pick them up. (tier-4 or so)
• data model extensions. This should be achieved either through a module containing the extensions or potentially placing the extension file somewhere. Profiles will need to be able to provide these customization.
• dynamic admin controlled customizations. It should be possible to enable / disable features, add / remove db entries from the admin to customize the instance. For instance, role assignation, static page creation, feature toggle.

Profiles
As the person in charge of installing an InvenioRDM instance, I want to select a "profile" that has a set of customizations more appropriate to my domain. Examples are "BioMedical" or "Physics" (better names needed). These profiles are available out of the box and you can create your own as a module (see 3.)

What can be customized

• choice of profile
• choice of controlled vocabularies
• layout and look (css and html) - at tier 1. through some configuration values, at tier 3 or 4 to make your own
• roles and permissions
• data model (at high cost to prevent chaos 😅 )
• static pages like the frontpage
• the "collections" strategy
• deployment configuration
• data backing volumes / hosts
• installed modules
• citation styles
• licenses

Documentation
Documentation to explain our approach to configuration should be written. Documentation should be placed at different locations with different audience in mind: in module for developers and on InvenioRDM website for less technical user.

Test that the integration of the different modules works for the different use cases in the scope.