While trying to validate our installation profile we encountered an error and I believe that the profile.xsd is missing

<xs:element name="description" type="xs:string" minOccurs="0" maxOccurs="1"/>

after Line 153 to allow <desctiption> tags for listItemLabels:

After adding the element, our XML Profile validated fine. See https://gist.github.com/karbecker/acbee20f2e2290b23fed8cef5e68cd8f#file-profile-xsd-L154

Hi!

I added a media screen in one of my custom occurrence type, but when I load a picture (.jpeg) and I try to visualize it I got this error message (it seems that try to find the representation from objects and not from occurrence):

Errors occurred when trying to access /index.php/editor/objects/ObjectEditor/GetRepresentationInfo/object_id/0/representation_id/98:
Invalid input

Why I can't find the GetRepresentationInfo method in occurrences/entities editor controller?

Thanks!

Creating a user with an e-mail address like

commander.o'[email protected]

results in an "Invalid email address" warning and the user cannot be saved.

Application version 1.7.6

See RFC3696:
https://tools.ietf.org/html/rfc3696#page-5

The JSON response from the service API returns the wrong JSON type for any part of the dictionary that's meant to be a JSON object if the object is empty.

For example, say list items are being requested with returnAllLocales: true for both preferred and nonpreferred labels. For an item with preferred labels and no nonpreferred labels, preferred results are returned like this (which is correct):

{
    "preferred_labels":
     {"1319":{"2":["Français"], "1":["English"]}},

...but the nonpreferred results are returned like this:

{
    "nonpreferred_labels":[],

Returning an array instead of a JSON object makes the response inconvenient to use in a language where arrays and objects/dictionaries are distinct types, since trying to index an array with anything other than an integer raises an exception.

I assume the bug here comes from the fact that, in PHP, arrays and associative arrays are the same type and the JSON serializer makes the wrong assumption about what to serialize an empty array to in this case?

I tested this on CollectiveAccess 1.3 - I'm afraid I haven't had the chance to set up a 1.4 installation (or an installation from git HEAD) to see if it's still an issue there.

Since ImageMagick 6.7.5-5 (released early 2012), the colorspace "sRGB" has to be used instead of "RGB". (https://www.imagemagick.org/discourse-server/viewtopic.php?t=20501). If RGB is used with newer versions, images appear too dark (see https://collectiveaccess.org/support/index.php?p=/discussion/299423/providence-v1-3-now-produces-images-either-too-light-or-too-dark)

I changed

array_unshift($va_ops['convert'], '-colorspace RGB');

to

array_unshift($va_ops['convert'], '-colorspace sRGB');

on https://github.com/collectiveaccess/providence/blob/master/app/lib/core/Plugins/Media/ImageMagick.php#L1273

It probably makes sense to make this configurable somewhere as long as there are still systems with older versions of ImageMagick in use.

The values for those two config options remove any limits from uploads/posts, so should not report an error.

There is an unresolved conflict starting at line 79 in themes/default/views/editor/collections/summary_html.php on the HEAD of master. This will need resolving.

I installed CA last week, practiced with for a few days, then emptied the database and deleted the files so I could re-install with a different Profile. It didn't work. I've also tried dropping the table and recreating it; in either case, SHOW TABLES FROM redacted returns an empty set.

The installer gets as far as /redacted/install/index.php?email=me%40redacted&profile=spectrum&page=2, then returns this error:

<b>Fatal error</b>:  Uncaught DatabaseException: Can't create table `redacted`.`ca_locales` (errno: -1 &quot;Internal error &lt; 0 (Not system error)&quot;) in /redacted/app/lib/core/Db/mysqli.php:316
Stack trace:
#0 /redacted/app/lib/core/Db/DbStatement.php(151): Db_mysqli-&gt;execute(Object(DbStatement), Object(DbStatement), '/*=============...', Array, NULL)
#1 /redacted/app/lib/core/Db.php(270): DbStatement-&gt;executeWithParamsAsArray(Array)
#2 /redacted/install/inc/Installer.php(484): Db-&gt;query('/*=============...')
#3 /redacted/install/inc/page2.php(69): Installer-&gt;loadSchema('caGetTableToBeL...')
#4 /redacted/install/index.php(152): require_once('/redacted...')
#5 {main}
  thrown in <b>/redacted/app/lib/core/Db/mysqli.php</b> on line <b>316</b><br />

The relevant log appears to be /var/log/mysql/error.log, which contains one of these per attempt:

InnoDB: Error: cannot load foreign constraint redacted/fk_ca_data_exporter_labels_locale_id: could not find the relevant record in SYS_FOREIGN

The query that appears to trigger the error is the first in schema_mysql.sql, i.e. "create table ca_locales". I also can't get the query to run in PHPMyAdmin, even when logged into PHPMyAdmin. Changing the name of the table in the query, e.g. to calocales, enables the query to run.

I've used git to diff last week's branch with the current HEAD, and found no significant changes; I also can't install from that branch. I have created a phpmyadmin table and user.

What am I doing wrong? What do I need to do to fix this (or what terms should I be searching for)?

We are facing an issue while removing items from a set in providence version 1.4 (with solr). Whenever an item is removed from a set it still appears in the search results for that set. We investigated this issue and found that item is correctly removed in the database however solr index is not updated. Which means when we make a search for a set after removing its item(s) it still gives results based on old solr index.

We also observed that in object editor if we ‘unlink’ an object from a set and at the same time change any other field of that object (for example object name) the issue is solved.
Which means solr indexing is performed when we update other information of an object but not when we solely unlink it from a set.

The problem is caused by the null value of the ‘$this->_FIELD_VALUE_CHANGED’ variable (‘update’ function in BaseModelWithAttributes.php). This variable is passed to ‘indexRow’ function (SearchIndexer.php). Where indexing is only performed if changed fields (‘$this->_FIELD_VALUE_CHANGED’) are not null. In case of removing items from a set ‘$this->_FIELD_VALUE_CHANGED’ is always null, hence no indexing is performed, leaving old index on solr.

On a fresh installation of Providence v1.4, I've tried uploading a DublinCore "Moving Image" object.
Although "queue_enabled" is set to "1" in "app/conf/global.php", according to the installation documentation, the browser waits forever, while "Magick" creates a 12 GB tempfile in "/tmp".

I guess my setup must be faulty, or is this the normal behavior?
No sarcasm here. I'm seriously asking :)

If so, could it be changed? We are planning to use it with lossless FFV1/PCM/AVI files of up to 4 hours (=approx. 100 GB per video), so I assume the memory consumption would be immense.
It works perfectly fine to transcode the videos using FFmpeg on an average PC with 4 GB of RAM and no swap configured.

The video used was from FFmpeg's sample collection:
http://samples.ffmpeg.org/benchmark/testsuite1/matrixbench_mpeg2.mpg

Many thanks and regards,
Peter B.

Hi,

Our CollectiveAccess has untranslated phrases, even though we translated nl_NL. If we compare our nl_NL with the English version we see a difference of strings and some strings are commented out. Could you please send us an updated version of nl_NL so that we can translate the missing strings.

An example:
"Show current location as 'breadcrumb' trail" is not translated in our version, because it is commented out in the Dutch and English version.

Kind regards,
Evy

I'm planning to translate some profiles (dublincore, base) where possible to German.
How could I commit my changes?

I guess, it could be handled like code-changes, so should I use git pull-requests or patches?

Regards,
Pb

While importing data in Providence, I am seeing repeated output of the word "primary" at the bottom of the screen.

Steps to reproduce error:

1. create data import mapping.
2. import xls data in Providence interface

While importing data on the ca_objects.extent field I realized that there is a mismatch between the metadata element description for ca_objects.extent in ISADG installation profile and the one described in the Object model PHP code.

Here it is the definition from the ISADG profile, the type is Text:

The description of the field in the model PHP code is for a number:

Negative numbers fail to validate when the datatype for a field is set to 'integer', irrespective of the minimal value allowed for the field. Negative 'numeric' values are valid, but there is no way to restrict to whole numbers.

Hi,

I am running an instance of Providence to evaluate it for my organization.

I am running it using the task queue, and I found that as an administrator I need more detail about the status of the queue. As of now the only available info is using the pending task widget on the dashboard.

I had a look at the current command line tools, but there is no one for listing the queue. Just to avoid overlapping. Is there anybody planning to develop it?

Thank you in advance!

Hi!

We noticed (we think we noticed) another error in CA’s OAI PMH Service.

When using the listRecords verb we need to be able to limit the response by modification date. Using &from=2018-05-23&until=2018-05-24 does not work, we instantly get back object records with <datestamp>2018-05-16T09:39:48Z</datestamp> and so on.

We noticed that in OAIPMHService.php in Line 512:

'limitToModifiedOn' => $vs_range, gets passed on to SearchEngine.php where in turn limitToModifiedOn appears only once in a commented help line. Nevertheless, filtering by modification date does not work at our end and it would be great if you could look into that as it is need for a functional OAI-PMH interface.

Thanks!

I've noticed that the duration of a video is displayed twice, but mismatching in the file representation view.

For example, the FFmpeg sample video:
http://samples.ffmpeg.org/benchmark/testsuite1/matrixbench_mpeg2.mpg

In the overview, the duration is shown as: "1 minute, 27 seconds"
In the media metadata block, it is shown as: "3mn 7s"

The second one is correct.

System: Providence v1.4 on Debian 7 (Wheezy) 64bit

Steps to Reproduce

1. Login as user with administrator privileges
2. Go to Manage > Access Control, this shows the "ListUsers" page which defaults to showing "full-access" users
3. Change the dropdown at the top to "deleted"

Expected Result

List of deleted users is displayed.

Actual Result

Dropdown switches back to "full-access" and the list of full access (not deleted) users is shown.

Impact

This makes it very difficult to find a deleted user. Once a user is deleted they go into a black hole and can only be recovered by guessing URLs (i.e. go to user 1 and increment the ID in the URL until the right user is found).

When importing records with the merge_on_preferred_labels; existing record policy, metadata elements that are repeatable end up being duplicated regardless of the fact that the contents are precisely the same. If the existing field and the importing field were checked as strings, they would be seen as equal.

I have been reviewing the Media Plugin code and I think it requires a complete refactor with regard to file checks, to make sure checks are run only once, and then cached, in a static variable, so you can use it on every next request to checkStatus.

As per the maintainer, that code was written back in PHP 4.x times and updated piecemeal over time, so we are certainly all for working on it.

The actual list of media items only becomes visible, all at once, after each of the images is loaded and the document's onload event is fired. This can take quite a while when you have 100+ items, even when everything is in the client's cache. During the time the page is not accessible to the user.

As all the thumbnail sizes are already pre-set in the HTML rendered by the backend, I don't see why it wouldn't be possible to do the layout already on DOMContentLoaded aka $(document).ready, before the images are loaded.

As current master, the event hooks for onload seem to be in:
video.js:1857
swfobject.js:5
ckeditor.js:6

I get the following error on install:

Directory providence/vendor/ezyang/htmlpurifier/library/HTMLPurifier/DefinitionCache/Serializer not writable, please chmod to 777 in providence/vendor/ezyang/htmlpurifier/library/HTMLPurifier/DefinitionCache/Serializer.php on line 297

It seems like HTMLPurifier should be writing its cache file to app/tmp/ instead of inside vendor/. Also, the recommendation for 777 permissions is not secure.

Using the latest develop branch code.

I am importing records with merge_on_idno_with_replace on but records are not updated.

I did the following. For a clean default profile installation I run an import of some records using a mapping. I then modifiy the mapping to set an existing field to a constant value, but records are not updated.

I am attaching the XLSX for the data and for the mapping.

Looking at the logs, there are a lot of % around, and I trace the code to here

and it looks like the code to collect the existing record is never executed, since variable $vb_idno_is_template is true.

I am also attaching logs.
log_2020-04-19.txt

Will you please have a look at it? I am afraid the latest changes introduced this regression.

nomenclator-import.xlsx
poblenou-places-10.xlsx

example :
elasticsearch/elasticsearch

• composer.lock : version 2.1.5
• actual version in vendor dir : 2.3.0

Me as a developer just executes composer install when I see a lock file, not really noticed the vendor directory. Had me looking for hours why I had certain errors.

We have to fix the composer.lock file. But I would also really like to completely remove the vendor directory. But I guess the reason why it's here in the first place is for sitebuilders. We can also just add a composer.phar and run php composer.phar install from the install script.

I'll open up a PR for the lock file.

We also need to change the format so more than one path can be specified, and each can be tried in turn. This way we can ship with all common paths specified and save the hassle of having to tweak it. For example: the Ghostscript gs executable may be in /usr/bin or /usr/local/bin on Linux depending upon how it was installed (Package vs. source)

Supporting multiple paths is probably best done using a configuration list. Eg. what we have now:

ghostscript_app = /usr/bin/gs

becomes

ghostscript_app = [/usr/bin/gs, /usr/local/bin/gs]

Hi,

Using code from develop branch.

I am moving an element on the place authority, and I get an unformatted string, like

Will be moved under %1 after next save.

See attached picture for an example:

The URL is:
http://localhost:8080/index.php/editor/places/PlaceEditor/Edit/Screen59/place_id/2

It is not clear to me if it is something related to locales or some bug in the code.

I also noticed that when I create a place in the place authority, I fill in the "place identifier" field, but after sending the form, the field is empty :( See attached image for the data in the form

And this is after sending the form, the "place identifier" field is empty:

The point is that I also realized there are some other %1 appearing on logs, for example, when running a data import using a mapping, with full DEBUG log, you will find lines like these:

log_2020-04-14.txt:2020-04-14 16:24:04 - DEBUG --> Finished building content tree for % at    0.1004 seconds [10]

I am sure that this message in the past was showing the name of the entity (idno) properly on the place of the % sign. See this log I had from previous versions:

log_2020-04-14.txt:2020-04-14 18:54:28 - DEBUG --> Finished building content tree for 0000000029 at    0.0093 seconds [29]

The date is not relevant, since I use to merge from the upstream branch some days after it is updated.

So I think it is something on the latest code, so I reviewed the commits, but I didn't find the reason.

I don't know if those three issues are connected, but I would appreciate some feedback.

Regards.

We noticed that if you add a lot of metadata within a screen there is a rendering problem that makes the browser terribly slow. The problem seems to be tied to the creation of the tooltip that is handled with a call to jquery for each metadata (we usually have 100/150 metadata elements). A workaround could be to disable the tooltips but the problem persists.

Dave

In the wiki appears a special sources named: filename and filepath, but: 1. Its impossible to put that special source in the mapping xlsx because it has to be in the same "source" column where i have to put the number of the Data xlsx where the path/to/file is

I have the latest develop code in a testing instance. I realized that I was having some issues with ImageMagick and GraphicsMagick and I reviewed the source code to track them.

I have it configured to use the queue to process large media (>500kb). My hosting was overloaded and it killed ImageMagick or GraphicsMagick.
I realized that the exec calls to ImageMagick and other external commands are not usually checked for the command (*nix) status code nor the expected output.
I am working on a patch for it, a function caExec that will perform some checks.

Hello

There's a little bug on caMediaPluginImageMagickInstalled function, there's a regexp check on image magick path : https://github.com/collectiveaccess/providence/blob/master/app/helpers/mediaPluginHelpers.php#L93

preg_match("/[^\/A-Za-z0-9\.:]+/", $ps_imagemagick_path)

It should be less restrictive, e.g. it is not possible to use - or _ or even in path...etc

Thanks

I've made a fresh install of Providence v1.4, and set the locale to "de_DE".
In the web-interface I then get a mix of the following languages:

• English (expected, due to not-yet-translated strings)
• German
• French
• Swedish

The French/Swedish texts seem to originate from the profiles "dublincore.xml" (French) and "base.xml" (Swedish).
It seems like the language taken from the XMLs is the last one listed as translation, even if the chosen locale (de_DE in my case) doesn't match.

Here's a screenshot with all 4 languages:

Hi,

I am preparing to create the ca_ES locale and I would like to know how do you create the POT file. My intuition is that the app/locale/en_US/message.po is the POT file, but not sure about it.

I read the documentation at https://docs.collectiveaccess.org/wiki/Creating_a_Translation , but it does not give details, it only mentions gettext tools, but not the command to generate them, nor a script to run it.

I started from the es_ES translation, and I would like to update the strings, since I have seen that there are new commits like the one at 3a81b17 (see comment 3a81b17#r36758368).

Will you please advice on how to update es_ES with these changes?

While running application for the external_applications.conf file, I found that my shared hosting platform needed to limit the resources of some of the commands.

I propose a new feature to allow configuring additional parameters on external_applications_conf, that will allow additional parameters for commands like ImageMagick convert, identify, and the like.

The configuration will be something like for commands convert and identify on ImageMagick:

imagemagick_convert_args=-limit thread 1
imagemagick_identify_args=-limit thread 1

What do you think?

Right now on master, loading the media page of an object fires 43 HTTP GET requests to load various JavaScript files, and I believe this will be the same for other pages in providence. I'd love to have a straightforward way to bundle these files into a single, minified file to improve page load time.

I see that the develop branch did away with the app/conf/javascript.conf file, so not sure if this is still relevant. But if you can give me a idea how to do this cleanly, I'd love to give it a shot in master.

Fresh install, getting:

Parse error: syntax error, unexpected 'class' (T_CLASS), expecting identifier (T_STRING) or variable (T_VARIABLE) or '{' or '$' in /var/www/html/app/lib/core/BaseModel.php on line 11277

Hi

I am developing (from develop branch) using Vagrantfile configuration.

I am continuously getting HTTP 500 Errors. I think they are related to the cache, but I am not familiar with it, so I don't know how to deal with them. It is not clear to me if the issue is related to something on my configuration or a latent bug on the develop branch.

Here it is a sample of the error on the /var/log/apache2/error.log file:

[Wed Jan 08 00:55:26.163389 2020] [php7:notice] [pid 1801] [client 10.0.2.2:58258] PHP Parse 
error:  syntax error, unexpected ''] = 1578444894;' (T_ENCAPSED_AND_WHITESPACE) in 
/vagrant/app/tmp/collectiveaccessCache/0fea6a13c52b4d47/25368f24b045ca84/38a865804f8fdcb6/57cd99682e939275/efd02694d21a801f/598ce1b6114171fe/70a527c70322dbf4/27d77c5ed98ce3de/aee6230f3abaa91f/e90ef2a0a8408df7.php 
on line 35, referer: http://localhost:8080/index.php/batch/MetadataImport/ImportData/

When it happens I run a clear-caches maintenance command and I have to reauth and run the command again, and it is solved. It is very annoying and time consuming, since it happens frequently.

Will you please advice on how to try to solve it?

I hereby attach the app/conf/local/app.conf file:

app.conf.txt

Thank you in advance.

I tried to implement providence in a VM using Vagrant and VirtualBox on my Windows machine, but it doesn't appear to work. The Vagrantfile is around 2 years old, are you no longer supporting Vagrant?

I'm in the middle of writing up a python wrapper for the API -- so my proficiency is certainly not php! -- and I came across an issue where only the display name of a label will return. (i.e. forename, middlename, surname, etc. all return null). Since I am requesting the api via "format=edit" for the purposes of making a tweak and sending back the changes, currently working this way erases the other fields. I confirmed this issue on both my setup and the CA demo.

If I'm not mistaken, adding "assumeDisplayField" => false to the two $va_labels pulls in the getItemInfoForEdit() function in ItemService.php should do the trick! I'm not sure how many people rely on the web API, but figured I share in case it could help.

I recently exported a configuration for moving to a fresh installation. I discovered that the XML produced by the configuration export tool is invalid. There are 2 main problems I found.

1. The content of element 'labels' is not complete. One of '{label}' is expected.
   
2. Attribute 'code' must appear on element 'restriction'.
   
3. Invalid content was found starting with element 'types'. One of '{type}' is expected.
   

Hi,

In order to contribute to this project I would like to avoid committing code that will not follow the coding standards of the project. I am using PHPStorm as of now and it reformats files and changes code layout.

Do you have any beatifier or template so I can automate code formatting?

I am using PHPStorm as of now, but I am open to try any other environment.

Regards.

The current file in master says 1.7.7 should it be 1.7.8?

Hi!

We recently noticed an error in the OAI PMH iterface. We worked out a Xepicur Mapping where the node epicur has a namespace associated with it. Having changed nothing, our xmlns from the epicur node was automatically moved to the metadata node of each record element:

This is not a valid XML as we need it.

We looked into OAIPMHService.php and after moving Line 592:

See https://gist.github.com/karbecker/adf65d668d512bc39f127c33562815c0

Please look into this and if I’m right feel free to update CA’s code. I hope this helps somone who has the same issue!

I've been profiling providence for a couple of days now, trying to find a way to increase performance of a search that requests a lot of related data. Now I've noticed that caGetOption is getting called a lot. And when I say a lot, I mean, A LOT. 100k function calls doesn't seem very good to me. And a lot of these calls, are just to get some data from an array that the method that calls caGetOption, already has.

Opinions?

Your project is very interesting, unfortunately the demo is not working anymore and needs to be updated: https://demo.collectiveaccess.org/

Hi, I tried to clone the repo, but getting an error when git tries to access the jquery-handsontable sub-repository since it isn't public.

Looking for fonts in:

/themes/default/css/din/28EC2C_0_0.woff

instead of:

/themes/default/css/fonts/28EC2C_0_0.woff

[PROV 1.5.1] : I have notice that in manage->administrator->configuration check - the "Application Version" is not correct, it shows 1.5 instead of 1.5.1.

Loading the media page of an object with 273 representations takes 11.2 seconds wall time on the server for rendering the HTML alone. Of this time, 5.4 seconds alone is spend inside a single call to caProcessRelationshipLookupLabel, which itself fires 1644 calls to SearchResult::get (presumably SQL queries). I am no expert in the nitty-gritty details of providence, but this seems a bit excessive and slow to me, given what information the rendered HTML contains, and how crucial this page is to day-to-day operations inside providence.

Happy to try out patches, etc!

Best