artefactual / archivematica-fpr-admin Goto Github PK

It'd be easier if we had only one place where we define our dependencies rather than two - the latter is prone to errors.

It's become a standard in the AM team but this project is not using it yet.
For now, see .travis.yml to know more about how we're running the tes.
There is also a mention in DEVELOPMENT.md.

Related: artefactual/archivematica#584

It reads v1.6.0-rc1

1.7.5 version of the fpr-admin is still on 17.12

The FIDO format id command (defined here: https://github.com/artefactual/archivematica-fpr-admin/blob/stable/1.6.x/fpr/migrations/pronom84_idcommands_all.json#L36 )

references a file that is installed as part of archivematicaCommon:
https://github.com/artefactual/archivematica/blob/stable/1.6.x/src/archivematicaCommon/lib/externals/fido/archivematica_format_extensions.xml

This lists some custom formats. It was added to Archivematica in 2014, because there were disk image formats that, at the time, we not listed in the PRONOM registry.

Now with PRONOM v88, all but one of those custom 'archivematica-x-fmt/*' entries have proper PRONOM IDs.

The Fido command should be updated, so that it no longer references this file. The file can be removed from archivematicaCommon.

The only entry that is still needed is for 'raw disk image' since there is no PRONOM entry for this. it should be possible to just include the following xml snippet in the fido fpr command directly:

<format> <puid>archivematica-fmt/4</puid> <name>Raw disk image</name> <extension>dd</extension> <extension>dd.001</extension> <extension>001</extension> </format>

This problem is blocking JiscSD/archivematica#13

If a transfer contains two OCR-able files with the same name, then the resulting .txt file transcriptions will both be written to the same filename under metadata/OCRfiles/ and the second will overwrite the first. There will still be two File model instances in the db, however, and the result of this will be that createMETS_v2.0 will fail at MCPClient/clientScripts/archivematicaCreateMETS2.py", line 800, in createFileSec
f = File.objects.get(**kwargs) (where kwargs = {"removedtime__isnull": True, fileGroupType: fileGroupIdentifier, "currentlocation": directoryPathSTR}) because there will be 2 OCR text files with the same currentlocation attribute.

See https://projects.artefactual.com/issues/11651

While working on #63 we found that we had two formats and their corresponding formatversions that had the very same description. The latest one added took the same slug causing MultipleObjectsReturned exceptions.

I think that the slug fields for these two models should be updated so they ensure uniqueness. This is something that the old django-slugfield seems to support out of the box. Perhaps we can also move to Django's SlugField?

Migration 0004 runs loaddata, using pronom_88.json. That json file has new fprules defined in it, for example:

    {
        "fields": {
            "count_not_okay": 0,
            "replaces": null,
            "count_attempts": 0,
            "uuid": "dfc4197d-d265-4eac-839d-b267d4ac4a9f",
            "format": "dc76c66b-8be8-45f4-a8c8-7d40a47a5b52",
            "lastmodified": "2016-12-14T14:04:19Z",
            "enabled": true,
            "command": "80314f3c-5d48-4ad1-a9f2-2c0f7c7b229d",
            "purpose": "characterization",
            "count_okay": 0
        },
        "model": "fpr.fprule",
        "pk": 712
    },

Each new instance of the fprule model defines a pk to use. If the database already has a row with that pk used, then the loaddata method tries to overwrite it. If that row has a uuid that is referred to in a replaces_id property of some other row in the database, the upgrade fails on an integrityError.

It is not necessary to pre-define the pk values. Omitting them works, the database will assign new keys automatically, and avoid this problem.

It would be better to avoid the use of loaddata altogether, ( see #38 ) but removing the pk's will fix the problem of not being able to upgrade Archivematica from 1.5.1 to 1.6.0 when the fpr has been customized.

Now: fpr/templates/...
Should be: fpr/templates/fpr/...

Because the root of templates/ is for project templates! More here: https://docs.djangoproject.com/en/1.8/intro/reusable-apps/. Our current approach is to symlink in the consumer... wrong! We can use django.template.loaders.app_directories.Loader to take care of that for us.

We should do something like:

from django.core.urlresolvers import reverse

return '<a class="revisions_link" href="{}">Revision history</a>'.format(reverse('revision_list', kwargs={'entity_name': revision_type, 'uuid': object_uuid}))

instead of:

return '<a class="revisions_link" href="/fpr/revisions/' + revision_type + '/' + object_uuid + '/">Revision History</a>'

This is causing the following issue:

After running manage.py migrate from the Dashboard on a fresh database:

 Applying fpr.0001_initial... OK
  Applying fpr.0002_initial_data... OK
  Applying fpr.0003_pronom_84.../usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_formatgroups_new' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

WARNING   py.warnings.find_fixtures:  /usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_formatgroups_new' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

/usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_formats_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

WARNING   py.warnings.find_fixtures:  /usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_formats_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

/usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_formatversions_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

WARNING   py.warnings.find_fixtures:  /usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_formatversions_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

/usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_fptools_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

WARNING   py.warnings.find_fixtures:  /usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_fptools_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

/usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_idtools_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

WARNING   py.warnings.find_fixtures:  /usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_idtools_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

/usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_idcommands_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

WARNING   py.warnings.find_fixtures:  /usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_idcommands_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

/usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_idrules_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

WARNING   py.warnings.find_fixtures:  /usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_idrules_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

/usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_fprules_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

WARNING   py.warnings.find_fixtures:  /usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom84_fprules_all' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

 OK
  Applying fpr.0004_pronom_88.../usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom_88' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

WARNING   py.warnings.find_fixtures:  /usr/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py:235: UserWarning: No fixture named 'pronom_88' found.
  warnings.warn("No fixture named '%s' found." % fixture_name)

 OK
  Applying fpr.0005_update_tool_versions... OK
  Applying fpr.0006_i18n_models... OK

The migration doesn't fail though.

We know using fixtures in our migrations is not good (see #38) but I think that's a different issue.

Occasionally, while editing a format through the Archivematica FPR UI, an error occurs. The error output looks like this:

(u'U', u'n', u'a', u'b', u'l', u'e', u' ', u't', u'o', u' ', u's', u'a', u'v', u'e', u',', u' ', u'a', u'n', u' ', u'a', u'c', u't', u'i', u'v', u'e', u' ', u'F', u'o', u'r', u'm', u'a', u't', u' ', u'V', u'e', u'r', u's', u'i', u'o', u'n', u' ', u' ', u'w', u'i', u't', u'h', u' ', u't', u'h', u'i', u's', u' ', u'p', u'r', u'o', u'n', u'o', u'm', u' ', u'i', u'd', u' ', u'a', u'l', u'r', u'e', u'a', u'd', u'y', u' ', u'e', u'x', u'i', u's', u't', u's', u'.')

Needless to say, it's not super helpful.

Working on this Django app in isolation showed a couple of situations where we make use of templates that belong to consumers.

• fpr/templates/app_layout.html extends layout_fluid.html - should the consumer include a local template instead?
• fpr/templates/format/list.html includes _fpr_update_button.html - should we revisit the update mechanism?

I'm not very experienced developing Django applications but this sounds like a bad practice unless it's documented?

Currently, app_layout.html points to static assets of dataTables based on how they're arranged in their consumers. Instead, the consumer should harvest static assets from its Django apps (including this one) during deployment. This can't be fixed in this repo but it's a good to keep the reminder.

This app also assumes that the consumer is providing jQuery. Perhaps we should find better way to match dependencies or create a custom dataTables shim where jQuery is embedded without conflicts with other existing jQuery versions.

  <link href="{% static 'vendor/datatables/css/jquery.dataTables.css %}" rel="stylesheet">
  <script src="{% static 'vendor/datatables/js/jquery.dataTables.min.js' %}"></script>

Not sure how to address this. At least, we should document the requirement.

STATIC_URL is a global only available in the dashboard.

The IDCommand that uses FIDO is linked to an old disabled version of the tool. It needs to link to the most recent version of the fido IDTOOL.

It's more annoying to have too many dependencies.