eadfetch is a small command line utility to retrieve EAD files from a running Archon content management system.

While attempting to export EAD files via the bulk download functions in Archon, we encountered two issues:

• The generated files are named using information from a collection's title. We ran into cases where the number of exported EAD records did not reflect what was in our Archon system. This appears to be due to collections with similar titles overwriting one another when exported.
• We had issues where exporting a sizable classification (~4000 records) from Archon would consistently fail. Altering various server resources, php, and Apache settings had no impact. This did not appear to be related to a specific record. Smaller classifications and individual collections would export just fine.

Rather than making chages to Archon, creating a small utility to download EAD files made the most sense for our case. The Archon EAD Fetcher is a more generalized version of what was originially written for others to use.

There are two options for installing:

• Install/build from source.
• Download a compiled binary.

Go 1.13+ is required to build from source. http.NewRequestWithContext from net/http is used, which was not added until Go 1.13. eadfetch has no dependencies outside of the Go standard library.

go get github.com/robertaltman/eadfetch

eadfetch can be installed as an executable. Download the appropriate compiled binary from the releases section. Place the binary in some sensible location and call directly. Open your terminal or command prompt and navigate to the location of the binary.

Linux/Mac
./eadfetch [flags]

Windows
eadfetch.exe [flags]

If you intend to use this tool with some regularity, consider adding the executable to your system path.

Aside from a running Archon instance, the only requirement is a CSV or JSON file containing information about the collections. For ease of use, the reference point was a straight download of the tblCollections_Collections table from the Archon database. In both cases, only the ID, CollectionIdentifier, and Title fields are relevant. Strictly speaking, only the ID field is required. Removing the unused fields from either file is not necessary.

If using a CSV file, make sure the columns are named accordingly. Order does not matter, but case does.

Similarly, if working with a JSON file:

[
    {
        "ID" : 1,
        "CollectionIdentifier" : "VF00001",
        "Title" : "Three Dimensional (3-D) Treatment Planning,
        ...
    }
    ...
]

Though untested, if in a real bind with no access to a download of the database table, you could theoretically create a CSV or JSON file containing only an ID field/column filled in with a sequence of numbers up to some arbirary limit. If a particular ID does not exist in the database, the tool will take note for an output report and move on.

Calling eadfetch -h will list the configurable options. Except for the -host setting, which defaults to http://127.0.0.1, most of the options should not require much attention.

#!/bin/bash
eadfetch -h

Usage of eadfetch:
  -eadname string
        Formatting options for naming the downloaded EAD files.
        Select from one or both in any order, separated by commas: CollectionIdentifier, Title.
        (e.g., -eadname Title,CollectionIdentifier).
        Default will result in a file name of 'ead_<ID column value>.xml'.
        The ID is always appendended to the filename to ensure they are distinct. Unsafe characters will be removed.
  -file string
        File to parse for Archon data. (default "collections-table.csv")
  -host string
        Hostname of Archon site (e.g., https://archon-site.edu). (default "http://127.0.0.1")
  -output string
        Output directory for fetched XML. (default "./ead_output")
  -ratelimit int
        Limit for the number of requests to be made per second. (default 4)
  -test int
        Specifiy a number for testing a limited number of collections from the input file.
  -timeout int
        Number of seconds to allow a request to take. (default 30)
  -workers int
        Number of request workers to initiate. (default 2)

Below is a more friendly description of each flag:

-eadname

Lets you use the information in the CollectionIdentifier and/or Title fields to name the retreived EAD files. Potentially unsafe characters will be replaced with underscores. In all cases the integer from the ID field will be appended to the name, preceeded by an underscore. If you do not need custom names, just let it run in default mode. This will result in files named ead_1.xml, ead_2.xml, ...

-file

The relative or full path to the CSV or JSON file containing the Archon collection information. If nothing is specified, eadfetch will look for a file named collections-table.csv in the directory where the command has been called.

-host

The full address to your Archon server. Do not include a trailing slash. This defaults to http://127.0.0.1; unless you are working from an installation on your local machine, you will want to set this.

-output

This is the directory where you would like the files to be downloaded. Can be a relative or full path. If the folder does not exist, eadfetch will attempt to create it. Defaults to a folder named ead_output relative to where the command has been called.

-ratelimit

Allows the requests to be throttled and not potentially overwhelm the server. A default rate of 4 requests per second has performed well. This limit is monitored by and shared across all workers.

-workers

Allows for multiple request workers to be instantiated. This is helpful as one worker dealing with a slow request will not block all work from proceeding.

-timeout

Sets a limit on the duration of time a single request is allowed to take. If the request exceeds the number of seconds indicated here, it will be cancelled so that the worker can continue with other requests. When a request is cancelled, it will be added back into the request queue to be tried again later.

-test

The test flag will allow you to run eadfetch, but limit the number of records it attempts to process. This is helpful as you do not need to create separate files for testing; just use the full CSV/JSON and run a small test batch.

#!/bin/bash
eadfetch -file my_collections.csv -host http://archon-site.edu -output my_ead_files -test 40

Fetching: 40 of 40 collections.
EAD XML fetching completed in 10.093588918s.
Writing report...
The report is now complete. Look for the file 'ead-fetch-report.csv' in the my_ead_files directory.
The last column will contain information about the status of a retrieval.

Files should now exist in the output directory. The directory will also contain a report/log file titled ead-fetch-report.csv indicating the status of each request. Refer to it to determine any download issues.

This project is licensed under the MIT License - see the LICENSE file for details