taskula / koha-migration-toolbox Goto Github PK

---------------------------------

 Voyager->Koha Migration Toolbox
 Copyright 2018 NatLibFi
 Copyright 2010 ByWater Solutions

---------------------------------

The files in this repository are free software: you can redistribute 
them and/or modify them under the terms of the GNU General Public 
License as published by the Free Software Foundation, either version 
3 of the License, or (at your option) any later version.

These files are distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

---------------------------------

1) Introduction

    This Voyager ETL-program is written by the National Library of Finland and Hypernova Oy
    based on the Voyager scripts by ByWater Solutions.
    These scripts are for use in
    migration and support of the Koha integrated library system.  We
    give them to the community, in the hope that others find them
    useful.  Most should be fairly self-explanatory; if you have specific
    questions, please direct them to the committers.

1.1) Credits

    To the National Library of Finland and Ere Maijala for providing the excellent USEMARCON
    FinMARC to MARC21 conversion package.

2) Submitting patches

    No one at ByWater/NatLibFi suggests that these scripts are the be-all,
    end-all of ILS migration tools; if you have enhancements that you are
    willing to share, please send them, and we will include them!
    A pull-request would be the most-welcome way to do that.

3) Installation instructions

This program is intended to be ran on two separate servers:
- the Voyager DB server and
- the Koha application server

a) Overview of the program flow:

- A small script and the relevant config is put to the VoyagerDB-server to extract all data
- Koha-server is configured to access the VoyagerDB-server via ssh and pulls a DB dump when needed
- Koha is configured with the desired configurations
- Transformation phase mapping tables and configurations match the configured Koha and Voyager instances.

- When the migrate-command is given:
-- Koha-server instructs the VoyagerDB to hand over the data
-- Data is pulled to Koha-server and unzipped
-- Data is transformed using mapping tables and program configurations
-- Transformed Koha-objects are stored to files
-- All configurations in the Koha Staff Client are backed up, and the Koha DB is cleared
-- Those files are read by bulk*Import.pl -scripts and relevant business objects are written to Koha via the Koha Perl API.
-- ElasticSearch reindexing is triggered.

b) Commands to Run

0. Run INSTALL.sh

NB! If installing module 'Carp::Always::Color' fails, change line

cpanm -L extlib --installdeps .

to

cpanm -L extlib --force --installdeps .

and re-run the script.

0. Install the VoyagerDB extraction script

- Configure ssh-connection to the Voyager DB-server from the Koha-server.
- Copy the "extractor/VoyagerExtractor" directory to the VoyagerDB-server and configure the scripts there as per instructions in extractor/VoyagerExtractor/README.
- Configure the extraction automation script. Set the script used to pull the Voyager data dump to the main configuration at
    ~/MMT-Voyager/config/main.yaml
  For example implementations take a look at
    extract/voyagerToKoha_*

1. Configure mapping tables.

- edit ~/MMT-Voyager/config/translationTables

2. Configure Koha

2.1 Follow the Koha-manual's implementation checklist closely

2.2 ExtendedPatronAttributes

- KITT2 needs 'statistic', which is repeatable, non-unique, authorized_value-category should be defined and used based on the patron_stat.yaml mapping table.
- Hetula needs 'SSN', which is non-repeatable, unique anonymized SSN-key reference to the real SSN-value

3. Extract Voyager data and download to the Koha-server, unzip to VoyagerExport-directory

  see the example pipeline script deployed during INSTALL.sh

3.1. Anonymization

  It is possible to anonymize the Patron data in Voyager, to make it's handling less sensitive.
  see extractor/VoyagerExtractor/extract.pl --help
  and adjust your pipeline to match

4. transform.pl to transform the data

  see migrate.pl and migrate.sh

  It is possible to run multiple ETL-pipelines by setting the MMT_HOME-environment variable.

5. Load to Koha

  see the example pipeline script deployed during INSTALL.sh

5.1 Preserving legacy database IDs

  see loader/KohaLoder/load.sh
  and adjust your pipeline to match