Ingestion of new proceedings causes inconsistency in the database about acl-anthology HOT 9 CLOSED

I've just tried to correct errors with W17-35 INLG 2017 which requires re-ingesting the all of the works from W17. This brought up problems with certain consistency checks that seem to have been recently added.

aclanthology@aclanthology:~/acl-anthology/public/pdf/W$ rake import:xml[true,"W17"]
(in /home/aclanthology/acl-anthology)
Seeding individual volume: W17.
PG::ForeignKeyViolation: ERROR: update or delete on table "papers" violates foreign key constraint "papers_people_paper_id_fkey" on table "papers_people"
DETAIL: Key (id)=(52374) is still referenced from table "papers_people".
: DELETE FROM papers WHERE volume_id IN (SELECT id FROM volumes WHERE anthology_id LIKE 'W17%');
rake aborted!
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR: update or delete on table "papers" violates foreign key constraint "papers_people_paper_id_fkey" on table "papers_people"
DETAIL: Key (id)=(52374) is still referenced from table "papers_people".
: DELETE FROM papers WHERE volume_id IN (SELECT id FROM volumes WHERE anthology_id LIKE 'W17%');
/home/aclanthology/.rvm/gems/ruby-2.0.0-p353@acl/gems/activerecord-4.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:128:in exec' /home/aclanthology/.rvm/gems/ruby-2.0.0-p353@acl/gems/activerecord-4.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:128:in block in execute'
/home/aclanthology/.rvm/gems/ruby-2.0.0-p353@acl/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:435:in block in log' /home/aclanthology/.rvm/gems/ruby-2.0.0-p353@acl/gems/activesupport-4.0.1/lib/active_support/notifications/instrumenter.rb:20:in instrument'
/home/aclanthology/.rvm/gems/ruby-2.0.0-p353@acl/gems/activerecord-4.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:430:in log' /home/aclanthology/.rvm/gems/ruby-2.0.0-p353@acl/gems/activerecord-4.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:127:in execute'
/home/aclanthology/acl-anthology/lib/tasks/xml_import.rake:330:in block (2 levels) in <top (required)>' /home/aclanthology/.rvm/gems/ruby-2.0.0-p353@acl/bin/ruby_executable_hooks:15:in eval'
/home/aclanthology/.rvm/gems/ruby-2.0.0-p353@acl/bin/ruby_executable_hooks:15:in `

from acl-anthology.

We need to fix this problem soon as without the ability to reingest a volume (say workshops from W17) which used to work as of one month ago, we can't ingest new proceedings or update ones.

@villalbamartin can you try ingesting the proceedings in the Saarlands VM at import/W17.xml?

from acl-anthology.

I'm on it. While it's not nice that something broke, we were expecting something like this to happen with the new checks.

from acl-anthology.

from acl-anthology.

Documenting this bug before I attempt to squash it.

The error is triggered by this line, which deletes all papers (if any) that belong to the volume that it's about to be imported - those papers either do not exist and nothing will be deleted, or they will be re-inserted briefly afterwards.

The problem is the table papers_people, which relates papers and authors. This table remains untouched, leading to the following sequence of events:

• All papers from the given volume are deleted. Even if they are re-inserted, their new ID means that they are effectively new papers.
• The table now contains references to deleted papers belonging to some authors.
• At some point, someone performs a search that involves one of those same authors.
• The search will then try to calculate how many papers that author has written. This process will fail for deleted papers (because the entry in the papers table is missing), the whole search will then fail, and you'll get the error we've seen recently.

These are the two steps I'll attempt to correct this:

• Delete the proper entries from papers_people before deleting the referenced entries.
• Make the entire ingestion process a transaction (if it isn't already), to avoid future inconsistencies.

from acl-anthology.

I've now added a line to delete the proper records, and ran rake import:xml[true,"W17"] without error. @knmnyn, could you confirm that things are working as expected? Note that I didn't run the entire ingestion pipeline, only the line that you mentioned in your report.

from acl-anthology.

@villalbamartin Thanks, that looks like it worked!

from acl-anthology.

Update: the consistency check disappeared when we recreated the database at some point, and yet the modification we did to xml_import.rake seems to have solved the problem, so I don't see the need to add an extra constraint to the database that we apparently don't need. So even though the fix is real and still around, the database consistency check is no longer there.

from acl-anthology.

Ok. Thanks, @villalbamartin !

from acl-anthology.