publici / fec-loader Goto Github PK

View Code? Open in Web Editor NEW

21.0 2.0 9.0 748 KB

Loads raw FEC filings into a database

License: MIT License

JavaScript 84.86% Dockerfile 1.44% Shell 13.70%

campaignfinance fec node etl elections

fec-loader's People

Contributors

Stargazers

Watchers

Forkers

pandringa robertdigital syyunn cimar tag-strategies digiwhale chriszs cqrollcall-datateam ryankelly-dc

fec-loader's Issues

Externalize filing queue

Perhaps using something like Kue. Still some architectural concerns to work out (how do we handle zip files?) but would allow for better failure modes and concurrent imports.

Add debts, loans and obligations, clovers and blue moons

Better handle 20130113.zip

Contains a README that says "There were no filings on this day."

Add way to flag committees in database for notification

Better handle empty "nofiles" zips

Import F3 summaries

We have F3X and F3P, but no F3s for some reason.

Stalls on ActBlue 2015 year-end

Filing 1047732. Temporarily excluded from zip import.

Modularize, decouple, simplify

Vary checker rate to match filing tempo

Debug new experimental copy stream import

The new stream support, based on Postgres COPY command, in the streeeams branch is way faster, but early testing on 20160907.zip has revealed a bunch of problems to fix. For instance, it appears to import the header and itemizations from filing number 1097996 but not the summary row? Also:

== importing 1098085 ==
inserted 237 rows from 1098085
Error: write after end
...
== importing 1098090 ==
inserted 300 rows from 1098090
Error: write after end
...
== importing 1097994 ==
processed 1,000 records in 0.13 seconds at 7,924 rows/second
...
processed 1,247,000 records in 230.03 seconds at 5,421 rows/second
{ error: extra data after last expected column
...
where: 'COPY fec_contributions, line 1246185: "1098041\tSA11AI\tC00401224\tSA11AI_52410847\tNULL\tNULL\tIND\tNULL\tWILSON\tBOB\tNULL\tNULL\tNULL\t64 SOUTH 25TH\t..."',
...
== importing 1098068 ==
inserted 105 rows from 1098068
Error: write after end
...
== importing 1098081 ==
{ error: invalid input syntax for integer: ""
...
  where: 'COPY fec_filings, line 1, column report_id: ""',
...
== importing 1098084 ==
{ error: insert or update on table "fec_expenditures" violates foreign key constraint "fec_expenditures_filing_id_fkey"
...
  detail: 'Key (filing_id)=(1098048) is not present in table "fec_filings".',
...

I need to figure out why it complains about not finding 1098048 in fec_filings when it's importing 1098084. Computer dyslexia? Is some part of it still importing 1098048 long after it's supposed to?

My guess is that once I figure out proper flow control for this, most of these (especially the "write after end" errors) will go away. But we may want to go back to wrapping the whole import in a transaction. And we definitely need tests.

Date/Time field value out of range

error inserting 996567:
{ SequelizeDatabaseError: date/time field value out of range: "2 1028"
at Query.formatError (/Users/rosie/Documents/fec-model/node_modules/sequelize/lib/dialects/postgres/query.js:357:14)
at Query. (/Users/rosie/Documents/fec-model/node_modules/sequelize/lib/dialects/postgres/query.js:88:19)
at Query.emit (events.js:180:13)
at Query.handleError (/Users/rosie/Documents/fec-model/node_modules/pg/lib/query.js:125:8)
at Connection. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/client.js:171:26)
at Connection.emit (events.js:180:13)
at Socket. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:109:12)
at Socket.emit (events.js:180:13)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onread (net.js:578:20)
name: 'SequelizeDatabaseError',
message: 'date/time field value out of range: "2 1028"',
parent:
{ error: date/time field value out of range: "2 1028"
at Connection.parseE (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:539:11)
at Connection.parseMessage (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:366:17)
at Socket. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:105:22)
at Socket.emit (events.js:180:13)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onread (net.js:578:20)
name: 'error',
length: 170,
severity: 'ERROR',
code: '22008',
detail: undefined,
hint: 'Perhaps you need a different "datestyle" setting.',
position: '10361',
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'datetime.c',
line: '3887',
routine: 'DateTimeParseError',
sql: 'INSERT INTO "fec_paper_contributions" ("id","filing_id","form_type","filer_committee_id_number","contributor_organization_name","contributor_last_name","contributor_first_name","contributor_middle_name","contributor_prefix","contributor_suffix","contributor_street_1","contributor_street_2","contributor_city","contributor_state","contributor_zip_code","election_code","election_other_description","contribution_date","contribution_amount","contribution_aggregate","contributor_employer","contributor_occupation","donor_committee_fec_id","memo_text_description","image_number") VALUES (DEFAULT,'996567','SA11A1','C00042366',NULL,'QUATE','RHODA',NULL,NULL,NULL,'3170 CHICAGO ST',NULL,'SAN DIEGO','CA','921176114',NULL,NULL,'2014-11-03 05:00:00.000 +00:00',' 35.00',' 453.00','AONIX','SOFTWARE ENGINEER',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311367'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-04 05:00:00.000 +00:00',' 35.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311367'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUATTROCCHI','ADA',NULL,NULL,NULL,'7200 CONNECTICUT AVE',NULL,'CHEVY CHASE','MD','208154932',NULL,NULL,'2014-10-19 04:00:00.000 +00:00',' 50.00',' 310.00','NONE','NOT EMPLOYED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311368'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-20 04:00:00.000 +00:00',' 50.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311368'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUATTROCCHI','ADA',NULL,NULL,NULL,'7200 CONNECTICUT AVE',NULL,'CHEVY CHASE','MD','208154932',NULL,NULL,'2014-10-25 04:00:00.000 +00:00',' 50.00',' 310.00','NONE','NOT EMPLOYED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311368'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-27 04:00:00.000 +00:00',' 50.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311369'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUATTROCCHI','ADA',NULL,NULL,NULL,'7200 CONNECTICUT AVE',NULL,'CHEVY CHASE','MD','208154932',NULL,NULL,'2014-11-01 04:00:00.000 +00:00',' 50.00',' 310.00','NONE','NOT EMPLOYED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311369'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-03 05:00:00.000 +00:00',' 50.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311369'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUATTROCCHI','VINCENT',NULL,NULL,NULL,'610 OHIO AVE',NULL,'LONG BEACH','CA','908141231',NULL,NULL,'2014-10-20 04:00:00.000 +00:00',' 50.00',' 260.00','SELF','REAL ESTATE',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311370'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-22 04:00:00.000 +00:00',' 50.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311370'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUBAIN','BASHAR',NULL,NULL,NULL,'680 FOREST RD',NULL,'WAYNE','PA','190872324',NULL,NULL,'2014-11-01 04:00:00.000 +00:00',' 80.00',' 230.00','GEOSTRUCTURES, INC.','GEOTECHNICAL ENGINEER',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311370'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-03 05:00:00.000 +00:00',' 80.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311371'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUIGLEY','MARTHA','QUIGLEY',NULL,NULL,'342 CORNELL AVE',NULL,'HAZARD','KY','41701',NULL,NULL,'2014-10-21 04:00:00.000 +00:00',' 15.00',' 210.00','CITY OF HAZARD','MUSEUM DIRECTOR',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311371'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-22 04:00:00.000 +00:00',' 15.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311371'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUIGLEY','MARTHA','QUIGLEY',NULL,NULL,'342 CORNELL AVE',NULL,'HAZARD','KY','41701',NULL,NULL,'2014-10-27 04:00:00.000 +00:00',' 20.00',' 210.00','CITY OF HAZARD','MUSEUM DIRECTOR',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311372'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-28 04:00:00.000 +00:00',' 20.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311372'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUIGLEY','MARTHA','QUIGLEY',NULL,NULL,'342 CORNELL AVE',NULL,'HAZARD','KY','41701',NULL,NULL,'2014-10-28 04:00:00.000 +00:00',' 15.00',' 210.00','CITY OF HAZARD','MUSEUM DIRECTOR',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311372'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-29 04:00:00.000 +00:00',' 15.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311373'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUIGLEY','MARTHA','QUIGLEY',NULL,NULL,'342 CORNELL AVE',NULL,'HAZARD','KY','41701',NULL,NULL,'2014-10-31 04:00:00.000 +00:00',' 15.00',' 210.00','CITY OF HAZARD','MUSEUM DIRECTOR',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311373'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-31 04:00:00.000 +00:00',' 15.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311373'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUIGLEY','MARTHA','QUIGLEY',NULL,NULL,'342 CORNELL AVE',NULL,'HAZARD','KY','41701',NULL,NULL,'2014-11-04 05:00:00.000 +00:00',' 15.00',' 210.00','CITY OF HAZARD','MUSEUM DIRECTOR',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311374'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-05 05:00:00.000 +00:00',' 15.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311374'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUIGLEY','MARTHA','QUIGLEY',NULL,NULL,'342 CORNELL AVE',NULL,'HAZARD','KY','41701',NULL,NULL,'2014-11-21 05:00:00.000 +00:00',' 15.00',' 210.00','CITY OF HAZARD','MUSEUM DIRECTOR',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311374'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-24 05:00:00.000 +00:00',' 15.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311375'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUILL','CYNTHIA',NULL,NULL,NULL,'563 W GENESEE STREET RD',NULL,'CAYUGA','NY','130343179',NULL,NULL,'2014-10-17 04:00:00.000 +00:00',' 80.00',' 230.00','NONE','NOT EMPLOYED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311375'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-20 04:00:00.000 +00:00',' 80.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311375'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','BARBARA',NULL,NULL,NULL,'16276 OAKFORD TRL',NULL,'FISHERS','IN','460377389',NULL,NULL,'2014-11-11 05:00:00.000 +00:00',' 8.00',' 208.00','NONE','RETIRED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311376'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-12 05:00:00.000 +00:00',' 8.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311376'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','CATHERINE',NULL,NULL,NULL,'N3W31717 TWIN OAKS DR',NULL,'DELAFIELD','WI','530182859',NULL,NULL,'2014-10-28 04:00:00.000 +00:00',' 15.00',' 285.00','SELF','ART THERAPIST',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311376'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-29 04:00:00.000 +00:00',' 15.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311377'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','CATHERINE',NULL,NULL,NULL,'N3W31717 TWIN OAKS DR',NULL,'DELAFIELD','WI','530182859',NULL,NULL,'2 1028',' 25.00',' 285.00','SELF','ART THERAPIST',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311377'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-29 04:00:00.000 +00:00',' 25.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311377'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','CATHERINE',NULL,NULL,NULL,'N3W31717 TWIN OAKS DR',NULL,'DELAFIELD','WI','530182859',NULL,NULL,'2014-11-04 05:00:00.000 +00:00',' 5.00',' 285.00','SELF','ART THERAPIST',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311378'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-05 05:00:00.000 +00:00',' 5.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311378'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','CATHERINE',NULL,NULL,NULL,'N3W31717 TWIN OAKS DR',NULL,'DELAFIELD','WI','530182859',NULL,NULL,'2014-11-13 05:00:00.000 +00:00',' 25.00',' 285.00','SELF','ART THERAPIST',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311378'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-14 05:00:00.000 +00:00',' 25.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311379'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','FRANK',NULL,NULL,NULL,'215 ZELLS MILL RD',NULL,'NEWPORT','VA','241284126',NULL,NULL,'2014-10-22 04:00:00.000 +00:00',' 100.00',' 250.00','VIRGINIA TECH','PROFESSOR',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311379'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-24 04:00:00.000 +00:00',' 100.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311379'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','JOSEPH',NULL,NULL,NULL,'1029 N JACKSON ST',NULL,'MILWAUKEE','WI','53202',NULL,NULL,'2014-10-21 04:00:00.000 +00:00',' 10.00',' 275.00','NONE','RETIRED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311380'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-22 04:00:00.000 +00:00',' 10.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311380'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','JOSEPH',NULL,NULL,NULL,'1029 N JACKSON ST',NULL,'MILWAUKEE','WI','53202',NULL,NULL,'2014-10-24 04:00:00.000 +00:00',' 35.00',' 275.00','NONE','RETIRED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311380'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-27 04:00:00.000 +00:00',' 35.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311381'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','JOSEPH',NULL,NULL,NULL,'1029 N JACKSON ST',NULL,'MILWAUKEE','WI','53202',NULL,NULL,'2014-10-27 04:00:00.000 +00:00',' 50.00',' 275.00','NONE','RETIRED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311381'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-28 04:00:00.000 +00:00',' 50.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311381'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','JOSEPH',NULL,NULL,NULL,'1029 N JACKSON ST',NULL,'MILWAUKEE','WI','53202',NULL,NULL,'2014-10-28 04:00:00.000 +00:00',' 10.00',' 275.00','NONE','RETIRED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311382'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-29 04:00:00.000 +00:00',' 10.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311382'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','JOSEPH',NULL,NULL,NULL,'1029 N JACKSON ST',NULL,'MILWAUKEE','WI','53202',NULL,NULL,'2014-10-31 04:00:00.000 +00:00',' 35.00',' 275.00','NONE','RETIRED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311382'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-31 04:00:00.000 +00:00',' 35.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311383'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','JOSEPH',NULL,NULL,NULL,'1029 N JACKSON ST',NULL,'MILWAUKEE','WI','53202',NULL,NULL,'2014-11-04 05:00:00.000 +00:00',' 10.00',' 275.00','NONE','RETIRED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311383'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-05 05:00:00.000 +00:00',' 10.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311383'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','MARK',NULL,NULL,NULL,'1819 ADAMS ST',NULL,'MADISON','WI','53711',NULL,NULL,'2014-10-28 04:00:00.000 +00:00',' 50.00',' 535.00','NONE','RETIRED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311384'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-29 04:00:00.000 +00:00',' 50.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311384'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','MARK',NULL,NULL,NULL,'1819 ADAMS ST',NULL,'MADISON','WI','53711',NULL,NULL,'2014-11-03 05:00:00.000 +00:00',' 50.00',' 535.00','NONE','RETIRED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311384'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-04 05:00:00.000 +00:00',' 50.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311385'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINN','WILLIAM','D',NULL,NULL,'8305 FOUNTAIN RIDGE DR',NULL,'PLANO','TX','750253933',NULL,NULL,'2014-11-01 04:00:00.000 +00:00',' 50.00',' 835.00','P3 GROUP','ELECTRICAL ENGINEER',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311385'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-03 05:00:00.000 +00:00',' 50.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311385'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINONES','RICK',NULL,NULL,NULL,'136 SOUTH ST',NULL,'FREEHOLD','NJ','077282040',NULL,NULL,'2014-11-03 05:00:00.000 +00:00',' 10.00',' 204.00','KOMPASS NORTH AMERICA','CONSULTANT',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311386'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-04 05:00:00.000 +00:00',' 10.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311386'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINONES','RICK',NULL,NULL,NULL,'136 SOUTH ST',NULL,'FREEHOLD','NJ','077282040',NULL,NULL,'2014-11-09 05:00:00.000 +00:00',' 15.00',' 204.00','KOMPASS NORTH AMERICA','CONSULTANT',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311386'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-10 05:00:00.000 +00:00',' 15.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311387'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'BURR','LINDA','QUINTON',NULL,NULL,'3006 INDIAN HILLS DR',NULL,'PROVO','UT','84604',NULL,NULL,'2014-10-21 04:00:00.000 +00:00',' 200.00',' 665.00','SELF','ATTORNEY',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311387'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-22 04:00:00.000 +00:00',' 200.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311387'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'BURR','LINDA','QUINTON',NULL,NULL,'3006 INDIAN HILLS DR',NULL,'PROVO','UT','84604',NULL,NULL,'2014-10-28 04:00:00.000 +00:00',' 100.00',' 665.00','SELF','ATTORNEY',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311388'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-29 04:00:00.000 +00:00',' 100.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311388'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'BURR','LINDA','QUINTON',NULL,NULL,'3006 INDIAN HILLS DR',NULL,'PROVO','UT','84604',NULL,NULL,'2014-11-04 05:00:00.000 +00:00',' 100.00',' 665.00','SELF','ATTORNEY',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311388'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-05 05:00:00.000 +00:00',' 100.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311389'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINTON','ELTON',NULL,NULL,NULL,'3079 E MARTINSBURG FIRE RD',NULL,'PALMYRA','IN','471646925',NULL,NULL,'2014-10-23 04:00:00.000 +00:00',' 5.00',' 342.00','NONE','NOT EMPLOYED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311389'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-24 04:00:00.000 +00:00',' 5.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311389'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINTON','ELTON',NULL,NULL,NULL,'3079 E MARTINSBURG FIRE RD',NULL,'PALMYRA','IN','471646925',NULL,NULL,'2014-10-25 04:00:00.000 +00:00',' 5.00',' 342.00','NONE','NOT EMPLOYED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311390'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-27 04:00:00.000 +00:00',' 5.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311390'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINTON','ELTON',NULL,NULL,NULL,'3079 E MARTINSBURG FIRE RD',NULL,'PALMYRA','IN','471646925',NULL,NULL,'2014-10-27 04:00:00.000 +00:00',' 5.00',' 342.00','NONE','NOT EMPLOYED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311390'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-10-28 04:00:00.000 +00:00',' 5.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311391'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINTON','ELTON',NULL,NULL,NULL,'3079 E MARTINSBURG FIRE RD',NULL,'PALMYRA','IN','471646925',NULL,NULL,'2014-11-01 04:00:00.000 +00:00',' 5.00',' 342.00','NONE','NOT EMPLOYED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311391'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-03 05:00:00.000 +00:00',' 5.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311391'),(DEFAULT,'996567','SA11A1','C00042366',NULL,'QUINTON','ELTON',NULL,NULL,NULL,'3079 E MARTINSBURG FIRE RD',NULL,'PALMYRA','IN','471646925',NULL,NULL,'2014-11-03 05:00:00.000 +00:00',' 5.00',' 342.00','NONE','NOT EMPLOYED',NULL,'* EARMARKED CONTRIBUTION: SEE BELOW','14021311392'),(DEFAULT,'996567','SA11A1','C00042366','ACTBLUE PAC',NULL,NULL,NULL,NULL,NULL,'PO BOX 441146',NULL,'WEST SOMERVILLE','MA','021440031',NULL,NULL,'2014-11-04 05:00:00.000 +00:00',' 5.00',' 5446563.87',NULL,'CONDUIT TOTAL LISTED IN AGG. FIELD','C00401224','[MEMO ITEM] NOTE: ABOVE CONTRIBUTION EARMARKED THROUGH THIS ORGANIZATION','14021311392');' }

Create spreadsheets on checker events

Reduce memory usage for large filings

Vastly simplify

Loan schema/validation errors

ERROR: missing data for column "loan_amount_original"
CONTEXT: COPY fec_loans, line 2: "SC2/10 C00475384 G383 L640 Causey Chad \N \N \N 205 South Main #203 \N Jonesboro AR 72401 Womack Lan..."

Load F99s, TEXT rows?

1051710

Oct 18 14:55:09 ARGUS node[37186]: downloaded 1051710
Oct 18 14:55:10 ARGUS node[37186]: == importing 1051710 ==
Oct 18 14:55:10 ARGUS node[37186]: /home/sysadmin/fec-loader/src/util/fec/import.js:198
Oct 18 14:55:10 ARGUS node[37186]:                         return model.match(row);
Oct 18 14:55:10 ARGUS node[37186]:                                     ^
Oct 18 14:55:10 ARGUS node[37186]: TypeError: Cannot read property 'match' of undefined

Set weeklies parser to import on a schedule

Parse while downloading?

Chokes on F3PS line in GARY JOHNSON 2012 post-general

== importing 992728 ==
processing 0 - 1 of 91
processing 1 - 3 of 91
already inserted 992728
undefined
rolling back transaction
Unhandled rejection Error: error called for 992728, but already finished

http://docquery.fec.gov/dcdev/posted/992728.fec

Add back in notification/alerts

Refactor checker to emit events

CLI

Denormalize amended?

Foreign key constraint error

{ SequelizeForeignKeyConstraintError: insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"
at Query.formatError (/Users/rosie/Documents/fec-model/node_modules/sequelize/lib/dialects/postgres/query.js:296:14)
at Query. (/Users/rosie/Documents/fec-model/node_modules/sequelize/lib/dialects/postgres/query.js:88:19)
at Query.emit (events.js:180:13)
at Query.handleError (/Users/rosie/Documents/fec-model/node_modules/pg/lib/query.js:125:8)
at Connection. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/client.js:171:26)
at Connection.emit (events.js:180:13)
at Socket. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:109:12)
at Socket.emit (events.js:180:13)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onread (net.js:578:20)
name: 'SequelizeForeignKeyConstraintError',
message: 'insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"',
parent:
{ error: insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"
at Connection.parseE (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:539:11)
at Connection.parseMessage (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:366:17)
at Socket. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:105:22)
at Socket.emit (events.js:180:13)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onread (net.js:578:20)
name: 'error',
length: 307,
severity: 'ERROR',
code: '23503',
detail: 'Key (filing_id)=(997646) is not present in table "fec_filings".',
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: 'public',
table: 'fec_contributions',
column: undefined,
dataType: undefined,
constraint: 'fec_contributions_filing_id_fkey',
file: 'ri_triggers.c',
line: '3324',
routine: 'ri_ReportViolation',
sql: 'INSERT INTO "fec_contributions" ("id","filing_id","form_type","filer_committee_id_number","contributor_organization_name","contributor_last_name","contributor_first_name","contributor_middle_name","contributor_prefix","contributor_suffix","contributor_street_1","contributor_street_2","contributor_city","contributor_state","contributor_zip_code","election_code","election_other_description","contribution_date","contribution_amount","contribution_aggregate","contributor_employer","contributor_occupation","donor_committee_fec_id","memo_text_description") VALUES (DEFAULT,'997646','SA11A1','C00547570',NULL,'HAUCK','MEGAN',NULL,NULL,NULL,'133 DUDDINGTON PLACE SE',NULL,'WASHINGTON','DC','200032625','CONSULTANT','G','2014-09-09 04:00:00.000 +00:00',' 1000.00','2014','NATHANSON',NULL,NULL,' 1000.00');' },
original:
{ error: insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"
at Connection.parseE (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:539:11)
at Connection.parseMessage (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:366:17)
at Socket. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:105:22)
at Socket.emit (events.js:180:13)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onread (net.js:578:20)
name: 'error',
length: 307,
severity: 'ERROR',
code: '23503',
detail: 'Key (filing_id)=(997646) is not present in table "fec_filings".',
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: 'public',
table: 'fec_contributions',
column: undefined,
dataType: undefined,
constraint: 'fec_contributions_filing_id_fkey',
file: 'ri_triggers.c',
line: '3324',
routine: 'ri_ReportViolation',
sql: 'INSERT INTO "fec_contributions" ("id","filing_id","form_type","filer_committee_id_number","contributor_organization_name","contributor_last_name","contributor_first_name","contributor_middle_name","contributor_prefix","contributor_suffix","contributor_street_1","contributor_street_2","contributor_city","contributor_state","contributor_zip_code","election_code","election_other_description","contribution_date","contribution_amount","contribution_aggregate","contributor_employer","contributor_occupation","donor_committee_fec_id","memo_text_description") VALUES (DEFAULT,'997646','SA11A1','C00547570',NULL,'HAUCK','MEGAN',NULL,NULL,NULL,'133 DUDDINGTON PLACE SE',NULL,'WASHINGTON','DC','200032625','CONSULTANT','G','2014-09-09 04:00:00.000 +00:00',' 1000.00','2014','NATHANSON',NULL,NULL,' 1000.00');' },
sql: 'INSERT INTO "fec_contributions" ("id","filing_id","form_type","filer_committee_id_number","contributor_organization_name","contributor_last_name","contributor_first_name","contributor_middle_name","contributor_prefix","contributor_suffix","contributor_street_1","contributor_street_2","contributor_city","contributor_state","contributor_zip_code","election_code","election_other_description","contribution_date","contribution_amount","contribution_aggregate","contributor_employer","contributor_occupation","donor_committee_fec_id","memo_text_description") VALUES (DEFAULT,'997646','SA11A1','C00547570',NULL,'HAUCK','MEGAN',NULL,NULL,NULL,'133 DUDDINGTON PLACE SE',NULL,'WASHINGTON','DC','200032625','CONSULTANT','G','2014-09-09 04:00:00.000 +00:00',' 1000.00','2014','NATHANSON',NULL,NULL,' 1000.00');',
fields: null,
table: 'fec_contributions',
value: undefined,
index: 'fec_contributions_filing_id_fkey' }

Add sanity checks

Unhandled rejection error

Running the script again to see if this magically disappears...here it is in the meantime

Unhandled rejection Error: error called for 767559, but already finished
at error (/Users/rosie/Documents/fec-loader/src/util/fec/import.js:61:19)
at tryCatcher (/Users/rosie/Documents/fec-model/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/Users/rosie/Documents/fec-model/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/Users/rosie/Documents/fec-model/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromise0 (/Users/rosie/Documents/fec-model/node_modules/bluebird/js/release/promise.js:614:10)
at Promise._settlePromises (/Users/rosie/Documents/fec-model/node_modules/bluebird/js/release/promise.js:689:18)
at Async._drainQueue (/Users/rosie/Documents/fec-model/node_modules/bluebird/js/release/async.js:133:16)
at Async._drainQueues (/Users/rosie/Documents/fec-model/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues [as _onImmediate] (/Users/rosie/Documents/fec-model/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:763:18)
at tryOnImmediate (timers.js:734:5)
at processImmediate (timers.js:716:5)

Upload spreadsheets to S3 or expose them with a server

Value too long for type character varying(30)

{ SequelizeDatabaseError: value too long for type character varying(30)
at Query.formatError (/Users/rosie/Documents/fec-model/node_modules/sequelize/lib/dialects/postgres/query.js:357:14)
at Query. (/Users/rosie/Documents/fec-model/node_modules/sequelize/lib/dialects/postgres/query.js:88:19)
at Query.emit (events.js:180:13)
at Query.handleError (/Users/rosie/Documents/fec-model/node_modules/pg/lib/query.js:125:8)
at Connection. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/client.js:171:26)
at Connection.emit (events.js:180:13)
at Socket. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:109:12)
at Socket.emit (events.js:180:13)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onread (net.js:578:20)
name: 'SequelizeDatabaseError',
message: 'value too long for type character varying(30)',
parent:
{ error: value too long for type character varying(30)
at Connection.parseE (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:539:11)
at Connection.parseMessage (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:366:17)
at Socket. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:105:22)
at Socket.emit (events.js:180:13)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onread (net.js:578:20)
name: 'error',
length: 98,
severity: 'ERROR',
code: '22001',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'varchar.c',
line: '624',
routine: 'varchar',
sql: 'INSERT INTO "fec_contributions" ("id","filing_id","form_type","filer_committee_id_number","contributor_organization_name","contributor_last_name","contributor_first_name","contributor_middle_name","contributor_prefix","contributor_suffix","contributor_street_1","contributor_street_2","contributor_city","contributor_state","contributor_zip_code","election_code","election_other_description","contribution_date","contribution_amount","contribution_aggregate","contributor_employer","contributor_occupation","donor_committee_fec_id","memo_text_description") VALUES (DEFAULT,'997772','SA11A1','C00437889',NULL,'SMITH','SONDRA','J',NULL,NULL,'2914 BAMLET RD',NULL,'ROYAL OAK','MI','480732979','DIRECTOR OF CORPORATE MARKETING','P','2014-04-30 04:00:00.000 +00:00',' 500.00','2014','BLUE CROSS BLUE SHIELD OF MICHIGAN',NULL,NULL,' 500.00');' },
original:
{ error: value too long for type character varying(30)
at Connection.parseE (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:539:11)
at Connection.parseMessage (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:366:17)
at Socket. (/Users/rosie/Documents/fec-model/node_modules/pg/lib/connection.js:105:22)
at Socket.emit (events.js:180:13)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onread (net.js:578:20)
name: 'error',
length: 98,
severity: 'ERROR',
code: '22001',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'varchar.c',
line: '624',
routine: 'varchar',
sql: 'INSERT INTO "fec_contributions" ("id","filing_id","form_type","filer_committee_id_number","contributor_organization_name","contributor_last_name","contributor_first_name","contributor_middle_name","contributor_prefix","contributor_suffix","contributor_street_1","contributor_street_2","contributor_city","contributor_state","contributor_zip_code","election_code","election_other_description","contribution_date","contribution_amount","contribution_aggregate","contributor_employer","contributor_occupation","donor_committee_fec_id","memo_text_description") VALUES (DEFAULT,'997772','SA11A1','C00437889',NULL,'SMITH','SONDRA','J',NULL,NULL,'2914 BAMLET RD',NULL,'ROYAL OAK','MI','480732979','DIRECTOR OF CORPORATE MARKETING','P','2014-04-30 04:00:00.000 +00:00',' 500.00','2014','BLUE CROSS BLUE SHIELD OF MICHIGAN',NULL,NULL,' 500.00');' },
sql: 'INSERT INTO "fec_contributions" ("id","filing_id","form_type","filer_committee_id_number","contributor_organization_name","contributor_last_name","contributor_first_name","contributor_middle_name","contributor_prefix","contributor_suffix","contributor_street_1","contributor_street_2","contributor_city","contributor_state","contributor_zip_code","election_code","election_other_description","contribution_date","contribution_amount","contribution_aggregate","contributor_employer","contributor_occupation","donor_committee_fec_id","memo_text_description") VALUES (DEFAULT,'997772','SA11A1','C00437889',NULL,'SMITH','SONDRA','J',NULL,NULL,'2914 BAMLET RD',NULL,'ROYAL OAK','MI','480732979','DIRECTOR OF CORPORATE MARKETING','P','2014-04-30 04:00:00.000 +00:00',' 500.00','2014','BLUE CROSS BLUE SHIELD OF MICHIGAN',NULL,NULL,' 500.00');' }

error inserting 904534:
    { error: invalid input syntax for type date: "  "

error inserting 1101352:
{ SequelizeBaseError: invalid input syntax for integer: "	"
    at Query.formatError (/Users/cskees/Desktop/work/fec-loader/node_modules/sequelize/lib/dialects/postgres/query.js:357:14)
    at Query.<anonymous> (/Users/cskees/Desktop/work/fec-loader/node_modules/sequelize/lib/dialects/postgres/query.js:88:19)
    at emitOne (events.js:115:13)
    at Query.emit (events.js:210:7)
    at Query.handleError (/Users/cskees/Desktop/work/fec-loader/node_modules/pg/lib/query.js:108:8)
    at Connection.<anonymous> (/Users/cskees/Desktop/work/fec-loader/node_modules/pg/lib/client.js:171:26)
    at emitOne (events.js:115:13)
    at Connection.emit (events.js:210:7)
    at Socket.<anonymous> (/Users/cskees/Desktop/work/fec-loader/node_modules/pg/lib/connection.js:109:12)
    at emitOne (events.js:115:13)
    at Socket.emit (events.js:210:7)
    at addChunk (_stream_readable.js:252:12)
    at readableAddChunk (_stream_readable.js:239:11)
    at Socket.Readable.push (_stream_readable.js:197:10)
    at TCP.onread (net.js:589:20)
  name: 'SequelizeDatabaseError',
  message: 'invalid input syntax for integer: "\t"',
  parent:
   { error: invalid input syntax for integer: "	"
       at Connection.parseE (/Users/cskees/Desktop/work/fec-loader/node_modules/pg/lib/connection.js:539:11)
       at Connection.parseMessage (/Users/cskees/Desktop/work/fec-loader/node_modules/pg/lib/connection.js:366:17)
       at Socket.<anonymous> (/Users/cskees/Desktop/work/fec-loader/node_modules/pg/lib/connection.js:105:22)
       at emitOne (events.js:115:13)
       at Socket.emit (events.js:210:7)
       at addChunk (_stream_readable.js:252:12)
       at readableAddChunk (_stream_readable.js:239:11)
       at Socket.Readable.push (_stream_readable.js:197:10)
       at TCP.onread (net.js:589:20)
     name: 'error',
     length: 95,
     severity: 'ERROR',
     code: '22P02',
     detail: undefined,
     hint: undefined,
     position: '236',
     internalPosition: undefined,
     internalQuery: undefined,
     where: undefined,
     schema: undefined,
     table: undefined,
     column: undefined,
     dataType: undefined,
     constraint: undefined,
     file: 'numutils.c',
     line: '62',
     routine: 'pg_atoi',
     sql: 'INSERT INTO "fec_filings" ("filing_id","record_type","ef_type","fec_version","soft_name","soft_ver","report_id","report_number","comment","created_date","updated_date") VALUES (\'1101352\',\'HDR\',\'FEC\',\'8.1\',\'FECfile\',\'8.1.0.4(f30)\',NULL,\'\t\',NULL,\'2018-02-21 02:58:18.806 +00:00\',\'2018-02-21 02:58:18.806 +00:00\');' },
  original:
   { error: invalid input syntax for integer: "	"
       at Connection.parseE (/Users/cskees/Desktop/work/fec-loader/node_modules/pg/lib/connection.js:539:11)
       at Connection.parseMessage (/Users/cskees/Desktop/work/fec-loader/node_modules/pg/lib/connection.js:366:17)
       at Socket.<anonymous> (/Users/cskees/Desktop/work/fec-loader/node_modules/pg/lib/connection.js:105:22)
       at emitOne (events.js:115:13)
       at Socket.emit (events.js:210:7)
       at addChunk (_stream_readable.js:252:12)
       at readableAddChunk (_stream_readable.js:239:11)
       at Socket.Readable.push (_stream_readable.js:197:10)
       at TCP.onread (net.js:589:20)
     name: 'error',
     length: 95,
     severity: 'ERROR',
     code: '22P02',
     detail: undefined,
     hint: undefined,
     position: '236',
     internalPosition: undefined,
     internalQuery: undefined,
     where: undefined,
     schema: undefined,
     table: undefined,
     column: undefined,
     dataType: undefined,
     constraint: undefined,
     file: 'numutils.c',
     line: '62',
     routine: 'pg_atoi',
     sql: 'INSERT INTO "fec_filings" ("filing_id","record_type","ef_type","fec_version","soft_name","soft_ver","report_id","report_number","comment","created_date","updated_date") VALUES (\'1101352\',\'HDR\',\'FEC\',\'8.1\',\'FECfile\',\'8.1.0.4(f30)\',NULL,\'\t\',NULL,\'2018-02-21 02:58:18.806 +00:00\',\'2018-02-21 02:58:18.806 +00:00\');' },
  sql: 'INSERT INTO "fec_filings" ("filing_id","record_type","ef_type","fec_version","soft_name","soft_ver","report_id","report_number","comment","created_date","updated_date") VALUES (\'1101352\',\'HDR\',\'FEC\',\'8.1\',\'FECfile\',\'8.1.0.4(f30)\',NULL,\'\t\',NULL,\'2018-02-21 02:58:18.806 +00:00\',\'2018-02-21 02:58:18.806 +00:00\');' }
undefined

== importing 945333 ==
   { error: insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"
...
== importing 976646 ==
   { error: date/time field value out of range: "4   0809"
...
== importing 996567 ==
   { error: date/time field value out of range: "2   1028"
...
== importing 997646 ==
   { error: insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"
...
== importing 997772 ==
   { error: value too long for type character varying(30)
...
== importing 999470 ==
   { error: date/time field value out of range: "0   1226"
...
== importing 1002778 ==
   { error: insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"
...
== importing 1004714 ==
   { error: insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"
...
== importing 1006071 ==
   { error: invalid input syntax for type date: "VT"
...
== importing 1006100 ==
   { error: insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"
...
== importing 1006155 ==
   { error: insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"
...
== importing 1009095 ==
   { error: insert or update on table "fec_contributions" violates foreign key constraint "fec_contributions_filing_id_fkey"
...
== importing 1034930 ==
   { error: invalid input syntax for type numeric: "* EARMARKED CONTRIBUTION: SEE BELOW"
...
== importing 1066901 ==
   { error: date/time field value out of range: "0   0329"

Add email notifier

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.