Problems loading 7M real data about questdb HOT 14 CLOSED

Hi Juan,

Your code looks ok, runs fine for me. I ran it for 10m of random strings - all ok. Questdb works only with primitive types, it would be skipping Double fields. If you have public data i can reproduce with i would appreciate it.

If you can export CSV file there is UI to load it up you can try.

• Run this main class: com.questdb.BootstrapMain, parameter is path to empty directory.
• Open browser on 'http://localhost:9000'
• Click on Upload button (second from top on left panel)
• Drag your csv into browser

If CSV has header row, it will name journal columns using it. Types are recognised automatically, but you can adjust manually and reload if you need.

Regards,
Vlad

from questdb.

I'm gonna try importing the CSV i've got.
Running the BootstrapMain I get the following error:

D:\JAC\Dropbox\des\Questdb\bin>CALL .\lcp.bat ..\lib\annotations-15.0.jar
QDB HTTP Server 3.0
Copyright (C) Appsicle 2014-2016, all rights reserved.

Exception in thread "main" java.lang.NullPointerException
at com.questdb.BootstrapMain.extractSite(BootstrapMain.java:128)
at com.questdb.BootstrapMain.main(BootstrapMain.java:68)
Presione una tecla para continuar . . .

Do i need install anything else? what's the "Site"?

Regarding the primitive types, this is the reason why i've only got several fields in questdb db ( int and String). The rest of fields (Double, Integer, Boolean and Dates) are skipped, aren't they?
If i changed the data type to the primitive ones they would be processed by the engine, correct?
I've got to see how to implement this. There is a problem with the Oracle database definition; some fileds allow NULL as a value. I'll see how to get by.

Regards,

Juan

from questdb.

"Site" is a directory unde resources, this is web site content that is being served when you run bootstrap. Usually maven packages it up in a jar. You can also run this from ide.

from questdb.

If it helps QuestDB interprets Double.NaN values as null.

from questdb.

A CVS File with 7878870 real bills was loaded with 1.5M errors in a INT field. I realized that it is possible to change the field definition and reload the file. Changing from INT to LONG everything was ok.
The header row and the first one look like the example below. In this case instead of using ; we are using | as file delimiter.

id_fatr|id_distribuidora_cne|id_comercializadora_cne|cups|contrato|num_factura|id_tipofactrec_cne|num_factura_rectificada|ticket_energetico|fecha_factura|fecha_grabacion|dir_suministro|municipio|documento|observaciones|id_tipofactper_cne|id_tipofactura_cne|id_tipoidentif_cne|cif|fecha_boe|modo_facturacion|validado|fecha_validado|fecha_envio_portal|documento_backup|fecha_firma|fecha_sappi|cod_solicitud|fecha_solicitud|tipo_moneda|fecha_valor|fecha_limite_pago|id_tarifa_cne|fecha_ini|fecha_fin|cupspm|num_meses|ind_at_med_bt|val_lec|val_atr_lec|ind_envio_csa|fecha_envio_csa|modo_control_potencia|total|saldo_factura|saldo_cobro|base_ie|coef_ie|pct_ie|ie|importe_alquileres|importe_intereses|base_ivaigic_n|pct_ivaigic_n|ivaigic_n|base_ivaigic_r|pct_ivaigic_r|ivaigic_r|base_ivaigic_n2|pct_ivaigic_n2|ivaigic_n2|base_ivaigic_r2|pct_ivaigic_r2|ivaigic_r2|fdesde_cch|fhasta_cch|ind_curvacarga|pvpc|id_envio_cumas|id_res_cumas|tipofactura|id_tarifa|fecha_ini_portal
35692563|0027|0644|ES0027123456789123GK0F|1000551687|0810030014177165|N|0810740000910954|V000105121686|2015-01-02 00:00:00|2015-01-03 03:00:06|NAVALIAS 9 39108 MOMPIA|39108|/20150103/0027.0644.F1.01.0810030014177165.V000105121686.xml|073227064401|1|01|NI|B39111111|2014-02-01 00:00:00||OK|2015-01-03 03:16:40|2015-01-03 03:36:28||2015-01-03 03:35:56|2015-01-03 03:39:37|073227064401|2015-01-02 00:00:00|02|2015-01-03 00:00:00|2015-01-23 00:00:00|001|2014-09-04 00:00:00|2014-12-04 00:00:00|ES0027123456789123GK0F|3.07|N|||1|2015-01-05 12:07:19|true|93.85|93.85|93.85|0.0|0.0|0.0|0.0|2.45|0.0|77.56|21.0|16.29|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|2014-09-04 00:00:00|2014-12-04 00:00:00|true|false||||176|
and so on (+7878869 rows)

Please, how can i use the SQl Console. I want to test the data and verify that everything is ok in the questDB

from questdb.

Dude, i've just finded out how to run sqls. I've got to tell you that this database doesn't run but flies at speed light. it blows my mind....

from questdb.

select cups,sum(total) from Factura.csv --> read in 2.207s 695,436 rows
select cups,count() from Factura.csv --> read in 1.777s 695,436 rows
select count() from Factura.csv --> 0.02s 1 rows (7878870 as result)
select sum(total) from Factura.csv --> read in 0.239s 1 rows

from questdb.

hi Juan, thanks. SQL is still in development, but most things are there (search, grouping, joins - including asof join and subqueries). But i guess you discovered some of them :)

Is that possible to have your data under non-disclosure agreement to troubleshoot the crash you initially reported?

from questdb.

data can be scrambled as long as it still causes crash...

from questdb.

The compressed file weights 537M.

from questdb.

do you use "slack" messaging?

from questdb.

i just uploaded 250MB file there no problem, i'm sure 537M should be ok too

from questdb.

Juan, issue should be resolved now. Thanks for your help resolving this bug.

from questdb.

Feel free to reopen in case this still is an issue

from questdb.