large dataset: page should not crash (or cli should not allow to push) about datahub-qa HOT 6 CLOSED

@OriHoch Thanks for reporting this. Please, could you check now, it works fine. Probably, it took time to pass pipeline and upload to the frontend. Good point about file size, for now, there is no limit for this, but we will take it into consideration.

from datahub-qa.

still crashes the page for me

I guess it's browser-related, I'm using Chrome Version 58.0.3029.110 (64-bit) - on Linux

I tested on Firefox 55.0.2 (64-bit) and it doesn't crash, but hangs and becomes very unresponsive, also table doesn't load:

from datahub-qa.

on firefox it worked eventually (not sure how long it took, but a long time)

It seems that currently the frontend loads all dataset to memory, this makes it unusable for large datasets

from datahub-qa.

@OriHoch thanks for report - and we had this high on the todo list already since we ran into it already during testing! We know what we want to do to fix for large datasets which is to cache a partial "preview" version of the dataset - loading into memory isn't going to work. (A nicer solution would be a frontend system that loads incrementally but that is quite a lot harder to do ...)

from datahub-qa.

@Mikanebu i believe this is fixed. Can you confirm and close if it is 😄

from datahub-qa.

@OriHoch Thank you for your patience. It is FIXED now, you just need to republish your data, so it triggers pipelines. If it is still taking a long time to load the page, it should be fixed here - datopian/assembler#32 - where we will use only 1k rows for preview. I am going to close this issue, for now, if something goes wrong, please, do not hesitate to reopen it.

from datahub-qa.