Excessive RAM usage for `DBI::dbWriteTable()` and `dplyr::collect()` about duckdb-r HOT 4 OPEN

Added an analysis of the reading behavior to https://github.com/krlmlr/duckdb-mem. Indeed, it seems that reading also consumes at least twice the size of the data read, which is surprising. We'll need to trace memory allocations to understand what's going on here. This is likely an issue in the glue, whereas the dbWriteTable() problems likely originate in the C++ core.

from duckdb-r.

I have started to investigate memory consumption with dbWriteTable() in https://github.com/krlmlr/duckdb-mem. The script https://github.com/krlmlr/duckdb-mem/blob/main/setup-manual-limited.R shows how the memory consumption is not enforced with a CREATE TABLE ... FROM ... statement. The only difference to the setup-register.R script in the same repository is that CREATE TABLE statement. @Tmonster: can you please take a look?

I'll also review memory usage for reading the table, might reorganize that repository a bit.

from duckdb-r.

Thanks for looking into this. I looked at the https://github.com/krlmlr/duckdb-mem, repo and I didnt see a reading scenario that includes the to_arrow() function like in that comment.

I was under the impression that it only requires 16GB for reading the 16GB file instead of 32GB. TIL in your repo that you can use "/usr/bin/time" to know how much memory a process used at its peak so I'll go check if my to_arrow() solution actually works.

bigdata <- tbl(con, "straight_from_memory") %>% collect() # 32 GB peak
bigdata <- tbl(con, "straight_from_memory") %>% to_arrow() %>% collect() # 16 GB peak

from duckdb-r.

Thanks, added Arrow. The usage is still 2x the data size, unfortunately.

from duckdb-r.