max feature count in one line about xlearn HOT 8 CLOSED

xlearn does not set the max id of feature for each line. But we will set a maximal buffer for each line. The buffer size is 100kb. If your data is beyond this buffer, xlearn will raise an error:

if (read_size > kMaxLineSize) {
   LOG(FATAL) << "Encountered a too-long line.    \
          Please check the data.";
 }

from xlearn.

If your data in one line is beyond this buffer, please tell us.

from xlearn.

OK, my dataset has many features in oneline, maybe 2w. feature_space is 10000000.
I've uploaded a sample data in my repo
@aksnzhy
https://github.com/HyperGroups/MachineLearning/tree/master/xlearn/problem
feature_length is {2719, 325, 10032, 2389, 2248}

my data can train by xgboost.

from xlearn.

@HyperGroups Hi, we have already fix this bug. You can try it. It's really a big size in one line of your data. Please let me know if there is any problem.

from xlearn.

@aksnzhy The problem in this issue has be fixed.

My machine's memory is 128G.

But I found in my dataset, there are some memory problem, my dataset is 24G, average sample size is 3000-15000, feature space is 1000w, actual is 300w.
xlearn_train failed with 'killed' error, training progress is 4 epoch.
I've also tried --no-norm --disk and --dis-lock-free

And another full dataset with 100G which can be trained by xgboost also be killed.
so I reduce my dataset to 24G to test.

xlearn_train train.data -m train.model -v test.data -x acc --no-norm --disk

from xlearn.

Can you provide me the detailed error information ? Thank you !

from xlearn.

I've emailed you, I can offer my dataset in Baidu Cloud if neccessary.
@aksnzhy

from xlearn.

Solved memory leak bug.

from xlearn.