reczoo / fuxictr Goto Github PK

words = words[0:self._topk_words] should be sorted.

The code of splitting heads is buggy, without transpose the shape. The buggy code impacts AutoInt, DESTINE, InterHAt

query = query.view(batch_size * self.num_heads, -1, self.attention_dim)
key = key.view(batch_size * self.num_heads, -1, self.attention_dim)
value = value.view(batch_size * self.num_heads, -1, self.attention_dim)

The buggy code is from the reference code https://zhuanlan.zhihu.com/p/47812375

The correct code should be:

query = query.view(batch_size, seq_len, self.num_heads, self.attention_dim).transpose(1, 2)
key = key.view(batch_size, seq_len, self.num_heads, self.attention_dim).transpose(1, 2)
value = value.view(batch_size, seq_len, self.num_heads, self.attention_dim).transpose(1, 2)

I don't find any examples for numerical + category + sequence dataset
maybe you can provide it

torch.jit.trace(fibinet, batch_data)
File "/anaconda3/envs/fuxictr3.8/lib/python3.8/site-packages/torch/jit/_trace.py", line 794, in trace
return trace_module(
File "/anaconda3/envs/fuxictr3.8/lib/python3.8/site-packages/torch/jit/_trace.py", line 1056, in trace_module
module._c._create_method_from_trace(
File "/anaconda3/envs/fuxictr3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/anaconda3/envs/fuxictr3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
TypeError: forward() takes 2 positional arguments but 3 were given

len(data_generator) != the actual batches in npz_block_dataloader

2024-04-14 11:01:33,212 P42965 INFO Evaluation @epoch 2 - batch 1060:
2024-04-14 16:15:11,489 P42965 INFO Evaluation @epoch 3 - batch 2120:
2024-04-14 21:28:40,049 P42965 INFO Evaluation @epoch 4 - batch 3180:
2024-04-15 02:43:36,228 P42965 INFO Evaluation @epoch 5 - batch 4240:

最近在FuxiCTR的基础上实现了自己的模型，由于显存吃紧，想使用torch.parallel模块在两块GPU上训练。但是无论如何设置，模型始终在一个GPU上运行。请问该如何解决？

Hey, I have a question about training DeepFM on originial dataset Criterio dataset. Is this possible with the code provided in the repository? Dataset presented in demo version has 19 colummns. I mean train_sample.csv, test_sample.csv etc. Are these columns from Criterio dataset? How can I use original Criterio dataset when the data from this dataset is numerical and the caterogical columns are hashed. How to handle no labels in test set?

Traceback (most recent call last):
File "run_expid.py", line 69, in
model.fit(train_gen, validation_data=valid_gen, **params)
File "/root/miniconda3/lib/python3.8/site-packages/fuxictr/pytorch/models/ctr_model.py", line 154, in fit
self.train_epoch(data_generator)
File "/root/miniconda3/lib/python3.8/site-packages/fuxictr/pytorch/models/ctr_model.py", line 210, in train_epoch
loss = self.train_step(batch_data)
File "/root/miniconda3/lib/python3.8/site-packages/fuxictr/pytorch/models/ctr_model.py", line 193, in train_step
loss = self.get_total_loss(batch_data)
File "/root/miniconda3/lib/python3.8/site-packages/fuxictr/pytorch/models/ctr_model.py", line 90, in get_total_loss
total_loss = self.add_loss(inputs) + self.add_regularization()
File "/root/FuxiCTR/model_zoo/FINAL/model/FINAL.py", line 107, in add_loss
return_dict = self.forward(inputs)
File "/root/FuxiCTR/model_zoo/FINAL/model/FINAL.py", line 87, in forward
y1 = self.forward1(feature_emb)
File "/root/FuxiCTR/model_zoo/FINAL/model/FINAL.py", line 96, in forward1
X = self.field_gate(X)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/FuxiCTR/model_zoo/FINAL/model/FINAL.py", line 132, in forward
if gate_residual == "concat":
NameError: name 'gate_residual' is not defined

in file FINAL.py, line 130:

    def forward(self, feature_emb):
        gates = self.linear(feature_emb.transpose(1, 2)).transpose(1, 2)
        if gate_residual == "concat":
            out = torch.cat([feature_emb, feature_emb * gates], dim=1) # b x 2f x d
        else:
            out = feature_emb + feature_emb * gates
        return out

the code if gate_residual == "concat": should be if self.gate_residual == "concat":

I am not sure whether the implementation of DIN here is consistent with the original official released code DIN or DIEN.
Given query_field (e.g., query Goods ID, query Cate ID) and history_field (e.g., Goods ID, Cate ID), in the original implementation, they first concatenate the embeddings of history_field (i.e., [Goods ID embedding, Cate ID embedding]) and then perform so-called "local activation unit (LAU)" with the concatenated query_field (i.e., [query Goods ID embedding, query Cate ID embedding]). In contrast, the implementation here seems to first perform the LAU then do the concatenation.
Specifically, the code in DIN.py:

# 1. perform the LAU for each history_field
for idx, (din_query_field, din_history_field) \
            in enumerate(zip(self.din_query_field, self.din_history_field)):
            item_emb = feature_emb_dict[din_query_field]
            history_sequence_emb = feature_emb_dict[din_history_field]
            pooled_history_emb = self.attention_layers[idx](item_emb, history_sequence_emb)
            feature_emb_dict[din_history_field] = pooled_history_emb
# 2. do the concatenation here
feature_emb = self.embedding_layer.dict2tensor(feature_emb_dict)

If my understanding is correct, I wonder whether this will affect the final performance compared with the original implementation?

Hello, I don't know how to modify the code detailly for running my model on avazu dataset.I have three csv file(train, valid, test) for avazu, which are placed in "data" directory.which files do I need to modify?please tell me how to transform "avazu" dataset to the format(H5) we want?

ypred is one dimension tensor,don't have the axis=1

In base_model.py ypred have been transform into 1 dimension

Suggest to update the logic of preprocessing for efficiency.

In many cases, a user's behavior sequence is the same for all training samples. In addition, the features of a user or an item are often the same for all training samples.

However, the current version of FuxiCTR receives the training dataset as a single DataFrame, so these features (e.g., a user's behavior sequence, the features of a user or an item) should be stored redundantly in that DataFrame, which consumes too much memory (especially, in large-scale dataset). Also, fit/transform of feature_preprocessor should be performed on redundant behavior sequences and features, which takes too long (especially, in large-scale dataset).

So, to operate more efficiently, I hope these redundancies are removed. To this end, I suggest to change the logic of preprocessing to receive user_df and item_df for each dataset together and fit/transform unique features (i.e., user_df and item_df).

想问一下，这个报错，没有看到具体的data_generator函数呀

https://fuxictr.github.io/ is not working

• Wukong: Towards a Scaling Law for Large-Scale Recommendation
• TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest
• DPN: Deep Pattern Network for Click-Through Rate Prediction
• MemoNet: Memorizing All Cross Features' Representations Efficiently via Multi-Hash Codebook Network for CTR Prediction
• Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems
• AdaEnsemble: Learning Adaptively Sparse Structured Ensemble Network for Click-Through Rate Prediction
• DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction
• AT4CTR: Auxiliary Match Tasks for Enhancing Click-Through Rate Prediction
• TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou
• End-to-end training of Multimodal Model and ranking Model
• Unified Visual Preference Learning for User Intent Understanding
• Scaling User Modeling: Large-scale Online User Representations for Ads Personalization in Meta

To update:
DESTINE
InterHAt
EDCN
MaskNet
DLRM
DSSM

Many hyperlinks are no longer valid, responding with 404. Please update.

For example:

In main README.md:

👉 See reusable dataset splits for CTR prediction.

In FuxiCTR/data/README.md:

Criteo: https://github.com/openbenchmark/BARS/blob/master/ctr_prediction/datasets/Criteo
Avazu: https://github.com/openbenchmark/BARS/blob/master/ctr_prediction/datasets/Avazu
Taobao: https://github.com/openbenchmark/BARS/blob/master/ctr_prediction/datasets/Taobao
KKBox: https://github.com/openbenchmark/BARS/blob/master/ctr_prediction/datasets/KKBox
MicroVideo1.7M: https://github.com/openbenchmark/BARS/blob/master/ctr_prediction/datasets/MicroVideo1.7M
Frappe: https://github.com/openbenchmark/BARS/blob/master/ctr_prediction/datasets/Frappe
Movielens: https://github.com/openbenchmark/BARS/blob/master/ctr_prediction/datasets/Movielens

与normalizer.py中的接口对应错误。

when i run "python run_expid.py --config /..../ --expid 191 --gpu 0" in my code,i get 2 results log .they are ..._tune_191_ahdfy68.log and ..._tune_051_os191usd.log.In second log file name,it also has the string of '191'.

For StandardScaler, looks like it supports NaN values, see class Normalizer:

null_index = np.isnan(X)

However, during preprocess, _fill_na() will fill na_value for non-string.
So

• for dtype=str, the X values will be string
• for dtype=float/int, the X values will be na_value

In the first case, np.isnan will throw an error because X elements are of string type.
In the second case, there is no point to normalize numbers if we have a na_value there.

Is this behavior expected or not?

Suggest to add a parameter description to some main methods ( such as basemodel.fit ( ) and some models ) to facilitate the user to read the code. Can refer to the https://github.com/shenweichen/DeepCTR-Torch parameter annotations and documentation.
I hope more comments to facilitate reading code.

Should field "sequence" share embeddings with the field "adgroup_id"? I found that the method "encoder.fit()" assigns the encoder such as tokenizer for each field. Since the given tiny datasets record the user historical behavior (ad sequence), then in my understanding that the id that appeared in the field "sequence" may also appear in the field "adgroup_id". As a result, it seems that the field "sequence" should share the same encoder (i.e., tokenizer) with the field "adgroup_id", but the demo "DeepFM_with_sequence_feature.py" gives separate encoders for these two fields.

您好，尝试您提供的deepFM等模型时，发现epoch为100时在criteo数据集上取得较好的效果，而推荐算法往往1个epoch就可以收敛，请问这里如何运行如此多个epoch，还能取得较好的效果？在百忙之中打扰到您~

1. The model raises errors occasionally
2. lsh_attention on long sequence has no mask for sequence padding positions

Where is the paper for FINAL model (FINAL: factorized interaction layer for ctr prediction)? I can't find it on the internet. Based on the source code, FINAL model uses multiplicative feature interactions. But I want to find the paper to gain more insight into the model. I would appreciate it if you could provide the paper.

In FuxiCTRv1, eval_steps seems to be set to a float number less than 1 to support multiple evaluations within one epoch.

However, in FuxiCTRv2, eval_steps needs to manually calculate and set the number of evaluation steps.

I suggest to be able to automatically calculate the number of steps for evaluation if eval_steps is less than 1.

我自己设计了一个非常简单的模型，非常简单（类似于MaskedNet），运行速度很快。因为现在毕业了没有GPU资源，所以几乎没有经过调参，但是准确率仅次于FinalMLP，请问下你们有兴趣接手一下吗？

https://github.com/SY575/DeepFFM

你好，有支持多卡训练的想法吗？

The paper uses concat(LN(e1), LN(e2),..., LN(ef))， but the code use nn.LayerNorm([feature_map.num_fields, embedding_dim]) . This makes normalization happen in the last two dimensions.

Hi, can you provide the config of the dataset and model about DIN on 'taobao_x1_001' ?

HFM adopts HolographicInteractionLayer in fuxictr/pytorch/layers/interaction.py.

However, HolographicInteractionLayer may be not available in Pytorch 1.10 because torch.rfft/torch.irfft has changed.\

Here is a solution for reference:

try:
    from torch import irfft
    from torch import rfft
except ImportError:
    from torch.fft import irfft2
    from torch.fft import rfft2
    def rfft(x, d):
        t = rfft2(x, dim = (-d))
        return torch.stack((t.real, t.imag), -1)
    def irfft(x, d, signal_sizes):
        return irfft2(torch.complex(x[:,:,0], x[:,:,1]), s = signal_sizes, dim = (-d))

41/5000
When I try to use sequence characteristics, the program has an array out of bounds error. Could you please give an example of sequence features.

Huggingface Datasets:

 dataset = load_dataset("parquet", data_files={split: data_blocks}, split=split)
 super().__init__(dataset=dataset, num_workers=8, batch_size=self.batch_size)

I want to save a model such as DCN_tf for serving. If I add 'model.save("path/to/model")' at the end of the run_expid.py, error occurs as
cannot be saved either because the input shape is not available or because the forward pass of the model is not defined.To define a forward pass, please override Model.call(). To specify an input shape, either call build(input_shape) directly, or call the model on actual data using Model(), Model.fit(), or Model.predict(). If you have a custom training step, please make sure to invoke the forward pass in train step through Model.__call__, i.e. model(inputs), as opposed to model.call().

I have added 'model(input)' before model.save() as the following code and it works. However, when I want to curl to call the service, I must specify the 'clk' which is already specified as label.

for i in train_gen:
    model(i)
    break
model.save("./model/SavedModels/1")

Thank you!

Hi, I found that MLP_Block didn't work properly due to BN when the length of the tensor input was greater than or equal to 3.

Here is a toy example.

a = torch.randn(16, 128, 64)
mlp_block = MLP_Block(input_dim=64, hidden_units=[1024,1024,1024],batch_norm=True)
mlp_block(a)

The output is:

RuntimeError: running_mean should contain 128 elements not 1024

We can't find a function called self.reduce_learning_rate() in Base model ,does it have change into self.lr_decay()?

streaming: False should be used in dataset_config