Comments (7)
Hi, @asimokby ,
I'm not sure what is the "answer" you mean. But according to the discussion with authors of LogAnomaly, they did not open-source their implementation due to several concerns. This project is the only one I can find in GitHub which combines DeepLog, LogAnomaly and LogRobust.
As for the inappropriate implementation of LogAnomaly. I was debugging LogAnomaly, and found out that the size of the input tensor of loganomaly model in logdeep/models/lstm.py
is (2048, L, 1) where 2048 means batch_size. L is the length of input. That means the input for each sequence in this batch is only a (L, 1) vector.
I'm not sure what the id (1 in the input size) is, but seems like there's one step missing which maps event ids into high-demensional vectors in data/hdfs/evene2semantic_vec.json
, like word embedding in NLP tasks.
I'm currently not interested in LogAnomaly after we discussed with the authors. So, if you want more details, you can reach out to them directly.
If you are studying log anomaly detection, There's one paper accepted by ICSE2021 that you may be interested.
If you have any other question, you can email me directly, my email address is linyang[AT]tju.edu.cn.
Best,
Lin Yang
from logdeep.
Hey @YangLin-George,
Have you got an answer to this?
from logdeep.
Hi@YangLin-George
You're right. An additional step to implement dLCE for template2vec is required.
By the way, why you are not interested in LogAnomaly? I am currently doing research on deep learning-based log anomaly detection and maybe you could leave me an email and we can discuss a little.
And I was wondering if you could please tell me which paper you mentioned in ICSE2021.
Thanks.
from logdeep.
Hi @KiteFlyKid ,
Thank you.
My email address is linyang[AT]tju.edu.cn. You can email me as much as you want, I'm happy to discuss with other researchers! :D
from logdeep.
Hi,
Thank you so much for this amazing project! I am recently playing with different methods here in this project, however, I do find something odd about the implementation of LogAnomaly. Here in this project, LogAnomaly is actually using log event ids within a window to predict the next event id, just the same as DeepLog, which seems wrong.
According to the paperwork of LogAnomaly, it seems using "semantic vectors" for the prediction (the prediction is an event id or semantic vector, either way is good).
So, as far as I can see, should we change the inputs of LogAnomaly from
Sequentials
andQuantitives
toSemantics
andQuantitives
?Please correct me if I misunderstood something. Thanks again for sharing the project!
Lin, Yang
After reading the original LogAnomaly paper, I found that this implementation is somewhat incorrect. In the LogAnomaly paper, they indeed use SemanticVector and CountVector as inputs for their LSTM model. However, they original authors did not describe how to use "attention" to connect these two inputs in their original paper. That is a big pity that they did not open-source their code!
from logdeep.
Hi @LeonYang95 Im trying to email you but the email format doesn't seem to be correct. How can I reach you?
from logdeep.
Hi @LeonYang95 Im trying to email you but the email format doesn't seem to be correct. How can I reach you?
Hi @alexjamesmx , you can try [email protected]
from logdeep.
Related Issues (20)
- hdfs parsing
- hdfs_train sequence file doesn't correspond to the sequence file generated for 100k structured file provided in the repository HOT 21
- hdfs文件夹下的event2semantic_vec.json这个文件是怎么用原始日志得到的 HOT 2
- 请问作者,data_read('template.txt')中template.txt文件是怎么得到的?第二个脚本里deepLog_hdfs_train.txt文件在data文件夹下也没看到 HOT 4
- 请问下,deeplog输出的这些指标是基于啥计算的,无监督的话咋知道哪些是对的,哪些是错的?最后有输出啥结果文件,找出有问题的日志窗口吗 HOT 1
- 关于 TP,FP,TN,FN的问题!
- 怎么生成训练数据hdfs_train呢? HOT 1
- '../result/deeplog/deeplog_last.pth 这个文件怎么产生 HOT 1
- prepare_log 这个的内容是什么 HOT 4
- Question about hdfs_train, hdfs_test_normal, and hdfs_test_abnormal HOT 1
- In HDFS templates count is 28? HOT 4
- An error occurs when the terminal command line runs
- Question about deeplog in logs Apache
- A data processing problem HOT 1
- One-hot encoding?
- F1 not achieved
- DeepLog hdfs original unpased data
- Possible implementation errors for session_windows
- In RobustLog's code, I didn't see the operation of weighting the semantic vector with TF-IDF
- Anomaly log file type detection and predict future log error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logdeep.