ibm / logdna-cos Goto Github PK

As it mentioned on IBM Cloud Docs, the Logpush job is currently pushing new log packages to COS bucket every 30 seconds or every 100,000 logs, whichever comes first. And with a solution with a high access, more than one file might be pushed per 30-second period or per 100,000 logs. So, in order to handle a higher number of files on COS bucket, the logdna-cos should be able to handle every package separately (in a serverless model).

The source code needs to be able to handle all the packaging process (by default, it is 20,000 in a single package) before start sending each one to the LogDNA.

The AS-IS scenario is different. It adds the 20,000 logs in a single package and then sends it to the LogDNA. It does this process for every 20,000 logs, but it does not wait to package everything. And when something happens during the "add logs and then send" process, like an exception (not mapped previously), it automatically exits and waits for the next time to redo everything again, which cause the duplication of logs on LogDNA (even though it only stores all new logs for 7 days, 14 days or 30 days on the IBM Log Analysis with LogDNA service).

List of all files to start contributing:

• CODE_OF_CONDUCT
• CONTRIBUTING

Also, adding a Contributing section on README.md with the link to the list of all contributors will make it easy to thank everyone who helped this repository (Idea from: Awesome IBM Cloud).

If you try to send a payload with more than 10 MB per request, LogDNA will return with an error. The size of each log depends on the number of fields added and the request result on IBM Cloud Internet Services (there is an example on README.md with all fields available on CIS).

As part of the improvement process, the buffer-split lib is a small package that allows the function to split all lines of logs by a "Buffer tag." It should be replaced by local code to allow its usage on IBM Cloud Functions without any additional process of zipping node_modules folder.

The current process is: download the object, parse the JSON, send to the LogDNA, upload the object, and delete the object. For IBM Cloud users, you can use the Private Endpoint to avoid any Public Bandwidth cost, but for those who are not using IBM Cloud, it can generate an extra charge.

So, to improve the process, the function should be able to copy the object instead of deploying the local one.

As announced in early 2020, the request (main dependency for the request-promise) library is deprecated. So it should be replaced by any other HTTP request library, like http (from Node.js core), axios, or any other library.