tom-draper / api-analytics Goto Github PK

I think it is because the row size is too large. My web application gets about 150k requests, now approximately 300k for 2 days. It took 10 minutes for me to get data from the api.

And I found that the api directly return all these requests data, which may be reason for the front-end dashboard to load so slowly. As it has to analyzed 30k rows data.

Maybe summarize the traffic data hour by hour, only count the total count of every endpoint to another database table might be a more wise idea to send the preprocessed and simplified data to dashboard and api? Hoping for your reply!😊

Hey!

This tool looks like what I need for my fastapi server. But after setup and some calls I am getting to metrics in the dashboard or through api calls.

Can someone else report if there are problems?

Hello!

Awesome project. For the rocket analytics, the "true path" is recorded. This ends up yielding endpoints in the UI like /api/things/some_unique_id, rather than likely the grouping that most people would want, like /api/things/<id>, which is the route name.

This would make all calls to /api/things/<id> grouped in the UI, which, I believe, is how most people would like to see the data grouped.

This can be fixed for rocket by encoding the route rather than the path (e.g., req.route().as_ref().uri.unwrap_or_else("UNKNOWN")).

I'm happy to contribute, but I wanted to verify that you are OK with this.

Hi,

Once again, thank you for this tool.

Installing the node-api-analytics package throws the following error (node v18, v20 and v22):

➜ npm i -S node-api-analytics
npm ERR! code 1
npm ERR! path /Users/me/tests/fastify-analytics/node_modules/node
npm ERR! command failed
npm ERR! command sh -c node installArchSpecificPackage
npm ERR! npm ERR! code ETARGET
npm ERR! npm ERR! notarget No matching version found for [email protected].
npm ERR! npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! npm ERR! notarget a package version that doesn't exist.
npm ERR! 
npm ERR! npm ERR! A complete log of this run can be found in: /Users/me/.npm/_logs/2024-05-31T11_31_55_761Z-debug-0.log
npm ERR! node:internal/modules/cjs/loader:1148
npm ERR!   throw err;
npm ERR!   ^
npm ERR! 
npm ERR! Error: Cannot find module 'node-bin-darwin-arm64/package.json'
npm ERR! Require stack:
npm ERR! - /Users/me/tests/fastify-analytics/node_modules/node/installArchSpecificPackage.js
npm ERR!     at Module._resolveFilename (node:internal/modules/cjs/loader:1145:15)
npm ERR!     at Function.resolve (node:internal/modules/helpers:190:19)
npm ERR!     at ChildProcess.<anonymous> (/Users/me/tests/fastify-analytics/node_modules/node-bin-setup/index.js:19:27)
npm ERR!     at ChildProcess.emit (node:events:519:28)
npm ERR!     at maybeClose (node:internal/child_process:1105:16)
npm ERR!     at ChildProcess._handle.onexit (node:internal/child_process:305:5) {
npm ERR!   code: 'MODULE_NOT_FOUND',
npm ERR!   requireStack: [
npm ERR!     '/Users/me/tests/fastify-analytics/node_modules/node/installArchSpecificPackage.js'
npm ERR!   ]
npm ERR! }
npm ERR! 
npm ERR! Node.js v20.13.1

npm ERR! A complete log of this run can be found in: /Users/me/.npm/_logs/2024-05-31T11_31_52_852Z-debug-0.log

Repro steps:

1. Start a new project npm init -y
2. Install npm i -S node-api-analytics

Thanks

Trying to use my key and getting 502 Bad Gateway in the network tab.

Hi, bro, I am using this library to monitor my axum app, I think I add this middleware layer to my main router rightly and send requests to my web app, but it seemed that it keeps finding no requests.

So does this logging has some update intervals such as every day that I can't see the recording instantly? Hoping for your reply!

Hi Tom!

Firstly I really appreciate your work you've done here.
I've tested the solution with one of my APIs, I love the way you're presenting the data.

I have some questions (pointed also on the screen below):

• Do you have any idea why I cannot see any locations?

• Should I see 0's at the response times pane? I assume it might be due to most of the requests being CURLs

• Is there an option/future plan to add an endpoint filtering mechanism? E.g. select endpoint X and see response times timeline only for that endpoint?

• I can see in the git repo, that there is some monitoring code. I assume it's not yet deployed to the www.apianalytics.dev server?

• I can see in the git repo that there is a full server code. Maybe you have a guide on how to start it up (ideally I would like to set up the server locally as a separate Docker container).

Please let me know your thoughts! I'm open to collaboration :)

I am having a POST function that currently takes up to 20 seconds. I see the request to the endpoint in my dashboard, but the response time seems to be 0 . Is there some kind of upper limit here?

Hi,
I don't seem to be able to get data from https://apianalytics-server.com/api/data. It returns an empty list. Even tho dashboard shows records.

Whenever I try to go to the analytics dashboard it is stuck on an infinite loading screen.
This is the snippet for loading the middleware (using FastAPI) app.add_middleware(Analytics, api_key="MY-API-KEY").
By looking at devtools I can see that https://api-analytics-server.vercel.app/api/requests/id-is-here always returns {"message":"Invalid user ID.","status":400}. Am I doing anything wrong or is this an issue with the package?

Throws:

thread 'actix-rt|system:0|arbiter:5' panicked at 'called `Option::unwrap()` on a `None` value', /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-analytics-1.0.8/src/analytics.rs:135:14

on almost every request.

Might be a ratelimit?

Currently the generate function seems to be broken: https://www.apianalytics.dev/generate

All time users changed to 0 even though it had over 100k users before.
Before:

After:

The response time graph also looks a bit odd now

Link to analytics: https://www.apianalytics.dev/dashboard/fb1f330ea13b4f4c91a8de83895e56e3

There are certain endpoints I want to hide from my dashboard and the overall requests count. For example, I have a "/status" endpoint that I have an uptime bot ping every 60 seconds and it shows on the dashboard. I don't want the /status endpoint to affect the rest of my analytics. This can also be done for enpoints that have to do with logins to token refreshes, or whatever.

I would like to request that users are able to filter out any number of endpoints and make it not shown on the dashboard at all. But, instead of just deleting them, maybe grey them out instead so I can still see it if I wanted to.

Will this api analytics have support for the rocket rust library? It is quite commonly used. Thanks.

I have been trying to use API ANALYTICS with Langchain Langserve but when I use add_routes with Langserve, API ANALYTICS is not picking them up. Is there anything I can do to make it work?

I had the same idea of building a lightweight middleware that can capture API metrics natively in the service.

1. What is your vision or goal with this project? Maybe, we can collaborate in building this project.
2. Also, what is the license for this project?

Hello @tom-draper , just wondering why i cant pull data api as stated in README.

response = requests.get("https://api-analytics-server.vercel.app/api/data/", headers=headers)

It says {"message":"Invalid API key."}

It would be useful if you could put some developer docs if someone wants to pull the project and try to run locally.

also a docker compose file would be much appreciated

There's an issue with logging requests in analytics dashboard.

Sometimes it captures all of the incoming requests, sometimes it doesn't

Usually, when I request same api 5, 6 times, it misses some requests hence not logging all of the requests.
I tried generating new API key to see if the issue is with the key and also tried to install 1.1.3 version but didn't help.

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1115, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 122, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 108, in call
response = await self.dispatch_func(request, call_next)
File "/usr/local/lib/python3.10/dist-packages/api_analytics/fastapi.py", line 25, in dispatch
'user_agent': request.headers['user-agent'],
File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 568, in getitem
raise KeyError(key)
KeyError: 'user-agent'

i tried integrating this on my FastAPI, and this is what i am getting when utilizing this on AWS environment.

Hi Tom,

Firstly, I want to express my gratitude for your project, it has proven to be quite valuable to my team. However, I'm facing an issue over the past few days regarding the dashboard not displaying data.

Upon inspecting the network activity, I observe that all the necessary data is being loaded as expected. So, I suspect that the recent commits made to tweak the dashboard might be the root cause of this problem.

Also, the analysis data, from the following endpoint https://www.apianalytics-server.com/api/requests/<secret-key>, is loading correctly and has a size of around 2-3MB. So, the issue doesn't seem to originate from there.

If there's any specific information or action required from my end to assist in resolving this matter, please let me know. I appreciate your help and the work you've put into this project.

Thanks once again.

I get this error:

from api_analytics.fastapi import Analytics
  File "/opt/anaconda3/lib/python3.8/site-packages/api_analytics/fastapi.py", line 4, in <module>
    from api_analytics.core import log_request
  File "/opt/anaconda3/lib/python3.8/site-packages/api_analytics/core.py", line 10, in <module>
    def _post_requests(api_key: str, requests_data: list[dict], framework: str):
TypeError: 'type' object is not subscriptable

Hello.

It is not possible to stop reqwest from using native-tls.
reqwest = { version = "0.11", features = ["json", "blocking"] }
defined here: https://github.com/tom-draper/api-analytics/blob/main/analytics/rust/actix/analytics/Cargo.toml

I prefer to use rustls-tls, which can be defined like this:
reqwest = { version = "0.11", default-features = false, features = ["json", "blocking", "rustls-tls"] }

A possible solution to this issue would add features to the analytics crate like this:

reqwest = { version = "0.11", default-features = false, features = ["json", "blocking"] }

...

[features]
default = ["native-tls"]

rustls = ["reqwest/rustls-tls"]
native-tls = ["reqwest/default-tls"]

My analytics site has incredibly long loading time (over 40 seconds)!
https://www.apianalytics.dev/dashboard/fb1f330ea13b4f4c91a8de83895e56e3

That's the analytics for this website: https://markdown-videos-api.jorgenkh.no/
The source code is available here in case you wanna look at it as a part of the debugging: https://github.com/Snailedlt/Markdown-Videos

Here's the console log before the loading is done:

And after the loading is done (the same error continues below the screenshot:

Here's the file it's pointing at:

Seems like this is an error, but perhaps adding some form of caching could also speed things up?

Looks like there may be a version bump needed for axum, and an update to the implementation of the Service trait.

the example with the latest axum version produces error:

63  |     .layer(Analytics::new(analytics_api_key.unwrap_or("").to_string())) // analytics middleware
    |      ----- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `tower_service::Service<axum::http::Request<Body>>` is not implemented for `AnalyticsMiddleware<Route>`
    |      |
    |      required by a bound introduced by this call
    |
    = help: the trait `tower_service::Service<http::request::Request<hyper::body::body::Body>>` is implemented for `AnalyticsMiddleware<S>`
note: required by a bound in `Router::<S>::layer`
   --> /home/thor/.cargo/registry/src/index.crates.io-6f17d22bba15001f/axum-0.7.4/src/routing/mod.rs:278:21
    |
275 |     pub fn layer<L>(self, layer: L) -> Router<S>
    |            ----- required by a bound in this associated function
...
278 |         L::Service: Service<Request> + Clone + Send + 'static,
    |                     ^^^^^^^^^^^^^^^^ required by this bound in `Router::<S>::layer`
    ```

Hello.

When I was looking into #5 I have noticed the "blocking" feature being used and was wondering why ... and you spawn a thread on every log request?

This might be no issue for services with not many requests, but when you have thousands of requests going in ... well...

I did as in the above example - statistics does not work, is not displayed on the site. There are no errors in the script related to the analytics module.

On the server based on FastAPI everything works fine.

Oct 26 23:19:55 Neurohost python3[2273106]: KeyError: 'HTTP_USER_AGENT'
Oct 26 23:19:55 Neurohost python3[2273106]: During handling of the above exception, another exception occurred:
Oct 26 23:19:55 Neurohost python3[2273106]: Traceback (most recent call last):
Oct 26 23:19:55 Neurohost python3[2273106]:   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 891, in finalize_request
Oct 26 23:19:55 Neurohost python3[2273106]:     response = self.process_response(response)
Oct 26 23:19:55 Neurohost python3[2273106]:   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1267, in process_response
Oct 26 23:19:55 Neurohost python3[2273106]:     response = self.ensure_sync(func)(response)
Oct 26 23:19:55 Neurohost python3[2273106]:   File "/usr/local/lib/python3.10/dist-packages/api_analytics/flask.py", line 24, in on_finish
Oct 26 23:19:55 Neurohost python3[2273106]:     'user_agent': request.headers['user-agent'],
Oct 26 23:19:55 Neurohost python3[2273106]:   File "/usr/local/lib/python3.10/dist-packages/werkzeug/datastructures/headers.py", line 493, in __getitem__
Oct 26 23:19:55 Neurohost python3[2273106]:     return self.environ[f"HTTP_{key}"]
Oct 26 23:19:55 Neurohost python3[2273106]: KeyError: 'HTTP_USER_AGENT'

In my server logs I see this error very often. If I disable analytics - the error disappears, everything works as it should.

Hello. So I want to use these analytics for my API, but there a following issues:

• stops counting at 1000 requests(?)
• shows every request as endpoint (causes extreme long endpoint list)
• it should be possible to filter out 404 from all data (on my API it will cause success rate to stay at almost 0% and show many endpoints which do not even exist)

endless list of endpoints...