Comments (4)
Hi, thanks for the issue and pull request. Could you explain how you want to use this so I can make sure I understand the intentions?
After a quick look at the pull request, I have a few questions.
- Why are you only logging status===200 requests?
- Should this new log be an option that can be enabled by a flag?
- Is there a reason to log both the old and new logs?
- Any reason "loggedInAs" is not included in the new log?
- Especially since a structured log format implies there will be dependencies on these logs, can you add tests to verify the logs are in the proper format?
from erddap.
Just curious: how is this different/better than the Tomcat request log which is also in a standard format?
from erddap.
A little background. we needed to see what had been downloaded going back as far as we could. our email setup hadn't been working due to some internal IT changes, therefore we fell back onto erddap's logging system, a grep did fetch the information required but only back about a year.
All of that work lead us to realising we needed a good
way to catch what was being requested. Currently we are investigating a way to duplicate traffic (i.e. monitor requests before they make it to erddap) - this is more of a challenge when we add in the https connections rather than http and the response tracking.
Given this is being driven from our remit to deliver data as a national data center (plus funders want to know), we thought that other institutes might want a simple
way to see what erddap was delivering out to their communities.
@BobSimons I see this as better, as it isn't dependent on tomcat's setup and without knowing every institutes setup it seems plausible that some places might have access to the erddap directories and not to tomcats logging.
@ChrisJohnNOAA These are all very good points and most of my answers resolve to because we need that info
, more details below:
- Agreed this log should include all the requests and it is up to other systems to visualise what might be required
i. we are using this system to track and report on downloaded/accessed data requests hence only 200s - happy to default to erddaps way of doing this
i. i would be interested in seeing if other users would want a way tofilter
this type of logging within the setup.xml - interesting, we don't want to remove logging from other places as we felt that other institutes might have designed ways/systems to pick this info out of the current logs.
- no, that is a good point and one we can look to include
- we can look to add tests in, given the current work underway around the way tests (and maven) are setup how would you like new tests to be added?
As stated this work is very much something we wanted to discuss with the community to see if it is of interest to anyone else, as we require this information for reporting purposes we will be continuing investigation around the traffic routing. If the idea in this issue is of use to the wider community then we'll swap to this way of collecting the information in the future.
from erddap.
Even if you only want successfully downloaded/accessed data, you probably want to include status codes besides 200. Specifically 206 (partial content) and redirects (300-308) might be useful. On a broader note, I'd be interested in feedback from others on if things like error requests would be useful here.
As for changes to the existing logging, my recommendation would be to have a setting that allows configuration of what logs should be generated. Most likely it should default to the current logging implementation, and have options for the new structured, and both together. Though I'd also appreciate feedback from others running ERDDAP servers on what they'd like to see here.
As for testing, my preference would be a JUnit style unit test, specifically testing the logging capabilities. However since I don't currently have an example JUnit test, I'd be happy with any test.
from erddap.
Related Issues (20)
- HTML Entity Name munging in XML listings HOT 1
- Search Multiple ERDDAPs seems to be broken HOT 1
- Overly restrictive S3 limitations HOT 3
- Outer axis overlap between files HOT 1
- Using XInclude HOT 2
- Unicode attributes and localized metadata HOT 46
- Inaccurate varName in logs when printing for null value attribute HOT 11
- Error: 'Error { code=404; message="Not Found: Currently unknown HOT 1
- Wrong link in the iso19115.xml metadata extraction ! HOT 5
- Dataset Citation and display info
- Translation tooltip images have extra encoding in paths HOT 3
- Fixed Value sourceNames are not excluded from duplicate column name detection HOT 3
- Trouble with accent or other special characters - Export encoding format HOT 20
- Datatypes changed when using .nc download HOT 5
- Please show loadDatasets progress diagnostics HOT 1
- How to request a protected dataset from a script ? HOT 19
- S3 Support HOT 1
- Relative URLs HOT 11
- ERDDAP not sending emails when emailUserName is not an email address
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from erddap.