jonbirge / logpager Goto Github PK

Much faster than using PHP to extract intermediate lines.

Color-coded by whether its an attack or valid connection. This may only be possible by caching IP geolocations or switching to database solution.

List them all at top of file as links

Highlights for 404s and other status codes that suggest hacking attempts.

That is, the most recent should be at the top.

Now that it is cached, it's not a problem.

Right now, especially on searches with lots of results, we're getting temporarily banned.

Just write a whois.php script to handle the whois requests locally and provide more information via the system whois interface.

Do an asynchronous reverse DNS of the IP addresses after downloaded from the server, using JavaScript.

If this is actually going to scale, it will need to periodically read the log files into a local SQL database. This might be too much scope creep, however.

• Backend PHP function to return modification time of given log
• Frontend remembers last date and checks every few seconds
• Frontend shows last check time and last changed time at top
• When update needed, update both graphics and table
• Animation to show that frontend script is running?

Optionally merge httpd and sshd logs into single coherent attack picture. O nly show failed entries.

PHP should just output a JSON structure with the log, and JS can render it into a table.

• Modify back-end logtail.php to output JSON array
• Front-end JavaScript to render table and add links to whois request
• Asychronously convert IPs to hostnames

Function that will take line start and end (measured from end of file).

From the logs it appears that when a search result is rendered as a table, the rDNS web service is hit twice per unique IP for some reason.

YAML file covering the following settings, at least initially:

• Log file locations and types
• Visualization parameters currently hardwired in JavaScript
• Whether or not to pull geolocations

Perhaps a heat map of hours versus days. The user should be able to click on a day and have that log come up as focus.

Next to each IP address, have a small link to search the log for that IP address.

Should go from 0-23 in heatmap

Start with given IP and determine IP range it belongs to via whois lookup. Search for any IP in logs from that range.

Pipe tail of log into grep, which should be much faster than using PHP. Clear log and indicate that search is happening.

Check line for whether it starts with a letter versus number, and handle different versions of timestamp.

Bubble plot on world map, sized by number from each region.

People shouldn't be able to just use the .php functions on their own.

Show attacks in one heatmap and normal traffic in another. Unless it's possible to show both in the same heatmap using a clever color scheme which adds red to green or something.

Add field and term parameters to search. If field isn't specified, than operate as normal searching everything in the line.

Add link to geolocation REST API at ipinfo.io or the like.

Use Google fonts and embed.

Add link below IP addresses

Set max number of log lines to even consider, counting from back. This can be handled with a simple UNIX "tail" command. Alternatively, or in addition, allow the log window to be specified in terms of time, and read dates backwards from the end of the log. The former will be a heck of a lot quicker, however.

Either batch them collecting them into a set of unique IPs for the given page on the front-end, or send all requests for a given page in one request to the back-end.

For now, just return the first 25 items found. Later, implement paged search results.

Use some sort of pie-type chart, and when the user clicks on a given IP address, all log entries from that address are shown. Possible just show the top N < 1/P that are above a certain fraction P.

Also, elide a few headings?

This will require using grep on the server to isolate only the lines with IP addresses in them using a regexp.

Have search create a temporary log file with the search results, and put front-end into mode where everything works the same except the temporary log file is used.

• Allow logtail.php take optional logfile as parameter
• Change front-end display to show that we're no longer paging through the access.log but the search results and what the search term was
• Maybe: have whatever offline statistics functionality exists run on temporary file so graphs (in the future) can persist and work with the search results

Make a given page available as option to log request. When page changes, programmatically change URL to reflect this and provide way to return. (If this is even possible.)

When drilling down to an hour after drilling down to an IP, the two searches should be intersected.

Maybe tooltips, too.

People may want to exclude consideration of certain IP addresses from the log, such as those associated with internal testing.

Have frontend script create a 'whois' link for each IP address that will run a local whois search and report the results in a separate div.

With a search term and page length parameter, there should be no reason for a separate search and tail function?

Not just update the table...

Alternatively, just treat null entries as "zero" log entries for display purposes.