Queries Fail due to invalid dates about enterprise-log-search-and-archive HOT 27 CLOSED

This indicates that the frontend is not finding any logs because there are no 
indexes listed.  The 1969 dates mean that the found "start" and "end" dates 
were "0."  Are you able to see any logs with any query?  What is the output of 
this query on the node?
mysql syslog -e "select * from v_indexes order by start"

I get a response "Empty set (0.00 sec)".
Something must be wrong with the indexer.
I have been trying to port ELSA to Arch Linux for some time now, and even after 
getting everything installed and configured it seems I'm still missing 
something!
The installer script simply doesn't work on Arch Linux, so I've had to package 
everything on my own... I'd love your help as the company I work for wants to 
make ELSA a pretty big part of our log parsing.
Perhaps we should merge this bug into the Arch Linux support bug and go from 
there?

Arch Linux uses a system known as the Arch Build System to package/install 
software. For my first ABS script handling ELSA, you can look at 
https://aur.archlinux.org/packages/el/elsa/PKGBUILD for a bash script that 
installs the files and dependencies needed for ELSA.

If you were to provide a tarball and rely on the end-user to configure each 
part of the entire 'ELSA' system individually, this would greatly increase the 
portability and extensiblity of your software. I will gladly help you as much 
as I can to get configuration for each part of ELSA written.

One thing I cannot figure out is why my indexes aren't getting indexed. I'm not 
sure if it's syslog-ng or sphinx that isn't doing its work. What can we do to 
figure out what part of the system isn't working?

Ok, let's try to get your setup working on Arch, then we'll see what's involved 
with the overall process to hopefully provide canonical support for Arch.

First things: If you remove any times listed and run a search for "seq" what do 
you get?  (seq is input in the initial test run so it should be there.)

Next: What do you have for indexes on your node?  You can find with:
mysql syslog -e "select * from v_indexes order by start"

"Invalid start or end: Wed Dec 31 19:00:00 1969 Wed Dec 31 19:00:00 1969 at 
/usr/local/elsa/web/lib/Query.pm line 656."
Even though both time boxes are blank, I still get this error when searching 
for "seq".

I have no indexes on my node, as that mysql command returns nothing.

I have changed the /etc/elsa_node.conf and /etc/sphinx/sphinx_elsa.conf files, 
and I've at least got something in my v_indexes table now. However, I have set 
up syslog-ng to take data from some Bro flatfiles and I still cannot see it 
when I make a query.

Do you see anything if you run the same query in archive mode?  You can switch 
to archive using the drop-down menu labeled "Index."

Running in archive mode is giving me a few results, but nothing related to what 
Bro is logging. Looking at the syslog db in my MySQL, there is no data 
currently being taken from Bro or syslog-ng.

Ok, let's make sure there's no problem with elsa.pl.  On the log node, run:
echo "testing 123" | perl elsa.pl -on
Are there any errors listed?

This is what I get:

isaac@archie ~ $ sudo bash -c "echo 'testing 123' | perl 
/usr/share/elsa/node/elsa.pl -on -c /etc/elsa/elsa_node.conf"
testing
isaac@archie ~ $

Searching for "testing" in both Archive and Index mode does not return any 
results.

Ok, look for any errors in the log file, there should be an indication of what 
it decided to do, since it didn't die with any fatal errors.

Where does the perl script log to?

Nevermind, I found node.log
I have a *lot* of lines that look like either of these two:

* ERROR [2012/07/10 13:39:08] /usr/share/elsa/node/Writer.pm (122) 
Writer::_sql_error_handler 28189 SQL_ERROR: DBD::mysql::st execute failed: 
called with 653 bind variables when 468 are needed, query: INSERT INTO 
syslog_data.syslogs_archive_1 (id, timestamp, host_id, program_id, class_id, 
msg, i0, i1, i2, i3, i4, i5, s0, s1, s2, s3, s4, s5) VALUES 
(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?
,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?
,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?
,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?
,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?
,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?
,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?
,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?
,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?
),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?
,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?
,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?),(?,?,?,?,?
,?,?,?,?,?,?,?,?,?,?,?,?,?)
* WARN [2012/07/10 13:39:09] /usr/share/elsa/node/Reader.pm (228) 
Reader::parse_line 28189 Missing required field class id

Uh oh, looks like realtime's not working for you.  Uncomment the "realtime" 
section in the elsa_node.conf file and restart syslog-ng.  Then hopefully your 
Bro logs start showing up.

After disabling realtime, I'm still unable to find anything bro-related, and I 
have many sets of lines similar to this in my node.log file

isaac@archie ~ $ tail -n 21 /srv/elsa/log/node.log
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/etc/sphinx.conf'...
WARNING: no such index 'temp_1014', skipping.
total 0 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 0 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
* TRACE [2012/07/10 13:53:28] /usr/share/elsa/node/Indexer.pm (1421) 
Indexer::_sphinx_index 8589 ran cmd: /usr/bin/sphinx-indexer --config 
/etc/sphinx.conf --rotate temp_1014 2>&1
* ERROR [2012/07/10 13:53:28] /usr/share/elsa/node/Indexer.pm (1440) 
Indexer::_sphinx_index 8589 Hit retry limit of 3
* ERROR [2012/07/10 13:53:28] /usr/share/elsa/node/Indexer.pm (1446) 
Indexer::_sphinx_index 8589 Indexing didn't work for temp_1014, output: $VAR1 = 
[
          'Sphinx 2.0.4-id64-release (r3135)',
          'Copyright (c) 2001-2012, Andrew Aksyonoff',
          'Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)',
          '',
          'using config file \'/etc/sphinx.conf\'...',
          'WARNING: no such index \'temp_1014\', skipping.',
          'total 0 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg',
          'total 0 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg'
        ];
* INFO [2012/07/10 13:53:28] /usr/share/elsa/node/Indexer.pm (1450) 
Indexer::_sphinx_index 8589 Indexed temp_1014 with 0 rows in 0.09198 seconds 
(0.00000 rows/sec)
* DEBUG [2012/07/10 13:53:28] /usr/share/elsa/node/Indexer.pm (437) 
Indexer::_validate_directory 8589 Wiping via index perm_1014

Your above message indicated you were using /etc/sphinx/sphinx_elsa.conf, but 
that error says it's trying to use /etc/sphinx.conf.  You may need to change 
the setting in your elsa_node.conf to match.

They are symlinked.

Ok, well is there a configuration for "temp_1014" in the sphinx.conf?  
Otherwise, it looks like you changed the setting for number of indexes but 
didn't recreate the sphinx.conf file.  (This can be done easily by simply 
deleting or moving it, ELSA will autocreate it.)

There is not a configuration for temp_1014. I've deleted the sphinx.conf file 
but elsa seems to have rebuilt it incorrectly.

Jul 10 14:22:39 archie searchd[15222]: ERROR: line too long in 
/etc/sphinx/sphinx_elsa.conf line 52182 col 1.

I have attached my sphinx.conf file.

I think you have way too many indexes. Set "num_indexes" down to something like 
400.

Alright, I've lowered the number of indexes. I can search for things in archive 
mode (a search for "bro" returned an error bro gave me upon restart of the 
node. Hurray!) but I get an error when making queries in the "index" mode:

No nodes available at /usr/local/elsa/web/lib/API.pm line 1771.

I'm not exactly sure what this means.

"No nodes available" implies a problem trying to connect to searchd.  Make sure 
that the port listed in elsa_web.conf for "nodes/<node>/mysql_port" matches the 
port that searchd is listening on (9306, by default, 3307 in older ELSA 
implementations).

I have gotten queries working (it was an iptables issue), but I do not seem to 
have any useful patterndb action going on. Queries are surprisingly blank.

What am I forgetting? This is what happens when I click "info" on a bro_http 
event.

I think it's related to my syslog-ng configuration, so I've also attached that.

I followed the Bro section of the Documentation page, by the way.

Ah, the problem is indeed in your syslog-ng.conf.  You are doing individual log 
{} statements for Bro, such as:
log { source(s_bro_communication); destination(d_elsa); };
But that doesn't do all of the rewriting, etc. like in the above:
log { 
    source(s_network);
    rewrite(r_host);
    rewrite(r_cisco_program);
    rewrite(r_snare);
    rewrite(r_pipes);
    parser(p_db);
    rewrite(r_extracted_host); 
    destination(d_elsa);
};

So, you need to add the Bro statements in like this:

log { 
    source(s_network);
        source(s_bro_communication);
        source(s_bro_conn);
        source(s_bro_dns);
        source(s_bro_http);
        source(s_bro_known_services);
        source(s_bro_notice);
        source(s_bro_software);
        source(s_bro_stderr);
        source(s_bro_stdout);
        source(s_bro_ssl);
        source(s_bro_weird);
    rewrite(r_host);
    rewrite(r_cisco_program);
    rewrite(r_snare);
    rewrite(r_pipes);
    parser(p_db);
    rewrite(r_extracted_host); 
    destination(d_elsa);
};

I notice that there are lines like this as well:
source s_bro_ssl { file("/var/log/bro/current/ssl.log" flags(no-parse) 
program_override("bro_ssl")); };
That have the "flags(no-parse)" option. Is that going to interfere with the 
patterndb parsing later on in the log directive?

No, the no-parse flag is separate and applies only to the log source.  
PatternDB parsing applies to all logs in the log {} chain it's in, regardless 
of source.

Closing for now due to inactivity.