sinks/graphite.py - invalid line received from client 127.0.0.1:60773, ignoring about statsite HOT 5 CLOSED

Just an update, I have started the graphite sinks without using statsite and it sends the info to graphite correctly.

Could the problem be with statsite?

python sinks/graphite.py localhost 2003 stats
om.msg.cri|1|1341308952
om.msg.cri|1|1341308961

graphite/storage/log/carbon-cache/carbon-cache-a/creates.log
03/07/2012 11:49:02 :: new metric stats.counts.om.msg.cri matched schema stats
03/07/2012 11:49:02 :: new metric stats.counts.om.msg.cri matched aggregation schema stats-counters
03/07/2012 11:49:02 :: creating database file /opt/graphite/storage/whisper/stats/counts/om/msg/cri.wsp (archive=[(10, 2160), (60, 10080), (600, 262800)] xff=0.0 agg=sum)
03/07/2012 11:49:28 :: new metric stats.counts.om.msg.min matched schema stats
03/07/2012 11:49:28 :: new metric stats.counts.om.msg.min matched aggregation schema min
03/07/2012 11:49:28 :: creating database file /opt/graphite/storage/whisper/stats/counts/om/msg/min.wsp (archive=[(10, 2160), (60, 10080), (600, 262800)] xff=0.1 agg=min)

from statsite.

Hey syepes,

This is very unusal. So it seems that the graphite sink is working but statsite is sending a bad record to it.
This is a definitely possibility, as statsite will passthough potentially invalid inputs. To see what it is doing,
try setting your streaming command to something like "cat >> /tmp/stream_out.txt", and then check to see
what statsite is emitting.

If you post it here, I can help you debug the output as well. Also, is statsite emitting any information?
Check your syslog for any messages.

from statsite.

I managed to more or less fix the issue about the "invalid line", but I am still not seeing the metrics in the graphite server.

The invalid line issue is that graphite.py is constantly sending to the graphite server the string "\n", even when it has not received any metrics and as a consequence it makes carbon give out errors.

Here is my workaround for this issue, note that I do not have any python experience I am more a Groovy guy ;-)

--- graphite_orig.py    2012-07-03 22:53:42.168137704 +0200
+++ graphite.py 2012-07-03 13:54:38.975779536 +0200
@@ -69,8 +69,12 @@
         """Tries to write a string to the socket, reconnecting on any errors"""
         for attempt in xrange(self.attempts):
             try:
-                self.sock.sendall(metric)
-                return
+                if len(metric) > 1:
+                   self.logger.info("write_metric: %s" % metric)
+                   self.sock.sendall(metric)
+                   return
+                else:
+                   return
             except socket.error:
                 self.logger.exception("Error while flushing to graphite. Reattempting...")
                 self.sock = self._create_socket()
@@ -80,7 +84,7 @@

 if __name__ == "__main__":
     # Initialize the logger
-    logging.basicConfig()
+    logging.basicConfig(filename='sinks_graphite.log',level=logging.DEBUG)

     # Intialize from our arguments
     graphite = GraphiteStore(*sys.argv[1:])

I even configured the log_level to DEBUG and there statsite does not show any errors in the syslog.

Thanks for the help!

Regards,
Sebastian

from statsite.

I think there must be some other issue with the graphite.py as it seems that statsite is receiving and streaming correctly the metrics, on the graphite server I see the listener connection but the metrics are not created. (Just for info, I have also set MAX_CREATES_PER_MINUTE = inf in the carbon.conf)

[statsite]
port = 8127
udp_port = 8127
log_level = INFO
flush_interval = 10
timer_eps = 0.01
stream_cmd = cat >> /tmp/stream_out.txt

Send metrics:
[/tmp] echo "om.msg.cri:1|c" | nc -w 0 localhost 8127
[/tmp] echo "om.msg.min:1|c" | nc -w 0 localhost 8127
[/tmp] echo "om.msg.min:1|c" | nc -w 0 localhost 8127
[/tmp] echo "om.msg.min:1|c" | nc -w 0 localhost 8127
[/tmp] echo "om.msg.cri:1|c" | nc -w 0 localhost 8127
[/tmp] echo "om.msg.cri:1|c" | nc -w 0 localhost 8127

[/tmp] cat stream_out.txt
counts.om.msg.cri|1.000000|1341350231
counts.om.msg.min|3.000000|1341350291
counts.om.msg.cri|2.000000|1341350301

from statsite.

Based on the data in stream_out.txt, I think that statsite is working as expected. We use the graphite sink in production as well, so I'm fairly confident that it is working as expected.

The issue probably is in the graphite server as you concluded. I'd check the permissions and stuff, to make sure that graphite is able to create the files on disk. Does graphite log any error messages?

from statsite.