Notifications with incorrect sequence numbers about pigpio HOT 21 CLOSED

I expect you have gone beyond the limit of what you can expect. That is an awful lot of sustained traffic. I'd guess the pipe is being filled faster than it can be emptied.

Start your program and then enter

pigs csi 512

Let it run for a while and try to count the sequence error messages. Then press ctrl c to abort the program.

You should get some stats.

Have a look at the pipe would block count and see if it matches the errors you saw.

from pigpio.

Thanks, I'll try that. I also noticed that the test program above is silly in that reads the notifications one by one. Reading up to 1000 notifications from the pipe at the same time gives far better results.

from pigpio.

I tried what you suggested with pigs csi 512 and the test program from the original post. There were 59 sequence number errors and the stats said that pipe would block 101 times:

version: 40
1000000
2000000
3000000
4000000
  sequence error - lastSeqno: 15026, seqno: 15174
    4405940
  sequence error - lastSeqno: 15846, seqno: 15905
    4406613
  sequence error - lastSeqno: 16245, seqno: 16307
    4406954
  sequence error - lastSeqno: 16647, seqno: 16708
    4407295
  sequence error - lastSeqno: 17048, seqno: 17131
    4407636
  sequence error - lastSeqno: 17471, seqno: 17509
    4407977
  sequence error - lastSeqno: 17849, seqno: 18313
    4408318
  sequence error - lastSeqno: 18653, seqno: 18706
    4408659
  sequence error - lastSeqno: 19046, seqno: 19106
    4409000
  sequence error - lastSeqno: 19446, seqno: 19506
    4409341
  sequence error - lastSeqno: 19846, seqno: 19906
    4409682
  sequence error - lastSeqno: 20246, seqno: 20306
    4410023
  sequence error - lastSeqno: 20646, seqno: 20706
    4410364
  sequence error - lastSeqno: 21106, seqno: 21448
    4410765
  sequence error - lastSeqno: 21845, seqno: 22304
    4411163
  sequence error - lastSeqno: 22644, seqno: 23106
    4411504
  sequence error - lastSeqno: 23446, seqno: 24306
    4411845
  sequence error - lastSeqno: 24646, seqno: 25107
    4412186
  sequence error - lastSeqno: 25853, seqno: 26308
    4412933
  sequence error - lastSeqno: 26648, seqno: 26708
    4413274
  sequence error - lastSeqno: 27048, seqno: 27107
    4413615
  sequence error - lastSeqno: 27447, seqno: 27506
    4413956
  sequence error - lastSeqno: 28274, seqno: 28710
    4414725
  sequence error - lastSeqno: 29446, seqno: 29506
    4415462
  sequence error - lastSeqno: 29846, seqno: 29904
    4415803
  sequence error - lastSeqno: 30244, seqno: 30707
    4416144
  sequence error - lastSeqno: 31047, seqno: 31106
    4416485
  sequence error - lastSeqno: 31446, seqno: 31504
    4416826
  sequence error - lastSeqno: 32245, seqno: 32355
    4417568
  sequence error - lastSeqno: 32695, seqno: 33105
    4417909
  sequence error - lastSeqno: 33445, seqno: 33559
    4418250
  sequence error - lastSeqno: 33899, seqno: 33904
    4418591
  sequence error - lastSeqno: 34643, seqno: 35104
    4419331
  sequence error - lastSeqno: 35844, seqno: 35904
    4420072
  sequence error - lastSeqno: 36244, seqno: 36314
    4420413
  sequence error - lastSeqno: 36654, seqno: 36706
    4420754
  sequence error - lastSeqno: 37046, seqno: 37503
    4421095
  sequence error - lastSeqno: 37843, seqno: 38305
    4421436
  sequence error - lastSeqno: 39043, seqno: 39105
    4422175
  sequence error - lastSeqno: 39445, seqno: 39905
    4422516
  sequence error - lastSeqno: 40245, seqno: 40305
    4422857
  sequence error - lastSeqno: 40645, seqno: 40704
    4423198
  sequence error - lastSeqno: 41105, seqno: 41447
    4423600
  sequence error - lastSeqno: 42299, seqno: 42641
    4424453
  sequence error - lastSeqno: 43042, seqno: 43098
    4424855
  sequence error - lastSeqno: 43438, seqno: 43902
    4425196
  sequence error - lastSeqno: 44242, seqno: 44298
    4425537
  sequence error - lastSeqno: 44638, seqno: 45102
    4425878
  sequence error - lastSeqno: 45841, seqno: 46302
    4426618
  sequence error - lastSeqno: 46642, seqno: 46718
    4426959
  sequence error - lastSeqno: 47058, seqno: 47500
    4427300
  sequence error - lastSeqno: 47840, seqno: 48301
    4427641
  sequence error - lastSeqno: 48746, seqno: 49088
    4428087
  sequence error - lastSeqno: 50241, seqno: 50699
    4429241
  sequence error - lastSeqno: 51039, seqno: 51114
    4429582
  sequence error - lastSeqno: 51454, seqno: 51899
    4429923
  sequence error - lastSeqno: 52239, seqno: 52300
    4430264
  sequence error - lastSeqno: 53041, seqno: 53100
    4431006
  sequence error - lastSeqno: 53440, seqno: 53499
    4431347
5000000
^C2015-11-20 20:37:44 sigHandler: Unhandled signal 2, terminating


#####################################################
If you didn't request stats please cut & paste the
following and e-mail to [email protected]
pigpio version=40 internals=200
micros=1 allocMode=0 dmaInitCbs=1 DMARestarts=0
samples 5353507 maxSamples 800 maxEmit 800 emitFrags 13315
cbTicks 12336, cbCalls 13373
pipe: good 26586, short 0, would block 101
alertTicks 13350, lateTicks 11, moreToDo 11
        0         0         0         0         0         0         0         0         0         0         0         1         0         0         0         0         0         0         0         1        45       218       902      2792     42388    978341     42108      2716       826       218        55        23        13         4         7         2         0         3         6        12        13        12         9         0         1         0         1         0         0         0 
#####################################################

However, as already mentioned, the original test program is silly as it reads the notifications one by one. Reading up to 1000 at the same time gives far better results.

The test code here is implemented in C for demonstration purposes. In reality, I'm using pigpio from a Node.js addon. Unfortunately, reading as many notifications as possible from the pipe in Node.js doesn't really improve things. The pipe is still being filled before it's emptied. If I'm not mistaken, the pipe has the default pipe size which is 65536 bytes. This is enough space for 65536/12=5416 notifications. The Node.js test program receives 80,000 notifications per second (unlike the C program above which receives 200,000 notifications per second). This means the pipe fills in 65536/12/80000=0.068 seconds, or 68 milliseconds. What I have noticed is that increasing the pipe size to 1048576 (the max possible) results in huge improvements for the Node.js code. There are no longer any sequence number errors and 80,000 notifications per second are easily handled.

The pipe size was modified by adding the following code

fcntl(fd, F_SETPIPE_SZ, 1048576);

directly after this line of code in pigpio.c

fd = open(name, O_RDWR|O_NONBLOCK);

Is there a way to configure the size of the pipe used for pigpio notifications?

from pigpio.

Do you do the open from C? I've briefly considered adding a parameter to gpioNotifyBegin but I'd rather leave that interface alone.

My current preference is for a new function gpioNotifyBeginWithSize taking a single parameter bufSiz. 0 for default, otherwise byte size. Would that suit you?

I am unlikely to export that beyond the C interface (at least until and unless I do a general tidy-up).

from pigpio.

Do you do the open from C? I've briefly considered adding a parameter to gpioNotifyBegin but I'd rather leave that interface alone.

I'm calling gpioNotifyOpen and gpioNotifyBegin from C++ which is more or less the same as calling them from C.

My current preference is for a new function gpioNotifyBeginWithSize taking a single parameter bufSiz. 0 for default, otherwise byte size. Would that suit you?

A new function that takes a parameter specifying the pipe size would be excellent and would suit me perfectly. I would have expected the new function to be called gpioNotifyOpenWithSize rather than gpioNotifyBeginWithSize as gpioNotifyBegin starts notifications on a previously opened handle, already has two parameters, and can be called multiple times for the same handle.

from pigpio.

The name was a typo on my part.

Provisionally I'll make the following change.

int gpioNotifyOpenWithSize(int bufSize)
{
   int i, slot, fd;
   char name[32];

   DBG(DBG_USER, "");

   CHECK_INITED;

   slot = -1;

   for (i=0; i<PI_NOTIFY_SLOTS; i++)
   {
      if (gpioNotify[i].state == PI_NOTIFY_CLOSED)
      {
         gpioNotify[i].state = PI_NOTIFY_OPENED;
         slot = i;
         break;
      }
   }

   if (slot < 0)
      SOFT_ERROR(PI_NO_HANDLE, "no handle");

   sprintf(name, "/dev/pigpio%d", slot);

   myCreatePipe(name, 0664);

   fd = open(name, O_RDWR|O_NONBLOCK);

   if (fd < 0)
   {
      gpioNotify[slot].state = PI_NOTIFY_CLOSED;
      SOFT_ERROR(PI_BAD_PATHNAME, "open %s failed (%m)", name);
   }

   if (bufSize != 0)
   {
      i = fcntl(fd, F_SETPIPE_SZ, bufSize);
      if (i != bufSize)
      {
         gpioNotify[slot].state = PI_NOTIFY_CLOSED;
         SOFT_ERROR(PI_BAD_PATHNAME,
            "fcntl %s size %d failed (%m)", name, bufSize);
      }
   }

   gpioNotify[slot].seqno = 0;
   gpioNotify[slot].bits  = 0;
   gpioNotify[slot].fd    = fd;
   gpioNotify[slot].pipe  = 1;
   gpioNotify[slot].max_emits  = MAX_EMITS;
   gpioNotify[slot].lastReportTick = gpioTick();

   closeOrphanedNotifications(slot, fd);

   return slot;
}

int gpioNotifyOpen(void)
{
   return gpioNotifyOpenWithSize(0);
}

from pigpio.

Excellent, thank you very much. I'll add the modification to my installation and let you know how it goes.

from pigpio.

I added the modifications to my installation and things are looking good :)

Two tests were performed from Node.js with bufSize set to 1048576.

The first test was 80,000 notifications per second with additional load (scp 4GB of files from a PC to the Pi 2.) Everything went well, there were no sequence number errors, every level change appears to have been detected, and the stats said that pipe would block was 0. The test ran for about 9 minutes. 44,014,932 notifications were processed.

The second test was 200,000 notifications per second without additional load. Everything went well, there were no sequence number errors, every level change appears to have been detected, and the stats said that pipe would block was 0. The test ran for a little over 8 minutes. 100,000,307 notifications were processed.

from pigpio.

As expected, the modified version of gpioNotifyOpen functions correctly too.

from pigpio.

I'll probably push out another release later on today (Sunday).

from pigpio.

👍

from pigpio.

Proposed change added to V41.

By the way they seem to be pretty impressive figures you are getting with Node.js.

One word of caution, I wouldn't recommend the use of the 1µs sampling rate in general. It's something I would only use if there was no other way of detecting edges. It puts an appreciable strain on the system.

from pigpio.

I just gave it a try and all looks well at 200,000 notifications per second. Up to 300,000 per second should work in Node.js, but there won't be much time left for doing much with them. The 1µs sampling rate was used in the tests as that's what was being used previously. I'll do some tests with other sampling rates.

One thing that I noticed with V41 compared to V40 is that the performance of gpioRead and gpioWrite appears to have dropped by about 50%. With V40 about 1,200,000 calls per second were possible from Node.js. With V41 it's about 600,000 calls per second. I haven't checked to see why this is the case but is there a V41 change that would explain it? It's not something that bothers me and it may be related to the JavaScript/Node.js side of things.

Once again, thanks a million.

from pigpio.

No changes were made to gpioRead/gpioWrite. Unfortunately it's not the sort of thing I benchmark. Was it just a busy loop spin? If you give me an idea of how you were calculating the figures I'll see if I can replicate your results.

from pigpio.

Yes, it was just a busy loop spin in JavaScript, for example here's the test for gpioRead: https://github.com/fivdi/pigpio/blob/master/test/digital-read-performance.js

Is V40 still available for download somewhere? I can double check to see if the figures are correct.

from pigpio.

I'm sure github has a way, may be simpler to download a zip from http://abyz.co.uk/rpi/pigpio/VERSIONS/

from pigpio.

Ok, thanks, I'll compare V40 and V41 to see what the figures are.

from pigpio.

The performance issue isn't related to pigpio. It appears to be related to the number of fields in objects used by the V8 JavaScript engine.

from pigpio.

Thanks for the update.

from pigpio.

Is V40 still available for download somewhere? I can double check to see if the figures are correct.

For anyone who wants to look at this code in the privacy of their own system they can grab a local copy of the code with:

• git clone https://github.com/joan2937/pigpio.git ./pigpio/ in your ~/src/ directory then change into the newly built local repository.
• Start up a copy of gitk other git management applications are available!.
• Select the text describing the commit at that point on the history and right click to Create a branch - give it a suitable name.
• Select the branch name (may need to Update / Reload the view using commands on the File menu) - then right click and Check out this branch.

The directory will than contain a local copy of the files as they were when that commit was made...

• Read up on git - noting that "git{any git command}--help" should bring up the manpage in a terminal window {that being a little quicker to type than "man git-{any git command}" if you have command line editing in place and are part way through typing a command that you need help with! 😀

To store changes you will also want git gui and git fetch installed / available - these two and gitk can call and will interoperate with each other.

Hope that brief start up guide is of use.

Oh, and if you wish to publish stuff for @joan2937 to use you can use the "Fork" command here to make your own copy here on github and use the git remote command to add your fork of this repository into your local system and then push to that and invite Joan to use your contributions with a "Pull Request" here.

from pigpio.

You should be able to get particular versions using the appropriate git commit hash. Alternatively I publish the versions at http://abyz.co.uk/rpi/pigpio/VERSIONS/ as ordinary zip/tar files.

from pigpio.