Server drops inbound messages and receives corrupted input. about alchemy-websockets HOT 35 OPEN

Very curious - thanks for the detailed info. We'll investigate.

from alchemy-websockets.

I am interested in hearing if this is just my system, or can others independently reproduce the issues with their systems on the provided test cases?

from alchemy-websockets.

I'm going to run these tests this evening; I'll see if @Kythera can as well.

from alchemy-websockets.

I am curious - has there been any progress on fixing this? I am really enjoying the ease and speed of this library, but I might have to try something else because of the corrupted data...? Just wondering :)

from alchemy-websockets.

I tried to resolve the problem for a while, but eventually migrated to using the Fleck library, https://github.com/statianzo/Fleck/ , which worked better for me without problems.

from alchemy-websockets.

Yeah, I completely failed to check this out; will be doing so this weekend.

from alchemy-websockets.

I like this library. For now I have increased the buffer size which seems to resolve the issue. It only seems to pop up when the data must be sent in packages. Maybe this helps?

from alchemy-websockets.

I think this is the best websocket server in C#, but I have the same issue with the library as the creator of this thread. It works for small data, but when I try to send lager amounts of data at once, the server will only receive "garbage".
Any chance this could be fixed?

from alchemy-websockets.

It happens to me too. I am sanding larger JSON packets between an alchemy server and alchemy client, and sometimes it arrives garbled. I used a workaround so when I encounter an unparseable JSON object I reset the dataframe (context.DataFrame.Reset()), and send a request to the server to send the data again. Usually it solves the problem.

from alchemy-websockets.

I am able to duplicate this issue. It appears when the server received 32768 bytes or more of data it starts not reporting it correctly to the server. It is pretty amazing that this issue is still outstanding.

from alchemy-websockets.

Additional news: I tested this out with the super websocket and it doesn't have this issue, so I don't think it isn't browser based.

from alchemy-websockets.

Another update: After testing with Safari 5.1.7 and Opera 12.14, there is no issue at all. However Chrome 24 drops message after 32768 bytes and Firefix 18 starts dropping the first 32-64 bytes if the message is greater than 1024 bytes. The major difference between the browsers that work (Safari and Opera) and the ones that do not (Chrome and Firefox) is the protocol used for the websockets. There appears to be a bug in the implementation for rfc6455, which Chrome and Firefox use. I personally don't know enough of the protocol to debug it myself so I would ask that someone please assist. Thanks.

from alchemy-websockets.

we're experiencing the same problem with our application in connection with every recent browser. any news?

from alchemy-websockets.

Same problem here. Has anyone solved this already ?

from alchemy-websockets.

our solution was using SuperWebSocket instead.

from alchemy-websockets.

I have no choice to drop Alchemy too because I'm loosing a lots of packet when sent very quick from another websocket client. I confirm with both WebSocket4net (nuget package) and Websocketsharp.

I did a small program that connect to server after connection is open I do a for loop to send 100 messages. The server print them and there is over half missing. I've switching to SuperWebSocket none were missing... haven't touch the client code. I was preferring Alchemy, less dependency and cleaner to run but it's missing packet is a showstopper.

from alchemy-websockets.

Thinking about using Alchemy for a websocket library.. any progress on this bug? Anyone done any investigations? Easy to fix, hard to fix?

from alchemy-websockets.

Problem still exists - and it's not even hard to reproduce. I have an app that just sends two messages upon connecting. 90% of the time, the second one is lost.

With that, Alchemy is completely worthless for me, all the code gone to waste because no one fixes this highly critical bug for two years.

Switched to Fleck, which - thanks to a similar architecture - was done within 20 minutes. No problems anymore.

Cannot recommend Alchemy at all.

from alchemy-websockets.

@swieser, did you try my fork (https://github.com/steforster/Alchemy-Websockets) also ?

from alchemy-websockets.

@steforster: you're a life-saver, thanks! I spent all morning trying to figure out why I was seeing this. Google led me here, I replaced the NuGet package with your fork and now everything is working perfectly. Cheers!

from alchemy-websockets.

@steforster I didn't try the fork, because Fleck worked flawlessly and I had no need to spend extra time porting my code back to Alchemy. Seeing Myndale's comment, however, I'm glad that someone actually steps up and fixes that (highly critical) bug, and applaud you for that.

from alchemy-websockets.

I have readed the code of Alchemy to include websocket it in my own network classes, and experienced the same problems. When sending big message, they come corrupted, and when I send two or more messages in a short time, I received only the first one !

So I start debugging the code, and re-read the Alchemy processing path step by step...

I have found that when a websocket send in a loop some small messages, they come in as aggregated ordered packets. I found that the Alcemy classe is not processing the remaining bytes when a packet contains the end of a message and the beginning of the next ! Hum... 👎

I have found that the Alchemy is not unmasking all payload segments correctly. In a random manner, some payload segments are not correctly unmasked. The index to read the mask seems to be wrong sometime.

So, I will implement the processing of the remaining bytes and give you a feedback. It will correct the multiple fast message sent and not received, and hope it will remove the big message problem...

from alchemy-websockets.

As you see above, I fixed this in https://github.com/steforster/Alchemy-Websockets and also posted a pull request.

from alchemy-websockets.

@steforster Thanks for you time, I've been lazy and just picked another library but alchemy footprint was much cleaner.

Question, did you made 2 simples test:

• Send 1000 messages and check if the 1000 messages arrives at the server in order.
• Have you try to send big messages if they are not getting cut, sending over 1 MB of data for instance.

I'm kinda tired to change my code and using yet another websocket library that failed basic testing.

Once you tell me both case are working I will try your fork in a live project.

from alchemy-websockets.

I did not send large messages but checked message order of fast appended messages.
Please feel free to improve the unit tests.
There are not many but they are on a client/server level.

from alchemy-websockets.

@steforster : THank you for your submit, it's cool. I've corrected the fast small message in my own code, this is ok. Now i'm experiencing some strange behaviore with big messages sent from the browser, 100 ko + messages are divided in 8048 octet each (standard buffer size), and some payload segments still randomly corrupted.

BUT :

I've noticed that if I slow down the socket by adding Thread.Sleep(2) before restarting Socket.Receive, then all is OK. So, it seems that a strange behaviore is linked to a race condition or IO Completion Port used for high performance Sockets and low GC pressure. But it is possible I've leave a bug in my own code, because I'm using only some small parts of the Alchemy lib (header, dataframe).

I'm starting to debug that now.

from alchemy-websockets.

First result in my debugging ; if you call immediatly ReceiveAsync then corruption appear randomly. This is the case when you do near nothing between the Receive call back and call to ReceiveAsync. Read this article - http://www.themissingdocs.net/wordpress/?p=615

This is my case : I simply queue the receved buffer to be processed by another thread, so the processing is nothing... This is possibly the case in Alchemy wich is only add the received buffer in the segment array, wich is too fast to avoid the race bug.

So I'm trying to find a simple but fast solution to go arround the problem, without GC pressure (many buffers, many structures) ;

• use many ASyncEventArgs in a circular manner
• integrate a non blocking delay before restart receiving...

from alchemy-websockets.

I would be surprised to see this error when using my fork. There I fixed several bugs concerning the async behaviour on server side.

from alchemy-websockets.

Yes, but you'd better execute the test of sending 100 times 1 Mo buffer and check if all received buffers are good. You cannot say "there is no problem" if you never test it !

What is true in Alchemy, is that the packet is fully processed BEFORE call ReceiveAsync, so, the time needed for this processing avoid the bug apparition !

In my rewritting I do the packet processing in another thread, and restart immediatly ReceiveAsync. I've test various solutions, with 100 EventArgs, with single buffer, etc... nothing is working. I still have the problem. The dellay between callback call and ReceiveAsync to avoid the bug is really thin, but it cause a sever penalty in case of high performance server.

I will start new tests, with unsafe pinned buffers this time.

from alchemy-websockets.

Nice to see some activity on this issue, great work. However, let me unsubscribe myself from this, since I am no longer using this library. Hope it gets resolved - good luck!

from alchemy-websockets.

Just to throw my 2c in the mix…I tested your fork on a facility containing a centralised server and about a dozen different clients all connected simultaneously and I saw heaps of corruption issues. Unfortunately I was under a tight deadline so I didn’t get time to track them down properly but my tests seemed to indicate it was due to Alchemy not being correctly thread-safe.

Switching to WebSocket4Net resolved all problems immediately and the system has been stable ever since, so I’m reasonably confident it’s not an issue with our code…unless of course Alchemy isn’t actually meant to be thread-safe?

Mark

From: Stefan Forster [mailto:[email protected]]
Sent: Thursday, 10 July 2014 9:39 PM
To: Olivine-Labs/Alchemy-Websockets
Cc: Myndale
Subject: Re: [Alchemy-Websockets] Server drops inbound messages and receives corrupted input. (#25)

I would be surprised to see this error when using my fork. There I fixed several bugs concerning the async behaviour on server side.

—
Reply to this email directly or view it on GitHub #25 (comment) . https://github.com/notifications/beacon/3477140__eyJzY29wZSI6Ik5ld3NpZXM6QmVhY29uIiwiZXhwaXJlcyI6MTcyMDYxMTU2NywiZGF0YSI6eyJpZCI6ODMxOTF9fQ==--9473078e69b4b6aef0c0b2b52be4380c24746fc3.gif

from alchemy-websockets.

I agree with @Myndale this library is not ready for prime time. Basic testing and multi-thread safety is a must.

from alchemy-websockets.

So, in my last research i've tested the ASync .Net API separatly and there is no problem with it. High perf, low garbage pressure. All is strange, because I can reproduce the behavioure of code presented with the article I posted yesterday. I will start to analyze the potential threading issues... The error can appear on a 8 core PC and not on a 2 core one, because the thread pool managing the IO Completion Port can be larger, and an unfinished message processing operation can be corrupted by the next one started by another threads.

So, I think to test it, everyone need to ;

• Use a multicore machine
• Process multi-megabyte messages

For the Alchemy implementation, I cannot say what would be the result.

from alchemy-websockets.

Ok, I've found my error : it's exactly a problem of bad protection of critical section and processing overlapping. Must add monitor to prevent that, and it's all ok. For Alchemy, I hope that the work done by steforster is solving all problems of threading, but I've haven't seen anymore problems in the code I extracted from the lib. Steforster seems to be really confident. Thank you all !

from alchemy-websockets.

The big message drop is not a bug, it's a hard limit in the code.
public UInt64 MaxFrameSize = 102400; //100kb in context.cs set the limit, so you can increase it.

The fork from steforster ( https://github.com/steforster/Alchemy-Websockets ) is not having any bug from what I've tested. I think the author should do the merge.

from alchemy-websockets.