peterhinch / micropython-iot Goto Github PK

Hi,
on my wipy, I need to implement a bridge, which connects the UART port with a Server and send the message back via UART.
So i created three looping tasks:

• READ_UART: Reading in Data from UART and writes to the queue READ_queue
• PROCESS_FRAME: waits for READ_queue and sends the frame with a "https Post message" to a server.
  The respond is then sent to a WRITE_queue
• WRITE_UART: waits for WRITE_QUEUE and sends data via UART.

Makes this approach sense?
What is the best way to make GET,PUT,POST.. requests on https asynchronously? I had a look at uaiohttps but i can not get it running. I used MicroRESTCli and it worked, but i think that this is not an asynchronous approach.
Also, i need to authorize first with a user and password to get a token for further communication.

Thanks so much in advance!

Title was too short to describe this correctly. I managed to lose a single message after reconnect because the client received 2 messages in a very short time but the unread message in self.evread got overwritten.
Reproducing this is however more difficult than it looks. It was basically a coincidence I found this during my tests with a print statement for each received package before it gets processed. I was not able to reproduce it since.

This is probably because the _do_qos() has no means of keeping a timely distance to the write coroutine. So in unfortunate circumstances these two will send directly after each other.
Maybe a global variable last_tx could be a workaround or using a callback on the client but that changes the API significantly.

@kevinkk525 You earlier mentioned the behaviour of the initial connect(). If I understand you correctly, there are circumstances in which it can block indefinitely. If this can ever occur it will prevent .bad_wifi() from running so the user has no chance of fixing the situation.

If the above is correct, I think we should get rid of it. I'll document the behaviour and invite users to issue connect() in their bad_wifi() implementation.

Hello Peter,
thanks a lot for your efforts!
I am not completely done with the code yet but I'd like to share some thoughts with you:

1. Qos/Guaranteed package delivery:
   The connection might be resilient but as you state in the README there is no guaranteed package delivery. This would have to be implemented in the App which is not a good idea in my opinion as you would have to implement it in every possible App manually or write a general app on top anyways. The common usecase is unlikely to accept package loss and handling package loss in every app needs more effort and also results in code bloat. Therefore I'd suggest implementing QoS directly into the driver

2. Authentication:
   It would be unwise to have a local server without any authentication other than sending a client_id. I'd recommend having at least a simple authentication using a username and password. I know it's not that safe as the connection is not encrypted and the data could be spoofed easily in local lan. At least it prevents the easiest ways of doing damage.

3. Session:
   This seems to be handled in the App. In mqtt you have the option to connect clean or reuse the previous session before the wifi outage possibly resulting in a load of new messages. If the app does not handle this, the server would flood the client with new messages and crash it. This makes it either neccessary to implement some session management or the server would have to dump messages or the client takes its chances. Another option would be number 4, at least for short outages. If the outage is longer then a clean session is needed anyways as after a few hours there could be hundreds of messages.

4. Flow control:
   As the main target of this driver are microcontrollers with limited RAM it could be interesting to implement a flow control. This could be done in a very easy way, bascially like having a socket lock with a timeout:

• server sends data to client
• qos handshake or similar between the two to ensure package delivery
• client signals the server when it is ready to accept new packages while server waits a configurable timeout
  This prevents crashing the client because of new messages and also gives it more control over the speed of processing messages. The value for this could be configured by the client on login.

5. Generic App Interface:
   The current implementation seems to need one client object per App. This is ok if you only use one App but how do you differ between mqtt messages, a "get" request or more? Spawning several clients has a huge overhead and would consume a lot of RAM.
   The solution would be to make the driver more generic by sending an additional header including an App-ID. Then server and client both know, where the message belongs to and use only one socket and driver instance for communication. An App then easily registers its callback for new messages and gets a basic set of functions for writing to the socket. This saves RAM and makes it very modular so you easily spawn a temporary App for some "GET" requests. The server would recognize the the app name and spawn a new server app for this with a uniqe App-ID.
   The exchanged messages would then be similar to mqtt having a bytes header first, something like: app_name (numeric), app_id, message_id, length, qos, duplicate

6. Cancellables
   I understand the use of cancellables and I did not test the difference in RAM usage between using it and having an alternative but by looking at the code, I assume that it uses a lot of RAM. Having an alternative should make the driver less RAM intense and the code easier. But again, this is just a thought as I do not have actually tested it yet.

7. Comparison to your mqtt_driver
   In the README you state that this driver uses less code than the mqtt implementation which is correct but in my opinion a weak comparison as the mqtt implementation has a lot more features and is therefore naturally more complex.

I'd be interested in your thoughts about this. I'm not sure my feedback fits the direction you were headed with the driver.
I'd happily contribute to the driver and write a server-side app for mqtt once the driver protocol is decided.

Thank you shared this impressive project.From the detail protocol content describe " keep alive ", it use the TCP protocol yet? But it can be used the UDP protocl.Thanks!

I guess this is a minor issue in the real application but if you are trying stuff and the microcontroller happens to be disconnected from the wifi, bad_wifi will be executed although it would just need to connect to the wifi as the credentials are stored.
Not sure if this should be fixed or just added to the README.

When trying to build a new server application I encountered the following problem:

The client connects and sends keepalives.
The server accepts and sends keepalives.
The client sends his id, server accepts it but does not answer.
Client.ok variable never gets true, although the led keeps flashing because of the keepalive.

Now according to the code of _readline, a flashing led should mean that cl.ok is True:

        while True:
            if line.endswith(b'\n'):
                self.ok = True  # Got at least 1 packet
                if len(line) > 1:
                    return line
                line = b''
                start = utime.ticks_ms()  # Blank line is keepalive
                if self.led is not None:
                    self.led(not self.led())
            d = self.sock.readline()

I actually have no idea why it behaves this way. Must have slipped myself. Hopefully you can find the problem as it would require the server to actually answer with a real message, not just with a keepalive. I don't think you intended it to be this way.

Same thing happens with your server application. Just comment out the writer loop and the client application will never start sending anything because cl.ok won't be True.

Once you manage to get cl.ok=True then it works and sends everything. Even if the connection breaks down and it does not get cl.ok=True afterwards, it will still send new messages despite the state of cl.ok.
In your example it just does not start because you check that state first.

In my short test when merging our functionality I came across the problem that the client received too many message and drops some. This happens if you use multiple concurrent writes on the server.
Therefore there should be a flow control only allowing a message to be sent every 50ms(?) to give the client time to process the message.

I know this may be a tough topic as it basically can't even be changed without integrating header support into the library and changing the way it receives and sends messages.

I try to explain my problem:
If you want to send a bytes object without converting it, you can easily end up sending an object with a new-line termination. This is especially problematic if you try to bring in a header support.
Let's take this bytearray as an example:
a=bytearray(b'\n\x03"\x00\x00')
The new-line character represents the message-id 10.
Now if I send this header, it will be split because of the new-line character.
I can however convert it first to a string representation and then convert it to json but that would implicate a huge overhead on performance and RAM.
The other option would be to convert the hex representation to an actual integer like 125129825 which can be re-converted using the struct module. Works but also has quite some overhead on performance, transmission length and RAM and is not dynamic, the header would have to have a fixed length.

The solution to this would be to implement a header support for every message containing at least these information: mid, len of the message, use-case (e.g. keepalive, ACK)
Then the library can just read those 4-5 bytes from the socket, figure out the length of the message and then receive the missing bytes. Then a newline-termination as sign of the end of the transmission is possible, however with ACK messages this is not needed as the socket will be empty until the client sends the ACK back.

So I'm basically asking you, if you would consider changing the protocol in that way or not. If not, my header support branch would be quite different, which I would like to prevent.
Implementing the header on top of the library would be an even bigger waste of resources so I'm not really considering this an option.
Changing the header format in any way is better than building on top of the library but still a huge waste of resources and that's why I wanted header support in the first place.
But even outside of header support, the current implementation of the library does not support a reliable way of sending bytes/bytearrays that some people might be interested in.

I published my header fork just now, in case you would like to have a look at it. Latest changes of master are not pulled yet: header

By simply changing this line in the client.py constructor the library could support not new-line terminated ids:
self.my_id = my_id if my_id.endswith("\n") else my_id+"\n"

In my opinion this would improve compatibility as you typically don't have new-line terminated id strings if you use them in other parts of your application too.
If you happen to forget to the new-line termination, the whole library acts crazy, although it sometimes works as the next keepalive provides the new-line termination for the id.

I believe in lines 93-95 in client.py the indentation is wrong and would only initialize the wifi correctly if wdog is False.

else:
            self._feed = lambda x: None

            self._sta_if = network.WLAN(network.STA_IF)
            ap = network.WLAN(network.AP_IF)  # create access-point interface
            ap.active(False) # deactivate the interface

When it should be:

else:
            self._feed = lambda x: None

self._sta_if = network.WLAN(network.STA_IF)
ap = network.WLAN(network.AP_IF)  # create access-point interface
ap.active(False) # deactivate the interface

If the first message (after the ID) is received twice, it won't be recognized because self._init in this line is True and mid does not get added to the bytearray:
if self._init or not mid or isnew(mid):

Every message afterwards will get correctly recognized.

How to reproduce:
Send two message with mid=0 after each other after sending the id.