We've actually contacted the author of the utp4j library and he personally suggested to rewrite it from scratch.
Main reasons being code quality issues and being in quite experimental state.

Still in the contact about some other details, but this was his main reply about the maintenance/taking over the project.
-- EDIT: After further conversation, the author agreed to transfer ownership (to me for now)

Current versions:

• utp4j (fork) https://github.com/PieterCarton/utp4j
• utp4j (original) https://github.com/Eragoneq/utp4j
• trustapp (fork) https://github.com/Eragoneq/trustchain-superapp

Current week progress update:

• Unifed the workflow on github
• Pushed changes to our forks
• Managed to exchange data between 2 phones (with uTP on local network) (253 KB/s)
• Fixed existing issue in utp4j library and updated dependencies

Current view in the app:

from tribler.

Weekly Update Benchmarking Team:

• upon investigation of UTP with 500ms latency, congestion window was tiny as expected
• 2 issues found:
  • when connection is very stable, UTP estimates the variance of the latency to be 0ms. Since UTP uses the variance to determine when a packet times out, this leads to phantom packet loss and limits window size
  • when latency is very high (~500ms), the congestion window only increases very slowly. Upon investigation, congestion window growth is heavily dependent on Round Trip Time. Increasing the congestion window gain parameter greatly increases performance in the benchmarks
  • max congestion window gain does not accurately respresent window gain in practice, there is factor 10 decrease in window gain https://github.com/PieterCarton/utp4j/blob/7bd348b8ca849d6c4ab2327fdd4a22ed23082bcf/src/main/java/net/utp4j/channels/impl/alg/UtpAlgorithm.java#L223

Benchmark results (performed on 4mB file):
(Default parameters coloured blue)
(Benchmark script: https://github.com/PieterCarton/UTPBenchmark/blob/186d5a8a10a0e6afb857330ed2d96cdbaac05663/benchmark.bash)

At 500ms delay, with 10ms jitter and 1% pkt loss

At 50ms delay, with 2ms jitter and 1% pkt loss

At 0ms delay with 1% pkt loss

from tribler.

Here is the demo of how it looks at the moment, as soon as we receive a package from one of these peers, the status indicator should turn to green.
Red: status idle
Orange: sent
Green: received.

Post-meeting notes:
These will be changed to the following:
Sent without ack: yellow
Sent with ack: Green
More than a minute without communication: yellow
More than 3 minutes without communication: red

from tribler.

This week progress:

• Managed to use IPv8 socket to transfer data
  • Using the low-level prefix in UDPEndpoint layer
  • Overriden socket to use new channel to be fillled with data
  • Connection works in every case that peer is visible in the community
• Created separate UtpCommunity to show clients capabilities
• For now receiver always accepts any incoming data
• Deleted old connection code
• Still working on custom payloads for heartbeat and transfer request

from tribler.

For the task to get the uTP transfer going, should we just make a completely fresh project and somehow already integrate it into the current project?

uTP Thesis - link

from tribler.

First task, get uTP working, minimal changes. No superapp integration yet.

from tribler.

• not as much progress as needed for a successful mid-term
• Found null pointer bug in uTP lib, typos, and changed internals
• not as much time spend on course as required (12-13 hours/week)
• EVA, IPv8, NAT puncturing initial investigations
• did small trails with different laptops 🚀

Sprint goal this week: structural measurement table of uTP. read example table
Quality of networking protocol determines the performance. Read background thesis with details
investigate tc utility, or netem wrapper

Measurement environment with packet loss and latency with reproducible performance test. One script to produce the example table with various conditions: 2 laptop and 1% packet loss rate

from tribler.

• organised into 2 teams:
  • 1. sub-app team and debug (@Eragoneq and @juul-j)
  • 2. benchmark team ()
• test mode inside Superapp
  • benchmark utility for connecting and profiling 2 phones
  • Enhancement of https://canyouseeme.org/ (without central server)
  • add to superapp, wireshark debugging, emulator port forwarding
  • proxy setup
  • no success yet 2 phones connecting with uTP
  • hint: no fancy overengineering, generic, universal connectivity API, minimal viable code.
• Benchmarking team
  • got the 11 years old testplan.csv to work MICROS_WAIT_BURST is only parameter being changed
  • 91 test case scenarios, 90 lines
  • burst size seems to be important says @CallumH2410
  • no idea yet why 3 KBit/sec or 9 MByte/sec
  • Structural testing of default parameter setting and changing 1 setting and the time plus determine outcome
  • Even Docket to Docker operational and profiled 🚀

FROM python:3.8-alpine
RUN apk add iproute2 vim iputils-ping python3 py3-pip openjdk11 bash 

ToDo:

• sprint and documentation how to get a connectable port from Android Studio with forwarding.
• exact line of code to uTP with null-pointer bug?
• rounding errors in uTP test code
• We can keep assignment as is; or make it more interesting:
  • focus exclusively on test transfer between 2 phones
  • just 1 networking debug screen, no new sub-app, no new dependancy, just add existing code to running code
  • debug screen can test transfer 2 hard-coded files: votes.csv (3.3. MBytes) and votes.csv (13.7 MBytes)
  • change the parameters for the test, such as,
    • packet size/mode,
    • packet drop 100% 🏎️
    • packet re-ordering
    • fast retransmit
    • congestion control.
    • How about 1-way receive mode, send mode?
    • 2-way send&receive mix traffic mode?
  • Decide between full tuning and both phones or 1 phone is the leader and automagically sets up the remote receiving parameters.
  • Make the uTP library mode "self-testing".
  • Inspiration from more (too generic) generic approach
  • no docker needed, extend the testplan.csv with minimal changes needed for 1 new column: packet drop rate {0..100%}. So all in same Java lib.

from tribler.

Varying Package size/mode

Skipped packages needed before fast retransmit

Target Delay of Congestion Control

Test performed with 1% package loss under 250 ms latency, transfer speed taken as average of 10 runs

Benchmark Program Location: https://github.com/iiljkic/utp4j/blob/master/src/main/java/net/utp4j/examples/configtest/ConfigTestWrite.java

from tribler.

Week Update:

• Forked UTP4J library and pushed previous modifications (see comment of @Eragoneq for more details surrounding situation)
• Implemented basic benchmarking "hand-shake": sending party in benchmark determines testplan to follow, receiving party follows plan
• Migrated benchmarking from docker to work on local machine
• Partially redid current benchmarks
  • Drastic improvement in performance compared to previous weeks when no latency is present (speeds of up to 50mB/s at its peak, avg. speed around 25mB/s).
  • Packet loss has little to no effect on these tranfer rates
  • However, even small amount of latency makes the transfer rate collapse (50x decrease in speed at 50ms latency)
• investigation of slow speed through wireshark
  • transmitter only actively sending packets fraction of the time, then idle for multiple ms
  • likely congestion window too small -> congestion control not optimally configured for medium to large latencies

Example: fast restransmission Benchmark without latency

Example: same fast restransmission Benchmark with 50ms latency

from tribler.

• "the other group is using the library wrong", inverse logic of client/server. Server == receiving in common case, resulting sequence number ==1 or random. ???? 😂
• 7390 line of code added. What do other master courses say about mixing fixes and global textual changes?
• Problems with utp4j lib:
  • overengineering, too abstract, boilerplate
  • bad inconsistent naming
  • CPU performance heavy, not optimised and not profiled. (5% cpu usage on laptop)
  • it collapses with latency, so is incapable of keeping the packet request queue filled (or window size wrong).
• please focus on the 500 ms case, which is bad 4G/5G networking. But still realistic.
• For mid-term, focus on an Android-14 compatible APK on this issue for transfer testing 👓
• Create an example IPv8 community with utp4j and use endpoint-send-ipv8 to send a single UDP via endpoint override
• Clear 500ms-case library/performance fixes could be cardinal outcome of your Master course project!

from tribler.

Starting off this week updates:

• Transferred the existing utp4j library to the organisation

  • Further pull requests should be reviewed by other team members
  • Post issues of library directly on the repo

• Managed to fetch peer data

• Sending over wireless network (local)

• Attempted to implement sending over IPv8 socket

  Current app version: apk

from tribler.

Callum: Was sick for most of last week so will need to do some catching up the upcoming week(s).
Some issues getting our forked repositories working based on space issues with my laptop (Michals contributions), got it working for physical device so managed to get some front-end changes in.
Will be looking to pick up the lost hours of last week in the upcoming week(s).

from tribler.

• Dream goal of MBytes/sec transfers in a phone-2-phone network
• uTP library has no test code for packet timings and formulas like RTO (retransmission timeout) {thesis})
  
• untested code of this complexity will have bugs 🤔
• parameter setting like max congestion window gain seem to be optimised in simulations, not 4G and 5G networks.
• Create various example unit tests for private void updateWindow() function which contains lots of complexity ?
  • loss of 1 packet
  • packet arriving late
  • focus on 3 packets and 10 packets?
• uTP owns the UDP socket()! deep integration with IPv8 requires big stripping action
• Read about IPv8 peer discovery, see basics
  • active demo community walking every 5 seconds
  • use 3 colors please plus incoming and outgoing connections.
  • orange, just heard recently about a peer, no connection.
  • red, heard about a peer over 60 seconds ago, never connected.
  • green, got an incoming packet in last 60 seconds
  • yellow, got an incoming packet in more than 60 seconds ago
  • grey, got an incoming packet in more than 120 seconds ago
  • remove dead peers after 300 seconds
• expand your puncture, send, and receive into full network debugger
  • see design and running code from 8 years ago 😲
  • #2131 (comment)
  • 
• Robustness: packet loss can happen to the initial send_binary(). Make sure to re-try the initial request for a uTP transmission, if nobody is answering you.
• awesome bonus, birthday attack details

from tribler.

This week:

• work on NAT puncturing
  • direct open-port connection works
  • less restrictive NAT works
  • still has errors
• making the send/receive more seamless
• integrating data size and status messages into requests
• updating the UI

from tribler.

Week 6 Report Benchmarking team:
Last week there were Prof. Pouwelse addressed some concerns about the complex piece of code that controls the congestion window

This week:

• We compared the UTP4J code to the UTP specification and LEDBAT specification, and it seemed to be up to specifications
• Unit tests were created to verify correctness of UTP4j congestion control, where we test the effect of latency caused by congestion
• Unit test were also made to test the packet time out behaviour
• Some benchmarks were rerun with larger file sizes, effectiveness of boosting congestion window gain is decreased
  Example:
  Benchmark of transfer of 50mB file at 50ms latency
  

from tribler.

Current UI Design Proof of Concept:

from tribler.

For the next week:

• The UI looks a bit rough. Need to keep working on it

• The receive button is irrelevant, needs to removed, or hidden. The server for receiving is running all times

• Separate port for UTP is used but there are still issues. Own public address/port not know, and there is reliable way to know if other peer is running UTP. One simplest option might be use third party to help bootstrap and discover UTP port. Any ideas, @qstokkink, what is the simplest and most elegant way to integrate UTP and IPv8?

• Focus on the demo with an app for the next week.

• The colours show the Pv8 status to see if peer is online.

• Think how to combine two status of the peer: UTP and IPv8

• Start working on the documentation.

from tribler.

@qstokkink, what is the simplest and most elegant way to integrate UTP and IPv8?

These are two opposites. The simple way is not elegant and the elegant way is not simple.

First, the difficult/elegant way would be to have channel separation for control logic and data streaming. You would have to puncture the NAT in a separate community with a separate socket and "hand off" this socket to your uTP streaming protocol. In a fantasy world, this would be on a separate physical channel as well (but this is obviously not realistic for consumer connections). You would need to use some form of DHT to couple the peer identity in the streaming community to the identity in your control logic community.

Second, the simple way: create a new IPv8 message to wrap your uTP packets. You will not get optimal performance, you will hinder other communities, and you will not have nice logical separation.. but it works 🤷 If your goal is to "just get something running", pick this option.

from tribler.

@qstokkink

I think that still the "difficult" solution would be actually easier if we're supposed to reuse the library code. As right now all the socket stuff, threads and listeners are quite coupled with the logic of uTP making it hard to just reuse the existing IPv8 socket. I've already attempted it and made quite a bit of progress, but still there would need to be quite a lot of refactoring required in the library to actually get it working well, which then makes it even less sense to use the library and probably just better to rewrite it from scratch with the separation (due to other known issues).

For now I was able to get the separate socket and everything working on local networks only reusing the IPv8 to exchange status messages about file transfer request and other metadata, then the separate socket is used to actually send binary data. I'm now considering what would be the way to actually properly get the NAT puncture to work. Do you have any suggestion on what to use for it to work in a simple way? I don't think it's possible to somehow open another port with the existing p2p connection, we'd still have to use 3rd party to establish the external ports and that's what I'm currently considering.

from tribler.

Weekly update:

• added logging of certain utp connection data in UTP debug fragment inside app
• started documentation of UTP benchmarking repository (link)

from tribler.

• {repeating} Dream goal of MBytes/sec transfers in a phone-2-phone network
• {repeating} please focus on the 500 ms case, which is bad 4G/5G networking. But still realistic.
  • 1 person 1 day
  • debugging, ensure it works, flawless as possible binary transfer
  • test setup: 3 hardware Android phone, 4 emulators, and the "Week 10" 2 test phones.
• After 8 weeks the effective magic setting for the congestion window increase is not in the code
• Please eliminate the server!!!
  • hard-coded IPv4 address
  • helpful to get something working
  • internal machine, will never scale
  • testing only
• IPv8 details: https://py-ipv8.readthedocs.io/en/latest/reference/peer_discovery.html
  • you talk to somebody
  • they already talk to somebody else
  • that somebody else also send you a pucture message 😲
  • this stranger is ready to receive your next "walk" in the network
  • 1 introduction-request message trigger 3 packet transfers (DoS 👎).
  • 
• Disagrees with @qstokkink unfortunately.
  • the design requirement is not efficiency, but fault tolerance. Hence use a single socket.
  • puncturing is a stochastic process that not always works. Thus avoid at all cost
  • we do not want "double puncture" logic in the networking stack. Puncture for signaling port and streaming port.
  • requires the 3 packets punctures for the stream socket 🤔
  • use IPv8 UDP transport port to transfer the uTP stream. 20 Bytes of overhead on 1500 Bytes is OK.
• Single repo for uTP. Include unit test, include benchmark

from tribler.

Great progress. The apk is working with speed up to 1 MB/s

For next week:

• Make sure you have running demo
• Finish the documentation, compile the benchmarks into one place and write the Gotchas into one file (big file transfer ..), what decisions you made
• Apk ready to be installed and repo with the code.

from tribler.