Problem: understanding the health, bugs and faults in deployed P2P software is hard
Goal: use a package with wide usage that solves this, instead of our own custom code.

Only enable with user consent (plus enabled by default in test releases).

Backend: http://graphite.wikidot.com/
Network deamon: https://github.com/etsy/statsd
Python wrapper: https://github.com/WoLpH/python-statsd/

Revision: 28434
KeyError: 'swiftmetadir'
Traceback (most recent call last):
File "/home/pouwelse/GITHUB/tribler/Tribler/Core/RawServer/RawServer.py", line 175, in listen_forever
func()
File "/home/pouwelse/GITHUB/tribler/Tribler/Core/Swift/SwiftDownloadImpl.py", line 145, in
network_create_engine_wrapper_lambda = lambda:self.network_create_engine_wrapper(lm_network_engine_wrapper_created_callback,pstate,lm_network_vod_event_callback,initialdlstatus)
File "/home/pouwelse/GITHUB/tribler/Tribler/Core/Swift/SwiftDownloadImpl.py", line 158, in network_create_engine_wrapper
self.sp.start_download(self)
File "/home/pouwelse/GITHUB/tribler/Tribler/Core/Swift/SwiftProcess.py", line 220, in start_download
metadir = d.get_swift_meta_dir()
File "/home/pouwelse/GITHUB/tribler/Tribler/Core/DownloadConfig.py", line 846, in get_swift_meta_dir
return self.dlconfig['swiftmetadir']
KeyError: 'swiftmetadir'

We've already moved the libswift mhash and mbinmap files to different dir, but it would be nice if we moved the swift multispec file to the same directory as well.

• 100Mbps sustained downloading

open issues:

• on-demand piece selection
• DHT export for .torrent collection and libswift announce
• for partial downloading: .part file
• when downloading multiple swarms in 1 directory: collision of identical files and oscillation

Goal: boost spam prevention
Approach: get more (anonymized) feedback on swarm quality automagically

The effort of identification of good content puts a burden on users.
Due to this effort we have a vocal minority which bears all responsibility for promoting the good content. They need to join channels using "mark as favorite". Others are the silent majority.

By automating part of the promote good swarms process we give a voice to the majority. The VLC player integration in Tribler gives us an opportunity to automate good content promotion. If a swarm consists of a single large video file we can use a simple heuristic: content is acceptable if over 75% of the content is played. If this threshold is reached it triggers a automatic vote in AllChannels. Auto-Votes are half the value of explicit votes in our spam-prevention algorithms plus at the swarm instead of channel level. Users can deactivate auto-voting in the setting menu (opt-out). To keep implementation complexity to a minimum, no state is kept on disk for playback progress. Only the viewing progress of the current swarm in playing VLC is used for auto-voting. Before casting a vote, check if not already done so in AllChannels.

With multiple multimedia files in a single swarm we need to devise a more complex heuristic. For instance, if more then 75% of the files or three files (whatever is reached first) are played for over 75%, an autovote is triggered.

To bootstrap the search, allchannels and other communities we currently require connectable servers.

The aim of this enhancement is that all connectable peers automatically form a community bootstrap overlay. An alternative is to use our DHT, but connectability and lack of NAT-awareness make this less attractive. Similar to the Kazaa 2001 and Buddycast 2005 design we probably better use a local host-cache with the 200+ peers which we've seen in every community.

The Tribler installer will no longer include a hard-coded list of 16 bootstrap servers, but a "peer_cache.txt" with a few hundred peers to find the network. On a daily basis we refresh this peer_cache with stable connectable peers. Plus we put this updated file in the Linux/Mac/Windows installers.

Aim: create a fully functional web browser which also support (anonymous) P2P video streaming.

Outcome: P2P-Browser for HTML5-based websites with any-format video embedding, Javascript and CSS.

Approach: build upon the new WxWidget HTML rendering in wxpython 2.9:
http://docs.wxwidgets.org/trunk/classwx_web_view.html
A browser in 800 lines: http://trac.wxwidgets.org/browser/wxWidgets/branches/SOC2011_WEBVIEW/samples/webview/webview.cpp
We can even play Youtube (and grab it for P2P seeding in Libswift it seems):

wxWebView *webview = wxWebView::New(this, wxID_ANY, "http://www.youtube.com/watch?v=zwDvF0NtgdU");
webview->RunScript("function onYouTubePlayerReady(playerId) 
     {document.getElementById('watch-player').playVideo();}");

For windows, HTML5 video tag support needs IE9 libs :-(

For P2P streaming we build upon our own upcoming IETF Internet Standard work: https://datatracker.ietf.org/doc/draft-ietf-ppsp-peer-protocol/
We use VLC to PlayVideo(), render in that windows and support any video format plus HTML5 power?
Old VLC adding patch exists: http://trac.wxwidgets.org/ticket/10264

P2P streaming support appears easy to add via the wxWebViewHandler. Assuming Python we can make a call to the libswift process and present the output of the HTTPGW as the inputstream for the web engine.
VLC will be used for playback. All mainstream containers+codecs will thus be supported, so we've created a HTML5 element that can play all formats. Bittorrent swarms can thus be easily be supported.

Limitation: web sites need to be specifically created to support Youtube-like P2P streaming.

Currently the only way to cancel/delete a download is to select the download and then move the mouse all the way up to the top toolbar.

Revision: 28434

Could not send search, AllChannelCommunity not found?
Traceback (most recent call last):
File "/home/pouwelse/GITHUB/tribler/Tribler/Core/APIImplementation/LaunchManyCore.py", line 611, in resume_download
pstate = self.load_download_pstate(filename)
File "/home/pouwelse/GITHUB/tribler/Tribler/Core/APIImplementation/LaunchManyCore.py", line 818, in load_download_pstate
pstate = pickle.load(f)
File "/usr/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatchkey
File "/usr/lib/python2.7/pickle.py", line 880, in load_eof
raise EOFError
EOFError
tlm: could not resume checkpoint /home/pouwelse/.Tribler/dlcheckpoints/8739d37e113763e171c59e5ba5e1229affba2e6e.pickle None

December 2008 docs: http://www.tribler.org/api/

wantpeers is forced to True once a user visited the Torrent details in the GUI. While this is correct, it does not set wantpeers back to False when the GUI is no longer showing the Torrent details. Currently wantpeers stays True until the download is finished.

See list_details.py:

class LibraryDetails(TorrentDetails):
    @warnWxThread
    def __init__(self, parent, torrent, bw_history):
        self.old_progress = -1
        self.refresh_counter = 0
        self.bw_history = bw_history
        TorrentDetails.__init__(self, parent, torrent)

        # Arno, 2012-07-17: Retrieving peerlist for the DownloadStates takes CPU
        # so only do it when needed for display.                                
        self.guiutility.library_manager.set_want_peers(True)

    def __del__(self):                                                          
        TorrentDetails.__del__(self)
        self.guiutility.library_manager.set_want_peers(False)

As a further optimization, we should not use wantpeers at all, unless we want the peers information about all the downloads (including background libswift downloads). Instead we should request information on a specific download. The control is already there in the SwiftDownload API. Use: download.set_moreinfo_stats(True) and the next callback will include detailed statistics.

The latest build fails on my 12.10 Ubuntu box.
http://jenkins.tribler.org/jenkins/job/Build-Tribler_Ubuntu-12-10_release-6.1/72/

Seems to no see that the required Libswift dependency is installed.

$ sudo dpkg -i /home/pouwelse/Desktop/Download_Dir_Chromium/tribler-swift_6.0.0+svn20121015.r28821-1ubuntu1_i386.deb /home/pouwelse/Desktop/Download_Dir_Chromium/tribler_6.0.0+svn20121015.r28821-1ubuntu1_all.deb 
(Reading database ... 226470 files and directories currently installed.)
Preparing to replace tribler-swift 6.0.0+svn20121015.r28821-1ubuntu1 (using .../tribler-swift_6.0.0+svn20121015.r28821-1ubuntu1_i386.deb) ...
Unpacking replacement tribler-swift ...
Preparing to replace tribler 6.0.0+svn20121015.r28821-1ubuntu1 (using .../tribler_6.0.0+svn20121015.r28821-1ubuntu1_all.deb) ...
Unpacking replacement tribler ...
Setting up tribler-swift (6.0.0+svn20121015.r28821-1ubuntu1) ...
dpkg: dependency problems prevent configuration of tribler:
 tribler depends on tribler-swift.

dpkg: error processing tribler (--install):
 dependency problems - leaving unconfigured
Processing triggers for bamfdaemon ...
Rebuilding /usr/share/applications/bamf.index...
Processing triggers for desktop-file-utils ...
Processing triggers for gnome-menus ...
Processing triggers for gconf2 ...
Processing triggers for man-db ...
Errors were encountered while processing:
 tribler

• Tribler anonymous downloads are fast

  Only 330KByte/sec is unacceptable for our video streaming use-case. Currently our GCM cypto or other bottleneck is slowing things down. #1882

• Tribler users never seed naked in Bittorrent swarms.

  Bittorrent swarms offer no protection, they leave people exposed and naked. Hidden seeding is the default in Tribler for sharing content and cannot be disabled. This ensures sufficient content becomes available with strong privacy in our darknet. Hidden seeding requires connectability of rendezvous and introduction points.

• Tribler can download from both hidden seeders and existing Bittorrent swarms.

  Tribler offers the best of both worlds. It always prefers downloading from hidden seeders using end-to-end encryption. If there are no hidden seeders available, you can anonymously download from normal Bittorrent seeders. Tribler offers this fallback option by default after trying for 180 seconds. This feature can be disabled easily if you prefer pure darknet-only downloads. Fallback option background: #885

• Tribler users never become an exit node, unless they explicitly volunteer.

  Our smooth backwards compatibility with Bittorrent comes with the cost of exposure in existing Bittorrent swarms. We need volunteers to run "exit nodes". These are the peers that receive encrypted messages from the darknet and unencrypted messages from Bittorrent peers. Running an exit node has proven to be a risky non-trivial task for the main Tor community. Lawyer-based attacks are very real for exit node volunteers. Ideally, only a single copy of each piece of content needs to be relayed by our exit nodes. If content is "pulled into the darknet" we can switch to end-to-end encryption and rely on hidden seeding. Ticket: #1174 @rjruigrok

• long-term tit-for-tat for anonymity

  Develop an open community to disseminate sharing ratio of participants. You need to help and relay traffic of others. Related experimental work in the Tor community: GoldStar, PAR, BRAIDS, LIRA, TEARS, and TorCoin (Tribler Tickets: partly #1230, also #5)

• Collective intelligence around reliability of anonymous relay partners

  Share with the rest of the network who is actively helping others and who is freeriding. Introduce a Bitcoin-like accounting system. Instead of pointless hashing we use proxy helping bandwidth as our proof-of-work. Our protocol follows the Tor specification as much as possible, our circuits have a maximum lifetime of 10 minutes and also terminate after 55MByte of traffic (security limiters). Similar to private Bittorrent communities we use the sharing ratio to identify freeriders. Double signed signature record is created after proxy helping. These proof-of-work records are shared in the community (like Bitcoin public ledger). Each peer builds a database with the reliability of their neighbors, visualize in a graph (locally; no central accountant, controlling server). #5

• real-time statistics of your relaying contributions and tunnel ratio

  Notify users they are freeriding and disable anonymous downloading. Be harsh, Tribler is only for social people!

• real-time anonymous tunnels reliability statistics

  Each member of the collective measures their perception of network performance and shows it to the user. Advanced programmer-minded feature and debugging. Add to the Tribler debug screen how many unique neighbors have been seen. Measure the proxy bandwidth donations and online duration (reliability) of yourself and neighbors. Show broken tunnel events. Show overall tunnel + hidden seeding reliability and performance for a tunnel tests and 50MB downloads.

• improved anti-spam measures, family filter accuracy and metadata in general

  Toward a Youtube-like experience within Tribler. With just as reliable metadata. @LipuFei

• credit mining for anonymous seeding

  Automatically boost poorly seeded anonymous swarms. Maximize the yield of your bandwidth by automatically detecting and anonymous seeding of swarms. #573 #655 #21, running code: #23

• darknet mode: deny any connection with strangers

  Here we start to deviate fundamentally from the random-circuit building approach, used in Tor. In our fully distributed setting it is difficult to protect yourself from various kind of attacks, like the Sybil attack. By not accepting connections from strangers we constrain a lot of attack classes. Every Tribler user that has a healthy set of neighbors to relay traffic with shall not interact with strangers.

• DTLS

  Besides our existing onion encryption we need additional point-to-point encryption. This is not needed for simple lawyer-based adversaries or snooping ISPs, but for more advanced attackers.

Move libswift files to different dir?

visualization of trust buildup for virtual identities.
Key issues: rendering, smoothing and node placement as nodes get added and pruned.

Depends: #20

initial prototype:

Determine the contributions of peers using our blockchain.

Track freeriders using MultiChain, compatible with anonymous seeding, relaying, and downloading mode.

Goal: superior performance compared to tit-for-tat, reduce freeriding and promote cooperation.
usage: accounting of bytes for seeding and Proxy bandwidth donations
Security: Sybil attack hardened, misreporting, lying, cheating, white washing and spoofing-proof.

Previously: Vlad thesis project, now Ewout!

#6 SUBTICKET
Firste step to enable creation of Tribler-specific probes
#6

Darknet search:

• chains of relay nodes forward anonymized queries
• no privacy leakage outside chain, minimized leakage between relays
• chains are formed between semantically-near peers (item co-occurency)
• this semantic overlay is formed exclusively in the encrypted domain
• use elliptic curve crypto protection
• keyword searches are forwarded through tunnels
• improve uncertainty and performance by caching of replies of others in the chain

We should control the free space and pause downloads and warn the user when we reach a given (configurable) threshold.

Goal: Create a Tor-like system which supports streaming
Approach: Fork Tor, make suitable for HD streaming

Re-use core messages from Tor: https://gitweb.torproject.org/torspec.git?a=blob_plain;hb=HEAD;f=tor-spec.txt
Add integrated NAT-traversal, user-donated proxies, bandwidth accounting and reward contributions with priorities.

Use Proxmox virtual machines to compare a Windows install of:

• uTorrent
• Tribler

Test the download speed and CPU usage automatically via Jenkins.
Workload: seed 250 files. Each file has a size of 1 GByte, thus 250GByte of seeding.

Download randomly 5 of the seeded files. Determine: upload speed, memory usage, CPU load and IO workload.
The workload somehow resembles that of a private Bittorrent community, mostly idle and then try to upload at full capacity to a single leecher.

Graph1: average upload over the 5 swarms. Single number for both clients. Plot with X-axis client name, Y-axis the average upload.

Graph2: upload progress in time. Show 5 lines for each of the 2 tested clients with the file completion percentage in time.

Example to follow/duplicate: http://speed.pypy.org/

Now it's just the roothash.
for 30 seconds

With some basic scripts to get a taste of SystemTap so we can start toying with Jenkins integration.
Part of #6.

We should add a way to check if there's a newer Tribler release. At least for when it crashes so we don't get reports for possibly fixed bugs.

Aim: Offer a similar experience as Youtube

• GUI: show animated "loading 51%" before video playback begins
• stream playback should begin directly after "loading 100%".
• Create a progress slider which indicates how far we are in the video clip
• keep track of "linear download progress", which is bytes completed from start
• GUI: show in grey how far we have downloaded and where a seek is instant
• seeking beyond current download progress will trigger the PiecePicker to switch there
• GracefulPause: automatically pause on video buffer underrun
• GracefulPause: display "busy animation" in middle of screen (ToDo check IO students).
• GracefulPause: automatically resume if buffer level is healthy again

Key problem: detect the video format, bitrate and duration info.
We need to download the header and give it to a video decoder for analysis or feed it to VLC as we go and extract info. Note, sometimes VLC requests bytes from the end of the file (for certain video formats).

Open problem: after the format, bitrate and duration is detected, when do we start playback?

Tribler uses 30% average CPU usage in idle of v6.1 on a single i7 core.

Dependancy: insight in performance #6 and fix inefficiency # 1 (this ticket is repurposed).

Old Mac Book Pro performance (idle, no downloads, no seeding):
55% average CPU usage on older 2.2GHz Intel core 2 duo (with 99% peaks)
360 IO operations/second at peak
309 network packets transmissions/second at peak

Most expensive GUI operation:
Viewing a subscribed channel with thousands of torrents.
It takes 15seconds to respond and CPU goes to 130-140% range

System monitor networking view and Tribler Debug info (walker,drops,tracker,etc)

IO view:

Point to latest code
General status, make empty and point to http://Github.com/tribler/libswift

The Developers Doc Portal

Goal: enable starting developers an immediate broad grasp of Tribler
Approach: single webpage with pointers to all documentation, low-maintenance, no sub-pages.

• Tribler basic functionality with screenshots: http://jenkins.tribler.org/jenkins/job/Test_tribler_full-ui-run_61x/lastBuild/
• Tribler architecture: modules and their functions
• Database structure: Tribler tables, Dispersy core and Dispersy communities
• Tribler API docs: Tribler download core, Libswift and Dispersy core
• Continous build system: Jenkins performance, unit test, and system tests
• Tribler, Dispersy and Libswift testing framework: DAS4 tests (Effort / Allchannels)
• Dispersy wire protocols: Walker, Allchannels, Search, BarterCast3
  https://github.com/Tribler/dispersy/blob/master/doc/wireprotocol_2.x
  SHA: b8eab4fb501213ae151372cd83a6131582e2ad6b
• Dispersy access control, security and certificate validation
• Dispersy core: event handling, internal data structures and fault behavior

How much can we autogenerate? (zero-maintenance..)

aim: determine exactly the CPU,memory, IO and bandwidth cost of running Tribler for 1hour

Run the SystemTap (#6) shell script with various probes into Jenkins
Single instance, complete GUI and live Tribler network: bootstrap, search, join channel and download.

Possible used for performance improvement progress with 5 minute and 1 week runtime variant. Determine bootstrap speed and memory leaks.

Not really usefull I guess.
For example, I can sort by swift ratio in the downloads panel without having it shown.

Currently the EffortCommunity is forcing the global wantpeers variable to true. This is unnecessarily also forcing the collection of all libTorrent peer-statistics, and thus should be prevented.

We should replace the wantpeers with a list of hashes of which the peer-statistics should be collected from, the EffortCommunity has to be modified to allow for this.

Johan, do you want to write a new one? if not, Ill get the one from tribler.org.

Help from strangers, without privacy leakage
etc.

As reported by Ron in build 36

Traceback (most recent call last):
  File "Tribler\Core\APIImplementation\LaunchManyCore.pyo", line 714, in network_checkpoint_callback
  File "Tribler\Core\Libtorrent\LibtorrentDownloadImpl.pyo", line 638, in network_stop
  File "Tribler\Core\Libtorrent\LibtorrentDownloadImpl.pyo", line 793, in network_get_persistent_state
  File "Tribler\Core\Libtorrent\LibtorrentDownloadImpl.pyo", line 604, in network_get_state
  File "Tribler\Core\DownloadState.pyo", line 113, in __init__
IndexError: list index out of range

Ability to seed 1 TByte of content using Tribler.

This requires announcing, say, 1000 swarms of content in the DHT.
This is a problem, as shown here: http://blog.libtorrent.org/2012/01/seeding-a-million-torrents/
The announce interval needs to be prolonged in order to reduce DHT announce traffic.
Perhaps the DHT cannot handle this and a new peer discovery method is required: #13.

As a quick partial fix we can put a cap on the maximum swarms to DHT announce. Then use a simple round-robin method to cycle slowly through all available swarms.

Three primary tasks for the whole Tribler research team:

• fully circular and self-sufficient token economy
  Create a richer incentive then tit-for-tat within Bittorrent. Deploy the micro-economy around tokens which we have been perfecting since the first deployed version in September 2007. Scientific micro-economy publications, presented recently at the Delft Blockchain Launch event:
  
  • Mining. Earn tokens by donating bandwidth. Credit mining effortlessly converts idle bandwidth into tokens. The mining boosts the health of the network. Underseeded swarms and popular swarms are identified and helped out. Dead swarms and spam are correctly dealt with. Users need to select a channel with content to boost. This signal from the user prevents spam. In a later stage we will auto-boost swarms in general.
  • Earn credits by relaying Tor-like encrypted traffic.
  • Get tokens for hidden seeding and being an exit node
  • Market. Sell and buy on our integrated decentralised marketplace. Convert bandwidth tokens into Ethereum or Bitcoin. Your karma as a merchant grows with each successful trade. Part of our Internet-of-Trust research.
  • Coin wallet. Move coins between a Tribler PC and Android devices. Offline storage inside Android smartphones. Transfer securely and effortlessly using simple QR codes without the need for Internet access. Focus on the "you can see it, you can spend it" security model.
• Tribler anonymous downloads are fast and secure
  Donations of encrypted relays are essential for performance. We require the token economy to fix that. Currently our software is still a bottleneck. Only 2.5MByte/sec is unacceptable for our video streaming use-case. Currently our GCM cypto, tunnel latency or other bottleneck are slowing things down. #1882 #2548 Replace Dispersy tunnels with clean code of IPv8
  • Refactor the video-on-demand part of Tribler
  • Refactor the download core of Tribler. Libtorrent wrapper #3418
• search, share, rate and enhance
  Vloggers, journalists, and creatives in general need to be able to use Tribler to share. The content within Tribler needs to become richer. Metadata needs to be perfect. Spam marking and voting needs to work. Relevance ranking and swarm popularity needs to be shown in search. Rich channels. We need to empower our community to sustain itself.
  • spam prevention, voting and spam marking
  • accurate swarm size estimations
  • Fast search with relevance ranking based on swarm size
  • rich metadata for channels
  • redesign of the AllChannel voting and channel pre-view mess. IPv8

Revision: 30061

After seeding for a few days and 47GByte uploaded it seems the peer candidate is clogging up. Is this really a problem? Seems not to slowdown the seeding speed.

Report upstream? See screenshot:

Credit investments going live:

Ability to Donate 1TByte of disk space to the community and get top download priority.
Critically depends on ticket: #5 and to lesser extend #21.
An undersupply detection mechanism seeks out swarms which are low on seeders.
The investment function download parts of these swarms and engages in partial seeding to earn credit. Seeding is done anonymously through proxies, critical dependency with #22.

Test idea: Tribler automatically fills 100MByte of disk, can be disabled on front-screen.

First test version is limited to few thousand Libswift test swarms which are included in Tribler and seeded. Desired outcome of the test is evenly spread out partial seeding. If we do artificial repeated downloads from 1 swarm, we boost demand and peers should see this as an attractive investment.

Idea: strongest possible reputation system by combining all existing science.

Problem: our scientists Rahim, Adele, Nitin and Dimitra did not combine their work.
Reason: optimal publication strategy is one clean and simple idea
There are no joint publications, no re-usage of algorithms and little collaboration.

We evaluated in isolation multiple enhancements to make accounting of contributions stronger and incentive compatible:

• max-flow calculation
• delete old information to reduce storage cost
• depletion of often used paths, provides sybil resilience with additional bookkeeping
• introduce negative trust, these negative edges increase sybil attack resilience
• friendship as an edge property to strengthen the trust overlay against sybils
• age as an peer property to strengthen against sybil attack
• biased random walk, taking into account peer and edge properties
• double signed signature (proof-of-work)
• append-only signatures, misreporting attack degrades it to weaker white-washing attack
• running total account of consumed and contributed resources, degrades misreporting attack
• transfer encrypted information, only provide decode key after proof-of-work is signed
• incremental signatures, reduce risk-taking by demanding signatures during transfer
• use n-hop Max-Path to reduce compulational cost
• reason from betweenness centrality node
• use full gossip

Reputation system proposals often assume either:

1. Tamper-proof and scalable storage of information
2. Scalable storage of information

Option number 1 is beyond the state-of-the-art, without assuming an honest central server with infinite capacity. Option number 2 is difficult to achieve. For scalability seasons, the storage layer will always be imperfect and will never offer both instant, accurate and complete recall of information. Thus we need to be able to tolerate limited accuracy or coverage in addition to the usual misreporting, lying and sybil creation.

A reputations system needs to determine (assuming background gossip of trust vectors):
 1 Who to talk to
 2 how to reach them
 3 What to say
 4 How much to trust

Existing work is focused on no.4 mostly, while the others have proven to be
much harder to solve. Nobody identified them as a problems and no
solutions exist.
Trust calculation is just 50 lines of code, while no. 1-3 is beyond
the state-of-the-art in epidemic protocols.

Missing topics:
3. "what to say", Efficient synchronisation of BarterCast (subgraph) trust vectors

Existing material:
 1. Who to talk to
    Biased random walk
 2. how to reach them
    Churn resiliance, state preservation, live overlay, NAT traversal, etc
 4. How much to trust
   Meulpolder 2008: Max-flow
   Rahim: various sybil-limiters
   Adele: rewards also effort
   Dimitra: betweenness centrality
   Nitin: sybil-limiters

Aim: realistic workload tracing of a live distributed (self-organizing) system
Run on clusters: Leaseweb boxes, DAS3? and report to Jenkins
Dispersy Sync as a possible use-case

Depends on #9

File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 370, in init
super(Member, self).init(public_key, private_key)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 154, in init
assert DispersyDatabase.has_instance(), "DispersyDatabase has not yet been created"
AssertionError: DispersyDatabase has not yet been created
E dispersy/callback:679 _loop |
Traceback (most recent call last):
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/callback.py", line 677, in _loop
result = call[0](*call[1], **call[2])
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/endpoint.py", line 136, in dispersythread_data_came_in
timestamp)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 1786, in on_incoming_packets
self._on_batch_cache(meta, batch)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 1858, in _on_batch_cache
messages = list(self._convert_batch_into_messages(batch))
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 2061, in _convert_batch_into_messages
yield conversion.decode_message(candidate, packet)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1408, in decode_message
return self._decode_message(candidate, data, verify, False)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1349, in _decode_message
decode_functions.authentication(placeholder)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1170, in _decode_member_authentication
members = [member for member in self._community.dispersy.get_members_from_id(member_id) if member.has_identity(self._community)]
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 705, in get_members_from_id
if public_key]
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 370, in init
super(Member, self).init(public_key, private_key)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 154, in init
assert DispersyDatabase.has_instance(), "DispersyDatabase has not yet been created"
AssertionError: DispersyDatabase has not yet been created
E dispersy/callback:679 _loop |
Traceback (most recent call last):
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/callback.py", line 677, in _loop
result = call[0](*call[1], **call[2])
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/endpoint.py", line 136, in dispersythread_data_came_in
timestamp)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 1786, in on_incoming_packets
self._on_batch_cache(meta, batch)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 1858, in _on_batch_cache
messages = list(self._convert_batch_into_messages(batch))
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 2061, in _convert_batch_into_messages
yield conversion.decode_message(candidate, packet)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1408, in decode_message
return self._decode_message(candidate, data, verify, False)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1349, in _decode_message
decode_functions.authentication(placeholder)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1170, in _decode_member_authentication
members = [member for member in self._community.dispersy.get_members_from_id(member_id) if member.has_identity(self._community)]
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 705, in get_members_from_id
if public_key]
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 370, in init
super(Member, self).init(public_key, private_key)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 154, in init
assert DispersyDatabase.has_instance(), "DispersyDatabase has not yet been created"
AssertionError: DispersyDatabase has not yet been created
E dispersy/callback:679 _loop |
Traceback (most recent call last):
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/callback.py", line 677, in _loop
result = call[0](*call[1], **call[2])
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/endpoint.py", line 136, in dispersythread_data_came_in
timestamp)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 1786, in on_incoming_packets
self._on_batch_cache(meta, batch)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 1858, in _on_batch_cache
messages = list(self._convert_batch_into_messages(batch))
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 2061, in _convert_batch_into_messages
yield conversion.decode_message(candidate, packet)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1408, in decode_message
return self._decode_message(candidate, data, verify, False)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1349, in _decode_message
decode_functions.authentication(placeholder)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1170, in _decode_member_authentication
members = [member for member in self._community.dispersy.get_members_from_id(member_id) if member.has_identity(self._community)]
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 705, in get_members_from_id
if public_key]
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 370, in init
super(Member, self).init(public_key, private_key)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 154, in init
assert DispersyDatabase.has_instance(), "DispersyDatabase has not yet been created"
AssertionError: DispersyDatabase has not yet been created
E dispersy/callback:679 _loop |
Traceback (most recent call last):
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/callback.py", line 677, in _loop
result = call[0](*call[1], **call[2])
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/endpoint.py", line 136, in dispersythread_data_came_in
timestamp)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 1786, in on_incoming_packets
self._on_batch_cache(meta, batch)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 1858, in _on_batch_cache
messages = list(self._convert_batch_into_messages(batch))
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 2061, in _convert_batch_into_messages
yield conversion.decode_message(candidate, packet)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1408, in decode_message
return self._decode_message(candidate, data, verify, False)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1349, in _decode_message
decode_functions.authentication(placeholder)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1170, in _decode_member_authentication
members = [member for member in self._community.dispersy.get_members_from_id(member_id) if member.has_identity(self._community)]
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 705, in get_members_from_id
if public_key]
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 370, in init
super(Member, self).init(public_key, private_key)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 154, in init
assert DispersyDatabase.has_instance(), "DispersyDatabase has not yet been created"
AssertionError: DispersyDatabase has not yet been created
E dispersy/callback:679 _loop |
Traceback (most recent call last):
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/callback.py", line 677, in _loop
result = call[0](*call[1], **call[2])
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/endpoint.py", line 136, in dispersythread_data_came_in
timestamp)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 1786, in on_incoming_packets
self._on_batch_cache(meta, batch)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 1858, in _on_batch_cache
messages = list(self._convert_batch_into_messages(batch))
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 2061, in _convert_batch_into_messages
yield conversion.decode_message(candidate, packet)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1408, in decode_message
return self._decode_message(candidate, data, verify, False)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1349, in _decode_message
decode_functions.authentication(placeholder)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/conversion.py", line 1170, in _decode_member_authentication
members = [member for member in self._community.dispersy.get_members_from_id(member_id) if member.has_identity(self._community)]
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/dispersy.py", line 705, in get_members_from_id
if public_key]
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 370, in init
super(Member, self).init(public_key, private_key)
File "/media/data/feina/code/delft/github/tribler/Tribler/dispersy/member.py", line 154, in init
assert DispersyDatabase.has_instance(), "DispersyDatabase has not yet been created"
AssertionError: DispersyDatabase has not yet been created
main: ONEXIT deleting instances
Client shutting down. Sleeping for a few seconds to allow other threads to finish
CmdGwCloseConnection: ENTER 19
cmd: Shutting down on CMD connection close
./tribler.sh 248,92s user 72,33s system 34% cpu 15:40,90 total

Swarmplayer 3000 has live streaming.
We prefer to keep that.
Bittorrent-based browser plugin either needs to be saved or cleaned up.

more brain dump regarding live. Live streaming to a HTML5 video element was implemented in the BitTorrent backend, not the swift backend. Swift at the moment just supports live streaming of MPEG-TS which doesn't require tricks like Ogg and (I think) WebM require. For that you need VLC or a non-broken Android library for playback.

The trick for Ogg is to replay the Ogg header pages that contain the stream info before sending the current live frames. So when a client tunes into a live Ogg stream it needs to:

1. Obtain the Ogg header pages
2. Parse the incoming live P2P byte stream for the start of an Ogg page
3. Send header pages to the player, followed by the pages from the P2P stream.

See

https://svn.tribler.org/abc/branches/release-m48-swarmplayer/Tribler/Video/Ogg.py

In the BitTorrent case I stored the Ogg header pages in the .torrent file to get them to the clients. For swift there is no metadata file so another solution is needed. TODO.

CU,
Arno

Details are unknown to me. Had needed a fix in the past.
Still one bug remains. Unit test needed.

Aim: dynamic instrumentation and live analysis
Purpose: systematic methodology to identify the top performance bottleneck
Depends: #7, #8, #9 and #19

Performance is not limited to pure CPU usage.
Performance: CPU, memory, Internet bandwidth, file system IO, database queries.
GUI responsiveness is also key.
It is not only average performance, but specifically peaks, stalls and freezes.

Problem is finding the workload of Tribler, as we don't now what a client is doing in the wild due to complexity of our system. reproducibility of the wild

Determine cost of various modules in Tribler:

• Cost of crypto (both CPU, latency and IO)
• SQL tracing (cost, variance, volume, IO, CPU, latency)
• Cost of Dispersy Bloom filter creation
• network thread
• Python, C++ Libswift

SystemTap GUI: http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=%2Fliaai%2Fstapgui%2Fdemo.htm

http://sourceware.org/systemtap/SystemTap_Beginners_Guide/introduction.html#goals
SystemTap is a tracing and probing tool that allows users to study and monitor the activities of the computer system (particularly, the kernel) in fine detail. It provides information similar to the output of tools like netstat, ps, top, and iostat; however, SystemTap is designed to provide more filtering and analysis options for collected information.
SystemTap provides the infrastructure to monitor the running Linux kernel for detailed analysis. This can assist administrators and developers in identifying the underlying cause of a bug or performance problem.
Without SystemTap, monitoring the activity of a running kernel would require a tedious instrument, recompile, install, and reboot sequence. SystemTap is designed to eliminate this, allowing users to gather the same information by simply running user-written SystemTap scripts.
However, SystemTap was initially designed for users with intermediate to advanced knowledge of the kernel. This makes SystemTap less useful to administrators or developers with limited knowledge of and experience with the Linux kernel. Moreover, much of the existing SystemTap documentation is similarly aimed at knowledgeable and experienced users. This makes learning the tool similarly difficult.

Expand SystemTap with DAS4-like features, supports 1000 instances of a
distributed app (beyond multi-agent system stuff). Cluster-Systemtap

LibtorrentDownloadImpl: alert performance_alert with message livedvd-amd64-multilib-2012.1: performance warning: max outstanding disk writes reached
LibtorrentDownloadImpl: alert performance_alert with message debian-6.0.5-i386-DVD-1.iso: performance warning: max outstanding disk writes reached
LibtorrentDownloadImpl: alert performance_alert with message livedvd-x86-amd64-32ul-2012.1: performance warning: max outstanding disk writes reached
LibtorrentDownloadImpl: alert performance_alert with message livedvd-x86-amd64-32ul-2012.1: performance warning: max outstanding disk writes reached
LibtorrentDownloadImpl: alert performance_alert with message livedvd-amd64-multilib-2012.1: performance warning: max outstanding disk writes reached
LibtorrentDownloadImpl: alert performance_alert with message livedvd-x86-amd64-32ul-2012.1: performance warning: max outstanding disk writes reached
LibtorrentDownloadImpl: alert performance_alert with message livedvd-x86-amd64-32ul-2012.1: performance warning: max outstanding disk writes reached
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
Traceback (most recent call last):
File "/media/data/feina/code/delft/github/tribler/Tribler/Main/Utility/GuiDBHandler.py", line 123, in wrapper
result = workerFn(_args, *_kwargs)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/RemoteTorrentHandler.py", line 272, in do_db
self.torrent_db.addExternalTorrent(tdef, extra_info = {'filename': swiftpath, 'swift_torrent_hash':sdef.get_roothash(), 'status':'good'})
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/SqliteCacheDBHandler.py", line 609, in addExternalTorrent
self._addTorrentToDB(torrentdef, source, extra_info, commit)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/SqliteCacheDBHandler.py", line 777, in _addTorrentToDB
self._db.update('Torrent', where=where, commit=False, *_database_dict)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/sqlitecachedb.py", line 724, in update
self.execute_write(sql, arg, commit)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/sqlitecachedb.py", line 2473, in execute_write
self._execute(sql, args)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/sqlitecachedb.py", line 2384, in invoke_func
return func(_args, **kwargs)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/sqlitecachedb.py", line 2551, in _execute
raise msg
ConstraintError: ConstraintError: column swift_torrent_hash is not unique

Traceback (most recent call last):
File "/media/data/feina/code/delft/github/tribler/Tribler/Main/Utility/GuiDBHandler.py", line 123, in wrapper
result = workerFn(_args, *_kwargs)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/RemoteTorrentHandler.py", line 272, in do_db
self.torrent_db.addExternalTorrent(tdef, extra_info = {'filename': swiftpath, 'swift_torrent_hash':sdef.get_roothash(), 'status':'good'})
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/SqliteCacheDBHandler.py", line 609, in addExternalTorrent
self._addTorrentToDB(torrentdef, source, extra_info, commit)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/SqliteCacheDBHandler.py", line 777, in _addTorrentToDB
self._db.update('Torrent', where=where, commit=False, *_database_dict)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/sqlitecachedb.py", line 724, in update
self.execute_write(sql, arg, commit)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/sqlitecachedb.py", line 2473, in execute_write
self._execute(sql, args)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/sqlitecachedb.py", line 2384, in invoke_func
return func(_args, **kwargs)
File "/media/data/feina/code/delft/github/tribler/Tribler/Core/CacheDB/sqlitecachedb.py", line 2551, in _execute
raise msg
ConstraintError: ConstraintError: column swift_torrent_hash is not unique

cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
SQLiteNoCacheDB.commitNow: BEGIN
Erased 25 torrents

Bug seen on Revision: 30061

The seeding statistics are incorrect in the library mouse-over notifications.
In the future this bartering information is important and we want do do flawless byte counting: #20.
After seeding a swarm for 47GByte and pausing this single swarm the stats go wrong. Both upload and download seem wrong.

Resuming the seeding seems to restore the stats correctly:

Goal: Test-only of Libswift byte counting.
Aim: prepare for larger goal of "rewarding good behavior" #5.
Methodology: idea is to use agile 2-weeks releases with new features and test them in the wild, after DAS4 success.

Statistics from Libswift are used on both GUI and
the BarterCast3 community (the Tribler trust overlay).

Current situation:
sesscb_states_callback is called every second and retrieves the peer information of all swarms from libswift when wantpeers is True. Wantpeers is set to True when an BarterCast3 community exists or a detailspanel is selected in the librarylist.

Desired situation:
Only every 60 seconds BarterCast3 sets wantpeers to True and retreives the peer information of all swarms. Every 1 second, the GUI gets the peer information from one specific swarm if a detailspanel is selected.
New event notifier: libswift connection closed + last counters for that connection

Future wish list:
-Public key inclusion, therefore no missing-identity messages problem. Alters the walker, only for this community.
-Simple churn test
-Edge traversal and teleport
-Top250 edges

I'm getting this message repeated all the time in current release-6.1 branch.

...
cannot open mbinmap for writing: No such file or directory
cannot open mbinmap for writing: No such file or directory
cannot open mbinmap for writing: No such file or directory
cannot open mbinmap for writing: No such file or directory
cannot open mbinmap for writing: No such file or directory
...

and also

...
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
cannot create/open hash file: No such file or directory
...

We going into a channel, the from channel "column" is showing.

Improve access to the exact released source code.

Over the past 7 years numerous releases have been made. Currently only old tags in SVN are available which point to the Python source code of old releases, for instance:
http://svn.tribler.org/abc/tags/tribler-3.3.1/ (RC1 from Feb 2006)
http://svn.tribler.org/abc/tags/tribler-5.0.0/