Ubuntu Linux build process fails with `./p2p/source/asio.hpp:24:15: fatal error: 'asio.hpp' file not found` about orchid HOT 15 CLOSED

@saurik the build process still isn't working, even with rustup installed.

from orchid.

I am going to guess that you forgot to use --recursive when initializing git submodules; this is probably being reported in git status. Regardless: we know the build is working now, as it is being built continually by GitHub CI. (On the off chance I'm wrong, I'll leave this bug open for a bit. To prove me wrong, show me that you have the file orchid/p2p/boost/libs/asio/include/boost/asio.hpp and are still getting this error.)

from orchid.

Good guess. I ran git submodule update --init --recursive and then make -C srv-shared.

Now fails with a different fatal error: 'boost/detail/indirect_traits.hpp' file not found

chrishobcroft@bountiful:~/orchid$ make -C srv-shared
make: Entering directory '/home/chrishobcroft/orchid/srv-shared'
[CC] lnx/x86_64 p2p/boost/libs/filesystem/src/directory.cpp
In file included from p2p/boost/libs/filesystem/src/directory.cpp:28:
In file included from ./p2p/boost/libs/filesystem/include/boost/filesystem/directory.hpp:26:
In file included from ./p2p/boost/libs/filesystem/include/boost/filesystem/path.hpp:29:
In file included from ./p2p/boost/libs/iterator/include/boost/iterator/iterator_facade.hpp:15:
./p2p/boost/libs/iterator/include/boost/iterator/detail/facade_iterator_category.hpp:28:12: fatal error: 'boost/detail/indirect_traits.hpp' file not found
#  include <boost/detail/indirect_traits.hpp>
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [env/output.mk:158: out-lnx/x86_64/./p2p/boost/libs/filesystem/src/directory.cpp.o] Error 1
make: Leaving directory '/home/chrishobcroft/orchid/srv-shared'

from orchid.

Hi

@chrishobcroft I had to personalize one .hpp file for it to build. It is about « » being experimental in gcc (even 8 strangely).

Try to look at .hpp file concerned and make it look like something like this :

#include <experimental/filesystem>

I am on a phone and don’t have the file under my eyes. You just have to add «experimental/»

@saurik how can I be a bandwidth provider and run a node when orchidd is built ?

from orchid.

@zaniko We do not use gcc to build orchid; the current assumption is that you are using libc++-9.

from orchid.

@chrishobcroft Can you 1) tell me if p2p/boost/libs/detail/include/boost/detail/indirect_traits.hpp exists; 2) give me the complete output of git status; and 3) remove the @ from $(job)@$(prefix) in the $(output)/%.cpp.o rule of env/output.mk and give me the clang command that is printed when you run make?

from orchid.

OK, I am currently working off the process documented in this branch at this commit.

It reads:

sudo apt-get update
sudo apt-get install \
     bison \
     clang \
     flex \
     gettext \
     git \
     gperf \
     groff \
     libc++-9 \
     lld \
     make \
     ninja-build \
     python3-pip \
     python3-setuptools \
     tcl
sudo pip3 install meson==0.51.2
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
git clone https://github.com/OrchidTechnologies/orchid.git
cd orchid/
git submodule update --init --recursive
make -C srv-shared

And now I'm getting:

chrishobcroft@bountiful:~/orchid$ make -C srv-shared
make: Entering directory '/home/chrishobcroft/orchid/srv-shared'
make: *** No rule to make target 'out-lnx/x86_64/orchidd', needed by 'all'. Stop.
make: Leaving directory '/home/chrishobcroft/orchid/srv-shared'

from orchid.

Specifically in response to your questions @saurik

1. tell me if p2p/boost/libs/detail/include/boost/detail/indirect_traits.hpp exists;

It does.

2. give me the complete output of git status; and

chrishobcroft@bountiful:~/orchid$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
	modified:   min-webrtc/libsrtp (modified content)
	modified:   p2p/SPCDNS (modified content)
	modified:   p2p/challenge-bypass-ristretto (modified content)
	modified:   p2p/cppcoro (modified content)
	modified:   p2p/eEVM (modified content)
	modified:   p2p/ethash (modified content)
	modified:   p2p/expected (modified content)
	modified:   p2p/libutp (modified content)
	modified:   vpn-shared/libmaxminddb (modified content)
	modified:   vpn-shared/zlib (modified content)
	modified:   vpn-windows/tap-windows6 (modified content)

no changes added to commit (use "git add" and/or "git commit -a")

3. remove the @ from $(job)@$(prefix) in the $(output)/%.cpp.o rule of env/output.mk and give me the clang command that is printed when you run make?

Done - same error as above, seems like a broader issue. I reverted the change.

chrishobcroft@bountiful:~/orchid$ make -C srv-shared
make: Entering directory '/home/chrishobcroft/orchid/srv-shared'
make: *** No rule to make target 'out-lnx/x86_64/orchidd', needed by 'all'. Stop.
make: Leaving directory '/home/chrishobcroft/orchid/srv-shared'

from orchid.

Ok, so that output from git status is showing--very clearly--that something is wrong with the code for a number of the projects that you have checked out. As this is definitely not a bug in Orchid, I am going to close this issue, but am happy to continue (occasionally) debugging your usage of git on this thread.

FWIW, my guess is that the code is missing--that folders such as p2p/libutp are empty--and that you failed to complete the sub module checkout. Can you 1) go into p2p/libutp and run git status to see if that is the case; 2) try a git submodule update --init --recursive to ensure it doesn't print any errors; and 3) contemplate deleting your working copy and starting over?

(I have heard good things about the book Pro Git, in case you would like to learn more about using git.)

from orchid.

OK, thanks @saurik I have started reading Pro Git. And thank you for being happy to continue (occasionally) to help debug my rookie usage of git.

My main objective it to get this build process working on Linux for myself and others, so we can all help build Orchid together, and start understanding what exactly it can do for us.

• I deployed a clean Ubuntu 18.04 hosted server, and ran the following:

sudo apt-get update
sudo apt-get install \
     bison \
     clang \
     flex \
     gettext \
     git \
     gperf \
     groff \
     libc++-9-dev \
     lld \
     make \
     ninja-build \
     python3-pip \
     python3-setuptools \
     tcl
sudo pip3 install meson==0.51.2
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
git clone https://github.com/OrchidTechnologies/orchid.git
cd orchid/
git submodule update --init --recursive
make -C srv-shared

• This failed with:

p2p/./openvpn3/client/ovpncli.cpp:24:10: fatal error: 'iostream' file not found
#include <iostream>
         ^~~~~~~~~~
1 error generated.
env/output.mk:155: recipe for target 'out-lnx/x86_64/./p2p/./openvpn3/client/ovpncli.cpp.o' failed
make: *** [out-lnx/x86_64/./p2p/./openvpn3/client/ovpncli.cpp.o] Error 1
make: Leaving directory '/home/ubuntu/orchid/srv-shared'

• Here's the git status:

ubuntu@VM-b4268bfc-3112-4eb8-a2dc-57f0e6701b59:~/orchid$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

• I ran git submodule update --init --recursive again, and no errors were reported.

• I went into p2p/libutp and ran git status with the following output:

HEAD detached at 2b364cb
nothing to commit, working tree clean

• I noticed that in all places in the install log, the string iostream appears within iostreams (with an "s"), so I changed the line in p2p/./openvpn3/client/ovpncli.cpp to iostreams (with an "s"), ran make -C srv-shared, to get a similar error:

p2p/./openvpn3/client/ovpncli.cpp:24:10: fatal error: 'iostreams' file not found
#include <iostreams>
         ^~~~~~~~~~~

• FWIW, p2p/boost/libs/detail/include/boost/detail/indirect_traits.hpp and orchid/p2p/boost/libs/asio/include/boost/asio.hpp also exist.

• Here is full session log

from orchid.

OK, I'm sorry: I just got mentally overloaded and was also actively working to fix a lot of stuff in the build to make it easier and I honestly feel bad every week or so that I kept choosing to not read this.

So iostream (no s) is part of the C++ standard library and is supposed to be coming from libc++ (which you did install). I don't understand how it could not find iostream given that that is just like, something that is effectively coming with the language :(. I do manually set -stdlib=libc++, but like, this isn't really even an issue with Orchid's code but like C++ compiler setup :(.

Can you just make a file (maybe test.cpp) containing nothing but #include <iostream> and then run clang++ -stdlib=libc++ -c -o test.o test.cpp to see if the compiler is even remotely set up correctly?

One thing you might consider is running env/setup-lnx.sh, which should attempt to install everything that I feel reasonable to just install for people (which does not include Rust). If you run env/setup-dkr.sh instead it will install everything I deploy on a totally blank docker container to build Orchid (and at that point I've even installed the Android NDK, and so you can do the "make debug=crossndk" build, which is much better as it is a reproducible build).

I dunno, man :(. This at least isn't a git issue this time, but it feels like some kind of weird C++ compiler issue.

from orchid.

@saurik you appear to have made some delicious progress on a client app for Linux, bravo!

I especially liked the commit comment:

I pretty much just reprogrammed the whole project. 

I was wondering whether perhaps someone might add a README for the app-linux repo, so that the community can understand how to use this thing, and perhaps how to build it ourselves.

Running ./orchidd-lnx_0.9.21 --help does provide a few pointers, but I'm stuck around The WebSocket handshake was declined by the remote peer [p2p/source/duplex-open.cpp:37] opening wss://goerli.infura.io:443/ws/v3/ef918************************eca2 (infura credentials have been obscured).

Also... I'm still guessing at exactly what this thing is supposed to do...

And for reference, I have two use cases in mind right now:

1. Serve livestream video content over a vpn requiring viewers pay per Byte using Orchid PMs
2. Some sort of way to run a shard of eth2 on an Orchid VPN, where peers need to pay each other per Byte using Orchid PMs

from orchid.

The orchidd --ws didn't work with wss:// URLs. FWIW, it also required a personal geth instance for various security reasons. More recent builds don't have that parameter at all. The app-linux folder is for the front-end GUI for the client application (which isn't even slightly ready to use yet on Linux: it compiles into something you can run, but there are no released binaries for it because it has no way to talk to the backend, so it is entirely pointless to use). End-user accessible documentation will be provided at such time as there are end-user accessible reasons to run this software.

As for your use cases: I personally don't think either is an appropriate usage of orchidd, as neither problem maps to "pay by the byte". Further, such servers could not be included in our node registry, because they don't provide network access. I feel like what you really want to do is just use our nanopayments mechanism to implement your video licensing (which should be pretty easy if you read the code for our lottery contract: take something like Wowza and then have the payment keys managed by the frontend paying for access by the quarter-second using nanopaments).

from orchid.

As for your use cases: I personally don't think either is an appropriate usage of orchidd, as neither problem maps to "pay by the byte".

I trust your judgment, you are far more familiar than me (or indeed anyone, probably) about what would and would not be "appropriate usage" for orchidd. One area we disagree though, is on "pay by the byte" vs. "pay by the (quarter)-second". I am actually looking for "pay by the byte" for reasons I explain below.

Further, such servers could not be included in our node registry, because they don't provide network access.

That's fine, I'm not interested in running something as part of Orchid's registry yet... not until there are interesting staking and revenue-earning opportunities which are permissionless and open to the public to participate in.

I feel like what you really want to do is just use our nanopayments mechanism to implement your video licensing (which should be pretty easy if you read the code for our lottery contract: take something like Wowza and then have the payment keys managed by the frontend paying for access by the quarter-second using nanopaments).

Yes, this is always an option, however a few refinements, as I alluded to earlier.

First, I'm already building on Livepeer's open-source video infrastructure services software, which has a public network specialising in Transcoding i.e. converting from e.g. 2160p to e.g. 1080p,720p,360p,256p,144p. This makes the content more accessible to those with weaker internet connections and less-powerful devices. Obviously a 144p stream requires far less bandwidth than a 2160p stream, thus metering "by the byte" would appear to help broadcasters serve a wider audience and charge a fair price based on the video resolution of the content they're consuming from a server.

Further, I discussed the concept of a "server-side paywall" with your colleague Jennifer, whereby two independent pieces of existing software (Orchid and Livepeer) may potentially at some point be able to run alongside each other, with a consumer paying the Orchid server in OXT per byte, and that Orchid server effectively proxies the hls video stream from the Livepeer media server (serving multiple transcodings) - a kind of Frankenstein's monster-like thing building on the shoulders of two Web 3.0 stalwarts:

And when I saw the --help options for orchidd, including flags for configuring something to do with "bandwidth pricing", I got excited that something like this might be possible with a bit of configuring of orchidd or orchidcd to get some dirty prototypes up and running, without needing to pen a line of code.

bandwidth pricing:
  --currency arg (=USD)                 currency used for price conversions
  --price arg (=0.03)                   price of bandwidth in currency / GB

I'm also interested to play around with the code for your android app, to see if I can reuse some of the code in there, to allow a "video livestream consumer" app to connect to a one-node VPN, where it's paying the Orchid server for serving it data, which the Orchid server effectively proxies from the Livepeer media server. And when it comes to mobile, it definitely does become a lot more about "per byte" than "per second" and bandwidth tends to be more of a constraint. I would pursue this further once I can get some kind of "charge in nanopayments per byte of livestream content" dirty prototype back end plugged together.

As a nirvana step, I'm interested in how to consolidate Livepeer's and Orchid's protocols at a smart contract level - i.e. Livepeer's PM mechanism (denominated in ETH) for paying for Transcoding services from the public network, and Livepeer's PoS rewards mechanisms (in LPT), and Orchid's OXT-based per-byte nanopayments system, with Uniswap in the mix (LPT / ETH / OXT swaps), to make a more coherent platform running through Ethereum for Livestream Media Rights owners to use, to balance their operating costs (transcoding - ETH), and their revenues (OXT for instant nano-royalties, LPT staking revenue) in order to provide a reliable service, and also to pay the creators.

So, what do orchidd and orchidcd actually do? I couldn't find anything in the readme. And what do you think?

from orchid.

@chrishobcroft So, while you haven't convinced me that "by the byte" is appropriate (as the CPU usage of these various bytes is not that comparable), I'm going to be more blunt about this: orchidcd and orchidd form a VPN server client and server solution, designed to tunnel IP packets. On the client end, you thereby either need to have a full simulated TCP stack, or you need to have a system-wide privileged tunnel device doing packet capture. Your problem simply has nothing to do with VPNs. What you should only want -- and this will be much much much less work for you -- is what you list as "a nirvana step": to just go directly to nanopayments. I can't speak to our roadmap, and so keeping only to what we have already developed (and also looking for the easiest possible solution, as I'd prefer you to not waste too much time), you should just make each HLS segment require a nanopayment (as their approximate size for this purpose is "just about perfect"), and (this makes it really easy, and you can do somethig more complex later) embed the cost information in your HLS chunk list as an extension. This solution will even work in a browser (assuming it has MSE; I can explain solutions that would work for an actual native HLS implementation, but it would behoove you to simply wait), as it doesn't require the user to install any extra software; meanwhile, it requires very very little modification to your client or your server. I thereby maintain that should really just be looking at our contract and technique for nanopayments, not our packet tunneling solution ;P.

from orchid.