Some versions of tar are not playing well with the current build script. Hence, when everything is tar'd up it includes the "dist" basedir which we don't want.

Placeholder for general code quality and best practices.

Further to Issue #10, we should improve the verbosity of our command line application to let user know that the FTE client/server has successfully started and is listening on the specified port.

When starting up as part of daemon I get this error:

Traceback (most recent call last):
  File "/usr/local/bin/fteproxy", line 30, in 
    import fte.conf
  File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 29, in 
    import fte.client.managed
ImportError: No module named managed
Traceback (most recent call last):
  File "/usr/local/bin/fteproxy", line 30, in 
    import fte.conf
  File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 29, in 
    import fte.client.managed
ImportError: No module named managed

The following error can be produced by attempting to bootstrap Tor over FTE.

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/fte/relay.py", line 56, in run
    [success, _data] = fte.io.recvall_from_socket(self._socket2)
  File "/usr/local/lib/python2.7/dist-packages/fte/io.py", line 48, in recvall_from_socket
    _data = sock.recv(bufsize)
  File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 78, in recv
    frag = self._decoder.pop()
  File "/usr/local/lib/python2.7/dist-packages/fte/record_layer.py", line 92, in pop
    except fte.encoder.DecodeFailureException:
AttributeError: 'module' object has no attribute 'DecodeFailureException'

The issue is intermittent and only occurs roughly 1 out of 5 times, or so.

We don't need the full GPLv3 as a header for each file. We should include a single LICENSE file, then have the standard GPLv3 short boilerplate.

The current TBB integration is rough: we have a hard-coded IP:port to our FTE server in Wisconsin.

We need to play nicer with the TBB and make it such that users can configure their Tor+FTE bridge via the Vidalia GUI.

Use the Coverity source code analysis service to detect any issues with our C++ source code.

We should add a command-line switch to enable FTE to run in the background. At the moment we run FTE in the background by using "&" on Linux. As a byproduct, we don't output much information unless the "--debug" switch is added.

We should:
(1) Add a "-d" switch for FTE to start as a daemon.
(2) When "-d" is not specified we should be a bit more chatty, so the user knows what is going on.

In the current release the FTE server binds to 127.0.0.1 on port 8080 and we forward all data to port 8081. This is probably not correct for most deployment. As a few examples of changes we want to make, we could make the following assumptions:

(1) FTE will bind to port 80.
(2) There will be a Tor bridge on the local machine running on port 9001, and we will forward all requests to this port.

However, should the FTE server reject a request to port 80 using a non-HTTP format? Or should the server just allow any supported format on any port?

The following AssertionError was encountered in normal operation of fteproxy:

Traceback (most recent call last):
 File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
   self.run()
 File "/usr/local/lib/python2.7/dist-packages/fte/relay.py", line 50, in run
   [success, _data] = fte.io.recvall_from_socket(self._socket1)
 File "/usr/local/lib/python2.7/dist-packages/fte/io.py", line 48, in recvall_from_socket
   _data = sock.recv(bufsize)
 File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 78, in recv
   frag = self._decoder.pop()
 File "/usr/local/lib/python2.7/dist-packages/fte/record_layer.py", line 87, in pop
   incoming_msg = self._decoder.decode(self._buffer)
 File "/usr/local/lib/python2.7/dist-packages/fte/encoder.py", line 119, in decode
   assert len(covertext) >= self._max_len, (len(covertext), self._max_len)
AssertionError: (145, 512)

At the moment, FTE format names are determined by their filename in the formats directory. This name is the only token used when performing regex negotiation. Hence, if a client has a regex name that matches the server, but the actual regex doesn't match, then the negotiation fails and the client will get unexpected results.

We should make it such that the naming scheme used for format negotiation includes a unique identifier, such as a hash of the regex.

Currently, we use long-term keys for encryption. This may be sufficient if we delegate all responsibility of security goals, such as Perfect Forward Secrecy, to the tunneled protocol. However, if there is something lightweight (e.g., a variant of DH) that could be implemented, then we might want to consider how it would improve the current FTE system.

Right now we simply output "Files .no_fte.txt and .yes_fte.txt are identical."

The "systemtests" script should be an opportunity for us to ensure that environmental variables are correctly set, and that fteproxy is installed properly.

When attempting to compile openfst-1.3.3 under Ubuntu 13.10, the following error occurs.

...
/bin/bash ../../libtool --tag=CXX   --mode=link g++  -g -O2   -o fstarcsort fstarcsort.o ../script/libfstscript.la ../lib/libfst.la -lm -ldl 
libtool: link: g++ -g -O2 -o .libs/fstarcsort fstarcsort.o  ../script/.libs/libfstscript.so ../lib/.libs/libfst.so -lm -ldl
../script/.libs/libfstscript.so: undefined reference to `dlopen'
../script/.libs/libfstscript.so: undefined reference to `dlerror'

configure command:

./configure --enable-fast-install --disable-dependency-tracking --prefix=/usr/local

GCC: 4.8.1

The command

fteproxy --stop

defaults as

fteproxy --mode client --stop

So, when

fteproxy --stop

is executed on an fteproxy server, the fteproxy server is not terminated and the command fails silently.

When installing FTE on the Tor-Cloud AMI instance within Amazon Web Services (AWS), the unit and system tests fail with the following error:

----------------------------------------------------------------------
ERROR: testLTB (fte.tests.bit_ops.TestEncoders)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/fte/tests/bit_ops.py", line 43, in testLTB
    self.assertEquals(fte.bit_ops.long_to_bytes(0xff), '\xFF')
  File "/usr/local/lib/python2.7/dist-packages/fte/bit_ops.py", line 39, in long_to_bytes
    bytestring = binascii.unhexlify(bytestring)
TypeError: Non-hexadecimal digit found

----------------------------------------------------------------------

Python version is 2.7.3
GCC 4.6.3
Ubuntu 12.04

Proxy appears to crash when client is negotiating with it. Could be build issue or incompatibility with client version. Could even be an unintended response to port scanning on HTTP ports (8080 in this case).

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/fte/relay.py", line 112, in run
    conn = self.onNewIncomingConnection(conn)
  File "/usr/local/lib/python2.7/dist-packages/fte/server.py", line 30, in onNewIncomingConnection
    socket = fte.wrap_socket(socket)
  File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 307, in wrap_socket
    K1, K2)
  File "/usr/local/lib/python2.7/dist-packages/fte/__init__.py", line 130, in __init__
    [negotiate_cell,remaining_buffer] = self._accept_negotiation(data)
TypeError: 'bool' object is not iterable

"make all" on Ubuntu 12.04 64-bit results in the following error:

/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.6/crtbeginT.o: relocation R_X86_64_32 against `__DTOR_END__' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/4.6/crtbeginT.o: could not read symbols: Bad value
collect2: ld returned 1 exit status
error: command 'g++' failed with exit status 1

GCC: 4.6

At the moment the core FTE code (rank/unrank/buildTable), our AE scheme (encrypt/decrypt) and FTE proxy (management of connections, etc.) are all jumbled into one program.

(1) We should separate the FTE code into a library that can be used as a C++ library, or as a Python module. This should be quite easy to do and is mostly a reorganization of directory structures.
(2) We may want to separate out our AE scheme in the same way, too.

Then that will leave use with our FTE proxying code that uses the AE scheme and FTE library to encrypt and encode messages. The main advantage of doing this is to increase transparency and modularity. This refactor should also be done in a way that allows us to swap in/out or disable/enable encryption/encoding for testing purposes.

We should have a good understanding of Tor's existing PyObfsproxy code and keep that in the back of our minds before we attempt this. It would be ideal if we could organize FTE code such that it works well with PyObfsproxy, esp. w.r.t. receiver-side parsing.

On a default OSX install, Gatekeeper prevents a user from running our Tor Browser Bundle.

http://support.apple.com/kb/ht5290

We either have to (1) jump through the hoops and pay the money to sign our releases or (2) explain to users how to get around this problem, probably by linking to the above Apple KB article.

For now, solution (2) should be fine, I assume.

-Kevin

Currently, fte_relay can only be run from its parent directory, because it assumes the location of the *.dfa files. This should not be the case.

Currently we use TCP internally for IPC, for control processes to communicate with worker processes, which is bad. We should investigate more refined methods (e.g. Python's multiprocessing module) for worker threads to receive/process incoming data streams.

Since our primary use case for FTE is as a type of pluggable transport for Tor, it makes sense to directly (and automatically) create the Tor Browser Bundles during the build process.

In the recent major refactor I removed the ability for client to dictate (in-band) the regex it wants to use. This is a regression and should be resolved.

The file

FTE/src/fte/cRegex.cc

should be documented, cleaned up and analyzed by a static analysis tool such as Coverity.

Instead of creating our own managed-proxy interface for Tor. Enable an (optional?) integration with pyptlib:

https://gitweb.torproject.org/pluggable-transports/pyptlib.git

We currently use boost in "FTE/src/fte/cRegex.cc" for a small number of features. This dependency is far too heavyweight and should be removed.

All Python code from this point should match the pep8 guidelines.

At the moment our build process is a hybrid between a Makefile and a Python setup.py file. What's more, the build step in the FTE directory depends upon file in "third-party" located in FTE's parent directory. This is bad form. We should organize the project such that either "./configure && make && make install" or "python setup.py install" works from the root directory on all Linux-based systems.

The current build.py is hard-coded, works for Ubuntu and has hacks for OSX.

We should transition to setup.py to facilitate cross-platform builds.

At the moment an FTE server can listen on only one port at a time. Hence, to listen on, say, ports 22 and 80, this requires two FTE instances to be running server-side, which requires twice the memory. However, if we setup FTE to take the list of server-side ports as a list, rather than an integer, this would all the FTE server to be much more clever and listen on multiple ports in the ServerDemuxer, without the need to have multiple FTE instances.

The build script should determine the number of cores on the build system and not naively pass "-j8" to make.

Figure out how we are going to integrate into Tor's BridgeDB system.

https://bridges.torproject.org/bridges

This is closely related to issues #18, #17, #14, #12, and #1, which all relate to how users acquire and use the FTE software in a typical setting.

gcc/g++ support code hardening flags such as -fstack-protector-all, -D_FORTIFY_SOURCE, -fPIE and some more. They should be added to the compiler flags to provide defence in depth.

We don't quite have the dependencies right for the Makefile. So "make -jN" for N>1 results in a build failure in some cases.

The current code release does not include any method to compile custom regular expressions into the necessary DFAs for use in the FTE framework. This issue focuses on adding an easy-to-use workflow for compiling new regular expressions. Should probably include a nice way to manage the regular expressions and add them to the help dialog.

At the moment the FTE client listens for incoming FTE connections on port 8079 and forwards them to 8080 on 127.0.0.1. The client uses intersection-http by default.

We should determine if our current intersection-http format is the right choice for a default format. We should also determine the correct default client-side port to listen for incoming Tor connections.

The build script should produce (1) FTE binaries and (2) Tor bundles that include the FTE binaries.

Determine automatic strategy for removing outgoing_max_len and incoming_max_len paramters from fte.wrap_socket interface.

Currently systemtests depends upon being able to download an external website. These tests should be completely self contained.

At the moment all FTE configuration parameters must be specified on the command line, which is already getting cumbersome. If we add new features, such as Issue#14, which enables the server admin to specify more than one server-side port, it would be ideal to specify this in a Tor-like configuration file.

The number of worker processes is currently not configurable and defaults to the number of cores on the system. In many cases it is desirable to restrict the number of worker processes to one, to ensure FTE doesn't consume too much CPU. Hence, the number of worker threads should be a command line configuration parameter.

When executing "fteproxy --stop" the following error occurs:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/local/bin/fteproxy", line 110, in run
    if fte.conf.getValue('runtime.console.debug'):
  File "/usr/local/lib/python2.7/dist-packages/fte/conf.py", line 24, in getValue
    return conf[key]
KeyError: 'runtime.console.debug'

We currently specify "#!/usr/bin/python" at the top of scripts. We should instead use "#! /usr/bin/env python"

Currently, the code has a single hard-coded encryption key that is used for the authenticated encryption scheme. In the future, we want to be able to allow the user to specify and use their own long-term keys.

Placeholder to investigate support for FTE on Windows.

At the moment, the FTE client can support only one FTE server IP address. In many cases the server may be down or inaccessible. Hence, it should be the case that the client can support a list of FTE servers and attempt to connect to them in order.

We should fully integrate in the TBB build process and not retrieve the TBB ourselves.

Learned formats don't take much disk space, but, given the current buildTable algorithm, they take orders of magnitude more memory than other formats. We should either (1) delete these formats or (2) keep them around, but disable them by default server-side. (This could be achieved by adding a server-side "--experimental" switch that loads learned formats only when the switch is specified.)