hive2hive / hive2hive Goto Github PK

Java library for secure, distributed, P2P-based file synchronization and sharing.

License: MIT License

Java 100.00%

synchronization file-sharing distributed security

hive2hive's Introduction

Hive2Hive

Hive2Hive is an open-source library, written in Java, for secure, distributed, P2P-based file synchronization and sharing. It is built on top of TomP2P, an advanced, high-performance DHT for multi-key-value pairs. The Hive2Hive project is licensed under the MIT License and any contribution is welcome.

Problems of common sync and sharing services

Although many well-known synchronization and sharing services exist, most of them exhibit the following drawbacks:

centralized client-server approaches, data resides in large, external data centers
- single-point-of-failure, vulnerable to targeted attacks
- often not scalable
private data is not encrypted
user control is revoked, no control over who has access to the private data
user is bound to the respective pricing and terms of service
no version control, no conflict management

Hive2Hive is the solution!

The Hive2Hive library addresses these issues by providing a free and open-sourced, distributed and scalable solution that focuses on maximum security and privacy of both users and data. Aside of this, it supports the whole feature set known from similar centralized approaches, such as Dropbox, OneDrive or Google Drive, and adds functionality for file versioning and conflict management. All packed in a clean, simple API.

There are many simple ways to improve this experience even more. Start to contribute now!

Check our GitHub Wiki to learn more about How To Use and How It Works.

Are you looking for a demo application?

PeerWasp, a Windows-based shell extension à la Dropbox.
H2H Console Client, an executable .jar console application.
Eclipse RCP Client, an RCP-based client.

API Demonstration
Features & Advantages
Installation
Documentation
Contribution
Contact

API Demonstration

The Hive2Hive library provides a simple API that is straightforward to use. (View Source) A short demonstration of the API and its basic usage are given here.

Network Management

Creating a P2P Network

Configuring and setting up a new P2P network is very easy. Just specify the configurations and setup an initial node.

The NetworkConfiguration and FileConfiguration factory classes may help to specify your configurations.
Create an initial peer node and connect it.

INetworkConfiguration netConfig = NetworkConfiguration.create("first");
IFileConfiguration fileConfig = FileConfiguration.createDefault();

IH2HNode peerNode = H2HNode.createNode(netConfig, fileConfig);
peerNode.connect();

Joining an Existing P2P Network

You may want to add other peer nodes to the created network. Any node can join by bootstrapping to another node that is already part of the network.

Specify the network configuration for the joining node (i.e., provide the bootstrap address of another node).
Create the new node and connect it. It will bootstrap according to its network configuration.

INetworkConfiguration netConfig2 = NetworkConfiguration.create("second", InetAddress.getByName("192.168.1.100"));
IH2HNode peerNode2 = H2HNode.createNode(netConfig2, fileConfig);
peerNode2.connect();

User Management

Once a peer node is connected to a network, users can interact with it. For this, each node provides a user management interface.

The user has to provide her credentials.
Login the user to the network. If it's the first login, she has to register herself, one-time.
Then, the user can interact with the network (i.e., file management is enabled).

IUserManager userManager = peerNode.getUserManager();

UserCredentials credentials = new UserCredentials("userId", "password", "pin");
Path rootDirectory = Paths.get("sample/path/to/rootDirectory");

if (!userManager.isRegistered(credentials.getUserId())) {
	userManager.register(credentials).await();
}
userManager.login(credentials, rootDirectory).await();

File Management

As soon as a user is logged in to the network, her files are automatically synchronized with the current node. Many further file operations are available. For this, each node provides a file management interface.

add / delete file
update / recover file
share file with another user
move file

IFileManager fileManager = peerNode.getFileManager();

File folder = new File("folderpath");
File file = new File(folder, "filepath");

fileManager.add(file);

fileManager.update(file);

fileManager.share(folder, "other-userId", PermissionType.WRITE);

IVersionSelector versionSelector = new IVersionSelector() {
	@Override
	public IFileVersion selectVersion(List<IFileVersion> availableVersions) {
		return availableVersions.get(0);
	}
};
fileManager.recover(file, versionSelector);

fileManager.move(folder, new File("other-folder"));

fileManager.delete(file);

File Watchdog

In order to keep track of changes in the local file system, a file observer is needed. This observer notifies its attached listeners about all file system events. Either use the provided H2HFileObserver and H2HFileObserverListener or implement your own observer adhering to the IFileObserver and IFileObserverListener interfaces.
The H2HFileObserverListener automatically synchronizes the Hive2Hive root folder with the network.

IFileObserverListener listener = new H2HFileObserverListener(fileManager);

IFileObserver observer = new H2HFileObserver(rootDirectory.toFile());
observer.addFileObserverListener(listener);
observer.start();

Features & Advantages

Hive2Hive offers the same basic functionality known from popular synchronization services (e.g., Dropbox).
On top of that, Hive2Hive provides additional features such as security and versioning.

File Synchronization
File Sharing and Access Permissions (read/write and read-only)
File Versioning and Conflict Management
File Watchdog / Change Detection (automated, configurable)
Security (configurable)
- Encryption of files and messages
- Authentication of data and messages
Users can use multiple clients (simulatenously)
Multiple users can use the same machine (simultaneously)

Using the Hive2Hive library is very simple and has several advantages:

P2P Decentralization
- Scalability
- Heterogeneity
- Reliability & Fault-Tolerance
no file size limits (configurable)
platform independent (JVM)
headless deployment possible^* (e.g., on a Raspberry Pi)
free & open-source
generously configurable & customizable
highly extendable
detailed documentation

^* Try our console-based org.hive2hive.client by just executing the library .jar.

And there is even more to come:

A demonstrative GUI client is waiting in the wings (see more)
REST API (see more)
Secure Bootstrapping (see more)
LAN Synchronization (see more)
Large File Alternatives (see more)
Same Network - Different Applications (see more)
and more

Installation

There are three easy ways to get and include the Hive2Hive library into your project.

If you just want to use the library, either refer to option 1 or 2.
If you want to contribute to the project, please refer to option 3.

Option 1: Add Maven dependency (recommended)
You can add the latest stable release as an Apache Maven dependency. Add the following to your pom.xml and make sure to select the most recent version.

<dependency>
  <groupId>org.hive2hive</groupId>
  <artifactId>org.hive2hive.core</artifactId>
  <version>1.X.X</version>
</dependency>

Option 2: Add JAR-file directly
In case you don't want to use Maven, you can just download the latest stable release that comes directly with all necessary sources. All required .jar-files are packed and delivered to you as a .zip.
Option 3: Clone from GitHub
If you want to contribute to the Hive2Hive library project, this is what you should do. Cloning from GitHub gets the bleeding edge of development and thus some sources might not be stable. So this option is not recommended if you just want to use the library.

Documentation

For more details and documentation of the Hive2Hive project, please visit the GitHub Wiki.
The source code itself is thoroughly documented using JavaDoc.

Contribution

The library is intended to be improved and extended so that we all profit from its capabilities. Unlike many other “secure” services, Hive2Hive discloses its implementation and is open for any sort of contribution and constructive criticism.

We believe that everyone can contribute to make Hive2Hive even better!

There are several things - from simple to complex - you can do to help:

Star and watch the project here on GitHub
Spread the word and help us getting attention (e.g., follow/tweet about @Hive2Hive)
Suggest and post your ideas about improvements or extensions on the issues page
Participate in discussions, share your expertise
Help us with the documentation: JavaDoc, GitHub Wiki, etc.
Help us with the implementation of (your) features
Fork the project and send your pull requests
Help the community by answering questions on StackOverflow (tagged with hive2hive)

Also, if you are a professional cryptographer with interest in this project, any feedback on the project is very welcome!

hive2hive's People

Contributors

Stargazers

Watchers

Forkers

shafiahmed graspeo kamyu104 dcale mebigfatguy benkoerber joielechong jjoergensen chenlonggang gillbates donsunsoft wolvesled albu89 bfrye91 jiangts mrkebubun prodigeni pierrecarre hamidaaa tempbottle patanric hifi-git igunawardana haiyang1987 shangdawei chrisidefix johnny0917 bbrangeo anarry mohanarunachalam wethinkall les-mockingjay deepk2u gavinljj yusongying mlufei deceivedagain mkane007 gtostock agackovka rihenperry aliasgarhada heokjin jiago rootyue teotikalki duyouhua liangcongqq cynthion zhoupan muxi166 salahdemni casals mmmika imbugs lmglg maniacs-satm bingzhang j-plus-plus 137x3w sts0mrg0 confidentmachine gravetii sura2k xinpan0214 alvin-reyes tikhomirov83 athleticas piercarloslavazza wenjun001 jiawenxiong androidqasim lordlee0702 alipro renatoyuzo polarbearzoo aktjerry leyuzhu touchrank-dev nothsa dustw austinsmom sagarbalwani mga1984 rivy tringl xiuxi mario-vbg clove506 sniperxiaojun zhj149 sriyanfernando the3null4player613310 dentou touandcim sandeepbeniwal funny2014 reference-project simpleman1984 will2github

hive2hive's Issues

Caching

Use caching where possible to improve the speed:

Getting Public keys from other users (see #19)
...

A easy-to-use library for caching is https://code.google.com/p/kitty-cache/

Cross-Platform signature

When signing with OS X and verifying the signature with Windows, the signature does not match (however, it should). Vice versa does not work either. Is there something wrong with usage of the bouncycastle's signature function?
We should fix that prior the 1.0.0 release since signatures are essentials for the security.

Nofitications

To enable real-time updates at logged in clients, file changes (deletions, additions, modifications) need to be sent to online clients directly. For this, we need a framework to easily broadcast the news.
In best case, clients should all be notified in parallel. However, it is not so bad if a message does not arrive (because of a network problem). A retransmission of it is optional since it's not critical.

File Observer

The H2H library should provide a sort of observer that serves notifications of changes in the file system. The following requirements shall be met:

notifications for file add, remove, modification
notifications should be OS independent
it should be possible to subscribe to notifications independently (Observer Pattern)

Cryptree Paper

It could be interesting to adapt the idea of the Wuala concept "Cryptree" which helps with the key management of shared folders.
(http://dcg.ethz.ch/publications/srds06.pdf)

Logout

The logout routine should do the following things:

stop all running processes
remove the client's entry from the locations map
persist the meta data

In case of an unfriendly leave, no master update is done here. This is done implicitly, see #15.

Refactoring: Processes

The process framework is good for modularization, however gets very complex for large chains. What can be improved here?

Digest from a H2HNode

According our meeting on 20.12. we would like to extend our H2HNode interface with an information method, which shows which files and folders of the logged in user currently are stored in the network.

File Synchronization - 1. Add

New data has been added on the current client node (H2H was offline) or on another (online or offline) client node. This has to be detected and trigger a synchronization.

No need to hold children in MetaFolder

We always update the parent meta folder if a new file is added in its directory. But maintaining a list of children there is not needed as far as I know. It is just an unnecessary overhead.
When removing this, we can simplify many processes:

add
remove
move
recover

BTW: We do not have a root meta folder.

Separate UPTasks and Notifications

The notification frameworks currently only supports private messages. For diverse operations on shared files users need to notify other users.
This is how we thought it:

Client sends notification to all his own clients
Client creates a UserProfileTask and adds it to the other user's queue in the DHT
Client notifies master of the other client that he should check his queue

Remove MetaFolder

Since we changed the concept slightly during the development, the MetaFolder is not used at the moment. When adding a folder, the MetaFolder is pushed to the DHT. If the MetaFolder stays unused, we could skip this step.

Refactor the H2HNode

The H2HNode is the first point of interaction with the user/developer. The interface should be well-organized, easy to use but still configurable and extendable.
At the moment, the H2HNode implements an IH2HNode interface, which is responsible for UserManagement and FileManagement. The H2HNode should be redesigned and split such that an object-oriented structure arises.

As a next step, the FileManager, holding the root of the current session and delivering some file search / retrieve functions could be improved: The root folder of the session should be stored in the H2HSession while the FileManager's methods become static and are moved to the FileUtil. This change may lead to many changes, especially in our test-cases, but I think it's worth it.

Delete old File-Versions

It is possible to configure the limit of maximum number of versions. When uploading a new version, the number of existing versions should be checked and old versions should be deleted.

UserProfile Manager

By running multiple processes concurrently (file upload / modification / deletion), the user profile is fetched multiple times. This leads to massive conflicts when putting it. To overcome this problem, we suggest to have a central instance that gets and puts the user profile.

Register - 1. ProcessManager

An initial framework for processes. A process can run in parallel and consists of several process steps which can be rolled back. The process manager keeps track of all running processes.

Goal: first process which checks if user id exists.

Content protection

Read-Only of content in the DHT. We use signatures to verify the owner and to ensure that only the owner can modify / delete his content in the network.

Open interfaces for put/get/remove/send

We once discussed that (parallel to the H2H functionality), we probably should keep a generic Put / Get functionality. The user can store anything to the DHT; This could support the developer when creating a specific app. Support for write protection and encryption would be a nice feature.

Client - Dummy Console Client

To test and verify the whole library and its APIs, a dummy console client shall be implemented. With this, use cases like register, login, add/remove file etc. should become testable.

Move file

User should be able to move a file from a private folder into another private folder. There are some operations on the DHT necessary in order that other clients know that the file has been moved. Until the operation is implemented completely, the user could delete and re-add the file at the new directory.

Join for IProcess

According to our metting on 20.12 we proposed a join method in our IProcess interface. This method should allow the users the create dependencies between the triggered processes (one waits for another).

Public key at 1st message receiption

When a user sends a message to another user for the first time, its public key is not in the cache yet and must be fetched from the DHT. This causes an unintended timeout.
We need to accept the message provisorily and then execute it after a real check of the key.

Register - 2. EncryptionUtil

A encryption util class which provides methods for generating keys, encrypting and decrypting.

Goal: encrypting user profile

Key Strength Management

The usage of different key sizes and encryption strengths should be consistently handled. A centralized place of definition and the usage of specific constants needs to be checked and validated.

Unshare

Unsharing should ensure that the other user does not have access to the files anymore. The other user(s) could keep the keys to locate and decrypt the files, thus a copy of the data must be re-encrypted with a new key pair and uploaded to the DHT.

Login - 3. AddLocation

Add entry into location map. But don't put the map now. See #8.

Login - 2. DecryptUserProfile

A single process step where the user profile gets decrypted.

File recovery

Since we store multiple versions of a file in the DHT, the user should be able to recover old versions. Thus, we should provide an operation to download a specific version, not only the newest one.

Share a folder (with all its contents) with another user. All subfolders should be shared as well (inherit the rights).

Defining Master Client

We have several Use Cases where the definition of the master client has to be done: Login, Logout and Client Node Detected As Offline (UC 7.6).
To my knowledge, we did not come up with one final solution but rather implemented this as part of the login process. We further discussed about taking the client with the lowest PeerAddress as master.
However, I propose to extract 2 separate processes to ensure consistency among all use cases:

a process that defines the master client
a process that periodically checks UC 7.6 and might start the process above

Further, each client should have a local flag about whether it is master or not. In my opinion, this flag should reside in the H2HStorageMemory class. Other processes, such as the handling of user messages, should respect this property and implement functionality accordingly. (i.e. also work if 2 or more masters are defined - by mistake - for a short duration)

Does somebody have any objections?

Register - 4. VersionConflictHandling

On a put conflicts should be detected. On every put the old version is taken as reference. E.g. Node A and B has version 4. Node A and B modifies the version. Both want to upload their version. Node B was faster (first serve) and stores version 5. Now node A wants to update, but sees that there is already a newer version. A conflict has been detected.

Goal: uploading user profile and location map without conflicts

File Synchronization - 4. Download

According to the other issues (#11, #12, #13) we need a downloading process. This is used when a client gets notified about a new file / version or after he logs in and detects the new version himself.

Login - 1. GetUserProfileStep

A single process step gets the encrypted user profile from the network. This involves the implementation of a GetProcessStep class, which provides all functionality and failure handling. The current solution with the methods in the ProcessStep class has to be removed.

Login - 4. OnlineCheck

Check online users of this client and define master. It could be possible that after an unfriendly leave some out-dated entries are stored in the location map.

This issue involves a first implementation of a message step, which includes also the message failure handling.

File Synchronization - File Observer

In order to synchronize the files (i.e. using the file synchronization APIs), a file observer running on the OS should be implemented. This observer is able to react to file changes, such as add/delete/modify, and invoke the respective APIs of the synchronization (see #11, #12, #13, #14).

Refactor FileTreeNode

The class 'FileTreeNode' is pretty ugly. We should consider following points:

FileTreeNode could be split into a class for files and a class for folders (with common parent).
The protection keys should be inherited
Find a good way to indicate that a folder / file is shared (probably using a flag or so).

H2HNode refactoring

According to our meeting of 20.12 we proposed the idea to split the H2HNode interface in two interfaces, where one is responisble for the setup and the other for the file operations. This would allow the user a more intuitiv useage.

Login - 5. HandleUserMessageQueue

If master, handle user message queue. This will involve a initial stub implementation, because there are currently no user message in the project. However a smart approach has to be proposed to handle a user message queue.

Bug in decrypting (keylength is wrong)

Sometimes I observe an error in the GetMetaDocumentStep when the fetched data should be decrypted. A stack will be added here, as soon as it happens again. Is this error because of a defective rollback?

List all files

For a better overview for the developer, we should provide methods to retrieve a list of all files of the user in the DHT and the file states on disk.

User Profile Tasks [former UserMessages]

So far User Messages are all of type BaseMessage. However, due to the recent changes they are no more sent by means of the MessageManager, but rather put/get globally. Following tasks emerge:

define UserMessage hierarchy/interfaces, might end up to be of type NetworkContent
sign/verify UserMessage

Threading: Synchronize Mutable Data

During the Process Framework refactoring I sometimes encountered strange and random behaviour and test fails that are impossible to reproduce. However, I figured out that the problems only occur in case of multithreaded access to mutable data in some classes, like the DataManager or NetworkManager.
Such behaviour (non-reproducibility) is typical for multithreaded applications when the accesses are not synchronized!

Configuration documentation

Our library allows several configurations (size of chunks, number of versions etc.). We have to documnet that on our webpage, so that future users have an ortientation.

File Synchronization - 2. Modify

Some data has been modified (on the local client node while H2H was offline or on another client node). This new version has to be synchronized with the other client nodes.

File Synchronization - 3. Remove

Some data has been locally or remote removed. The online going client node has to remove this data locally or respectively in the network.

Register - 3. UserProfile/LocationMap

A user profile can be created which all contains keys (also domain key). The user profile can be encrypted by the user's password.
A location map which is public, read only and contains peer address and role (user messages).

Goal: creating initial user profile and location map

Message Encryption

All messages sent over the network shall be asymmetrically encrypted. The message is encrypted with the receiver's public key. The receiver decrypts the message with its private key.

IProcess States Documentation

We implemented a process framework wih several states. We have to document them on our webpage. But this issue can wait till the process framework refactoring has been done, which certainly will change some states.

Extend & update website

The website (hive2hive.com) only covers basic topics yet. We should extend it because it's probably the most important way to 'communicate' with users and developers. After a discussion, we came up with following tasks:

~~Redo the 'Overview' section --> @Biocodr~~
~~'How-to-use' must be updated to the newest version (after #47 ) --> @Biocodr~~
~~Advanced Usage: 'Model' should show basic idea --> @nicoruti~~
~~Advanced Usage: 'Security aspects' describe all security relevant decisions and principles (Encryption, Content protection, Versioning, Concurrent Modification, Login, ...) --> @ippes~~
~~Advanced usage: 'Implemented Functionality' (currently 'sequence diagrams') should briefly show how the functions (login, add, ...) are implemented. --> @nicoruti~~
~~Advanced Usage: 'How-to-extend' shows how to use the code and extend the existing interfaces with custom functionality --> @Biocodr~~
~~We should merge the 'about', 'contact' and 'roadmap' page to a single menu because they are less important~~

Further we need to decide what to show at the landing page. I like the way our sliders at the front page look, but they are less useful for a quick overlook. Maybe we can redo these sliders too or even remove them.

Cleanup test framework

Actually, this is a bit messy. Use clear structure for our tests. Mockito could help (https://code.google.com/p/mockito/).