Glad to hear this !!! It's seems we will make new paradigm of data migration :-)

1. My expiriense with FUSE via repertory already allowed me to create ext4 on fuse and sucsesfully mount - problems was only with updates, full file system started to upload again to the blockchain. I used ISCSI targed for posibillity to create block device with data source of target on fuse - here you can find all details here https://bitbucket.org/blockstorage/repertory/issues/86/cant-disable-cache
   If you can make BUSE work with user - it's very nice also and it increases the chances of success.

2. Script for posibillity to convert existing contracts it will be nice also - I have a lot of contracts for my experiments with SIA. Posibillity to return contracts into siad will be also greate for users who will in the process of investigation SIA and will try to use blockchain by diffeent ways. But this functionality not major on this moment.

3. Our priority to find a way for posibility to make ext4 on SIA and try to improve perfomance and manage the redunancy of data in the blockchain. Perfomance like you understanding will deppend from hosts analises and methods of contracts usage. Redunancy it's seems will be more easy task. Performance will deppend from cache and internet speed. Local cache must be managed like in repertory, but only to be "chunk based cache" instead "file based cache". We have a lot of copies our data becuase we need this for our security, but in the same way we can increase the download/upload speed. For example our 1 file stored on 10 or 100 hosts and we like advanced professional user have a greate internet channel. Each host a slow internet connection, but if we will upload data by small parts on several hosts in one time - we can increase our speed. For example 100 hosts, redunancy 10x - in this case we can increase speed near the 10 times if will shard our file on 10 parts and upload it togeather. Other methods to increase data speed can be used also in the same time - it's like for RAID 10 - best performance and best repairs. On the future may be will have sense to implement algoritms like RAID TEC/RAID 5 - this algoritms have a less perfomance but will decrase the costs of data storage.

Tomorrow I will try to make ext4 on fuse via "user" and let you now about current results.

from user.

user can only be used with contracts that you formed with user. It can't use your siad contracts, unless you manually convert them. (And if you do, you won't be able to use them with siad anymore.) If you want to convert existing contracts, let me know and I will write a script for you.

Alternatively, you can form new contracts with user form and renew them with user renew. These commands will spend coins from your walrus wallet. You can choose which hosts to use by looking at siac hostdb -v or https://siastats.info/hosts.

If you have a walrus_addr and shard_addr in your config.toml, siad will not be used at all.

By the way, both shard and walrus store their own full copy of the blockchain. If you don't want to store two copies, you can use shard.lukechampine.com as your shard server.

I haven't tried storing a database with FUSE, so let me know if it works! If you truly want to "format your Sia hosts as ext4," you may need to use something like BUSE instead of FUSE. I would have to write a bit of code to make BUSE work with user, but I don't think it would be too difficult.

from user.

can't upload data to the mounted FUSE

./user mount /mnt/user_meta /mnt/user
Mounted!

cp /opt/install.sh /mnt/user/install.sh

root@ubuntu-SIA-1:/opt#

But I don't to see a copied file
Error only via Midnight Commander in process of copy file to the mounted DIR I have the next error:
Cannot chown target file "/mnt/user/install.sh" No such file or directory (2) Skip/Skip all/Retry/Abort

10 contracts was created manually before

/opt/install.sh - in source location have saize 0

Files which have a size have can be copied vi MC but with other errors
like

Cannot chown target file /mnt/user/test.dat
Function not implemented (38) Skip/Skip all/Retry/Abort
and
Cannot chown target file /mnt/user/test.dat
Not such file or directory (2) Skip/Skip all/Retry/Abort

But file will created.

mount | grep /mnt/user

pathfs.pathInode on /mnt/user type fuse.pathfs.pathInode (rw,nosuid,nodev,relatime,user_id=0,group_id=0)

df#
df: /tmp/.mount_repertODQGmM: Transport endpoint is not connected
df: /mnt/user: Function not implemented
and I can't see size of disk in this folder
and
Midnight Commander don't show any size of disk in folder also

from user.

Cannot create target file "/mnt/user/consensus.db"
Input Output Error (5)
Skip/Skip all/Retry/Abort

And file was deleted

But with next try start of action I can sussesfull copy the file

If will try to interrup copy file then have posibillities to keep part copy of file writed, but in case of this error process interrupting fully and uploaded data losted

from user.

big files ner 2gigs i can't uploading becuse have cyclik error input/output errors

and errors in terminal with user command

./user mount /user_meta /user

/bin/fusermount: failed to access mountpoint /user: No such file or directory
could not mount: fusermount exited with code 256
root@ubuntu-16gb-fsn1-SIA-1:/opt/Sia# ./user mount /mnt/user_meta /mnt/user
Mounted!
Truncate wagerr-3.0.1-x86_64-linux-gnu.tar.gz: could not upload to some hosts:
de8a5bca: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector
12221826: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector
d9cd1249: Write: couldn't write RPC ID: WriteRequest: write tcp 78.47.85.129:38240->72.66.79.45:9982: i/o timeout
Write consensus.db: could not upload to some hosts:
de8a5bca: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector
12221826: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector
Write consensus.db: could not upload to some hosts:
de8a5bca: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector
12221826: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector
7962d480: Write: couldn't write RPC ID: WriteRequest: write tcp 78.47.85.129:46446->50.197.241.165:9982: i/o timeout
d9cd1249: Write: couldn't write RPC ID: WriteRequest: write tcp 78.47.85.129:42358->72.66.79.45:9982: i/o timeout
Truncate consensus.db: could not upload to some hosts:
de8a5bca: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector
12221826: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector
Write consensus.db: could not upload to some hosts:
de8a5bca: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector
12221826: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector
d9cd1249: Write: couldn't write RPC ID: WriteRequest: write tcp 78.47.85.129:43888->72.66.79.45:9982: i/o timeout
Write consensus.db: could not upload to some hosts:
de8a5bca: Write: couldn't read signature response: ReadResponse: not enough storage remaining to accept sector

from user.

/mnt/user/consensus.db size 9 502 620 B

r# mkfs.ext4 /mnt/user/consensus.db
mke2fs 1.44.1 (24-Mar-2018)
Creating filesystem with 9280 1k blocks and 2320 inodes
Filesystem UUID: bdc743e4-123d-4a09-a1d9-05bae39cbd27
Superblock backups stored on blocks:
8193

Allocating group tables: done
mkfs.ext4: Input/output error while zeroing block 9216 at end of filesystem
Writing inode tables: done
Creating journal (1024 blocks): mkfs.ext4: Input/output error
while trying to create journal
root@ubuntu-16gb-fsn1-SIA-1:/mnt/user#

from user.

finaly after remount in /mnt/user_meta not one file was stored and in /mnt/user all data was a lost

from user.

additinaly after second remount Midnight commander show /mnt/user like /mnt/~user warited by rad.

:~# ls /mnt
ls: cannot access '/mnt/sia_virtual': Input/output error
ls: cannot access '/mnt/user': No such file or directory
HC_Volume_3608062 sdb sdc SIA_FS sia_storage sia_virtual user user_meta

df

df: /tmp/.mount_repertODQGmM: Transport endpoint is not connected
df: /mnt/user: Function not implemented
Filesystem 1K-blocks Used Available Use% Mounted on
udev 8001152 0 8001152 0% /dev
tmpfs 1603840 1228 1602612 1% /run
/dev/sda1 157482440 107666612 43371896 72% /
tmpfs 8019180 23176 7996004 1% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 8019180 0 8019180 0% /sys/fs/cgroup
/dev/loop0 90752 90752 0 100% /snap/go/4668
/dev/loop1 90752 90752 0 100% /snap/go/4762
/dev/loop2 91264 91264 0 100% /snap/core/8039
/dev/loop3 91264 91264 0 100% /snap/core/7917
/dev/sdb 30832548 6137820 23105480 21% /mnt/sdb
tmpfs 1603836 0 1603836 0% /run/user/119
tmpfs 1603836 0 1603836 0% /run/user/0
repertory 4742490224 1748544324 2993945900 37% /mnt/sia_virtual

from user.

Dear Lucke,
please let to know, how many time required from your side for posibillity to fix issues with FUSY becuase it not working fully like with siad like with walrus & shard server. If you think BUSY will be better for our needs let try to implement BUSY. Second impotant moment will be bandwidth performance - but only after we have BUSY/FUSY ... I have also much ideas, but think all other ideas must be considering after we have FUSY/BUSY, nice network perfomance & automations processes by contracts.

Best Regards.

from user.

regarding fuse.
current issue it's that Midnight Commander say errors - but files which have size can be copied, but errors present also. After I trying unmount storage by CTRL+C in the terminal with user mount ... I have mistakes in process unmountig. I have 10 contracts created manually via siad then switched to walrus & shard, transfered some money to the walrus and tring again - same mistakes.

a# ./user mount /mnt/user_meta /mnt/user
Mounted!
^CUnmounting...
Error during close: could not upload to some hosts:
86b2eb34: Write: couldn't read Merkle proof response: ReadResponse: communication error: renter is requesting revision after the revision deadline
de8a5bca: Write: couldn't read Merkle proof response: ReadResponse: communication error: renter is requesting revision after the revision deadline
b7b3f35f: Write: couldn't read Merkle proof response: ReadResponse: communication error: renter is requesting revision after the revision deadline
43cd88ca: Write: couldn't read Merkle proof response: ReadResponse: communication error: renter is requesting revision after the revision deadline
12221826: Write: couldn't read Merkle proof response: ReadResponse: communication error: renter is requesting revision after the revision deadline
17878417: Write: couldn't read Merkle proof response: ReadResponse: communication error: renter is requesting revision after the revision deadline
d9cd1249: Write: couldn't read Merkle proof response: ReadResponse: communication error: renter is requesting revision after the revision deadline
7962d480: Write: couldn't read Merkle proof response: ReadResponse: communication error: renter is requesting revision after the revision deadline
55086fd6: Write: couldn't read Merkle proof response: ReadResponse: communication error: renter is requesting revision after the revision deadline
80123282: Write: couldn't read Merkle proof response: ReadResponse: communication error: renter is requesting revision after the revision deadline

from user.