amccurry / pack Goto Github PK

Create test external error generation framework to test error handling on the iscsi and wal services.

Actively repair blocks from WAL after volume has been shutdown so that WAL entries can be released.

Add create volume form in admin application.

Check volume not in use before detachment of volume.

Blocks in external store that are no longer referenced in either the current blocks or in snapshots should be deleted.

The current block file implementation requires more reading during block file open than is required. If the block file indexes starting positions were calculated and stored in the meta data the logic required to open the block file would simplified and should be faster.

Volumes should be allowed to be put into a readonly mode or created as readonly from a snapshot of another volume.

This command will allow thin provisioning of data in the remote storage by allowing fstrim to be used on the mounted OS.

Create new volumes from snapshots.

Add multiple host directory support for block cache and WAL storage.

The compactor should be able to operate without the need to be restarted from time to time. This could have been an issue with where the compactors were running at the time of the issue. The servers were running out of memory from time to time. So this could be a non-issue.

This needed for writing blocks into hdfs. If clients are using multipathing then one and only one server should deliver the wal data to hdfs.

The idea is to prime the write ahead log with data before enabling the new log for normal writes. This might allow for more seamless WAL roles.

Deal with failed local storage for both WAL and block cache stores.

Investigate compressing blocks in remote store.

Create CSD and Parcel for Cloudera Manager deployment.

Needs a cli for managing volumes and getting status from pack.

Cache removal listener is now a blocking call. Some aspects of the call should be made async.

This should help clean up old disk space in the block storage system.

The current WAL implementation should be made into a replicated service to avoid single points of failure.

Snapshots of volumes should be allowed, both online and offline. Online snapshots may produce unexpected results given that the FS may be changing. So for online snapshots FS freeze and sync should be employed.

Read blocks from external store when a sequential read is detected.

Currently pack copies all the files on the server's classpath before it start a mount. That way upgrades won't disrupt current running mounts. We can improve this by having the server maintain the copies and when all the files are not in use by a running mount the old libraries can be GCed.

Local block cache and state should be removed on volume detachment.

Store the current cache state (which blocks are currently cached) for the given volume so that on reopen the state can be restored. This should allow similar performance after a volume is moved to a new machine.

Investigate using TOKEN authentication for performance improvement.