Git Product home page Git Product logo

random-access-file's Introduction

random-access-file

Continuous reading or writing to a file using random offsets and lengths

npm install random-access-file

Why?

If you are receiving a file in multiple pieces in a distributed system it can be useful to write these pieces to disk one by one in various places throughout the file without having to open and close a file descriptor all the time.

random-access-file allows you to do just this.

Usage

const RandomAccessFile = require('random-access-file')

const file = new RandomAccessFile('my-file.txt')

file.write(10, Buffer.from('hello'), function(err) {
  // write a buffer to offset 10
  file.read(10, 5, function(err, buffer) {
    console.log(buffer) // read 5 bytes from offset 10
    file.close(function() {
      console.log('file is closed')
    })
  })
})

file will use an open file descriptor. When you are done with the file you should call file.close().

API

const file = new RandomAccessFile(filename, [options])

Create a new file. Options include:

{
  truncate: false, // truncate the file before reading / writing
  size: someSize, // truncate the file to this size first
  readable: true, // should the file be opened as readable?
  writable: true,  // should the file be opened as writable?
  lock: false, // lock the file
  sparse: false // mark the file as sparse
}

file.write(offset, buffer, [callback])

Write a buffer at a specific offset.

file.read(offset, length, callback)

Read a buffer at a specific offset. Callback is called with the buffer read.

file.del(offset, length, callback)

Delete a portion of the file. Any partial file blocks in the deleted portion are zeroed and, if the file is sparse, the remaining file blocks unlinked in-place.

file.truncate(offset, callback)

Truncate the file length to this offset.

file.stat(callback)

Stat the storage. Should return an object with useful information about the underlying storage, including:

{
  size: number // how many bytes of data is stored?
}

file.close([callback])

Close the underlying file descriptor.

file.unlink([callback])

Unlink the underlying file.

file.on('open')

Emitted when the file descriptor has been opened. You can access the fd using file.fd. You do not need to wait for this event before doing any reads/writes.

file.on('close')

Emitted when the file has been closed.

License

MIT

random-access-file's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

random-access-file's Issues

Error: Offset is out of bounds

Hello.
I use webtorrent in node, os windows 8.1 and got error:

fs.js:620 Uncaught Error: Offset is out of bounds
    fs.read @ fs.js:620
    onread  @ C:\Projects\launcher\src\main_app\node_modules\random-access-file\index.js:84
    wrapper @ fs.js:614

I can wrap fs.read in try/catch and it solves the problem:

try {
  fs.read(self.fd, buf, buf.length - offset, length, offset, onread)
}
catch (e) {
  onread(e)
}

How better to solve this problem?

Support for promises

Hey! I'm curious as to why none of the random access modules return promises? I love this project and would like to use existing modules and write my own but while experimenting I keep finding myself on top of a pyramid of doom :-)

List files

It would be helpful if there was an API call to list all of the available files.

Missing NTFS/ReFS sparse support

NTFS requires files to be created with the FILE_ATTRIBUTE_SPARSE_FILE attribute to treat them as sparse. This is different from other modern OSes where you don’t need to do anything special.

I tested with fsutil.exe sparse queryflag supposedly_sparse.file and various calls to check file sizes, and found that the files aren’t marked as sparse files on NTFS.

Read after close gives this

Trace
    at Class.destroy (/Users/findutnyy/Code/indutny/hyperfeed/node_modules/from2/index.js:76:15)
    at check (/Users/findutnyy/Code/indutny/hyperfeed/node_modules/from2/index.js:57:28)
    at /Users/findutnyy/Code/indutny/hyperfeed/node_modules/hypercore/index.js:974:21
    at onread (/Users/findutnyy/Code/indutny/hyperfeed/node_modules/random-access-file/index.js:101:14)
    at FSReqWrap.wrapper [as oncomplete] (fs.js:628:17)
events.js:163
      throw er; // Unhandled 'error' event

propose project merge with my align-block-file

we are trying to do the same thing here, and if we merge our api then yak shaving (like getting it fast in indexeddb) will benefit all our projects.

so, this would mean adding methods from each other's apis, and writing a test suite (and benchmark suite!) that can be reused.

so the union of
https://github.com/flumedb/aligned-block-file
plus
https://github.com/mafintosh/random-access-file

methods already shared: read except that RAF uses start, length and ABF uses start, end

then RAF has the following methods

  • write (ABF currently only has append)
  • del (ABF more accurately calls this truncate)
  • end (curious about how this is used, because not every RAF implementation has a place for this data?)
  • close
  • unlink
  • events "open" "close" ABF uses observables. I'd be content to switch to observables or support both.

ABF has

  • readUInt{32,48,64}BE since ABF internally has an aligned buffer thing, these methods can actually be made really fast, rather than reading a 4-8 byte Buffer and then converting it to a number then throwing it away. It doesn't allocate buffer memory, but creating a buffer object will add more memory to the heap than it contains!
  • append (would be easy to add this to RAF)
  • size returns current size... should probably remove this method?
  • offset observable.
  • truncate could be an alias for RAF.del

The main difficulty here would is we need to settle on either start,end or start,length and that is a breaking change for someone... we could choose based on which has less code to fix or which pattern is used more consistently in the node apis (it seems that both are present, but i'm not clear on the rationale for when to use a particular one)

Fully don't work with ElectronAPI.

image

When try read file as: console.log(file.read(551846, 4, 'err')) it caused error.
Please, fix it, library good, but i don't can it use because it not work. Thanks for attention.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.