brianmario / bzip2-ruby Goto Github PK

Please see this build of steam-condenser for more details.

At least compatibility with ruby-head should be fixed.

Consider the test dataset from issue #22:

bunzip2 the test dataset
bzip2 the test dataset with Bzip2::Write in test1.bz2
bzip2 the test dataset with command line bzip2 in test2.bz2

Notice the huge size difference between test1.bz2 and test2.bz2
Notice that bzcat test1.bz2 | md5sum differs from bzcat test2.bz2 | md5sum

Versions:
Linux orsted 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux
bzip2, a block-sorting file compressor. Version 1.0.6, 6-Sept-2010.
bzip2-ruby (0.2.7)

... so people can finally "enjoy" bzip2 with Ruby 1.9.x :-)

I am experiencing nasty premature termination of Bzip2::Reader. I am including a test script and a data file located here:

ftp://ftp_20130926_12853:[email protected]

Notice the difference in output:

./test.rb | wc
13206 16508 952823

bzcat bzip_test.bz2 | wc
14604484 18255605 994881701

Versions:

Linux orsted 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux
ruby 1.9.2p180 (2011-02-18 revision 30909) [x86_64-linux]
bzip2-ruby (0.2.7)

Hi Brian,

Thanks for the great work with this. Unfortunately, I'm running into a malloc error when trying to zip or unzip very large files (685MB zipped, 2.1GB unzipped). I'm running this on a machine w/ 4GB ram, so I think it may actually a limitation in the memory size that ruby itself allows. Are there any plans to implement a buffer to unzip larger files? Unfortunately I'm not familiar with C well enough to edit the bindings myself, hoping this is on your radar. Thanks again!

Erik Kastman

Reader does, but not Writer.

I'd love to use this library, but without any documentation, it's totally useless.
This doesn't help either because, for example, Bzip2.new and .open are private in your library.

Can you provide documentation?
Thank you.

bzip2.c:6:21: fatal error: version.h: No such file or directory

yardoc is unable to compile the documentation and fails with the following message:

[warn]: Load Order / Name Resolution Problem on Bzip2Singleton:
[warn]: -
[warn]: Something is trying to call child on object Bzip2Singleton before it has been recognized.
[warn]: This error usually means that you need to modify the order in which you parse files
[warn]: so that Bzip2Singleton is parsed before methods or other objects attempt to access it.
[warn]: -
[warn]: YARD will recover from this error and continue to parse but you *may* have problems
[warn]: with your generated documentation. You should probably fix this.
[warn]: -
/usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/code_objects/proxy.rb:218:in `rescue in method_missing': Proxy cannot call method #child on object 'Bzip2Singleton' (YARD::CodeObjects::ProxyMethodError)
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/code_objects/proxy.rb:215:in `method_missing'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/parser/c_parser.rb:108:in `handle_alias'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/parser/c_parser.rb:415:in `block in parse_aliases'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/parser/c_parser.rb:408:in `scan'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/parser/c_parser.rb:408:in `parse_aliases'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/parser/c_parser.rb:18:in `parse'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/parser/source_parser.rb:438:in `parse'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/parser/source_parser.rb:361:in `parse_in_order'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/parser/source_parser.rb:89:in `block in parse'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/logging.rb:66:in `enter_level'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/parser/source_parser.rb:88:in `parse'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard.rb:18:in `parse'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/cli/yardoc.rb:215:in `run'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/lib/yard/cli/command.rb:13:in `run'
      from /usr/lib64/ruby/gems/1.9.1/gems/yard-0.7.4/bin/yardoc:4:in `<top (required)>'
      from /usr/bin/yardoc:19:in `load'
      from /usr/bin/yardoc:19:in `<main>'

[colin@bricktop plas]$ gem install bzip2-ruby
Building native extensions.  This could take a while...
ERROR:  Error installing bzip2-ruby:
    ERROR: Failed to build gem native extension.

/home/colin/.rvm/rubies/ruby-head/bin/ruby extconf.rb
checking for bzlib.h... yes
checking for BZ2_bzWriteOpen() in -lbz2... yes
creating Makefile

make
compiling bzip2.c
bzip2.c: In function ‘bz_next_available’:
bzip2.c:779:18: warning: operation on ‘bzf->buf’ may be undefined
bzip2.c: In function ‘bz_reader_ungetc’:
bzip2.c:981:18: warning: operation on ‘bzf->buf’ may be undefined
bzip2.c: In function ‘bz_reader_ungets’:
bzip2.c:1025:18: warning: operation on ‘bzf->buf’ may be undefined
bzip2.c: In function ‘Init_bzip2_ext’:
bzip2.c:1726:5: error: ISO C90 forbids mixed declarations and code
make: *** [bzip2.o] Error 1


Gem files will remain installed in /home/colin/.rvm/gems/ruby-head@plas/gems/bzip2-ruby-0.2.7 for inspection.
Results logged to /home/colin/.rvm/gems/ruby-head@plas/gems/bzip2-ruby-0.2.7/ext/gem_make.out

gem install bzip2-ruby --version 0.2.6 works fine for me.

Versions:

[colin@bricktop plas]$ gem --version
1.3.7
[colin@bricktop plas]$ ruby --version
ruby 1.9.3dev (2010-12-18 trunk 30244) [x86_64-linux]
[colin@bricktop plas]$ dpkg --list | grep bz2
ii  libbz2-1.0                                1.0.5-6                                    high-quality block-sorting file compressor library - runtime
ii  libbz2-dev                                1.0.5-6                                    high-quality block-sorting file compressor library - development
[colin@bricktop plas]$ lsb_release -dr
Description:    Ubuntu natty (development branch)
Release:    11.04

Hi,

in contrast to regular file handles and gzip filehandles, the bzip2 library does not support the tell method, telling the position in the current (uncompressed) stream. It's an incompatibility. Would be nice to have. :-)

regards
Hadmut

Hi

I'm trying to use bzip2-ruby to stream-in .tar.bz2 files via Gem::Package::TarReader.

BRIEF: It doesn't work yet due to this:

NoMethodError: undefined method `pos' for #<Bzip2::Reader:0xaa2bf5c>

Details follow.

require "bzip2-ruby"
require "rubygems/package"

#fn = "/path/to/file.tar"
fn = "/path/to/file.tar.bz2"
#fn = "/path/to/file.tar.gz"

stream = if fn.match /\.bz2\z/
  Bzip2::Reader.open(fn)
elsif fn.match /\.gz\z/
  Zlib::GzipReader.open(fn)
elsif fn.match /\.tar\z/
  File.open(fn)
else
  raise "Error: Don't know how to handle '#{fn}', aborting"
end

untar = Gem::Package::TarReader.new(stream)
untar.each do |entry|
  ...
end

Plain tar and .tar.gz work. .tar.bz2 doesn't, here's the exception info:

NoMethodError: undefined method `pos' for #<Bzip2::Reader:0xaad6394>
        from /home/rbdev/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/package/tar_reader.rb:42:in `initialize'
        from /home/rbdev/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/package/tar_reader.rb:23:in `new'
        from /home/rbdev/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/package/tar_reader.rb:23:in `new'

Thank you.

Alex

RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can imagine, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.

There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!

Appendix:

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies

p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.