intake / python-snappy Goto Github PK

How should I go about it? It works OK in nodejs without any issues.

It appears that the module-level functions snappy.compress and snappy.uncompress have bitrotted and fallen behind the new framed version produced/consumed by the command-line interface.

Is the framed version adopted enough that those should be made consistent? Regardless, having this inconsistency seems like the wrong thing to do.

$ echo 'hello' | python -m snappy -c > hello.sz
$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import snappy
snappy.uncompress(open('hello.sz').read())
Traceback (most recent call last):
File "", line 1, in
snappy.UncompressError: Error while decompressing: invalid input

open('hello.sz').read()
'\xff\x06\x00\x00sNaPpY\x01\n\x00\x00SU\xffShello\n'
d = open('hello.sz').read()
snappy.uncompress(d)
Traceback (most recent call last):
File "", line 1, in
snappy.UncompressError: Error while decompressing: invalid input

snappy.compress('hello\n')
'\x06\x14hello\n'

from cStringIO import StringIO

fh = StringIO()
snappy.stream_decompress(StringIO(d), fh)
fh.getvalue()
'hello\n'

It sounds like a framing format is used over plain encrypted blocks (which is good). But it is not clear what format this would be, given that Snappy does not have standard one.
That is, there are multiple ones

• one used by https://github.com/dain/snappy (pure java snappy)
• one described in https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt

I'd be guessing it might be second one, but would want to know, to make sure I am using compatible implementations.

This drove me crazy before my morning coffee sank in and I realized something was wrong when trying to install the dependency off the link.

Google moved the project to github and it splashes with an "organization" page. The download touts for the snappy library on http://google.github.io/snappy default to the current MASTER git source - not an 'easily installable version'. The active git version - which is linked to on the front page - needs to be installed via a package maintainer's workflow and requires a lot of environment setup (which is a complete f****** pain).

If click into the project's full github page, and then the releases page (https://github.com/google/snappy/releases) to find an installable version (with configure and everything set up straight), the install process is a straightforward configure/make/install -- even on the core osx distribution.

After having installed 1) snappy from github.com/google/snappy , and 2) sudo pip install python-snappy, in python 2.7 environment, I can not "import snappy". It threw errors like this:

>>> import snappy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/snappy.py", line 50, in <module>
    from snappy_cffi import UncompressError, compress, decompress, \
  File "/usr/local/lib/python2.7/dist-packages/snappy_cffi.py", line 3, in <module>
    from cffi import FFI
ImportError: cannot import name FFI

Importing cffi, made even worse, with more errors.

How to fix this issue?

atso% uname -a
Darwin andrew-tsos-macbook-pro.local 10.5.0 Darwin Kernel Version 10.5.0: Fri Nov 5 23:20:39 PDT 2010; root:xnu-1504.9.17~1/RELEASE_I386 i386

atso% gcc --version
i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

atso% pip install python-snappy
Downloading/unpacking python-snappy
Running setup.py egg_info for package python-snappy
Installing collected packages: python-snappy
Running setup.py install for python-snappy
building 'snappy' extension
/usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c snappymodule.cc -o build/temp.macosx-10.6-x86_64-2.6/snappymodule.o
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for C/ObjC but not for C++
snappymodule.cc:31:22: error: snappy-c.h: No such file or directory
snappymodule.cc: In function ‘PyObject* snappy__compress(PyObject_, PyObject_)’:
snappymodule.cc:62: error: ‘snappy_status’ was not declared in this scope
snappymodule.cc:62: error: expected ;' before ‘status’ snappymodule.cc:75: error: ‘snappy_max_compressed_length’ was not declared in this scope snappymodule.cc:79: error: ‘status’ was not declared in this scope snappymodule.cc:79: error: ‘snappy_compress’ was not declared in this scope snappymodule.cc:81: error: ‘SNAPPY_OK’ was not declared in this scope snappymodule.cc: In function ‘PyObject* snappy__uncompress(PyObject*, PyObject*)’: snappymodule.cc:107: error: ‘snappy_status’ was not declared in this scope snappymodule.cc:107: error: expected;' before ‘status’
snappymodule.cc:120: error: ‘status’ was not declared in this scope
snappymodule.cc:120: error: ‘snappy_uncompressed_length’ was not declared in this scope
snappymodule.cc:121: error: ‘SNAPPY_OK’ was not declared in this scope
snappymodule.cc:128: error: ‘snappy_uncompress’ was not declared in this scope
snappymodule.cc:129: error: ‘SNAPPY_OK’ was not declared in this scope
snappymodule.cc: In function ‘PyObject* snappy__is_valid_compressed_buffer(PyObject_, PyObject_)’:
snappymodule.cc:151: error: ‘snappy_status’ was not declared in this scope
snappymodule.cc:151: error: expected `;' before ‘status’
snappymodule.cc:156: error: ‘status’ was not declared in this scope
snappymodule.cc:156: error: ‘snappy_validate_compressed_buffer’ was not declared in this scope
snappymodule.cc:157: error: ‘SNAPPY_OK’ was not declared in this scope
snappymodule.cc: At global scope:
snappymodule.cc:41: warning: ‘_state’ defined but not used
error: command '/usr/bin/gcc-4.2' failed with exit status 1
Complete output from command /Users/atso/.virtualenv/subscribed/bin/python -c "import setuptools; file='/Users/atso/.virtualenv/subscribed/build/python-snappy/setup.py'; execfile('/Users/atso/.virtualenv/subscribed/build/python-snappy/setup.py')" install --single-version-externally-managed --record /var/folders/8u/8uRxBUB4FseEZBUr-aAi3k+++TI/-Tmp-/pip-ZMZFjJ-record/install-record.txt --install-headers /Users/atso/.virtualenv/subscribed/lib/include:
running install

running build

running build_ext

building 'snappy' extension

/usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c snappymodule.cc -o build/temp.macosx-10.6-x86_64-2.6/snappymodule.o

cc1plus: warning: command line option "-Wstrict-prototypes" is valid for C/ObjC but not for C++

snappymodule.cc:31:22: error: snappy-c.h: No such file or directory

snappymodule.cc: In function ‘PyObject* snappy__compress(PyObject_, PyObject_)’:

snappymodule.cc:62: error: ‘snappy_status’ was not declared in this scope

snappymodule.cc:62: error: expected `;' before ‘status’

snappymodule.cc:75: error: ‘snappy_max_compressed_length’ was not declared in this scope

snappymodule.cc:79: error: ‘status’ was not declared in this scope

snappymodule.cc:79: error: ‘snappy_compress’ was not declared in this scope

snappymodule.cc:81: error: ‘SNAPPY_OK’ was not declared in this scope

snappymodule.cc: In function ‘PyObject* snappy__uncompress(PyObject_, PyObject_)’:

snappymodule.cc:107: error: ‘snappy_status’ was not declared in this scope

snappymodule.cc:107: error: expected `;' before ‘status’

snappymodule.cc:120: error: ‘status’ was not declared in this scope

snappymodule.cc:120: error: ‘snappy_uncompressed_length’ was not declared in this scope

snappymodule.cc:121: error: ‘SNAPPY_OK’ was not declared in this scope

snappymodule.cc:128: error: ‘snappy_uncompress’ was not declared in this scope

snappymodule.cc:129: error: ‘SNAPPY_OK’ was not declared in this scope

snappymodule.cc: In function ‘PyObject* snappy__is_valid_compressed_buffer(PyObject_, PyObject_)’:

snappymodule.cc:151: error: ‘snappy_status’ was not declared in this scope

snappymodule.cc:151: error: expected `;' before ‘status’

snappymodule.cc:156: error: ‘status’ was not declared in this scope

snappymodule.cc:156: error: ‘snappy_validate_compressed_buffer’ was not declared in this scope

snappymodule.cc:157: error: ‘SNAPPY_OK’ was not declared in this scope

snappymodule.cc: At global scope:

snappymodule.cc:41: warning: ‘_state’ defined but not used

error: command '/usr/bin/gcc-4.2' failed with exit status 1

Command /Users/atso/.virtualenv/subscribed/bin/python -c "import setuptools; file='/Users/atso/.virtualenv/subscribed/build/python-snappy/setup.py'; execfile('/Users/atso/.virtualenv/subscribed/build/python-snappy/setup.py')" install --single-version-externally-managed --record /var/folders/8u/8uRxBUB4FseEZBUr-aAi3k+++TI/-Tmp-/pip-ZMZFjJ-record/install-record.txt --install-headers /Users/atso/.virtualenv/subscribed/lib/include failed with error code 1

The install_requires in setup.py does not mention the dependencies. Everytime I try to install snappy with command 'pip install snappy', it fails for library dependency and I need to take trial and error approach to install it.

I realise this is a somewhat nonsensical use-case, but if I do the following under Python 2.7.8 I get an error:

>>> import snappy
>>> snappy.decompress(snappy.compress(''))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: Objects/stringobject.c:3903: bad argument to internal function

Under Python 2.6.6 it seems to work (returns ''). If it helps, I noticed this trying to build avro (https://github.com/apache/avro/tree/trunk/lang/py) under Python 2.7.8.

The ominous lack of commits and mounting number of outstanding issues and pull requests inescapably suggests this once-optimistic project to now be yet another a shambling GitHub zombie. ^{...and just in time for Halloween!}

This saddens us all. Would any intrepid unpaid volunteer with an open pull request mind producing an official, well-maintained fork of this project under a different name? I'm partial to mrsnips myself, but anything will do. 🦂

Let's just do something, people. It's time to fight commit inactivity with renewed vigour!

I built snappy with brew install snappy or brew install snappy # snappy library from Google

it have a problem

-bash: brew: command not found

how to find it?

This is a follow-on to issue #3, streaming compression support.

Snappy framing format:

http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt

Is there a way to make snappy.decompress() work directly on a memoryview without casting it to another datatype? Casting makes a copy of the data which is undesirable in our case.

compressed = snappy.compress('The quick brown fox jumped over the lazy dog.')
print(snappy.decompress(buffer(compressed)))      # works on Python 2.7
print(snappy.decompress(memoryview(compressed)))  # fails on Python 2.7 and Python 3
# Python 2: TypeError: argument 1 must be string or read-only buffer, not memoryview
# Python 3: TypeError: argument 1 must be read-only bytes-like object, not memoryview

https://github.com/cclauss/snappy-hack/blob/master/snappy_hack.py

it works !!
try:
CPPFLAGS="-I/usr/local/include -L/usr/local/lib -stdlib=libc++ " pip install python-snappy

This library appears to be both useful to the Python community and currently unmaintained. This is normal, developers sometimes move on, but we should decide what to do.

@andrix would you be willing to transfer ownership of this library to someone else? Also would you be willing to give someone else ownership rights over the package on PyPI?

Hello, I am not a programmer so I apologize if this is not the right place for this or a trivial issue. I am trying to install python-snappy in cygwin as a dependency for another program - avro - and I get the following error. I have installed snappy version 1.0.5 and I am trying to install the current python-snappy-0.4.

building 'snappy' extension
gcc -fno-strict-aliasing -g -O2 -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/includ e/python2.6 -c snappymodule.cc -o build/temp.cygwin-1.7.16-i686-2.6/snappymodule.o
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
snappymodule.cc:43: warning: '_state' defined but not used
g++ -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.7.16-i686-2.6/snappymodule.o -L/usr/lib/ python2.6/config -lsnappy -lpython2.6 -o build/lib.cygwin-1.7.16-i686-2.6/snappy.dll
/usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: cannot find -lsnappy
collect2: ld returned 1 exit status
error: command 'g++' failed with exit status 1

Any help would be appreciated, thanks in advance.

I'm on Anaconda's Python 3.4 on RHEL.

I installed Snappy and then installed python-snappy with PIP to:
/usr/local/lib/python3.4/site-packages/
but it cannot be imported with:

import snappy

ImportError: No module named 'snappy'

Hi,

I have been chasing a problem over the last days that I initially thought belonged to dask-distributed, later to fastparquet and eventually to snappy.

What I am doing is very simple. I am taking the larger file of kaggle/wsdm, feeding into a python script that converts in to parquet.
`
fp.write(os.path.join(parquet_dir,"user_logs.parq"),
df,
compression="snappy",
write_index=True,
fixed_text={"msno":44},
file_scheme="hive",
row_group_offsets=100000,
append=True

                  )

`

Essentially a 33GB ascii csv, that once in parquet is about 10GB.

The process is done on linux/ubuntu
dask-workers in linux ingest it happily.

However, same file in windows with identical spec windows 7 prof fully patched fails.

Same python lib versions as well
`
distributed.worker - WARNING - Compute Failed
Function: _read_parquet_row_group
args: (<dask.bytes.core.OpenFileCreator object at 0x000000000A876828>, '/home/julian/wsdm/parquet/user_logs.parq/part.2707.parq
uet', 'msno', 'num_100', <class 'fastparquet.parquet_thrift.parquet.ttypes.RowGroup'>
columns: [<class 'fastparquet.parquet_thrift.parquet.ttypes.ColumnChunk'>
file_offset: 1258524
file_path: part.2707.parquet
meta_data: <class 'fastparquet.parquet_thrift.parquet.ttypes.ColumnMetaData'>
codec: 1
data_page_offset: 4
dictionary_page_offset: None
encoding_stats: [<class 'fastparquet.parquet_thrift.parquet.ttypes.PageEncodingStats'>
count: 1
encoding: 0
page_type: 0
]
encodings: [3, 4, 0]
index_page_offset: None
key_value_metadata: []
num_values: 100000
path_in_schema: ['msno']
statistics: <class 'fastparquet.parquet_thrift.parquet.ttypes.Statistics'>
distinct_count: None
max: b'zzqmzWghWZupF3neoC+fNKRkzN6l0BpmRjR1ZUpWnIE='
min: b'++9nzvvMyez1LasLEYdlEA/toFCTbLhhK6NVozPiQOg='
null_count: 0

total_compressed_size: 1258520
total_unco
kwargs: {}
Exception: MemoryError()

`

I'm having trouble using snappy on a TLS-encrypted connection. At that stage, I am not yet buffering data. However, I'll call ssl.wrap_socket (actually gevent.ssl.wrap_socket, which is a monkey-patched version of the original), and then start filtering all reads and writes through Snappy. I'm getting "UncompressError: stream missing snappy identifier" every time. However, Snappy on its own works perfectly, and TLS on its own works perfectly.

Are there any hangups or advice that I should be aware of?

I have a parallel function which is compressing large number of files using python-snappy. Is there a way to merge all the returned compressed streams into one snappy file?

Thanks in advance

import snappy
Traceback (most recent call last):
File "", line 1, in
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/snappy/init.py", line 10, in
from .SnapPy import (AbelianGroup, HolonomyGroup, FundamentalGroup,
File "SnapPycore.pxi", line 38, in init SnapPy (cython/SnapPy.c:104679)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/spherogram/init.py", line 3, in
from .links import *
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/spherogram/links/init.py", line 1, in
from .links import Crossing, Strand, Link
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/spherogram/links/links.py", line 2, in
from .invariants import Link
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/spherogram/links/invariants.py", line 399
raise ValueError, 'The link diagram is split.'
^
SyntaxError: invalid syntax

Seems like make is missing an include for os x.

[andrew-tsos-macbook-pro] atso% port list |grep snappy
snappy @1.0.4 archivers/snappy

[andrew-tsos-macbook-pro] atso% uname -a
Darwin andrew-tsos-macbook-pro.local 10.5.0 Darwin Kernel Version 10.5.0: Fri Nov 5 23:20:39 PDT 2010; root:xnu-1504.9.17~1/RELEASE_I386 i386

[andrew-tsos-macbook-pro] atso% pip install python-snappy
Downloading/unpacking python-snappy
Running setup.py egg_info for package python-snappy

Installing collected packages: python-snappy
Running setup.py install for python-snappy
building 'snappy' extension
cc -arch x86_64 -fPIC -Wimplicit -I/Users/atso/.virtualenv/subscribed-pypy-1.6/include -c snappymodule.cc -o build/temp.macosx-10.6-x86_64-2.7/snappymodule.o
snappymodule.cc:31:22: error: snappy-c.h: No such file or directory
snappymodule.cc: In function ‘PyObject* snappy__compress(PyObject_, PyObject_)’:
snappymodule.cc:62: error: ‘snappy_status’ was not declared in this scope
snappymodule.cc:62: error: expected ;' before ‘status’ snappymodule.cc:75: error: ‘snappy_max_compressed_length’ was not declared in this scope snappymodule.cc:79: error: ‘status’ was not declared in this scope snappymodule.cc:79: error: ‘snappy_compress’ was not declared in this scope snappymodule.cc:81: error: ‘SNAPPY_OK’ was not declared in this scope snappymodule.cc: In function ‘PyObject* snappy__uncompress(PyObject*, PyObject*)’: snappymodule.cc:107: error: ‘snappy_status’ was not declared in this scope snappymodule.cc:107: error: expected;' before ‘status’
snappymodule.cc:120: error: ‘status’ was not declared in this scope
snappymodule.cc:120: error: ‘snappy_uncompressed_length’ was not declared in this scope
snappymodule.cc:121: error: ‘SNAPPY_OK’ was not declared in this scope
snappymodule.cc:128: error: ‘snappy_uncompress’ was not declared in this scope
snappymodule.cc:129: error: ‘SNAPPY_OK’ was not declared in this scope
snappymodule.cc: In function ‘PyObject* snappy__is_valid_compressed_buffer(PyObject_, PyObject_)’:
snappymodule.cc:151: error: ‘snappy_status’ was not declared in this scope
snappymodule.cc:151: error: expected `;' before ‘status’
snappymodule.cc:156: error: ‘status’ was not declared in this scope
snappymodule.cc:156: error: ‘snappy_validate_compressed_buffer’ was not declared in this scope
snappymodule.cc:157: error: ‘SNAPPY_OK’ was not declared in this scope
error: command 'cc' failed with exit status 1
Complete output from command /Users/atso/.virtualenv/subscribed-pypy-1.6/bin/pypy -c "import setuptools;file='/Users/atso/.virtualenv/subscribed-pypy-1.6/build/python-snappy/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --single-version-externally-managed --record /var/folders/8u/8uRxBUB4FseEZBUr-aAi3k+++TI/-Tmp-/pip-Alvsc4-record/install-record.txt --install-headers /Users/atso/.virtualenv/subscribed-pypy-1.6/include/site/python2.7:
running install

running build

running build_ext

building 'snappy' extension

cc -arch x86_64 -fPIC -Wimplicit -I/Users/atso/.virtualenv/subscribed-pypy-1.6/include -c snappymodule.cc -o build/temp.macosx-10.6-x86_64-2.7/snappymodule.o

snappymodule.cc:31:22: error: snappy-c.h: No such file or directory

snappymodule.cc: In function ‘PyObject* snappy__compress(PyObject_, PyObject_)’:

snappymodule.cc:62: error: ‘snappy_status’ was not declared in this scope

snappymodule.cc:62: error: expected `;' before ‘status’

snappymodule.cc:75: error: ‘snappy_max_compressed_length’ was not declared in this scope

snappymodule.cc:79: error: ‘status’ was not declared in this scope

snappymodule.cc:79: error: ‘snappy_compress’ was not declared in this scope

snappymodule.cc:81: error: ‘SNAPPY_OK’ was not declared in this scope

snappymodule.cc: In function ‘PyObject* snappy__uncompress(PyObject_, PyObject_)’:

snappymodule.cc:107: error: ‘snappy_status’ was not declared in this scope

snappymodule.cc:107: error: expected `;' before ‘status’

snappymodule.cc:120: error: ‘status’ was not declared in this scope

snappymodule.cc:120: error: ‘snappy_uncompressed_length’ was not declared in this scope

snappymodule.cc:121: error: ‘SNAPPY_OK’ was not declared in this scope

snappymodule.cc:128: error: ‘snappy_uncompress’ was not declared in this scope

snappymodule.cc:129: error: ‘SNAPPY_OK’ was not declared in this scope

snappymodule.cc: In function ‘PyObject* snappy__is_valid_compressed_buffer(PyObject_, PyObject_)’:

snappymodule.cc:151: error: ‘snappy_status’ was not declared in this scope

snappymodule.cc:151: error: expected `;' before ‘status’

snappymodule.cc:156: error: ‘status’ was not declared in this scope

snappymodule.cc:156: error: ‘snappy_validate_compressed_buffer’ was not declared in this scope

snappymodule.cc:157: error: ‘SNAPPY_OK’ was not declared in this scope

error: command 'cc' failed with exit status 1

Command /Users/atso/.virtualenv/subscribed-pypy-1.6/bin/pypy -c "import setuptools;file='/Users/atso/.virtualenv/subscribed-pypy-1.6/build/python-snappy/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --single-version-externally-managed --record /var/folders/8u/8uRxBUB4FseEZBUr-aAi3k+++TI/-Tmp-/pip-Alvsc4-record/install-record.txt --install-headers /Users/atso/.virtualenv/subscribed-pypy-1.6/include/site/python2.7 failed with error code 1
Storing complete log in /Users/atso/.pip/pip.log

Hi.
Would you mind to include hadoop snappy compression format support? Here I've just implemented this (compression & decompression): https://github.com/aleks-v-k/python-snappy/tree/hadoop_snappy_format

I can make a PR if you will accept such changes.

I am trying to upgrade python-snappy and the install is failing:

C:\>pip install python-snappy --upgrade
Collecting python-snappy
  Using cached python-snappy-0.5.1.tar.gz
Building wheels for collected packages: python-snappy
  Running setup.py bdist_wheel for python-snappy ... error
  Complete output from command C:\Users\my.name\AppData\Local\Continuum\Anaconda2\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\my.name\\appdata\\local\\temp\\pip-build-wukcy4\\python-snappy\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d c:\users\my.name\appdata\local\temp\tmprao6jypip-wheel- --python-tag cp27:
  WARNING: '.' not a valid package name; please use only .-separated package names in setup.py
  running bdist_wheel
  running build
  running build_py
  package init file '__init__.py' not found (or not a regular file)
  creating build
  creating build\lib.win-amd64-2.7
  copying setup.py -> build\lib.win-amd64-2.7
  copying snappy.py -> build\lib.win-amd64-2.7
  copying snappy_cffi.py -> build\lib.win-amd64-2.7
  copying test_snappy.py -> build\lib.win-amd64-2.7
  copying test_snappy_cffi.py -> build\lib.win-amd64-2.7
  running build_ext
  building '_snappy' extension
  creating build\temp.win-amd64-2.7
  creating build\temp.win-amd64-2.7\Release
  C:\Users\my.name\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Users\my.name\AppData\Local\Continuum\Anaconda2\include -IC:\Users\my.name\AppData\Local\Continuum\Anaconda2\PC /Tpsnappymodule.cc /Fobuild\temp.win-amd64-2.7\Release\snappymodule.obj
  snappymodule.cc
  C:\Users\my.name\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Users\my.name\AppData\Local\Continuum\Anaconda2\include -IC:\Users\my.name\AppData\Local\Continuum\Anaconda2\PC /Tccrc32c.c /Fobuild\temp.win-amd64-2.7\Release\crc32c.obj
  crc32c.c
  c:\users\my.name\appdata\local\temp\pip-build-wukcy4\python-snappy\crc32c.h(56) : error C2054: expected '(' to follow 'inline'
  c:\users\my.name\appdata\local\temp\pip-build-wukcy4\python-snappy\crc32c.h(57) : error C2085: 'crc_init' : not in formal parameter list
  c:\users\my.name\appdata\local\temp\pip-build-wukcy4\python-snappy\crc32c.h(57) : error C2143: syntax error : missing ';' before '{'
  c:\users\my.name\appdata\local\temp\pip-build-wukcy4\python-snappy\crc32c.h(79) : error C2054: expected '(' to follow 'inline'
  c:\users\my.name\appdata\local\temp\pip-build-wukcy4\python-snappy\crc32c.h(80) : error C2085: 'crc_finalize' : not in formal parameter list
  c:\users\my.name\appdata\local\temp\pip-build-wukcy4\python-snappy\crc32c.h(80) : error C2143: syntax error : missing ';' before '{'
  error: command 'C:\\Users\\my.name\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe' failed with exit status 2

I'm using Python 2.7.13 with Anaconda on Windows 10 and I have already succesfully installed python-snappy into my Anaconda environment in a previous version around the end of march.
Back then I went through several steps as detailed here to get snappy working on Windows 10 and I succeeded, everything was working fine then.

Now the upgrade fails and I have no clue why that C error is appearing.

Could it be a problem with the stdint.h?

Python-snappy build fails in Mac OSX 10.9. Even though the Snappy-c library is installed, python build fails to find the dependancies.

cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c crc32c.c -o build/temp.macosx-10.9-intel-2.7/crc32c.o
c++ -bundle -undefined dynamic_lookup -arch x86_64 -arch i386 -Wl,-F. build/temp.macosx-10.9-intel-2.7/snappymodule.o build/temp.macosx-10.9-intel-2.7/crc32c.o -lsnappy -o build/lib.macosx-10.9-intel-2.7/_snappy.so
ld: library not found for -lsnappy
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'c++' failed with exit status 1

When I try to install the module I get this error
how do I fix it?

c:\Users\X\Desktop\python-snappy-master>python setup.py build
C:\Python27\lib\distutils\dist.py:267: UserWarning: Unknown distribution option:
'install_requires'
warnings.warn(msg)
running build
running build_py
package init file 'init.py' not found (or not a regular file)
package init file 'init.py' not found (or not a regular file)
running build_ext
building '_snappy' extension
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox
/MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\PC /Tpsnappymodule.cc
/Fobuild\temp.win32-2.7\Release\snappymodule.obj
snappymodule.cc
snappymodule.cc(31) : fatal error C1083: Cannot open include file: 'snappy-c.h':
No such file or directory
error: command 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\c
l.exe' failed with exit status 2

c:\Users\X\Desktop\python-snappy-master>cd c:\users\X\Desktop\pyth
on-snappy-master

Using Anaconda here, but it also didn't work when not in the anaconda environment.

import snappy
Traceback (most recent call last):
File "", line 1, in
File "/Users/johnmin/src/anaconda3/lib/python3.5/site-packages/snappy/init.py", line 10, in
from .SnapPy import (AbelianGroup, HolonomyGroup, FundamentalGroup,
File "SnapPycore.pxi", line 38, in init SnapPy (cython/SnapPy.c:104679)
File "/Users/johnmin/src/anaconda3/lib/python3.5/site-packages/spherogram/init.py", line 3, in
from .links import *
File "/Users/johnmin/src/anaconda3/lib/python3.5/site-packages/spherogram/links/init.py", line 1, in
from .links import Crossing, Strand, Link
File "/Users/johnmin/src/anaconda3/lib/python3.5/site-packages/spherogram/links/links.py", line 1, in
from .links_base import Crossing, Strand, CrossingStrand, CrossingEntryPoint
File "/Users/johnmin/src/anaconda3/lib/python3.5/site-packages/spherogram/links/links_base.py", line 12, in
from ordered_set import OrderedSet
ImportError: No module named 'ordered_set'

It failed for a first time so I figured I should first try to upgrade pip and then try again.

pip install --upgrade pip 
pip install python-snappy

I get the following output:

------------------------------------------------------------
C:\Python27\Scripts\pip-script.py run on 01/31/14 11:13:45
Downloading/unpacking python-snappy
  Getting page https://pypi.python.org/simple/python-snappy/
  URLs to search for versions for python-snappy:
  * https://pypi.python.org/simple/python-snappy/
  Analyzing links from page https://pypi.python.org/simple/python-snappy/
    Found link https://pypi.python.org/packages/source/p/python-snappy/python-snappy-0.2.1.tar.gz#md5=c34c46eb4c3f3673b1d3e85652edddd8 (from https://pypi.python.org/simple/python-snappy/), version: 0.2.1
    Found link https://pypi.python.org/packages/source/p/python-snappy/python-snappy-0.2.2.tar.gz#md5=8ea73ee4df62fa73d6a41a4a715f5df9 (from https://pypi.python.org/simple/python-snappy/), version: 0.2.2
    Found link https://pypi.python.org/packages/source/p/python-snappy/python-snappy-0.2.tar.gz#md5=6c86e44cc1bf4077cf4c7e70f1da2744 (from https://pypi.python.org/simple/python-snappy/), version: 0.2
    Found link https://pypi.python.org/packages/source/p/python-snappy/python-snappy-0.3.1.tar.gz#md5=872da91e53b63fb51e766cf3e67bc4fc (from https://pypi.python.org/simple/python-snappy/), version: 0.3.1
    Found link https://pypi.python.org/packages/source/p/python-snappy/python-snappy-0.3.2.tar.gz#md5=94ec3eb54a780fac3b15a6c141af973f (from https://pypi.python.org/simple/python-snappy/), version: 0.3.2
    Found link https://pypi.python.org/packages/source/p/python-snappy/python-snappy-0.3.tar.gz#md5=e3cac7f7872a3cdc2e719978626c96f0 (from https://pypi.python.org/simple/python-snappy/), version: 0.3
    Found link https://pypi.python.org/packages/source/p/python-snappy/python-snappy-0.4.tar.gz#md5=3e82ec29ddf47e63cb0c56ba48ea516c (from https://pypi.python.org/simple/python-snappy/), version: 0.4
    Found link https://pypi.python.org/packages/source/p/python-snappy/python-snappy-0.5.tar.gz#md5=044fb0ac21888c0b4eee0c4f15e75504 (from https://pypi.python.org/simple/python-snappy/), version: 0.5
    Skipping link http://code.google.com/p/snappy (from https://pypi.python.org/simple/python-snappy/); not a file
  Using version 0.5 (newest of versions: 0.5, 0.4, 0.3.2, 0.3.1, 0.3, 0.2.2, 0.2.1, 0.2)
  Downloading python-snappy-0.5.tar.gz
  Downloading from URL https://pypi.python.org/packages/source/p/python-snappy/python-snappy-0.5.tar.gz#md5=044fb0ac21888c0b4eee0c4f15e75504 (from https://pypi.python.org/simple/python-snappy/)
  Running setup.py (path:c:\users\sjuul\appdata\local\temp\pip_build_sjuul\python-snappy\setup.py) egg_info for package python-snappy
    running egg_info
    creating pip-egg-info\python_snappy.egg-info
    writing pip-egg-info\python_snappy.egg-info\PKG-INFO
    writing top-level names to pip-egg-info\python_snappy.egg-info\top_level.txt
    writing dependency_links to pip-egg-info\python_snappy.egg-info\dependency_links.txt
    writing manifest file 'pip-egg-info\python_snappy.egg-info\SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'pip-egg-info\python_snappy.egg-info\SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pip-egg-info\python_snappy.egg-info\SOURCES.txt'
  Source in c:\users\sjuul\appdata\local\temp\pip_build_sjuul\python-snappy has version 0.5, which satisfies requirement python-snappy
Installing collected packages: python-snappy
  Running setup.py install for python-snappy
    Running command C:\Python27\python.exe -c "import setuptools, tokenize;__file__='c:\\users\\sjuul\\appdata\\local\\temp\\pip_build_sjuul\\python-snappy\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\users\sjuul\appdata\local\temp\pip-ijfbau-record\install-record.txt --single-version-externally-managed --compile
    running install
    running build
    running build_py
    creating build
    creating build\lib.win32-2.7
    copying snappy.py -> build\lib.win32-2.7
    running build_ext
    building '_snappy' extension
    creating build\temp.win32-2.7
    creating build\temp.win32-2.7\Release
    c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\PC /Tpsnappymodule.cc /Fobuild\temp.win32-2.7\Release\snappymodule.obj
    snappymodule.cc
    snappymodule.cc(31) : fatal error C1083: Cannot open include file: 'snappy-c.h': No such file or directory
    error: command '"c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe"' failed with exit status 2
    Complete output from command C:\Python27\python.exe -c "import setuptools, tokenize;__file__='c:\\users\\sjuul\\appdata\\local\\temp\\pip_build_sjuul\\python-snappy\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\users\sjuul\appdata\local\temp\pip-ijfbau-record\install-record.txt --single-version-externally-managed --compile:
    running install

running build

running build_py

creating build

creating build\lib.win32-2.7

copying snappy.py -> build\lib.win32-2.7

running build_ext

building '_snappy' extension

creating build\temp.win32-2.7

creating build\temp.win32-2.7\Release

c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\PC /Tpsnappymodule.cc /Fobuild\temp.win32-2.7\Release\snappymodule.obj

snappymodule.cc

snappymodule.cc(31) : fatal error C1083: Cannot open include file: 'snappy-c.h': No such file or directory

error: command '"c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe"' failed with exit status 2

----------------------------------------
Cleaning up...
  Removing temporary dir c:\users\sjuul\appdata\local\temp\pip_build_sjuul...
Command C:\Python27\python.exe -c "import setuptools, tokenize;__file__='c:\\users\\sjuul\\appdata\\local\\temp\\pip_build_sjuul\\python-snappy\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\users\sjuul\appdata\local\temp\pip-ijfbau-record\install-record.txt --single-version-externally-managed --compile failed with error code 1 in c:\users\sjuul\appdata\local\temp\pip_build_sjuul\python-snappy
Exception information:
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\basecommand.py", line 122, in main
    status = self.run(options, args)
  File "c:\python27\lib\site-packages\pip\commands\install.py", line 279, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "c:\python27\lib\site-packages\pip\req.py", line 1380, in install
    requirement.install(install_options, global_options, *args, **kwargs)
  File "c:\python27\lib\site-packages\pip\req.py", line 699, in install
    cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
  File "c:\python27\lib\site-packages\pip\util.py", line 697, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command C:\Python27\python.exe -c "import setuptools, tokenize;__file__='c:\\users\\sjuul\\appdata\\local\\temp\\pip_build_sjuul\\python-snappy\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\users\sjuul\appdata\local\temp\pip-ijfbau-record\install-record.txt --single-version-externally-managed --compile failed with error code 1 in c:\users\sjuul\appdata\local\temp\pip_build_sjuul\python-snappy

Version 0.5.1 can be installed for pypy3 or pypy3.5, but 0.5.2 installs in a broken state.

(snappypypy35) ubuntu@ip-10-122-30-65:~$ pip install python-snappy==0.5.1
Collecting python-snappy==0.5.1
  Downloading https://files.pythonhosted.org/packages/4c/8f/09f0a11dbcaedf9b0758a37ab2bb77d6c34b9c29afc4d0440019c152d2af/python-snappy-0.5.1.tar.gz
Requirement already satisfied: cffi in /opt/python/pypy3.5/lib_pypy (from python-snappy==0.5.1) (1.11.5)
Building wheels for collected packages: python-snappy
  Running setup.py bdist_wheel for python-snappy ... done
  Stored in directory: /home/ubuntu/.cache/pip/wheels/b8/2a/3d/528db6296278d2660142b513e2923177e6b1ac19a71021d1c3
Successfully built python-snappy
Installing collected packages: python-snappy
Successfully installed python-snappy-0.5.1
(snappypypy35) ubuntu@ip-10-122-30-65:~$ pypy3
Python 3.5.3 (fdd60ed87e941677e8ea11acf9f1819466521bf2, Apr 27 2018, 15:39:57)
[PyPy 6.0.0 with GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> import snappy
>>>> 
(snappypypy35) ubuntu@ip-10-122-30-65:~$ pip uninstall python-snappy
Uninstalling python-snappy-0.5.1:
  Would remove:
    /home/ubuntu/snappypypy35/site-packages/python_snappy-0.5.1.dist-info/*
    /home/ubuntu/snappypypy35/site-packages/setup.py
    /home/ubuntu/snappypypy35/site-packages/snappy.py
    /home/ubuntu/snappypypy35/site-packages/snappy_cffi.py
    /home/ubuntu/snappypypy35/site-packages/test_snappy.py
    /home/ubuntu/snappypypy35/site-packages/test_snappy_cffi.py
Proceed (y/n)? y
  Successfully uninstalled python-snappy-0.5.1
(snappypypy35) ubuntu@ip-10-122-30-65:~$ pip install python-snappy
Collecting python-snappy
Requirement already satisfied: cffi>=1.0.0 in /opt/python/pypy3.5/lib_pypy (from python-snappy) (1.11.5)
Installing collected packages: python-snappy
Successfully installed python-snappy-0.5.2
(snappypypy35) ubuntu@ip-10-122-30-65:~$ pypy3
Python 3.5.3 (fdd60ed87e941677e8ea11acf9f1819466521bf2, Apr 27 2018, 15:39:57)
[PyPy 6.0.0 with GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> import snappy
Traceback (most recent call last):
  File "/home/ubuntu/snappypypy35/site-packages/snappy/snappy.py", line 47, in <module>
    from ._snappy import UncompressError, compress, decompress, \
ImportError: No module named 'snappy._snappy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/snappypypy35/site-packages/snappy/__init__.py", line 1, in <module>
    from .snappy import (
  File "/home/ubuntu/snappypypy35/site-packages/snappy/snappy.py", line 50, in <module>
    from .snappy_cffi import UncompressError, compress, decompress, \
  File "/home/ubuntu/snappypypy35/site-packages/snappy/snappy_cffi.py", line 1, in <module>
    from _snappy_cffi import ffi, lib
ImportError: No module named '_snappy_cffi'

pypy (based on Python 2.7) seems to work fine.

Hello,

I fail to run pip install python-snappy on my virtualenv with Python 3.4.4

I built snappy with brew install snappy and also when I do pip3 install python-snappy with my standard python 3.4.4 version (no venv activated) everything works fine.

My error is

(paul) ➜  python git:(master) ✗ CPPFLAGS="-I/usr/local/include -L/usr/local/lib" pip install python-snappy

Collecting python-snappy
  Using cached python-snappy-0.5.1.tar.gz
Building wheels for collected packages: python-snappy
  Running setup.py bdist_wheel for python-snappy ... error
  Complete output from command /Users/labs/venvs/paul/bin/python3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/pf/l5_q3mn94jb9wddqtv5h23th0000gn/T/pip-build-rcpw6591/python-snappy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/pf/l5_q3mn94jb9wddqtv5h23th0000gn/T/tmps8cfe0d6pip-wheel- --python-tag cp34:
  WARNING: '.' not a valid package name; please use only .-separated package names in setup.py
  running bdist_wheel
  running build
  running build_py
  package init file '__init__.py' not found (or not a regular file)
  creating build
  creating build/lib.macosx-10.6-intel-3.4
  copying setup.py -> build/lib.macosx-10.6-intel-3.4
  copying snappy.py -> build/lib.macosx-10.6-intel-3.4
  copying snappy_cffi.py -> build/lib.macosx-10.6-intel-3.4
  copying test_snappy.py -> build/lib.macosx-10.6-intel-3.4
  copying test_snappy_cffi.py -> build/lib.macosx-10.6-intel-3.4
  running build_ext
  building '_snappy' extension
  creating build/temp.macosx-10.6-intel-3.4
  /usr/local/bin/clang -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -g -I/usr/local/include -L/usr/local/lib -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c snappymodule.cc -o build/temp.macosx-10.6-intel-3.4/snappymodule.o
  clang-5.0: warning: argument unused during compilation: '-L/usr/local/lib' [-Wunused-command-line-argument]
  /usr/local/bin/clang -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -g -I/usr/local/include -L/usr/local/lib -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c crc32c.c -o build/temp.macosx-10.6-intel-3.4/crc32c.o
  clang-5.0: warning: argument unused during compilation: '-L/usr/local/lib' [-Wunused-command-line-argument]
  /usr/local/bin/clang++ -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g -I/usr/local/include -L/usr/local/lib build/temp.macosx-10.6-intel-3.4/snappymodule.o build/temp.macosx-10.6-intel-3.4/crc32c.o -lsnappy -o build/lib.macosx-10.6-intel-3.4/_snappy.so
  ld: warning: ignoring file /usr/local/lib/libsnappy.dylib, file was built for x86_64 which is not the architecture being linked (i386): /usr/local/lib/libsnappy.dylib
  ld: in '/usr/local/lib/libunwind.dylib', file was built for x86_64 which is not the architecture being linked (i386): /usr/local/lib/libunwind.dylib for architecture i386
  clang-5.0: error: linker command failed with exit code 1 (use -v to see invocation)
  error: command '/usr/local/bin/clang++' failed with exit status 1

  ----------------------------------------
  Failed building wheel for python-snappy
  Running setup.py clean for python-snappy
Failed to build python-snappy
Installing collected packages: python-snappy
  Running setup.py install for python-snappy ... error
    Complete output from command /Users/labs/venvs/paul/bin/python3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/pf/l5_q3mn94jb9wddqtv5h23th0000gn/T/pip-build-rcpw6591/python-snappy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/pf/l5_q3mn94jb9wddqtv5h23th0000gn/T/pip-nld38g3m-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/labs/venvs/paul/bin/../include/site/python3.4/python-snappy:
    WARNING: '.' not a valid package name; please use only .-separated package names in setup.py
    running install
    running build
    running build_py
    package init file '__init__.py' not found (or not a regular file)
    creating build
    creating build/lib.macosx-10.6-intel-3.4
    copying setup.py -> build/lib.macosx-10.6-intel-3.4
    copying snappy.py -> build/lib.macosx-10.6-intel-3.4
    copying snappy_cffi.py -> build/lib.macosx-10.6-intel-3.4
    copying test_snappy.py -> build/lib.macosx-10.6-intel-3.4
    copying test_snappy_cffi.py -> build/lib.macosx-10.6-intel-3.4
    running build_ext
    building '_snappy' extension
    creating build/temp.macosx-10.6-intel-3.4
    /usr/local/bin/clang -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -g -I/usr/local/include -L/usr/local/lib -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c snappymodule.cc -o build/temp.macosx-10.6-intel-3.4/snappymodule.o
    clang-5.0: warning: argument unused during compilation: '-L/usr/local/lib' [-Wunused-command-line-argument]
    /usr/local/bin/clang -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -g -I/usr/local/include -L/usr/local/lib -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c crc32c.c -o build/temp.macosx-10.6-intel-3.4/crc32c.o
    clang-5.0: warning: argument unused during compilation: '-L/usr/local/lib' [-Wunused-command-line-argument]
    /usr/local/bin/clang++ -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g -I/usr/local/include -L/usr/local/lib build/temp.macosx-10.6-intel-3.4/snappymodule.o build/temp.macosx-10.6-intel-3.4/crc32c.o -lsnappy -o build/lib.macosx-10.6-intel-3.4/_snappy.so
    ld: warning: ignoring file /usr/local/lib/libsnappy.dylib, file was built for x86_64 which is not the architecture being linked (i386): /usr/local/lib/libsnappy.dylib
    ld: in '/usr/local/lib/libunwind.dylib', file was built for x86_64 which is not the architecture being linked (i386): /usr/local/lib/libunwind.dylib for architecture i386
    clang-5.0: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command '/usr/local/bin/clang++' failed with exit status 1

    ----------------------------------------
Command "/Users/labs/venvs/paul/bin/python3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/pf/l5_q3mn94jb9wddqtv5h23th0000gn/T/pip-build-rcpw6591/python-snappy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/pf/l5_q3mn94jb9wddqtv5h23th0000gn/T/pip-nld38g3m-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/labs/venvs/paul/bin/../include/site/python3.4/python-snappy" failed with error code 1 in /private/var/folders/pf/l5_q3mn94jb9wddqtv5h23th0000gn/T/pip-build-rcpw6591/python-snappy/ 

I think my libsnappy was built for 64 bit only and I'm trying to use it with a 32-bit program but this goes beyond my understanding.
I tried brew install snappy --universal as I read some similar issues for pip installing other programs but it failed again.

I googled it and there's nothing about such a case.

Many thanks for reading my issue.

Similar to #69 , but it's "No module named snappy._snappy_cffi error" rather than "No module named '_snappy_cffi'"

I install the snappy python from

python setup.py build
python setup.py install

My python version is python 3.6.4 with anaconda 3 on centos 6.

When trying to use snappy inside AWS Lambda I am getting:
module initialization error: [Errno 2] No such file or directory: '/var/task/__pycache__/_cffi__x762f05ffx6bf5342b.c'

Hello,

I've downloaded the latest SVN version from snappy and successfully compiled it.
When trying to use python-snappy I get no compression. Worse, my string gets bigger:

Eg.:
text : 'hello world!' size 12
compressed : '??hello world!' size 14
uncompressed: 'hello world!' size 12

It passed all the test_snappy.py tests but it makes no sense.

Is there an explanation to this behavior?

It would be very useful if the source of snappy could be included in order not to depend
on an external library installation.
Any plan on this?

Thanks!
Paolo

#72 allows users to compress memoryviews via snappy.compress interface.
However, StreamCompressor.add_chunk raises TypeError when generating crc:

env36/lib/python3.6/site-packages/snappy/snappy.py:139: in add_chunk
    crc = _masked_crc32c(chunk)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

data = <memory at 0x7fb4e4cad648>

    def _masked_crc32c(data):
        # see the framing format specification
>       crc = _crc32c(data)
E       TypeError: argument 1 must be read-only bytes-like object, not memoryview

Looks like snappy__crc32c should be refactored to also support the buffer protocol.

I'm creating an AWS Lambda layer for snappy. Here is my code.

docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.7 \
    /bin/bash -c "yum install -y snappy-devel; pip install python-snappy -t pkg/python
mv pkg/python/snappy/_snappy.cpython-37m-x86_64-linux-gnu.so \
    pkg/python/snappy/_snappy_cffi.cpython-37m-x86_64-linux-gnu.so

The second line is necessary, otherwise I run into issues with the missing `._snappy_cffi module. I'm thinking this is a bug, but I'm not sure what the solution is. This is just a workaround.

When running python-snappy in CLI, in case of an error the traceback is written to stderr, eg:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.5/dist-packages/snappy/__main__.py", line 89, in <module>
    cmdline_main()
  File "/usr/local/lib/python3.5/dist-packages/snappy/__main__.py", line 85, in cmdline_main
    method(args.infile, args.outfile, **additional_args)
  File "/usr/local/lib/python3.5/dist-packages/snappy/hadoop_snappy.py", line 207, in stream_decompress
    start_chunk=start_chunk
  File "/usr/local/lib/python3.5/dist-packages/snappy/snappy.py", line 323, in stream_decompress
    decompressor.flush()  # makes sure the stream ended well
  File "/usr/local/lib/python3.5/dist-packages/snappy/hadoop_snappy.py", line 181, in flush
    raise UncompressError("chunk truncated")
snappy.UncompressError: chunk truncated

Can the traceback be ommited? Only the error message is required.

During the package review for fedora, the reviewer spotted that _snappy is in the global namespace. Since it is private it may be a good idea to move it in its own namespace. See https://bugzilla.redhat.com/show_bug.cgi?id=1244514#c10

This came up in a pykafka compression issue.

Parsely/pykafka#508

So byte arrays above a certain size causes a seg fault.

(pypy) ➜ cat s.py
from uuid import uuid4
import snappy

payload = b''.join([uuid4().bytes for i in range(10)])

c = snappy.compress(payload)
assert snappy.decompress(c) == payload
(pypy) ➜  python s.py
[1]    4587 segmentation fault (core dumped)  python s.py
(pypy) ➜  python
Python 2.7.10 (bbd45126bc69, Mar 18 2016, 21:35:08)
[PyPy 5.0.1 with GCC 4.8.4] on linux2

while

from uuid import uuid4
import snappy

payload = b''.join([uuid4().bytes for i in range(5)])

c = snappy.compress(payload)
assert snappy.decompress(c) == payload

passes

Running pip install python-snappy on osx 10.11.4 with snappy 1.1.3 installed via homebrew (https://github.com/Homebrew/homebrew-core/blob/83f27cf3bfdc1b545fe0c0b4d7318b981a94b123/Formula/snappy.rb) fails with:

➤ pip install python-snappy                                                                                                                                         ~
Collecting python-snappy
  Using cached python-snappy-0.5.tar.gz
Building wheels for collected packages: python-snappy
  Running setup.py bdist_wheel for python-snappy ... error
  Complete output from command /Users/x/t/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/9s/qssw6wq122bc0mxq_c1_mvs80000gn/T/pip-build-OUeKMW/python-snappy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /var/folders/9s/qssw6wq122bc0mxq_c1_mvs80000gn/T/tmpARscR6pip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.11-x86_64-2.7
  copying snappy.py -> build/lib.macosx-10.11-x86_64-2.7
  running build_ext
  building '_snappy' extension
  creating build/temp.macosx-10.11-x86_64-2.7
  clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c snappymodule.cc -o build/temp.macosx-10.11-x86_64-2.7/snappymodule.o
  snappymodule.cc:31:10: fatal error: 'snappy-c.h' file not found
  #include <snappy-c.h>
           ^
  1 error generated.
  error: command 'clang' failed with exit status 1

  ----------------------------------------
  Failed building wheel for python-snappy
  Running setup.py clean for python-snappy
Failed to build python-snappy
Installing collected packages: python-snappy
  Running setup.py install for python-snappy ... error
    Complete output from command /Users/x/t/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/9s/qssw6wq122bc0mxq_c1_mvs80000gn/T/pip-build-OUeKMW/python-snappy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/9s/qssw6wq122bc0mxq_c1_mvs80000gn/T/pip-0RkgyT-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/x/t/bin/../include/site/python2.7/python-snappy:
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.11-x86_64-2.7
    copying snappy.py -> build/lib.macosx-10.11-x86_64-2.7
    running build_ext
    building '_snappy' extension
    creating build/temp.macosx-10.11-x86_64-2.7
    clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c snappymodule.cc -o build/temp.macosx-10.11-x86_64-2.7/snappymodule.o
    snappymodule.cc:31:10: fatal error: 'snappy-c.h' file not found
    #include <snappy-c.h>
             ^
    1 error generated.
    error: command 'clang' failed with exit status 1

    ----------------------------------------
Command "/Users/x/t/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/9s/qssw6wq122bc0mxq_c1_mvs80000gn/T/pip-build-OUeKMW/python-snappy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/9s/qssw6wq122bc0mxq_c1_mvs80000gn/T/pip-0RkgyT-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/x/t/bin/../include/site/python2.7/python-snappy" failed with error code 1 in /private/var/folders/9s/qssw6wq122bc0mxq_c1_mvs80000gn/T/pip-build-OUeKMW/python-snappy/

A workaround is to tell pip to look in /usr/local when compiling: pip install --global-option=build_ext --global-option='-I/usr/local/include' --global-option='-L/usr/local/lib' python-snappy

Hi,

Can you please include a license file (both in github and in the pypi release)?

When I used snappy module to decompress a file it occurs the issue

snappy.UncompressError: stream missing snappy identifier

It would simplify installation if we weren't required to compile locally but could instead get the wheel via pip

Is there a standard container format we can use for holding snappy compressed blocks of data?

It wouldn't be hard to write a streaming compressor that reads lets say 4 MB of input at a time in a thread (or process) and shoves output into a file-like object that can be read from as it is being filled up. I started to play with this myself, but I don't know much about compression container formats and I don't want to 1) reinvent the wheel or 2) make something that no other snappy based tool can decompress ...

If I want to compress or uncompress only a view of my string/data, I don't see a way to do that without creating a copy. Do you think this can be added.

hi,
When I use compress function to process some large data, it raises "CompressError: Error ocurred while compressing string".
For example:

>>> import snappy
>>> r = snappy.compress('a'*10000000)
CompressError: Error ocurred while compressing string

By examining snappymodule.cc , I believe the error was caused by using of uninitialized real_size, in
https://github.com/andrix/python-snappy/blob/master/snappymodule.cc#L79

status = snappy_compress(input, input_size, compressed, &real_size);

The snappy_compress function in snappy-c.cc in snappy:

snappy_status snappy_compress(const char* input,
                              size_t input_length,
                              char* compressed,
                              size_t *compressed_length) {
  if (*compressed_length < snappy_max_compressed_length(input_length)) {
    return SNAPPY_BUFFER_TOO_SMALL;
  }
  snappy::RawCompress(input, input_length, compressed, compressed_length);
  return SNAPPY_OK;
}

Here *compressed_length is real_size which is not initialized yet. Thus the result of the if statement is undefined(on my machine, *compressed_length gives value like 468038 and fails data w/ larger size).

I've changed real_size to max_comp_size in snappymodule.cc and it works like a charm.

My machine: Python2.6, python-snappy-0.3.1 and snappy-1.0.3 on Linux X86_64.

B.R.

Is it possible to add the version attribute?

Installing the python-snappy wrappings for (Google's) snappy via:
$ pip install python-snappy
installs the python package as snappy. This conflicts with the existing Python package SnapPy which can be installed via:
$ pip install snappy
As a result, if SnapPy is already installed then many systems become confused. For example, dask can use (Google's) snappy and so looks tries to see whether import snappy works. If SnapPy has been installed then this will succeed but load the wrong package and as a result dask will crash.

I tried to install python-snappy on a virtualenv with python 2.7. It failed

` x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c snappy/snappymodule.cc -o build/temp.linux-x86_64-2.7/snappy/snappymodule.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
snappy/snappymodule.cc:31:22: fatal error: snappy-c.h: Aucun fichier ou dossier de ce type
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Failed building wheel for python-snappy
Running setup.py clean for python-snappy
Successfully built cyvcf2 peddy
Failed to build python-snappy
Installing collected packages: sqlalchemy, numpy, cyvcf2, peddy, geneimpacts, python-snappy
Running setup.py install for geneimpacts ... done
Running setup.py install for python-snappy ... error
Complete output from command /home/aurelien.beliard/.virtualenvs/varapp/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-1c10wm/python-snappy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-CzAHUl-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/aurelien.beliard/.virtualenvs/varapp/include/site/python2.7/python-snappy:
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'cffi_modules'
warnings.warn(msg)
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/snappy
copying snappy/init.py -> build/lib.linux-x86_64-2.7/snappy
copying snappy/snappy_cffi.py -> build/lib.linux-x86_64-2.7/snappy
copying snappy/snappy.py -> build/lib.linux-x86_64-2.7/snappy
copying snappy/snappy_cffi_builder.py -> build/lib.linux-x86_64-2.7/snappy
copying snappy/main.py -> build/lib.linux-x86_64-2.7/snappy
copying snappy/snappy_formats.py -> build/lib.linux-x86_64-2.7/snappy
copying snappy/hadoop_snappy.py -> build/lib.linux-x86_64-2.7/snappy
running build_ext
building 'snappy._snappy' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/snappy
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c snappy/snappymodule.cc -o build/temp.linux-x86_64-2.7/snappy/snappymodule.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
snappy/snappymodule.cc:31:22: fatal error: snappy-c.h: Aucun fichier ou dossier de ce type
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
`

I tried to reinstall libsnappy-dev to correct it, but i have another problem
I tried to install cffi but there already is a problem:

static struct module_state _state;
                              ^
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c snappy/crc32c.c -o build/temp.linux-x86_64-2.7/snappy/crc32c.o
  c++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/snappy/snappymodule.o build/temp.linux-x86_64-2.7/snappy/crc32c.o -lsnappy -o build/lib.linux-x86_64-2.7/snappy/_snappy.so
  c++: error: unrecognized command line option ‘-Wdate-time’
  c++: error: unrecognized command line option ‘-fstack-protector-strong’
  c++: error: unrecognized command line option ‘-Wdate-time’
  c++: error: unrecognized command line option ‘-fstack-protector-strong’
  error: command 'c++' failed with exit status 1

Hello,
python-snappy depends on Snappy, but how to install Snappy ? I downloaded the snappy-1.1.1.tar.gz file ans unzipped it, but I have no idea of the way to install it on my Mac. Could you help ?
Jean-Baptiste

I try to use snappy which use cffi with a non-root user that can't create the pycache/ dir.
I got :

STDERR: Traceback (most recent call last):
      File "/opt/graphite/pypy/bin/django-admin", line 11, in <module>
        sys.exit(execute_from_command_line())
      File "/opt/graphite/pypy/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
        utility.execute()
      File "/opt/graphite/pypy/site-packages/django/core/management/__init__.py", line 302, in execute
        settings.INSTALLED_APPS
      File "/opt/graphite/pypy/site-packages/django/conf/__init__.py", line 55, in __getattr__
        self._setup(name)
      File "/opt/graphite/pypy/site-packages/django/conf/__init__.py", line 43, in _setup
        self._wrapped = Settings(settings_module)
      File "/opt/graphite/pypy/site-packages/django/conf/__init__.py", line 99, in __init__
        mod = importlib.import_module(self.SETTINGS_MODULE)
      File "/opt/pypy/pypy-5.7.1-linux_x86_64-portable/lib-python/2.7/importlib/__init__.py", line 37, in import_module
        __import__(name)
      File "/opt/graphite/webapp/graphite/settings.py", line 160, in <module>
        from graphite.local_settings import *  # noqa
      File "/opt/graphite/webapp/graphite/local_settings.py", line 53, in <module>
        from graphite.local_settings_dynamic import *
      File "/opt/graphite/webapp/graphite/local_settings_dynamic.py", line 1, in <module>
        from biggraphite.plugins import graphite
      File "/opt/graphite/pypy/site-packages/biggraphite/plugins/graphite.py", line 30, in <module>
        from biggraphite import graphite_utils
      File "/opt/graphite/pypy/site-packages/biggraphite/graphite_utils.py", line 17, in <module>
        from biggraphite import utils as bg_utils
      File "/opt/graphite/pypy/site-packages/biggraphite/utils.py", line 22, in <module>
        from biggraphite.drivers import cassandra as bg_cassandra
      File "/opt/graphite/pypy/site-packages/biggraphite/drivers/cassandra.py", line 34, in <module>
        from cassandra import cluster as c_cluster
      File "/opt/graphite/pypy/site-packages/cassandra/cluster.py", line 46, in <module>
        from cassandra.connection import (ConnectionException, ConnectionShutdown,
      File "/opt/graphite/pypy/site-packages/cassandra/connection.py", line 80, in <module>
        import snappy
      File "/opt/graphite/pypy/site-packages/snappy.py", line 50, in <module>
        from snappy_cffi import UncompressError, compress, decompress, \
      File "/opt/graphite/pypy/site-packages/snappy_cffi.py", line 169, in <module>
        ''', libraries=["snappy"])
      File "/opt/pypy/pypy-5.7.1-linux_x86_64-portable/lib_pypy/cffi/api.py", line 438, in verify
        lib = self.verifier.load_library()
      File "/opt/pypy/pypy-5.7.1-linux_x86_64-portable/lib_pypy/cffi/verifier.py", line 113, in load_library
        self._write_source()
      File "/opt/pypy/pypy-5.7.1-linux_x86_64-portable/lib_pypy/cffi/verifier.py", line 202, in _write_source
        with open(self.sourcefilename, "w") as fp:
    IOError: [Errno 2] No such file or directory: '/opt/graphite/pypy/site-packages/__pycache__/_cffi__g762f05ffx6bf5342b.c'

As far as I understand, it calls a deprecated method of cffi (http://cffi.readthedocs.io/en/latest/whatsnew.html?#v1-0-0).
A workaround would be to create pycache/ directory at the installation as root.