Comments (4)
mrocklin
commented on August 14, 2024
You will need to install the snappy compressor before using it. The fastparquet library does not include snappy by default.
I don't see any conda packages for snappy for Windows on Anaconda.org. You may have to install the Snappy compresssor and the python-snappy library yourself if you want to use it.
from fastparquet.
rsheftel
commented on August 14, 2024
The windows binary is available here: http://www.lfd.uci.edu/~gohlke/pythonlibs/#python-snappy
I had that installed, but no success. I also tried on Linux after installing snappy using conda, again I got the same error message. Do I need to register snappy with fastparquet so that it knows about its existence on my machine?
from fastparquet.
mrocklin
commented on August 14, 2024
It looks like if snappy is importable from python then it will be added.
See
https://github.com/dask/fastparquet/blob/master/fastparquet/compression.py#L41-L46
You might want to ensure that you are able to import snappy
You might need to install not just the snappy system library, but also the
python-snappy library.
…On Sun, Feb 26, 2017 at 8:39 PM, rsheftel ***@***.***> wrote:
The windows binary is available here: http://www.lfd.uci.edu/~
gohlke/pythonlibs/#python-snappy
I had that installed, but no success. I also tried on Linux after
installing snappy using conda, again I got the same error message. Do I
need to register snappy with fastparquet so that it knows about its
existence on my machine?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#91 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AASszCxxljxPnXsRB9XTRvUdSOIyR50Kks5rgilSgaJpZM4MMJs1>
.
from fastparquet.
martindurant
commented on August 14, 2024
It seems that python-snappy is not built for windows, but maybe andrix/python-snappy#30 allow it to work.
The underlying library is listed for windows, but not built, see conda-forge/snappy-feedstock#4 ; but it is available on other channels, such as the standard Anaconda one.
from fastparquet.
Related Issues (20)
- sdist for 2023.4.0 is missing on PyPI HOT 2
- Can't read parquet files created by pyspark using ParquetFile.iter_row_groups() HOT 2
- TypeError: 'NoneType' object is not iterable HOT 2
- write to buffer support HOT 2
- AttributeError: 'ParquetFile' object has no attribute '_columns_dtype' when reading files without pandas metadata HOT 1
- Incorrect value returned for overflow timestamps in micros format for V1 footers HOT 4
- row_filter=True fails if parquet file wasn't made by fastparquet HOT 1
- ValueError: Seek before start of file using custom open_with function or S3 file object HOT 3
- pandas nullable opt-out not working as intented HOT 3
- API Documentation Page Is Empty HOT 1
- Potential Parquet File Metadata Corruption After Process Timeout HOT 7
- Fastparquet not outputting DATE logical type while using to_parquet of pandas HOT 4
- fastparquet encoding issue. HOT 20
- BUG: reading boolean column with RLE encoding gives wrong values HOT 4
- fastparquet cannot read a categorical column that contains NaNs only HOT 2
- to_pandas(): cramjam.DecompressionError: snappy: output buffer (size = 262144) is smaller than required (size = 1048576) HOT 1
- BUG: dataframe.empty with non-nano pd.DatetimeTZDtype HOT 2
- a python-3.12 windows wheel HOT 13
- Some `fastparquet`-related tests are failing on Python 3.10 HOT 10
- Regression due to `_from_sequence` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fastparquet.