Git Product home page Git Product logo

flupy's Introduction

flupy

Tests Codestyle Black

Python version PyPI version License Download count


Documentation: https://flupy.readthedocs.io/en/latest/

Source Code: https://github.com/olirice/flupy


Overview

Flupy implements a fluent interface for operating on python iterables. All flupy methods return generators and are evaluated lazily. This allows expressions to transform arbitrary size data in extremely limited memory.

You can think of flupy as a light weight, 0 dependency, pure python alternative to the excellent Apache Spark project.

Setup

Requirements

  • Python 3.6+

Installation

Install flupy with pip:

$ pip install flupy

Library

from itertools import count
from flupy import flu

# Processing an infinite sequence in constant memory
pipeline = (
    flu(count())
    .map(lambda x: x**2)
    .filter(lambda x: x % 517 == 0)
    .chunk(5)
    .take(3)
)

for item in pipeline:
  print(item)

# Returns:
# [0, 267289, 1069156, 2405601, 4276624]
# [6682225, 9622404, 13097161, 17106496, 21650409]
# [26728900, 32341969, 38489616, 45171841, 52388644]

CLI

The flupy command line interface brings the same syntax for lazy piplines to your shell. Inputs to the flu command are auto-populated into a Fluent context named _.

$ flu -h
usage: flu [-h] [-f FILE] [-i [IMPORT [IMPORT ...]]] command

flupy: a fluent interface for python

positional arguments:
  command               flupy command to execute on input

optional arguments:
  -h, --help            show this help message and exit
  -f FILE, --file FILE  path to input file
  -i [IMPORT [IMPORT ...]], --import [IMPORT [IMPORT ...]]
                        modules to import
                        Syntax: <module>:<object>:<alias>
                        Examples:
                                'import os' = '-i os'
                                'import os as op_sys' = '-i os::op_sys'
                                'from os import environ' = '-i os:environ'
                                'from os import environ as env' = '-i os:environ:env'

flupy's People

Contributors

jasiek avatar olirice avatar scottwedge avatar thejaminator avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

flupy's Issues

Change return annotation to `Fluent`?

screen shot 2018-12-15 at 9 38 10 am

This code runs fine, but pylint thinks I am working with an arbitrary generator, not a Fluent. Is there a return annotation somewhere that can be changed to Fluent?

P.S. This is a nice project.

Discussion regarding Fluent.collect return type

Previously in in this MR

@olirice mentioned that

Any chance you'd be willing to take a look at Fluent.collect? it currently returns a Collection[T] but I'd love to have it pick up the container type from the container_type argument and return List[T] or Set[T] depending on the callable that gets passed. So far I haven't been able to figure that out.

I messed around, and i think unfortunately its not totally possible without support for higher-kind types.

See this example here and here. Both would require users to install a custom mypy plugin. Which probably goes against the philosophy of this being a light weight library :)
Guido did reply to the issue created on mypy so there is some hope of it coming one day to standard myppy

We could instead support some standard library types with the overloads operator, such as Tuple, List, Set. That should be nice enough for most people.

But we will lose the type information of the Collection type for the other non-defined types

[New Feature] Folding/Reduce

Hi @olirice very cool project. I've been looking for something like this for some time! Transforming collections in this way wasn't a pleasure for me at all

sum(map(lambda, filter(lambda, list)))

I would be glad to see one more function that would be useful - foldLeft or foldRight or reduce as in scala.

Ex.

from flupy import flu
class Collector:
    __init__(self):
        self.nums = []

collector = flu([1,2,3]). \
                 filter(lambda x: x>1). \
                 foldLeft(Collector(), lambda x, acc: acc.nums.append(x))  

assert collector.nums == [2,3]

Btw I think that there is a small error in the documentation https://flupy.readthedocs.io/en/latest/api.html#flupy.Fluent.group_by - Shouldn't there be 2 and 3 instead of 4 and 9?

`flupy` CLI does not work on Windows

Ran into this issue while in the process of adding flupy (and alembic-utils) to conda-forge here: conda-forge/staged-recipes#26023.

Seems to be due to this bit here:

from signal import SIG_DFL, SIGPIPE, signal

signal.SIGPIPE is not available on Windows.

Traceback:

>flu --help 
Traceback (most recent call last):
  File "C:\bld\flupy_1712887240528\_test_env\Scripts\flu-script.py", line 5, in <module>
    from flupy.cli.cli import main
  File "C:\bld\flupy_1712887240528\_test_env\Lib\site-packages\flupy\cli\cli.py", line 4, in <module>
    from signal import SIG_DFL, SIGPIPE, signal
ImportError: cannot import name 'SIGPIPE' from 'signal' (C:\bld\flupy_1712887240528\_test_env\Lib\signal.py). Did you mean: 'SIGFPE'?

Saying flupy "brings lazy piplines to your shell" is comical.

You know, because lazy pipelines provide the fundamental paradigm around which shell programming was designed, decades before Python--let alone flupy--existed.

Not a bit deal, of course. I just had to chuckle when I read that line. ๐Ÿคทโ€โ™‚๏ธ

No support for Python 3.10.

pip install flupy in Python 3.10 causes output like this below. Is this as simple as adding 3.10 to setup.py, or is there some nasty bug which has prevented you from doing that?

(venv) PS ...> pip install flupy
Collecting flupy
Using cached flupy-1.1.8.tar.gz (12 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
ERROR: Command errored out with exit status 1:
command: ...\python.exe' '...\venv\lib\site-packages\pip_vendor\pep517\in_process_in_process.py' get_requires_for_build_wheel '...\Temp\tmpaqtyz1ok'
cwd: ...\Temp\pip-install-opd7zhl6\flupy_7a930ec7e41d487391822b70fd7bef1e
Complete output (1 lines):
Python 3.6+ is required.

WARNING: Discarding https://files.pythonhosted.org/packages/b9/28/34591f80faa91a517650a1537f6faca99308dafd49de5e4edf400cadc777/flupy-1.1.8.tar.gz#sha256=6be9b46f737053d4058414477220dc35339fc71a2a7ea53a6498b5add4642e8f (from https://pypi.org/simple/flupy/). Command errored out with exit status 1: '...\temp\venv\Scripts\python.exe' 'C:\Users\magnus.lycka\work\temp\venv\lib\site-packages\pip_vendor\pep517\in_process_in_process.py' get_requires_for_build_wheel '...\Temp\tmpaqtyz1ok' Check the logs for full command output.

`to_list` not in 1.1.9

__version__ = "1.1.9", docs say there's to_list and to_set, repository source has both methods, however pypi package doesn't have them.

"foldleft" support?

I think foldleft summarizing method could be added, because reduce requires a function with two same-type parameters. foldleft/foldright method is more generalizing. The behavior of this API likes what reduce does in functools module.

## return ["1", "2", "3", "4"]
flu([1, 2, 3, 4])\
 .foldleft(lambda x, y: x + str(y), '')

## same as `functools.reduce`
reduce(lambda x, y: x + str(y), [1, 2, 3, 4], '')

filter does not work with type guards

Summary

python 3.10 introduced type guards similar to that of typescript. however, flupy does not seem to support this feature.

image

from typing import Any, TypedDict, TypeGuard

from flupy import flu


class Person(TypedDict):
    name: str
    age: int


def is_person(val: Any) -> TypeGuard[Person]:
    try:
        return (
            isinstance(val, dict)
            and isinstance(val["name"], str)
            and isinstance(val["age"], int)
        )
    except KeyError:
        return False


def get_age(val: Person):
    return f"Age: {val['age']}"


result = (
    flu([Person(name="Alice", age=20), 3, "afds", {"name": "Eve"}])
    .filter(is_person)
    .map(get_age)
    .collect()
)

Python 3.5+?

Amazing project :-)

Why it can't be python 3.5? I could give it a try in the following weeks in some py3.5 projects I have, just to provide some use cases.

TypeError: '<' not supported between instances of 'str' and 'int'

When toying around with your library and testing some snippets of your API reference I noticed that the first example of group_by results in an error when comparing str with int:

>>> from flupy import flu
>>> flu.group_by([1, 'a', 'a', 1, 1]).collect()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bjoosten/.pyenv/versions/3.6.3/lib/python3.6/site-packages/flupy/fluent.py", line 37, in wrapper
    return func(*args, **kwargs)
  File "/Users/bjoosten/.pyenv/versions/3.6.3/lib/python3.6/site-packages/flupy/fluent.py", line 235, in group_by
    gen = self.sort(key) if sort else self
  File "/Users/bjoosten/.pyenv/versions/3.6.3/lib/python3.6/site-packages/flupy/fluent.py", line 37, in wrapper
    return func(*args, **kwargs)
  File "/Users/bjoosten/.pyenv/versions/3.6.3/lib/python3.6/site-packages/flupy/fluent.py", line 204, in sort
    return Fluent(sorted(self, key=key, reverse=reverse))
TypeError: '<' not supported between instances of 'str' and 'int'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.