rust-data-science / ulist Goto Github PK

The case-when-then-else-end statement is supported by all the SQL databases, and it would be good if ulist can support this method.

Although comparing floating numbers is not as simple as Integers due to the precision issues. It still makes sense to implement eq, ne, ge and le operators for the FloatList.

Actually, it does not makes sense to implement sum or mean methods for BooleanList because there is not addition or division methods for boolean type. We should write:

bool_list.astype('int').sum()
bool_list.astype('int').mean()

But people are quite comfortable to use sum or mean for BooleanList, so I have to compromise.

Implement apply method like this:

>>> import ulist as ul

>>> arr = ul.arange(3)
>>> arr.apply(lambda x: x < 1)
UltraFastList([True, False, False])

Sometimes we need to know the index of the maximum or minimum value in an array or list. If there are multiple indexes, just return the first one.

Get the ulist elements by multiple indexes, for example:

import ulist as ul

arr = ul.from_seq([1, 2, 3, 4, 5], dtype='int')
indexes = ul.IndexList([0, 2, 4])
arr[indexes]

[1, 3, 5]

The IndexList can slice the ulist array by its indexes. For example:

import ulist as ul
idx = IndexList([0, 2, 4])
arr = ul.from_seq([1, 2, 3, 5, 7, 11])
arr[idx]

[1, 3, 7]

Currently, I use wish list to write down my new ideas. It's a bit awkward to use issues to manage these ideas, and it's just like one person talking to oneself.

Try to reduce the numbers of in-place methods in ulist, because I personally think inplace method is not a good idea. For most of the case, we don't need to save memory so much. An 1d array/list should't crush your computer's memory. And in future, we can implement in-place mode for people who care about the memory that much.

Concat two ulist together by union_all method, which is equivalent to the SQL UNION ALL keyword.

Implement a select function which is equivalent to numpy.select. This would be the dependency of case-when-then-else-end implementation.
See: https://numpy.org/doc/stable/reference/generated/numpy.select.html

Partial sort the elements of NumericalList, for example:

import ulist as ul

arr = ul.from_seq([4, 2, 1, 3, 5, 0], dtype='int')
arr.partition(3)

[2, 1, 0, 3, 4, 5]

It is very common to implement types conversion for containers with different types. For example, pandas and numpy both implement astype method for Series and ndarray.

If a method can be implement by Python and almost has not overhead. Then we should consider move this method out of Rust codes. For example,

def mean(self):
    return self.sum() / self.size()

We only need to implement sum and size in rust and wrap them by Python. Then implement mean by "pure" Python codes.

Consider using the tricks in this issue PyO3/pyo3#696 to reduce duplicate code blocks in the lib.rs file.

Also needs to find a better way to handle dynamic type, instead of implementing 3 select functions in control_flow.rs for every possibility.
see: https://oswalt.dev/2021/06/polymorphism-in-rust/

Compare the main methods of ulist with numpy and generate the benchmarking report.

Need to write some examples more useful and attractive to users.

This method return the elements stored as dictionary keys and their counts are stored as dictionary values.
For example:

[1, 1, 2, 2, 3] => {1: 2, 2: 2, 3: 1}

Note: f32 doesn’t implement Eq because NaN is not equal to itself, so it would be impossible to retrieve a HashMap entry with NaN as the key. So cannot implement the counter method for it now.

It's very common to implement var method for int and bool array. Both Pandas and Numpy have implemented this. Just use existing methods in Ulist can do this.

Implement StringList to handle string type elements. Simply inherit the methods of List trait for the time being.

In Ulist project, both assertion and exception are used to make the program robust. But there is best practice for using assertion and exception we should follow.

See: https://stackoverflow.com/questions/944592/best-practice-for-using-assert
and : https://stackoverflow.com/questions/20076182/what-are-the-implications-of-using-assert-vs-raise-exception

Compute the median of IntegerList and FloatList.

Need to create a Github action to auto generate coverage report, and then create a new badge to show in the Readme.md file. Please refer to: https://github.com/nedbat/coveragepy/blob/master/.github/workflows/coverage.yml

Like SQL WHERE statement, implement a where method to filter the ulist by given condition. For example:

arr = ul.arange(5)
arr.where(lambda x: x > 1)
UltraFastList([2, 3, 4])