rust-ndarray / ndarray Goto Github PK

Two reasons:

• Blas offers this as axpy, so it can be optimized (note: axpy is for vectors, not strided matrix)
• It's a common case.

This operation is already _available efficiently_ using .zip_mut_with:

a1.zip_mut_with(&a2, |x, &y| *x += k * y)

The question is if this is so common and the zip_mut_with so ugly that it warrants its own method.

cc @vbarrielle

Maybe these are more convenient as returning non-references and having a method like .index_set(Axis, Ix) for the mutable case.

See http://docs.scipy.org/doc/numpy-1.10.1/user/basics.broadcasting.html

A more general form of broadcasting, it's like two-sided broadcasting, broadcast to a common shape that's not the same as either left or right shape.

In #898 - internal function co_broadcast, but available externally for arithmetic ops.

Make available as a public interface.

General dimensions, like numpy::dot.

Restricted to float types, for transparent blas support.

Need good low level documentation about the striding scheme to go with these.

Main win is syntax, array[[1, 2]] is pretty sweet.

The &[] and &mut [] in array views are actually dead weight.

They don't serve a purpose, since the raw pointer member is all that's needed. Carrying a slice is not possible for mutable views anyway (mutable views), since disjoint mutable views can't carry slices of the whole region (would alias) and neither can carry a restricted slice of just their view (not contig.).

Compilling ndarray stopped with error
.cargo/registry/src/github.com-0a35038f75765ae4/ndarray-0.2.0/src/arrayformat.rs:46:23: 46:34 error: no method named alternate found for type &mut core::fmt::Formatter<'_> in the current scope
.cargo/registry/src/github.com-0a35038f75765ae4/ndarray-0.2.0/src/arrayformat.rs:46 if !f.alternate() {

Ubuntu 15.04
rust 1.4

We want to allow a (for example) 2D-array to serialize and deserialize equally, regardless of using Dimension type (Ix, Ix) or Vec<Ix>.

Part of #10

We probably want to mimic numpy here by using a bitfield for keeping track of memory layout. c order, f order (both possible at the same time), as well as neither order. Maybe some blas-related property is relevant down the line. May even be a Cell field to update it on the fly? (This makes the type !Sync which we don't want..)

In some cases it can be nice to be able to specify a specific stride when constructing a new array (for instance to wrap over foreign data).

Like .iter(), but visits the array elements in "arbitrary" order. Will use memory order or whatever order is most efficient.

Investigate using fixed size arrays for .shape() and .strides()

What are your thoughts on implementing something similar to http://dask.pydata.org/en/latest/ on top of ndarrays? I suspect parallelized computations on submatrices should be pretty natural to do in the Rust framework, and it seems you've already created sub-array view functions. Do you agree?

(Community Edits below)

Actionable sub issues:

• Send/Sync splittable array views are already present
• Implement Rayon parallel iterator traits for Axis Iter #248 #252
• Implement Rayon parallel iterator traits for element iterators / for Array itself #252
• alternative methods of collecting an axis_iter to ndarray matrix #249
• Parallel Iter for AxisChunksIter
• Parallel support for Array::map_inplace #288
• Parallel support for Array::map -> Array
• Parallel lock step function application (Zip) #288

Just brainstorming

A "builder pattern" approach may be viable here to allow n-ary zipping operations. We could even allow masking (?) which is a very valuable operation.

zip_mut(&mut a).zip(&b).zip_mut(&mut c).apply(|x, y, z| *x = *y + *z);

// This is intended to do the same as this would in numpy
// mask = c > d;
// a[mask] = b[mask]
zip_mut(&mut a).zip(&b).mask(&c, &d, <_>::gt).apply(|aelt, belt| *aelt = *belt);

Once again, you happen to have just what I need 😄

Any chance this will find its way onto crates.io any time soon?

Maybe we can merge these errors. If not, we can simplify ShapeError so that it doesn't allocate?

Would it make sense to have a.le(b) (also a.leq(b), a.ge(b), a.geq(b)) return boolean arrays? (where a, b : PartialOrd and properly broadcastable).

See rust-lang/rust/issues/29717

Use to select impl for unrolled dot product and scalar sum.

We need to be stricter (at least in the docs) and hide trait methods. These traits are not intended to be used as more than trait bounds.

This is a bit tricky, since it may require some kind of scalar trait to be able to have both a + b (arrays) and a + x (x is a scalar) both available due to trait coherence.

I'm trying to write a simple mean function that calculates the mean of an array for a given axis. Looking at sum I cam up with the following, but now I can't figure out how to divide the sum by the number of elements.
If I try normal divison I get: error: the trait 'core::ops::Div<usize>' is not implemented for the type 'A'. If I try the idiv version it tells me error: mismatched types: expected 'A', found 'usize' (expected type parameter, found usize) [E0308]

Do you have any hints?

trait Stats<A, S, D>
        where A: Clone + Add<Output = A> + Div<Output = A>,
              D: RemoveAxis
{
    fn mean(&self, Axis) -> OwnedArray<A, <D as RemoveAxis>::Smaller>;
}

impl<A, S, D> Stats<A, S, D> for ArrayBase<S, D>
    where A: Clone + Add<Output = A> + Div<Output = A> + Div,
          S: Data<Elem = A>,
          D: Dimension + RemoveAxis
{
    fn mean(&self, axis: Axis) -> OwnedArray<A, <D as RemoveAxis>::Smaller> {
        let n = self.shape()[axis.axis()];
        println!("{:?}", self.shape());
        let mut res = self.subview(axis, 0).to_owned();
        for i in 1..n {
            let view = self.subview(axis, i);
            res.iadd(&view);
        }
        // res.idiv(&arr0::<A>(n));
        res / n
    }
}

In the example at the bottom, addition of two arrays yields a new one, but also mutates one of the inputs. This is because the add() function in the Add trait is defined in terms of the in-place iadd() (from the macro impl_binary_op macro):

impl<'a, A, S, S2, D, E> $trt<&'a ArrayBase<S2, E>> for ArrayBase<S, D>
    where A: Clone + $trt<A, Output=A>,
          S: DataMut<Elem=A>,
          S2: Data<Elem=A>,
          D: Dimension,
          E: Dimension,
{
    type Output = ArrayBase<S, D>;
    fn $mth (mut self, rhs: &ArrayBase<S2, E>) -> ArrayBase<S, D>
    {
        self.$imth(rhs);
        self
    }
}

This is fine, if self is an OwnedArray or an Array because they are consumed anyways, but if instead an ArrayViewMut is passed in, the underlying data changes. Is this intended? I think one shouldn't expect the array to get changed when addition is performed on a view.

extern crate ndarray;

use ndarray::{Array,OwnedArray};

fn main() {
    let r = 0..4;
    let v = r.map(|x| x as f64).collect::<Vec<_>>();

    let mut a = OwnedArray::from_vec(v.clone()).into_shape((2,2)).unwrap();
    let b = OwnedArray::from_vec(v.clone()).into_shape((2,2)).unwrap();

    let bv = b.view();

    {
        let x = a.view_mut() + bv;
        println!("{:?}", x);
    }

    println!("{:?}", a);
}

Where we have by-value self it may make sense.

We could have strongly typed Axis arguments (instead of current usize).

It even allows mixing dynamically bounds checked axis numbers and statically checked ones:

.subview(Axis(0), 1) vs. .subview(Axis0, 1) would both be possible at the same time.

This would allow using a.dim()[Axis(0)], which is not that much of a win. However, when implementing a method that takes an Axis as an argument, this could be a real win:
a.dim().slice()[axis.axis()] vs a.dim()[axis].

What do you think?

Building version 0.4.0-alpha.1 with the dependency ndarray = "~0.4.0" in Cargo.toml gives the following error:

% cargo build --verbose
unable to get packages from source

Caused by:
  failed to unpack package `ndarray v0.4.0-alpha.1`

Caused by:
  No such file or directory (os error 2)
zsh: exit 101   cargo build --verbose

Apparently this was an issue with the recent cargo build and is now fixed. According to this comment, crates for which this problem still occurs should be reuploaded with a recent cargo version.

You put ndarray onto crates.io just a few days before this error report was posted, so I think this might be related.

Consider this case:

If you're interested, a tricky question remains, imagine a 4 x 4 matrix in row major order. The columns of the matrix are technically both F-order and C-order since they are one dimensional (but with a leading dimension of 4).

Anyway, adjacent columns do not overlap, but their memory interleaves very closely. Is this compatible with blas? 😄

This will be needed. Iterator perf is abysmal with u32.

The idea is to add a trait for just f32, f64 that allows access to all relevant ndarray API, as long as the element has that trait.

We now have Any and ScalarOperand involved in various bounds for technical reasons, so the trait would for example include those.

• mutate in place stepping for the iterator
• Pass index as slice
• allow indexing with regular arrays (#225)
• No allocation for small case #285 #301
• .zip_mut_with improvement reduces binary operation custom stride case overhead for ixdyn #292
• Fancier slicing, selecting subviews #215

Yesterday I was window shopping for new computer languages and I found rust which looks awesome if not yet mainstream. I like array-based approaches to certain problems, so I checked for ndarray support and found this github library. Nice!

Anyway, I noticed that this crate uses u32 for indexing.
http://bluss.github.io/rust-ndarray/doc/ndarray/struct.Array.html
In numpy it is increasingly common for users to have large arrays that cannot be indexed by such small integer types. I personally don't have enough RAM for this, but this could change in the future. Or you might want the same interface for arrays on disk as in RAM, or the same interface for sparse arrays that could more easily hold more entries. But of course using unnecessarily large integer types for indexing slows the code and increases the memory usage...

99% of numerical support exists to support floating-point f32 and f64.

We can make arithmetic and numerics less generic (stricter trait bounds).

As it is now, sum() is for example implemented for all elements where A: Clone + Add<A, Target=A>.
Arithmetic operations like Sub are implemented with similar Clone + Sub bounds.

We would benefit some if we could make these implementations stricter. For example using A: Copy allows us to use uninitialized memory efficiently (important for whenever you allocate a new array for the result).

See https://github.com/mikkyang/rust-blas

doesn't look impossible.

Using slices, I want to set the columns or rows of the 2×2 matrix (i.e. Array) a to the vectors u or v, respectively. This works when slicing along the first axis (getting row-slices), but not along the second:

#[macro_use(s)]
extern crate ndarray;

use ndarray::OwnedArray;

fn main() {
    let u: Vec<f64> = vec![1.,2.];
    let v: Vec<f64> = vec![3.,4.];

    let mut a = OwnedArray::from_elem((2,2), 0f64);

    // Works
    a.slice_mut(s![0..1, ..]).assign(&OwnedArray::from_vec(u.clone()));
    a.slice_mut(s![1..2, ..]).assign(&OwnedArray::from_vec(v.clone()));

    // Could not broadcast array from shape: [2] to: [2, 1]
    a.slice_mut(s![.., 0..1]).assign(&OwnedArray::from_vec(u.clone()));
    a.slice_mut(s![.., 1..2]).assign(&OwnedArray::from_vec(v.clone()));
}

I suppose the problem is that ndarray can broadcast from [2] to [1,2], but not to the reverse.

f-contiguous can be reshaped into other f-order arrays.

I was trying to make a crate depend on the version 0.4.0-alpha.1 that's on crates.io, but if I specify the dependency as ndarray = "0.4" I get the cargo error unable to get packages from source, and if I try to put the exact version number cargo errs out telling me it can't parse the cargo.toml.

However this works fine when specifying the dependency via git.

So maybe alpha versions should not be uploaded to crates.io as it seems they can"t be used from there.

We need user feedback on scalar ops.

This is more of an open question than an issue -- do you plan doing anything about the structured types? Kind of like numpy handles recarrays (http://docs.scipy.org/doc/numpy-1.10.1/user/basics.rec.html). I was thinking of trying to use rust-ndarray as the data backend for the hdf5 library, but I wonder what to do with the structured non-scalar types...