The following code runs in 29.3 s on GPU and in 30.7 s on CPU, profile.

import t3f
import tensorflow as tf
import numpy as np

m = 10 * np.ones(15)
A = t3f.random_matrix((m, m), tt_rank=30)
b = t3f.random_matrix((m, None), tt_rank=30)
matvec = t3f.round(t3f.matmul(A, b), max_tt_rank=20)

with tf.Session() as sess:
    %timeit matvec.eval()

A few places where dtype is hardcoded to be float32
https://github.com/Bihaqo/t3f/blob/develop/t3f/ops.py#L433
https://github.com/Bihaqo/t3f/blob/develop/t3f/ops.py#L1151
https://github.com/Bihaqo/t3f/blob/develop/t3f/initializers.py#L90
https://github.com/Bihaqo/t3f/blob/develop/t3f/initializers.py#L181
https://github.com/Bihaqo/t3f/blob/develop/t3f/ops.py#L1199
https://github.com/Bihaqo/t3f/blob/develop/t3f/initializers.py#L112
https://github.com/Bihaqo/t3f/blob/develop/t3f/initializers.py#L112

Rewrite readme to avoid non-existing tt_init_op = t3f.initialize_from_tensor(tt_W_1, W_1)

Printing fails:
TypeError: %d format: a number is required, not NoneType

Should be something like
A ? element batch of ...

Check if using tf.while_loop instead of for loop in loops w.r.t. the TT-cores is faster and reasonably maintainable.

Test TensorNet, both example in the readme and some experiment from the paper.

I have the following problem that I don't know how to get around... would really appreciate any advice.

I want to form model inputs into TT cores and use them to instantiate a TensorTrain (I can't do to_tt_tensor because the number of inputs is too large -- the full tensor would be huge)

However, since TensorTrain does tt_cores = list(tt_cores), I get the (Tensorflow) error

TypeError: `Tensor` objects are not iterable when eager execution is not enabled. To iterate over this tensor use `tf.map_fn`.

(tf.map_fn doesn't solve the problem, of course). Eager execution doesn't work at present:

import tensorflow as tf
import tensorflow.contrib.eager as tfe
import t3f

tfe.enable_eager_execution()

shape = (3, 4, 4, 5, 7, 5)
initialization = t3f.random_tensor(shape, tt_rank=5)
estimated = t3f.get_variable('estimated', initializer=initialization)

yielding

ValueError: ValueError: Variable estimated does not exist, or was not created with t3f.get_tt_variable(). Did you mean to set reuse=None in VarScope?

It looks like t3f.ops.multiply doesn't support batches. It would be nice if it could also broadcast if we need to multiply a tt-batch by a single tt-matrix.

It works for TensorTrainBatch, but not for TensorTrain because of sum([batch_size]) and Nones

We need functions lazy_shape, lazy_raw_shape, and lazy_tt_ranks that return static data if available, and dynamic otherwise.

Project(something tf.float64, something tf.float64) doesn't work because it inits rhs and lhs with float32 ones

The following code fails
a = t3f.random_tensor((3, 3, 3), tt_rank=2)
b = t3f.random_tensor_batch((3, 3, 3), tt_rank=2, batch_size=5)
t3f.multiply(a, b)
with error
ValueError: The arguments should have the same shape.,
but should be possible via broadcasting

t3f.multiply(b, b) also produces error

#15 (comment)

With @test_util.run_all_in_graph_and_eager_modes

Add a simple SGD tensor completion example

add test

T3f on CPU scales worse than lineary with e.g. multipliying a batch of TT-matrices by a TT-matrix. This is weird.

Also, matrix-by-matrix multiplication is slower than that of TTPY.

See profile_cpu_vs_gpu branch

We can implement round of element-wise product cheaper than t3f.round(t3f.multiply(a, b)).

If anybody needs it, I'll implement.

I would like to contract all but a few indices in a TT. At the moment, I am doing this by defining TT matrices of shape ((1,1,1,..., n_ext), (m_1, ..., m_N)), and then contracting the m indices (with t3f.matmul()) with some other TT matrix.

The problem is that I'm interested in large N, so that I end up getting:

2018-03-08 14:55:41.571939: F tensorflow/core/framework/tensor_shape.cc:243] Check failed: ndims_byte() < MaxDimensions() (unsigned char value 254 vs. 254)Too many dimensions in tensor

It would be nice to avoid this by allowing a TT to have both 3d and 4d cores. t3f.full() could still give a matrix. t3f.matmul() would require the 4d cores to be in the same locations.

For similar reasons, it would be good to be able to have genuine matrix-vector multiplication. At the moment I do this with a shape = ((m_1, .... m_N), (1,1,....)) TT matrix, also contributing to the "too many dimensions" issue.

Right now Frobenious norm has a flag "differentiable" which chooses between fast and stable QR version and differentiable naive version.

It seems to be easy to manually implement backward pass for the QR version: we need to orthogonalize the tensor in both directions and then for differentiating w.r.t. k-th core we can represent the norm as:
||A||^2_F = ||U_1 U_2 ... U_k-1 (L_k-1 G_k R_k) U_k+1 ... ||^2 = ||L_k-1 G_k R_k||^2

where L_k-1 and R_k doesn't depend on G_k.

If anyone lacks a stable and differentiable Frobenius norm, I'll implement.

The following code
projected_grad = t3f.riemannian.project(grad_tt, weight_tt)
where grad_tt and weight_tt are TensorTrain objects gives the following error
ValueError: batch size is not available for a TensorTrain object.

However, the same piece of code with t3f.riemannian.project_sum(grad_tt, weight_tt) instead works

Hello, I use Tensorflow not keras. Unfortunately, when I replace fullconnected layer with TTDense ,I get the message :
Traceback (most recent call last): File "mnist-tflayers.py", line 140, in <module> launch_train_with_config(config, SimpleTrainer()) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/train/interface.py", line 91, in launch_train_with_config extra_callbacks=config.extra_callbacks) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/train/base.py", line 331, in train_with_defaults steps_per_epoch, starting_epoch, max_epoch) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/train/base.py", line 301, in train self.setup_callbacks(callbacks, monitors) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/utils/argtools.py", line 182, in wrapper return func(*args, **kwargs) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/train/base.py", line 211, in setup_callbacks self._callbacks.setup_graph(weakref.proxy(self)) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/callbacks/base.py", line 52, in setup_graph self._setup_graph() File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/callbacks/group.py", line 66, in _setup_graph cb.setup_graph(self.trainer) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/callbacks/base.py", line 52, in setup_graph self._setup_graph() File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/callbacks/inference_runner.py", line 142, in _setup_graph self._input_source, self.trainer.tower_func) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/graph_builder/predict.py", line 49, in build return tower_fn(*inputs) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/tfutils/tower.py", line 207, in __call__ output = self._tower_fn(*args) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/tensorpack/graph_builder/model_desc.py", line 234, in _build_graph_get_cost ret = self.build_graph(*inputs) File "mnist-tflayers.py", line 54, in build_graph logits = TTDense(row_dims=[512, 1, 1, 1], column_dims=[1, 2, 5, 1], tt_rank=16)(l) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/keras/engine/topology.py", line 592, in __call__ self.build(input_shapes[0]) File "/home/wc/ResNet/ok/utils/tt_dense.py", line 60, in build self.W = t3f.get_variable(name, initializer=initializer) File "/home/wc/tfpy27/local/lib/python2.7/site-packages/t3f/variables.py", line 73, in get_variable 'set reuse=None in VarScope?' % name) ValueError: ValueError: Variable tt_dense_matrix_1 does not exist, or was not created with t3f.get_tt_variable(). Did you mean to set reuse=None in VarScope?
what I should do to correct code ? My implement is here :link
thanks a lot.

Right now all initializers return TT objects and we can set the variable with them. TF likes when initializers are functions which can generate tensor on demand (like tf.contrib.layers.xavier_initializer), this way we may avoid potential problems with things like tf.while_loop

Change the order of lhs and rhs einsums to
(1, 2, 0, 3), 'aikd,sabc,bije,scjkf->sdef'
and
(1, 2, 0, 3), 'cikf,sdef,bije,sajkd->sabc'

and change the complexity to
O(d R n r2 ry s + d R n r ry2 s + d R2 n2 r ry s)
or
O(d R2 n2 r ry s) given that both r and ry are < Rn

move to t3f.layers, make a version without keras dependencies, rename rows, cols => inputs, outputs

https://github.com/Bihaqo/t3f/blob/develop/variables.py#L53
Should probably be if reuse:

It seems that TF is caching intermediate computation which makes t3f appear very fast in profiling. Fix!

Zeros, ones, eye, Laplace...

The following code doesn't work

a = t3f.random_tensor_batch((2, 3), batch_size=3)
n = tf.placeholder(tf.int32)
sess = tf.InteractiveSession()
sess.run(t3f.full(a[n:3]), {n: 1})

The following code produces an error
x = t3f.random_tensor_batch((3, 3, 3), tt_rank=2)
t3f.frobenius_norm_squared(x, differentiable=True)

In #132 I create name scopes for ops. It may also make sense to name the results of the ops, e.g. a = t3f.matmul(...) => a.name = 't3f_matmul:0'

t3f.flat_inner(x_tt, w) when x_tt is a batch of unknown size and w is a single object returns tensor of unknown number of dimensions

Hi all,
I am probably doing something wrong but a simple testing of creation of a zero TT matrix with the following code returns an error probably due to an integer overflow

`import t3f

b = [2, 40, 32, 20, 12, 8, 4, 24, 8, 8, 8, 24, 4, 24, 6, 20, 6, 6, 4, 80, 20, 72, 6, 32, 6]
hshape = [b,b]
X = t3f.initializers.matrix_zeros(hshape)
print(X)`

Running the code gives: the error

File "", line 1, in
File "/opt/anaconda2/lib/python2.7/site-packages/t3f/tensor_train.py", line 103, in str
shape = self.get_shape()
File "/opt/anaconda2/lib/python2.7/site-packages/t3f/tensor_train_base.py", line 49, in get_shape
return tf.TensorShape((m, n))
File "/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/tensor_shape.py", line 458, in init
self._dims = [as_dimension(d) for d in dims_iter]
File "/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/tensor_shape.py", line 399, in as_dimension
return Dimension(value)
File "/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/tensor_shape.py", line 39, in init
raise ValueError("Dimension %d must be >= 0" % self._value)
ValueError: Dimension -2305843009213693952 must be >= 0

lazy_shape overflows for large tensors and matrices
ValueError: Dimension -2101438300051996672 must be >= 0

Make a good implementation of TensorNet, add experiments with it to one of the repos, add links to it to all others and to the paper.

I just finished reading Chicolki's monograph on tensor networks for dimensionality reduction and I'm excited to start using t3f for some problems I'm working on.

I noticed in the last chapter of the monograph that I should be able to represent my input tensor directly as a set of TT cores but I'm having trouble finding a good example of this being done at scale in t3f and the examples in the codebase.

The problem I want to start with is using t3f to factorize a PMI matrix at scale using HOSVD. My data is going to contain several 10's of millions (to possibly billions) of items that I need to map to this PMI matrix, meaning that I'll end up with a square matrix of size N*N to factorize. I might be able to sparsify this matrix a tad using the normalized PMI but I have not checked exact metrics as of yet.

I'm working with several SMP systems with >=3tb ram that are equipped with several older GPUs (K80's) on each node.

I'm also not seeing an easy way to turn even a large-scale sparse matrix into a TT tensor, since it doesn't appear that the t3f API supports going directly from a common sparse library into the TT format without having to convert it to its dense form.

Forgive me if I've missed details which are documented publicly. I have looked through the published documentation and example notebooks and have not been able to figure out where to start.

If doing full documentation is too hard, at least generate a simple page with the list of function with descriptions from docstrings.

matmul doesn't work for Variable inputs (multiplying TT tensor by tf.Variable), only for tf.Tensor inputs

It's not obvious what should we return when all the indexes are specified exactly -- a number (0-dimensional tf.Tensor) or a 0-dimensional TensorTrain.

The first option is kind of more intuitive. The second option is better because in all other cases, indexing returns a TensorTrain object. But implementing the second option will force as to support 0D TensorTrain objects which is not nice.

t3f.ops.multiply doesn't support unknown shapes as it checks that outputs of get_shape() coincide.