Scalar, Vector, Matrix and Tensor

Foreword

When one starts working with Pickle Tensor and Safetensors data structures, one works with tensors without really being clear about what one is actually working with.

At first glance, the tensor was a black box for me in this sense, containing data without me being interested in the extent to which I was already familiar with this concept.

Over the time, I looked at some theories about it and realized that I was already more than familiar with the tensor theory behind it.

I am writing down the brief results of my observations here.

Scalar, Vector, Matrix and Tensor

General Note

Scalars, vectors and matrices, which are more common to most people, are special cases of a tensor. Or, conversely, the tensor is the extension of theories to scalars, vectors and matrices.

All scalars, vectors and matrices are not per se tensors, although all tensors of rank 0 are scalars, although all tensors of rank 1 are vectors and although all tensors of rank 2 are matrices in principle.

When we are talking about machine learning, we are dealing with the programm technical structure of a tensor and not only with the mathematical structure of a tensor.

For structuring purposes I am using the square brackets below.

Scalar

A scalar is a Zero-Order Tensor or a single numerical value like the ones below.

x = 1
y = 3.14159
z = x + y i, where i is the imaginary number √-1

As one can see, the numerical value can be an integer number, a floating point number or a complex number and so on. It is all a time a single number.

We can write the scalars introduced above also in following notation

x = [1]
y = [3.14159]
z = [x + y i]

Vector

A vector is a First-Order Tensor like the both below.

v = [1  1  1 ...]
w = [1
     1
     1
     ...]

Such a vector can be a row vector or a column vector. In this manner v is a row vector and w is a column vector. If one has more than 1 elements, then it is a vector. If there will be only 1 element, it is again a scalar.

Matrix

A matrix is a Second-Order Tensor like the matrices A, B, C and D below.

A = [1  0
     0  1]
B = [1  0  0
     0  1  0
     0  0  1] 
B = [1  0  0  0
     0  1  0  0
     0  0  1  0
     0  0  0  1]
D = [1  0  0  0  0  ...
     0  1  0  0  0  ...
     0  0  1  0  0  ...
     0  0  0  1  0  ...
     0  0  0  0  1  ...
     ...]

The presented matrices A, B, C and D are in this case identity matrices with ones on the main diagonal.

Tensor

A tensor is an extension of the theories of scalars, vectors and matrices with respect to higher dimensions.

Such a tensor is a multi-dimensional object like the one below.

T = [[[1  0]
      [0  1]],
     [[1  0]
      [0  1]]]

The introduced tensor T consists of two matrices in this special and simple case. Two row vectors build a identity matrix. Together this results in the tensor representation. The tensor T is a tensor of rank three or third order.

The described concept can be extended to fourth, fifth and higher order tensors.

Dimension, Order or Rank of a Tensor

Looking at the structure of a tensor, one can state, that the number of countable square brackets at beginning and end of the tensor are the rank of the tensor. This has to be checked, if this is all the time valid.

Program Technical Representation

Tensor in General

The program technical representation of the tensor T from the last chapter looks like

#!/usr/bin/python

import torch
T = torch.tensor([[[1,0], [0,1]], [[1,0], [0,1]]], dtype=torch.int32)
print(T)

This short Python example using Torch constructs a tensor, which we can find in Pickle Tensor as well as Safetensors files. The declaration what we can find in the tensor is done by

dtype=torch.int32

To be able to play around with tensors I wrote a small tensor demo script [10]. It creates a tensor of fourth-order with random values, prints the tensor in short form using torch as well as numpy, prints the rank, prints the number of elements and the dtype of the created tensor.

Complex Tensor of Order Zero

For what follows I omit the leading Shebang and that one has to import the Python module torch.

The construction of a Complex Zero-Order Tensor is done in two steps. This is special case and that is the reason that I am presenting it here.

x = 1
y = 1

real = torch.tensor([x], dtype=torch.float32)
imag = torch.tensor([y], dtype=torch.float32)

z = torch.complex(real, imag)
print(z)

Now one can check size and the dtype of created object z.

z.size()
z.dtype

This would output using the print command.

torch.Size([1])
torch.complex64

One can now see that the given type float32 for real and imaginary part is conversed to complex64 of the complex number.

A Python script for demonstration purposes can be found at [10].

A Little Bit Theory

Especially a tensor is related to the basius in which he is defined. For a second-order tensor for example everybody knows, that one has obviously three directions in a right-angled or oblique-angled co-ordinate system. Mostly used is x, y and z for the description of the related directions.

For a second-order tensor (matrix) a multiplication looks typically like this

C = A ⋅ B

Correctly one has to say in which basis a operation is done.

 C = A e ⋅ B e

where e is the column containing the basis vectors. If one multiplies a 3 x 3 matrix A with another 3 x 3 matrix B one will get a 3 x 3 matrix C again in basis e.

In tensor calculus one will find a new type of multiplication

T = U ⊗ V
T = (u ⋅ e) ⊗ (v ⋅ f)  where e and f basis vectors

This is no longer a simple multiplication, it is a transformation.

In the context of a tensor and a tensor product, this means the following. For n ⋅ m basis vectors, the dimensionality of U ⊗ V is equal to the product of the dimension of U and V. Dimensions are multiplied and not added in a tensor product [9].

Both multiplications should not be mixed up with the so-called cross product using the sign ×

C = A × B

It must also be noted that especially in tensor calculus one deals with upper and lower indices. This may lead to confusions.

Side Note

I used in the preceding text the designations zero-order tensor, first-order tensor, second order tensor, third-order tensor, fourth-order tensor, fifth-order tensor and so on for the classification of the tensors.

Other terms are also used to describe tensors. Here are a few examples. 2D or 2-D is used for a second-order tensor and 3D or 3-D is used for a third-order tensor. This denotation may lead to some confusions.