Basic Operations on Tensors
This article is part two of the Linear Algebra series. The first part gave a basic introduction to the fundamentals of Linear Algebra: scalars, vectors, matrices and tensors. The article also discussed scalars being 0th order tensors, vectors being 1st order tensors and matrices being 2nd order tensors. In this article, we'll see the basic operations that can be performed on tensors. The following tensor operations are discussed.
- Addition
- Broadcasting
- Multiplication (including dot product and Hadamard Product)
- Transpose
- Inverse
Let's get started.
Addition
A normal matrix addition involves element-wise addition. Let us consider two matrices \(A\) and \(B\) and their sum resulting in a new matrix \(C\).
\[A_{ij} + B_{ij} = C_{ij}\]
For the addition of two matrices, the dimensions of the matrices must match. The final resulting matrix would also be of the same dimension.
Example:
If \(A = \begin{pmatrix} 1 & 2\\ 3 & 4 \end{pmatrix}, B = \begin{pmatrix} 5 & 6\\ 7 & 8 \end{pmatrix}\), then the addition of the matrices would go like this:
\[\begin{pmatrix} 1 & 2\\ 3 & 4 \end{pmatrix} + \begin{pmatrix} 5 & 6\\ 7 & 8 \end{pmatrix} = \begin{pmatrix} 6 & 8\\ 10 & 12 \end{pmatrix}\]
Broadcasting
This is a special type of tensor addition operation where a matrix and a vector are added. Earlier, we saw that the addition requires two tensors to have the same dimension for element-wise sum. However, we already know that a vector is a 1-D array while a matrix is a 2-D array. Therefore, broadcasting is basically a programmatic approach of adding a matrix and a vector. The vector is automatically replicated to match the dimension of the matrix it is going to be added.
Here's the mathematical notation of broadcasting: \(A_{ij} + v_{j} = C_{ij}\), where \(A_{ij}\) is a matrix and \(v_{j}\) is a vector. The dimension of the vector, however, should match either the number of rows or the number of columns of the matrix it is going to be added to. Only this way, the vector can replicate itself for addition.
NumPy and MATLAB both support broadcasting.
Example:
Let us suppose, \(A = \begin{pmatrix} 1 & 3 & 5\\ 2 & 4 & 6 \end{pmatrix}, v = \begin{bmatrix} 1 \\ 2 \end{bmatrix}\)
Although the dimensions do not match, the vector is replicated as a part of broadcasting and the following result is produced after addition:
\[C = \begin{pmatrix} 2 & 4 & 6\\ 4 & 6 & 8 \end{pmatrix}\]
Multiplication
Multiplication of a matrix \(A\) of dimension \(m*n\) and a matrix \(B\) of dimension \(n*q\) is given by:
\[C_{ij} = \sum_{k}A_{ik}B_{kj}\]
For multiplication, the number of columns of the first matrix must match the number of rows of the second matrix.
Example:
Let's suppose two matrices, \(A = \begin{pmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{pmatrix}\) and \(B = \begin{pmatrix} 1 & 2 \\ 3 & 1\\ 2 & 3 \end{pmatrix}\)
\[C_{ij} = \begin{pmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{pmatrix} * \begin{pmatrix} 1 & 2 \\ 3 & 1\\ 2 & 3 \end{pmatrix} = \begin{pmatrix} 1*1+2*3+3*2 & 1*2+2*1+3*3\\ 4*1+5*3+6*2 & 4*2+5*1+6*3 \end{pmatrix}\]
\[C_{ij} = \begin{pmatrix} 13 & 13\\ 31 & 31 \end{pmatrix}\]
Multiplication of a matrix and a vector (column vector) is also possible if the dimension conditions match.
Hadamard Product
The Hadamard product involves element-wise multiplication. Multiplying matrices must be of the same dimensions.
\[C_{ij} = A_{ij} \odot B_{ij}\]
Example:
\[\begin{pmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{pmatrix} \odot \begin{pmatrix} 2 & 2 & 3\\ 4 & 2 & 1 \end{pmatrix} = \begin{pmatrix} 2 & 4 & 9\\ 16 & 10 & 6 \end{pmatrix}\]
Dot Product
A dot product of two vectors results in a scalar. The dot product between vectors \(\overset{\rightarrow}{x}\) and \(\overset{\rightarrow}{y}\) is defined as (in vector notation):
\[\overset{\rightarrow}{x}.\overset{\rightarrow}{y} = a\]
\[\sum_{i}x_{i}y_{i} = a\]
where, \(a\) is scalar.
Example:
Let's suppose \(\overset{\rightarrow}{x} = \begin{bmatrix} 1 & 2 & 3 & 4 \end{bmatrix}\) and \(\overset{\rightarrow}{y} = \begin{bmatrix} 2 & 3 & 1 & 4 \end{bmatrix}\)
\[= \begin{bmatrix} 1 & 2 & 3 & 4 \end{bmatrix} . \begin{bmatrix} 2 & 3 & 1 & 4 \end{bmatrix}\]
\[= \begin{bmatrix} 1*2+2*3+3*1+4*4\end{bmatrix} = 27\]
In matrix notation, the dot product is written as:
\( x^{T}*y = a\) or \(x*y^{T} = a\)
where the superscript \(T\) denotes the transpose operation. The transpose makes sure that the two vectors are compatible for multiplication. The transpose operation is performed on any one of the multiplying vectors.
Transpose
The transpose of a matrix \(A\) which denoted by \(B = A^{T}\), is given by \(B_{ji} = A_{ij}\).
Example:
Let's suppose a matrix \(A = \begin{pmatrix} 1 & 2 & 3\\ 3 & 4 & 5 \end{pmatrix}\). The transpose \( A^{T}\) is:
\(B = A^{T} = \begin{pmatrix} 1 & 3\\ 2 & 4\\ 3 & 5 \end{pmatrix}\)
Inverse
An inverse matrix \((A^{-1})\) is a matrix which, when multiplied by the original matrix \((A)\), gives an identity matrix \((I)\). An identity matrix is a special kind of square matrix that has ones on the diagonal, with all other elements as zero.
\[I = A^{-1}A = AA^{-1}\]
An identity matrix \(I\) of dimension \(3*3\) is defined as:
\[I = \begin{pmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{pmatrix}\]
Not all square matrices will have an inverse. In the case of non-square matrices, we have Moore-Penrose pseudo inverse for the generalization of the matrix inverse.
That's all for this post. See you in the next one.