The Nerv Matrix Package

Part of the Nerv toolkit.

Description

Underlying structure

In the begining is could be useful to know something about the underlying structure of a Nerv matrix. Please keep in mind that matrice in Nerv is row-major.
Every matrix object is a encapsulation of a C struct that describes the attributes of this matrix.

typedef struct Matrix {
    size_t stride;              /* size of a row */
    long ncol, nrow, nmax;    /* dimension of the matrix, nmax is simply nrow * ncol */
    union {
        float *f;
        double *d;
        long *i;
    } data;                   /* pointer to actual storage */
    long *data_ref;
} Matrix;

It is worth mentioning that that data_ref is a counter which counts the number of references to its memory space, mind that it will also be increased when a row of the matrix is referenced(col = m[2]). A Nerv matrix will deallocate its space when this counter is decreased to zero. Also note that all assigning operation in Nerv is reference copy, you can use copy_tod or copy_toh method to copy value. Also, row assigning operations like m1[2]=m2[3] is forbidden in Nerv.

Class hierarchy

The class hierarchy of the matrix classes can be clearly observed in matrix/init.c. First there is a abstract base class Nerv.Matrix, which is inherited by Nerv.CuMatrix and Nerv.MMatrix(also abstract).
Finally, there is Nerv.CuMatrixFloat, Nerv.CuMatrixDouble, inheriting Nerv.CuMatrix, and Nerv.MMatrixFloat, Nerv.MMatrixDouble, Nerv.MMatrixInt , inheriting Nerv.MMatrix.

Methods

Mind that usually a matrix object can only do calculation with matrix of its own type(a Nerv.CuMatrixFloat matrix can only do add operation with a Nerv.CuMatrixFloat).
In the methods description below, Matrix could be Nerv.CuMatrixFloat, Nerv.CuMatrixDouble, Nerv.MMatrixFloat or Nerv.MMatrixDouble. Element_type could be float or double, respectively. * Matrix = Matrix(int nrow, int ncol)
Returns a Matrix object of nrow rows and ncol columns. * Element_type = Matrix.get_elem(Matrix self, int index)
Returns the element value at the specific index(treating the matrix as a vector). The index should be less than nmax of the matrix. * void Matrix.set_elem(Matrix self, int index, Element_type value)
Set the value at index to be value. * int Matrix.ncol(Matrix self)
Get ncol, the number of columns. * int Matrix.nrow(Matrix self)
Get nrow, the number of rows. * int Matrix.get_dataref_value(Matrix self)
Returns the value(not a pointer) of space the data_ref pointer pointed to. This function is mainly for debugging.
* Matrix/Element_type, boolean Matrix.__index__(Matrix self, int index)
If the matrix has more than one row, will return the row at index as a Matrix . Otherwise it will return the value at index. * void Matrix.__newindex__(Matrix self, int index, Element_type value)
Set the element at index to be value.

Matrix Matrix.create(Matrix a)
Return a new Matrix of a's size(of the same number of rows and columns).
Matrix Matrix.colsum(Matrix self)
Return a new Matrix of size (1,self.ncol), which stores the sum of all columns of Matrix self.
Matrix Matrix.rowsum(Matrix self)
Return a new Matrix of size (self.nrow,1), which stores the sum of all rows of Matrix self.
Matrix Matrix.rowmax(Matrix self)
Return a new Matrix of size (self.nrow,1), which stores the max value of all rows of Matrix self.
Matrix Matrix.rowmax_idx(Matrix self)
Return two new Matrix of size (self.nrow,1), which stores the max value of all rows of Matrix self, and its corresponding column indices(start from zero).
Matrix Matrix.trans(Matrix self)
Return a new Matrix of size (self.ncol,self.nrow), which stores the transpose of Matrix self.
void Matrix.copy_fromh(Matrix self, MMatrix a)
Copy the content of a MMatrix a to Matrix self, they should be of the same size.
void Matrix.copy_fromd(Matrix self, CuMatrix a)
Copy the content of a CuMatrix a to Matrix self, they should be of the same size.
void Matrix.copy_toh(Matrix self, MMatrix a)
Copy the content of the Matrix self to a MMatrix a.
void Matrix.copy_tod(Matrix self, CuMatrix a)
Copy the content of the Matrix self to a CuMatrix a.
void Matrix.copy_rows_fromh_by_idx(Matrix self, MMatrix ma, MMatrixInt idx, int idx_begin)
idx should be a row vector. This function copy the rows of ma to self according to idx, in other words, it assigns ma[idx[i+idx_begin]] to self[i].
void Matrix.copy_rows_fromd_by_idx(Matrix self, Matrix b, Matrix idx, int idx_begin)
idx needs to a row vector matrix, it stacks the rows of index idx of the CuMatrix b and copies to self. idx_begin is used as an offset in the idx index array.
void Matrix.update_select_rows(Matrix self, Matrix err, Matrix idx, double alpha, double beta)
Update selected rows of self, i.e. self[idx[i]] = self[idx[i]] * (1 - beta * alpha) + alpha * err[i].
void Matrix.add(Matrix self, Matrix ma, Matrix mb, Element_type alpha, Element_type beta)
It sets the content of Matrix self to be alpha * ma + beta * mb.Matrix ma,mb,self should be of the same size.
void Matrix.mul(Matrix self, Matrix ma, Matrix mb, Element_type alpha, Element_type beta, [string ta, string tb])
It sets the content of Matrix self to be beta * self + alpha * ma * mb. ta and tb is optional, if ta is 'T', then ma will be transposed, also if tb is 'T', mb will be transposed.
void Matrix.add_row(Matrix self, Matrix va, Element_type beta)
Add beta * va to every row of Matrix self.
void Matrix.fill(Matrix self, Element_type value)
Fill the content of Matrix self to be value.
void Matrix.sigmoid(Matrix self, Matrix ma)
Set the element of Matrix self to be elementwise-sigmoid of ma.
void Matrix.sigmoid_grad(Matrix self, Matrix err, Matrix output)
Set the element of Matrix self, to be self[i][j]=err[i][j]*output[i][j]*(1-output[i][j]). This function is used to propagate sigmoid layer error.
Matrix Matrix.softmax(Matrix self, Matrix a)
Calculate a row-by-row softmax of Matrix a and save the result in self. Returns a new self.nrow*1 index matrix that stores the index of the maximum value of each row.
void Matrix.mul_elem(Matrix self, Matrix ma, Matrix mb)
Calculate element-wise multiplication of Matrix ma and mb, store the result in self.
void Matrix.log_elem(Matrix self, Matrix ma)
Calculate element-wise log of Matrix ma, store the result in self.
void Matrix.expand_frm(Matrix self, Matrix a, int context)
Treating each row of a as speech feature, and do a feature expansion. The self should of size (a.nrow, a.ncol * (context * 2 + 1)). self[i] will be (a[i-context] a[i-context+1] ... a[i] a[i+1] a[i+context]). a[0] and a[nrow] will be copied to extend the index range.
void Matrix.rearrange_frm(Matrix self, Matrix a, int step)
Rearrange a according to its feature dimension. The step is the length of context. So, self[i][j] will be assigned a[i][j / step + (j % step) * (a.ncol / step)]. a and self should be of the same size and step should be divisible by a.ncol.
void Matrix.scale_row(Matrix self, Matrix scale)
Scale each column of self according to a vector scale. scale should be of size 1 * self.ncol.
Matrix Matrix.__add__(Matrix ma, Matrix mb)
Returns a new Matrix which stores the result of ma+mb.
Matrix Matrix.__sub__(Matrix ma, Matrix mb)
Returns a new Matrix which stores the result of ma-mb.
Matrix Matrix.__mul__(Matrix ma, Matrix mb)
Returns a new Matrix which stores the result of ma*mb.
CuMatrix CuMatrix.new_from_host(MMatrix m)
Return a new CuMatrix which is a copy of m.
MMatrix CuMatrix.new_to_host(CuMatrix self)
Return a new MMatrix which is a copy of self.
string Matrix.__tostring__(Matrix self)
Returns a string containing values of Matrix self.

MMatrix MMatrix.load(ChunkData chunk)
Return a new MMatrix loaded from the file position in chunk.
void MMatrix.save(MMatrix self, ChunkFileHandle chunk)
Write self to the file position in chunk.
void MMatrix.copy_from(MMatrix ma, MMatrix mb,[int b_bgein, int b_end, int a_begin])
Copy a part of mb(rows of index [b_begin..b_end)) to ma beginning at row index a_begin. If not specified, b_begin will be 0, b_end will be b.nrow, a_begin will be 0.

Examples

Use get_dataref_value to test Nerv's matrix space allocation.

m = 10
n = 10
fm = nerv.MMatrixFloat(m, n)
dm = nerv.MMatrixDouble(m, n)
for i = 0, m - 1 do
    for j = 0, n - 1 do
        t = i / (j + 1)
        fm[i][j] = t
        dm[i][j] = t
    end
end
print("test fm:get_dataref_value:", fm:get_dataref_value())
print("forced a garbade collect")
collectgarbage("collect")
print("test fm:get_dataref_value:", fm:get_dataref_value())
print(fm)
print(dm)

Test some Matrix calculations.

m = 4
n = 4
fm = nerv.CuMatrixFloat(m, n)
dm = nerv.CuMatrixDouble(m, n)
for i = 0, m - 1 do
    for j = 0, n - 1 do
        -- local t = math.random(10)
        t = i / (j + 1)
        fm[i][j] = t
        dm[i][j] = t
    end
end
print(fm)
fs = fm:create()
fs:softmax(fm)
-- print(fs)
print(dm)
ds = dm:create()
ds:softmax(dm)
-- print(ds)
print(fs)
print(fs + fs)
print(ds + ds)
print(fs - fs)
print(ds - ds)
a = fs:create()
a:mul_elem(fs, fs)
print(a)
a:log_elem(fs)
print(a)