Calculate the determinant of the matrix