Differential Matrix Calculus

The Derivative of a Matrix

Consider $\mathbf{x}(t)$ where ${\mathbf{x}}^{\prime }={\left[\begin{array}{cccc}{x}_1& x_2& ...& x_n\end{array}\right]}^{\prime }$ and $t$ is a scalar. The derivative of $\mathbf{x}(t)$ with respect to time $t$ is:

$$\frac{d\mathbf{x}}{dt}=\left[\begin{array}{c}\frac{d{x}_1}{dt}\\ \frac{d{x}_2}{dt}\\ ⋮\\ \frac{d{x}_n}{dt}\end{array}\right]$$

The derivative of a matrix with respect to a scalar is just the derivative of all the values. Similarly for an $m\times n$ matrix $\mathbf{X}$

$$\frac{d\mathbf{X}}{dt}=\left[\begin{array}{ccc}\frac{d{x}_{11}}{dt}& \frac{d{x}_{12}}{dt}& \dots \frac{d{x}_{1n}}{dt}\\ \frac{d{x}_{21}}{dt}& \frac{d{x}_{22}}{dt}& \dots \frac{d{x}_{2n}}{dt}\\ ⋮& ⋮& \ddots ⋮\\ \frac{d{x}_{m1}}{dt}& \frac{d{x}_{m2}}{dt}& \dots \frac{d{x}_{mn}}{dt}\end{array}\right]$$

Vector-Valued Functions

The set of $m$ functions on the same $n$ variables can be represented as a vector-valued function $\mathbf{f}$ over the vector $x$

$$\mathbf{f}(\mathbf{x})=\left[\begin{array}{c}{f}_1(x_1,x_2,...,x_n)\\ f_2(x_1,x_2,...,x_n)\\ ⋮\\ f_m(x_1,x_2,...,x_n)\end{array}\right]$$

Each element of the vector $\mathbf{f}$ is a function of the $\mathbf{n}$ variables $x_1,x_2,...x_n$

• $\mathbf{f}$ is an $m\times 1$ vector function over $x$
• $\mathbf{x}$ is an $n\times 1$ vector

The Matrix Form of the Chain Rule

If $\mathbf{f}=\mathbf{f}(\mathbf{x})$ and $\mathbf{x}=(\mathbf{\theta })$ such that $\mathbf{f}=\mathbf{f}(\mathbf{x}(\mathbf{\theta }))$:

$$\frac{\partial \mathbf{f}}{\partial \mathbf{\theta }}=\frac{\partial \mathbf{f}}{\partial \mathbf{x}}\cdot \frac{\partial \mathbf{x}}{\partial \mathbf{\theta }}$$

This is the same as the scalar case, but note that matrix multiplication is not commutative so the order matters.

The Jacobian Matrix

The derivative of a vector function $\mathbf{f}$ with respect to a column vector $\mathbf{x}$ is defined formally as the $m\times n$ Jacobian matrix:

$$\mathbf{f}=\left[\begin{array}{c}{f}_1\\ f_2\\ ⋮\\ f_m\end{array}\right]\mathbf{x}=\left[\begin{array}{c}{f}_1\\ f_2\\ ⋮\\ f_m\end{array}\right]$$

$$\mathbf{J}=\frac{\partial \mathbf{f}}{\partial \mathbf{x}}=\left[\begin{array}{cccc}\frac{\partial f_1}{\partial x_1}& \frac{\partial f_1}{\partial x_2}& \dots & \frac{\partial f_1}{\partial x_n}\\ \frac{\partial f_2}{\partial x_1}& \frac{\partial f_2}{\partial x_2}& \dots & \frac{\partial f_2}{\partial x_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial f_m}{\partial x_1}& \frac{\partial f_m}{\partial x_2}& \dots & \frac{\partial f_m}{\partial x_n}\end{array}\right]$$

$${\mathbf{J}}_{ij}=\frac{\partial f_i}{\partial x_j}$$

The Jacobian matrix is the derivative of a multivariate function, representing the best linear approximation to a differentiable function near a point. Geometrically, it defines a tangent plane to the function $\mathbf{f}$ at the point $\mathbf{x}$

Linearisation of a Matrix Differential Equation

Assume that ${\mathbf{x}}^{\ast }$ is a stationary point (equilibrium state) of a non-linear system described by a matrix differential equation:

$$\dot{\mathbf{x}}(t)=\mathbf{f}\left[\mathbf{x}(t)\right]\mathbf{f}\left[{\mathbf{x}}^{\ast }(t)\right]=0\mathbf{f}(\mathbf{x})=\left[\begin{array}{c}{f}_1(x_1,x_2,...,x_n)\\ f_2(x_1,x_2,...,x_n)\\ ⋮\\ f_m(x_1,x_2,...,x_n)\end{array}\right]$$

The linearisation of this system is the evaluation of the Jacobian matrix at ${\mathbf{x}}^{\ast }$. The linearised equation is $\dot{\mathbf{x}}(t)=\mathbf{Ax}(t)$, with the matrix of constants $A$.

$$\mathbf{A}=\frac{\partial \mathbf{f}}{\partial \mathbf{x}}{|}_{\mathbf{x}={\mathbf{x}}^{\ast }}=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ a_{11}& a_{22}& \dots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \dots & a_{mn}\end{array}\right]$$

Example

Linearise the system around an equilibrium state:

$$\frac{d}{d\mathbf{x}}\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]=\mathbf{f}(\mathbf{x})=\left[\begin{array}{c}\sigma (x_1-x_2)\\ x_1(\rho -x_3)-x_2\\ x_1{x}_2-\beta x_3\end{array}\right]$$

$\sigma$, $\rho$, and $\beta$ are parameters. At it's equilibrium, $\mathbf{f}(\mathbf{x})=0$

$$\mathbf{f}(\mathbf{x})=\left[\begin{array}{c}\sigma (x_1-x_2)\\ x_1(\rho -x_3)-x_2\\ x_1{x}_2-\beta x_3\end{array}\right]=0$$

There are three solutions to this system of algebraic equations, but we're interested in the one at the origin where ${\left[\begin{array}{ccc}{x}_1& x_2& x_3\end{array}\right]}^{\prime }={\left[\begin{array}{ccc}0& 0& 0\end{array}\right]}^{\prime }$. Evaluating the Jacobian at this point:

$$\mathbf{J}=\frac{\partial \mathbf{f}}{\partial \mathbf{x}}=\left[\begin{array}{ccc}\frac{\partial f_1}{\partial x_1}& \frac{\partial f_1}{\partial x_2}& \frac{\partial f_1}{\partial x_3}\\ \frac{\partial f_2}{\partial x_1}& \frac{\partial f_2}{\partial x_2}& \frac{\partial f_2}{\partial x_3}\\ \frac{\partial f_3}{\partial x_1}& \frac{\partial f_3}{\partial x_2}& \frac{\partial f_3}{\partial x_3}\end{array}\right]=\left[\begin{array}{ccc}\sigma & -\sigma & 0\\ \rho -x_3& -1& -x_1\\ x_2& x_1& -\beta \end{array}\right]$$

$$\mathbf{J}({\mathbf{x}}^{\ast })=\mathbf{J}(0)\frac{\partial \mathbf{f}}{\partial \mathbf{x}}{|}_{\mathbf{x}=0}=\left[\begin{array}{ccc}\sigma & -\sigma & 0\\ \rho & -1& 0\\ 0& 0& -\beta \end{array}\right]$$

The linearised equation is therefore:

$$\frac{d\mathbf{f}}{d\mathbf{x}}=\mathbf{Ax}=\left[\begin{array}{ccc}\sigma & -\sigma & 0\\ \rho & -1& 0\\ 0& 0& -\beta \end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]$$

The Derivative of a Scalar Function With Respect to a Vector

If $Q(\mathbf{x})=Q(x_1,x_2,...,x_n)$ is a scalar quantity that depends on a vector $\mathbf{x}$ of $n$ variables, then the derivative of $Q$ with respect to \mathbf is a row vector:

$$\frac{\partial Q}{\partial \mathbf{x}}=\left[\begin{array}{cccc}\frac{\partial Q}{\partial x_1}& \frac{\partial Q}{\partial x_2}& \dots & \frac{\partial Q}{\partial x_n}\end{array}\right]=\nabla Q$$

This is the gradient $\mathrm{grad}(Q)$ or nabla ($\nabla Q$)

The Derivative of the Quadratic Form

Using an auxillary result

$$\alpha ={\mathbf{y}}^{\prime }\mathbf{Bx}\frac{\partial \alpha }{\partial \mathbf{x}}={\mathbf{y}}^{\prime }\mathbf{B}\frac{\partial \alpha }{\partial \mathbf{y}}={\mathbf{x}}^{\prime }{\mathbf{B}}^{\prime }$$

We can compute the derivative of a quadratic form $Q={\mathbf{x}}^{\prime }\mathbf{Ax}$:

$$\frac{\partial Q}{\partial \mathbf{x}}=\frac{\partial }{\partial \mathbf{x}}\left({\mathbf{x}}^{\prime }\mathbf{Ax}\right)+\frac{\partial }{\partial ({\mathbf{x}}^{\prime })}\left({\mathbf{x}}^{\prime }\mathbf{Ax}\right)={\mathbf{x}}^{\prime }\mathbf{A}+{\mathbf{x}}^{\prime }{\mathbf{A}}^{\prime }={\mathbf{x}}^{\prime }(\mathbf{A}+{\mathbf{A}}^{\prime })$$

Since $A$ is symmetric by definition of the quadratic form, $\mathbf{A}={\mathbf{A}}^{\prime }$, the derivative of the quadratic form is a row vector:

$$\frac{\partial Q}{\partial \mathbf{x}}=2{\mathbf{x}}^{\prime }\mathbf{A}$$

Example

Consider the polynomial $Q(x_1,x_2)=x_1^2+4x_1{x}_2+2x_2^2$. Find $\nabla Q$. First putting the equation into quadratic form:

$$Q(x_1,x_2)=\left[\begin{array}{cc}{x}_1& x_2\end{array}\right]\left[\begin{array}{cc}1& 2\\ 1& 2\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]$$

The derivative $\frac{\partial Q}{\partial \mathbf{x}}=2{\mathbf{x}}^{\prime }\mathbf{A}$:

$$\frac{\partial Q}{\partial \mathbf{x}}=2{\mathbf{x}}^{\prime }\mathbf{A}=2\left[\begin{array}{cc}{x}_1& x_2\end{array}\right]\left[\begin{array}{cc}1& 2\\ 1& 2\end{array}\right]=\left[\begin{array}{cc}2x_1+4x_2& 4x_1+4x_2\end{array}\right]$$