最小二乘法矩阵求导

原创 人工智能 机器学习

1. 求导法则

本文采用矩阵求导中的分母布局，即：分子横向，分母纵向

• 乘法公式：\(\frac{dv^{T}u}{dx} = \frac{du}{dx}v + \frac{dv}{dx}u\)
• 加法公式：\(\frac{d(u+v)}{dx} = \frac{du}{dx} + \frac{dv}{dx}\)

2. 两个常用例子

例子 1

\(f(x) = A^{T}X\), 其中\(A^{T} = \begin{pmatrix} a_{1}, & a_{2}, & ... & a_{n} \end{pmatrix}\), \(X^{T} = \begin{pmatrix} x_{1}, & x_{2}, & ... & x_{n} \end{pmatrix}\)

\[ \begin{aligned} sol:~~f(x) =A^{T}X = \sum_{i=1}^{n}a_{i}x_{i} \\ \frac{df(x)}{dx} = \begin{pmatrix} \frac{df(x)}{dx_{1}} \\ \frac{df(x)}{dx_{2}}\\ ... \\ \frac{df(x)}{dx_{n}} \end{pmatrix} = \begin{pmatrix} a_{1} \\ a_{2} \\ ... \\ a_{n} \end{pmatrix} = A \end{aligned} \]

所以：\(\frac{dA^{T}X}{dx} = \frac{dX^{T}A}{dx} = A\)

例子2

\(f(x) = X^{T}AX\)，其中\(X^{T} = \begin{pmatrix} x_{1}, & x_{2}, & ... & x_{n} \end{pmatrix}\)，\(A = \begin{pmatrix} a_{11} & a_{12} & ... & a_{1n} \\ a_{21} & a_{22} & ... & a_{2n} \\ ... & ... & ... & ... \\ a_{n1} & a_{n2} & ... & a_{nn} \end{pmatrix}\)

\[ sol:~f(x) = X^{T}AX = \begin{pmatrix} x_{1}, & x_{2}, & ... & x_{n} \end{pmatrix} \begin{pmatrix} a_{11} & a_{12} & ... & a_{1n} \\ a_{21} & a_{22} & ... & a_{2n} \\ ... & ... & ... & ... \\ a_{n1} & a_{n2} & ... & a_{nn} \end{pmatrix} \begin{pmatrix} x_{1} \\ x_{2} \\ ... \\ x_{n} \end{pmatrix} = \sum_{i=1}^{n}\sum_{j=1}^{n}a_{ij}x_{i}x_{j} \]

化简得

\[ \begin{aligned} \frac{df(x)}{dx} = \begin{pmatrix} \frac{df(x)}{dx_{1}} \\ \frac{df(x)}{dx_{2}}\\ ... \\ \frac{df(x)}{dx_{n}} \end{pmatrix} = \begin{pmatrix} \sum_{j=1}^{n}a_{1j}x_{j}+\sum_{i=1}^{n}a_{i1}x_{i} \\ \sum_{j=1}^{n}a_{2j}x_{j}+\sum_{i=1}^{n}a_{i2}x_{i} \\ ... \\ \sum_{j=1}^{n}a_{nj}x_{j}+\sum_{i=1}^{n}a_{in}x_{i} \end{pmatrix} \\ ~\\ = \begin{pmatrix} \sum_{j=1}^{n}a_{1j}x_{j} \\ \sum_{j=1}^{n}a_{2j}x_{j} \\ ... \\ \sum_{j=1}^{n}a_{nj}x_{j} \end{pmatrix} + \begin{pmatrix} \sum_{i=1}^{n}a_{i1}x_{i} \\ \sum_{i=1}^{n}a_{i2}x_{i} \\ ... \\ \sum_{i=1}^{n}a_{in}x_{i} \end{pmatrix} = AX+A^{T}X \end{aligned} \]

所以：\(\frac{dX^{T}AX}{dx} = AX+A^{T}X\)

从上面两个例子中可以得到两个结论：

• \(\frac{dA^{T}X}{dx} = \frac{dX^{T}A}{dx} = A\)
• \(\frac{dX^{T}AX}{dx} = AX+A^{T}X\)

接下来我们会用到上面的两个结论

3. 最小二乘法

3.1 没有加权的回归

各个参数形式如下：

\[ Y = \begin{pmatrix} y_{1} \\ y_{2} \\ ... \\ y_{n} \end{pmatrix}_{n\times 1}~~~ X = \begin{pmatrix} x_{1}^{T} \\ x_{2}^{T} \\ ... \\ x_{n}^{T} \end{pmatrix}_{n\times p}~~~ w = \begin{pmatrix} w_{1} \\ w_{2} \\ ... \\ w_{n} \end{pmatrix}_{p\times 1} \]

将最小二乘表示成矩阵相乘的形式

\[ \begin{aligned} L(w) & = \sum_{i=1}^{n}(y_{i}-x_{i}^{T}w)^{2} \\ & = ||Y-Xw||^{2} \\ & = (Y-Xw)^{T}(Y-Xw) \\ & = (Y^{T}-w^{T}X^{T})(Y-Xw) \\ & = (Y^{T}Y-Y^{T}Xw-w^{T}X^{T}Y+w^{T}X^{T}Xw) \end{aligned} \]

对上述形式的矩阵求导得到最终的结果

\[ \begin{aligned} \frac{L(w)}{dw} & = \frac{d(Y^{T}Y)}{dw} - \frac{d(Y^{T}Xw)}{dw} - \frac{d(w^{T}X^{T}Y)}{dw} + \frac{d(w^{T}X^{T}Xw)}{dw} \\ ~~ \\ & = 0 - X^{T}Y - X^{T}Y + 2X^{T}Xw \\ & = 0 \end{aligned} \]

整理得：<>\(-X^{T}Y-X^{T}Y+2X^{T}Xw=0,w^{*}=(X^{T}X)^{-1}X^{T}Y\)</>, 将\(w^{*}\)带入原式

\[ \begin{aligned} Xw^{*} = X (X^{T}X)^{-1}X^{T}Y = \hat Y = \hat H Y \\ \hat H = X (X^{T}X)^{-1}X^{T} \end{aligned} \]

3.2 加权回归

各个参数形式与没有加权的回归一致
将最小二乘表示成矩阵相乘的形式

\[ \begin{aligned} L(w) & = \sum_{i=1}^{n}r_{i}(y_{i}-x_{i}^{T}w)^{2} \\ & = r||Y-Xw||^{2} \\ & = (Y-Xw)^{T}r(Y-Xw) \\ & = (Y^{T}-w^{T}X^{T})r(Y-Xw) \\ & = (Y^{T}rY-Y^{T}rXw-w^{T}X^{T}rY+w^{T}X^{T}rXw) \end{aligned} \]

对上述形式的矩阵求导得到最终的结果

\[ \begin{aligned} \frac{L(w)}{dw} & = \frac{d(Y^{T}rY)}{dw} - \frac{d(Y^{T}rXw)}{dw} - \frac{d(w^{T}X^{T}rY)}{dw} + \frac{d(w^{T}X^{T}rXw)}{dw} \\ ~~ \\ & = 0 - X^{T}rY - X^{T}rY + 2X^{T}rXw \\ & = 0 \end{aligned} \]

整理得：<>\(-X^{T}rY - X^{T}rY + 2X^{T}rXw = 0, w^{*} = (X^{T}rX)^{-1}X^{T}rY\)</>，将\(w^{*}\)带入原式

\[ \begin{aligned} Xw^{*} = X(X^{T}rX)^{-1}X^{T}rY = \hat Y = \hat H Y \\ \hat H = X(X^{T}rX)^{-1}X^{T}r \end{aligned} \]

本文由 Yonghui Wang 创作，采用 知识共享署名4.0 国际许可协议进行许可
本站文章除注明转载/出处外，均为本站原创或翻译，转载前请务必署名
最后编辑时间为: Dec 19, 2024 12:06 pm