证明Logistic回归的损失函数是凸函数

Last updated Sep 11, 2021 Edit Source

• All
• Machine learning
• Logistic regression
• Convex optimization
• Cost function

Tags: #MachineLearning #LogisticRegression #ConvexOptimization #CostFunction

Part.13_Cost_Function-Logistic_Regression(ML_Andrew.Ng.) $$\begin{aligned} h&=g(X \theta) \\ J(\theta)&=-\frac{1}{m} \cdot\left[y^{T} \log (h)+(1-y)^{T} \log (1-h)\right] \end{aligned}$$

在梯度下降里面我们已经求出了一阶导数了:

$$h(x)=g\left(\theta^Tx\right) $$ $$\begin{aligned} \frac{\partial}{\partial \theta_{j}} h(x)&= \frac{\partial}{\partial \theta_{j}} g\left(\theta^Tx\right)\\ &=g\left(\theta^Tx\right)\left(1-g\left(\theta^Tx\right)\right) \frac{\partial}{\partial \theta_{j}} \theta^Tx\\ &=g\left(\theta^Tx\right)\left(1-g\left(\theta^Tx\right)\right)x_j\\ &=h(x)\left(1-h(x)\right)x_j \end{aligned}$$

$$\begin{aligned} \frac{\partial^2}{\partial \theta_{j}^2} J(\theta) &=\frac{\partial}{\partial \theta_{j}}\left[\frac{1}{m} \sum_{i=1}^{m}\left(h(x^{(i)})-y^{(i)}\right) x^{(i)}{j}\right]\\ &=\frac{1}{m} \sum{i=1}^{m}\left(\frac{\partial}{\partial \theta_{j}}h(x^{(i)})\right) x^{(i)}{j} \\ &=\frac{1}{m} \sum{i=1}^{m}g\left(\theta^Tx\right)\left(1-g\left(\theta^Tx\right)\right)\left(x_j^{(i)}\right)^2 \\ &=\frac{1}{m} \sum_{i=1}^{m}h(x)\left(1-h(x)\right)\left(x_j^{(i)}\right)^2 \end{aligned}$$

• 接下来判断二阶导数的负号: 对于Sigmoid函数的导函数部分, 观察函数图像, 蓝色曲线为导函数始终大于零: Logistic Function 同时$\left(x_j^{(i)}\right)^2$也非负, 所以二阶导数恒大于等于零, 函数为凸函数, 证毕.