\widehat{P}(\mathbf{x})=\sum_{k=1}^{K}\pi_{k}\mathcal{N}(\mathbf{x}|\mu_{k},\Sigma_{k}),\quad \sum_{k=1}^{K}\pi_{k}=1
$$
-
전통적인 density estimation 방법
-
$K$ = mixtures 개수
-
$\pi_k$ = Gaussian mixture에 할당된 weight
-
👍 flexible, 정확하게 approximate 가능
</aside>
-
$z_{k}\in\{0, 1\}$: 각각의 data pt.에 대해 어떤 mixture를 선택할지에 대한 binary random variable
$$
\widehat{P}(\mathbf{x}|\mathbf{z})=\prod_{k=1}^{K}\mathcal{N}(\mathbf{x}|\mu_{k},\Sigma_{k})^{z_{k}},\quad \widehat{P}(\mathbf{z})=\prod_{k=1}^{K}\pi_{k}^{z_{k}}
$$
-
For a given data
$\mathcal{D}=\{\mathbf{x}{1},\ldots,\mathbf{x}{N}\}$,
the log-likelihood function =
$$
\ln \widehat{P}(\mathcal{D})=\sum_{n=1}^{N}\ln\left(\sum_{k=1}^{K}\pi_{k}\mathcal{N}(\mathbf{x}{n}|\mu{k},\Sigma_{k})\right)
$$
-
Expectation-Maximization (EM) algorithm으로 MLE 문제 풀 수 있음
-
Set the posterior probabilities (or responsibilities)
$$
\gamma(z_{nk}):=\frac{\widehat{P}(\mathbf{x}{n},z{k}=1)}{\widehat{P}(\mathbf{x}{n})}=\frac{\pi{k}\mathcal{N}(\mathbf{x}{n}|\mu{k},\Sigma_{k})}{\sum_{j}\pi_{j}\mathcal{N}(\mathbf{x}{n}|\mu{j},\Sigma_{j})},\quad 1\leq n\leq N,1\leq k \leq K
$$
-
$\mathbf{x}_{n}$이라는 data pt.가 주어졌을 때 $k$번째 mixture가 선택될 확률
→ $k$번째 mixture가 가지는 일종의 설명력
→ 특정 mixture $k$에 들어갈 확률
= (n번째 data가 k번째에 들어갈 확률) / (전체 density)
-
$\pi_k$, $\mu_k$, $\Sigma_k$ 다 알아야 계산 가능 → MLE 통해서 formula 찾을 것!
-
$ln\hat{P}(\mathcal{D})$ 미분
-
w.r.t $\mu_k$ and setting it to zero
$$
N_{k}=\sum_{n=1}^{N}\gamma(z_{nk}),\quad \mu_{k}=\frac{1}{N_{k}}\sum_{n=1}^{N}\gamma(z_{nk})\mathbf{x}_{n}
$$
-
w.r.t $\Sigma_k$ and setting it to zero
$$
\Sigma_{k}=\frac{1}{N_{k}}\sum_{n=1}^{N}\gamma(z_{nk})(\mathbf{x}{n}-\mu{k})(\mathbf{x}{n}-\mu{k})^{\top},\quad \pi_{k}=\frac{N_{k}}{N}
$$
- 그러나, 위의 식들은 not closed-form solution to the maximum likelihood
- ∵ responsibilities $\{\gamma(z_{nk}):n=1,\ldots,N,k=1,\ldots,K\}$
and parameters $\{\mu_{k},\Sigma_{k},\pi_{k}:k=1,\ldots, K\}$ are interconnected each other
- 그러나, 계산을 반복하면 GMM의 log-likelihood function
$\ln\widehat{P}(\mathcal{D})$은 단조롭게 증가하다가 수렴할 것!
[Algorithm] (Expectation-Maximization for Gaussian Mixture Models)
- Input: initial parameters $\{\mu_{k},\Sigma_{k},\pi_{k}:k=1,\ldots K\}$, data $\mathcal{D}$
- Output: converged parameters $\{\mu_{k},\Sigma_{k},\pi_{k}:k=1,\ldots K\}$
<aside>
⌨️ Repeat:
-
(E-step) For $\mathbf{x}_{n}\in \mathcal{D}:$
For $k\in\{1,\ldots,K\}:$
compute $\gamma(z_{nk})$ with the parameters $\mu_{k},\Sigma_{k},\pi_{k}$.
-
(M-step) For $k\in\{1,\ldots,K\}:$
update $N_{k},\mu_{k},\Sigma_{k},\pi_{k}$.
Return $\{\mu_{k},\Sigma_{k},\pi_{k}:k=1,\ldots K\}$.
</aside>
