3.2 The Neyman-Pearson Fundamental Lemma

Family of distributions is called simple if the family consists of a single element. Otherwise, we call it composit.

Suppose $H, K$ are both simple and $X$ is discrete. Let \(P_{0}, P_{1}\) be corresponding distribution. Let $S$ be criticial reagion. Since \(P(X \in S) = \sum_{x \in S}P(X = x)\), the problem can be stated as following optimization problem;

\[\begin{align} \min & & & \sum_{x \notin S} P_{1}(X = x) \nonumber \\ \mathrm{subject\ to} & & & \sum_{x \in S} P_{0}(X = x) \le \alpha, \label{chap03_3_6} \end{align}\] \[r(x) := \frac{ P_{0}(X = x) }{ P_{1}(X = x) }\] \[P_{0}(X \in S) = \sum_{x:r(x) > c} P(X = x) = \alpha\]

Theorem 3.2.1

• $P_{0}, P_{1}$
  • distribution of $X$
• $\mu$
  • measure over \((\mathcal{X}, \mathcal{A})\)
• $p_{0}, p_{1}$
  • radon nikodym derivative of $P_{i}$ with respect to $\mu$
• \(H = \{P_{0}\}\),
  • hypothesis
• \(K = \{P_{1}\}\),
  • alternatives
• $\alpha \in [0, 1]$,

(1)

\[\begin{eqnarray} \exists \phi: \text{ test}, \ \exists k, \gamma \in \mathbb{R} \ \text{ s.t. } \ \ \mathrm{E}_{0} \left[ \phi \right] & = & \alpha \label{chap03_03_07} \\ \phi(x) & = & \begin{cases} 1 & (p_{1}(x) > kp_{0}(x)) \\ \gamma & (p_{1}(x) = kp_{0}(x)) \\ 0 & p_{1}(x) < kp_{0}(x) \end{cases} \nonumber \\ & = & 1_{\{p_{1} > kp_{0}\}}(x) + \gamma 1_{\{p_{1} = kp_{0}\}}(x) \quad \mu \text{-a.e. } x \label{chap03_03_08} \end{eqnarray}\]

(2) If test $\phi$ satisifes \(\eqref{chap03_03_07}\) and \(\eqref{chap03_03_08}\), then $\phi$ is most powerful test at level $\alpha$.

(3) If $\alpha \in (0, 1)$ and test $\phi$ is the most powerful test at level $\alpha$, then there exists $k, \gamma \in \mathbb{R}$ such that \(\eqref{chap03_03_08}\) satisfies.

proof.

proof of (1)

For $\alpha = 0$,

\[\begin{eqnarray} F(z) & := & P_{0} \left( \left\{ x \in \mathcal{X} \mid p_{1}(x) \le z p_{0}(x) \right\} \right) \nonumber \\ & = & \int_{\mathcal{X}} 1_{\{\frac{p_{1}}{p_{0}} \le z\}}(x) p_{0}(x) \ \mu(dx) \label{theorem_fundamental_neyman_peason_lemma_02_def_of_cdf} \end{eqnarray}\]

$F$ is cumulative distribution function, that is, $F$ satisfies following propeties:

• (a) $F(+\infty) = 1$,
• (b) $F(-\infty) = 0$,
• (c) $F$ is right continuous,
• (d) $F$ is non decreasing.

Indeed, for (a), by monoton convergence theorem, we have

\[\begin{eqnarray} \int_{\mathcal{X}} 1_{\{x \mid \frac{p_{1}(x)}{p_{0}(x)} \le z\}}(x) p_{0}(x) \ \mu(dx) & \nearrow & \int_{\mathcal{X}} p_{0}(x) \ \mu(dx) \quad (z \rightarrow \infty) \nonumber \\ & = & 1 . \nonumber \end{eqnarray}\]

(b) is obivous since

\[P(p_{i}(x) < 0) = 0 .\]

For (c), let $z \in \mathbb{R}$ be fixed. For all sequence \(\{z_{n}\}_{n \in \mathbb{N}}\), \(z_{n} \searrow z\), we have

\[\begin{eqnarray} \int_{\mathcal{X}} 1_{\{\frac{p_{1}}{p_{0}} \le z_{n}\}}(x) p_{0}(x) \ \mu(dx) - \int_{\mathcal{X}} 1_{\{\frac{p_{1}}{p_{0}} \le z\}}(x) p_{0}(x) \ \mu(dx) & = & \int_{\mathcal{X}} ( 1_{\{\frac{p_{1}}{p_{0}} \le z_{n}\}}(x) - 1_{\{\frac{p_{1}}{p_{0}} \le z\}}(x) ) p_{0}(x) \ \mu(dx) . \nonumber \end{eqnarray}\]

By taking the limit of equation above as $n \rightarrow \infty$, it converges to 0 by Lebesgue dominated convergence theorem. Hence (c) holds. Finally we show (d). But this is obvious since integrand satisifies that

\[\forall z < z^{\prime}, \ \Rightarrow \ \forall x \in \mathcal{X}, \ 0 \le 1_{\{\frac{p_{1}}{p_{0}} \le z\}}(x) p_{0}(x) < 1_{\{\frac{p_{1}}{p_{0}} \le z^{\prime}\}}(x) p_{0}(x) .\]

Therefore, $F$ is cumulative distribution. Since $F$ is non decreasing and right continuous,

\[\begin{equation} 0 < \forall \alpha < 1, \ \exists k \in \mathbb{R}, \ \text{ s.t. } \ F(k-) \le 1 - \alpha \le F(k) . \label{theorem_fundamental_neyman_peason_lemma_01} \end{equation}\]

Indeed, let $\alpha \in (0, 1)$ be fixed. We denote

\[\begin{eqnarray} A & := & \{ x \in \mathbb{R} \mid 1 - \alpha \le F(x) \} \nonumber \\ k & := & \inf A \nonumber \\ B & := & \{ x \in \mathbb{R} \mid F(x) \le 1 - \alpha \} . \nonumber \\ c & := & \sup B . \nonumber \end{eqnarray}\]

By right-continuity,

\[\forall \{k_{n}\}, \ k_{n} \searrow k, \ \lim_{n \rightarrow \infty} F(k_{n}) = F(k) \ge 1 - \alpha .\]

Then $F$ is non decreasing so that

\[\forall \{k_{n}\}_{n \in \mathbb{N}}, \ k_{n} \nearrow k, \ \Rightarrow \ k_{n} \le c .\]

We can easily check this. Suppose there exists $n$ such that \(c < k_{n}\). $F$ is non decreasing so that \(F(c) < F(k_{n})\). If \(1 - \alpha \le F(k_{n})\), \(k_{n} \in A\) but this contradict to \(k_{n} \le k\). In other hand, if we assume \(F(k_{n}) < 1 - \alpha\), \(k_{n} \in B\) so that \(k_{n} \le c\). Therefore the above statement hold. By combining our observations, we have

\[\lim_{n \rightarrow \infty} F(k_{n}) \le F(c) \le 1 - \alpha \le F(k) .\]

The equation \(\eqref{theorem_fundamental_neyman_peason_lemma_01}\) holds. Now we define constant $\gamma$

\[\gamma := \begin{cases} 0 & F(k) = F(k-) = 1 - \alpha \\ \frac{F(k) - (1 - \alpha)}{F(k) - F(k-)} & \text{otherwise} \end{cases} ,\]

and test $\phi$

\[\phi(x) := 1_{\{p_{1} > k p_{0}\}}(x) + \gamma 1_{\{p_{1} = k p_{0}\}}(x) .\]

Then we have

\[\begin{eqnarray} \mathrm{E}_{0} \left[ \phi \right] & = & P_{0}(p_{1} > k p_{0}) + \gamma P_{0}(p_{1} = k p_{0}) \nonumber \\ & = & 1 - F(k) + \gamma (F(k) - F(k-)) \nonumber \\ & = & \begin{cases} 1 - (1- \alpha) & F(k) = F(k-) = 1 - \alpha \\ 1 - F(k) - ( F(k) - (1 - \alpha) ) & \text{otherwise} \end{cases} \nonumber \\ & = & \alpha \end{eqnarray}\]

proof of (2)

Let $\phi$ be test. There exist $k, \gamma \in \mathbb{R}$ such that $\phi$ satisfies that \(\eqref{chap03_03_07}\) and \(\eqref{chap03_03_08}\). For all test $\phi^{\prime}: \mathcal{X} \rightarrow [0, 1]$ at level $\alpha$, from \(\eqref{chap03_03_08}\),

\[(\phi - \phi^{\prime}) (p_{1} - k p_{0}) \ge 0 \quad \mu \text{-a.e.}\]

Hence the integral of the equation satisfies

\[\int_{\mathcal{X}} (\phi - \phi^{\prime}) (p_{1} - k p_{0}) \ \mu(x) \ge 0\]

Form the equation above,

\[\begin{eqnarray} \int_{\mathcal{X}} (\phi - \phi^{\prime}) p_{1} \ \mu(x) & \ge & k \int_{\mathcal{X}} (\phi - \phi^{\prime}) p_{0} \ \mu(x) \nonumber \\ & = & k \mathrm{E}_{0} \left[ \phi \right] - k \mathrm{E}_{0} \left[ \phi^{\prime} \right] \nonumber \\ & = & k \left( \alpha - \mathrm{E}_{0} \left[ \phi^{\prime} \right] \right) \ge 0 \quad (\because \phi^{\prime} \text{ is test at level } \alpha) \nonumber \end{eqnarray}\]

Therefore

\[\mathrm{E}_{1} \left[ \phi \right] \ge \mathrm{E}_{1} \left[ \phi^{\prime} \right] .\]

proof of (3)

Let $\phi^{\prime}$ be most powerful test at level $\alpha$. As we have already shown in (1), there exist the most powerful test $\phi$ at level $\alpha$. By definition of $\phi$ and (2),

\[\int_{\mathcal{X}} (\phi - \phi^{\prime}) (p_{1} - k p_{2}) \ \mu(x) \ge 0 .\]

In other hand, $\phi^{\prime}$ is most powerful test at level $\alpha$ so that we have

\[\mathrm{E}_{1} \left[ \phi^{\prime} \right] \ge \mathrm{E}_{1} \left[ \phi \right] .\]

$\Box$

Corollary 3.2.1

• $\alpha \in (0, 1)$,
• $\beta \in [0, 1]$
  • powe of the most powerful level $\alpha$ test for hypothesis \(\Theta_{0} := \{0\}\) and alternatives \(\Theta_{1} := \{1\}\).

Then

\[P_{0} \neq P_{1} \Rightarrow \alpha < \beta .\]

proof

Since $\phi \equiv \alpha$ is a level-$\alpha$ test , which has power $\alpha$, $\alpha \le \beta$. If $\alpha = \beta$, the test $\phi(x) \equiv \alpha$ is most poweful and by Therorem 3.2.1 (iii) must satisfy \(\eqref{chap03_03_08}\). Then $p_{0} = p_{1}$ $\mu$-a.e. and henc $P_{0} = P_{1}$.

$\Box$