3.5 Confidence Bounds

• The theory of UMP one-sided tests can be applied to the problem of obtaining a lower or upper bound for a real-valued paramter $\theta$,
• The discussion of lower and upper bounds completely symmetric

Definition confidence bound

• $(\mathcal{X}, \mathcal{A})$,
  • measurable sp
• $(\mathcal{T}, \mathcal{B})$,
  • measurable sp
• $X: \mathcal{X} \rightarrow \mathcal{X}$,
  • identity function
• $\alpha \in [0, 1]$
• $1 - \alpha$,
  • Confidene level
• $\underline{\theta}: \mathcal{X} \rightarrow \mathbb{R}$,
  • Funtion of observation
• $\overline{\theta}: \mathcal{X} \rightarrow \mathbb{R}$,

$\underline{\theta}$ is said to be lower confidnece bound for $\theta$ if

\[\begin{eqnarray} \forall \theta \in \Theta, \ P_{\theta}(\underline{\theta}(X) \le \theta) \ge 1 - \alpha. \label{chap03_03_21} \end{eqnarray}\]

We denote by $\mathcal{H}_{\alpha}$ a set of all lower confidence bound for $\theta$, that is,

\[\mathcal{H}_{\alpha} := \{ \underline{\theta}^{\prime}:\mathcal{X} \rightarrow \mathbb{R} \mid \forall \theta, \ P_{\theta}(\underline{\theta}^{\prime}(X) \le \theta) \ge 1 - \alpha \} .\]

Infimum of LHS of above equation,

\[\inf_{\theta \in \Theta} P_{\theta}(\underline{\theta}(X) \le \theta)\]

is called confidence coefficient of $\underline{\theta}$.

$\underline{\theta} \in \mathcal{H}_{\alpha}$ is said to be uniformly most accurate lower confidence bound for $\theta$ at confidence level $1 - \alpha$ if

\[\forall \theta^{\prime} < \theta, \ P_{\theta}(\underline{\theta}(X) \le \theta^{\prime}) = \min_{\underline{\theta}^{\prime} \in \mathcal{H}_{\alpha}} P_{\theta}(\underline{\theta}^{\prime}(X) \le \theta^{\prime}) .\]

Similary, $\overline{\theta}$ is said to be upper confidence bound for $\theta$ at confidence level $1 - \alpha$ if

\[\forall \theta, \ P_{\theta}(\underline{\theta}(X) \ge \theta) \ge 1 - \alpha.\]

■

Remark

• In practical, confidence coefficent is equal to confidence level

■

Now we consider loss function for lower confidece function.

• $L: \mathbb{R} \times \mathbb{R} \rightarrow \mathbb{R}_{\ge 0}$,
  1. lower confidence level $\underline{\theta}$ should underestimate $\theta$

\[\theta^{\prime} < \theta, \ L(\theta, \theta^{\prime}) \ge 0 .\]

In this definition, we minimize the following equation,

\[\mathrm{E}_{\theta} \left[ L(\theta, \underline{\theta}) \right] .\]

We extend the definition of confidence bound to sets.

Definition confidence sets

• $S(x) \subseteq \Theta \ (x \in \mathcal{X})$,

$S(X)$ is said to constitute a family of confidence sets at confidence level $ 1 - \alpha$ if

\[\begin{eqnarray} \forall \theta \in \Theta, \ P_{\theta}(\theta \in S(X)) \ge 1 - \alpha \label{chap03_03_24} \end{eqnarray}\]

■

Remark

Confidence set for lower confidence bound $\underline{\theta}$ is defined by

\[x \in \mathcal{X}, \ S(x) := \{ \theta \in \Theta \mid \underline{\theta}(x) \le \theta < \infty \} .\]

■

Confidence sets defined in this section is only related to non-randomized test, that is,

\[\exists S_{1} \subseteq \mathcal{X}, \text{ s.t. } \phi(x) = 1_{S_{1}}(x) .\]

Under this assumption, we can take decision function as \(\delta(x) := 1_{S_{1}}(x)\), then corresponding acceptance region and critical region are $S_{0} := \mathcal{X} \setminus S_{1}$ and $S_{1}$ respectively. Moreover,

\[\begin{eqnarray} \forall \theta \in \Theta, \ P_{\theta}(X \in S_{1}) & = & \mathrm{E}_{\theta} \left[ \phi(X) \right] . \nonumber \end{eqnarray}\]

It follows that if acceptance region $S_{0}$ is level $\alpha$ test, then

\[\begin{eqnarray} \forall \theta \in \Theta_{0}, \ \mathrm{E}_{\theta} \left[ \phi(X) \right] & = & \mathrm{E}_{\theta} \left[ 1_{S_{1}}(X) \right] \nonumber \\ & = & 1 - \mathrm{E}_{\theta} \left[ 1_{S_{0}}(X) \right] \nonumber \\ & = & 1 - P_{\theta}(X \in S_{0}) \nonumber \\ & \le & \alpha \end{eqnarray}\]

Remark

Generally, test $\phi$ is a randomized test so that corresponding decision rule is randomized. A decision function is randomized means that $\delta:\mathcal{X} \times D \rightarrow [0, 1]$ where $D$ is decision space. For randomized test, decision space is \(D := \{0, 1\}\),

\[\delta(x, d) = (1 - \phi(x))\epsilon_{0} + \phi(x)\epsilon_{1}\]

where $\epsilon_{d}$ is dirac delta measure over \((D, 2^{D})\).

■

Thereom 3.5.1

• $\alpha \in [0, 1]$,
• \(\Theta_{0}(\theta) := \{\theta\} \ (\theta \in \Theta)\),
  • hypothesis
• \(\Theta_{1}(\theta) := \{\theta^{\prime} \mid \theta \neq \theta^{\prime}\} \ (\theta \in \Theta)\),
  • alternatives
• $A(\theta) \subseteq \mathcal{X}$,
  • given
  • Acceptance region of level $\alpha$ test for hypothesis $\Theta_{0}$
• $K(\theta) \subseteq \mathcal{X}$,
  • given
  • critical region of level $\alpha$ test for alternatives $\Theta_{1}$
• $A(\theta) \cup K(\theta) = \mathcal{X}$,
• $\mathcal{S}_{\alpha}$,
  • set of all level $1 - \alpha$ families of confidence sets for $\theta$.

Let

\[x \in \mathcal{X}, \ S(x) := \{ \theta \in \Theta \mid x \in A(\theta), \} .\]

Then

• (i) $S(x) \in \mathcal{S}_{\alpha}$
• (ii) If for all $\theta_{0}$, $A(\theta_{0})$ is UMP for testing $\Theta_{0}(\theta_{0})$ at level $\alpha$ againts the alternatives $\Theta_{1}(\theta_{0})$, then

\[\theta \in \Theta_{1}(\theta_{0}), \ S = \arg\min_{S^{\prime} \in \mathcal{S}} P_{\theta_{0}}(\theta_{0} \in S^{\prime}(X))\]

proof

(i)

By definition of $S(x)$, for every $\theta \in \Theta$,

\[\begin{eqnarray} \theta \in S(x) \Leftrightarrow x \in A(\theta) . \label{chap03_03_25} \end{eqnarray}\]

Hence

\[\forall \theta \in \Theta, \ \{ x \in \mathcal{X} \mid \theta \in S(x) \} = \{ x \in \mathcal{X} \mid x \in A(\theta) \}\]

Since $A(\theta)$ is acceptance region, for all $\theta \in \Theta$,

\[\begin{eqnarray} P_{\theta} ( \{ x \in \mathcal{X} \mid \theta \in S(x) \} ) & = & P_{\theta}(X \in A(\theta)) \nonumber \\ & = & (P_{\theta}(X \in (A(\theta) \cup K(\theta))) - P_{\theta}(X \in K(\theta))) \nonumber \\ & = & 1 - \sup_{\theta^{\prime} \in \Theta_{0}(\theta)} P_{\theta^{\prime}}(X \in K(\theta^{\prime})) \quad (\because \Theta_{0}(\theta) = \{\theta\}) \nonumber \\ & \ge & 1 - \alpha \quad (\because \text{level }\alpha \text{ test}) \nonumber \end{eqnarray} .\]

(ii)

Let \(S^{*}(x) \in \mathcal{S}_{\alpha}\) and $\theta_{0} \in \Theta$ be fixed. If \(A^{*}(\theta) := \{x \mid \theta \in S^{*}(x)\}\), then

\[\begin{eqnarray} \theta \in \Theta, \ P_{\theta}(X \in A^{*}(\theta)) & = & P_{\theta}(\theta \in S^{*}(X)) \nonumber \\ & \ge & 1 - \alpha \nonumber \end{eqnarray}\]

so that $A^{*}(\theta_{0})$ is the acceptance region of a level $\alpha$ test of $\Theta_{0}(\theta_{0})$. Since $A(\theta_{0})$ is acceptance region of UMP test for hypothesis $\Theta_{0}(\theta_{0})$ against alternative $\Theta_{1}(\theta_{0})$,

\[\forall \theta \in \Theta_{1}(\theta_{0}), \ P_{\theta}(X \in A^{*}(\theta_{0})) \ge P_{\theta}(X \in A(\theta_{0})) .\]

Hence

\[\forall \theta \in \Theta_{1}(\theta_{0}), \ P_{\theta}(\theta_{0} \in S^{*}(X)) \ge P_{\theta}(\theta_{0} \in S(X)), .\]

$\Box$

Corollary 3.5.1

• $T:\mathcal{X} \rightarrow \mathbb{R}$,
• $p_{\theta}(x) \ \theta \in \Theta$,
  • family of densities
  • MLR in $T$
• $F_{\theta}(t)$,
  • cumulative distribuiton function of $T$
  • $\forall \theta \in \Theta$, $F_{\theta}$ is continuous
  • $\forall x \in \mathbb{R}$, $F_{\cdot}(x)$ is continuous
• $\alpha \in [0, 1]$,

Then follwoing statements hold:

• (i) There exists a uniformly most accurate confidence bound $\underline{\theta}$ for $\theta$ at confidence level $1 - \alpha$, that is,

\[\forall \theta^{\prime} < \theta, P_{\theta}(\underline{\theta}(X) \le \theta^{\prime}) = \min_{\underline{\theta}^{\prime} \in \mathcal{H}_{\alpha}} P_{\theta}(\underline{\theta}^{\prime}(X) \le \theta^{\prime}) ,\]

• (ii) If $x$ denotes the observed values of $X$ and $t = T(x)$, and if the equation

\[\begin{equation} \exists \hat{\theta} \in \Theta \text{ s.t. } F_{\hat{\theta}}(t) = 1 - \alpha . \label{chap03_03_26} \end{equation}\]

has a solution, $\hat{\theta}$ is unique solution of \(\eqref{chap03_03_26}\) and $\underline{\theta}(x) \equiv \hat{\theta}$,

proof

(i)

For each $\theta_{0} \in \Theta$, by theorem 3.4.1, there exists a constant $C(\theta_{0}), \gamma(\theta_{0})$ such that

\[\begin{eqnarray} \Theta_{0}(\theta_{0}) & := & \{ \theta \in \Theta \mid \theta \le \theta_{0} \} \nonumber \\ \Theta_{1}(\theta_{0}) & := & \{ \theta \in \Theta \mid \theta > \theta_{0} \} \nonumber \\ S_{0}^{\prime}(\theta_{0}) & := & \{ x \in \mathcal{X} \mid T(x) < C(\theta_{0}) \} \nonumber \\ S_{1}(\theta_{0}) & := & \{ x \in \mathcal{X} \mid T(x) > C(\theta_{0}) \} \nonumber \\ S^{\prime}(\theta_{0}) & := & \{ x \in \mathcal{X} \mid T(x) = C(\theta_{0}) \} \nonumber \\ \phi(x;\theta_{0}) & := & \gamma(\theta_{0}) 1_{S^{\prime}(\theta_{0})}(x) + 1_{S_{1}(\theta_{0})}(x) \nonumber \\ \mathrm{E}_{\theta_{0}} \left[ \phi(\cdot ;\theta_{0}) \right] & = & \alpha, \nonumber \end{eqnarray}\]

where $\phi(\cdot;\theta_{0})$ is UMP test at level $\alpha$ for hypothesis $\Theta_{0}(\theta_{0})$ against alternatives $\Theta_{1}(\theta_{0})$. By our assumpations, $S^{\prime}(\theta_{0}) \equiv \emptyset$ and $\gamma(\theta_{0}) \equiv 0$. It follows that

\[\mathrm{E}_{\theta_{0}} \left[ \phi(\cdot ;\theta_{0}) \right] = P_{\theta_{0}}(T(X) > C(\theta_{0})) = \alpha.\]

and $S_{0}(\theta_{0}) := S_{0}^{\prime}(\theta_{0}) \cup S^{\prime}(\theta_{0})$ and $S_{1}(\theta_{0})$ are acceptance region and cretical region of the UMP test, respectively. Now we show that for each $\theta_{0} < \theta_{1}$, $C(\theta_{0}) < C(\theta_{1})$. By theorem 3.4.1 (iii), power function is strictly increasing in $\theta$ so that

\[\begin{eqnarray} \alpha & = & \mathrm{E}_{\theta_{1}} \left[ \phi(\cdot ;\theta_{1}) \right] \nonumber \\ & = & \mathrm{E}_{\theta_{0}} \left[ \phi(\cdot ;\theta_{0}) \right] \nonumber \\ & < & \mathrm{E}_{\theta_{1}} \left[ \phi(\cdot ;\theta_{0}) \right] \nonumber \end{eqnarray}\]

Hence

\[P_{\theta_{1}}(T(X) > C(\theta_{0})) > P_{\theta_{1}}(T(X) > C(\theta_{1})) = \alpha .\]

By monotonicity of measure, $C(\theta_{0}) > C(\theta_{1})$. Moreover, $C$ is a continuous. TBD

We show that

\[\begin{eqnarray} \underline{\theta}(x) & := & \inf \{ \theta \in \Theta \mid T(x) \le C(\theta) \} \nonumber \\ & = & \inf \{ \theta \in \Theta \mid x \in S_{0}(\theta) \} \nonumber \end{eqnarray}\]

Let

\[\begin{eqnarray} S(x) & := & \{ \theta \in \Theta \mid x \in S_{0}(\theta) \} \nonumber \\ & = & \{ \theta \in \Theta \mid T(x) \le C(\theta) \} \nonumber . \end{eqnarray}\]

We show that

\[\begin{equation} \forall \theta \in \Theta, \ \{ x \in \mathcal{X} \mid \theta \in S(x) \} = \{ x \in \mathcal{X} \mid \underline{\theta}(x) \le \theta \} \label{chap03_corollary_03_05_01_equivallent_expression_of_confidence_set} . \end{equation}\]

Indeed,

\[\begin{eqnarray} \forall \theta \in \Theta, \ \theta \in S(x) & \Leftrightarrow & T(x) \le C(\theta) \nonumber \\ & \Leftrightarrow & \underline{\theta}(x) \le \theta \nonumber \end{eqnarray}\]

Moreover, $S(x)$ is confidence set defined in Theorem 3.5.1 so that $S(x)$ is a family of confidence sets for $\theta$ at confidnece level $1 - \alpha$ and

\[\begin{eqnarray} \theta \in \Theta_{1}(\theta_{0}), \ P_{\theta_{0}}(\theta \in S^{\prime}(X)) & = & P_{\theta_{0}}(\underline{\theta}(X) \le \theta) \quad (\because \eqref{chap03_corollary_03_05_01_equivallent_expression_of_confidence_set}) \nonumber \\ & = & \min_{S^{\prime} \in \mathcal{S}_{\alpha}} P_{\theta_{0}}(\theta \in S^{\prime}(X)) \quad (\because \text{theorem 3.5.1}) \nonumber \end{eqnarray} .\]

It is easy to show that

\[\left\{ S^{\prime}(x) := \{ \theta \mid \underline{\theta}^{\prime}(x) \le \theta \} \mid \underline{\theta}^{\prime}(x) \in \mathcal{H}_{\alpha} \right\} \subseteq \mathcal{S}_{\alpha} .\]

Thus, $\underline{\theta}$ is uniformly most accurate confidence bound.

(ii)

It follows from Theorem 3.4.1 that $F_{\theta}(t)$ is a stritly decreasing function of $\theta$. Thus, \(\eqref{chap03_03_26}\) has at most one solution and by our assumptions of continuity it has a solution.

Let $t$ be fixed. Suppose that $\hat{\theta} \in \Theta$ is the solution for $t$ of \(\eqref{chap03_03_26}\).

\[\begin{eqnarray} & & P_{\hat{\theta}}(T(X) > C(\hat{\theta})) = \alpha \nonumber \\ & \Leftrightarrow & 1 - P_{\hat{\theta}}(T(X) \le C(\hat{\theta})) = \alpha \nonumber \\ & \Leftrightarrow & 1 - \alpha = P_{\hat{\theta}}(T(X) \le C(\hat{\theta})) \nonumber \end{eqnarray}\]

$\Box$