1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 1. 目录2. 相关先修内容 (待补充) 2.1. 概率 2.2. 大数定律 Law of Large Numbers 2.3. 中心极限定律 Central Limit Theorem 2.4. 其他 (待更) 2.5. 补充 3. 各种概率分布 (待补充) 3.1. 离散型分布 3.2. 连续型分布 4. 抽样方法 4.1. The Inversion Method 4.2. The Grid Method 4.3. The Rejection Method 4.4. The Sampling/Importance Resampling Method 4.5. The Stochastic Representation Method 4.6. The Conditional Sampling Method 5. 统计中的优化算法 (待更) 5.1. 基础 5.2. The Newton-Raphson Algorithm 5.3. The Fisher Scoring Algorithm 5.4. The EM Algorithm 5.5. The ECM Algorithm 5.6. The MM Algorithms 6. 经典蒙特卡洛积分与黎曼和估计 (待更) 6.1. Classical Monte Carlo Integration 6.2. The Riemannian Sum Estimator 7. 贝叶斯统计 Bayesian Statistics (待更)8. 随机过程 (待更) 8.1. 马尔可夫链 8.2. 平稳过程 8. 蒙特卡洛方法 MCMC Methods (待更)9. Bootstrap Methods (待补充) 9.1. Parametric Bootstrap 9.2. Non-Parametric Bootstrap 9.3. Hypothesis Testing with the Bootstrapj
随机变量：设 ( Ω , F , P ) (\Omega, \mathcal{F}, P) ( Ω , F , P ) X = X ( ω ) X=X(\omega) X = X ( ω ) Ω \Omega Ω
{ ω ∣ X ( ω ) ≤ x } ∈ F \{\omega | X(\omega)\leq x\} \in \mathcal{F}
{ ω ∣ X ( ω ) ≤ x } ∈ F
则称 X = X ( ω ) X=X(\omega) X = X ( ω )
大数定律主要有两种表现形式：弱大数定律 和强大数定律 。定律的两种形式都肯定无疑地表明，样本均值
X n ‾ = 1 n ( X 1 + ⋯ + X 2 ) X n ‾ → μ as n → ∞ \overline{X_n} = \frac{1}{n} (X_1 + \cdots + X_2) \qquad \overline{X_n} \to \mu \text{ as $n \to \infty$}
X n = n 1 ( X 1 + ⋯ + X 2 ) X n → μ as n → ∞
其中 X 1 , X 2 , … { {X_1}, {X_2}, \dots } X 1 , X 2 , … E ( X 1 ) = E ( X 2 ) = ⋯ = μ {E(X_1)=E(X_2)=\cdots=\mu} E ( X 1 ) = E ( X 2 ) = ⋯ = μ X j {X_j} X j E ( X j ) {E(X_j)} E ( X j )
弱大数定律(Weak law)也称为辛钦定理 (Khinchin’s law)，陈述为：样本均值依概率收敛于期望值。ϵ {\epsilon} ϵ
l i m n → ∞ P ( ∣ X n ‾ − μ ∣ > ϵ ) = 0 {lim_{n \to \infty} P(\lvert \overline{X_n}-\mu \rvert > \epsilon)=0}
l i m n → ∞ P ( ∣ X n − μ ∣ > ϵ ) = 0
强大数定律(Strong law)指出，样本均值以概率1 (almost surely) 收敛于期望值。P ( l i m n → ∞ X n ‾ = μ ) = 1 {P(lim_{n \to \infty} \overline{X_n}=\mu)=1} P ( l i m n → ∞ X n = μ ) = 1
**殆必收敛 converges almost surely**
设样本空间S为闭区间[0,1]，且该区间上的概率分布为均匀分布。定义随机变量X n ( s ) = s + s n X_n(s)=s+s^n X n ( s ) = s + s n X ( s ) = s X(s)=s X ( s ) = s s ∈ [ 0 , 1 ) s\in [0,1) s ∈ [ 0 , 1 ) n → ∞ n\to\infty n → ∞ s n → 0 s^n\to 0 s n → 0 X n ( 1 ) = 2 X_n(1)=2 X n ( 1 ) = 2 X n ( 1 ) X_n(1) X n ( 1 ) P r ( [ 0 , 1 ) ) = 1 Pr([0,1))=1 P r ( [ 0 , 1 ) ) = 1 X n X_n X n
**依概率收敛，但非殆必收敛 converges in probability**
设样本空间S为闭区间[0,1]，且该区间上的概率分布为均匀分布。定义随机变量序列X如下：
X 1 ( s ) = s + I [ 0 , 1 ] ( s ) X 2 ( s ) = s + I [ 0 , 1 / 2 ] ( s ) X 3 ( s ) = s + I [ 1 / 2 , 1 ] ( s ) X 4 ( s ) = s + I [ 0 , 1 / 3 ] ( s ) X 5 ( s ) = s + I [ 1 / 3 , 2 / 3 ] ( s ) X 6 ( s ) = s + I [ 2 / 3 , 1 ] ( s ) . . . X_1(s)=s+I_{[0,1]}(s)\\
X_2(s)=s+I_{[0,1/2]}(s)\\
X_3(s)=s+I_{[1/2,1]}(s)\\
X_4(s)=s+I_{[0,1/3]}(s)\\
X_5(s)=s+I_{[1/3,2/3]}(s)\\
X_6(s)=s+I_{[2/3,1]}(s)\\
... X 1 ( s ) = s + I [ 0 , 1 ] ( s ) X 2 ( s ) = s + I [ 0 , 1 / 2 ] ( s ) X 3 ( s ) = s + I [ 1 / 2 , 1 ] ( s ) X 4 ( s ) = s + I [ 0 , 1 / 3 ] ( s ) X 5 ( s ) = s + I [ 1 / 3 , 2 / 3 ] ( s ) X 6 ( s ) = s + I [ 2 / 3 , 1 ] ( s ) . . .
令X ( s ) = s X(s)=s X ( s ) = s n → ∞ n\to\infty n → ∞ P r ( ∣ X n − X ∣ ≥ ϵ ) Pr(|X_n-X|\geq\epsilon) P r ( ∣ X n − X ∣ ≥ ϵ ) X n X_n X n X n X_n X n s ∈ S s\in S s ∈ S X n ( s ) → s = X ( s ) X_n(s)\to s=X(s) X n ( s ) → s = X ( s ) X n ( s ) X_n(s) X n ( s ) X 1 ( s ) = X 2 ( s ) = X 5 ( s ) = ⋯ = 1 + 3 8 X 3 ( s ) = X 4 ( s ) = X 6 ( s ) = ⋯ = 3 8 X_1(s)=X_2(s)=X_5(s)=\cdots=1+\frac{3}{8}\quad X_3(s)=X_4(s)=X_6(s)=\cdots=\frac{3}{8} X 1 ( s ) = X 2 ( s ) = X 5 ( s ) = ⋯ = 1 + 8 3 X 3 ( s ) = X 4 ( s ) = X 6 ( s ) = ⋯ = 8 3
所以殆必收敛蕴含依概率收敛，不过，依概率收敛的随机变量序列必存在殆必收敛的子列 (见Resnick 1999 第6.3节)。
converges almost surely
converges in probability
待更
切比雪夫不等式、协方差与相关系数
待更
**四个统计方向 (field)**
Classical Methods: Likelihood / Frequentist
Bayesian Methods
Fiducial Methods (信任方法)
Bootstrap Methods (自助方法) **四个研究内容**
点估计(point estimate): Constrained MLE / Unconstrained MLE / Generalized Moment Estimate / Robust Estimate
区间估计(interval estimate): S+d / Bootstrap
假设检验(Testing Hypothesis)
模型的拟合与选择: goodness of fit test / variable selection / AIC / BIC **四个检验方法**
Wald Test
Likelihood Ratio Test
Score Test
Exact Test **四个算法**
Newton-type algorithm
EM-type algorithm
MM-type algorithm
Mode-sharing algorithm **四大数据类型**
Continuous型
Discrete型
区间型 如(0, 1)内的连续型
寿命型 如(Censoring), Longitudinal **四大模型**
Without Covaviate (Distribution)
With Covaviate (Linear, Generialized, Non-linear)
Cox Model
Random Effect Models 均匀分布 X ~ U(a, b)
描述了有限个数值拥有相同概率的概率分布。
支撑集 k ∈ { a , a + 1 , a + 2 , … , b − 1 , b } k \in \{ a, a + 1, a + 2, \ldots, b - 1, b\} k ∈ { a , a + 1 , a + 2 , … , b − 1 , b }
概率质量函数
p ( X = k ) = { 1 n for a < k < b 0 otherwise n = b − a + 1 p( X = k ) = \left\{ \begin{matrix}
\frac{1}{n} & \textrm{ for a < k < b} \\
0 & \textrm{ otherwise} \\
\end{matrix} \right. \qquad n = b - a + 1 p ( X = k ) = { n 1 0 for a < k < b otherwise n = b − a + 1
累计分布函数
F ( k ) = { 0 k < a k − a + 1 n a ≤ k < b 1 b ≤ k F( k ) = \left\{ \begin{matrix}
0 & \textrm{ k < a} \\
\frac{k - a + 1}{n} & \textrm{ $a \leq k < b$} \\
1 & \textrm{ $b \leq k$} \\
\end{matrix} \right. F ( k ) = ⎩ ⎨ ⎧ 0 n k − a + 1 1 k < a a ≤ k < b b ≤ k
数学期望
E ( X ) = a + b 2 E( X ) = \frac{a + b}{2}
E ( X ) = 2 a + b
方差
var ( X ) = n 2 − 1 12 \text{var}( X ) = \frac{n^{2} - 1}{12}
var ( X ) = 1 2 n 2 − 1
伯努利/两点分布 X ~ Bernoulli§
描述了若伯努利试验成功，则伯努利随机变量取值为1。若伯努利试验失败，则伯努利随机变量取值为0的概率分布。
支撑集 k ∈ { 0 , 1 } k \in \{ 0,\ 1\} k ∈ { 0 , 1 }
概率质量函数
p ( X = k ) = p k ( 1 − p ) 1 − k = { 1 − p k = 0 p k = 1 p( X = k ) = p^{k}{(1 - p)}^{1 - k} = \left\{ \begin{matrix}
1 - p & \textrm{ k = 0 }\\
p & \textrm{ k = 1 }\\
\end{matrix} \right. p ( X = k ) = p k ( 1 − p ) 1 − k = { 1 − p p k = 0 k = 1
累计分布函数
F ( k ) = { 0 k < 1 1 − p 0 ≤ k < 1 1 1 ≤ k F( k ) = \left\{ \begin{matrix}
0 & \textrm{ k < 1 }\\
1 - p & \textrm{ $0 \leq k < 1$} \\
1 & \textrm{ $1 \leq k$} \\
\end{matrix} \right. F ( k ) = ⎩ ⎨ ⎧ 0 1 − p 1 k < 1 0 ≤ k < 1 1 ≤ k
数学期望
E ( X ) = p E( X ) = p
E ( X ) = p
方差
var ( X ) = p ( 1 − p ) \text{var}( X ) = p(1 - p)
var ( X ) = p ( 1 − p )
二项分布 X ~ B(n, p)
描述了n次试验中正好得到k次成功的概率分布。
支撑集 k ∈ { 0 , 1 , … , n } k \in \{ 0,\ 1,\ \ldots,\ n\} k ∈ { 0 , 1 , … , n }
概率质量函数
p ( X = k ; n , p ) = ( n k ) p k ( 1 − p ) n − k p( X = k;n,p ) = {n\choose k} p^k (1 - p)^{n - k}
p ( X = k ; n , p ) = ( k n ) p k ( 1 − p ) n − k
( n k ) = n ! k ! ( n − k ) ! {n\choose k} = \frac{n!}{k!( n - k )!}
( k n ) = k ! ( n − k ) ! n !
累计分布函数
F ( k ; n , p ) = ∑ i = 0 k ( n i ) p i ( 1 − p ) n − i F( k;n,p ) = \sum_{i = 0}^{k}{ {n\choose i} p^i(1 - p)^{n - i}}
F ( k ; n , p ) = i = 0 ∑ k ( i n ) p i ( 1 − p ) n − i
数学期望
E ( X ) = n p E( X ) = np
E ( X ) = n p
方差
var ( X ) = n p ( 1 − p ) \text{var}( X ) = np(1 - p)
var ( X ) = n p ( 1 − p )
多项式分布 X ~ Multinomial (n, p1, p2, …, pk)
是二项分布的一般化。
支撑集x i ∈ { 0 , 1 , … , n } ∑ x i = n i ∈ { 1 , 2 , … , k } x_{i} \in \left\{ 0,\ 1,\ \ldots,\ n \right\}\quad\sum_{}^{}x_{i} = n\quad i \in \{ 1,\ 2,\ \ldots,\ k\} x i ∈ { 0 , 1 , … , n } ∑ x i = n i ∈ { 1 , 2 , … , k }
概率质量函数
p ( X ⃗ = { x 1 , x 2 , … , x k } T ; n , p ) = { n ! x 1 ! … x k ! p x 1 ⋯ p x k w h e n ∑ x i = n 0 otherwise p( \vec{X} = {\{ x_1, x_2, \ldots, x_k\}}^T;n,p ) = \left\{ \begin{matrix}
\frac{n!}{x_1 !\ldots x_k !} p^{x_1}\cdots p^{x_k} & \textrm{ $when\sum_{}^{}x_{i} = n$} \\
0 & \textrm{ otherwise }\\
\end{matrix} \right. p ( X = { x 1 , x 2 , … , x k } T ; n , p ) = { x 1 ! … x k ! n ! p x 1 ⋯ p x k 0 w h e n ∑ x i = n otherwise
累计分布函数比较复杂。
数学期望
E ( X i ) = n p i E( X_{i} ) = np_{i}
E ( X i ) = n p i
方差/协方差
var ( X i ) = n p i ( 1 − p i ) \text{var}( X_{i} ) = np_{i}(1 - p_{i})
var ( X i ) = n p i ( 1 − p i )
Cov ( X i , X j ) = − n p i p j i ≠ j \text{Cov}( X_{i},\ X_{j} ) = - np_{i}p_{j}\quad i \neq j
Cov ( X i , X j ) = − n p i p j i = j
泊松分布 X ~ P(λ)
在二项分布的伯努利试验中，如果试验次数n很大，二项分布的概率p很小，且乘积λ=
支撑集 k ∈ { 0 , 1 , … } k \in \{ 0,\ 1,\ \ldots\} k ∈ { 0 , 1 , … }
概率质量函数
p ( X = k ) = λ k k ! e − λ p( X = k ) = \frac{\lambda^k}{k!}e^{- \lambda}
p ( X = k ) = k ! λ k e − λ
累计分布函数
F ( k ; λ ) = e − λ ∑ i = 0 k λ i i ! = Γ ( k + 1 , λ ) k ! F( k;\lambda ) = e^{- \lambda}\sum_{i = 0}^{k}\frac{\lambda^i}{i!} = \frac{\Gamma( k + 1,\lambda )}{k!}
F ( k ; λ ) = e − λ i = 0 ∑ k i ! λ i = k ! Γ ( k + 1 , λ )
Γ ( s , x ) = ∫ x ∞ t s − 1 e − t d t \Gamma( s,x ) = \int_{x}^{\infty}{t^{s - 1}e^{- t}dt}
Γ ( s , x ) = ∫ x ∞ t s − 1 e − t d t
数学期望
E ( X ) = λ E( X ) = \lambda
E ( X ) = λ
方差
var ( X ) = λ \text{var}( X ) = \lambda
var ( X ) = λ
几何分布 X ~ G§
描述了在伯努利试验中，得到一次成功所需要的试验次数或失败的次数。前一种形式经常被称作shifted
支撑集 k ∈ { 1 , 2 , … } k \in \{ 1,\ 2,\ \ldots\} k ∈ { 1 , 2 , … }
概率质量函数
p ( X = k ; p ) = p ( 1 − p ) k − 1 p( X = k;p ) = p(1 - p)^{k - 1}
p ( X = k ; p ) = p ( 1 − p ) k − 1
累计分布函数
F ( k ; p ) = 1 − ( 1 − p ) k F( k;p ) = 1 - (1 - p)^k
F ( k ; p ) = 1 − ( 1 − p ) k
数学期望
E ( X ) = 1 / p E( X ) = 1/p
E ( X ) = 1 / p
方差
var ( X ) = ( 1 − p ) / p 2 \text{var}( X ) = (1 - p)/p^2
var ( X ) = ( 1 − p ) / p 2
超几何分布 X ~ H(n, K, N)
描述了由含有K个指定种类物件的有限N个物件中抽出n个物件，成功抽出该指定种类的物件的个数k（不归还
支撑集 k ∈ { 1 , 2 , … } k \in \{ 1, 2, \ldots\} k ∈ { 1 , 2 , … }
概率质量函数
p ( X = k ; n , K , N ) = ( K k ) ( N n ) ( N − K n − k ) p(X=k;n,K,N)=\frac{K\choose k }{N\choose n} {N-K\choose n-k}
p ( X = k ; n , K , N ) = ( n N ) ( k K ) ( n − k N − K )
累计分布函数比较复杂
数学期望
E ( X ) = n K / N E( X ) = nK/N
E ( X ) = n K / N
方差比较复杂
均匀分布 X ~ U(a, b)
支撑集 a ≤ x ≤ b a \leq x \leq b a ≤ x ≤ b
概率密度函数
f ( x ) = { 1 b − a f o r a ≤ x ≤ b 0 otherwise f( x ) = \left\{ \begin{matrix}
\frac{1}{b - a} & \textrm{$ for\ a \leq x \leq b$} \\
0 & \textrm{ otherwise} \\
\end{matrix} \right. f ( x ) = { b − a 1 0 f or a ≤ x ≤ b otherwise
累计分布函数
F ( x ) = P r ( X < x ) = { 0 k < a x − a b − a a ≤ k < b 1 b ≤ k F( x ) = Pr(X < x) = \left\{ \begin{matrix}
0 & \textrm{ k < a} \\
\frac{x - a}{b - a} & \textrm{$ a \leq k < b$} \\
1 & \textrm{$ b \leq k$} \\
\end{matrix} \right. F ( x ) = P r ( X < x ) = ⎩ ⎨ ⎧ 0 b − a x − a 1 k < a a ≤ k < b b ≤ k
数学期望
E ( X ) = a + b 2 E(X) = \frac{a + b}{2}
E ( X ) = 2 a + b
方差
D ( X ) = ( b − a ) 2 12 D(X) = \frac{(b - a)^2}{12}
D ( X ) = 1 2 ( b − a ) 2
指数分布 X ~ Exp(λ)
指数分布可以用来表示独立随机事件发生的时间间隔。
支撑集 x ∈ [ 0 , + ∞ ) x \in \lbrack 0,\ + \infty) x ∈ [ 0 , + ∞ )
概率密度函数
f ( x ) = λ e − λ x for x ≥ 0 f( x ) = \lambda e^{- \lambda x}\quad \text{for $x \geq 0$}
f ( x ) = λ e − λ x for x ≥ 0
累计分布函数
F ( x ) = P r ( X < x ) = { 0 x < 0 1 − e − λ x 0 ≤ x F( x ) = Pr(X < x) = \left\{ \begin{matrix}
0 & \textrm{ x < 0} \\
1 - e^{- \lambda x} & \textrm{$ 0 \leq x$} \\
\end{matrix} \right. F ( x ) = P r ( X < x ) = { 0 1 − e − λ x x < 0 0 ≤ x
数学期望
E ( X ) = 1 λ E(X) = \frac{1}{\lambda}
E ( X ) = λ 1
方差
D ( X ) = 1 λ 2 D(X) = \frac{1}{\lambda^2}
D ( X ) = λ 2 1
正态分布 X ~ B(μ, σ2 )
正态分布在统计学上十分重要，经常用在自然和社会科学来代表一个不明的随机变量。
支撑集 x ∈ ( − ∞ , + ∞ ) x \in ( - \infty, + \infty) x ∈ ( − ∞ , + ∞ )
概率密度函数
f ( x ) = 1 σ 2 π e ( x − μ ) 2 − 2 σ 2 f( x ) = \frac{1}{\sigma\sqrt{2\pi}}e^{\frac{(x - \mu)^2}{- 2\sigma^2}}
f ( x ) = σ 2 π 1 e − 2 σ 2 ( x − μ ) 2
累计分布函数
F ( x ) = P r ( X < x ) = 1 2 [ 1 + erf ( x − μ σ 2 ) ] F( x ) = Pr(X < x) = \frac{1}{2}\lbrack 1 + \text{erf}( \frac{x - \mu}{\sigma\sqrt{2}} )\rbrack
F ( x ) = P r ( X < x ) = 2 1 [ 1 + erf ( σ 2 x − μ ) ]
erf ( x ) = 1 π ∫ − x x e − t 2 d t \text{erf}( x ) = \frac{1}{\sqrt{\pi}}\int_{- x}^{x}{e^{- t^{2}}dt}
erf ( x ) = π 1 ∫ − x x e − t 2 d t
数学期望
E ( X ) = μ E(X) = \mu
E ( X ) = μ
方差
D ( X ) = σ 2 D(X) = \sigma^{2}
D ( X ) = σ 2
伽马分布 X ~ Γ(α, β)
伽玛分布是统计学的一种连续概率函数。伽玛分布中的参数α，称为形状参数，β称为尺度参数，假设随机变数X为
支撑集 x ∈ [ 0 , + ∞ ) x \in \lbrack 0, + \infty) x ∈ [ 0 , + ∞ )
概率密度函数，取λ = 1 / β \lambda = 1/\beta λ = 1 / β
f ( x ) = x α − 1 λ α e − λ x Γ ( α ) x > 0 f( x ) = \frac{x^{\alpha - 1}\lambda^{\alpha}e^{- \lambda x}}{\Gamma( \alpha )}\quad x > 0
f ( x ) = Γ ( α ) x α − 1 λ α e − λ x x > 0
Gamma函数特征
{ Γ ( α ) = ( α − 1 ) ! i f α i s Z + Γ ( α ) = ( α − 1 ) ! Γ ( α − 1 ) i f α i s R + Γ ( 1 / 2 ) = π Γ ( 1 ) = 1 \left\{ \begin{matrix}
Γ( \alpha ) = ( \alpha - 1 )! & \textrm{$ if\ \alpha\ is \mathbb{Z}^{+} $} \\
Γ( \alpha ) = ( \alpha - 1 )!\Gamma( \alpha - 1 ) & \textrm{$ if\ α\ is\ \mathbb{R}^{+} $} \\
\Gamma( 1/2 ) = \sqrt{\pi} & \textrm{$ \Gamma( 1 ) = 1 $} \\
\end{matrix} \right. ⎩ ⎨ ⎧ Γ ( α ) = ( α − 1 ) ! Γ ( α ) = ( α − 1 ) ! Γ ( α − 1 ) Γ ( 1 / 2 ) = π i f α i s Z + i f α i s R + Γ ( 1 ) = 1
累计分布函数比较复杂
数学期望
E ( X ) = α β E( X ) = αβ
E ( X ) = α β
方差
D ( X ) = β 2 α D( X ) = \beta^{2}\alpha
D ( X ) = β 2 α
贝塔分布 X ~ Be(α, β)
指一组定义在(0,1)区间的连续概率分布，有两个参数α，β>0。
支撑集 x ∈ ( 0 , 1 ) x \in (0, 1) x ∈ ( 0 , 1 )
概率密度函数
f ( x ) = x α − 1 ( 1 − x ) β − 1 B ( α , β ) x > 0 f( x ) = \frac{x^{\alpha - 1}{(1 - x)}^{\beta - 1}}{Β( \alpha,\beta )}\quad x > 0
f ( x ) = B ( α , β ) x α − 1 ( 1 − x ) β − 1 x > 0
B ( α , β ) = ∫ 0 1 u α − 1 ( 1 − u ) β − 1 d u Β( \alpha,\beta ) = \int_{0}^{1}{u^{\alpha - 1}{(1 - u)}^{\beta - 1}}du
B ( α , β ) = ∫ 0 1 u α − 1 ( 1 − u ) β − 1 d u
Beta函数特征
B ( α , β ) = Γ ( α ) Γ ( β ) Γ ( α + β ) Β( \alpha,\beta ) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha + \beta)}
B ( α , β ) = Γ ( α + β ) Γ ( α ) Γ ( β )
累计分布函数比较复杂
数学期望
E ( X ) = α α + β E( X ) = \frac{\alpha}{\alpha + \beta}
E ( X ) = α + β α
方差
D ( X ) = α β ( α + β ) 2 ( α + β + 1 ) D( X ) = \frac{αβ}{( \alpha + \beta )^{2}(\alpha + \beta + 1)}
D ( X ) = ( α + β ) 2 ( α + β + 1 ) α β
其他分布
截断正态分布 X ~ TN(n, p)
Dirichlet分布 X
假设对于一个随机变量 X 有概率质量函数 pmf
p ( X = x i ) = p i where p i > 0 , i = 1 , 2 , … , m and ∑ i = 1 m p i = 1 p(X=x_i)=p_i\quad \textrm{where $p_i>0, i=1, 2,\ldots, m$ and $\sum_{i=1}^{m}{p_i=1}$}
p ( X = x i ) = p i where p i > 0 , i = 1 , 2 , … , m and ∑ i = 1 m p i = 1
其中 m 可以是 finite or infinite.
那么 从 U(0, 1)中均匀抽样，得到 Y 与 X 的概率累积分布比较，落在哪个区间即对应哪个值。X的概率取值服从均匀分布。
**算法过程**
Step1. Draw Y from U(0,1);
Step2. If Y < p1, set X = x1 and stop;
Else if Y < p1+p2, set X = x2 and stop;
…
Else if Y < ∑pi, set X = xi and stop;
…
**Note.**
通常将较大 pi 的抽样放在靠近 0 的一侧，会更高效。
不一定要将所有 pi 都先计算出来。比如用 Inversion Method 生成泊松分布样本时，先执行 Step1，然后依次计算出 pi，再求和比较。如果符合条件，就不必计算后续的 pi 了。 假设对于一个随机变量 X 有概率累积分布函数 cdf
F ( x ) = P r ( X < x ) = ∫ − ∞ x f ( t ) d t F(x)=Pr(X<x)=\int_{-\infty}^x f(t)dt
F ( x ) = P r ( X < x ) = ∫ − ∞ x f ( t ) d t
那么 令 Y = y ~ U(0, 1)。从 y = F(x)求解 x = F-1 (y)。F(x)是服从(0, 1)上的均匀分布的。
**算法过程**
Step1. Get the cdf F(x) of X;
Step2. Let x = F-1(y);
Step3. Draw Y from U(0, 1);
Step4. Return X = F-1(Y).
**Note.**
证明随机变量 X 有连续的 cdf, 则随机变量 Y=F(X)~U(0, 1) (这里的 F(·)被视作一个函数)。只需证 F(y)=y
F ( y ) = P r ( Y ≤ y ) = P r ( F ( X ) ≤ y ) F(y)=Pr(Y \leq y)=Pr(F(X)\leq y)
F ( y ) = P r ( Y ≤ y ) = P r ( F ( X ) ≤ y )
对于 Y→X 的映射，有
x = F − 1 ( y ) = inf { x : F ( x ) ≥ y } where y ∈ ( 0 , 1 ) x=F^{-1}(y)=\text{inf}\{x:F(x)\geq y\}\quad \textrm{where $y\in (0,1)$}
x = F − 1 ( y ) = inf { x : F ( x ) ≥ y } where y ∈ ( 0 , 1 )
易知F ( F − 1 ( y ) ) ≡ F ( x ) ≡ y F(F^{-1}(y))\equiv F(x)\equiv y F ( F − 1 ( y ) ) ≡ F ( x ) ≡ y
P r ( F ( X ) ≤ y ) = P r ( X ≤ x ) = F ( x ) = F ( F − 1 ( y ) ) = y Pr(F(X)\leq y)=Pr(X\leq x)=F(x)=F(F^{-1}(y))=y
P r ( F ( X ) ≤ y ) = P r ( X ≤ x ) = F ( x ) = F ( F − 1 ( y ) ) = y
得证。
假设对于一个随机变量 X 有概率密度函数 pdf
X ∼ f X ( x ) where S X is finite X \sim f_X(x)\quad \text{where $S_X$ is finite}
X ∼ f X ( x ) where S X is finite
那么 在支撑集 S_X 上选取合适的格点集合 {xi} 覆盖 S_X 。计算该格点集合对应的概率密度函数值集合 {fx(xi)} 并作归一化
p i = f X ( x i ) ∑ i = 1 d f X ( x i ) i = 1 , 2 , … , d p_i=\frac{f_X(x_i)}{\sum_{i=1}^d{f_X(x_i)}}\quad i=1, 2, \ldots,d
p i = ∑ i = 1 d f X ( x i ) f X ( x i ) i = 1 , 2 , … , d
就得到了随机变量 X 的近似 X’
X ′ ∼ FDiscrete d ( { x i } , { p i } ) X' \sim \text{FDiscrete}_d(\{x_i\},\{p_i\})
X ′ ∼ FDiscrete d ( { x i } , { p i } )
**算法过程**
Step1. Generate an appropriate set {xi}, which covers S X S_X S X
Step2. Get the {pi};
Step3. Sample from X’ ~ FDiscrete({xi},{pi}).
**Note.**
主要是处理 IM 连续型计算复杂的情况，将其简化至离散型，然后再用 IM 或其他方法抽
假设对于一个随机变量 X 有概率密度函数 pdf
X ∼ f X ( x ) where S X is finite X \sim f_X(x)\quad \text{where $S_X$ is finite}
X ∼ f X ( x ) where S X is finite
且有常数 c≥1 和包络密度函数 g(x)，满足
f ( x ) ≤ c g ( x ) ∀ x ∈ S X f(x)\leq cg(x)\quad \forall x\in S_X
f ( x ) ≤ c g ( x ) ∀ x ∈ S X
那么 由必须满足的条件：
f ( x ) ≤ c g ( x ) ∀ x ∈ S X f(x)\leq cg(x)\quad \forall x\in S_X
f ( x ) ≤ c g ( x ) ∀ x ∈ S X
据此可以得到
f Y ( x ∣ Z ≤ f ( Y ) c g ( Y ) ) = P r ( Z ≤ f ( Y ) c g ( Y ) ∣ Y = x ) ⋅ g ( x ) P r ( Z ≤ f ( Y ) c g ( Y ) ) = P r ( Z ≤ f ( Y ) c g ( Y ) ) ∫ S X P r ( Z ≤ f ( Y ) c g ( Y ) ∣ Y = x ) g ( x ) d x g ( x ) = f ( x ) c g ( x ) g ( x ) ∫ S X f ( x ) c g ( x ) g ( x ) d x = f ( x ) c 1 / c = f ( x ) f_Y(x|Z\leq \frac{f(Y)}{cg(Y)})=\frac{Pr(Z\leq \frac{f(Y)}{cg(Y)}|Y=x)\cdot g(x)}{Pr(Z\leq \frac{f(Y)}{cg(Y)})}\\
=\frac{Pr(Z\leq \frac{f(Y)}{cg(Y)})}{\int_{S_X}{Pr(Z\leq \frac{f(Y)}{cg(Y)}|Y=x)}g(x)dx} g(x)\\
=\frac{\frac{f(x)}{cg(x)} g(x)}{\int_{S_X}{\frac{f(x)}{cg(x)}g(x)dx}}=\frac{\frac{f(x)}{c}}{1/c}=f(x) f Y ( x ∣ Z ≤ c g ( Y ) f ( Y ) ) = P r ( Z ≤ c g ( Y ) f ( Y ) ) P r ( Z ≤ c g ( Y ) f ( Y ) ∣ Y = x ) ⋅ g ( x ) = ∫ S X P r ( Z ≤ c g ( Y ) f ( Y ) ∣ Y = x ) g ( x ) d x P r ( Z ≤ c g ( Y ) f ( Y ) ) g ( x ) = ∫ S X c g ( x ) f ( x ) g ( x ) d x c g ( x ) f ( x ) g ( x ) = 1 / c c f ( x ) = f ( x )
由上式可知，仅当
f ( x ) ≤ c g ( x ) ∀ x ∈ S X f(x)\leq cg(x)\quad \forall x\in S_X
f ( x ) ≤ c g ( x ) ∀ x ∈ S X
满足时，fY(x|Z≤f(Y)/cg(Y))=f(x)才成立，否则 fY(x|Z≤f(Y)/cg(Y))=g(x)。所以很自然地,
c = { c o p t ∣ c o p t ≥ f ( x ) g ( x ) ∀ x ∈ S X } c=\{ c_{opt}|c_{opt}\geq \frac{f(x)}{g(x)}\quad \forall x\in S_X \}
c = { c o p t ∣ c o p t ≥ g ( x ) f ( x ) ∀ x ∈ S X }
由于
P r ( Z ≤ f ( Y ) c g ( Y ) ) = ∫ S X P r ( Z ≤ f ( Y ) c g ( Y ) ∣ Y = x ) g ( x ) d x = ∫ S X f ( x ) c g ( x ) g ( x ) d x = 1 / c Pr(Z\leq \frac{f(Y)}{cg(Y)})=\int_{S_X} Pr(Z\leq \frac{f(Y)}{cg(Y)}\big|Y=x)g(x)dx\\
=\int_{S_X} \frac{f(x)}{cg(x)}g(x)dx=1/c P r ( Z ≤ c g ( Y ) f ( Y ) ) = ∫ S X P r ( Z ≤ c g ( Y ) f ( Y ) ∣ ∣ Y = x ) g ( x ) d x = ∫ S X c g ( x ) f ( x ) g ( x ) d x = 1 / c
代表了抽样有效的效率，所以 c 应当尽可能小。故对于一个满足要求的函数簇g θ ( x ) g_{\theta}(x) g θ ( x )
c = m i n { m a x θ ( f ( x ) g θ ( x ) ) } c=min\{ max_\theta (\frac{f(x)}{g_{\theta}(x)}) \}
c = m i n { m a x θ ( g θ ( x ) f ( x ) ) }
**算法过程**
Step1. Draw Z ~ U(0, 1) and independently draw Y ~ g(·) ;
Step2. If Z ≤ f(Y) / [cg(Y)], return X = Y, otherwise, go to Step 1.
**Note.**
求包络参数 c 第一步取最大值是要满足充分条件 c≥f(x)/g(x)。第二步取最小值是对于一簇满足要求的 g(x)，要让接受概率最小。
接受概率为 1/c。
求 f(x)/g(x)最大值时可以改用求 logf(x)-logg(x)最大值。
g(x)的条件有：1）和 f(x)一样的支撑集；2）比 f(x)更大的变化[variance/dispersion]；3）从 g(x)中抽样更简单。
log-concave 函数适合用 piece-wise exponential envelopes。（log-concave 就是 log 后是 concave 函数，可以用分段指数函数来作为包络）。
假设对于一个随机变量 X 有概率密度函数 pdf
X ∼ f X ( x ) where S X is finite X \sim f_X(x)\quad \text{where $S_X$ is finite}
X ∼ f X ( x ) where S X is finite
以及一个相同支撑集上的概率密度函数pdf
X ∼ g ( x ) X \sim g(x)
X ∼ g ( x )
那么 已知
f ( x ) = f ( x ) g ( x ) g ( x ) ∀ x ∈ S X f(x)=\frac{f(x)}{g(x)} g(x) \quad \forall x \in S_X
f ( x ) = g ( x ) f ( x ) g ( x ) ∀ x ∈ S X
则么可以将从 f(x)的抽样过程看成两个过程：Y~g(x)和 X~f(Y)/g(Y)。若要令抽样的 X 概率分布完全等同于 f(x)，首先应当抽出一个数量无穷大的{Y}，再由f(y)/g(y)抽出 X。这个问题反而变复杂了。所以引入误差，使得
w ( X j ) = f ( X j ) X j w(X^j)=\frac{f(X^j)}{X^j}
w ( X j ) = X j f ( X j )
ω j ( X j ) = w ( X j ) ∑ w ( X j ) \omega_j(X^j)=\frac{w(X^j)}{\sum{w(X^j)}}
ω j ( X j ) = ∑ w ( X j ) w ( X j )
相当于使用了 The Grid Method 。再进行重抽样抽出 m 个样本。
P r ( X ∗ ≤ x ∗ ) = ∑ j = 1 J ω j × I ( X j ≤ x ∗ ) = ∑ j = 1 J [ w ( X j ) × I ( X j ≤ x ∗ ) ] ∑ w ( X j ) = 1 J ∑ j = 1 J [ w ( X j ) × I ( X j ≤ x ∗ ) ] 1 J ∑ w ( X j ) Pr(X^*\leq x^*) = \sum_{j=1}^J{\omega_j \times I(X^j\leq x^*)}\\
=\frac{\sum_{j=1}^J {[w(X^j)\times I(X^j\leq x^*)]}}{\sum{w(X^j)}}\\
=\frac{\frac{1}{J}\sum_{j=1}^J {[w(X^j)\times I(X^j\leq x^*)]}}{\frac{1}{J} \sum{w(X^j)}}
P r ( X ∗ ≤ x ∗ ) = j = 1 ∑ J ω j × I ( X j ≤ x ∗ ) = ∑ w ( X j ) ∑ j = 1 J [ w ( X j ) × I ( X j ≤ x ∗ ) ] = J 1 ∑ w ( X j ) J 1 ∑ j = 1 J [ w ( X j ) × I ( X j ≤ x ∗ ) ]
其中I ( x ) I(x) I ( x )
mean ( w ( X j ) ) = ∫ w ( x ) × I ( x ≤ x ∗ ) × g ( x ) d x ∫ w ( x ) g ( x ) d x \text{mean}(w(X^j))=\frac{\int {w(x)\times I(x\leq x^*)\times g(x) dx}}{\int w(x)g(x)dx}
mean ( w ( X j ) ) = ∫ w ( x ) g ( x ) d x ∫ w ( x ) × I ( x ≤ x ∗ ) × g ( x ) d x
故
lim J → ∞ P r ( X ∗ ≤ x ∗ ) = ∫ w ( x ) × I ( x ≤ x ∗ ) × g ( x ) d x ∫ w ( x ) g ( x ) d x = ∫ − ∞ x ∗ f ( x ) d x \lim_{J\to \infty} Pr(X^*\leq x^*)=\frac{\int w(x)\times I(x\leq x^*)\times g(x)dx}{\int w(x)g(x)dx}\\
=\int_{-\infty}^{x^*} f(x)dx J → ∞ lim P r ( X ∗ ≤ x ∗ ) = ∫ w ( x ) g ( x ) d x ∫ w ( x ) × I ( x ≤ x ∗ ) × g ( x ) d x = ∫ − ∞ x ∗ f ( x ) d x
**算法过程**
Step1. Generate X1, X2, … , XJ iid g(·) [Sampling step] ;
Step2. Select a subset { X i } i = 1 m \{X^i\}_{i=1}^m { X i } i = 1 m { X j } j = 1 J \{X^j\}_{j=1}^J { X j } j = 1 J [Importanceresampling step] .
**Note.**
J/m → ∞时即是完全等同，一般取 J/m≥10 或 J/m=20 比较有效。
SIR 得到的是原分布的近似 ，重抽样过程采用了类似 The Grid Method 类似的思想。
假设对于一个随机变量 X 有概率密度函数 pdf
X ∼ f X ( x ) where S X is finite X \sim f_X(x)\quad \text{where $S_X$ is finite}
X ∼ f X ( x ) where S X is finite
那么 如果随机变量 X 和 Y 有相同的分布，则记作𝑋 ⇔ 𝑌 （⇔上有个"d"，下文相同），其中，相同分布的意思是
F ( x ) = P r ( X ≤ x ) ⇔ P r ( Y ≤ y ) = F ( y ) let x = y F(x)=Pr(X\leq x) \Leftrightarrow Pr(Y\leq y)=F(y)\quad \text{let $x=y$}
F ( x ) = P r ( X ≤ x ) ⇔ P r ( Y ≤ y ) = F ( y ) let x = y
比如某个 X 的累积分布 F(x)关于(0, 0.5)对称，那么取 Y=-X，有
F ( y ) = P r ( Y ≤ y ) = P r ( − X ≤ x ) = P r ( X ≥ − x ) = 1 − P r ( X ≤ − x ) = 1 − F ( − x ) = F ( x ) F(y) =Pr(Y\leq y)=Pr(-X\leq x)\\
=Pr(X\geq -x)=1-Pr(X\leq -x)\\
=1-F(-x)=F(x) F ( y ) = P r ( Y ≤ y ) = P r ( − X ≤ x ) = P r ( X ≥ − x ) = 1 − P r ( X ≤ − x ) = 1 − F ( − x ) = F ( x )
所以Y=-X ⇔ X。若𝑋 ⇔ 𝑌且有𝑋 = ℎ(𝑌)，则有f Y ( y ) d y = f X ( x ) d x f_Y(y)dy=f_X(x)dx f Y ( y ) d y = f X ( x ) d x
f Y ( y ) = f X ( x ) ∣ d x d y ∣ = f X ( h ( y ) ) ∣ d h ( x ) d y ∣ = f X ( h ( y ) ) ∣ Δ h ( y ) ∣ f_Y(y)=f_X(x)\Big|\frac{dx}{dy}\Big|=f_X(h(y))\Big|\frac{dh(x)}{dy}\Big|=f_X(h(y))|\Delta h(y)|
f Y ( y ) = f X ( x ) ∣ ∣ ∣ d y d x ∣ ∣ ∣ = f X ( h ( y ) ) ∣ ∣ ∣ d y d h ( x ) ∣ ∣ ∣ = f X ( h ( y ) ) ∣ Δ h ( y ) ∣
同理，一个y对应多个x时，X = h i ( Y ) X=h_i(Y) X = h i ( Y )
f Y ( y ) = ∑ f X ( h i ( y ) ) ∣ Δ h i ( y ) ∣ f_Y(y)=\sum f_X(h_i(y))|\Delta h_i(y)|
f Y ( y ) = ∑ f X ( h i ( y ) ) ∣ Δ h i ( y ) ∣
**算法过程**
这里是一对二的情况。
Step1. Draw Z ∼ U(0, 1) and independently draw Y ∼ fY (·);
Step2. Set X1 = h1(Y) and X2 = h2(Y);
Step3. If Z ≤ { 1 + f X ( X 2 ) f X ( X 1 ) ∣ Δ G ( X 1 ) Δ G ( X 2 ) ∣ } Z\leq \{1+\frac{f_X(X_2)}{f_X(X_1)}\Big|\frac{\Delta G(X_1)}{\Delta G(X_2)}\Big|\} Z ≤ { 1 + f X ( X 1 ) f X ( X 2 ) ∣ ∣ ∣ Δ G ( X 2 ) Δ G ( X 1 ) ∣ ∣ ∣ }
**Note.**
逆方法可以看作 SR 方法的特例：没有 Step3；
beta 分布可以通过 gamma 分布生成;
inverse gamma 通过 gamma;
chi-squared and log-normal via normal;
Student’s t- and F-distribution via normal and chi-squared；
Dirichlet distribution 可以通过 independent gamma distributions;
multivariate normal distribution 通过 uni-normal；
multivariate t-distribution 通过 multinormal and chi-squared;
Wishart 通过 multinormal。
假设对于一个随机变量 X ⃗ = ( X 1 , … , X d ) T \vec X=(X_1,\ldots,X_d)^T X = ( X 1 , … , X d ) T
X ⃗ ∼ f X ( x ⃗ ) = f X ( x 1 , x 2 , … , x d ) where S X is finite \vec X \sim f_X(\vec x)=f_X(x_1,x_2,\ldots,x_d)\quad \text{where $S_X$ is finite}
X ∼ f X ( x ) = f X ( x 1 , x 2 , … , x d ) where S X is finite
该式可以写作f X ( x 1 , x 2 , … , x d ) = f 1 ( x 1 ) ∏ i = 2 d f i ( x i ∣ x 1 , x 2 , … , x i − 1 ) f_X(x_1,x_2,\ldots,x_d)=f_1(x_1)\prod_{i=2}^d f_i(x_i|x_1,x_2,\ldots,x_{i-1}) f X ( x 1 , x 2 , … , x d ) = f 1 ( x 1 ) ∏ i = 2 d f i ( x i ∣ x 1 , x 2 , … , x i − 1 )
**算法过程**
Step1. Draw X1 from f1(x1);
Step2. Draw X2 from f2(x2|x1);
Step3. Draw X3 from f3(x3|x1, x2);
…
Stepi. Draw Xd from fd(xd|x1, x2, …, xd-1).
待更
待更
**What's the bootstrap?**
Bootstrap 是统计推断中基于数据计算的方法。
The bootstrap is a data-based method for statistical inference. Its introduction into statistics is relatively recent because the method is computationally intensive.
**The purpose of the bootstrap.**
Bootstrap提供了一种通用方法来获得估计量的标准误差估计（S e ^ ( θ ^ ) {\widehat {Se}(\hat \theta)} S e ( θ ^ )
Bootstrap同样应用于假设检验中。
First, the bootstrap approach provides a general method for obtaining estimated standard errors S e ^ ( θ ^ ) {\widehat {Se}(\hat \theta)} S e ( θ ^ ) confidence intervals C I CI C I
Second, the bootstrap approach can also be applied to testing hypotheses to calculate a bootstrap p-value or to provide an upper quantile point of the distribution of a test-statistic when its density is not available in closed-form .
**How to describe the parametric bootstrap for computing CIs of parameters of interest by one sentence?**
Suppose that we have a method to calculate the point estimator θ ^ \hat \theta θ ^
Thus, the key is how to generate a bootstrap sample.
Let { X i } i = 1 n {\{Xi\}}_{i=1}^{n} { X i } i = 1 n θ = P r ( X 1 = 1 ) \theta = Pr(X_1 =1) θ = P r ( X 1 = 1 )
Let X ⃗ = ( X 1 , ⋯ , X n ) T \vec{X}=(X_1,\cdots,X_n)^T X = ( X 1 , ⋯ , X n ) T x ⃗ = ( x 1 , ⋯ , x n ) T \vec{x}=(x_1,\cdots,x_n)^T x = ( x 1 , ⋯ , x n ) T
L ( θ ) = ∏ i = 1 n θ x i ( 1 − θ ) 1 − x i , 0 ≤ θ ≤ 1 L(\theta)=\prod_{i=1}^n{\theta^{x_i}(1-\theta)^{1-x_i}},\qquad 0\leq\theta\leq1
L ( θ ) = i = 1 ∏ n θ x i ( 1 − θ ) 1 − x i , 0 ≤ θ ≤ 1
so that the MLE of θ is the sample mean defined by:
θ ^ = 1 n ∑ i = 1 n x i = s ( x ⃗ ) \hat\theta=\frac{1}{n}\sum_{i=1}^n{x_i=s(\vec{x})}
θ ^ = n 1 i = 1 ∑ n x i = s ( x )
Note that
n θ ^ = ∑ i = 1 n X i ∼ B i n o m i a l ( n , θ ) n\hat\theta=\sum_{i=1}^n{X_i}\sim{Binomial(n,\theta)}
n θ ^ = i = 1 ∑ n X i ∼ B i n o m i a l ( n , θ )
then
E ( θ ^ ) = θ , a n d V a r ( θ ^ ) = θ ( 1 − θ ) / n E(\hat\theta)=\theta,\qquad and \qquad Var(\hat\theta)=\theta(1-\theta)/n
E ( θ ^ ) = θ , a n d V a r ( θ ^ ) = θ ( 1 − θ ) / n
According to Central Limit Theorem , we have
θ ^ − θ θ ( 1 − θ ) / n → Z ∼ N ( 0 , 1 ) \frac{\hat\theta-\theta}{\sqrt{\theta(1-\theta)/n}}\to Z \sim N(0,1)
θ ( 1 − θ ) / n θ ^ − θ → Z ∼ N ( 0 , 1 )
Namely, [ θ ^ − E ( θ ^ ) ] / [ V a r ( θ ^ ) ] 1 2 [\hat\theta-E(\hat\theta)]/[Var(\hat\theta)]^\frac{1}{2} [ θ ^ − E ( θ ^ ) ] / [ V a r ( θ ^ ) ] 2 1 coverges in distribution to a random variable following N(0,1). Based on limiting properties of MLE, we have approximately
θ ^ − θ θ ( 1 − θ ) / n ∼ N ( 0 , 1 ) a s n → ∞ \frac{\hat\theta-\theta}{\sqrt{\theta(1-\theta)/n}} \thicksim N(0,1)\qquad as \qquad n \to \infty
θ ( 1 − θ ) / n θ ^ − θ ∼ N ( 0 , 1 ) a s n → ∞
Let z α z_\alpha z α α \alpha α P r ( Z ≥ z α ) = α Pr(Z \geq z_\alpha)=\alpha P r ( Z ≥ z α ) = α 100 ( 1 − α ) % 100(1-\alpha)\% 1 0 0 ( 1 − α ) %
1 − α = P r ( − z α / 2 ≤ θ ^ − θ σ ^ ≤ z α / 2 ) = P r ( θ ^ − z α / 2 σ ^ ≤ θ ≤ θ ^ + z α / 2 σ ^ ) 1-\alpha=Pr(-z_{\alpha/2} \leq \frac{\hat\theta-\theta}{\hat\sigma} \leq z_{\alpha/2})=Pr(\hat\theta-z_{\alpha/2}\hat\sigma \leq \theta \leq \hat\theta + z_{\alpha/2}\hat\sigma)
1 − α = P r ( − z α / 2 ≤ σ ^ θ ^ − θ ≤ z α / 2 ) = P r ( θ ^ − z α / 2 σ ^ ≤ θ ≤ θ ^ + z α / 2 σ ^ )
Thus, [ θ ^ l , θ ^ u ] = [ θ ^ − z α / 2 σ ^ , θ ^ + z α / 2 σ ^ ] [\hat\theta_l, \hat\theta_u]=[\hat\theta-z_{\alpha/2}\hat\sigma,\hat\theta+z_{\alpha/2}\hat\sigma] [ θ ^ l , θ ^ u ] = [ θ ^ − z α / 2 σ ^ , θ ^ + z α / 2 σ ^ ] σ ^ = θ ^ ( 1 − θ ^ ) / n \hat\sigma=\sqrt{\hat\theta(1-\hat\theta)/n} σ ^ = θ ^ ( 1 − θ ^ ) / n
**Two problems with the asymptotic CI**
即便对于大容量样本，当θ真实值接近0时，CI下界会低于0；当θ真实值接近1时，CI下界会高于1。这是无效的。
对于小到中等容量样本，asymptotic CI不够可靠。
First, even though for large sample size n, the lower bound may be beyond zero when the true value of θ is close to zero while the upper bound may be beyond 1 when the true value of θ is near to 1.
Second, for small to moderate sample sizes, the asymptotic CI is not reliable.
假设有分布 x ∼ f ( x ; θ ) x \sim f(x;\theta) x ∼ f ( x ; θ )
**算法过程**
Step1. 计算参数的点估计(point estimator) θ ^ = s ( x ⃗ ) \hat\theta=s(\vec x) θ ^ = s ( x )
Step2. 生成独立同分布bootstrap sample X ⃗ ∗ = ( x 1 ∗ , ⋯ , x n ∗ ) T \vec X^*=(x_1^*,\cdots,x_n^*)^T X ∗ = ( x 1 ∗ , ⋯ , x n ∗ ) T { x i ∗ } i = 1 n ∼ f ( x ; θ ^ ) \{x_i^*\}_{i=1}^n\sim f(x;\hat\theta) { x i ∗ } i = 1 n ∼ f ( x ; θ ^ ) θ ^ ∗ = s ( x ⃗ ∗ ) \hat\theta^*=s(\vec x^*) θ ^ ∗ = s ( x ∗ )
Step3. 独立地重复G次 Step2 ，获得G bootstrap replication { θ ^ ∗ ( g ) } g = 1 G \{\hat\theta^*(g)\}_{g=1}^G { θ ^ ∗ ( g ) } g = 1 G
Step4. 与此同时，标准误差 S e ( θ ^ ) {Se}(\hat \theta) S e ( θ ^ )
S e ^ ∗ ( θ ^ ) = 1 G − 1 ∑ g = 1 G [ θ ^ ∗ ( g ) − θ ‾ ∗ ] 2 w h e r e θ ‾ ∗ = [ θ ^ ∗ ( 1 ) + ⋯ + θ ^ ∗ ( G ) ] / G \widehat {Se}^*(\hat \theta)=\sqrt{\frac{1}{G-1}\sum_{g=1}^G[\hat\theta^*(g)-\overline \theta^*]^2} \\ where \qquad \overline\theta^*=[\hat\theta^*(1)+\cdots+\hat\theta^*(G)]/G
S e ∗ ( θ ^ ) = G − 1 1 g = 1 ∑ G [ θ ^ ∗ ( g ) − θ ∗ ] 2 w h e r e θ ∗ = [ θ ^ ∗ ( 1 ) + ⋯ + θ ^ ∗ ( G ) ] / G
Step5. 如果{ θ ^ ∗ ( g ) } g = 1 G \{\hat\theta^*(g)\}_{g=1}^G { θ ^ ∗ ( g ) } g = 1 G 100 ( 1 − α ) % 100(1-\alpha)\% 1 0 0 ( 1 − α ) % θ \theta θ [ θ ^ l ∗ , θ ^ u ∗ ] = [ θ ‾ ∗ − z α / 2 S e ^ ∗ ( θ ^ ) , θ ‾ ∗ + z α / 2 S e ^ ∗ ( θ ^ ) ] [\hat\theta_l^*, \hat\theta_u^*]=[\overline\theta^*-z_{\alpha/2}\widehat {Se}^*(\hat \theta),\overline\theta^*+z_{\alpha/2}\widehat {Se}^*(\hat \theta)] [ θ ^ l ∗ , θ ^ u ∗ ] = [ θ ∗ − z α / 2 S e ∗ ( θ ^ ) , θ ∗ + z α / 2 S e ∗ ( θ ^ ) ]
Note : in standard normal distribution, z α / 2 ≈ 1.96 z_{\alpha/2} \approx 1.96 z α / 2 ≈ 1 . 9 6
Step6. 如果远非正态分布或Step5 中的CI是无效的，则100 ( 1 − α ) % 100(1-\alpha)\% 1 0 0 ( 1 − α ) % θ \theta θ [ θ ^ L ∗ , θ ^ U ∗ ] [\hat\theta_L^*, \hat\theta_U^*] [ θ ^ L ∗ , θ ^ U ∗ ] θ ^ L ∗ \hat\theta_L^* θ ^ L ∗ { θ ^ ∗ ( g ) } g = 1 G \{\hat\theta^*(g)\}_{g=1}^G { θ ^ ∗ ( g ) } g = 1 G ( α / 2 ) G (\alpha/2)G ( α / 2 ) G θ ^ U ∗ \hat\theta_U^* θ ^ U ∗ ( 1 − α / 2 ) G (1-\alpha/2)G ( 1 − α / 2 ) G
For example , when α = 0.05 \alpha=0.05 α = 0 . 0 5 θ ^ L ∗ \hat\theta_L^* θ ^ L ∗ θ ^ U ∗ \hat\theta_U^* θ ^ U ∗ θ ^ ∗ ( 1 ) , ⋯ , θ ^ ∗ ( 1000 ) \hat\theta^*(1),\cdots,\hat\theta^*(1000) θ ^ ∗ ( 1 ) , ⋯ , θ ^ ∗ ( 1 0 0 0 )
**Why need the non-parametric bootstrap?**
In many real applications, the form of density function is unknown .
It is desirable to use a non-parametric bootstrap method to obtain the estimated standard error of an estimator (e.g., the least square estimator (LSE) ) or the BCI for a population parameter (e.g., the mean of population distribution).
The key is how to generate a bootstrap sample from the empirical cdf .
假设分布(the empirical cdf)形式为 x ∼ F ^ n ( x ; θ ) = F ^ n ( x ) = 1 n ∑ i = 1 n I ( x i ≤ x ) x \sim \hat F_n(x;\theta)=\hat F_n(x)=\frac{1}{n} \sum_{i=1}^n I_{(x_i \leq x)} x ∼ F ^ n ( x ; θ ) = F ^ n ( x ) = n 1 ∑ i = 1 n I ( x i ≤ x ) x 1 ≤ x 2 ≤ ⋯ ≤ x n x_1\leq x_2\leq \cdots \leq x_n x 1 ≤ x 2 ≤ ⋯ ≤ x n F ^ n \hat F_n F ^ n θ \theta θ θ ^ = T ( F ^ n ) = s ( x ⃗ ) \hat\theta=T(\hat F_n)=s(\vec x) θ ^ = T ( F ^ n ) = s ( x )
**算法过程**
Step1. 计算参数的点估计(point estimator) θ ^ = s ( x ⃗ ) \hat\theta=s(\vec x) θ ^ = s ( x )
Step2. 生成独立同分布bootstrap sample X ⃗ ∗ = ( x 1 ∗ , ⋯ , x n ∗ ) T \vec X^*=(x_1^*,\cdots,x_n^*)^T X ∗ = ( x 1 ∗ , ⋯ , x n ∗ ) T { x i ∗ } i = 1 n ∼ F ^ n ( x ) \{x_i^*\}_{i=1}^n\sim \hat F_n(x) { x i ∗ } i = 1 n ∼ F ^ n ( x ) θ ^ ∗ = s ( x ⃗ ∗ ) \hat\theta^*=s(\vec x^*) θ ^ ∗ = s ( x ∗ )
Step3. 独立地重复G次 Step2 ，获得G bootstrap replication { θ ^ ∗ ( g ) } g = 1 G \{\hat\theta^*(g)\}_{g=1}^G { θ ^ ∗ ( g ) } g = 1 G
Step4. 与此同时，标准误差S e ( θ ^ ) {Se}(\hat \theta) S e ( θ ^ )
S e ^ ∗ ( θ ^ ) = 1 G − 1 ∑ g = 1 G [ θ ^ ∗ ( g ) − θ ‾ ∗ ] 2 w h e r e θ ‾ ∗ = [ θ ^ ∗ ( 1 ) + ⋯ + θ ^ ∗ ( G ) ] / G \widehat {Se}^*(\hat \theta)=\sqrt{\frac{1}{G-1}\sum_{g=1}^G[\hat\theta^*(g)-\overline \theta^*]^2} \\ where \qquad \overline\theta^*=[\hat\theta^*(1)+\cdots+\hat\theta^*(G)]/G
S e ∗ ( θ ^ ) = G − 1 1 g = 1 ∑ G [ θ ^ ∗ ( g ) − θ ∗ ] 2 w h e r e θ ∗ = [ θ ^ ∗ ( 1 ) + ⋯ + θ ^ ∗ ( G ) ] / G
Step5. 如果{ θ ^ ∗ ( g ) } g = 1 G \{\hat\theta^*(g)\}_{g=1}^G { θ ^ ∗ ( g ) } g = 1 G 100 ( 1 − α ) % 100(1-\alpha)\% 1 0 0 ( 1 − α ) % θ \theta θ [ θ ^ l ∗ , θ ^ u ∗ ] = [ θ ‾ ∗ − z α / 2 S e ^ ∗ ( θ ^ ) , θ ‾ ∗ + z α / 2 S e ^ ∗ ( θ ^ ) ] [\hat\theta_l^*, \hat\theta_u^*]=[\overline\theta^*-z_{\alpha/2}\widehat {Se}^*(\hat \theta),\overline\theta^*+z_{\alpha/2}\widehat {Se}^*(\hat \theta)] [ θ ^ l ∗ , θ ^ u ∗ ] = [ θ ∗ − z α / 2 S e ∗ ( θ ^ ) , θ ∗ + z α / 2 S e ∗ ( θ ^ ) ]
Note : in standard normal distribution, z α / 2 ≈ 1.96 z_{\alpha/2} \approx 1.96 z α / 2 ≈ 1 . 9 6
Step6. 如果远非正态分布或Step5 中的CI是无效的，则100 ( 1 − α ) % 100(1-\alpha)\% 1 0 0 ( 1 − α ) % θ \theta θ [ θ ^ L ∗ , θ ^ U ∗ ] [\hat\theta_L^*, \hat\theta_U^*] [ θ ^ L ∗ , θ ^ U ∗ ] θ ^ L ∗ \hat\theta_L^* θ ^ L ∗ { θ ^ ∗ ( g ) } g = 1 G \{\hat\theta^*(g)\}_{g=1}^G { θ ^ ∗ ( g ) } g = 1 G ( α / 2 ) G (\alpha/2)G ( α / 2 ) G θ ^ U ∗ \hat\theta_U^* θ ^ U ∗ ( 1 − α / 2 ) G (1-\alpha/2)G ( 1 − α / 2 ) G
**注:**
从F ^ n ( x ) \hat F_n(x) F ^ n ( x ) x ⃗ ∗ = ( x 1 ∗ , ⋯ , x n ∗ ) T \vec x^*=(x_1^*,\cdots,x_n^*)^T x ∗ = ( x 1 ∗ , ⋯ , x n ∗ ) T x i ∗ x_i^* x i ∗ x 1 ∗ = x 7 , x 2 ∗ = x 3 , ⋯ , x n ∗ = x 7 x_1^*=x_7, x_2^*=x_3, \cdots,x_n^*=x_7 x 1 ∗ = x 7 , x 2 ∗ = x 3 , ⋯ , x n ∗ = x 7
待整理...
