Content

Introduction to ARMA models Invertibility General ARMA models ACVF and ACF for a causal $ARMA(p, q)$ model <- Go Back

401-4623-00: Time Series Analysis

Section 6

ARMA Models

Swiss Federal Institute of Technology Zurich

Eidgenössische Technische Hochschule Zürich

Last Edit Date: 12/22/2024

Disclaimer and Term of Use:

1. We do not guarantee the accuracy and completeness of the summary content. Some of the course material may not be included, and some of the content in the summary may not be correct. You should use this file properly and legally. We are not responsible for any results from using this file

2. This personal note is adapted from Professor Fadoua Balabdaoui and Professor Nicolai Meinshausen. Please contact us to delete this file if you think your rights have been violated.

3. This work is licensed under a Creative Commons Attribution 4.0 International License.

---

Introduction to ARMA models

Definition: A time series $\{X_t\}_t$ is said to be an $ARMA(1, 1)$ process if it is stationary and satisfies the equations

$$\underbrace{X_t - \phi X_{t-1}}_{AR} = \underbrace{Z_t + \theta Z_{t-1}}_{MA}$$

for $t \in \mathbb{Z}$, $Z_t \in WN(0, \sigma^2)$, and $\theta, \phi \in \mathbb{R}$ such that $\theta + \phi \neq 0$.

We can rewrite $X_t - \phi X_{t-1} = Z_t + \theta Z_{t-1}$ as

$$\phi(B) X_t = \theta(B) Z_t$$

where $\phi(z) = 1 - \phi_z$ and $\theta(z) = 1 + \theta_z$.

Question: For what values of $\phi$ and $\theta$ the equation $X_t - \phi X_{t-1} = Z_t + \theta Z_{t-1}$ admit a solution?

1. First case - $|\phi| < 1$

   Recall that $X_t - \phi X_{t-1} = Z_t + \theta Z_{t-1} \Leftrightarrow \phi(B) X_t = \theta(B)Z_t$. We have already shown that $\chi(B) = \sum_{j=0}^{\infty} \phi^j B^j$ is the inverse of $\Phi(B)$ as of $\chi(B) \circ \Phi(B) = I$.

   Hence, it holds that

   $$ \begin{align} X_t &= \left( \chi(B) \circ \theta(B) \right) Z_t\\ &= \left( \sum_{j=0}^{\infty} \phi^j B^j \right) \left( 1 + \theta B \right) Z_t\\ &= \left( \sum_{j=0}^{\infty} \phi^j B^j + \theta \sum_{j=0}^{\infty} \phi^j B^{j+1} \right) Z_t\\ &= \left( \sum_{j=0}^{\infty} \phi^j B^j + \theta \sum_{j=1}^{\infty} \phi^{j-1}B^j \right) Z_t\\ &= \left( 1 + \sum_{j=1}^{\infty} (\phi^j + \theta \phi^{j-1}) B^j \right) Z_t\\ &= \left( 1 + (\phi + \theta) \sum_{j=1}^{\infty} \phi^{j-1} B^j \right) Z_t\\ &= Z_t + (\phi + \theta) \sum_{j=1}^{\infty} \phi^{j-1}Z_{z-j} \end{align} $$

2. Second case - $|\phi| > 1$

   We still have that

   $$\phi(B)X_t = \theta(B) Z_t.$$

   We need to invert $\phi(B) = 1 - \phi B$

   $$ \begin{align} \frac{1}{1 - \phi z} &= \frac{1}{\phi z (\frac{1}{\phi z} - 1)}\\ &= -\frac{1}{\phi z} \frac{1}{1 - \frac{1}{\phi z}}\\ &= -\frac{1}{\phi z} \sum_{j = 0}^{\infty} \left( \frac{1}{\phi z} \right)^j\\ &= -\sum_{j=0}^{\infty} \frac{1}{\phi^{j+1} z^{j+1}}\\ &= -\sum_{j=1}^{\infty} \frac{1}{\phi^j z^j}\\ &= \sum_{j=-\infty}^{\infty} \chi_j z^j \end{align} $$

   where $\chi_j = \begin{cases} -\phi^j & j \le 1\\ 0 & \text{otherwise} \end{cases}$.

   $$\sum_{j=-\infty}^{\infty} \left| \chi_j \right| = \sum_{j=-\infty}^{-1}\left| \phi \right|^j = \sum_{j=1}^{\infty} \frac{1}{|\phi|^j} < \infty$$

   as $\frac{1}{|\phi|} < 1$, hence, the inverse of $\phi(B)$ in case is given by $\chi(B) = -\sum_{j=1}^{\infty} \frac{1}{\phi^j} B^{-j}$, ($\chi(B) \circ \phi(B) = I$).

   This means that

   $$ \begin{align} &\phi(B) X_t = \theta(B) Z_t\\ \Rightarrow & X_t = (\chi(B) \circ \theta(B)) Z_t\\ &= - \left( \sum_{j=1}^{\infty} \frac{1}{\phi^j} B^{-j} \right) (1 + \theta B) Z_t\\ &= - \left( \sum_{j=1}^{\infty} \frac{1}{\phi^j} B^{-j} + \theta \sum_{j=1}^{\infty} \frac{1}{\phi^j} B^{-j+1} \right) Z_t\\ &= - \left( \sum_{j=1}^{\infty} \frac{1}{\phi^j} B^{-j} + \theta \sum_{j=0}^{\infty} \frac{1}{\phi^{j+1}} B^{-j} \right) Z_t\\ &= - \left( \frac{\theta}{\phi} \sum_{j=1}^{\infty} \left( \frac{1}{\phi^j} B^{-j} + \frac{\theta}{\phi^{j+1}} B^{-j} \right) \right) Z_t\\ &= - \left( \frac{\theta}{\phi} + (\phi + \theta) \sum_{j=1}^{\infty} \frac{1}{\phi^{j+1}} B^{-j} \right)Z_t\\ &= -\frac{\theta}{\phi} Z_t - (\phi + \theta) \sum_{j=1}^{\infty} \frac{1}{\phi^{j+1}} Z_{t+j} \end{align} $$

3. Third case - $\phi \in \{-1, 1\}$

   Assume that $\phi = 1$, for $k \in Z+$

   $$ \begin{align} X_t - X_{t-1} &= Z_t + \theta Z_{t-1}\\ X_{t-1} - X_{t-1} &= Z_{t-1} + \theta Z_{t-2}\\ &\vdots\\ X_{t-k+1} - X_{t-k} &= Z_{t-k+1} + \theta Z_{t-k} \end{align} $$

   Taking the sum yields

   $$ \begin{align} X_t - X_{t-k} &= \sum_{j=0}^{k-1} Z_{t-j} + \theta \sum_{j=1}^k Z_{t-j}\\ &= Z_t + \theta Z_{t-k} + (1+\theta) \sum_{j=1}^{k-1} Z_{t-j} \end{align} $$

   If $\{X_t\}_t$ were stationary, we would have

   • $\mathrm{Var}(X_t - X_{t-k}) = 2 \gamma_x(0) - 2\gamma_x(k) = \sigma^2 (1+\theta^2+(1+\theta)^2(k-1))$

   • $\gamma_x(k) = \gamma_x(0) - \frac{\sigma^2}{2}(1+\theta^2+(1+\theta)^2(k-1))$

   • $\lim_{k \rightarrow \infty} \gamma_x(k) = - \infty$ because $(1 + \theta)^2 > 0$

   which contradicts with the fact that $|\gamma_x(k)| \le \gamma_x(0)$.

   Hence, no stationary solution exists in this case. A similar reason can be applied for $\phi = -1$.

Conclusion: A stationary solution for the $ARMA(1, 1)$ model exists if and ony if $\phi \notin \{-1, 1\}$.

• For $|\phi| < 1$, the model is causal.

• For $|\phi| > 1$, the model is non-causal.

Invertibility

It means that the noise $\{Z_t\}_t$ depends only on the present and past terms of $\{X_t\}_t$.

Question: For what value of $\theta$, is the $ARMA(1, 1)$ model invertible?

We have $X_t - \phi X_{t - 1} = Z_t + \theta Z_{t-1} \Leftrightarrow Z_t - (-\theta) Z_{t-1} = X_t + (-\phi) X_{t-1}$.

We can put $\phi' = -\theta$ and $\theta' = -\phi$ and write $Z_t - \phi' Z_{t-1} = X_t + \theta X_{t-1}$.

1. First case - $|\phi'| < 1 \Leftrightarrow |\theta| < 1$

   We have (after switch the $AR$ and $MA$ parts) that

   $$ \begin{align} Z_t &= X_t + (\phi' + \theta') \sum_{j=1}^{\infty} (\phi')^{j-1} X_{t-j}\\ &= X_t - (\phi + \theta) \sum_{j=1}^{\infty} (-1)^{j-1} \theta^{j-1} X_{t-j} \end{align} $$

   This means the model is invertible in this case.

2. Second case - $|\phi'| > 1 \Leftrightarrow |\theta| > 1$

   $$ \begin{align} Z_t &= -\frac{\theta'}{\phi'} X_t - (\phi' + \theta') \sum_{j=1}^{\infty} \frac{1}{(\phi')^{j+1}} X_{t+j}\\ &= -\frac{\phi}{\theta} X_t + (\phi + \theta) \sum_{j=1}^{\infty} (-1)^{j+1} \frac{1}{\theta^{j+1}} X_{t + j} \end{align} $$

   This means the model is not invertible in this case.

3. Third case - $\phi' \in \{-1, 1\} \Leftrightarrow \theta \in \{-1, 1\}$

   We can show that $Z_t$ cannot be written as $\sum_{j=0}^{\infty} \psi_j X_{t-j}$ nor under $\sum_{j=0}^{\infty} \psi_j X_{t+j}$ for $\{\psi_j\}_j$ such that $\sum_{j=-\infty}^{\infty} |\psi_j| < 0$.

General ARMA models

Definition: A time series $\{X_t\}_t$ is said to be an $ARMA(p, q)$ process for some integers $p \ge 0$ and $q \ge 0$, if $\{X_t\}_t$ is stationary and satisfies the equations.

$$X_t - \phi_1 X_{t-1} - \cdots - \phi_p X_{t-p} = Z_t + \theta_1 Z_{t-1} + \cdots + \theta_q Z_{t-q}, \forall t \in \mathbb{Z}$$

where $\{Z_t\}_t \sim WN(0, \sigma^2)$ and $\phi_1, \cdots, \phi_p, \theta_1, \cdots, \theta_q \in \mathbb{R}$ such that

• $\phi_p \neq 0$ and $\phi_q \neq 0$

• the polynomials $\phi(z) = 1- \phi_1 z - \cdots - \phi_p z^p$ , $\theta(z) = 1 + \theta_1 z + \cdots + \theta_q z^q$ for $z \in \mathbb{C}$ have no common factors.

The equation can be rewritten as $\phi(B)X_t = \theta(B)Z_t$ with $B$ as the backward shift operator.

Since any complex polynomial admits zeros in $\mathbb{C}$, we can find $a_1, \cdots, a_p \in \mathbb{C}$ such that $\phi(z) = (1-a_1 z) \cdots (1-a_p z)$ and $\prod_{i=1}^p (-a_i) = -\phi_p$ or $\phi_p = (-1)^{p-1} \prod_{i=1}^p a_i$.

Note that $\frac{1}{a_1}, \cdots, \frac{1}{a_p}$ are the zeros of $z \rightarrow \phi(z)$ in $\mathbb{C}$. Similarly, $\exists b_1, \cdots, b_q \in \mathbb{C}$ such that $\theta(z) = (1 - b_1 z) \cdots (1 - b_1 z)$ and $\prod_{i = 1}^q (-b_i) = \theta_q$ or $\theta_q = (-1)^q \prod_{i = 1}^q b_i$. $\frac{1}{b_1}, \cdots, \frac{1}{b_q}$ are the zeros of $z \rightarrow \theta(z)$ in $\mathbb{C}$.

Remark:

• Saying that the polynomials $\phi$ and $\theta$ have no common factors is equivalent to saying that they do not have a common zero

  $$\left\{ \frac{1}{a_1}, \cdots, \frac{1}{a_p} \right\} \cap \left\{ \frac{1}{b_1}, \cdots, \frac{1}{b_p} \right\} = \emptyset.$$

• The zeros of $\phi$ and $\theta$ are not necessarily real numbers. However, since $\phi_1, \cdots, \phi_p \in \mathbb{R}$ and $\theta_1, \cdots, \theta_q \in \mathbb{R}$ is holds that:

  $\frac{1}{a_i} \in \mathbb{C}, \mathbb{R} \Rightarrow \frac{1}{\bar{a}_i}$ must be also a zero of $\phi$ and $\frac{1}{b_j} \in \mathbb{C}, \mathbb{R} \Rightarrow \frac{1}{\bar{b}_i}$ must be also a zero of $\theta$.

Example

1. Consider $\{X_t\}_t$ such that

   $$X_t + \frac{1}{4}X_{t-1} - \frac{3}{8}X_{t-2} = Z_t - 3Z_{t-1}, \forall t \in \mathbb{Z}.$$

   The AR part:

   $$ \begin{align} \Phi(z) &= -\frac{3}{8} z^2 + \frac{1}{4} z + 1\\ &= -\frac{1}{8} (3z^2 - 2z -8)\\ \end{align} $$

   solve the equation and get the roots of $\Phi$: $z_1 = -\frac{4}{3}$ and $z_2 = 2$.

   The MA part:

   We have that $\Phi(z) = \left( 1 + \frac{3}{4}z \right) \left( 1 - \frac{z}{2} \right)$ and $\{X_t\}_t$ satisfies

   $$\left( 1 + \frac{3}{4}B \right) \left( 1 - \frac{1}{2}B \right) X_t = (1-3B)Z_t.$$

   Now we need to find the inverse of $\Phi(B) = \left( 1 + \frac{3}{4}B \right)\left( 1 - \frac{1}{2}B \right)$

   $$\frac{1}{\Phi(z)} = \frac{1}{\left(1 + \frac{3}{4}z\right)\left(1 - \frac{1}{2}z\right)} = \frac{a}{1 + \frac{3}{4}z} + \frac{b}{1 - \frac{1}{2}z}$$

   from some $a$ and $b$ in $\mathbb{R}$.

   There are different ways to do this. The simplest one is to write

   $$\frac{a}{1 + \frac{3}{4}z} + \frac{b}{1 - \frac{1}{2}z} = \frac{a\left( 1 - \frac{1}{2}z \right) + b \left( 1 + \frac{3}{4}z \right)}{\left( 1 + \frac{3}{4}z \right)\left( 1 - \frac{1}{2}z \right)}$$

   Hence $\forall z \notin \{-\frac{4}{3}, 2\}$,

   $$ \begin{align} & a\left( 1 - \frac{1}{2}z \right) + b\left( 1 + \frac{3}{4}z \right) = 1\\ \Leftrightarrow & a + b + z \left( -\frac{a}{2} + \frac{3b}{4} \right) = 1\\ \Leftrightarrow & \begin{cases} a + b = 1 \\ -\frac{a}{2} + \frac{3b}{4} = 0 \end{cases}\\ \Leftrightarrow & \begin{cases} b = 1-a \\ -\frac{a}{2} + \frac{3}{4}(1-a) = 0 \end{cases}\\ \Leftrightarrow & \begin{cases} a = \frac{3}{5} \\ b=\frac{2}{5} \end{cases} \end{align} $$

   This implies that

   $$\frac{1}{\phi(z)} = \frac{3}{5} \sum_{j=0}^{\infty} \left(-\frac{3}{4}\right)^j z^j + \frac{2}{5} \sum_{j=0}^{\infty} \frac{1}{2^j} z^j$$

   for $z$, $\left| \frac{3}{4} z \right| < 1$ and $\left| \frac{z}{2} \right| < 1$ that is $\left| z \right| < \frac{4}{3}.$

   Hence, the inverse of $\Phi(B)$ is given by

   $$\chi(B) = \sum_{j=0}^{\infty} \left( \frac{3}{5} \left(-\frac{3}{4}\right)^j + \frac{2}{5} \frac{1}{2^j} \right) B^j$$

   It follows that

   $$ \begin{align} X_t &= \left( \sum_{j=0}^{\infty} \left( \frac{3}{5} \left(-\frac{3}{4}\right) + \frac{2}{5} \frac{1}{2^j} \right) B^j \right) (1 - 3B) Z_t\\ &= \left( \sum_{j=0}^{\infty} \left( \frac{3}{5} \left(-\frac{3}{4}\right) + \frac{2}{5} \frac{1}{2^j} \right) B^j - \sum_{j=0}^{\infty} \left( \frac{9}{5} \left(-\frac{3}{4}\right) + \frac{6}{5} \frac{1}{2^j} \right) B^{j+1} \right) Z_t\\ &= Z_t + \sum_{j=1}^{\infty} \left( \frac{3}{5} \left(-\frac{3}{4}\right) + \frac{2}{5} \frac{1}{2^j} - \frac{9}{5} \left(-\frac{3}{4}\right) - \frac{6}{5} \frac{1}{2^j} \right) Z_{t-j}\\ &= Z_t - \sum_{j=1}^{\infty} \left( \frac{9}{4} \left( -\frac{3}{4} \right)^{j - 1} + \frac{1}{2^{j-1}} \right) Z_{t - j}. \end{align} $$

   As a result, we have

   $$X_t = Z_t - \sum_{j=1}^{\infty} \left( \frac{9}{4} \left( -\frac{3}{4} \right)^{j - 1} + \frac{1}{2^{j-1}} \right) Z_{t - j}.$$

2. Consider $\{X_t\}_t$ such that

   $$X_t + \frac{1}{4} X_{t - 1} - \frac{3}{8} X_{t-2} = Z_t + \frac{1}{2}Z_{t-1} - \frac{1}{2}Z_{t-2}, \forall t \in \mathbb{Z}.$$

   The AR part:

   Same as the previous example: $z_1 = -\frac{4}{3}$, $z_2 = 2$.

   The MA part:

   $\theta (z) = -\frac{1}{2}z^2 + \frac{1}{2}z + 1 = -\frac{1}{2}(z^2 - z - 2)$

   Solve the equation and get the zeros of $\theta$, which are $z_1 = -1$ and $z_2 = 2$.

   Then $\theta(z) = \left( 1 - \frac{1}{2} z \right) \left( 1 + z \right)$. Using operations, we can rewrite the equations as

   $$\left(1 - \frac{1}{2}B \right) \left( 1 + \frac{3}{4} \right) X_t = \left( 1 - \frac{1}{2}B \right) (1 + B) Z_t.$$

   Let $\chi(B) = \sum_{j=0}^{\infty} \frac{1}{2^j} B^j$. Then $\chi(B)$ is the inverse of $\left( 1 - \frac{1}{2}B \right)$, or equivalently $\chi(B) \left( 1 - \frac{B}{2} \right) = 1$, the identity.

   Hence

   $$ \begin{align} &\chi(B)\left( 1 - \frac{1}{2}B \right) \left( 1 + \frac{3}{4}B \right)X_t = \chi(B) \left( 1 - \frac{1}{2} \right) (1+B) Z_t\\ \Leftrightarrow & \left( 1 + \frac{3}{4}B \right)X_t = (1 + B)Z_t. \end{align} $$

   Thus, the seemingly $ARMA(2, 2)$ process turns out to be an $ARMA(1, 1)$ process because of the common root 2.

   The inverse of $1 + \frac{3}{4}B$ is $\sum_{j=0}^{\infty} \left( -\frac{3}{4} \right)^j B^j$ and hence

   $$ \begin{align} X_t &= \left( \sum_{j=0}^{\infty} \left( -\frac{3}{4} \right)^j B^j \right) (1+B) Z_t\\ &= Z_t + \sum_{j=1}^{\infty} \left( \left(-\frac{3}{4}\right)^j + \left(-\frac{3}{4}\right)^{j-1} \right) Z_{t-j}\\ &= Z_t + \sum_{j=1}^{\infty} \left( -\frac{3}{4} \right)^{j-1} \left( 1 - \frac{3}{4} \right) Z_{t-j}\\ &= Z_t + \frac{1}{4} \sum_{j=1}^{\infty} \left( -\frac{3}{4} \right)^{j-1} Z_{t-j} \end{align} $$

   As a result, we have $X_t = Z_t + \frac{1}{4} \sum_{j=1}^{\infty} \left( -\frac{3}{4} \right)^{j-1} Z_{t-j}$.

3. Consider $\{X_t\}_t$ such that

   $$X_t - \frac{4}{3} X_{t-1} + \frac{1}{3} X_{t-2} = Z_t + \frac{1}{4} Z_{t-1}, t \in \mathbb{Z}.$$

   The AR part:

   $\Phi(z) = \frac{1}{3} z^2 - \frac{4}{3} z + 1 = \frac{1}{3}(z^2 - 4z + 3)$.

   Solving this equation and the zeros of $\Phi$ are $z_1 = 1$ and $z_2 = 3$.

   The MA part:

   Suppose there exists a stationary solution to $X_t - \phi_1 X_{t-1} - \cdots - \phi_p X_{t-p} = Z_t + \theta_1 Z_{t-1} + \cdots + \theta_q Z_{t-q}, \forall t \in \mathbb{Z}$. Then, this solution $\{X_t\}_t$ satisfies

   $$(1-B)\left( 1- \frac{1}{3}B \right)X_t = \left( 1 + \frac{1}{4}B \right)Z_t.$$

   Write $Y_t = \left(1 - \frac{1}{3} B\right)X_t$. We know that $\{Y_t\}_t$ should also be stationary. Furthermore, $\{Y_t\}_t$ satisfies

   $$(1 - B) \left( 1 - \frac{1}{3}B \right) X_t = (1 + \frac{1}{4} B)Z_t.$$

   Write $Y_t = \left( 1 - \frac{1}{3}B \right)X_t$. We know that $\{Y_t\}_t$ should also be stationary. Furthermore, $\{Y_t\}_t$ satisfies,

   $$(1 - B)Y_t = \left( 1 + \frac{1}{4} B \right)Z_t$$

   that is $ARMA(1, 1)$ equations with $\phi = 1$. We know that such equations do not admit a stationary solution.

Theorem

Consider the equations

$$X_t - \phi_1 X_{t-1} - \cdots - \phi_p X_{t-p} = Z_t + \theta_1 Z_{t-1} + \cdots + \theta_q Z_{t-q}, \forall t \in \mathbb{Z}$$

where $\{Z_t\}_t \sim WN(0, \sigma^2)$, $\phi_p \neq 0$, $\theta_q \neq 0$ and the polynomials $z \rightarrow \Phi(z)$ and $z \rightarrow \theta(z)$ have no common factors.

Then, these equations admit an almost surely unique stationary solution $\{X_t\}_t$ if and only if

$$\Phi(z) \neq 0, \forall z \in \mathbb{C} : |z| = 1,$$

that is $\Phi$ admits no root on the unit disk. Furthermore, $\{X_t\}_t$ is causal, that is, $X_t = \sum_{j=0}^{\infty} \psi_j Z_{t-j}, \forall t \in \mathbb{Z}$ for some real sequence $\{\psi_j\}_{j \ge 0}$ such that $\sum_{j=0}^{\infty} < \infty$ if and only if

$$\Phi(z) \neq 0, \forall z \in \mathbb{C} : |z| \le 1,$$

that is $\Phi$ admits no root in the unit disk.

Suppose that $\{X_t\}_t \sim ARMA(p, q)$ such that $\{X_t\}_t$ is causal. Then

$$\Phi(B) X_t = \theta(B)Z_t$$

$$X_t = \psi(B) Z_t$$

with $\Phi(z) = 1 - \phi_1 z - \cdots - \phi_p z^p$, $\theta(z) = 1 + \theta_1 z + \cdots + \theta_q z^q$, and $\psi(z) = \sum_{j=0}^{\infty} \psi_j z^j$ for some real sequence $\{\psi_j\}_j$ such that $\sum_{j=0}^{\infty} < \infty$.

Quesition: How do we determine $\psi_j, j \ge 0$?

Answer:

• If we can easily find the inverse of $\Phi(B)$, $\chi(B)$, say, then $\psi(B) = \chi(B) \theta(B)$ and we can use the expression

  $$\psi_i = \sum_{k = -\infty}^{\infty} \chi_k \theta_{j-k} = \sum_{k=-\infty}^{\infty} \theta_k \chi_{j-k}.$$

  Here $\theta_k = \begin{cases} 1 & k = 0\\ 0 & k \ge q + 1, k < 0 \end{cases}$.

• In general, we can write that

  $$\Phi(B) \psi(B) = \theta(B).$$

  Put $\phi_0 = -1$. This means that $\Phi(z) -\phi_0 - \phi_1 z - \cdots - \phi_p z^p$. Then

  $$\theta_j = \sum_{k=0}^p (-\Phi_k) \psi_{j-k}, \forall j \ge 0.$$

  Since $\psi_{j-k} = 0$, $\forall k$ such that $k > j$ we have that

  $$\theta_j = -\sum_{k=0}^{\min(p, j) \phi_k \psi_{j-k}}, \forall j \ge 0.$$

  For $j = 0$: $\theta_0 = 1 = -\phi_0 \psi_0 = \psi_0 \Leftrightarrow \psi_0 = 1$

  For $j > 0$: $\theta_j = -\phi_0 \psi_{j} - \sum_{k = 1}^{\min(p, j)} \phi_k \psi_{j - k} = \psi_j - \sum_{k = 1}^{\min(p, j)} \phi_k \psi_{j - k}$

  $$\Leftrightarrow \psi_j = \theta_j + \sum_{k=1}^{\min(p, j)} \phi_k \psi_{j-k}.$$

Remark: For $j \ge q + 1$, $\theta_j = 0$ and hence $\psi_j = \sum_{k=1}^{\min(p, j)} \phi_k \psi_{j-k}$.

Recall that an $ARMA(p,q)$ model is invertible if there exists a real sequence $\{X_j\}_{j\ge 0}$ such that $\sum_{j=0} |\pi_j| < \infty$ and $Z_t = \sum_{j=0}^{\infty} \pi_j X_{t-j}, \forall t \in \mathbb{Z}$.

Theorem

An $ARMA(p,q)$ model is invertible if and only if the polynomial $\theta$ admits no zero (root) in the unit disk, that is, $\theta(z) \neq 0, \forall z \in \mathbb{C} : |z| \le 1$.

Question: How can we determine $\pi_j$, $j \ge 0$?

Answer: The easiest way to do this is to switch the roles of the $AR$ and $MA$ parts and write

$$Z_t - \phi_1' Z_{t-1} - \cdots - \phi_q' Z_{t-q} = X_t + \theta_1'X_{t-p} + \cdots + \theta_p' X_{t-p}$$

with $\phi_i' = -\theta_i$ for $1 \le i \le q$ and $\theta_i' = -\phi_j$ for $1 \le j \le p$.

Then $\theta_j' = \sum_{k=0}^{\min(q, j)} \phi_k' \pi_{j-k} \Leftrightarrow -\phi_j = \sum_{k=0}^{\min(q, j)} \theta_k \pi_{j-k}$, $\forall j \ge 0$.

• For $j = 0$, $\phi_0 = 1$, $1 = \theta_0 \pi_0 \Rightarrow \pi_0 = 1$.

• For $j \ge 1$, $-\phi_j = \pi_j + \sum_{k=1}^{\min(q, j)} \theta_k \pi_{j-k}$, hence $\pi_j = -\phi_j - \sum_{k=1}^{\min(q,j)} \theta_k \pi_{j-k}$.

• For $j \ge p + 1$, $\phi_j = 0$ and hence $\pi_j = -\sum_{k=1}^{\min(q, j)} \theta_k \pi_{j-k}$.

ACVF and ACF for a causal $ARMA(p, q)$ model

There are many ways of doing this. Let us now focus on a very straightforward approach.

We know that $X_t = \sum_{j=0}^{\infty} \psi_j Z_{t-j}, t \in \mathbb{Z}$, where $\begin{cases} \psi_j = \theta_j + \sum_{k=1}^{\min(p, j)} \phi_k \psi_{j-k} & \theta_0 = 1\\ \psi_0 = 0 & \theta_j = 0, \forall j \ge q+1 \end{cases}$.

This gives a recursive way of computing the coefficients $\psi_j, j \ge 0$. Then, $\forall h \in \mathbb{Z}$, plugged in the expressions (already established for linear processes)

$$\gamma_x(h) = \sigma^2 \sum_{j=0}^{\infty} \psi_j \psi_{j+|h|}$$

$$\rho_x(h) = \frac{\sum_{j=0}^{\infty} \psi_j \psi_{j+|h|}}{\sum_{j=0}^{\infty} \psi_j^2}$$

Example

1. Let $\phi \in (-1, 1)$ and $\theta: \theta \neq -\phi$. Consider the $ARMA(1, 1)$ equations

   $$X_t - \phi X_{t - 1} = Z_t + \theta Z_{t - 1}, t \in \mathbb{Z}$$

   where $\{Z_t\}_t \sim WN(0, \sigma^2)$.

   We have already established that

   $$X_t = Z_t + (\phi + \theta) \sum_{j=1}^{\infty} \phi^{j-1} Z_{t-j}, t \in \mathbb{Z}.$$

   Then

   $$ \psi_j = \begin{cases} 1 & j = 0\\ (\phi + \theta)\phi^{j-1} & j \ge 1 \end{cases} $$

   Therefore

   $$ \begin{align} \gamma_x(0) &= \sigma^2 \sum_{j=0}^{\infty} \psi_j^2\\ &= \sigma^2 \left( 1 + \sum_{j = 1}^{\infty} \psi_j^2 \right)\\ &= \sigma^2 \left( 1 + (\phi + \theta)^2 \sum_{j=1}^{\infty} (\phi^2)^{j-1} \right)\\ &= \sigma^2 \left( 1 + \frac{(\phi + \theta)^2}{1 - \phi^2} \right) \end{align} $$

   For $|h| \ge 1$

   $$ \begin{align} \gamma_x(h) &= \sigma^2 \left( (\phi + \theta) \phi^{|h|-1} + \sum_{j=1}^{\infty} (\phi+\theta)^2 \phi^{j - 1} \phi^{j+|h|-1} \right)\\ &= \sigma^2 \left( (\phi + \theta) \phi^{|h| - 1} + \frac{(\phi + \theta)^2 \phi^{|h|}}{1 - \phi^2} \right)\\ &= \sigma^2 (\phi + \theta) \phi^{|h| - 1} \left( 1 + \frac{(\phi + \theta)^2}{1 - \phi^2} \right)\\ &= \frac{\sigma^2 (\phi + \theta) \phi^{|h|-1} (1 + \phi\theta)}{1 - \phi^2} \end{align} $$

   Note that $\gamma_x(h) = \phi^{|h| - 1} \gamma_x(1)$ where $\gamma_x(1) = \frac{\sigma^2 (\phi + \theta) (1 + \phi \theta)}{1 + \phi^2}$.

   $$\rho_x(h) = \begin{cases} 1 & h = 0\\ \phi^{|h| - 1}\rho_x(1) & |h| \ge 1 \end{cases}$$

   with $\rho_x(1) = \frac{(\phi + \theta)(1 + \phi \theta)}{1 + 2\phi \theta + \theta^2}$.

2. Consider the $MA(q)$ process

   $$X_t = Z_t + \theta_1 Z_{t - 1} + \cdots + \theta_q Z_{t-q}$$

   with $\{Z_t\}_t \sim WN(0, \sigma^2)$.

   Then $X_t = \sum_{j = 0}^{\infty} \psi_j Z_{t - j}$ with $\psi_j = \begin{cases} \theta_j & j = 0, \cdots, q \\ 0 & \text{otherwise} \end{cases}$, $\theta_0 = 1$.

   $$ \begin{align} \gamma_x(h) &= \sigma^2 \sum_{j = 0}^{\infty} \psi_j \psi_{j+|h|}\\ &= \sigma^2 \sum_{j=0}^{\infty} \mathbb{1}\{0\le j\le q\} \theta_{j + |h|} \mathbb{1}\{0 \le j + |h| \le q \}\\ &= \sigma^2 \sum_{j=0}^{\infty} \theta_j \theta_{j+|h|} \mathbb{1}\{0 \le j \le q, 0 \le j + |h| \le q\}\\ &= \begin{cases} 0 & |h| > q\\ \sigma^2 \sum_{j=0}^{q-|h|} \theta_j \theta_{j+|h|} & |h| \le q\end{cases}\\ \rho_x(h) &= \begin{cases} \frac{\sum_{j = 1}^{q - |h|} \theta_j \theta_{j+|h|}}{\sum_{j = 0}^{q} \theta_j^2} & |h| \le q\\ 0 & \text{otherwise} \end{cases} \end{align} $$

© Sample Academy