### The Markov Property

A stochastic process (“a mathematical object usually defined as a collection of random variables”) is said to have the Markov Property if, conditional on the present value, the future is independent on the past.

Let’s firstly introduce some notation: let **S** be a countable set called the *state space* and let **X **= (X_{t}: t ≥ 0) be a sequence of random variables taking values in **S**.

Then, the sequence **X** is called a Markov Chain if it satisfies the **Markov Property**:

for all t ≥ 0 and all x_{0}, x_{1}, …, x_{t} ϵ S.

Notation is simplified in the case where the Markov chain is **homogeneous**. This is when for all i, j ϵ S, the conditional probability P(X_{t+1} = j | X_{t} = i) does not depend on the value of *t*.

**Examples**

**Branching Process**: The branching process is a simple model of the growth of a population; each member of the *n*^{th }generation has a number of offspring that is independent of the past; X_{n} = size of the n^{th} generation.
**Random Walk**: A particle performs a random walk on the line: let Z_{1}, Z_{2}, …, be independent with P(Z_{i} = 1) = p and P(Z_{i} = -1) = 1-p, then X_{n} =

Z_{1} + … + Z_{n}; at each epoch of time, it jumps a random distance that is independent of the previous jumps.
**Poisson Process**: the Poisson process satisfies a Markov property in which time is a *continuous* variable rather than a discrete variable, and thus the Poisson process is an example of a *continuous*-time Markov chain; the Markov property still holds as arrivals after time *t* are independent of arrivals before this time.

### Two Quantities

For simplification (and as this is only an *intro* to Markov chains) we’ll assume that the Markov chains are homogeneous.

Two quantities that are needed in order to calculate the probabilities in a chain are the:

**transition matrix**: P = (p_{i,j}: i,j ϵ S) given by p_{i,j} = P(X_{1} = j | X_{0} = i);
**initial distribution**: λ = (λ_{i} : i ϵ S) given by λ_{i} = P(X_{0} = i).

As we have assumed homogeneity we have that

p_{i,j} = P(Xn+1 = j | Xn = i) for n ≥ 0.

These quantities are characterised in the following way:

#### Proposition:

a) The vector λ is a *distribution * in that λ_{i} ≥ 0 for i ϵ S and the sum of λ_{i} over i = 1.

b) The matrix P = (p_{i,j}) is a *stochastic matrix* in that p_{i,j} ≥ 0 for i, j ϵ S and the sum of p_{i,j} over j = 1 for i ϵ S (i.e. that P row sums to 1)

#### Proof:

a) As λ_{i} is a probability, it is clearly non-negative. Additionally, the sum of λ_{i} over i = the sum of P(X_{0} = i) over i = 1.

b) Since p_{i,j} is a probability, it is non-negative. Finally, the sum of p_{i,j} over j = the sum of P(Xn+1 = j | Xn = i) over j = P(Xn+1 ϵ S | Xn = i) = 1.

Hope you enjoyed this brief introduction to a Markov Chain!

M x