### The Markov Property

A stochastic process (“a mathematical object usually defined as a collection of random variables”) is said to have the Markov Property if, conditional on the present value, the future is independent on the past.

Let’s firstly introduce some notation: let **S** be a countable set called the *state space* and let **X **= (X_{t}: t ≥ 0) be a sequence of random variables taking values in **S**.

Then, the sequence **X** is called a Markov Chain if it satisfies the **Markov Property**:

for all t ≥ 0 and all x_{0}, x_{1}, …, x_{t} ϵ S.

Notation is simplified in the case where the Markov chain is **homogeneous**. This is when for all i, j ϵ S, the conditional probability P(X_{t+1} = j | X_{t} = i) does not depend on the value of *t*.

**Examples**

**Branching Process**: The branching process is a simple model of the growth of a population; each member of the*n*^{th }generation has a number of offspring that is independent of the past; X_{n}= size of the n^{th}generation.**Random Walk**: A particle performs a random walk on the line: let Z_{1}, Z_{2}, …, be independent with P(Z_{i}= 1) = p and P(Z_{i}= -1) = 1-p, then X_{n}=

Z_{1}+ … + Z_{n}; at each epoch of time, it jumps a random distance that is independent of the previous jumps.**Poisson Process**: the Poisson process satisfies a Markov property in which time is a*continuous*variable rather than a discrete variable, and thus the Poisson process is an example of a*continuous*-time Markov chain; the Markov property still holds as arrivals after time*t*are independent of arrivals before this time.

### Two Quantities

For simplification (and as this is only an *intro* to Markov chains) we’ll assume that the Markov chains are homogeneous.

Two quantities that are needed in order to calculate the probabilities in a chain are the:

**transition matrix**: P = (p_{i,j}: i,j ϵ S) given by p_{i,j}= P(X_{1}= j | X_{0}= i);**initial distribution**: λ = (λ_{i}: i ϵ S) given by λ_{i}= P(X_{0}= i).

As we have assumed homogeneity we have that

p_{i,j} = P(Xn+1 = j | Xn = i) for n ≥ 0.

These quantities are characterised in the following way:

## Proposition:

a) The vector λ is a

distributionin that λ_{i}≥ 0 for i ϵ S and the sum of λ_{i}over i = 1.b) The matrix P = (p

_{i,j}) is astochastic matrixin that p_{i,j}≥ 0 for i, j ϵ S and the sum of p_{i,j}over j = 1 for i ϵ S (i.e. that P row sums to 1)## Proof:

a) As λ

_{i}is a probability, it is clearly non-negative. Additionally, the sum of λ_{i}over i = the sum of P(X_{0}= i) over i = 1.b) Since p

_{i,j}is a probability, it is non-negative. Finally, the sum of p_{i,j}over j = the sum of P(Xn+1 = j | Xn = i) over j = P(Xn+1 ϵ S | Xn = i) = 1.

Hope you enjoyed this brief introduction to a Markov Chain!

M x