Clear all symbols from previous evaluations to avoid problems

Clear["Global`*"]

Examples of simple Markov Chains

Auxiliary function for applying the transition matrix to the probability vector and printing results

ShowResultsPiW[pi_,W_,niter_]:=
Module[{Wfin,p,λ,v},
Print["W = ",MatrixForm[N[W]]];
p=pi;
Print["π(0) = ",p];
Wfin=IdentityMatrix[Length[p]];
Do[
Wfin=Wfin.W;
p=N[p.W];
Print["π(",k,") = ",p],
{k,1,niter}
];
Print["W^",niter," = ",MatrixForm[N[Wfin]]];
{λ,v}=Eigensystem[Transpose[W]];
Do[ 
If[
Abs[Total[v[[i]]]]>0.0001,v[[i]]=v[[i]]/Total[v[[i]]]],(* normalization *)
{i,1,2}
];
Print["Eigenvalues and left eigenvectors of W in rows:"];
Print[λ];
Print[MatrixForm[N[v]]];
Return[];
];

Problematic network

See Nezbeda, Kolafa and Kotrla: Introduction to Molecular Simulations, Prague 2002 (in Czech), page 46

Let there be a network which is not very reliable:
1) the probability the network will work tomorrow if it works today is 60%
2) and the probability the network will not work tomorrow if it does not work today is 70%.

Questions:
1) What is the probability the network will work in k days if it works today?
2) Does this probability depend on k? Does it depend on the initial state of the network?

(* vector pi contains probabilities whether the network works or doesn't *)(* initial state - the network certainly works today *)
pi={1.0,0.0};
(* transition matrix - probabilities of going from one state to another *)
W={{0.6,0.4},{0.3,0.7}} ;
ShowResultsPiW[pi,W,20]

Random flipping of a die over an edge

Let us have a regular six-sided die. Put it on the table and start flipping each time over a random edge. Thus
1) the probability the vertical side will appear at the top is 1/4
2) and the probability the top or bottom side will appear at the top is 0.

Questions:
1) What is the probability a certain side will appear at the top after k flips if there is 1 at the beginning?
2) Again: Does this probability depend on k? Does it depend on the initial state?

(* vector pi contains probabilities of each side to be at the top *)(* initial state - the side 1 is certaily up *)
n=6;
pi=ConstantArray[0.0,n];
pi[[1]]=1.0;
(* transition matrix - probabilities of going from one state to another *)
W=ConstantArray[1/4,{n,n}];
Do[
W[[i,i]]=0;
W[[i,n-i+1]]=0,
{i,1,n}
];
ShowResultsPiW[pi,W,20]

Random walk through n points on the line

Let us consider a random walk through n points on the line. A walker can go with the same probability to the left or right neighbor except at the end points where he must go to the only neighbour. 

Questions:
1) What is the probability of finding the walker at a given point after k steps if he starts at a certain point at the beginning?
2) And again: Does this probability depend on k? Does it depend on the initial state? (This time the results is different!)

(* vector pi contains probabilities of finding the walker at corresponding points *)
(* initial state *)
n=5;
pi=ConstantArray[0.0,n];
pi[[2]]=1.0;
(* transition matrix *)
W=ConstantArray[0.0,{n,n}];
W[[1,2]]=1.0;
W[[n,n-1]]=1.0;
Do[
W[[i,i-1]]=0.5;
W[[i,i+1]]=0.5,
{i,2,n-1}
];
ShowResultsPiW[pi,W,20]

Random walk through n points on the line - with possibility to stay at any point

Let us consider a random walk through n points on the line. A walker can go with the same probability to the nearest neighbors or stay at the same point where he is.

(* vector pi contains probabilities of finding the walker at corresponding points *)
(* initial state *)
n=5;
pi=ConstantArray[0.0,n];
pi[[3]]=1.0;
(* transition matrix - probabilities of going from one state to another *)
W=ConstantArray[0.0,{n,n}];
W[[1,1;;2]]=0.5;
W[[n,n-1;;n]]=0.5;
Do[
W[[i,i-1;;i+1]]=1.0/3,
{i,2,n-1}
];
ShowResultsPiW[pi,W,30]

Random walk through n points on the line - Metropolis algorithm

Let us again consider a random walk through n points on the line but this time we prescribe probabilities at each point and decide at each step whether the trial step is accepted or not in the way as in Metropolis algorithm.

(* vector pi contains probabilities of each side to be at the top *)
(* initial state *)
n=5;
pi=ConstantArray[0.0,n];
pi[[2]]=1.0;
(* prescribed probabilities at each point (~probability density) *)
p0 = {0.1,0.2,0.3,0.3,0.1};
(* transition matrix *)
W=ConstantArray[0.0,{n,n}];
(* off-diagonal elements *)
Do[
(* 1/3 for the probability of taking step to the right or to the left or staying *)
W[[i,i+1]]=If[p0[[i+1]]≥p0[[i]],1,p0[[i+1]]/p0[[i]]]/3;
W[[i+1,i]]=If[p0[[i]]≥p0[[i+1]],1,p0[[i]]/p0[[i+1]]]/3,
{i,1,n-1}
];
(* diagonal elements *)
W[[1,1]]=1-W[[1,2]];
Do[
W[[i,i]]=1-W[[i,i-1]]-W[[i,i+1]],
{i,2,n-1}
];
W[[n,n]]=1-W[[n,n-1]];
ShowResultsPiW[pi,W,30]