Read Sections #29.1 (Stochastic Processes) and #29.2 (Markov Chains), and #29.3 (Chapman- Kolmogorov Equations), of the attached, and write a summary report.
Note that the summary report has to be prepared on a word processor (e.g., MS Word), and it has to be submitted through our class canvas system. Your report will be formatted with the following traits: ? The title page should include course title, student name, and the date. ? There is no page limit but the article summary should be at least 3 pages long, single spaced
throughout. ? Use a standard font (Times New Roman 12). ? Use 1 inch margins for top, bottom, left, and right. ? Use proper punctuation, spelling, and grammar. ? All pages (with the exception of the title page) should be numbered.
1
29C H A P T E R Markov Chains
Chapter 16 focused on decision making in the face of uncertainty about one futureevent (learning the true state of nature). However, some decisions need to take into account uncertainty about many future events. We now begin laying the groundwork for decision making in this broader context.
In particular, this chapter presents probability models for processes that evolve over time in a probabilistic manner. Such processes are called stochastic processes. After briefly introducing general stochastic processes in the first section, the remainder of the chapter focuses on a special kind called a Markov chain. Markov chains have the special prop- erty that probabilities involving how the process will evolve in the future depend only on the present state of the process, and so are independent of events in the past. Many processes fit this description, so Markov chains provide an especially important kind of probability model.
For example, Chap. 17 mentioned that continuous-time Markov chains (described in Sec. 29.8) are used to formulate most of the basic models of queueing theory. Markov chains also provided the foundation for the study of Markov decision models in Chap. 19. There are a wide variety of other applications of Markov chains as well. A considerable number of books and articles present some of these applications. One is Selected Refer- ence 4, which describes applications in such diverse areas as the classification of customers, DNA sequencing, the analysis of genetic networks, the estimation of sales demand over time, and credit rating. Selected Reference 6 focuses on applications in fi- nance and Selected Reference 3 describes applications for analyzing baseball strategy. The list goes on and on, but let us turn now to a description of stochastic processes in general and Markov chains in particular.
? 29.1 STOCHASTIC PROCESSES A stochastic process is defined as an indexed collection of random variables {Xt}, where the index t runs through a given set T. Often T is taken to be the set of non- negative integers, and Xt represents a measurable characteristic of interest at time t. For example, Xt might represent the inventory level of a particular product at the end of week t.
hil23453_ch29_001-036.qxd 1/22/1970 10:52 PM Page 1
Confirming Pages
2 CHAPTER 29 MARKOV CHAINS
Stochastic processes are of interest for describing the behavior of a system operating over some period of time. A stochastic process often has the following structure.
The current status of the system can fall into any one of M 1 mutually exclusive cate- gories called states. For notational convenience, these states are labeled 0, 1, . . . , M. The random variable Xt represents the state of the system at time t, so its only possible values are 0, 1, . . . , M. The system is observed at particular points of time, labeled t 0, 1, 2, . . . . Thus, the stochastic process {Xt} {X0, X1, X2, . . .} provides a mathematical representation of how the status of the physical system evolves over time.
This kind of process is referred to as being a discrete time stochastic process with a finite state space. Except for Sec. 29.8, this will be the only kind of stochastic process con- sidered in this chapter. (Section 29.8 describes a certain continuous time stochastic process.)
A Weather Example
The weather in the town of Centerville can change rather quickly from day to day. However, the chances of being dry (no rain) tomorrow are somewhat larger if it is dry today than if it rains today. In particular, the probability of being dry tomorrow is 0.8 if it is dry today, but is only 0.6 if it rains today. These probabilities do not change if information about the weather before today is also taken into account.
The evolution of the weather from day to day in Centerville is a stochastic process. Starting on some initial day (labeled as day 0), the weather is observed on each day t, for t 0, 1, 2, . . . . The state of the system on day t can be either
State 0 Day t is dry
or
State 1 Day t has rain.
Thus, for t 0, 1, 2, . . . , the random variable Xt takes on the values,
Xt The stochastic process {Xt} {X0, X1, X2, . . .} provides a mathematical representation of how the status of the weather in Centerville evolves over time.
An Inventory Example
Daves Photography Store has the following inventory problem. The store stocks a par- ticular model camera that can be ordered weekly. Let D1, D2, . . . represent the demand for this camera (the number of units that would be sold if the inventory is not depleted) during the first week, second week, . . . , respectively, so the random variable Dt (for t 1, 2, . . .) is
Dt number of cameras that would be sold in week t if the inventory is not depleted. (This number includes lost sales when the inventory is depleted.)
It is assumed that the Dt are independent and identically distributed random variables hav- ing a Poisson distribution with a mean of 1. Let X0 represent the number of cameras on hand at the outset, X1 the number of cameras on hand at the end of week 1, X2 the num- ber of cameras on hand at the end of week 2, and so on, so the random variable Xt (for t 0, 1, 2, . . .) is
Xt number of cameras on hand at the end of week t.
if day t is dry if day t has rain.
0 1
hil23453_ch29_001-036.qxd 1/22/1970 10:52 PM Page 2
Confirming Pages
29.2 MARKOV CHAINS 3
Assume that X0 3, so that week 1 begins with three cameras on hand.
{Xt} {X0, X1, X2, . . .}
is a stochastic process where the random variable Xt represents the state of the system at time t, namely,
State at time t number of cameras on hand at the end of week t.
As the owner of the store, Dave would like to learn more about how the status of this sto- chastic process evolves over time while using the current ordering policy described below.
At the end of each week t (Saturday night), the store places an order that is delivered in time for the next opening of the store on Monday. The store uses the following order policy:
If Xt 0, order 3 cameras. If Xt 0, do not order any cameras.
Thus, the inventory level fluctuates between a minimum of zero cameras and a maximum of three cameras, so the possible states of the system at time t (the end of week t) are
Possible states 0, 1, 2, or 3 cameras on hand.
Since each random variable Xt (t 0, 1, 2, . . .) represents the state of the system at the end of week t, its only possible values are 0, 1, 2, or 3. The random variables Xt are dependent and may be evaluated iteratively by the expression
Xt1 for t 0, 1, 2, . . . .
These examples are used for illustrative purposes throughout many of the following sections. Section 29.2 further defines the particular type of stochastic process considered in this chapter.
if Xt 0 if Xt 1,
max{3 Dt1, 0} max{Xt Dt1, 0}
? 29.2 MARKOV CHAINS
Assumptions regarding the joint distribution of X0, X1, . . . are necessary to obtain ana- lytical results. One assumption that leads to analytical tractability is that the stochastic process is a Markov chain, which has the following key property:
A stochastic process {Xt} is said to have the Markovian property if P{Xt1 j?X0 k0, X1 k1, . . . , Xt1 kt1, Xt i} P{Xt1 j?Xt i}, for t 0, 1, . . . and every sequence i, j, k0, k1, . . . , kt1.
In words, this Markovian property says that the conditional probability of any future event, given any past events and the present state Xt i, is independent of the past events and depends only upon the present state.
A stochastic process {Xt} (t 0, 1, . . .) is a Markov chain if it has the Markovian property.
The conditional probabilities P{Xt1 j?Xt i} for a Markov chain are called (one- step) transition probabilities. If, for each i and j,
P{Xt1 j?Xt i} P{X1 j?X0 i}, for all t 1, 2, . . . ,
then the (one-step) transition probabilities are said to be stationary. Thus, having stationary transition probabilities implies that the transition probabilities do not change
hil23453_ch29_001-036.qxd 1/22/1970 10:52 PM Page 3
Confirming Pages
4 CHAPTER 29 MARKOV CHAINS
1For n 0, pij (0) is just P{X0 j?X0 i} and hence is 1 when i j and is 0 when i j.
over time. The existence of stationary (one-step) transition probabilities also implies that, for each i, j, and n (n 0, 1, 2, . . .),
P{Xtn j?Xt i} P{Xn j?X0 i}
for all t 0, 1, . . . . These conditional probabilities are called n-step transition probabilities. To simplify notation with stationary transition probabilities, let
pij P{Xt1 j?Xt i}, pij
(n) P{Xtn j?Xt i}.
Thus, the n-step transition probability pij (n) is just the conditional probability that the sys-
tem will be in state j after exactly n steps (time units), given that it starts in state i at any time t. When n 1, note that pij
(1) pij 1.
Because the pij (n) are conditional probabilities, they must be nonnegative, and since
the process must make a transition into some state, they must satisfy the properties
pij (n) 0, for all i and j; n 0, 1, 2, . . . ,
and
M
j0 pij
(n) 1 for all i; n 0, 1, 2, . . . .
A convenient way of showing all the n-step transition probabilities is the n-step transition matrix
State 0 1 M
P(n)
Note that the transition probability in a particular row and column is for the transition from the row state to the column state. When n 1, we drop the superscript n and sim- ply refer to this as the transition matrix.
The Markov chains to be considered in this chapter have the following properties:
1. A finite number of states. 2. Stationary transition probabilities.
We also will assume that we know the initial probabilities P{X0 i} for all i.
Formulating the Weather Example as a Markov Chain
For the weather example introduced in the preceding section, recall that the evolution of the weather in Centerville from day to day has been formulated as a stochastic process {Xt} (t 0, 1, 2, . . .) where
Xt 0 if day t is dry1 if day t has rain.
? ? ? ? ? ?
p(n)0M p(n)1M
p(n)MM
p01 (n)
p11 (n)
p(n)M1
p00 (n)
p10 (n)
p(n)M0
? ? ? ? ? ?
0
1
M
hil23453_ch29_001-036.qxd 1/22/1970 10:52 PM Page 4
Confirming Pages
29.3 MARKOV CHAINS 5
P{Xt1 0?Xt 0} 0.8,
P{Xt1 0?Xt 1} 0.6.
Furthermore, because these probabilities do not change if information about the weather before today (day t) is also taken into account,
P{Xt1 0?X0 k0, X1 k1, . . . , Xt1 kt1, Xt 0} P{Xt1 0?Xt 0} P{Xt1 0?X0 k0, X1 k1, . . . , Xt1 kt1, Xt 1} P{Xt1 0?Xt 1}
for t 0, 1, . . . and every sequence k0, k1, . . . , kt1. These equations also must hold if Xt1 0 is replaced by Xt1 1. (The reason is that states 0 and 1 are mutually exclusive and the only possible states, so the probabilities of the two states must sum to 1.) There- fore, the stochastic process has the Markovian property, so the process is a Markov chain.
Using the notation introduced in this section, the (one-step) transition probabilities are
p00 P{Xt1 0?Xt 0} 0.8, p10 P{Xt1 0?Xt 1} 0.6
for all t 1, 2, . . . , so these are stationary transition probabilities. Furthermore,
p00 p01 1, so p01 1 0.8 0.2, p10 p11 1, so p11 1 0.6 0.4.
Therefore, the (one-step) transition matrix is
P where these transition probabilities are for the transition from the row state to the column state. Keep in mind that state 0 means that the day is dry, whereas state 1 signifies that the day has rain, so these transition probabilities give the probability of the state the weather will be in tomorrow, given the state of the weather today.
The state transition diagram in Fig. 29.1 graphically depicts the same information provided by the transition matrix. The two nodes (circle) represent the two possible states for the weather, and the arrows show the possible transitions (including back to the same state) from one day to the next. Each of the transition probabilities is given next to the corresponding arrow.
The n-step transition matrices for this example will be shown in the next section.
1 0.2 0.4
0 0.8 0.6
State 0 1
1 p01 p11
0 p00 p10
State 0 1
10
0.2
0.6
0.8 0.4
? FIGURE 29.1 The state transition diagram for the weather example.
hil23453_ch29_001-036.qxd 1/22/1970 10:53 PM Page 5
Confirming Pages
6 CHAPTER 29 MARKOV CHAINS
Formulating the Inventory Example as a Markov Chain
Returning to the inventory example developed in the preceding section, recall that Xt is the number of cameras in stock at the end of week t (before ordering any more), so Xt represents the state of the system at time t (the end of week t). Given that the current state is Xt i, the expression at the end of Sec. 29.1 indicates that Xt1 depends only on Dt1 (the demand in week t 1) and Xt. Since Xt1 is independent of any past history of the inventory system prior to time t, the stochastic process {Xt} (t 0, 1, . . .) has the Markovian property and so is a Markov chain.
Now consider how to obtain the (one-step) transition probabilities, i.e., the elements of the (one-step) transition matrix
P
given that Dt1 has a Poisson distribution with a mean of 1. Thus,
P{Dt1 n} (1)
n
ne !
1
, for n 0, 1, . . . ,
so (to three significant digits)
P{Dt1 0} e 1 0.368,
P{Dt1 1} e 1 0.368,
P{Dt1 2} 1 2
e1 0.184,
P{Dt1 3} 1 P{Dt1 2} 1 (0.368 0.368 0.184) 0.080.
For the first row of P, we are dealing with a transition from state Xt 0 to some state Xt1. As indicated at the end of Sec. 29.1,
Xt1 max{3 Dt1, 0} if Xt 0.
Therefore, for the transition to Xt1 3 or Xt1 2 or Xt1 1,
p03 P{Dt1 0} 0.368, p02 P{Dt1 1} 0.368, p01 P{Dt1 2} 0.184.
A transition from Xt 0 to Xt1 0 implies that the demand for cameras in week t 1 is 3 or more after 3 cameras are added to the depleted inventory at the beginning of the week, so
p00 P{Dt1 3} 0.080.
For the other rows of P, the formula at the end of Sec. 29.1 for the next state is
Xt1 max {Xt Dt1, 0} if Xt 1.
This implies that Xt1 Xt, so p12 0, p13 0, and p23 0. For the other transitions,
p11 P{Dt1 0} 0.368,
p10 P{Dt1 1) 1 P{Dt1 0} 0.632,
p22 P{Dt1 0} 0.368,
p21 P{Dt1 1} 0.368,
p20 P{Dt1 2} 1 P{Dt1 1} 1 (0.368 0.368) 0.264.
? ? ? ? ? ?
3
p03 p13 p23 p33
2
p02 p12 p22 p32
1
p01 p11 p21 p31
0
p00 p10 p20 p30
? ? ? ? ? ??
State
0
1
2
3
hil23453_ch29_001-036.qxd 1/22/1970 10:53 PM Page 6
Confirming Pages
29.2 MARKOV CHAINS 7
For the last row of P, week t 1 begins with 3 cameras in inventory, so the calculations for the transition probabilities are exactly the same as for the first row. Consequently, the complete transition matrix (to three significant digits) is
P
The information given by this transition matrix can also be depicted graphically with the state transition diagram in Fig. 29.2. The four possible states for the number of cam- eras on hand at the end of a week are represented by the four nodes (circles) in the dia- gram. The arrows show the possible transitions from one state to another, or sometimes from a state back to itself, when the camera store goes from the end of one week to the end of the next week. The number next to each arrow gives the probability of that particular transition occurring next when the camera store is in the state at the base of the arrow.
Additional Examples of Markov Chains
A Stock Example. Consider the following model for the value of a stock. At the end of a given day, the price is recorded. If the stock has gone up, the probability that it will go up tomorrow is 0.7. If the stock has gone down, the probability that it will go up tomorrow is only 0.5. (For simplicity, we will count the stock staying the same as a decrease.) This is a Markov chain, where the possible states for each day are as follows:
State 0: The stock increased on this day. State 1: The stock decreased on this day.
The transition matrix that shows each probability of going from a particular state today to a particular state tomorrow is given by
? ? ? ? ? ?
3
0.368
0
0
0.368
2
0.368
0
0.368
0.368
1
0.184
0.368
0.368
0.184
0
0.080
0.632
0.264
0.080
? ? ? ? ? ?
State
0
1
2
3
0 1
2 3
0.080
0.080
0.184
0.368
0.368
0.368
0.3680.264 0.184
0.632
0.368
0.368
0.368
? FIGURE 29.2 The state transition diagram for the inventory example.
hil23453_ch29_001-036.qxd 1/22/1970 10:53 PM Page 7
Confirming Pages
P The form of the state transition diagram for this example is exactly the same as for
the weather example shown in Fig. 29.1, so we will not repeat it here. The only differ- ence is that the transition probabilities in the diagram are slightly different (0.7 replaces 0.8, 0.3 replaces 0.2, and 0.5 replaces both 0.6 and 0.4 in Fig. 29.1).
A Second Stock Example. Suppose now that the stock market model is changed so that the stocks going up tomorrow depends upon whether it increased today and yesterday. In particular, if the stock has increased for the past two days, it will increase tomorrow with probability 0.9. If the stock increased today but decreased yesterday, then it will increase tomorrow with probability 0.6. If the stock decreased today but increased yesterday, then it will increase tomorrow with probability 0.5. Finally, if the stock decreased for the past two days, then it will increase tomorrow with probability 0.3. If we define the state as repre- senting whether the stock goes up or down today, the system is no longer a Markov chain. However, we can transform the system to a Markov chain by defining the states as follows:2
State 0: The stock increased both today and yesterday. State 1: The stock increased today and decreased yesterday. State 2: The stock decreased today and increased yesterday. State 3: The stock decreased both today and yesterday.
This leads to a four-state Markov chain with the following transition matrix:
P
Figure 29.3 shows the state transition diagram for this example. An interesting feature of the example revealed by both this diagram and all the values of 0 in the transition matrix is that so many of the transitions from state i to state j are impossible in one step. In other words, pij 0 for 8 of the 16 entries in the transition matrix. However, check out how it always is possible to go from any state i to any state j (including j i) in two steps. The same holds true for three steps, four steps, and so forth. Thus, pij
(n) 0 for n 2, 3, . . . for all i and j.
A Gambling Example. Another example involves gambling. Suppose that a player has $1 and with each play of the game wins $1 with probability p 0 or loses $1 with probability 1 p 0. The game ends when the player either accumulates $3 or goes broke. This game is a Markov chain with the states representing the players current hold- ing of money, that is, 0, $1, $2, or $3, and with the transition matrix given by
P
? ? ? ? ? ?
3
0
0
p
1
2
0
p
0
0
1
0
0
1 p
0
0
1
1 p
0
0
? ? ? ? ? ?
State
0
1
2
3
? ? ? ? ? ?
3
0
0
0.5
0.7
2
0.1
0.4
0
0
1
0
0
0.5
0.3
0
0.9
0.6
0
0
? ? ? ? ? ?
State
0
1
2
3
1
0.3
0.5
0
0.7
0.5
State
0
1
8 CHAPTER 29 MARKOV CHAINS
2We again are counting the stock staying the same as a decrease. This example demonstrates that Markov chains are able to incorporate arbitrary amounts of history, but at the cost of significantly increasing the number of states.
hil23453_ch29_001-036.qxd 1/22/1970 10:53 PM Page 8
Confirming Pages
? 29.3 CHAPMAN-KOLMOGOROV EQUATIONS
29.3 CHAPMAN-KOLMOGOROV EQUATIONS 9
The state transition diagram for this example is shown in Fig. 29.4. This diagram demonstrates that once the process enters either state 0 or state 3, it will stay in that state forever after, since p00 1 and p33 1. States 0 and 3 are examples of what are called an absorbing state (a state that is never left once the process enters that state). We will focus on analyzing absorbing states in Sec. 29.7.
Note that in both the inventory and gambling examples, the numeric labeling of the states that the process reaches coincides with the physical expression of the systemi.e., actual in- ventory levels and the players holding of money, respectivelywhereas the numeric label- ing of the states in the weather and stock examples has no physical significance.
2 3
0.5
0.4
0.5
0.30.1
0.7
0 1 0.6
0.9
2 3 1
0 1 1-r
1-r
r
r
1
? FIGURE 29.3 The state transition diagram for the second stock example.
? FIGURE 29.4 The state transition diagram for the gambling example.
Section 29.2 introduced the n-step transition probability pij (n). The following Chapman-
Kolmogorov equations provide a method for computing these n-step transition probabilities:
pij (n)
M
k0 pik
(m)pkj (nm), for all i 0, 1, . . . , M,
j 0, 1, . . . , M, and any m 1, 2, . . . , n 1,
n m 1, m 2, . . . .3
3These equations also hold in a trivial sense when m 0 or m n, but m 1, 2, . . . , n 1 are the only interesting cases.
hil23453_ch29_001-036.qxd 1/22/1970 10:53 PM Page 9
Confirming Pages
10 CHAPTER 29 MARKOV CHAINS
These equations point out that in going from state i to state j in n steps, the process will be in some state k after exactly m (less than n) steps. Thus, pik
(m) pkj (nm) is just the con-
ditional probability that, given a starting point of state i, the process goes to state k after m steps and then to state j in n m steps. Therefore, summing these conditional proba- bilities over all possible k must yield pij
(n). The special cases of m 1 and m n 1 lead to the expressions
pij (n)
M
k0 pikpkj
(n1)
and
pij (n)
M
k0 pik
(n1)pkj,
for all states i and j. These expressions enable the n-step transition probabilities to be obtained from the one-step transition probabilities recursively. This recursive relationship is best explained in matrix notation (see Appendix 4). For n 2, these expressions become
pij (2)
M
k0 pikpkj, for all states i and j,
where the pij (2) are the elements of a matrix P(2). Also note that these elements are obtained
by multiplying the matrix of one-step transition probabilities by itself; i.e.,
P(2) P P P2.
In the same manner, the above expressions for pij (n) when m 1 and m n 1 indicate
that the matrix of n-step transition probabilities is
P(n) PP(n1) P(n1)P PPn1 Pn1P Pn.
Thus, the n-step transition probability matrix Pn can be obtained by computing the nth power of the one-step transition matrix P.
n-Step Transition Matrices for the Weather Example
For the weather example introduced in Sec. 29.1, we now will use the above formulas to calculate various n-step transition matrices from the (one-step) transition matrix P that was obtained in Sec. 29.2. To start, the two-step transition matrix is
P(2) P P . Thus, if the weather is in state 0 (dry) on a particular day, the probability of being in state 0 two days later is 0.76 and the probability of being in state 1 (rain) then is 0.24. Similarly, if the weather is in state 1 now, the probability of being in state 0 two days later is 0.72 whereas the probability of being in state 1 then is 0.28.
The probabilities of the state of the weather three, four, or five days into the future also can be read in the same way from the three-step, four-step, and five-step transition matrices calculated to three significant digits below.
0.76 0.24 0.72 0.28
0.8 0.2 0.6 0.4
0.8 0.2 0.6 0.4
hil23453_ch29_001-036.qxd 1/22/1970 10:53 PM Page 10
Confirming Pages
29.3 CHAPMAN-KOLMOGOROV EQUATIONS 11
P(3) P3 P P2 P(4) P4 P P3 P(5) P5 P P4 Note that the five-step transition matrix has the interesting feature that the two rows
have identical entries (after rounding to three significant digits). This reflects the fact that the probability of the weather being in a particular state is essentially independent of the state of the weather five days before. Thus, the probabilities in either row of this five-step transition matrix are referred to as the steady-state probabilities of this Markov chain.
We will expand further on the subject of the steady-state probabilities of a Markov chain, including how to derive them more directly, at the beginning of Sec. 29.5.
n-Step Transition Matrices for the Inventory Example
Returning to the inventory example included in Sec. 29.1, we now will calculate its n-step transition matrices to three decimal places for n = 2, 4, and 8. To start, its one-step transition matrix P obtained in Sec. 29.2 can be used to calculate the two-step transition matrix P(2)
Read more
Applied Sciences
Architecture and Design
Biology
Business & Finance
Chemistry
Computer Science
Geography
Geology
Education
Engineering
English
Environmental science
Spanish
Government
History
Human Resource Management
Information Systems
Law
Literature
Mathematics
Nursing
Physics
Political Science
Psychology
Reading
Science
Social Science
Home
Homework Answers
Blog
Archive
Tags
Reviews
Contact
twitterfacebook
Copyright © 2021 SweetStudy.com
