Decision Making II: POMDPs and Belief-Space Planning
Planning over beliefs instead of states: the alpha-vector value function, why exact solving explodes, and how sampled lookahead makes a robot pay to find out where it is.
Thus, even though the value function is defined over a continuum, it can be represented on a digital computer—up to the accuracy of floating point numbers.
In this chapter
Chapter 21 ended with a policy that is optimal and useless. It answers "what should I do in state ?" for every state in the map, and Rusty has never once been in a position to look up the answer, because Rusty does not know . It knows .
This chapter closes the book's central circle. The belief that Parts II through IV taught you to maintain becomes the thing you plan over. That single move — from planning in state space to planning in belief space — is what makes a robot pay for a sensor reading, hug a wall it could have driven past, or take a detour whose only payoff is finding out where it is. None of those behaviors is hand-coded here. They are what the Bellman equation returns once the state of the decision problem is a distribution.
The price is severe and worth naming up front: exact POMDP solving is combinatorial, not merely expensive. Most of this chapter is therefore about the four honest ways out — collapse the uncertainty (QMDP), sample the belief space (PBVI), sample the future (POMCP, DESPOT), or compress the belief to a statistic you can plan over (AMDP). Each one is exactly one paragraph of theory and one page of Rust, and each one fails somewhere you can see.
Two doors, and a microphone that costs a dollar
Rusty is in the Hallway in front of two identical doors. Behind one is the charging dock. Behind the other is an open stairwell. The doors look the same, the floor looks the same, and the map is no help: this corridor is symmetric.
There is a microphone. Pressing it costs one unit of battery and returns a direction — left or right — that is correct 85% of the time. Opening the right door earns ; opening the wrong one costs and a trip to the repair bench.
That is the whole problem, and it is the oldest toy in the field: Kaelbling, Littman and Cassandra's tiger problem (1998), which we will use for every number in this chapter because every number in it can be checked by hand. Play it before reading further. The widget opens on the optimal pilot; take control with You play and try to beat it.
Three things are worth noticing, and each is a section of this chapter.
Sensing is an action, and it has a price. Listening does not move the robot and does not open anything. Its entire value is that it moves the belief. In Chapter 21's MDP there was no such action — every action was worth something because of where it took you. Here an action is worth something because of what it tells you, and the Bellman equation has to be able to say so.
The value function lives over the belief, and it is made of straight lines. The lower panel is the actual object this chapter is about: plotted against . Each faint gray line is one -vector; the bold curve is their upper envelope. The policy is not stored anywhere — it is read off the envelope, one action per linear piece.
Certainty is worth money, and the shape of says how much. is convex. It is lowest in the middle, where the robot knows nothing, and highest at the ends, where it knows everything. The vertical gap between the chord and the curve is not a metaphor — it is the expected value of perfect information, in the same units as the reward. At the tiger's converged numbers give , so a free oracle would be worth just over nine points, and a microphone at one point per press is a bargain. That is why the optimal policy burns battery on presses that accomplish nothing physical.
Building intuition
Averaging over worlds is not planning under uncertainty
The tempting move — and the one almost every first implementation makes — is this: I do not know which world I am in, so let me solve each candidate world, then average the answers. Compute for the tiger-left world and the tiger-right world, weight by , act greedily.
That is a real algorithm, it is called QMDP, and we will implement it. It is also structurally incapable of the behavior in the widget above, for a reason the draft of Probabilistic Robotics states flatly: "the planning problem in partially observable environment cannot be solved by considering all possible environments and averaging the solution."
The reason is that averaging happens before the max. In each individual world the robot knows where the tiger is, so listening is a waste of a battery unit; the microphone earns its keep only in the counterfactual where you might have been wrong. Average first and that counterfactual is gone. Switch the widget's pilot to QMDP and watch the envelope collapse from nine linear pieces to three — and look at the middle one. It is horizontal: QMDP assigns listening the same value at every belief, because under its assumption the fog lifts anyway. A flat piece is an action whose worth does not depend on what you know, which is precisely an action with no informational value.
Why is convex, in one paragraph and no algebra
Suppose someone offers you a choice. Option A: you are told, honestly, which of two belief states or you are in — a coin decides, with probability and . Option B: you are told nothing, and you must act on the mixture .
Option A is worth , because you get to run the best policy for whichever one came up. Option B is worth , and whatever policy is best there was available under option A too. So
That is convexity, and it is the formal statement of information has non-negative value. The gap between the two sides at any belief is what the robot should be willing to pay to have its uncertainty resolved. Read the widget's envelope again with that in mind: the deeper the sag in the middle, the more the microphone is worth.
The mathematics
Notation
| Symbol | Meaning |
|---|---|
| Finite state, action and observation sets. Sizes |X|, |U|, |Z|. | |
| The belief — exactly bel(x) from Chapter 5, now serving as the state of a planning problem. | |
| The belief simplex. A segment for two states, a triangle for three. | |
| The belief transition: one Bayes-filter predict-and-correct, written as a function. | |
| Evidence under a belief — the branching probability of the belief MDP. Chapter 5’s η⁻¹. | |
| Belief reward. Linear in b, which is where the whole theory comes from. | |
| An α-vector: one linear piece of V, tagged with the action it commits to. | |
| The α-vector set at horizon T. V_T(b) = max_k ⟨α^(k), b⟩. | |
| Backprojection of α^(k) through (u, z): what that piece is worth seen from x, before observing. | |
| The AMDP compression: most likely state plus one number of uncertainty. |
Notation collision. -vectors are standard POMDP vocabulary and have nothing whatever to do with the motion-noise parameters of Chapter 9. We write value vectors with a superscript, , noise parameters with a subscript, , and never both in one equation.
The POMDP tuple
A partially observable Markov decision process is Chapter 21's MDP with one thing added and one thing taken away:
Added: the observation set and the measurement model — which is Chapter 10's object, now allowed to depend on which action you took, because "take a photograph" and "drive forward" do not return the same kind of reading. Taken away: the assumption that the agent is ever told . It sees only and , and starts from a prior .
Everything else is inherited. In particular is still Chapter 21's payoff, collected on taking in , and still buys convergence.
The tiger, completely specified, is the instance every number below refers to:
| — the tiger is behind the left or the right door | |
| — the growl came from the left or the right | |
| (the hearing-accuracy slider) | |
| correct door / tiger door | / |
| after opening | the tiger is re-placed uniformly and the next growl is pure noise, so the belief resets to |
The belief transition is the Bayes filter, wearing a new hat
Nothing new happens here at all, and that is the point worth making loudly. Given a belief, an action and an observation, the next belief is
which is Chapter 5's two lines — predict, then correct — with the integral become a sum. The normalizer is the evidence, and here it stops being housekeeping and becomes load-bearing, because it is the transition probability of the belief MDP:
Two consequences that shape everything downstream. First, the belief dynamics are deterministic given : all the stochasticity of a POMDP sits in which observation arrives. Second, the belief MDP's branching factor is , not — a tree over action–observation histories, which is exactly the object POMCP builds later in this chapter.
Run the tiger numbers by hand. From , hearing :
A second, consistent :
and one contradicting from there takes it straight back down. From , a single returns the belief to exactly — the two growls cancel, because the likelihood ratio is the reciprocal. That is the ladder the marker climbs in the widget above, and the whole optimal policy is a statement about where on that ladder you may stop.
The belief MDP
DerivationA POMDP is an MDP over the belief simplex
Statement. Define the belief MDP with
Then an optimal policy of this MDP, composed with the Bayes filter, is an optimal policy of the POMDP.
Step 1 — the belief is a sufficient statistic of the history. This is Chapter 5's completeness argument, not a new one: under the Markov assumption, is all of that matters for predicting and . The filter is a lossless compressor of history with respect to the future.
Step 2 — write the objective in terms of histories. A policy for a POMDP is by definition a map from histories to actions, and its expected return is with the expectation over states and observations. Condition the inner expectation on : since depends on only through its expectation under , the -th term is .
Step 3 — collapse histories to beliefs. By Step 1, two histories with the same belief have the same distribution over every future quantity, so no optimal policy can benefit from distinguishing them. The policy may therefore be taken to be a function of alone.
Step 4 — read off the induced process. Given and , the next belief is determined by which arrives, and arrives with probability . That is a bona fide MDP over the simplex, with reward . Chapter 21's Bellman equation applies verbatim, giving the result above.
What it costs. The state space is now a continuum — for states, a -dimensional simplex — so "for all " in Chapter 21's value-iteration sweep is now "for all points of a continuum". Every technique in this chapter is a way of coping with that one sentence.
The value function is piecewise-linear and convex
This is the theorem that makes the chapter possible — Sondik's, published with Smallwood in 1973. Every exact POMDP algorithm since is a way of maintaining the object it describes.
DerivationV_T is a maximum of finitely many linear functions
Statement. For every finite horizon there is a finite set of vectors with . Consequently is convex and piecewise linear.
Step 1 — the base case is the immediate reward. With one step to go there is nothing to anticipate: . So , of size . For the tiger these are three vectors you can write down without a computer:
Step 2 — substitute the inductive hypothesis into the Bellman backup.
Step 3 — the normalizer cancels. This is the trick. Write the inner product out and notice that 's normalizer is precisely the factor sitting in front of it:
A quantity that looked like (probability) × (value of a normalized posterior) is in fact linear in . Name that linear functional's coefficient vector the backprojection:
so that exactly. Read as: what is worth, seen from state , through the pair — the value discounted by how likely that observation was.
Step 4 — a sum of maxima is a max over tuples. Now
and since the -terms are independent, choosing the best for each separately is the same as choosing the best assignment jointly:
Step 5 — read off and count it. Each pair (action, assignment) contributes one vector, so
Step 6 — convexity is free. is a pointwise maximum of linear functions, and a pointwise maximum of convex functions is convex.
The interpretive payoff. An -vector is not "a value". It is a whole conditional plan: take action now, and if you then see , continue with the plan that represents. is that plan's expected return under , and picks the best plan for the belief you are actually in. That is why the envelope's breakpoints are the policy's decision thresholds: they are where one conditional plan overtakes another.
The explosion, watched live
The count in Step 5 is the whole story. For the tiger, , and then
This is doubly exponential in the horizon, on a problem with two states. It is not a scaling problem to be solved with a bigger machine; it is a combinatorial one, and every practical method in this chapter exists because of this line.
Pruning rescues it, and by an enormous margin — but only because most of those vectors are never the maximum anywhere on the simplex. Turn the toggle off in the widget below and watch the counter run away while the picture stops being drawable.
For the record, here is what our solver actually produces on the tiger at , accuracy — candidates generated versus vectors that survive:
| horizon | 1 | 2 | 3 | 4 | 5 | 6 | 8 | 10 | 12 |
|---|---|---|---|---|---|---|---|---|---|
| cross-sum candidates | 3 | 27 | 75 | 243 | 147 | 507 | 1 083 | 2 187 | 4 107 |
| kept after pruning | 3 | 5 | 9 | 7 | 13 | 15 | 25 | 27 | 35 |
Two honest observations. The candidate counts stay small only because each row is computed from the pruned previous row — that is the feedback loop that keeps exact value iteration alive at all. And the kept counts are not monotone: at the minimal set is smaller than at . Nothing is wrong. The minimal representation of a value function has no obligation to grow, and a solver that assumes it does will over-prune or under-prune.
Where this sits on the complexity map. Deciding whether a finite-horizon POMDP has a policy achieving a given value is PSPACE-complete (Papadimitriou and Tsitsiklis, 1987) — harder than NP-complete under the usual assumptions, and — unless — strictly harder than the fully observable case of Chapter 21, which the same paper places in P. Approximation in this chapter is therefore structural, not laziness: there is no machine and no clever data structure that makes the exact problem go away.
Pruning, and what our pruning misses
Given a candidate set , the minimal set is — the vectors that own at least one witness point. Three tests, in increasing cost:
- Duplicate merge. As value iteration converges the backup keeps re-deriving the same linear piece through different observation branches. Without a merge, grows without bound while stands still. Sort lexicographically and sweep: .
- Pointwise dominance. If for every , then can be deleted. Free, exact, and it never removes a vector it should keep — but it misses vectors that are dominated only by combinations of others.
- Witness test. survives iff the linear program subject to for all , , has . This is exact, and it is one LP per candidate.
We implement 1 and 2 always. For two states we then do something better than an LP: with every -vector is the line , and the minimal set is exactly the upper envelope of a set of lines — a convex hull, computable exactly in with no numerical tolerance games. That is what draws the bold curve in both widgets, and it is why the thresholds quoted in this chapter are exact rather than sampled. For three or more states we fall back to a witness search: test the simplex corners, then a few thousand Dirichlet-sampled interior points.
Be clear about what the witness search misses. It can only ever keep too few vectors: a piece whose ownership region is smaller than the sampling density is silently dropped, and the value function loses a sliver it should have had. It never keeps too many. In exchange we avoid a dependency on an LP solver, which for the problem sizes in this book is the right trade — but a production solver should use the LP, and Thrun et al. §16.2.4 writes it out.
QMDP, and exactly what it cannot see
The cheapest approximation is to keep only the first backup and stop. Assume that after one action the fog lifts entirely and the state becomes observable. Then the value of doing in belief is just the average, over states, of Chapter 21's :
This is an -vector set with exactly members, for any horizon, and it costs one Chapter 21 solve. It is also an upper bound: , because it prices a relaxed problem in which someone tells you the state after one step, and more information cannot hurt. On the tiger the bound is spectacularly loose — against , a factor of ten — but it is sound everywhere and free, which is why it is the standard upper bound in DESPOT's and SARSOP's branch-and-bound. A bad estimator can still be a useful certificate.
Now the sharp statement of what QMDP gives up.
DerivationQMDP is blind to the sensor
Statement. Let and be two actions with identical transition models and identical rewards, differing only in their observation models. Then for every belief — QMDP is exactly indifferent between them, no matter how much better one sensor is than the other.
Step 1. here is the value function of the underlying, fully observable MDP. Its defining equation, , contains no term involving . The observation model is not an input.
Step 2. Hence depends on only through and , which by hypothesis agree for and . So componentwise.
Step 3. .
Corollary — the failure, stated exactly. Give the tiger agent two microphones at the same price: a sharp one ( accurate) and a dull one (). QMDP's -vectors for the two actions are bit-identical — both — and it picks between them by tie-break. The exact solver picks the sharp microphone at every belief. This is the precise sense in which QMDP "never values information": it cannot distinguish actions that differ only in what they reveal. The Rust test suite below pins it as an equality assertion, which is the most honest possible way to document a limitation.
There is a subtler failure that the widget above is built to expose, and it matters more in practice. QMDP is not incapable of choosing the microphone in the tiger problem — listening also happens to be a cheap way of not dying, so it gets chosen for the wrong reason. What QMDP gets wrong is when to stop: it opens a door as soon as , where the optimal policy waits for .
Work out where comes from, because it is a two-line calculation. Under full observability the optimal policy is "open the correct door", worth , so at . Then for both states, while . Opening beats listening when , i.e. — and notice that the number contains no reference to the hearing accuracy at all. Drag the accuracy slider in the widget from to and QMDP's threshold does not move by one digit. That is Step 1 of the derivation, made visible.
A cautionary note about benchmarks. At the default accuracy the two pilots score identically, because the reachable belief ladder from is and steps clean over the gap between and : both thresholds say "listen once more, then open". A tournament run at would certify QMDP as optimal.
Move the slider in either direction and the agreement dissolves. Over 20 000 paired rounds — same tiger placement, same growl stream, every pilot:
| hearing accuracy | listens per round, optimal / QMDP | mauled, optimal / QMDP | score per round |
|---|---|---|---|
| / | / | / | |
| / | / | / | |
| / | / | / | |
| / | / | / |
The last row is the one to remember. With a -accurate microphone a single growl already puts the belief at , which clears QMDP's fixed — so QMDP opens after one listen and is eaten roughly eighteen times as often as the optimal pilot, which pays one more unit for a second opinion. A better sensor made the approximation worse. An approximation that looks perfect on your benchmark may be one parameter away from being dangerous, in either direction.
PBVI: keep the value function only where the robot can get to
The exact backup maintains everywhere on the simplex. Point-based value iteration (Pineau, Gordon and Thrun, 2003) keeps it only at a finite set of belief points, and — this is the whole idea — keeps exactly one -vector per point:
Crucially you never build the cross-sum. For a fixed , the best assignment is found one observation at a time — — because the -terms are independent. The backup therefore costs and returns at most vectors. Doubly exponential becomes polynomial, and the price is that is a lower bound on : every is the value of a real conditional plan, just not necessarily the best one somewhere else on the simplex.
The error bound is , with the density of in the simplex. Be honest about it: on the tiger with and the prefactor is , so with our five belief points () the bound reads "the error is at most ", while the actual error at is . The bound proves convergence as densifies. It certifies nothing at any you would actually run.
What does matter is which points you pick, and that is PBVI's real contribution: expand along reachable beliefs, , rather than gridding the simplex. Switch the Alpha Forge to PBVI mode and take the belief count down to three. With the envelope collapses onto QMDP's answer — threshold exactly — because the corners of the simplex cannot see that listening is worth anything at . Add the reachable point and the threshold jumps to ; add its mirror and it settles at , within of the exact . Five well-chosen points buy most of what the exact solver spends millions of vectors on. SARSOP (Kurniawati, Hsu and Lee, 2008) industrialized precisely this observation, by targeting beliefs reachable under near-optimal policies.
Online planning: POMCP searches histories, not states
Everything so far computes a policy for all beliefs, offline. The modern robotics answer is to give that up. At each timestep, plan only from the belief you are actually in, spend a fixed compute budget, act, and re-plan.
POMCP (Silver and Veness, 2010) is Monte-Carlo tree search over action–observation histories, and it fits this book unreasonably well because it needs two things you already built:
- A generative model : a black box that, given , samples . That is the Chapter 4 simulator, verbatim. POMCP never sees a transition matrix, which is why it scales to state spaces you could not write one for.
- A particle set for the root belief. That is Chapter 8's object, and Chapter 12's MCL produces one every frame.
The mechanism: each simulation samples a state from the root particles, then walks down the tree picking actions by UCB1 — — stepping the generative model at each node, and appending the sampled state to whichever child node the generated observation leads to. When it falls off the tree it runs a rollout policy to the horizon and backs the return up along the path.
The beautiful part is the belief representation. Nobody runs a Bayes filter inside the tree. The particles that happen to reach node are, by construction, distributed according to : the observation branching does the conditioning, and the surviving particles are the posterior. Sampling is the filter.
Two caveats, both inherited from Chapter 8 and both real. Particle deprivation gets worse with depth: a node three observations down may hold four particles, and its value estimate is noise. Particle deprivation gets worse on re-rooting: when you execute and observe , the new root is node , whose particles are all you have — which is why practical implementations reinvigorate with noise, exactly as Augmented MCL does. And the UCB1 constant is not a tuning afterthought: set it too small in the widget above and both opening arms get exactly one pull, the unlucky one draws the tiger, and neither is ever tried again. The planner then listens forever on the strength of a single rollout.
DESPOT, in one page
POMCP's weakness is that the tree branches on every observation that happens to be sampled, so with many observations it becomes a bush of singleton nodes. DESPOT (Ye et al., 2017) fixes this with one idea: fix scenarios in advance.
A scenario is a start state plus a stream of random numbers. Determinize the generative model so that is a function. Now a policy's outcome under scenario is deterministic, and the sparse tree contains only the observation branches that actually occur under the scenarios — the branching factor becomes instead of , and the whole tree evaluates all scenarios at once.
Two consequences worth remembering. First, the value estimated on scenarios overfits, exactly as empirical risk does: a policy can be tailored to the sampled futures. DESPOT's regularized objective, , penalizes policy size and comes with a regret bound in terms of the representation size of the optimal policy — the analogue of a generalization bound, and the reason the regularization is not an optional extra. Second, it is anytime and bounded: the search is guided by an upper bound (QMDP serves) and a lower bound (any default policy), and the gap between them at the root is a certificate you can stop on. It is a genuinely small amount of code once POMCP exists — which is why it is this chapter's stretch exercise rather than its main listing.
AMDP: compress the belief, then use Chapter 21 unchanged
For Rusty in the Apartment, none of the above applies directly: over is an infinite-dimensional object and is a LiDAR scan. The oldest robotics answer is also still one of the best: throw almost all of the belief away and keep two numbers.
Where I probably am, and how sure I am. Plan in that augmented space with Chapter 21's value iteration — nothing else changes — and you get the augmented MDP (Roy and Thrun, 1999; Thrun et al. §16.5). For a 2-D isotropic Gaussian belief the entropy coordinate collapses to a single position standard deviation: the differential entropy — the continuous analogue of the sum above, which may be negative, and only its differences matter here — is
so "minimize entropy" and "minimize " are the same instruction, and has the advantage of being drawable as the width of a ribbon.
The transitions in the augmented space need , and here Chapter 6 hands us the answer in one line of information form:
Prediction adds variance; correction adds information. The only genuinely new modelling object is , the localizability of a place: how much information one scan taken there delivers. We model it as , where is the distance to the nearest wall the robot can actually localize against and is the LiDAR range. A scan is informative when there is geometry inside it — and note that occupancy and information are two different maps, because a long featureless wall constrains distance-to-wall and nothing else, which is exactly the degeneracy Chapter 16's scan matcher suffers from.
The compression lies, and here is where. is only a legitimate state if it is a sufficient statistic for value — for every the robot may encounter. The draft says it plainly: "In practice, this assumption will rarely hold true." The clean counterexample is the one Chapter 12 already showed you. A belief split evenly between two symmetric corridors and a belief smeared uniformly over one large room can have the same entropy and the same MAP cell, and they demand opposite actions: the first needs a disambiguating detour, the second needs patience. cannot tell them apart. Chapter 24 builds a widget whose entire job is to exploit this.
The algorithms
- In
- a finite POMDP and a horizon T
- Out
- Γ_1 … Γ_T, with pre- and post-prune counts
- for to do
- for all do
- for all do
- for all assignments do
- return
This is Thrun et al.'s Table 16.1 with the pruning step promoted from a remark to line 8, and with the action tag kept on each vector so that a policy can be read off without a second pass.
- In
- a candidate α-vector set
- Out
- the minimal subset with the same upper envelope
- merge vectors agreeing componentwise to tolerance
- if then return the upper envelope of the lines
- the simplex corners Dirichlet samples
- return
- In
- a POMDP — its observation model is not read
- Out
- Γ_QMDP, an upper bound on V*
- on
- return
- In
- a POMDP, a belief set B, an iteration count
- Out
- Γ with |Γ| ≤ |B|, a lower bound on V*
- repeat times
- compute all from
- for all do
- for all do with
- , deduplicated
- return
- In
- root particle set, simulation budget, exploration constant
- Out
- an action; anytime — stop whenever the budget runs out
- repeat times: sample ;
- return (most-visited, not highest-valued)
- simulate:
- if then return
- if is a leaf then expand ; ; return
- ; append to the particle set of
- ; ;
- return
- In
- a simulator, a (cell, σ-bin) grid
- Out
- V and π over the augmented space
- for all and : estimate from the motion model
- , binned
- run on the resulting finite MDP
- return
Line 3 is the only place uncertainty enters the reward, and it is worth pausing on: the collision term is for a 2-D isotropic Gaussian error, which is a Rayleigh tail, , with the clearance to the nearest wall. A robot that is uncertain is charged for being uncertain in proportion to how tight the space is — which is why the emergent behavior is not "always hug walls" but "hug walls where it pays".
A worked example you can check by hand
Everything in this section is doable on paper in about fifteen minutes, and every number is pinned by a test in the next section.
and for the tiger
is the immediate-reward set, written above: , , .
For we need the backprojections. Under listen the transition is the identity, so — no sum survives. Take the assignment "if I hear left, follow ; if I hear right, follow ":
Under open the transition resets to and the growl carries nothing, so every backprojection is regardless of . With both branches following , each backprojection is , and there are two observations to sum over:
Note what just happened: because opening resets the world, every assignment with the same total gives the same vector. Nine assignments collapse to three distinct candidates before pruning even starts.
Grind through all assignments and exactly five vectors survive pruning:
| action | the plan it encodes | |
|---|---|---|
| open | open the left door now | |
| listen | listen; open left on , listen again on | |
| listen | listen twice, commit to nothing | |
| listen | listen; open right on , listen again on | |
| open | open the right door now |
Their upper envelope has four breakpoints, and each is a two-line calculation. Where does "open right now" overtake "listen, then open right"?
and where does the do-nothing plan give way to it?
So the two-step policy is: open right above , listen below it, and by symmetry open left below .
The converged answer, and a chain that closes
Run the backup to convergence and the tiger's minimal set has vectors and exactly three policy regions:
with values , , and . Those four numbers are not independent, and checking that they close is the best possible test that the solver is right:
(The residual digit in the last two lines is rounding, not error: run the solver at full precision and the three identities close to six decimals. The book's test asserts them to . If you read rather than in the widget above, that is not a discrepancy either — it solves to horizon so that the accuracy slider stays interactive, and the last of the threshold is carried by the discounted tail beyond step twenty.)
Read the second and third lines as sentences. From complete ignorance, the optimal thing to do is pay one unit, and you will find yourself at . From , pay one more unit; with probability the growl agrees and you land at , which is past the threshold, so you open; with probability it disagrees and you are back where you started. Listen, listen, open — and the number is the entire reason it is two listens and not one or three.
Implementation in Rust
The crate is crates/ch22_pomdp. It depends on ch21_mdp (for QMDP's inner solve and for the
augmented-MDP lab), on ch08_particles (for POMCP's root beliefs), and on nothing else that is not
in the book's stack.
The model: dimensions as types
use nalgebra::{SMatrix, SVector};
/// A finite POMDP with `S` states, `A` actions and `Z` observations.
///
/// The dimensions are const generics because they are genuinely fixed by the
/// problem, and because it makes the compiler check the one thing that is easy
/// to get wrong: whether a matrix is indexed [x][x'] or [x'][x].
pub struct FinitePomdp<const S: usize, const A: usize, const Z: usize> {
/// `t[u]` is row-stochastic: row `x`, column `x'` holds p(x' | x, u).
pub t: [SMatrix<f64, S, S>; A],
/// `o[u]` is row-stochastic: row `x'`, column `z` holds p(z | x', u).
/// Note the *next* state: the observation follows the transition.
pub o: [SMatrix<f64, S, Z>; A],
/// r(x, u), with Chapter 21's convention — collected on taking u in x.
pub r: SMatrix<f64, S, A>,
pub gamma: f64,
}
pub type Belief<const S: usize> = SVector<f64, S>;
impl<const S: usize, const A: usize, const Z: usize> FinitePomdp<S, A, Z> {
/// b̄(x') = Σ_x p(x' | x, u) b(x). Prediction: the step that loses information.
///
/// The transpose is the whole content of "rows are `x`, columns are `x'`".
pub fn predict(&self, b: &Belief<S>, u: usize) -> Belief<S> {
self.t[u].transpose() * b
}
/// τ(b, u, z), together with the evidence p(z | b, u).
///
/// The evidence is returned rather than discarded because in a POMDP it is
/// not a diagnostic — it is the branching probability of the belief MDP,
/// and the PWLC derivation turns on it cancelling against this normalizer.
pub fn update(&self, b: &Belief<S>, u: usize, z: usize) -> (Belief<S>, f64) {
let bar = self.predict(b, u);
let mut un = bar.component_mul(&self.o[u].column(z).into_owned());
let evidence = un.sum();
if evidence > 0.0 {
un /= evidence;
}
(un, evidence)
}
/// ρ(b, u) = Σ_x b(x) r(x, u). Linear in b — the load-bearing fact.
pub fn reward(&self, b: &Belief<S>, u: usize) -> f64 {
self.r.column(u).dot(b)
}
/// The underlying MDP, with the observation model **dropped on the floor**.
///
/// This method is the QMDP indifference theorem, expressed as code: nothing
/// downstream of here can possibly depend on how good the sensor is.
pub fn underlying_mdp(&self) -> ch21_mdp::Mdp<A> { /* … */ }
}
pub const LISTEN: usize = 0;
pub const OPEN_LEFT: usize = 1;
pub const OPEN_RIGHT: usize = 2;
/// Kaelbling, Littman & Cassandra (1998), with this book's discount.
pub fn tiger(accuracy: f64, gamma: f64) -> FinitePomdp<2, 3, 2> {
let a = accuracy;
let stay = SMatrix::<f64, 2, 2>::identity();
// Opening a door ends the round: the tiger is re-placed uniformly and the
// growl that follows carries no information at all.
let reset = SMatrix::<f64, 2, 2>::repeat(0.5);
FinitePomdp {
t: [stay, reset, reset],
o: [SMatrix::<f64, 2, 2>::new(a, 1.0 - a, 1.0 - a, a), reset, reset],
r: SMatrix::<f64, 2, 3>::new(
-1.0, -100.0, 10.0, // tiger-left: listen, open-left, open-right
-1.0, 10.0, -100.0, // tiger-right: …
),
gamma,
}
}The exact backup, and the pruning that saves it
use nalgebra::SVector;
use crate::model::{Belief, FinitePomdp};
/// One linear piece of V, tagged with the action it commits to at step one.
///
/// The tag is not decoration. Without it you would have to re-derive the
/// policy from the value function, which for a PWLC function means recomputing
/// the argmax of the backup — the expensive half of value iteration.
#[derive(Clone, Debug)]
pub struct AlphaVec<const S: usize> {
pub v: SVector<f64, S>,
pub action: usize,
}
impl<const S: usize> AlphaVec<S> {
#[inline]
pub fn value_at(&self, b: &Belief<S>) -> f64 {
self.v.dot(b)
}
}
/// g^{u,z}_k(x) = Σ_{x'} p(z | x', u) p(x' | x, u) α^(k)(x'), indexed [u][z][k].
///
/// In matrix form this is one scaled matrix–vector product: componentwise-scale
/// α by the observation column, then push it back through the transition. That
/// is the whole "backprojection" — value seen from x, through (u, z).
pub(crate) fn backprojections<const S: usize, const A: usize, const Z: usize>(
m: &FinitePomdp<S, A, Z>,
set: &[AlphaVec<S>],
) -> Vec<Vec<Vec<SVector<f64, S>>>> {
(0..A)
.map(|u| {
(0..Z)
.map(|z| {
let col = m.o[u].column(z).into_owned();
set.iter().map(|a| m.t[u] * a.v.component_mul(&col)).collect()
})
.collect()
})
.collect()
}
/// The exact backup: one α-vector per (action, assignment of one surviving
/// vector to each observation). Returns |U|·|Γ|^|Z| candidates — the
/// combinatorial explosion, un-hidden.
pub fn backup<const S: usize, const A: usize, const Z: usize>(
m: &FinitePomdp<S, A, Z>,
set: &[AlphaVec<S>],
) -> Vec<AlphaVec<S>> {
let g = backprojections(m, set);
let k = set.len();
let mut out = Vec::with_capacity(A * k.pow(Z as u32));
for u in 0..A {
let mut pick = [0usize; Z];
loop {
let mut v = m.r.column(u).into_owned();
for z in 0..Z {
v += m.gamma * g[u][z][pick[z]];
}
out.push(AlphaVec { v, action: u });
// Odometer over Γ^{|Z|}: one α index per observation.
let mut z = Z;
let carried = loop {
if z == 0 {
break true;
}
z -= 1;
pick[z] += 1;
if pick[z] < k {
break false;
}
pick[z] = 0;
};
if carried {
break;
}
}
}
out
}
/// Merge vectors that agree componentwise to `tol`.
///
/// Not cosmetic. As value iteration converges the backup keeps re-deriving the
/// *same* linear piece through different observation branches, and without this
/// merge |Γ| grows without bound while V stands still.
pub(crate) fn dedupe<const S: usize>(set: &[AlphaVec<S>], tol: f64) -> Vec<AlphaVec<S>> { /* … */ }
/// Prune to the minimal set.
///
/// For two states this is exact and cheap: every α is a *line* over
/// b = (t, 1−t), so the minimal set is the upper envelope of a set of lines,
/// which a monotone-chain hull gives in O(n log n) with no tolerance games —
/// and the breakpoints it returns are the policy's decision thresholds.
pub fn prune<const S: usize>(set: &[AlphaVec<S>]) -> Vec<AlphaVec<S>> {
let merged = dedupe(set, 1e-6);
if S == 2 {
return upper_envelope(&merged).into_iter().map(|(i, _)| merged[i].clone()).collect();
}
// Otherwise: free pointwise dominance, then a witness *search*. This can
// only ever keep too few vectors, never too many — see the chapter text.
let survivors = drop_dominated(&merged, 1e-9);
witness_survivors(&survivors, 4096, 0xC0FFEE)
}
/// V(b) = max_k ⟨α^(k), b⟩, together with the action the winning piece commands.
pub fn value_at<const S: usize>(set: &[AlphaVec<S>], b: &Belief<S>) -> (f64, usize) { /* … */ }
/// The upper envelope's decision thresholds. Only meaningful for |X| = 2, where
/// the envelope is a chain of line segments and its breakpoints *are* the policy.
pub struct Breakpoints {
pub open_left: f64,
pub open_right: f64,
}
pub fn policy_breakpoints<const S: usize>(set: &[AlphaVec<S>]) -> Breakpoints { /* … */ }
/// Back up until a dense sweep of the simplex stops moving. The Bellman backup
/// is a γ-contraction (Chapter 21's proof, unchanged), so this terminates.
pub fn solve_to_convergence<const S: usize, const A: usize, const Z: usize>(
m: &FinitePomdp<S, A, Z>,
tol: f64,
) -> Vec<AlphaVec<S>> { /* … */ }
pub struct Stage<const S: usize> {
pub horizon: usize,
/// |U|·|Γ_{t−1}|^{|Z|} — the analytic count, reported even when we refuse
/// to materialize it. The widget's runaway counter is this field.
pub raw: u128,
pub gamma: Vec<AlphaVec<S>>,
}
/// Thrun et al., Table 16.1 — `finite_world_POMDP`.
pub fn finite_world_pomdp<const S: usize, const A: usize, const Z: usize>(
m: &FinitePomdp<S, A, Z>,
horizon: usize,
) -> Vec<Stage<S>> {
let mut set: Vec<AlphaVec<S>> =
(0..A).map(|u| AlphaVec { v: m.r.column(u).into_owned(), action: u }).collect();
set = prune(&set);
let mut stages = vec![Stage { horizon: 1, raw: A as u128, gamma: set.clone() }];
for t in 2..=horizon {
let raw = (A as u128) * (set.len() as u128).pow(Z as u32);
set = prune(&backup(m, &set));
stages.push(Stage { horizon: t, raw, gamma: set.clone() });
}
stages
}QMDP and PBVI: the upper bound and the lower one
use nalgebra::SVector;
use crate::exact::AlphaVec;
use crate::model::FinitePomdp;
/// Γ_QMDP = { Q*(·, u) }_u — one vector per action, for every horizon.
///
/// Note what is *not* passed to `value_iteration`: `m.o`. That omission is the
/// theorem. An action's QMDP value cannot depend on what it reveals, because
/// the function that computes it never receives the observation model.
pub fn qmdp<const S: usize, const A: usize, const Z: usize>(
m: &FinitePomdp<S, A, Z>,
) -> Vec<AlphaVec<S>> {
let mdp = m.underlying_mdp();
let sol = ch21_mdp::value_iteration(&mdp, 1e-10, 100_000);
(0..A)
.map(|u| AlphaVec {
v: SVector::from_fn(|x, _| mdp.q(sol.v.as_slice(), x, u)),
action: u,
})
.collect()
}use crate::exact::{backprojections, dedupe, AlphaVec};
use crate::model::{Belief, FinitePomdp};
/// Pineau, Gordon & Thrun (2003): back up at `B` only, one vector per point.
///
/// The cross-sum is never built. For a fixed b the best assignment factorizes
/// over observations, so the inner argmax is |Z| independent scans instead of
/// one search over |Γ|^|Z| tuples — which is the entire complexity win.
pub fn point_backup<const S: usize, const A: usize, const Z: usize>(
m: &FinitePomdp<S, A, Z>,
set: &[AlphaVec<S>],
beliefs: &[Belief<S>],
) -> Vec<AlphaVec<S>> {
let g = backprojections(m, set);
let mut out = Vec::with_capacity(beliefs.len());
for b in beliefs {
let mut best: Option<AlphaVec<S>> = None;
for u in 0..A {
let mut v = m.r.column(u).into_owned();
for z in 0..Z {
let k = (0..set.len())
.max_by(|&i, &j| g[u][z][i].dot(b).total_cmp(&g[u][z][j].dot(b)))
.unwrap_or(0);
v += m.gamma * g[u][z][k];
}
if best.as_ref().is_none_or(|c| v.dot(b) > c.v.dot(b)) {
best = Some(AlphaVec { v, action: u });
}
}
out.extend(best);
}
dedupe(&out, 1e-9)
}
/// PBVI's belief-set expansion: B ← B ∪ { τ(b,u,z) }.
///
/// Reachability is the point. Gridding the simplex wastes points on beliefs no
/// filter will ever produce; this walks the same tree the robot will walk.
pub fn expand<const S: usize, const A: usize, const Z: usize>(
m: &FinitePomdp<S, A, Z>,
beliefs: &[Belief<S>],
tol: f64,
) -> Vec<Belief<S>> {
let mut out = beliefs.to_vec();
for b in beliefs {
for u in 0..A {
for z in 0..Z {
let (bp, evidence) = m.update(b, u, z);
if evidence > 1e-9 && !out.iter().any(|q| (q - bp).amax() < tol) {
out.push(bp);
}
}
}
}
out
}POMCP: planning with the simulator you already have
use rand::rngs::SmallRng;
use rand::Rng;
/// The black box POMCP plans with. For the labs in this chapter it is
/// literally the Chapter 4 simulator: no transition matrix exists anywhere.
pub trait Generative {
type State: Clone;
const N_ACTIONS: usize;
/// (x, u) ↦ (x', z, r). Observations are interned to `u32` so that a laser
/// scan can be hashed into a discrete branch without changing this trait.
fn step(&self, x: &Self::State, u: usize, rng: &mut SmallRng) -> (Self::State, u32, f64);
fn terminal(&self, _x: &Self::State) -> bool {
false
}
/// The default policy a leaf is evaluated with, and the thing to tune first
/// when the budget is small. On the tiger, "listen twice then guess" is
/// worth 23 points of root value over uniform-random at 16 simulations, 2
/// points at 256, and nothing at all by 1024: a good rollout buys you the
/// early part of the anytime curve, which is the part a control loop gets.
fn rollout(&self, _x: &Self::State, _depth: usize, rng: &mut SmallRng) -> usize {
rng.random_range(0..Self::N_ACTIONS)
}
}
/// A history node. `particles` is the belief — Chapter 8's object, arriving by
/// natural selection rather than by a filter update.
struct BeliefNode<X> {
n: u32,
particles: Vec<X>,
arms: Vec<u32>,
obs: Option<u32>,
}
/// A bandit arm: N(h, u) and the running mean Q(h, u).
struct ActionNode {
action: usize,
n: u32,
q: f64,
children: Vec<(u32, u32)>, // (observation, belief-node id)
}
pub struct Pomcp<G: Generative> {
model: G,
beliefs: Vec<BeliefNode<G::State>>,
actions: Vec<ActionNode>,
root: usize,
/// UCB1 exploration weight. Scale it to the *spread of returns*, not to 1:
/// on the tiger the returns span 110, and c = 1 explores nothing.
pub c: f64,
pub max_depth: usize,
pub gamma: f64,
pub simulations: u64,
}
impl<G: Generative> Pomcp<G> {
/// Run `n` more simulations. Anytime: stop whenever the control loop says so.
pub fn search(&mut self, n: u64, rng: &mut SmallRng) {
for _ in 0..n {
let i = rng.random_range(0..self.beliefs[self.root].particles.len());
let x = self.beliefs[self.root].particles[i].clone();
self.simulate(x, self.root, 0, rng);
self.simulations += 1;
}
}
fn simulate(&mut self, x: G::State, node: usize, depth: usize, rng: &mut SmallRng) -> f64 {
if depth >= self.max_depth {
return 0.0;
}
if self.beliefs[node].arms.is_empty() {
self.expand(node);
self.beliefs[node].n += 1;
// A leaf's value is a rollout, not zero. Returning zero here is the
// single most common way to make MCTS pathologically pessimistic.
return self.rollout_value(x, depth, rng);
}
let arm = self.select_ucb1(node);
let u = self.actions[arm].action;
let (xp, z, r) = self.model.step(&x, u, rng);
let child = self.child_for(arm, z);
// The child's belief is *built* from the particles that reach it: an
// unweighted particle filter, running inside the search tree.
self.beliefs[child].particles.push(xp.clone());
let future = if self.model.terminal(&xp) {
0.0
} else {
self.simulate(xp, child, depth + 1, rng)
};
let ret = r + self.gamma * future;
self.beliefs[node].n += 1;
let a = &mut self.actions[arm];
a.n += 1;
a.q += (ret - a.q) / f64::from(a.n);
ret
}
/// UCB1: exploit Q, but keep an eye on arms you have barely tried.
fn select_ucb1(&self, node: usize) -> usize {
let n = f64::from(self.beliefs[node].n + 1);
let mut best = (f64::NEG_INFINITY, self.beliefs[node].arms[0] as usize);
for &arm in &self.beliefs[node].arms {
let a = &self.actions[arm as usize];
if a.n == 0 {
return arm as usize; // every arm gets one free pull
}
let score = a.q + self.c * (n.ln() / f64::from(a.n)).sqrt();
if score > best.0 {
best = (score, arm as usize);
}
}
best.1
}
/// Pure bookkeeping: create one action node per arm, find-or-create the
/// belief node for an observation, and roll a default policy to the horizon.
fn expand(&mut self, node: usize) { /* … */ }
fn child_for(&mut self, arm: usize, z: u32) -> usize { /* … */ }
fn rollout_value(&self, x: G::State, depth: usize, rng: &mut SmallRng) -> f64 { /* … */ }
/// The recommendation: the *most visited* arm, not the highest-valued one.
/// A high Q on two pulls is a rumour; a high visit count is a decision.
pub fn best_action(&self) -> usize { /* … */ }
/// Execute u, observe z: re-root at `huz` and keep its particles as the new
/// belief. Reinvigorate them — depth-three nodes are exactly where Chapter
/// 8's particle deprivation reappears.
pub fn advance(&mut self, u: usize, z: u32, rng: &mut SmallRng) { /* … */ }
}The worked example, as tests
use approx::assert_relative_eq;
use ch22_pomdp::{exact::*, model::*, qmdp::qmdp};
use nalgebra::SVector;
/// A tiger with two microphones at the same price, listed sharp-first. The
/// dynamics and the rewards of actions 0 and 1 are identical by construction;
/// only `o[0]` and `o[1]` differ.
const SHARP: usize = 0;
const DULL: usize = 1;
/// Complete ignorance: the belief every round of the tiger starts from.
fn half() -> Belief<2> {
SVector::new(0.5, 0.5)
}
#[test]
fn belief_ladder_matches_the_text() {
let m = tiger(0.85, 0.95);
let (b1, ev1) = m.update(&half(), LISTEN, 0);
assert_relative_eq!(b1[0], 0.85, epsilon = 1e-12);
assert_relative_eq!(ev1, 0.5, epsilon = 1e-12); // a first growl is a coin flip
let (b2, ev2) = m.update(&b1, LISTEN, 0);
assert_relative_eq!(b2[0], 0.7225 / 0.745, epsilon = 1e-12); // 0.969799…
assert_relative_eq!(ev2, 0.745, epsilon = 1e-12);
// A contradicting growl undoes exactly one consistent one.
let (b3, _) = m.update(&b1, LISTEN, 1);
assert_relative_eq!(b3[0], 0.5, epsilon = 1e-12);
}
#[test]
fn gamma_two_has_five_survivors_and_the_right_threshold() {
let m = tiger(0.85, 0.95);
let stages = finite_world_pomdp(&m, 2);
assert_eq!(stages[1].raw, 27); // |U| · |Γ₁|^|Z| = 3 · 3²
assert_eq!(stages[1].gamma.len(), 5);
let listen_then_open = stages[1]
.gamma
.iter()
.find(|a| a.action == LISTEN && a.v[0] > 0.0)
.expect("the plan that listens, then opens right on z_L");
assert_relative_eq!(listen_then_open.v[0], 6.9325, epsilon = 1e-9);
assert_relative_eq!(listen_then_open.v[1], -16.0575, epsilon = 1e-9);
let thresholds = policy_breakpoints(&stages[1].gamma);
assert_relative_eq!(thresholds.open_right, 84.8925 / 87.01, epsilon = 1e-9);
}
#[test]
fn converged_policy_and_the_value_chain() {
let m = tiger(0.85, 0.95);
let g = solve_to_convergence(&m, 1e-9);
assert_eq!(g.len(), 9);
let t = policy_breakpoints(&g);
assert_relative_eq!(t.open_right, 0.960346, epsilon = 1e-4);
assert_relative_eq!(t.open_left, 1.0 - t.open_right, epsilon = 1e-9); // symmetry
let v = |p: f64| value_at(&g, &SVector::new(p, 1.0 - p)).0;
// The three identities from the text — the chain has to close.
assert_relative_eq!(v(1.0), 10.0 + 0.95 * v(0.5), epsilon = 1e-3);
assert_relative_eq!(v(0.5), -1.0 + 0.95 * v(0.85), epsilon = 1e-3);
assert_relative_eq!(
v(0.85),
-1.0 + 0.95 * (0.745 * v(0.969799) + 0.255 * v(0.5)),
epsilon = 1e-3
);
}
#[test]
fn qmdp_cannot_tell_a_good_microphone_from_a_bad_one() {
// Two listening actions: identical cost, identical (identity) dynamics,
// wildly different sensors. The exact solver always picks the sharp one.
let m = tiger_with_two_microphones(0.95, 0.55);
let g = qmdp(&m);
assert_relative_eq!(g[SHARP].v, g[DULL].v, epsilon = 1e-12); // the theorem
let exact = solve_to_convergence(&m, 1e-9);
assert_eq!(value_at(&exact, &half()).1, SHARP);
}
#[test]
fn qmdp_threshold_ignores_the_sensor_entirely() {
for accuracy in [0.55, 0.7, 0.85, 0.99] {
let t = policy_breakpoints(&qmdp(&tiger(accuracy, 0.95)));
assert_relative_eq!(t.open_right, 0.9, epsilon = 1e-9);
}
}The widgets in this chapter run the TypeScript twin of every listing above — lib/pomdp/finite.ts,
lib/pomdp/pomcp.ts, lib/pomdp/amdp.ts. The thresholds printed on screen, the vector counts, and
the tournament returns all come out of those functions, so if you redo the algebra by hand you get
the number on the screen.
Putting it together: why robots hug walls
The tiger is a two-state cartoon. Rusty's belief lives over , and its observation is a LiDAR scan — a POMDP with a continuum of states and effectively a continuum of observations. Exact solving is not merely intractable here; the objects do not exist.
So we do the compression. The state becomes (cell, -bin): free cells uncertainty bins plus one absorbing terminal, states, solved by Chapter 21's value iteration in about forty Gauss–Seidel sweeps. Two policies come out of the same code. One is told to pretend is always at its floor — the certainty-equivalent planner, which is what "plan a path on the MAP estimate, then track it" amounts to. The other is given the honest dynamics.
Nothing in the map says "prefer walls". The reward function contains a step cost, a collision hazard, and a doorway bonus; the word "wall" appears only inside , as the distance to something a scan can lock onto. Yet the purple pilot leaves the straight line, spends about extra steps in the neighbourhood of the textured south wall, pinches its uncertainty ribbon back down, and only then turns for the door — arriving through the gap of the time against the straight pilot's over 400 seeded runs.
Then push the LiDAR range slider past m. The information field floods the room, both pilots reach , and the coastal behavior evaporates. This is not a disappointment; it is the honest boundary of the phenomenon, and it reproduces Figure 16.6 of the Thrun draft, where entropy at the goal is plotted against sensor range and the two curves meet. Coastal navigation is a statement about when sensing is informative, not a law of robotics. A robot with GPS should drive straight.
What this chapter is not
Continuous observations. POMCP branches on observations, so a continuous observation space gives every node exactly one child and the tree degenerates into a set of independent rollouts. The fixes — progressive widening, observation clustering, adaptive discretization — are a live research area; Hoerger et al. (2024) is a good entry point, and Lauri et al. (2023) surveys the landscape.
Belief-space trajectory optimization. For unimodal Gaussian beliefs you can skip trees entirely and run a continuous optimizer over a trajectory of pairs, with the covariance propagated by an EKF inside the cost. It is fast and it is what most manipulation systems do; it also cannot represent the tiger, because a Gaussian cannot be bimodal. Chapter 23 takes the continuous-control ground with sampling instead.
Learning the policy. Everything here assumes a model. Deep RL over histories learns the same object without one, and pays in sample complexity and in the loss of every guarantee in this chapter. Chapter 25 makes the case for keeping learning inside the Bayesian frame rather than replacing it.
Where this goes next: Chapter 23 is a receding-horizon POMDP solver in disguise — sampled futures, reweighted, re-planned every frame. Chapter 24 formulates active SLAM as a POMDP and then approximates it exactly the way this chapter licenses, with the AMDP's failure mode as the thing to watch. And Chapter 26 wires the decision layer to everything else.
Exercises
- Foundation exerciseDifficulty 2 of 3Compute Γ₂ by hand, all 27 of it
Enumerate the candidate vectors for the tiger at , accuracy . The text asserts that the nine assignments collapse to three distinct vectors while the nine ones do not. Verify it, and identify the two separate reasons — one is a property of the transition and observation matrices after opening, the other is an accident of the particular numbers in . Then prune to the five survivors in the table above, and verify the breakpoint from the two lines that cross there.
- Foundation exerciseDifficulty 3 of 3Convexity without α-vectors
Prove directly from the definition of as a supremum over policies that is convex in , without using the piecewise-linear representation. (Hint: a policy's expected return is linear in the initial belief; is a pointwise supremum of such functions.) Then explain in one paragraph why convexity is the formal statement of "certainty is worth money", and say what the sag below the chord at equals in dollars.
- Conceptual exerciseDifficulty 2 of 3Predict the divergence, then verify
In the Tiger Door Console, the optimal threshold at accuracy is and QMDP's is , yet the two pilots make identical decisions. The belief after consistent growls from is , and the two policies differ exactly when a rung of that ladder lands in the gap between the thresholds. Using only that formula and a calculator, predict the set of accuracies at which they diverge — it is a union of bands, not an interval, and happens to fall in a gap between two of them. Verify with the slider, then explain why raising the sensor accuracy from to makes QMDP worse. Finally: which of the two thresholds moves when you change , and which cannot?
- Conceptual exerciseDifficulty 2 of 3Find where the coast stops paying
In the Coastal Navigator, find the LiDAR range at which the two pilots' arrival rates come within one standard error of each other, using enough seeded runs that the answer means something. Then relate what you see to : at that range, what fraction of the room is informative? Finally, describe a concrete belief in the Apartment for which the AMDP compression would give the planner actively misleading advice, and say which action it would wrongly recommend.
- Practical exerciseDifficulty 2 of 3Implement QMDP on Chapter 21's solver
qmdp.rsships as a stub. Implement it on top ofch21_mdp::value_iteration— the only work is buildingunderlying_mdp(), and the only thing to get right is that the observation model must not appear anywhere in the conversion. Then reproduce the chapter's paired tournament: 20 000 rounds at accuracies , and , with the optimal, QMDP and impatient (open the more likely door immediately) pilots sharing one seeded growl stream per round. Your maul rates at should land near and . - Practical exerciseDifficulty 3 of 3POMCP over an MCL belief
Wire
Pomcpto the Chapter 12 MCL filter and the Chapter 4 simulator, on a T-junction with two mirror-image corridors and a bimodal particle belief. The generative model is the simulator; the root particle set is MCL's output, resampled to 300. Report how often the planner drives past the disambiguating doorway before committing to a corridor, as a function of the simulation budget — and find the budget below which it commits blind. Then break it deliberately: shrink the UCB1 constant until a single unlucky rollout retires the correct arm. - Practical exerciseDifficulty 3 of 3DESPOT's determinized scenarios (stretch)
Add a
search_despottopomcp.rs: draw scenarios (a start state plus a seeded random stream each), share them across the whole tree so that every node evaluates all at once, and branch only on the observations those scenarios actually produce. Use QMDP as the upper bound and "listen twice then guess" as the lower one. Compare regret against simulation count with POMCP on the tiger and on your T-junction world, and report where the regularization term starts to matter.
References
- Smallwood, R. D. and Sondik, E. J. (1973) The Optimal Control of Partially Observable Markov Processes over a Finite Horizon. Operations Research 21(5), 1071–1088.doi:10.1287/opre.21.5.1071 (opens in a new tab)
Where piecewise-linear convexity comes from. This chapter's centerpiece derivation is their theorem, restated in modern α-vector notation; the paper is also the origin of the exact backup our exact.rs implements.
- Papadimitriou, C. H. and Tsitsiklis, J. N. (1987) The Complexity of Markov Decision Processes. Mathematics of Operations Research 12(3), 441–450.doi:10.1287/moor.12.3.441 (opens in a new tab)
The PSPACE-completeness result quoted in the complexity box. The same paper shows the fully observable case is only P-complete, which is exactly the gap Chapter 21 was living in.
- Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. (1998) Planning and Acting in Partially Observable Stochastic Domains. Artificial Intelligence 101(1–2), 99–134.doi:10.1016/S0004-3702(98)00023-X (opens in a new tab)
The paper that made POMDPs legible to a generation, and the source of the tiger problem this chapter runs every number on. Read §3–5 alongside the mathematics section here.
- Roy, N. and Thrun, S. (1999) Coastal Navigation with Mobile Robots. Advances in Neural Information Processing Systems 12 (NIPS 1999).link to Coastal Navigation with Mobile Robots (opens in a new tab)
The augmented-MDP compression and the wall-hugging behavior the Coastal Navigator reproduces. Their Figure comparing entropy at the goal against sensor range is the experiment the widget's slider runs live.
- Pineau, J., Gordon, G., and Thrun, S. (2003) Point-based value iteration: An anytime algorithm for POMDPs. IJCAI 2003, 1025–1032.link to Point-based value iteration: An anytime algorithm for POMDPs (opens in a new tab)
PBVI, its reachable-belief expansion, and the error bound quoted (and criticized) above. The algorithm is thirty lines; the insight — approximate the belief set, not the value function — is the one that stuck.
- Thrun, S., Burgard, W., and Fox, D. (2005) Probabilistic Robotics. MIT Press.link to Probabilistic Robotics (opens in a new tab)
Chapter 16 is this chapter's baseline: the finite-world exact solver of Table 16.1, the LP pruning we replace with an envelope, and §16.5's augmented MDP. The epigraph is from §16.2 of the 1999–2000 draft, which differs from the published edition.
- Kurniawati, H., Hsu, D., and Lee, W. S. (2008) SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces. Robotics: Science and Systems IV.link to SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces (opens in a new tab)
PBVI industrialized: sample beliefs reachable under near-optimal policies rather than under any policy. Still the offline solver to reach for when your state space is small enough to have one.
- Silver, D. and Veness, J. (2010) Monte-Carlo Planning in Large POMDPs. Advances in Neural Information Processing Systems 23 (NIPS 2010).link to Monte-Carlo Planning in Large POMDPs (opens in a new tab)
POMCP: the algorithm in this chapter's fifth box, and the reason a robot with a simulator and a particle filter already owns most of a POMDP solver.
- Ye, N., Somani, A., Hsu, D., and Lee, W. S. (2017) DESPOT: Online POMDP Planning with Regularization. Journal of Artificial Intelligence Research 58, 231–266.doi:10.1613/jair.5328 (opens in a new tab)
The journal version of the 2013 NIPS paper: determinized scenarios, the regret bound in terms of policy representation size, and the regularization that stops a K-scenario tree from overfitting its own futures.
- Lauri, M., Hsu, D., and Pajarinen, J. (2023) Partially Observable Markov Decision Processes in Robotics: A Survey. IEEE Transactions on Robotics 39(1), 21–40.doi:10.1109/TRO.2022.3200138 (opens in a new tab)
The current map of the field, written for roboticists. The best single answer to 'which solver should I actually use for my problem', and the place to go for continuous-space and multi-robot variants this chapter skips.
- Hoerger, M., Kurniawati, H., Kroese, D., and Ye, N. (2024) Adaptive Discretization using Voronoi Trees for Continuous POMDPs. The International Journal of Robotics Research 43(9), 1283–1298.doi:10.1177/02783649231188984 (opens in a new tab)
A recent answer to the continuous-action problem POMCP degenerates on: refine the action space where the value function says it matters. Read it after building the tree widget's model and asking what happens when u is a real number.
