\ie \cf

$\newcommand{\norm}[1]{\\|#1\\|} \newcommand{\rank}{\text{rank}} \newcommand{\tr}{\text{tr}} \newcommand{\mor}{\text{Mor}} \newcommand{\hom}{\text{Hom}} \newcommand{\id}{\text{id}} \newcommand{\obj}{\text{obj}} \newcommand{\pr}{\text{pr}} \newcommand{\ker}{\text{ker}} \newcommand{\coker}{\text{coker}} \newcommand{\im}{\text{im}} \newcommand{\bbA}{\mathbb A} \newcommand{\bbB}{\mathbb B} \newcommand{\bbC}{\mathbb C} \newcommand{\bbD}{\mathbb D} \newcommand{\bbE}{\mathbb E} \newcommand{\bbF}{\mathbb F} \newcommand{\bbG}{\mathbb G} \newcommand{\bbH}{\mathbb H} \newcommand{\bbI}{\mathbb I} \newcommand{\bbJ}{\mathbb J} \newcommand{\bbK}{\mathbb K} \newcommand{\bbL}{\mathbb L} \newcommand{\bbM}{\mathbb M} \newcommand{\bbN}{\mathbb N} \newcommand{\bbO}{\mathbb O} \newcommand{\bbP}{\mathbb P} \newcommand{\bbQ}{\mathbb Q} \newcommand{\bbR}{\mathbb R} \newcommand{\bbS}{\mathbb S} \newcommand{\bbT}{\mathbb T} \newcommand{\bbU}{\mathbb U} \newcommand{\bbV}{\mathbb V} \newcommand{\bbW}{\mathbb W} \newcommand{\bbX}{\mathbb X} \newcommand{\bbY}{\mathbb Y} \newcommand{\bbZ}{\mathbb Z} \newcommand{\rmA}{\mathfrak A} \newcommand{\rmB}{\mathfrak B} \newcommand{\rmC}{\mathfrak C} \newcommand{\rmD}{\mathfrak D} \newcommand{\rmE}{\mathfrak E} \newcommand{\rmF}{\mathfrak F} \newcommand{\rmG}{\mathfrak G} \newcommand{\rmH}{\mathfrak H} \newcommand{\rmI}{\mathfrak I} \newcommand{\rmJ}{\mathfrak J} \newcommand{\rmK}{\mathfrak K} \newcommand{\rmL}{\mathfrak L} \newcommand{\rmM}{\mathfrak M} \newcommand{\rmN}{\mathfrak N} \newcommand{\rmO}{\mathfrak O} \newcommand{\rmP}{\mathfrak P} \newcommand{\rmQ}{\mathfrak Q} \newcommand{\rmR}{\mathfrak R} \newcommand{\rmS}{\mathfrak S} \newcommand{\rmT}{\mathfrak T} \newcommand{\rmU}{\mathfrak U} \newcommand{\rmV}{\mathfrak V} \newcommand{\rmW}{\mathfrak W} \newcommand{\rmX}{\mathfrak X} \newcommand{\rmY}{\mathfrak Y} \newcommand{\rmZ}{\mathfrak Z} \newcommand{\paren}[1]{( #1 )} \newcommand{\Paren}[1]{\left( #1 \right)} \newcommand{\bigparen}[1]{\bigl( #1 \bigr)} \newcommand{\Bigparen}[1]{\Bigl( #1 \Bigr)} \newcommand{\biggparen}[1]{\biggl( #1 \biggr)} \newcommand{\Biggparen}[1]{\Biggl( #1 \Biggr)} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\Abs}[1]{\left\lvert #1 \right\rvert} \newcommand{\bigabs}[1]{\bigl\lvert #1 \bigr\rvert} \newcommand{\Bigabs}[1]{\Bigl\lvert #1 \Bigr\rvert} \newcommand{\biggabs}[1]{\biggl\lvert #1 \biggr\rvert} \newcommand{\Biggabs}[1]{\Biggl\lvert #1 \Biggr\rvert} \newcommand{\card}[1]{\left| #1 \right|} \newcommand{\Card}[1]{\left\lvert #1 \right\rvert} \newcommand{\bigcard}[1]{\bigl\lvert #1 \bigr\rvert} \newcommand{\Bigcard}[1]{\Bigl\lvert #1 \Bigr\rvert} \newcommand{\biggcard}[1]{\biggl\lvert #1 \biggr\rvert} \newcommand{\Biggcard}[1]{\Biggl\lvert #1 \Biggr\rvert} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\Norm}[1]{\left\lVert #1 \right\rVert} \newcommand{\bignorm}[1]{\bigl\lVert #1 \bigr\rVert} \newcommand{\Bignorm}[1]{\Bigl\lVert #1 \Bigr\rVert} \newcommand{\biggnorm}[1]{\biggl\lVert #1 \biggr\rVert} \newcommand{\Biggnorm}[1]{\Biggl\lVert #1 \Biggr\rVert} \newcommand{\iprod}[1]{\langle #1 \rangle} \newcommand{\Iprod}[1]{\left\langle #1 \right\rangle} \newcommand{\bigiprod}[1]{\bigl\langle #1 \bigr\rangle} \newcommand{\Bigiprod}[1]{\Bigl\langle #1 \Bigr\rangle} \newcommand{\biggiprod}[1]{\biggl\langle #1 \biggr\rangle} \newcommand{\Biggiprod}[1]{\Biggl\langle #1 \Biggr\rangle} \newcommand{\set}[1]{\lbrace #1 \rbrace} \newcommand{\Set}[1]{\left\lbrace #1 \right\rbrace} \newcommand{\bigset}[1]{\bigl\lbrace #1 \bigr\rbrace} \newcommand{\Bigset}[1]{\Bigl\lbrace #1 \Bigr\rbrace} \newcommand{\biggset}[1]{\biggl\lbrace #1 \biggr\rbrace} \newcommand{\Biggset}[1]{\Biggl\lbrace #1 \Biggr\rbrace} \newcommand{\bracket}[1]{\lbrack #1 \rbrack} \newcommand{\Bracket}[1]{\left\lbrack #1 \right\rbrack} \newcommand{\bigbracket}[1]{\bigl\lbrack #1 \bigr\rbrack} \newcommand{\Bigbracket}[1]{\Bigl\lbrack #1 \Bigr\rbrack} \newcommand{\biggbracket}[1]{\biggl\lbrack #1 \biggr\rbrack} \newcommand{\Biggbracket}[1]{\Biggl\lbrack #1 \Biggr\rbrack} \newcommand{\ucorner}[1]{\ulcorner #1 \urcorner} \newcommand{\Ucorner}[1]{\left\ulcorner #1 \right\urcorner} \newcommand{\bigucorner}[1]{\bigl\ulcorner #1 \bigr\urcorner} \newcommand{\Bigucorner}[1]{\Bigl\ulcorner #1 \Bigr\urcorner} \newcommand{\biggucorner}[1]{\biggl\ulcorner #1 \biggr\urcorner} \newcommand{\Biggucorner}[1]{\Biggl\ulcorner #1 \Biggr\urcorner} \newcommand{\ceil}[1]{\lceil #1 \rceil} \newcommand{\Ceil}[1]{\left\lceil #1 \right\rceil} \newcommand{\bigceil}[1]{\bigl\lceil #1 \bigr\rceil} \newcommand{\Bigceil}[1]{\Bigl\lceil #1 \Bigr\rceil} \newcommand{\biggceil}[1]{\biggl\lceil #1 \biggr\rceil} \newcommand{\Biggceil}[1]{\Biggl\lceil #1 \Biggr\rceil} \newcommand{\floor}[1]{\lfloor #1 \rfloor} \newcommand{\Floor}[1]{\left\lfloor #1 \right\rfloor} \newcommand{\bigfloor}[1]{\bigl\lfloor #1 \bigr\rfloor} \newcommand{\Bigfloor}[1]{\Bigl\lfloor #1 \Bigr\rfloor} \newcommand{\biggfloor}[1]{\biggl\lfloor #1 \biggr\rfloor} \newcommand{\Biggfloor}[1]{\Biggl\lfloor #1 \Biggr\rfloor} \newcommand{\lcorner}[1]{\llcorner #1 \lrcorner} \newcommand{\Lcorner}[1]{\left\llcorner #1 \right\lrcorner} \newcommand{\biglcorner}[1]{\bigl\llcorner #1 \bigr\lrcorner} \newcommand{\Biglcorner}[1]{\Bigl\llcorner #1 \Bigr\lrcorner} \newcommand{\bigglcorner}[1]{\biggl\llcorner #1 \biggr\lrcorner} \newcommand{\Bigglcorner}[1]{\Biggl\llcorner #1 \Biggr\lrcorner} \newcommand{\Rnn}{\bbR_{\ge 0}} \DeclareMathOperator{\Tr}{Tr} \DeclareMathOperator{\min}{min} \DeclareMathOperator{\max}{max} \DeclareMathOperator{\argmax}{arg\,max} \DeclareMathOperator{\argmin}{arg\,min} \DeclareMathOperator{\E}{\mathbb E} \DeclareMathOperator{\pE}{\tilde{\mathbb E}} \DeclareMathOperator{\Pr}{\mathbb P} \DeclareMathOperator{\Det}{\textbf{Det}} \DeclareMathOperator{\Perm}{\textbf{Perm}} \DeclareMathOperator{\Sym}{\textbf{Sym}} \DeclareMathOperator{\Pow}{\textbf{Pow}} \DeclareMathOperator{\Gal}{\textbf{Gal}} \DeclareMathOperator{\Aut}{\textbf{Aut}} \notag$

$\newcommand{\1}{\ensuremath{\mathrm{1}}} \notag$

$\newcommand{\transpose}[1]{\ensuremath{#1{}^{\mkern-2mu\intercal}}} \newcommand{\dyad}[1]{\ensuremath{#1#1{}^{\mkern-2mu\intercal}}} \newcommand{\nchoose}[1]{\ensuremath} \newcommand{\generated}[1]{\ensuremath{\langle #1 \rangle}} \notag$

\newcommand{\eqdef}{\mathbin{\stackrel{\rm def}{=}}}  \newcommand{\R} % real numbers  \newcommand{\N}} % natural numbers  \newcommand{\Z} % integers  \newcommand{\F} % a field  \newcommand{\Q} % the rationals  \newcommand{\C}{\mathbb{C}} % the complexes  \newcommand{\poly}}  \newcommand{\polylog}}  \newcommand{\loglog}}}  \newcommand{\zo}{\{0,1\}}  \newcommand{\suchthat}  \newcommand{\pr}[1]{\Pr\left[#1\right]}  \newcommand{\deffont}{\em}  \newcommand{\getsr}{\mathbin{\stackrel{\mbox{\tiny R}}{\gets}}}  \newcommand{\Exp}{\mathop{\mathrm E}\displaylimits} % expectation  \newcommand{\Var}{\mathop{\mathrm Var}\displaylimits} % variance  \newcommand{\xor}{\oplus}  \newcommand{\GF}{\mathrm{GF}}  \newcommand{\eps}{\varepsilon}  \notag

$\newcommand{\eps}{\varepsilon} \notag$

$\newcommand{\MIP}{\class{MIP}}$

\no

$\newcommand{\textprob}[1]{\text{#1}} \newcommand{\mathprob}[1]{\textbf{#1}} \newcommand{\Satisfiability}{\textprob{Satisfiability}} \newcommand{\Mor}{\textprob{Mor}} \newcommand{\Hom}{\textprob{Hom}} \newcommand{\cok}{\textprob{cok }} \newcommand{\ker}{\textprob{ker }} \newcommand{\im}{\textprob{im }} \newcommand{\coim}{\textprob{coim }} \newcommand{\id}{\textprob{id}}$

Chapter 1 |

| Previous

Mathematical Issues

100%

In this post, we take a slight detour into making rigorous some of the handwaving earlier on we did to establish the basic problem. For those of you who don't care about definitional considerations, you may want to skip this post.

What really is a basic problem?

We can summarize our setup of the basic problem into the following steps:

The controller observes $x_k$ and applies $u_k=\mu_k(x_k)$
The disturbance is generated by $w_k\sim P_k(\cdot\mid x_k,u_k)$
The cost $g_k(x_k,u_k,w_k)$ is incurred and added to the previous costs
The next state is generated by $x_{k+1}=f_k(x_k,u_k,w_k)$

Of course, if we are at the final state, we skip the last step and instead add the terminal cost $g_n(x_n)$ to finish the process. This all seems fine, and it is, until we want to do anything interesting with the basic problem.

Minimizing Costs, rigorously

In our Dynamic Programming Algorithm, we define our "cost-to-go" $V_k$ by evaluting the expected value of $g_k$. This, of course, assumes $g_k$ can be interpreted as a random variable. Let's try to do that.

In order to do so without much effort, we make some basic assumptions about basic problems (haha). In particular, we'd like to assume that $P_k(\cdot\mid x_k,u_k)$ for any $x_k,u_k$ are distributions over at most countable sets, and for any policy, the cost $g_k$ at any step is finite.

In this case, we can define a new term $\tilde{g_k}$ as follows:$$\tilde{g_k}(x_k,\mu_k(x_k))=E_{w_k\sim P_k(\cdot\mid x_k,\mu_k(x_k))}\Set{g_k(x_k,\mu_k(x_k),w_k)\mid x_k,\mu_k(x_k)}$$In other words, $\tilde{g_k}$ is a random variable dependent on $x_k,\mu_k(x_k)$. The expected value is well defined because of the countability of the state space of $P_k$ and finiteness of the cost. Thus, we can let the total cost be $$V_\pi(x_0)=E_{x_1,\ldots,x_n}\Set{g_n(x_n)+\sum\limits_{k=0}^{n-1}\tilde{g_k}(x_k,\mu_k(x_k))}$$This cost can be interpreted as a random variable by letting the state space be the cartesian product of $S_k$ for each $k$.

Show that with our new definition of the total cost, the Dynamic Programming Algorithm generates an optimal policy.

I've been a tad bit sloppy with how I use $g_k$ as I both use it as $g^*_k(x_k,u_k,x_{k+1})$ (the cost is dependent on the state, the action, and the resulting state) and as $\hat{g_k}(x_k,u_k,w_k)$ (the cost is dependent on the state, the action, and the noise term). We can express the latter in terms of the former:$$\hat{g_k}(x_k,u_k,w_k)=g^*_k(x_k,u_k,f(x_k,u_k,w_k))$$So why use both? The answer is that the literature isn't consistent on defining $g_k$. Bertsekas insists on the $\hat{g_k}$ variation since it allows for slightly more general formulations, but others use the $g^*_k$ variation since it avoids some of the random variable complexity in this post.

Top