In exercise 8.5, I believe that stochastic environments refers to stochastic state transition as well as reward. Thus, the model table should represent p(s', r|s, a) and not only p(r| s, s', a), creating a distribution model rather than a sample model. When queried the model would generate sample transitions following p(s', r | s, a).

I believe that in order to cope with changing environments some sort of exploratory reward should be given following the intuition of Dyna-Q+. Decaying experience would take time to shift the estimate of Q.

The first point is valid: alpha is not small enough. But the second point is not relevant. The 'down and up again' behaviour of the error is caused by the initialization value of 0.5, which is exactly the true value of the state c. This means during the first few episodes when state c is not updated yet, it has the 0 error. But once it gets updated, the estimated value for state c will fluctuate around the true value for state c but not very close (the higher alpha, the more fluctuating).

Therefore, the initialization of exactly 0.5 causes the down and up again phenomenon.

The following figure empirically demonstrates situations when states are initialized with a different value. It's clear only 0.5 initialization has this issue.

If we break down the squared error of each state a,b,c,d and e (all initialized with 0.5) over training episodes, it again shows state c is the cause.

Hello,

I'm not sure about the solution here.

Firstly, it seems the Bellman optimality equation for q_{*}(s,a) is wrong. Looks like there's a typo where the sum $r + \gamma*max_{a'}q_{*}(s',a')$ has been replaced into the product r_{\gamma}*max_{a'}q_{*}(s',a').

This error has then been propagated to the two solutions given as examples.

If we ignore this typo, I agree with the solution for q_{*}(high, search).

However, I think the answer for q_{*}(high, wait) is wrong. If the action 'wait' is taken, then the next state s' must be 'high' for which the next actions a' = {'high', 'wait'} are possible. I think the answer should be

q_{*}{high, wait) = [r_{wait} + \gamma * max{q_{*}(high, wait), q_{*}(high, search)}]

What do you think?

Hi, the off-policy algorithm suggesed in the pdf is based on the alg on page 110, which is a MC algorithm (running weighted mean, full-episode returns, Q approximations). It may be clearer if the solution is based on the TD algorithm given in page 120 (bootstrapping, single-step rewards, V approximations). Here is a suggestion:

Input: target policy π, behavior policy b with coverage of π
Algorithm param: step size α∈(0,1], discount factor γ∈[0,1]

Initialize V(s) for all s∈S+ arbitrarily and V(terminal) = 0.
For each episode:
  S <- Initial state
  For each step while S not terminal:
      A <- sample b(a|S)
      R, S' <- take action A
      ρ <- π(A|S)/b(A|S)
      Vπ(S) <- Vπ(S) + α[ρR + γVπ(S') - Vπ(S)]
      S <- S'

What do you think? Best,

Line 151 : pi[(i, j)] = 0 Why are all pi[(i,j)] initialized to 0???

Shouldn't pi[(i,j)] -> a ?? i.e., randomly mapped to actions in set A = {-5,-4,-3,-2,-1, 0, 1,2,3,4,5}

Being more precise in the solution, I think s belongs to S (not S+), since the dynamics would not make much sense for the terminal state, i.e., there are no possible next states or even actions.

Hi I'm currently going through the exercises for this book as well: https://github.com/KimMatt/RL_Projects

In your solution to 10.6 how did you get from

Hi, in Policy Improvement of ex4.5, isn't is more clear and succinct to update the policy by selecting the action with maximum q value for every state? such as following:

policy_stable <- true For each s \in S: old_action <- \pi (s) \pi(s) <- argmax_{a} q(s,a) ......(same as original solution)

I think the update equations for Double Expected Sarsa with epsilon-greedy target policy can be:

Q_{1}(S_{t},A_{t})\leftarrow Q_{1}(S_{t},A_{t}) + \alpha\left[R_{t+1}+\gamma\sum_a\pi(a|S_{t+1})Q_{2}(S_{t+1},a)-Q_{1}(S_{t},A_{t})\right]

where

\pi(a|s)=\begin{cases}1-\epsilon+\frac{\epsilon}{|A(s)|}, & if a=argmax_{a}(Q_{1}(s,a')+Q_{2}(s,a'))\\\frac{\epsilon}{|A(s)|}, & otherwise\end{cases}

Numerical error: when gamma = 0.9, v_right = 9.5 and v_left = 5.3, rounded to two significant figures. The former sums over odd powers of gamma, the latter even powers.

Might also want to check gamma = 0.5

Hi, I think that one has to multiply the value of node s' (v_\pi (s')) by the discounting factor gamma. Thank you for your work, it is helping me a lot!

I also found the wording of this question confusing. My best guess is to be "how would the differential TD(0) algorithm be different from tabular TD(0)?" Like you, I also came up with the update formula for the weight vector. (10.10) gives us the TD error, assuming we have the average reward estimate R_bar. From there, I think the only thing you're missing to create the differential TD(0) algorithm is the update for R_bar, which uses the TD error.

In tabular TD(0), we have a single line that updates V(S). For differential TD(0), I think we need to expand that to the following 3 lines to update the weights vector.

Let me know if you think that sounds reasonable.

Also, since you have done a lot of work to produce these solutions, you might want to see if Rich Sutton would honor the offer to provide book solutions if you email him your answers :) He said he would on his site! http://incompleteideas.net/book/solutions.html. Your answers have been invaluable as I work through the textbook, and I'd also be curious to know how close you are to the book solutions.

Hello, I have submitted my code for the exercise 4.7, in place of the current one.

• It is optimized using vectorization and memoization. It takes under a minute to compute the optimal policy.
• The code is configurable through global constants
• The additional constraints from the exercise 4.7 have been implemented
• The code produces a pretty plot of the policy and of the value function

I also didn't get results exactly equal to that of the book, but they're very close.

Plot of the base problem as presented in example 4.2:

After introducing the two nonlinearities:

In your pseudocode for calculating q*, if π is deterministic (as stated in initialization and in pseudocode given for v*), then you don't need to loop on all a∈A in step 2 and you don't need to a to ponderate on all a' for the Q(s,a) calculation.

Again, in step 3 you shouldn't loop on a because you get old-action with the deterministic policy.

Thanks for considering this fix ;) Have a nice day !

The state 15 (with state 13's dynamics changed), isn't equivalent to state 13. It is further away from the upper left terminal state but closer to lower right (left, right and down are equivalent to state 13, but up makes it closer to lower right than up in state 13). I ran your script 4.2.py (by the way going left and right in state 15 leads to 12 and 14 respectively and not to state 15 as written in your script), added a print in the draw function for state 15 and you can see that the decimals are not the same as for state 13. You have to recalculate the whole game. All the states changed slightly (those further away changed less) if you take the decimals into account (compared to running your script for 4.1 which by the way prints value-1 in the board for some weird reason but the accurate state value list is ok). Thanks for your efforts in providing a correction for the exercises !