It seems like the clamping of the network's output in the update is a bit excessive? Given values of x in [0, 1] (valid probabilities), as x -> 0, log(x) -> -infinity, so clamping the minimum value makes sense, but log(1) = 0, so there's no issues with the max value. Pinging @tudor-berariu as well.

Empirically, this might be an issue. I'm running my Rainbow agent with a minimum clamp of 0.001 (arbitrarily chosen), and get the following rewards and Q-values on Space Invaders (the Q-values are in line with what is reported in the Double DQN paper; unfortunately I do not have reported Q-values for Rainbow):

Whereas when I use a minimum clamp of 0.01 and maximum clamp of 0.99 as in this repo, I get the following, which indicates that this prevents the network from accurately estimating Q (note that this is the first time I've ever seen Q-values so far from what I got above, so the issue clearly lies with the clamping):

In def _get_categorical(self, next_states, rewards, mask), when "b" happens to be an integer (e.g., bellman_op clamped to be self.v_max, so b=51), the floor and ceil indexe values, "l" and "u" will be equal. This seems to cause trouble to the distribution projection, as the category "b" will be projected to nowhere.

Closes #1. Not sure if this is the best way to deal with the problem, but it seems to cover a few cases - let me know if you have a better solution.

So to provide an example on the issue, the update relies on qa_probs * (u.float() - b) and qa_probs * (b - l.float()), but if b happens to contain any ints (e.g. in a terminal state where all probability mass is concentrated in one location), then both of these parts of the update turn into qa_probs * 0 and hence the network tries matching a vector of 0s with its softmax output, which will obviously cause problems. Due to the nature of this edge case I believe it has a worse effect on environments with more terminal transitions.

Closes #3. As discussed previously, a max clamp shouldn't be needed. I haven't decoupled the effects of a min clamp of 0.01 versus 0.001, but the latter seems to work fine for my experiments.