Right, and if we want to be able to go backwards on a sequence, then we want to allow _negative_ numbers of arrows. So the "graph" that will take us to the left on the Fibonacci sequence is \$\left( \begin{array}{cc} 0 & 1 \\\ 1 & -1 \end{array} \right) .\$ I don't have a good interpretation of what that means as a path count, maybe just the standard difference between path counts? I don't think that works...

@KeithEPerterson a Markov chain is just a normalized path count. (Well if you have rational probabilities. :) )