Thank you very much for this course, John, reading a lecture or doing an exercise on my train to/from work makes my day (that's ~5h/week dedication).
Since the beginning, I was unable to keep up with your speed, but it did not matter at all. The lectures (and the solved exercises by our fellas!) are a great permanent resource for self-paced learning. Your explanations are a perfect complement to the sketches. And the fact that you brought together so many people spotting mistakes/typos/suggestions/puzzles to improve Spivak and Fong's draft should be an example to the mathematical community.
I prefer to go slow and do every proof myself than follow your rythm. This is the magic of this non-traditional course. And, personally, it also helps to try to sit every two weeks with a coworker doing the same, to confront our solutions in the non-virtual world. If we didn't have the 'true' solutions to the book exercises, we probably wouldn't be doing the course as seriously we are.
My recommendation, therefore, is that you do not give up but do not get fed up either. This should be pleasant for everyone, especially you. Know that your effort is valued, that it makes a huge impact, and follow your own rythm. Even one lecture per week would be fine. If you have to pause for a while, do so. We won't blame you, we thank you for everything you already gave.
And good luck with the gym! Mens sana in corpore sano.