Graham, I apologize for implying that you thought that the model was not good enough. I now think that you want me to apply a stricter set of rules on how to discover new behaviors.
> "The point is to test the model produced by Eureqa on data that Eureqa has never seen."
OK, first of all that is what Eureqa "kind of" does. It splits the data into a training interval and a validation interval. It uses the training interval to establish an error criteria (min error, max corr coeff, etc) and then continually tests the result on the unfitted validation interval. The way the Eureqa algorithm works is that it will stop when the error criteria on the training interval no longer improves the score on the validation interval (see BTW note below @ ).
The practical matter is that I have personally observed that one can get pretty much similar results if one uses the entire interval for training, and not bothering with the split data approach.
That is just a consequence of dealing with an oscillating yet stationary time-series . The stationarity of the time series is what I think causes this. Say I take the first half of the time-series for training and obtain a fit for the entire interval. Then I run the complement where I use the second half and validate on the first half. For this application the differences turn out to be small because the stationarity of the time series imposes that the same Fourier components are on both first and second half of the time series. And then could take the average of the two fits -- and that is what usually converges to the the fit of the entire time series.
For a non-stationary time series, all bets are off because you aren't even close to being assured that the same Fourier components are on both sides of the interval. That's where you can have problems with over-fitting.
And you are right in that I have no interest in making predictions at the present time. I want to use all available information to determine what the underlying physical factors are. There are no rules when it comes to that and you can "cheat" as much as you want -- you will either lock in to the fundamental formula or you won't. When I am done, I don't think it will be important to even mention that I used Eureqa.
@ BTW, some would argue that Eureqa cheats by evaluating the result on the validation interval, and uses that criteria to continue the search. If it was completely blind, it wouldn't be allowed to do that. Yet, it obviously does that from what I have observed, and that is likely one of the ingredients in how it can uncover new behaviors. That's probably also why the tool is so popular -- in that it takes a pragmatic approach. It's not used for predictions as much as trying to reveal the hidden internal behaviors of a time series.