I think this is classic boundary-value limited behavior. To stretch a testing regime that includes an interval of 100 years to another 200 years back in time requires that the forcing periodicity has enough precision to accommodate phase drift. Otherwise the agreement between model and data gradually gets out of phase.
Earlier, I was using 2.33 years for the QBO period, and now have settled on 2.36 years. There are more than 100 QBO cycles over 300 years so that a difference of 0.03 years in period is enough to cause a shift of at least one average QBO cycle. That may be enough to cause the model to lose coherence with the data over the entire interval, as the QBO is providing the main boundary-value forcing to keep it in phase.
TODO: Next up I will look at Jim's list of past El Nino event dates and see if they match with the data and model.
Comment Source:I think this is classic boundary-value limited behavior. To stretch a testing regime that includes an interval of 100 years to another 200 years back in time requires that the forcing periodicity has enough precision to accommodate phase drift. Otherwise the agreement between model and data gradually gets out of phase.
Earlier, I was using 2.33 years for the QBO period, and now have settled on 2.36 years. There are more than 100 QBO cycles over 300 years so that a difference of 0.03 years in period is enough to cause a shift of at least one average QBO cycle. That may be enough to cause the model to lose coherence with the data over the entire interval, as the QBO is providing the main boundary-value forcing to keep it in phase.
TODO: Next up I will look at Jim's list of past El Nino event dates and see if they match with the data and model.
Note that the common spectral peak found (2.37) matches the 2.366 year period I used in #97 for producing the ENSO model. There are also side-lobes that are evidence of the jitter observed on both QBO and CW. Sidorenkov claims that those lobes track with the same 2-to-1 ratio.
Comment Source:In [comment #97](http://forum.azimuthproject.org/discussion/comment/14618/#Comment_14618), I made the association that the Chandler Wobble period is half the QBO period.
[This white paper by Sidorenkov](http://syrte.obspm.fr/jsr/journees2014/pdf/Sidorenkov.pdf) makes the same association.
This is an excerpt from the paper
Note that the common spectral peak found (2.37) matches the 2.366 year period I used in #97 for producing the ENSO model. There are also side-lobes that are evidence of the jitter observed on both QBO and CW. Sidorenkov claims that those lobes track with the same 2-to-1 ratio.
I'm completely saturated with other work, but Paul's stuff looks interesting. If someone wants to write a blog article about it for Azimuth, just follow the usual recipe: put it on the wiki following the directions in How to blog, get people to criticize it and fix mistakes, etc. I'll join in.
Comment Source:I'm completely saturated with other work, but Paul's stuff looks interesting. If someone wants to write a blog article about it for Azimuth, just follow the usual recipe: put it on the wiki following the directions in [How to blog](http://www.azimuthproject.org/azimuth/show/How+to#blog), get people to criticize it and fix mistakes, etc. I'll join in.
John, I can take a crack on getting a start. Of course none of this is published yet, which is why I am using the forum instead of the wiki to get a discussion going.
Couple of other citations supporting comment #103 above:
Furuya, Masato, Yozo Hamano, and Isao Naito. "Quasi‐periodic wind signal as a possible excitation of Chandler wobble." Journal of Geophysical Research: Solid Earth (1978–2012) 101.B11 (1996): 25537-25546.
"In a narrow-band analysis of the excitation using least squares fit sinusoids, we found that both the amplitude and phase of the inferred excitation near the Chandler band indicate their strong dependence on the assumed Chandler period and that an assumed CW period of 431 days caused atmospheric and observed excitations to agree most closely."
Ponte, Rui M., and Detlef Stammer. "Role of ocean currents and bottom pressure variability on seasonal polar motion." Journal of Geophysical Research: Oceans (1978–2012) 104.C10 (1999): 23393-23409. pdf
Comment Source:John, I can take a crack on getting a start. Of course none of this is published yet, which is why I am using the forum instead of the wiki to get a discussion going.
---
Couple of other citations supporting comment #103 above:
Furuya, Masato, Yozo Hamano, and Isao Naito. "Quasi‐periodic wind signal as a possible excitation of Chandler wobble." Journal of Geophysical Research: Solid Earth (1978–2012) 101.B11 (1996): 25537-25546.
> "In a narrow-band analysis of the excitation using least squares fit sinusoids, we found that both the amplitude and phase of the inferred excitation near the Chandler band indicate their strong dependence on the assumed Chandler period and that an assumed CW period of 431 days caused atmospheric and observed excitations to agree most closely."
Ponte, Rui M., and Detlef Stammer. "Role of ocean currents and bottom pressure variability on seasonal polar motion." Journal of Geophysical Research: Oceans (1978–2012) 104.C10 (1999): 23393-23409. [pdf](http://onlinelibrary.wiley.com/doi/10.1029/1999JC900222/pdf)
This is OT re. ENSO but it's an attempt at a wavelet solution to something not entirely dissimilar, and you never know, might be connected via the polar bridge:
Comment Source:This is OT re. ENSO but it's an attempt at a wavelet solution to something not entirely dissimilar, and you never know, might be connected via the polar bridge:
Oleg Pokrovsky, [Impact of Atlantic and Pacific decadal oscillation on ice extent in Russian Arctic seas (2010)](http://www.arcticfrontiers.com/downloads/arctic-frontiers-2010/conference-presentations-7/wednesday-27-january-2010/part-i-ice-and-climate-including-paleo-climate/1017-07-pokrovsky/file) (presentation slides).
The SOI metric for ENSO is really leaning negative.
Once it hits -20 for a sustained number of months, we are back to 1998 conditions.
Look how much it dives over the course in the few recent days of May.
All of my model fits so far have shown this negative excursion starting around 2015, but that doesn't really confirm much.
Comment Source:The SOI metric for ENSO is really leaning negative.
Once it hits -20 for a sustained number of months, we are back to 1998 conditions.
Look how much it dives over the course in the few recent days of May.
All of my model fits so far have shown this negative excursion starting around 2015, but that doesn't really confirm much.
Jim, I may have. I just don't think a prediction is a game changer. The sign-metric I am using only fits the correct sign of an excursion at an 80% level at most. So that once every 5 years it will have the wrong sign.
Comment Source:Jim, I may have. I just don't think a prediction is a game changer. The sign-metric I am using only fits the correct sign of an excursion at an 80% level at most. So that once every 5 years it will have the wrong sign.
Another look at recent ENSO activity. When the SOI reaches that -20 mark and stays, that is nearing 1998 territory for significant ocean sloshing amplitude:
Comment Source:Another look at recent ENSO activity. When the SOI reaches that -20 mark and stays, that is nearing 1998 territory for significant ocean sloshing amplitude:
There are a large number of methods and programming pages you and Dara have covered which I think should be written up on the wiki.
For example I have no idea that you were using a sign metric in:
The sign-metric I am using only fits the correct sign of an excursion at an 80% level at most.
I'd find the write-up of these details fascinating :).
I'm hoping to take up a suggestion of Dave Tanzer's to work on Azimuth's educational goals; which I think of as needing some kind of applied green maths and practical sustainability engineering curriculum.
Comment Source:There are a large number of methods and programming pages you and Dara have covered which I think should be written up on the wiki.
For example I have no idea that you were using a sign metric in:
> The sign-metric I am using only fits the correct sign of an excursion at an 80% level at most.
I'd find the write-up of these details fascinating :).
I'm hoping to take up a suggestion of Dave Tanzer's to work on Azimuth's educational goals; which I think of as needing some kind of applied green maths and practical sustainability engineering curriculum.
Jim, I don't know why I am not using the Wiki. It's not that I am not accustomed to it, as I started using a wiki at work when they first came out years ago.
I guess my issues are several-fold:
I would rather do work using a semantic wiki or feed information into a semantic web format (which is my context modeling project)
Don't want to give the impression that these results are iron-clad, which is what a wiki has grown to signify
When I sense that there is some support and someone can start duplicating and perhaps extending the work, that would be a good time to consider structuring some sort of educational module.
BTW, the sign-metric is simply counting the fraction of the time that the model and data shows the same sign-excursion. If it always agrees it is 1.0 and if it the excursions are random then 0.5.
Comment Source:Jim, I don't know why I am not using the Wiki. It's not that I am not accustomed to it, as I started using a wiki at work when they first came out years ago.
I guess my issues are several-fold:
* I would rather do work using a semantic wiki or feed information into a semantic web format (which is my context modeling project)
* Don't want to give the impression that these results are iron-clad, which is what a wiki has grown to signify
When I sense that there is some support and someone can start duplicating and perhaps extending the work, that would be a good time to consider structuring some sort of educational module.
BTW, the sign-metric is simply counting the fraction of the time that the model and data shows the same sign-excursion. If it always agrees it is 1.0 and if it the excursions are random then 0.5.
A blog and a forum work great as a scratchpad for thinking out loud. I used to run a blog called http://mobjectivist.blogspot.com, which was mainly about peak oil. At some point I decided to cull from the interesting posts and write a book on the topic "The Oil ConunDrum". Then I stopped writing to that blog, like you stop writing to a filled-up lab notebook. As it turns out, the blog contained all sorts of charts and graphs hosted by an imaging service and those started to disappear. Glad that I put something together for posterity, as the blog is filled with missing links now !
I guess my point is that one does what can to maintain and organize their thoughts, and it is a judgement call to anticipate what you will need to maintain in the future.
In thinking out loud about ENSO, one of the false starts may be about the impact of TSI variations. In the ongoing discussion, I started to believe that it was an important driver but have now started to downplay its relevance. On a Wiki making these kinds of linear narrative changes is awkward -- do you just delete the false start, or do you explain why the change was made? Lots of space can be expended on what is not the correct approach. I am waiting for the point when I can pull the trigger and be confident that I have the right mechanisms in place to lay it all out.
Comment Source:A blog and a forum work great as a scratchpad for thinking out loud. I used to run a blog called http://mobjectivist.blogspot.com, which was mainly about peak oil. At some point I decided to cull from the interesting posts and write a book on the topic "The Oil ConunDrum". Then I stopped writing to that blog, like you stop writing to a filled-up lab notebook. As it turns out, the blog contained all sorts of charts and graphs hosted by an imaging service and those started to disappear. Glad that I put something together for posterity, as the blog is filled with missing links now !
I guess my point is that one does what can to maintain and organize their thoughts, and it is a judgement call to anticipate what you will need to maintain in the future.
In thinking out loud about ENSO, one of the false starts may be about the impact of TSI variations. In the ongoing discussion, I started to believe that it was an important driver but have now started to downplay its relevance. On a Wiki making these kinds of linear narrative changes is awkward -- do you just delete the false start, or do you explain why the change was made? Lots of space can be expended on what is not the correct approach. I am waiting for the point when I can pull the trigger and be confident that I have the right mechanisms in place to lay it all out.
The point I am trying to get across is that a regular periodic function can create the erratic behavior that results in ENSO.
So when we start with this:
it results in this
Here is the beginning of an elevator pitch : The periodic harmony behind the equatorial Pacific dynamics are obscured by the non-linear sloshing transformation of the ocean's volume.
My premise has always been that since days, seasons, and the tides are so periodic why do the ocean's necessarily have to be chaotic? Just because they appear chaotic doesn't mean that they are. This is the basic hydrodynamic equation that justifies a predictable deterministic outcome.
Comment Source:I have a new blog post called The Hidden Harmony of ENSO here -- http://ContextEarth.com/2015/05/31/the-hidden-harmony-of-enso
The point I am trying to get across is that a regular periodic function can create the erratic behavior that results in ENSO.
So when we start with this:
it results in this
Here is the beginning of an elevator pitch : The periodic harmony behind the equatorial Pacific dynamics are obscured by the non-linear sloshing transformation of the ocean's volume.
My premise has always been that since days, seasons, and the tides are so periodic why do the ocean's necessarily have to be chaotic? Just because they *appear* chaotic doesn't mean that they are. This is the basic hydrodynamic equation that justifies a predictable deterministic outcome.
Thanks Graham,
I see how that is accomplished via the experiments category. The table of contents shows the linear narrative of the experimental approaches taken. It makes it more like a mini-blog within a wiki.
I am thinking that with my previous comment #115, it could capture successive refinements of the most concise and plausible ENSO model fits. The table of contents would track the evolution effectively.
Comment Source:Thanks Graham,
I see how that is accomplished via the experiments category. The table of contents shows the linear narrative of the experimental approaches taken. It makes it more like a mini-blog within a wiki.
I am thinking that with my previous comment #115, it could capture successive refinements of the most concise and plausible ENSO model fits. The table of contents would track the evolution effectively.
WebHubTel: just to second what Graham is saying, I think you should put your work on one or more "Experiments" pages in the wiki. It's easier to organize things there: for example, you can have a page that's organized in a chronological narrative and also have a section or page that describes the "current state of the art", etc. The Forum here is designed for conversations, and it seems to work pretty well for that, but if I want to understand what you've done it can be a bit difficult to extract it from a collection of conversations.
Comment Source:WebHubTel: just to second what Graham is saying, I think you should put your work on one or more "Experiments" pages in the wiki. It's easier to organize things there: for example, you can have a page that's organized in a chronological narrative _and also_ have a section or page that describes the "current state of the art", etc. The Forum here is designed for conversations, and it seems to work pretty well for that, but if I want to understand what you've done it can be a bit difficult to extract it from a collection of conversations.
Comment Source:John, OK I will start using an Experiments page and see how that works.
Here is a start [azimuth/show/Experiments+in+ENSO+modeling](http://www.azimuthproject.org/azimuth/show/Experiments+in+ENSO+modeling)
Of course this model is totally at odds with the Zebiac-Cane model, which is also described in the wiki. What the wiki says is that the Zebiac-Cane model shows "somewhat realistic ENSO behavior", yet I have never actually seen a fit of the Z-C model to any extended ENSO time-series data.
This is where it gets tricky in modifying the Wiki narrative to include the sloshing model alongside the Z-C model and the others described there. Do we list it alongside the others as a candidate of equal standing? I am not going to make that call until I get some buy-in from other wiki contributors.
Comment Source:I have the first wiki "draft" in place for the sloshing ENSO model here: [azimuth/show/Experiments+in+ENSO+modeling](http://www.azimuthproject.org/azimuth/show/Experiments+in+ENSO+modeling)
Of course this model is totally at odds with the Zebiac-Cane model, which is also described in the wiki. What the wiki says is that the Zebiac-Cane model shows *"somewhat realistic ENSO behavior"*, yet I have never actually seen a fit of the Z-C model to any extended ENSO time-series data.
This is where it gets tricky in modifying the Wiki narrative to include the sloshing model alongside the Z-C model and the others described there. Do we list it alongside the others as a candidate of equal standing? I am not going to make that call until I get some buy-in from other wiki contributors.
There's an initial wiki page on the ZC model you can link to and perhaps add to and improve there and review from your experiments page?
Comment Source:There's an initial wiki page on the ZC model you can link to and perhaps add to and improve there and review from your experiments page?
Jim,
I placed a bi-directional link between the ENSO wiki page and this experimental modeling page via /show/ENSO#azimuth_experiments.
Also am not sure what to add about the ZC model other than what I said above, and that must be vetted by a second opinion. The most I was able to find was that the ZB model "produced plausible simulations of ENSO". Yet there are no charts demonstrating the plausibility.
Comment Source:Jim,
I placed a bi-directional link between the ENSO wiki page and this experimental modeling page via [/show/ENSO#azimuth_experiments](http://www.azimuthproject.org/azimuth/show/ENSO#azimuth_experiments).
Also am not sure what to add about the ZC model other than what I said above, and that must be vetted by a second opinion. The most I was able to find was that the ZB model ["produced plausible simulations of ENSO"](http://www.asp.ucar.edu/colloquium/2000/Lectures/battisti1.html). Yet there are no charts demonstrating the plausibility.
I am starting to look more at the distinctions between my sloshing formulation and that of the parametric recharge oscillator, which is one of the favorite models in the literature [1]. Look at Eq 8 and 9 in Stein et al, which leads to theor Eq 27.
$ T''(t) + \lambda(1-T^2)T'(t) + T(t) = F(t) $
This is the basic wave equation, with a first derivative drag term that has a non-linear behavior, decreasing the drag for larger excursions of T. I ignore this drag term and assume the behavior is largely inviscid and that any drag effect is minor.
With the forcing term F(t) consisting of QBO and Chandler wobble, most of the ENSO behavior is simply modeled. Add in the Mathieu/Hill modulation and the details start to emerge, which is what I have added to the Azimuth Wiki page.
The prevailing question is why aren't the climate scientists working the angle of using known forcing factors in the formulations that they originally devised? It's like an electrical engineer not using Kirchoff's and Ohm's laws in evaluating the response of a resonant circuit. Instead, they bypass the simple path and immediately assume that the road must be via chaotic bifurcations and the messiness that approach entails.
It's enough to drive one up the wall :) And that is not the path of least resistance :):)
[1] K. Stein, A. Timmermann, N. Schneider, F.-F. Jin, and M. F. Stuecker, “ENSO seasonal synchronization theory,” Journal of Climate, vol. 27, no. 14, pp. 5285–5310, 2014. PDF
Take a look at these two papers that just came out as well. This is the Jin-Neelin-Ghil model. which is an extension of [1]. I have the PDFs available.
[2] D. Mukhin, E. Loskutov, A. Mukhina, A. Feigin, I. Zaliapin, and M. Ghil, “Predicting critical transitions in ENSO models. Part I: Methodology and simple models with memory,” Journal of Climate, vol. 28, no. 5, pp. 1940–1961, 2015.
[3] D. Mukhin, D. Kondrashov, E. Loskutov, A. Gavrilov, A. Feigin, and M. Ghil, “Predicting critical transitions in ENSO models. Part II: Spatially dependent models,” Journal of Climate, vol. 28, no. 5, pp. 1962–1976, 2015.
Comment Source:I am starting to look more at the distinctions between my sloshing formulation and that of the parametric recharge oscillator, which is one of the favorite models in the literature [1]. Look at Eq 8 and 9 in Stein et al, which leads to theor Eq 27.
$ T''(t) + \lambda(1-T^2)T'(t) + T(t) = F(t) $
This is the basic wave equation, with a first derivative drag term that has a non-linear behavior, decreasing the drag for larger excursions of T. I ignore this drag term and assume the behavior is largely inviscid and that any drag effect is minor.
With the forcing term F(t) consisting of QBO and Chandler wobble, most of the ENSO behavior is simply modeled. Add in the Mathieu/Hill modulation and the details start to emerge, which is what I have added to the Azimuth Wiki page.
The prevailing question is why aren't the climate scientists working the angle of using known forcing factors in the formulations that they originally devised? It's like an electrical engineer not using Kirchoff's and Ohm's laws in evaluating the response of a resonant circuit. Instead, they bypass the simple path and immediately assume that the road must be via chaotic bifurcations and the messiness that approach entails.
It's enough to drive one up the wall :) And that is not the path of least resistance :) :)
[1] K. Stein, A. Timmermann, N. Schneider, F.-F. Jin, and M. F. Stuecker, “ENSO seasonal synchronization theory,” Journal of Climate, vol. 27, no. 14, pp. 5285–5310, 2014. [PDF](http://www.researchgate.net/profile/Malte_Stuecker/publication/267098864_ENSO_seasonal_synchronization_theory/links/5445d08a0cf2f14fb80f05e0.pdf)
Take a look at these two papers that just came out as well. This is the Jin-Neelin-Ghil model. which is an extension of [1]. I have the PDFs available.
[2] D. Mukhin, E. Loskutov, A. Mukhina, A. Feigin, I. Zaliapin, and M. Ghil, “Predicting critical transitions in ENSO models. Part I: Methodology and simple models with memory,” Journal of Climate, vol. 28, no. 5, pp. 1940–1961, 2015.
[3] D. Mukhin, D. Kondrashov, E. Loskutov, A. Gavrilov, A. Feigin, and M. Ghil, “Predicting critical transitions in ENSO models. Part II: Spatially dependent models,” Journal of Climate, vol. 28, no. 5, pp. 1962–1976, 2015.
This looks like a breakthrough paper on ENSO [1]. The 2nd author Abarca-del-Rio has done a lot of research on understanding the earth's angular momentum and he has helped me in the past with analysis.
As I read through the paper, it appears that it is right up the alley for the mathematical physicists on Azimuth.
The rather strong assertion that they make is
"The most important result of this study is that the so-called SOI anomaly corresponds to the dynamics of a nolinear oscillator having complex regularities and exhibits an acceptable level of accuracy of average non-linear predictability in the range between 2 and 4 years of time span."
And the eye-opener to me is the difficulty they have with the El Nino event starting in 1981. That is precisely the point in time that has been most problematic in the deterministic model of ENSO that I have been working on (see the first comment in this long thread). You can see it in the figure below which is reproduced from their paper. It appears as if the event is delayed according to the model but the data forces it to loop back.
I had suggested this anomaly has something to do with a discrete Pacific Ocean shift, but it may also be related to a peak in the strength of TSI activity, which is shown in the wiki page.
[1] H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015. PDF
Comment Source:This looks like a breakthrough paper on ENSO [1]. The 2nd author Abarca-del-Rio has done a lot of research on understanding the earth's angular momentum and he has helped me in the past with analysis.
As I read through the paper, it appears that it is right up the alley for the mathematical physicists on Azimuth.
The rather strong assertion that they make is
> "The most important result of this study is that the so-called SOI anomaly corresponds to the dynamics of a nolinear oscillator having complex regularities and exhibits an acceptable level of accuracy of average non-linear predictability in the range between 2 and 4 years of time span."
And the eye-opener to me is the difficulty they have with the El Nino event starting in 1981. That is precisely the point in time that has been most problematic in the deterministic model of ENSO that I have been working on (see the [first comment](/discussion/1608/enso-revisit/p1) in this long thread). You can see it in the figure below which is reproduced from their paper. It appears as if the event is delayed according to the model but the data forces it to loop back.
I had suggested this anomaly has something to do with a discrete Pacific Ocean shift, but it may also be related to a peak in the strength of TSI activity, which [is shown in the wiki page](http://www.azimuthproject.org/azimuth/show/Experiments+in+ENSO+modeling).
[1] H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015. [PDF](http://arxiv.org/pdf/1506.04066.pdf)
Comment Source:Another potentially breakthrough paper
The 11-year solar cycle in current reanalyses: a (non)linear attribution study of the middle atmosphere
http://www.atmos-chem-phys.net/15/6879/2015/acp-15-6879-2015.html
This paper
[1] H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015. [pdf](http://arxiv.org/pdf/1506.04066.pdfO
has me thinking about the importance of a Pacific Ocean event occurring around 1980 impacting the ENSO dynamics. The model fit I am using up to 1980 is excellent, which I will explain in a later comment. But after 1980 it looks to effect a phase reversal --
This is a plot of the LHS vs the RHS of the DiffEq
f''(t)+kf(t) versus F(t)
where F(t) is the forcing function. Note that right near 1980, the values hit a degenerate inflection point coinciding with a value of zero. What I think this means is that a slight push in any direction could cause the motion to reverse sign. So if some forcing noise (such as volcanic eruptions or earthquakes, see El Chichon 1982) causes f(t) to jump above or below zero, that is the direction that the signal may head, regardless on whether the f(t) region is convex or concave.
So this may be a metastable behavior that we are detecting. I have to find some examples of this in the literature to see if this is a plausible explanation.
Also, get a load of the model fit before 1980. This is a combination of QBO for the fine detail and Chandler wobble, TSI and long-term tidal beat periods for the decadal variability. Once the fit gets above a 0.9 correlation coefficient, it really starts to lock in to place.
Comment Source:This paper
[1] H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015. [pdf](http://arxiv.org/pdf/1506.04066.pdfO
has me thinking about the importance of a Pacific Ocean event occurring around 1980 impacting the ENSO dynamics. The model fit I am using up to 1980 is excellent, which I will explain in a later comment. But after 1980 it looks to effect a phase reversal --
This is a plot of the LHS vs the RHS of the DiffEq
f''(t)+kf(t) versus F(t)
where F(t) is the forcing function. Note that right near 1980, the values hit a degenerate inflection point coinciding with a value of zero. What I think this means is that a slight push in any direction could cause the motion to reverse sign. So if some forcing noise (such as volcanic eruptions or earthquakes, see El Chichon 1982) causes f(t) to jump above or below zero, that is the direction that the signal may head, regardless on whether the f(t) region is convex or concave.
So this may be a metastable behavior that we are detecting. I have to find some examples of this in the literature to see if this is a plausible explanation.
Also, get a load of the model fit before 1980. This is a combination of QBO for the fine detail and Chandler wobble, TSI and long-term tidal beat periods for the decadal variability. Once the fit gets above a 0.9 correlation coefficient, it really starts to lock in to place.
Anyone getting excited yet about the ENSO model ? :)
Again this is a plot of the LHS vs the RHS of the SOI differential wave equation
SOI''(t)+k SOI(t) versus F(t)
where F(t) is the forcing function.
I added some forcing changes at 1980 to get the full 1880 to 2013 fit
I also pushed the characteristic freq back to a period of 4.25 years, and the QBO emerges as a very strong factor.
The bottom panel is the power spectra of the residual noise (see middle panel "diff"). It's mostly flat, with the rise to the right corresponding to jitter about the main QBO period of 2.33 years.
IMO this is definitely not an overfit, but just a different way of looking at the model described in the ARXIV ENSO sloshing paper. What one can do is extend the forcing and then do an integration of the LHS to project the ENSO waveform into the future.
Where are the pro climate scientists on this? This is so fundamentally basic that it boggles the mind.
Comment Source:Anyone getting excited yet about the ENSO model ? :)
Again this is a plot of the LHS vs the RHS of the SOI differential wave equation
SOI''(t)+k SOI(t) versus F(t)
where F(t) is the forcing function.
I added some forcing changes at 1980 to get the full 1880 to 2013 fit
I also pushed the characteristic freq back to a period of 4.25 years, and the QBO emerges as a very strong factor.
The bottom panel is the power spectra of the residual noise (see middle panel "diff"). It's mostly flat, with the rise to the right corresponding to jitter about the main QBO period of 2.33 years.
IMO this is definitely not an overfit, but just a different way of looking at the model described in the ARXIV ENSO sloshing paper. What one can do is extend the forcing and then do an integration of the LHS to project the ENSO waveform into the future.
Where are the pro climate scientists on this? This is so fundamentally basic that it boggles the mind.
Comment Source:Comparison of a wavelet scalogram of transformed SOI data against the model
Horizontal time scale is in months from 1880.
blog post here:
[http://contextearth.com/2015/07/11/enso-transformation/](http://contextearth.com/2015/07/11/enso-transformation/)
Experimenting with the ENSO model and the transition that I detect and Astudillo also observe (see comment #125), it may be that a phase reversal is happening. What I did was fit a model between 1880 and 1980 with a correlation coefficient approaching 0.8. I then extended the model as an out-of-band forecast beyond 1880 and noticed that the model appeared sign reversed with respect to the transformed data,
I reversed only the forcing sign and the correlation coefficient maintained a value close to 0.8. See the curve below. I am using essentially the same forcing parameters of QBO, Chandler wobble, TSI, and tidal periods as in the previous comments but calculating using a spreadsheet instead of Mathematica.
I mentioned this before (comment #127), but if a second-order DiffEq shows an inflection point at amplitudes close to zero, the direction that the curve will take may be very sensitive to a forcing disturbance. Just like in a cavity, the standing wave's initial direction behaves like a metastable point. This could explain much as the disturbance may be very subtle and therefore a highly plausible effect. For example, It could be a peak in the TSI right around 1980.
Comment Source:Experimenting with the ENSO model and the transition that I detect and Astudillo also observe (see comment #125), it may be that a phase reversal is happening. What I did was fit a model between 1880 and 1980 with a correlation coefficient approaching 0.8. I then extended the model as an out-of-band forecast beyond 1880 and noticed that the model appeared sign reversed with respect to the transformed data,
I reversed only the forcing sign and the correlation coefficient maintained a value close to 0.8. See the curve below. I am using essentially the same forcing parameters of QBO, Chandler wobble, TSI, and tidal periods as in the previous comments but calculating using a spreadsheet instead of Mathematica.
I mentioned this before (comment #127), but if a second-order DiffEq shows an inflection point at amplitudes close to zero, the direction that the curve will take may be very sensitive to a forcing disturbance. Just like in a cavity, the standing wave's initial direction behaves like a metastable point. This could explain much as the disturbance may be very subtle and therefore a highly plausible effect. For example, It could be a peak in the TSI right around 1980.
The striking commonality of the forcing factors between the QBO and ENSO models is the most critical piece in solidifying the overall concept. In other words, putting effort into either model improves both, at least that is what I am seeing.
Comment Source:This thread has been dormant for a while because the activity has been on the QBO recently.
https://forum.azimuthproject.org/discussion/1640/predictability-of-the-quasi-biennial-oscillation#latest
The striking commonality of the forcing factors between the QBO and ENSO models is the most critical piece in solidifying the overall concept. In other words, putting effort into either model improves both, at least that is what I am seeing.
This is cool. I regressed (i.e. trained) the ENSO model on the UEP coral proxy data from 1880 to 1977 and then used that solution to back extrapolate to 1650.
The correlation coefficient over the 100-year training interval is 0.74 and over the entire 300+ year region it is 0.54. The waveform looks jagged because there is only one data point per year.
This uses the same parameters as the ENSO SOI model and the QBO model.
The multiple regression takes less than a second to compute.
Comment Source:This is cool. I regressed (i.e. trained) the ENSO model on the UEP coral proxy data from 1880 to 1977 and then used that solution to back extrapolate to 1650.
The correlation coefficient over the 100-year training interval is 0.74 and over the entire 300+ year region it is 0.54. The waveform looks jagged because there is only one data point per year.
This uses the same parameters as the ENSO SOI model and the QBO model.
The multiple regression takes less than a second to compute.
Over the validation interval, the sign of the excursion matches 143 out of 230 times (i.e. for each year from 1650 to 1880). Using the binomial theorem with P=0.5, this number of matches (or greater) should occur with probability 0.00008 strictly due to chance.
On the training side (which doesn't count), the sign of the excursion matches 70 out of 97 times (each year from 1880 to 1977). This number of matches could occur by chance with probability 0.000007. Even though the number is less than 0.00008, it is meaningless since the training side is prone to overfitting .
However, on the validation side, the probability is highly significant because it is an out-of-band blind test. The pattern in the underlying oscillations is very likely deterministic as also shown by Astudillo, but now extended over 3 centuries plus some.
[1]H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015.
Comment Source:http://ContextEarth.com/2015/10/03/raising-the-bar-on-enso-model-validation
Over the validation interval, the sign of the excursion matches 143 out of 230 times (i.e. for each year from 1650 to 1880). Using the binomial theorem with P=0.5, this number of matches (or greater) should occur with probability 0.00008 strictly due to chance.
http://stattrek.com/online-calculator/binomial.aspx
On the training side (which doesn't count), the sign of the excursion matches 70 out of 97 times (each year from 1880 to 1977). This number of matches could occur by chance with probability 0.000007. Even though the number is less than 0.00008, it is meaningless since the training side is prone to overfitting .
However, on the validation side, the probability is highly significant because it is an out-of-band blind test. The pattern in the underlying oscillations is very likely deterministic as also shown by Astudillo, but now extended over 3 centuries plus some.
[1]H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015.
Comment Source:Wild training interval for only 5 coefficients over a 15 year span
ENSO shows very stationary time-series traits if the data is transformed via the wave equation. http://contextearth.com/2015/07/11/enso-transformation/
I figured out an intriguing ENSO pattern, based on precisely defining sidebands +/- on each side of the exact biennial [1] period. The sidebands appear to follow the period of the three triaxial wobbles in the earth's angular momentum [2]. It is able to extrapolate most of the wave-equation transformed curve by fitting to a short interval. This is simply a consequence of the interval containing enough information to reconstruct the rest of the stationary time series.
Its not a lot different from what the sloshing formulation I started with, but the symmetry and canonical form is now much more readily apparent. The three sideband periods are 6.5 year, 14.3 year, and 18.6 year, which you can understand from reading the fractured English in reference [2].
[1] S.-R. Yeo and K.-Y. Kim, “Global warming, low-frequency variability, and biennial oscillation: an attempt to understand the physical mechanisms driving major ENSO events,” Climate Dynamics, vol. 43, no. 3–4, pp. 771–786, 2014. http://link.springer.com/article/10.1007/s00382-013-1862-1/fulltext.html
[2] Wang, Wen-Jun, W.-B. Shen, and H.-W. Zhang, “Verifications for Multiple Solutions of Triaxial Earth Rotation,” IERS Workshop on Conventions Bureau International des Poids et Mesures (BIPM), Sep. 2007. http://www1.bipm.org/utils/en/events/iers/Wang.pdf
Comment Source:I figured out an intriguing ENSO pattern, based on precisely defining sidebands +/- on each side of the exact biennial [1] period. The sidebands appear to follow the period of the three triaxial wobbles in the earth's angular momentum [2]. It is able to extrapolate most of the [wave-equation transformed curve](http://contextearth.com/2015/07/11/enso-transformation/) by fitting to a short interval. This is simply a consequence of the interval containing enough information to reconstruct the rest of the stationary time series.
Its not a lot different from what the sloshing formulation I started with, but the symmetry and canonical form is now much more readily apparent. The three sideband periods are 6.5 year, 14.3 year, and 18.6 year, which you can understand from reading the fractured English in reference [2].
[1] S.-R. Yeo and K.-Y. Kim, “Global warming, low-frequency variability, and biennial oscillation: an attempt to understand the physical mechanisms driving major ENSO events,” Climate Dynamics, vol. 43, no. 3–4, pp. 771–786, 2014. http://link.springer.com/article/10.1007/s00382-013-1862-1/fulltext.html
[2] Wang, Wen-Jun, W.-B. Shen, and H.-W. Zhang, “Verifications for Multiple Solutions of Triaxial Earth Rotation,” IERS Workshop on Conventions Bureau International des Poids et Mesures (BIPM), Sep. 2007. http://www1.bipm.org/utils/en/events/iers/Wang.pdf
"Interestingly, the El Nino events are aware of reason for Earth accelerations while La Nina events for Earth decelerations. El Nino events have discovered of quasi-period of 2~7 yr. Notice that bifurcation may take place in the process of Earth’s free acceleration for LOD, maybe the El Nino events are caused by the free accelerations of periods 7.3 yr with bifurcations. In 7.3 years, the ocean water near the equator may be translated about 0.402 m as huge matter redistribution for water fluid persisting in period of intra annual to one side of the ocean. So the water flows to pile up in one side of Pacific Ocean eastward or reflux westward. The pile-up of the equatorial water brings warmth eastward or westward and changes the inherent equilibrium of the equatorial atmosphere so that causes the weather to change yielding El Nino or La Nina. Hence this may be the dynamical force causing ENSO. Evidently, there are some terms of forced fluctuations in LOD. But as known, forced fluctuations may not cause accelerations. Forced fluctuations are made from the variations of atmosphere and ocean flood while free fluctuations yield centrifugal accelerations. Only free fluctuations may cause accelerations and further centrifugal forces that yield water flowing periodically one way of a direction."
"However, the periods are difficult to explain the ENSO occurring near Equator in Pacific Ocean. We deduce here the free fluctuations for LOD may have inherent period of 14.6 yr from the triaxiality of real Earth. As known, the ENSO events have no inherent periods but from 1.5 to 7 yr as. We make some discussions here for the phenomenon. As discussed in the text, the model of rotation equations has been seen evidently as nonlinear and we solve the model with linearized approach of direct product decomposition. The real result of the solution must be treated in product formulation. So the linearized solution of period 14.6 yr must be affected by product of other fluctuations in the process. Thus the inherent period of 14.6 may become instable and may yield bifurcation to split into series of semi-periods such as 7.3 yr, 3.65 yr, 1.825 yr and such on. This may be the different periods for ENSO to occur as pseudo-periods 7 yr, 4 yr and 2 yr. On the other hand, the centrifugal force of acceleration amount 7.55 mm/yr2 also may not stable in the nonlinear process. Some times the centrifugal force is strengthened and other times decreased so that the ENSO events may burst strong some times and weak other times. Generally, the ENSO events occur in this nonlinear process unstably caused by the free fluctuations of LOD. Here we discover the dynamical force of free accelerations of rotation velocity. This model solves the resources of dynamical force for ENSO but is to study further and deeper. In order to test the two free acceleration variations deduced in theoretically in this study of data series for El Nino and Southern Oscillation (ENSO) events, subharmonic beating and synchronic need to be noticed. The appearance quasi-period of ENSO may not be identical to the prediction here. The main quasi-period of ENSO may be correlated to the free oscillation periods but with variations of subharmonic beating and nonlinear coupling. Anyway, the free acceleration of rotation velocity may be clue to discuss quasi-period of ENSO."
and conclusion
"Nonlinear dynamics makes the Earth rotation breaking a revolution so that updated the whole polar motion science. We must humor the conformance of the advancing history without against the tidal current. Professor E Grafarend [2005] published a course about nonlinearity in geodesy as well as in geophysics. Therefore, Earth rotation theory must be reformed to a new Convention in which the two free wobbles should be stated clearly and the similar accompanied consequences as those of the Chandler wobble. Simultaneously, the whole theory of Earth rotation must face to new challenges of two stable and an unstable component solutions with chaotic final state."
They were arguing that the ENSO period would come about by bifurcating the 14-year period down to the known ENSO range of 2-7 years. While I think the key is that the 14-year period is multiplied against the 2-year biennial period, generating the time-series profile that matches the ENSO data.
Comment Source:Here is my narrative to elaborate on the previous comment: http://contextearth.com/2015/12/12/biennial-connection-from-qbo-to-enso/
Figured some of this out by trying to decipher [ref [2] in prior comment](http://www1.bipm.org/utils/en/events/iers/Wang.pdf).
>"Interestingly, the El Nino events are aware of reason for Earth accelerations while La Nina events for Earth decelerations. El Nino events have discovered of quasi-period of 2~7 yr. Notice that bifurcation may take place in the process of Earth’s free acceleration for LOD, maybe the El Nino events are caused by the free accelerations of periods 7.3 yr with bifurcations. In 7.3 years, the ocean water near the equator may be translated about 0.402 m as huge matter redistribution for water fluid persisting in period of intra annual to one side of the ocean. So the water flows to pile up in one side of Pacific Ocean eastward or reflux westward. The pile-up of the equatorial water brings warmth eastward or westward and changes the inherent equilibrium of the equatorial atmosphere so that causes the weather to change yielding El Nino or La Nina. Hence this may be the dynamical force causing ENSO. Evidently, there are some terms of forced fluctuations in LOD. But as known, forced fluctuations may not cause accelerations. Forced fluctuations are made from the variations of atmosphere and ocean flood while free fluctuations yield centrifugal accelerations. Only free fluctuations may cause accelerations and further centrifugal forces that yield water flowing periodically one way of a direction."
> "However, the periods are difficult to explain the ENSO occurring near Equator in Pacific Ocean. We deduce here the free fluctuations for LOD may have inherent period of 14.6 yr from the triaxiality of real Earth. As known, the ENSO events have no inherent periods but from 1.5 to 7 yr as. We make some discussions here for the phenomenon. As discussed in the text, the model of rotation equations has been seen evidently as nonlinear and we solve the model with linearized approach of direct product decomposition. The real result of the solution must be treated in product formulation. So the linearized solution of period 14.6 yr must be affected by product of other fluctuations in the process. Thus the inherent period of 14.6 may become instable and may yield bifurcation to split into series of semi-periods such as 7.3 yr, 3.65 yr, 1.825 yr and such on. This may be the different periods for ENSO to occur as pseudo-periods 7 yr, 4 yr and 2 yr. On the other hand, the centrifugal force of acceleration amount 7.55 mm/yr2 also may not stable in the nonlinear process. Some times the centrifugal force is strengthened and other times decreased so that the ENSO events may burst strong some times and weak other times. Generally, the ENSO events occur in this nonlinear process unstably caused by the free fluctuations of LOD. Here we discover the dynamical force of free accelerations of rotation velocity. This model solves the resources of dynamical force for ENSO but is to study further and deeper. In order to test the two free acceleration variations deduced in theoretically in this study of data series for El Nino and Southern Oscillation (ENSO) events, subharmonic beating and synchronic need to be noticed. The appearance quasi-period of ENSO may not be identical to the prediction here. The main quasi-period of ENSO may be correlated to the free oscillation periods but with variations of subharmonic beating and nonlinear coupling. Anyway, the free acceleration of rotation velocity may be clue to discuss quasi-period of ENSO."
and conclusion
> "Nonlinear dynamics makes the Earth rotation breaking a revolution so that updated the whole polar motion science. We must humor the conformance of the advancing history without against the tidal current. Professor E Grafarend [2005] published a course about nonlinearity in geodesy as well as in geophysics. Therefore, Earth rotation theory must be reformed to a new Convention in which the two free wobbles should be stated clearly and the similar accompanied consequences as those of the Chandler wobble. Simultaneously, the whole theory of Earth rotation must face to new challenges of two stable and an unstable component solutions with chaotic final state."
They were arguing that the ENSO period would come about by bifurcating the 14-year period down to the known ENSO range of 2-7 years. While I think the key is that the 14-year period is multiplied against the 2-year biennial period, generating the time-series profile that matches the ENSO data.
Integrating the wave equation transform approach, the biennial modulation isn't 100% and longer term factors are unmodulated to a degree.
Comment Source:Integrating the wave equation transform approach, the biennial modulation isn't 100% and longer term factors are unmodulated to a degree.
Here is a training interval starting in 1953, adding the unmodulated long terms
I went back to an older paper [1] that was discussed here before. In retrospect, these guys were tantalizingly close to figuring out ENSO. They had the biennial factor in their Eq(1) but then didn't seem to pursue it.
[1]K.-Y. Kim, J. J. O’Brien, and A. I. Barcilon, “The principal physical modes of variability over the tropical Pacific,” Earth Interactions, vol. 7, no. 3, pp. 1–32, 2003. PDF
Comment Source:Here is a training interval starting in 1953, adding the unmodulated long terms
I went back to an older paper [1] that was [discussed here before](https://forum.azimuthproject.org/discussion/comment/12639/#Comment_12639). In retrospect, these guys were tantalizingly close to figuring out ENSO. They had the biennial factor in their Eq(1) but then didn't seem to pursue it.
[1]K.-Y. Kim, J. J. O’Brien, and A. I. Barcilon, “The principal physical modes of variability over the tropical Pacific,” Earth Interactions, vol. 7, no. 3, pp. 1–32, 2003.
[PDF](http://journals.ametsoc.org/doi/pdf/10.1175/1087-3562(2003)007%3C0001%3ATPPMOV%3E2.0.CO%3B2)
This Chinese research team predicted a plausible mechanism for the forcing of ENSO.
Wang, Wenjun. “Free acceleration of Earth rotation and energy source of ENSO.” 37th COSPAR Scientific Assembly. Vol. 37. 2008.
“Earth is a triaxial body. Triaxial body rotation has Euler dynamic model of nonlinearity with A¡B¡C. We develop a decomposition theorem for obtaining three true solutions for the rotation with two stable wobbles of Chandler and another decadal of period 14.6 a. The other may be an unstable inverted pendulum of one way sway. The nonlinear coupling of the two free wobbles provide free accelerations for the rotation or LOD with a period of 7 month and another 14.6 a. The fluctuation of period 14.6 a may cause bifurcation cascade in nonlinear coupling and a series of periods 7.3, 3.65 and 1.825 a may be observed with free accelerations and decelerations so that cause ocean water near the equator flow eastward and westward. This makes the energy source of ENSO in eigen periods of 7, 4, 2 a. Therefore in this study we verify that the energy source of ENSO is the free acceleration of Earth rotation. ”
I modeled a 7 month periodic forcing as Wang suggested, and sure enough, it showed up as a sharp factor centered at 0.5791 years = 6.95 months (albeit within the noise of ENSO). This is a 2nd derivative of the time-series, so it exaggerates the higher frequencies: Most of the filtered signal is in the indicated biennial area, which contains the observed ~14.6 year modulated signal.
Tie this into Meehl's ongoing research on the biennial component to ENSO and it can shore up some loose ends in our understanding:
Meehl, Gerald A., and Julie M. Arblaster. "Relating the strength of the tropospheric biennial oscillation (TBO) to the phase of the Interdecadal Pacific Oscillation (IPO)." Geophysical Research Letters 39.20 (2012).
Yet there are strong voices against this kind of approach in climate science. I read through the following paper, and they claim just what the title says -- picking out biennial oscillations in the time series could be just the result of a random pattern in the white noise.
Stuecker, M. F., Timmermann, A., Yoon, J., & Jin, F. F. (2015). Tropospheric Biennial Oscillation (TBO) indistinguishable from white noise. Geophysical Research Letters, 42(18), 7785-7791.
Note that one of the authors is Jin, who has one of the delayed action oscillator models for ENSO
Jin, Fei-Fei. "An equatorial ocean recharge paradigm for ENSO. Part I: Conceptual model." Journal of the Atmospheric Sciences 54.7 (1997): 811-829.
Comment Source:This Chinese research team predicted a plausible mechanism for the forcing of ENSO.
Wang, Wenjun. “Free acceleration of Earth rotation and energy source of ENSO.” 37th COSPAR Scientific Assembly. Vol. 37. 2008.
> “Earth is a triaxial body. Triaxial body rotation has Euler dynamic model of nonlinearity with A¡B¡C. We develop a decomposition theorem for obtaining three true solutions for the rotation with two stable wobbles of Chandler and another decadal of period 14.6 a. The other may be an unstable inverted pendulum of one way sway. The nonlinear coupling of the two free wobbles provide free accelerations for the rotation or LOD with a period of 7 month and another 14.6 a. The fluctuation of period 14.6 a may cause bifurcation cascade in nonlinear coupling and a series of periods 7.3, 3.65 and 1.825 a may be observed with free accelerations and decelerations so that cause ocean water near the equator flow eastward and westward. This makes the energy source of ENSO in eigen periods of 7, 4, 2 a. Therefore in this study we verify that the energy source of ENSO is the free acceleration of Earth rotation. ”
I modeled a 7 month periodic forcing as Wang suggested, and sure enough, it showed up as a sharp factor centered at 0.5791 years = 6.95 months (albeit within the noise of ENSO). This is a 2nd derivative of the time-series, so it exaggerates the higher frequencies: Most of the filtered signal is in the indicated biennial area, which contains the observed ~14.6 year modulated signal.
Tie this into Meehl's ongoing research on the biennial component to ENSO and it can shore up some loose ends in our understanding:
Meehl, Gerald A., and Julie M. Arblaster. "Relating the strength of the tropospheric biennial oscillation (TBO) to the phase of the Interdecadal Pacific Oscillation (IPO)." Geophysical Research Letters 39.20 (2012).
Yet there are strong voices against this kind of approach in climate science. I read through the following paper, and they claim just what the title says -- picking out biennial oscillations in the time series could be just the result of a random pattern in the white noise.
Stuecker, M. F., Timmermann, A., Yoon, J., & Jin, F. F. (2015). Tropospheric Biennial Oscillation (TBO) indistinguishable from white noise. Geophysical Research Letters, 42(18), 7785-7791.
Note that one of the authors is Jin, who has one of the delayed action oscillator models for ENSO
Jin, Fei-Fei. "An equatorial ocean recharge paradigm for ENSO. Part I: Conceptual model." Journal of the Atmospheric Sciences 54.7 (1997): 811-829.
I know everyone likes to see symmetry in nature. Here is the ENSO biennial signal modulated with longer-term periods described in the preceding comments
This is essentially a periodogram of the sidebands, where I transformed the x-axis to reflect the period instead of the frequency.
Note the nice symmetry about a 2.0077 biennial year period, which was chosen to align the discrete 2048 point FFT with the 1 month per sample data series. Keeping track of the 15-year phase reversal starting in 1981 is important, as without that, it would likely not be observable.
As a reminder, the + and - sidebands come about from this modulation:
Comment Source:I know everyone likes to see symmetry in nature. Here is the ENSO biennial signal modulated with longer-term periods described in the preceding comments
This is essentially a periodogram of the sidebands, where I transformed the x-axis to reflect the period instead of the frequency.
Note the nice symmetry about a 2.0077 biennial year period, which was chosen to align the discrete 2048 point FFT with the 1 month per sample data series. Keeping track of the 15-year phase reversal starting in 1981 is important, as without that, it would likely not be observable.
As a reminder, the + and - sidebands come about from this modulation:
$ sin(\pi t) \cdot sin(\omega_m t) = \frac{1}{2} ( cos(\pi t - \omega_m t) - cos(\pi t + \omega_m t) ) $
Here is a very short training interval, from 1940 to 1960, using the 7 bolded frequency factors from the previous chart. Evaluate how well it extrapolates outside of the interval.
Areas highlighted in yellow are where the predicted phase is reversed. This is on the wave equation transformed data, with the ENSO signal inverted from 1981 to 1996.
Comment Source:Here is a very short training interval, from 1940 to 1960, using the 7 bolded frequency factors from the previous chart. Evaluate how well it extrapolates outside of the interval.
Areas highlighted in yellow are where the predicted phase is reversed. This is on the wave equation transformed data, with the ENSO signal inverted from 1981 to 1996.
I'm sure it is documented someplace on Azimuth, but might you post a short description of the machine learning ("ML") algorithm used with these, and possibly a pointer to a longer description?
I have another application involving time series where I might want to try an ML algorithm.
Thank you!
Comment Source:@WebHubTel,
I'm sure it is documented someplace on Azimuth, but might you post a short description of the machine learning ("ML") algorithm used with these, and possibly a pointer to a longer description?
I have another application involving time series where I might want to try an ML algorithm.
Thank you!
HNY Paul and Jan,
My "haven't done list" has hoping I could get Paul and anybody else to add to my cut and pasted abstracts on ML: Deep learning eg. about Eureqa and symbolic regression robustness in general.
Comment Source:HNY Paul and Jan,
My "haven't done list" has hoping I could get Paul and anybody else to add to my cut and pasted abstracts on ML: [[Deep learning]] eg. about Eureqa and symbolic regression robustness in general.
Jan,
The main machine learning algorithm I use is embedded in a tool called Eureqa by Nutonian.
Everything that I know about Eureqa is contained in the User Guide. This includes all the details of how to search for embedded differential equations in the data.
Comment Source:Jan,
The main machine learning algorithm I use is embedded in a tool called [Eureqa](http://www.nutonian.com/products/eureqa/) by Nutonian.
Everything that I know about Eureqa is contained in the [User Guide](http://formulize.nutonian.com/eureqa-user-guide.html). This includes all the details of how to search for embedded differential equations in the data.
BTW, the results from comments such as #141 above are not from any sophisticated machine learning. That is simply multiple linear regression where I "train" the factor set by focusing on an sub-interval, and then check to see how it works outside that interval.
Eureqa does this as well with their symbolic regression algorithm but how they mix the "in-band" and "out-of-bad" intervals is a bit of a mystery to me. They somehow use only the "in-band" interval for training, yet still peek at how well it works on the "out-of-band" data during their exhaustive ML search. It may be that they only allow the error to drop slightly when computing "out-of-band" fit, otherwise they do not continue with the search on the "in-band" interval. That's understandable if they want to prove that the fit is stationary over a sub-interval as well as over a longer interval, otherwise it is cheating according to the strict precepts of no look-ahead for out-of-band validation.
I am just guessing here because the guts of the algorithm are proprietary. They make money off this tool so I imagine that whatever algorithm they use is OK as long as they have satisfied customers.
Comment Source:BTW, the results from comments such as [#141](#Comment_15092) above are not from any sophisticated machine learning. That is simply multiple linear regression where I "train" the factor set by focusing on an sub-interval, and then check to see how it works outside that interval.
Eureqa does this as well with their symbolic regression algorithm but how they mix the "in-band" and "out-of-bad" intervals is a bit of a mystery to me. They somehow use only the "in-band" interval for training, yet still peek at how well it works on the "out-of-band" data during their exhaustive ML search. It may be that they only allow the error to drop slightly when computing "out-of-band" fit, otherwise they do not continue with the search on the "in-band" interval. That's understandable if they want to prove that the fit is stationary over a sub-interval as well as over a longer interval, otherwise it is cheating according to the strict precepts of no look-ahead for out-of-band validation.
I am just guessing here because the guts of the algorithm are proprietary. They make money off this tool so I imagine that whatever algorithm they use is OK as long as they have satisfied customers.
Thanks, Jim. I'm very much interested in developing and coding up ML algorithms for these kinds of series independently of any product. I've seen some remarkable success in my own work from boosting, and hope to adapt it to series. Thanks.
Comment Source:Thanks, Jim. I'm very much interested in developing and coding up ML algorithms for these kinds of series independently of any product. I've seen some remarkable success in my own work from boosting, and hope to adapt it to series. Thanks.
The two examples of machine learning that I have, one for QBO and one for ENSO, show the strengths and weaknesses of the basic approach. The generic machine learning of Eureqa was able to set one on a trail by doing a heavy-lifting search of possibilities, but wan't able to close the book, IMO. It all depends on what rules go in to the machine learning algorithm. With respect to QBO machine learning, Eureqa hinted at the possibility of signal aliasing, but was unable to use that information to further refine a search. For example, it had no idea as to the aliased vs unaliased value matching sets and it actually seemed accidental that the aliased signals were even found. And with ENSO, Eureqa likely could have detected a frequency modulated signal but I was able to spot it from the set of individual factors that Eureqa supplied.
Also I could point out that one could even see the aliasing and modulation from a frequency power spectra, especially the modulation, whereas Eureqa was actually violating the Nyquist criteria when it suggested the higher frequency aliased values, which sets it apart from a Nyquist-limited power spectra.
Like all things AI, the power is dependent on the rules that go in to the inference engine. If the rules don't exist then it won't find the deeper meaning of any result the algorithm supplies. In other words, there is still much interpretation that needs to occur.
Comment Source:The two examples of machine learning that I have, one for QBO and one for ENSO, show the strengths and weaknesses of the basic approach. The generic machine learning of Eureqa was able to set one on a trail by doing a heavy-lifting search of possibilities, but wan't able to close the book, IMO. It all depends on what rules go in to the machine learning algorithm. With respect to QBO machine learning, Eureqa hinted at the possibility of signal aliasing, but was unable to use that information to further refine a search. For example, it had no idea as to the aliased vs unaliased value matching sets and it actually seemed accidental that the aliased signals were even found. And with ENSO, Eureqa likely could have detected a frequency modulated signal but I was able to spot it from the set of individual factors that Eureqa supplied.
Also I could point out that one could even see the aliasing and modulation from a frequency power spectra, especially the modulation, whereas Eureqa was actually violating the Nyquist criteria when it suggested the higher frequency aliased values, which sets it apart from a Nyquist-limited power spectra.
Like all things AI, the power is dependent on the rules that go in to the inference engine. If the rules don't exist then it won't find the deeper meaning of any result the algorithm supplies. In other words, there is still much interpretation that needs to occur.
In fitting to ENSO, I have never looked at data beyond the date September 2013.
I did this intentionally because I wanted to leave some out-of-band data for validation.
The following should be taken with a grain of salt but this is the SOI model extrapolation for years beyond 2013.
It does not prove anything because the accuracy of the model is <80% in getting the sign of the excursion right, but it does nail the current El Ni~no, which some say has just peaked. See the green up-arrow. It is a strong peak in terms of width but doesn't match the deep excursion of the 1998 peak. (Recall that for SOI, negative means higher temperatures)
Comment Source:In fitting to ENSO, I have never looked at data beyond the date September 2013.
I did this intentionally because I wanted to leave some out-of-band data for validation.
The following should be taken with a grain of salt but this is the SOI model extrapolation for years beyond 2013.
It does not prove anything because the accuracy of the model is <80% in getting the sign of the excursion right, but it does nail the current El Ni~no, which [some say has just peaked](http://www.wunderground.com/news/el-nino-noaa-january-2016-update). See the green up-arrow. It is a strong peak in terms of width but doesn't match the deep excursion of the 1998 peak. (Recall that for SOI, negative means higher temperatures)
So, if I were digging into this, I'd like to know how the "accuracy of the model" is assessed. In particular I wonder if that's Mean Integrated Square Error ("MISE") or something else. It's not at all clear that MISE is the appropriate measure for a non-stationary series and, especially, for a series which has chaotic contributions. The score, in the latter two instances, should penalize for the result not being smooth, since it is the estimator, at any point, of all possible futures. Chasing a particular future, since it is unattainable, is not a useful goal.
Comment Source:So, if I were digging into this, I'd like to know how the "accuracy of the model" is assessed. In particular I wonder if that's Mean Integrated Square Error ("MISE") or something else. It's not at all clear that MISE is the appropriate measure for a non-stationary series and, especially, for a series which has chaotic contributions. The score, in the latter two instances, should penalize for the result not being smooth, since it is the estimator, at any point, of all possible futures. Chasing a particular future, since it is unattainable, is not a useful goal.
So would I. I've been looking at the ENSO and QBO in tandem, and since I discovered that the QBO is stationary and predictable according to the lunisolar tidal parameters, my working assumption is that ENSO may be as well.
Yet, ENSO is a tougher nut to crack, largely because the sub-yearly behavior is so sensitive to localized weather conditions, such as cyclonic activity. In comparison, QBO does show stationary patterns in the fine sub-yearly structure -- I can rationalize this as the QBO data is taken from the stratosphere, which is removed from the lower troposphere instabilities.
So QBO is largely stationary at both sub-yearly and longer time-scales, while ENSO likely only on time scales greater than a year.
Comment Source:So would I. I've been looking at the ENSO and QBO in tandem, and since I discovered that the QBO is stationary and predictable according to the lunisolar tidal parameters, my working assumption is that ENSO may be as well.
Yet, ENSO is a tougher nut to crack, largely because the sub-yearly behavior is so sensitive to localized weather conditions, such as cyclonic activity. In comparison, QBO does show stationary patterns in the fine sub-yearly structure -- I can rationalize this as the QBO data is taken from the stratosphere, which is removed from the lower troposphere instabilities.
So QBO is largely stationary at both sub-yearly and longer time-scales, while ENSO likely only on time scales greater than a year.
Comments
Jim, Thanks. I am always looking for substantiating citations.
Jim, Thanks. I am always looking for substantiating citations.I think this is classic boundary-value limited behavior. To stretch a testing regime that includes an interval of 100 years to another 200 years back in time requires that the forcing periodicity has enough precision to accommodate phase drift. Otherwise the agreement between model and data gradually gets out of phase.
Earlier, I was using 2.33 years for the QBO period, and now have settled on 2.36 years. There are more than 100 QBO cycles over 300 years so that a difference of 0.03 years in period is enough to cause a shift of at least one average QBO cycle. That may be enough to cause the model to lose coherence with the data over the entire interval, as the QBO is providing the main boundary-value forcing to keep it in phase.
TODO: Next up I will look at Jim's list of past El Nino event dates and see if they match with the data and model.
I think this is classic boundary-value limited behavior. To stretch a testing regime that includes an interval of 100 years to another 200 years back in time requires that the forcing periodicity has enough precision to accommodate phase drift. Otherwise the agreement between model and data gradually gets out of phase. Earlier, I was using 2.33 years for the QBO period, and now have settled on 2.36 years. There are more than 100 QBO cycles over 300 years so that a difference of 0.03 years in period is enough to cause a shift of at least one average QBO cycle. That may be enough to cause the model to lose coherence with the data over the entire interval, as the QBO is providing the main boundary-value forcing to keep it in phase. TODO: Next up I will look at Jim's list of past El Nino event dates and see if they match with the data and model.In comment #97, I made the association that the Chandler Wobble period is half the QBO period.
This white paper by Sidorenkov makes the same association.
This is an excerpt from the paper
Note that the common spectral peak found (2.37) matches the 2.366 year period I used in #97 for producing the ENSO model. There are also side-lobes that are evidence of the jitter observed on both QBO and CW. Sidorenkov claims that those lobes track with the same 2-to-1 ratio.
In [comment #97](http://forum.azimuthproject.org/discussion/comment/14618/#Comment_14618), I made the association that the Chandler Wobble period is half the QBO period. [This white paper by Sidorenkov](http://syrte.obspm.fr/jsr/journees2014/pdf/Sidorenkov.pdf) makes the same association. This is an excerpt from the paper  Note that the common spectral peak found (2.37) matches the 2.366 year period I used in #97 for producing the ENSO model. There are also side-lobes that are evidence of the jitter observed on both QBO and CW. Sidorenkov claims that those lobes track with the same 2-to-1 ratio.I'm completely saturated with other work, but Paul's stuff looks interesting. If someone wants to write a blog article about it for Azimuth, just follow the usual recipe: put it on the wiki following the directions in How to blog, get people to criticize it and fix mistakes, etc. I'll join in.
I'm completely saturated with other work, but Paul's stuff looks interesting. If someone wants to write a blog article about it for Azimuth, just follow the usual recipe: put it on the wiki following the directions in [How to blog](http://www.azimuthproject.org/azimuth/show/How+to#blog), get people to criticize it and fix mistakes, etc. I'll join in.John, I can take a crack on getting a start. Of course none of this is published yet, which is why I am using the forum instead of the wiki to get a discussion going.
Couple of other citations supporting comment #103 above:
Furuya, Masato, Yozo Hamano, and Isao Naito. "Quasi‐periodic wind signal as a possible excitation of Chandler wobble." Journal of Geophysical Research: Solid Earth (1978–2012) 101.B11 (1996): 25537-25546.
Ponte, Rui M., and Detlef Stammer. "Role of ocean currents and bottom pressure variability on seasonal polar motion." Journal of Geophysical Research: Oceans (1978–2012) 104.C10 (1999): 23393-23409. pdf
John, I can take a crack on getting a start. Of course none of this is published yet, which is why I am using the forum instead of the wiki to get a discussion going. --- Couple of other citations supporting comment #103 above: Furuya, Masato, Yozo Hamano, and Isao Naito. "Quasi‐periodic wind signal as a possible excitation of Chandler wobble." Journal of Geophysical Research: Solid Earth (1978–2012) 101.B11 (1996): 25537-25546. > "In a narrow-band analysis of the excitation using least squares fit sinusoids, we found that both the amplitude and phase of the inferred excitation near the Chandler band indicate their strong dependence on the assumed Chandler period and that an assumed CW period of 431 days caused atmospheric and observed excitations to agree most closely." Ponte, Rui M., and Detlef Stammer. "Role of ocean currents and bottom pressure variability on seasonal polar motion." Journal of Geophysical Research: Oceans (1978–2012) 104.C10 (1999): 23393-23409. [pdf](http://onlinelibrary.wiley.com/doi/10.1029/1999JC900222/pdf)This is OT re. ENSO but it's an attempt at a wavelet solution to something not entirely dissimilar, and you never know, might be connected via the polar bridge:
Oleg Pokrovsky, Impact of Atlantic and Pacific decadal oscillation on ice extent in Russian Arctic seas (2010) (presentation slides).
This is OT re. ENSO but it's an attempt at a wavelet solution to something not entirely dissimilar, and you never know, might be connected via the polar bridge: Oleg Pokrovsky, [Impact of Atlantic and Pacific decadal oscillation on ice extent in Russian Arctic seas (2010)](http://www.arcticfrontiers.com/downloads/arctic-frontiers-2010/conference-presentations-7/wednesday-27-january-2010/part-i-ice-and-climate-including-paleo-climate/1017-07-pokrovsky/file) (presentation slides).The SOI metric for ENSO is really leaning negative.
Once it hits -20 for a sustained number of months, we are back to 1998 conditions.
Look how much it dives over the course in the few recent days of May.
All of my model fits so far have shown this negative excursion starting around 2015, but that doesn't really confirm much.
The SOI metric for ENSO is really leaning negative. Once it hits -20 for a sustained number of months, we are back to 1998 conditions. Look how much it dives over the course in the few recent days of May.  All of my model fits so far have shown this negative excursion starting around 2015, but that doesn't really confirm much.I scrolled back up this thread to #68 where I can see a big drop in 2014 (corrected). Did you post results up to 2015 somewhere?
I scrolled back up this thread to #68 where I can see a big drop in 2014 (corrected). Did you post results up to 2015 somewhere?Jim, I may have. I just don't think a prediction is a game changer. The sign-metric I am using only fits the correct sign of an excursion at an 80% level at most. So that once every 5 years it will have the wrong sign.
Jim, I may have. I just don't think a prediction is a game changer. The sign-metric I am using only fits the correct sign of an excursion at an 80% level at most. So that once every 5 years it will have the wrong sign.Another look at recent ENSO activity. When the SOI reaches that -20 mark and stays, that is nearing 1998 territory for significant ocean sloshing amplitude:
Another look at recent ENSO activity. When the SOI reaches that -20 mark and stays, that is nearing 1998 territory for significant ocean sloshing amplitude: There are a large number of methods and programming pages you and Dara have covered which I think should be written up on the wiki.
For example I have no idea that you were using a sign metric in:
I'd find the write-up of these details fascinating :).
I'm hoping to take up a suggestion of Dave Tanzer's to work on Azimuth's educational goals; which I think of as needing some kind of applied green maths and practical sustainability engineering curriculum.
There are a large number of methods and programming pages you and Dara have covered which I think should be written up on the wiki. For example I have no idea that you were using a sign metric in: > The sign-metric I am using only fits the correct sign of an excursion at an 80% level at most. I'd find the write-up of these details fascinating :). I'm hoping to take up a suggestion of Dave Tanzer's to work on Azimuth's educational goals; which I think of as needing some kind of applied green maths and practical sustainability engineering curriculum.Jim, I don't know why I am not using the Wiki. It's not that I am not accustomed to it, as I started using a wiki at work when they first came out years ago.
I guess my issues are several-fold:
When I sense that there is some support and someone can start duplicating and perhaps extending the work, that would be a good time to consider structuring some sort of educational module.
BTW, the sign-metric is simply counting the fraction of the time that the model and data shows the same sign-excursion. If it always agrees it is 1.0 and if it the excursions are random then 0.5.
Jim, I don't know why I am not using the Wiki. It's not that I am not accustomed to it, as I started using a wiki at work when they first came out years ago. I guess my issues are several-fold: * I would rather do work using a semantic wiki or feed information into a semantic web format (which is my context modeling project) * Don't want to give the impression that these results are iron-clad, which is what a wiki has grown to signify When I sense that there is some support and someone can start duplicating and perhaps extending the work, that would be a good time to consider structuring some sort of educational module. BTW, the sign-metric is simply counting the fraction of the time that the model and data shows the same sign-excursion. If it always agrees it is 1.0 and if it the excursions are random then 0.5.Sure thing. The Azimuth wiki can just link to any explanations on your blog.
Sure thing. The Azimuth wiki can just link to any explanations on your blog.A blog and a forum work great as a scratchpad for thinking out loud. I used to run a blog called http://mobjectivist.blogspot.com, which was mainly about peak oil. At some point I decided to cull from the interesting posts and write a book on the topic "The Oil ConunDrum". Then I stopped writing to that blog, like you stop writing to a filled-up lab notebook. As it turns out, the blog contained all sorts of charts and graphs hosted by an imaging service and those started to disappear. Glad that I put something together for posterity, as the blog is filled with missing links now !
I guess my point is that one does what can to maintain and organize their thoughts, and it is a judgement call to anticipate what you will need to maintain in the future.
In thinking out loud about ENSO, one of the false starts may be about the impact of TSI variations. In the ongoing discussion, I started to believe that it was an important driver but have now started to downplay its relevance. On a Wiki making these kinds of linear narrative changes is awkward -- do you just delete the false start, or do you explain why the change was made? Lots of space can be expended on what is not the correct approach. I am waiting for the point when I can pull the trigger and be confident that I have the right mechanisms in place to lay it all out.
A blog and a forum work great as a scratchpad for thinking out loud. I used to run a blog called http://mobjectivist.blogspot.com, which was mainly about peak oil. At some point I decided to cull from the interesting posts and write a book on the topic "The Oil ConunDrum". Then I stopped writing to that blog, like you stop writing to a filled-up lab notebook. As it turns out, the blog contained all sorts of charts and graphs hosted by an imaging service and those started to disappear. Glad that I put something together for posterity, as the blog is filled with missing links now ! I guess my point is that one does what can to maintain and organize their thoughts, and it is a judgement call to anticipate what you will need to maintain in the future. In thinking out loud about ENSO, one of the false starts may be about the impact of TSI variations. In the ongoing discussion, I started to believe that it was an important driver but have now started to downplay its relevance. On a Wiki making these kinds of linear narrative changes is awkward -- do you just delete the false start, or do you explain why the change was made? Lots of space can be expended on what is not the correct approach. I am waiting for the point when I can pull the trigger and be confident that I have the right mechanisms in place to lay it all out.I have a new blog post called The Hidden Harmony of ENSO here -- http://ContextEarth.com/2015/05/31/the-hidden-harmony-of-enso
The point I am trying to get across is that a regular periodic function can create the erratic behavior that results in ENSO.
So when we start with this:
it results in this
Here is the beginning of an elevator pitch : The periodic harmony behind the equatorial Pacific dynamics are obscured by the non-linear sloshing transformation of the ocean's volume.
My premise has always been that since days, seasons, and the tides are so periodic why do the ocean's necessarily have to be chaotic? Just because they appear chaotic doesn't mean that they are. This is the basic hydrodynamic equation that justifies a predictable deterministic outcome.
I have a new blog post called The Hidden Harmony of ENSO here -- http://ContextEarth.com/2015/05/31/the-hidden-harmony-of-enso The point I am trying to get across is that a regular periodic function can create the erratic behavior that results in ENSO. So when we start with this:  it results in this  Here is the beginning of an elevator pitch : The periodic harmony behind the equatorial Pacific dynamics are obscured by the non-linear sloshing transformation of the ocean's volume. My premise has always been that since days, seasons, and the tides are so periodic why do the ocean's necessarily have to be chaotic? Just because they *appear* chaotic doesn't mean that they are. This is the basic hydrodynamic equation that justifies a predictable deterministic outcome.WebHubTel,
We have an experiments category which would suit your work.
WebHubTel, We have an [experiments category](http://www.azimuthproject.org/azimuth/list/experiments) which would suit your work.Thanks Graham, I see how that is accomplished via the experiments category. The table of contents shows the linear narrative of the experimental approaches taken. It makes it more like a mini-blog within a wiki.
I am thinking that with my previous comment #115, it could capture successive refinements of the most concise and plausible ENSO model fits. The table of contents would track the evolution effectively.
Thanks Graham, I see how that is accomplished via the experiments category. The table of contents shows the linear narrative of the experimental approaches taken. It makes it more like a mini-blog within a wiki. I am thinking that with my previous comment #115, it could capture successive refinements of the most concise and plausible ENSO model fits. The table of contents would track the evolution effectively.WebHubTel: just to second what Graham is saying, I think you should put your work on one or more "Experiments" pages in the wiki. It's easier to organize things there: for example, you can have a page that's organized in a chronological narrative and also have a section or page that describes the "current state of the art", etc. The Forum here is designed for conversations, and it seems to work pretty well for that, but if I want to understand what you've done it can be a bit difficult to extract it from a collection of conversations.
WebHubTel: just to second what Graham is saying, I think you should put your work on one or more "Experiments" pages in the wiki. It's easier to organize things there: for example, you can have a page that's organized in a chronological narrative _and also_ have a section or page that describes the "current state of the art", etc. The Forum here is designed for conversations, and it seems to work pretty well for that, but if I want to understand what you've done it can be a bit difficult to extract it from a collection of conversations.John, OK I will start using an Experiments page and see how that works.
Here is a start azimuth/show/Experiments+in+ENSO+modeling
John, OK I will start using an Experiments page and see how that works. Here is a start [azimuth/show/Experiments+in+ENSO+modeling](http://www.azimuthproject.org/azimuth/show/Experiments+in+ENSO+modeling)I have the first wiki "draft" in place for the sloshing ENSO model here: azimuth/show/Experiments+in+ENSO+modeling
Of course this model is totally at odds with the Zebiac-Cane model, which is also described in the wiki. What the wiki says is that the Zebiac-Cane model shows "somewhat realistic ENSO behavior", yet I have never actually seen a fit of the Z-C model to any extended ENSO time-series data.
This is where it gets tricky in modifying the Wiki narrative to include the sloshing model alongside the Z-C model and the others described there. Do we list it alongside the others as a candidate of equal standing? I am not going to make that call until I get some buy-in from other wiki contributors.
I have the first wiki "draft" in place for the sloshing ENSO model here: [azimuth/show/Experiments+in+ENSO+modeling](http://www.azimuthproject.org/azimuth/show/Experiments+in+ENSO+modeling) Of course this model is totally at odds with the Zebiac-Cane model, which is also described in the wiki. What the wiki says is that the Zebiac-Cane model shows *"somewhat realistic ENSO behavior"*, yet I have never actually seen a fit of the Z-C model to any extended ENSO time-series data. This is where it gets tricky in modifying the Wiki narrative to include the sloshing model alongside the Z-C model and the others described there. Do we list it alongside the others as a candidate of equal standing? I am not going to make that call until I get some buy-in from other wiki contributors.There's an initial wiki page on the ZC model you can link to and perhaps add to and improve there and review from your experiments page?
There's an initial wiki page on the ZC model you can link to and perhaps add to and improve there and review from your experiments page?Jim, I placed a bi-directional link between the ENSO wiki page and this experimental modeling page via /show/ENSO#azimuth_experiments.
Also am not sure what to add about the ZC model other than what I said above, and that must be vetted by a second opinion. The most I was able to find was that the ZB model "produced plausible simulations of ENSO". Yet there are no charts demonstrating the plausibility.
Jim, I placed a bi-directional link between the ENSO wiki page and this experimental modeling page via [/show/ENSO#azimuth_experiments](http://www.azimuthproject.org/azimuth/show/ENSO#azimuth_experiments). Also am not sure what to add about the ZC model other than what I said above, and that must be vetted by a second opinion. The most I was able to find was that the ZB model ["produced plausible simulations of ENSO"](http://www.asp.ucar.edu/colloquium/2000/Lectures/battisti1.html). Yet there are no charts demonstrating the plausibility.+1
+1I am starting to look more at the distinctions between my sloshing formulation and that of the parametric recharge oscillator, which is one of the favorite models in the literature [1]. Look at Eq 8 and 9 in Stein et al, which leads to theor Eq 27.
$ T''(t) + \lambda(1-T^2)T'(t) + T(t) = F(t) $
This is the basic wave equation, with a first derivative drag term that has a non-linear behavior, decreasing the drag for larger excursions of T. I ignore this drag term and assume the behavior is largely inviscid and that any drag effect is minor.
With the forcing term F(t) consisting of QBO and Chandler wobble, most of the ENSO behavior is simply modeled. Add in the Mathieu/Hill modulation and the details start to emerge, which is what I have added to the Azimuth Wiki page.
The prevailing question is why aren't the climate scientists working the angle of using known forcing factors in the formulations that they originally devised? It's like an electrical engineer not using Kirchoff's and Ohm's laws in evaluating the response of a resonant circuit. Instead, they bypass the simple path and immediately assume that the road must be via chaotic bifurcations and the messiness that approach entails.
It's enough to drive one up the wall :) And that is not the path of least resistance :) :)
[1] K. Stein, A. Timmermann, N. Schneider, F.-F. Jin, and M. F. Stuecker, “ENSO seasonal synchronization theory,” Journal of Climate, vol. 27, no. 14, pp. 5285–5310, 2014. PDF
Take a look at these two papers that just came out as well. This is the Jin-Neelin-Ghil model. which is an extension of [1]. I have the PDFs available.
[2] D. Mukhin, E. Loskutov, A. Mukhina, A. Feigin, I. Zaliapin, and M. Ghil, “Predicting critical transitions in ENSO models. Part I: Methodology and simple models with memory,” Journal of Climate, vol. 28, no. 5, pp. 1940–1961, 2015.
[3] D. Mukhin, D. Kondrashov, E. Loskutov, A. Gavrilov, A. Feigin, and M. Ghil, “Predicting critical transitions in ENSO models. Part II: Spatially dependent models,” Journal of Climate, vol. 28, no. 5, pp. 1962–1976, 2015.
I am starting to look more at the distinctions between my sloshing formulation and that of the parametric recharge oscillator, which is one of the favorite models in the literature [1]. Look at Eq 8 and 9 in Stein et al, which leads to theor Eq 27. $ T''(t) + \lambda(1-T^2)T'(t) + T(t) = F(t) $ This is the basic wave equation, with a first derivative drag term that has a non-linear behavior, decreasing the drag for larger excursions of T. I ignore this drag term and assume the behavior is largely inviscid and that any drag effect is minor. With the forcing term F(t) consisting of QBO and Chandler wobble, most of the ENSO behavior is simply modeled. Add in the Mathieu/Hill modulation and the details start to emerge, which is what I have added to the Azimuth Wiki page. The prevailing question is why aren't the climate scientists working the angle of using known forcing factors in the formulations that they originally devised? It's like an electrical engineer not using Kirchoff's and Ohm's laws in evaluating the response of a resonant circuit. Instead, they bypass the simple path and immediately assume that the road must be via chaotic bifurcations and the messiness that approach entails. It's enough to drive one up the wall :) And that is not the path of least resistance :) :) [1] K. Stein, A. Timmermann, N. Schneider, F.-F. Jin, and M. F. Stuecker, “ENSO seasonal synchronization theory,” Journal of Climate, vol. 27, no. 14, pp. 5285–5310, 2014. [PDF](http://www.researchgate.net/profile/Malte_Stuecker/publication/267098864_ENSO_seasonal_synchronization_theory/links/5445d08a0cf2f14fb80f05e0.pdf) Take a look at these two papers that just came out as well. This is the Jin-Neelin-Ghil model. which is an extension of [1]. I have the PDFs available. [2] D. Mukhin, E. Loskutov, A. Mukhina, A. Feigin, I. Zaliapin, and M. Ghil, “Predicting critical transitions in ENSO models. Part I: Methodology and simple models with memory,” Journal of Climate, vol. 28, no. 5, pp. 1940–1961, 2015. [3] D. Mukhin, D. Kondrashov, E. Loskutov, A. Gavrilov, A. Feigin, and M. Ghil, “Predicting critical transitions in ENSO models. Part II: Spatially dependent models,” Journal of Climate, vol. 28, no. 5, pp. 1962–1976, 2015.This looks like a breakthrough paper on ENSO [1]. The 2nd author Abarca-del-Rio has done a lot of research on understanding the earth's angular momentum and he has helped me in the past with analysis.
As I read through the paper, it appears that it is right up the alley for the mathematical physicists on Azimuth.
The rather strong assertion that they make is
And the eye-opener to me is the difficulty they have with the El Nino event starting in 1981. That is precisely the point in time that has been most problematic in the deterministic model of ENSO that I have been working on (see the first comment in this long thread). You can see it in the figure below which is reproduced from their paper. It appears as if the event is delayed according to the model but the data forces it to loop back.
I had suggested this anomaly has something to do with a discrete Pacific Ocean shift, but it may also be related to a peak in the strength of TSI activity, which is shown in the wiki page.
[1] H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015. PDF
This looks like a breakthrough paper on ENSO [1]. The 2nd author Abarca-del-Rio has done a lot of research on understanding the earth's angular momentum and he has helped me in the past with analysis. As I read through the paper, it appears that it is right up the alley for the mathematical physicists on Azimuth. The rather strong assertion that they make is > "The most important result of this study is that the so-called SOI anomaly corresponds to the dynamics of a nolinear oscillator having complex regularities and exhibits an acceptable level of accuracy of average non-linear predictability in the range between 2 and 4 years of time span." And the eye-opener to me is the difficulty they have with the El Nino event starting in 1981. That is precisely the point in time that has been most problematic in the deterministic model of ENSO that I have been working on (see the [first comment](/discussion/1608/enso-revisit/p1) in this long thread). You can see it in the figure below which is reproduced from their paper. It appears as if the event is delayed according to the model but the data forces it to loop back.  I had suggested this anomaly has something to do with a discrete Pacific Ocean shift, but it may also be related to a peak in the strength of TSI activity, which [is shown in the wiki page](http://www.azimuthproject.org/azimuth/show/Experiments+in+ENSO+modeling). [1] H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015. [PDF](http://arxiv.org/pdf/1506.04066.pdf)Another potentially breakthrough paper
The 11-year solar cycle in current reanalyses: a (non)linear attribution study of the middle atmosphere
http://www.atmos-chem-phys.net/15/6879/2015/acp-15-6879-2015.html
Another potentially breakthrough paper The 11-year solar cycle in current reanalyses: a (non)linear attribution study of the middle atmosphere http://www.atmos-chem-phys.net/15/6879/2015/acp-15-6879-2015.htmlThis paper [1] H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015. [pdf](http://arxiv.org/pdf/1506.04066.pdfO
has me thinking about the importance of a Pacific Ocean event occurring around 1980 impacting the ENSO dynamics. The model fit I am using up to 1980 is excellent, which I will explain in a later comment. But after 1980 it looks to effect a phase reversal --
This is a plot of the LHS vs the RHS of the DiffEq
f''(t)+kf(t) versus F(t)
where F(t) is the forcing function. Note that right near 1980, the values hit a degenerate inflection point coinciding with a value of zero. What I think this means is that a slight push in any direction could cause the motion to reverse sign. So if some forcing noise (such as volcanic eruptions or earthquakes, see El Chichon 1982) causes f(t) to jump above or below zero, that is the direction that the signal may head, regardless on whether the f(t) region is convex or concave.
So this may be a metastable behavior that we are detecting. I have to find some examples of this in the literature to see if this is a plausible explanation.
Also, get a load of the model fit before 1980. This is a combination of QBO for the fine detail and Chandler wobble, TSI and long-term tidal beat periods for the decadal variability. Once the fit gets above a 0.9 correlation coefficient, it really starts to lock in to place.
This paper [1] H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015. [pdf](http://arxiv.org/pdf/1506.04066.pdfO has me thinking about the importance of a Pacific Ocean event occurring around 1980 impacting the ENSO dynamics. The model fit I am using up to 1980 is excellent, which I will explain in a later comment. But after 1980 it looks to effect a phase reversal --  This is a plot of the LHS vs the RHS of the DiffEq f''(t)+kf(t) versus F(t) where F(t) is the forcing function. Note that right near 1980, the values hit a degenerate inflection point coinciding with a value of zero. What I think this means is that a slight push in any direction could cause the motion to reverse sign. So if some forcing noise (such as volcanic eruptions or earthquakes, see El Chichon 1982) causes f(t) to jump above or below zero, that is the direction that the signal may head, regardless on whether the f(t) region is convex or concave. So this may be a metastable behavior that we are detecting. I have to find some examples of this in the literature to see if this is a plausible explanation. Also, get a load of the model fit before 1980. This is a combination of QBO for the fine detail and Chandler wobble, TSI and long-term tidal beat periods for the decadal variability. Once the fit gets above a 0.9 correlation coefficient, it really starts to lock in to place.Anyone getting excited yet about the ENSO model ? :)
Again this is a plot of the LHS vs the RHS of the SOI differential wave equation
SOI''(t)+k SOI(t) versus F(t)
where F(t) is the forcing function.
I added some forcing changes at 1980 to get the full 1880 to 2013 fit
I also pushed the characteristic freq back to a period of 4.25 years, and the QBO emerges as a very strong factor.
The bottom panel is the power spectra of the residual noise (see middle panel "diff"). It's mostly flat, with the rise to the right corresponding to jitter about the main QBO period of 2.33 years.
IMO this is definitely not an overfit, but just a different way of looking at the model described in the ARXIV ENSO sloshing paper. What one can do is extend the forcing and then do an integration of the LHS to project the ENSO waveform into the future.
Where are the pro climate scientists on this? This is so fundamentally basic that it boggles the mind.
Anyone getting excited yet about the ENSO model ? :) Again this is a plot of the LHS vs the RHS of the SOI differential wave equation SOI''(t)+k SOI(t) versus F(t) where F(t) is the forcing function. I added some forcing changes at 1980 to get the full 1880 to 2013 fit  I also pushed the characteristic freq back to a period of 4.25 years, and the QBO emerges as a very strong factor. The bottom panel is the power spectra of the residual noise (see middle panel "diff"). It's mostly flat, with the rise to the right corresponding to jitter about the main QBO period of 2.33 years. IMO this is definitely not an overfit, but just a different way of looking at the model described in the ARXIV ENSO sloshing paper. What one can do is extend the forcing and then do an integration of the LHS to project the ENSO waveform into the future. Where are the pro climate scientists on this? This is so fundamentally basic that it boggles the mind.Comparison of a wavelet scalogram of transformed SOI data against the model
Horizontal time scale is in months from 1880.
blog post here: http://contextearth.com/2015/07/11/enso-transformation/
Comparison of a wavelet scalogram of transformed SOI data against the model  Horizontal time scale is in months from 1880. blog post here: [http://contextearth.com/2015/07/11/enso-transformation/](http://contextearth.com/2015/07/11/enso-transformation/)Experimenting with the ENSO model and the transition that I detect and Astudillo also observe (see comment #125), it may be that a phase reversal is happening. What I did was fit a model between 1880 and 1980 with a correlation coefficient approaching 0.8. I then extended the model as an out-of-band forecast beyond 1880 and noticed that the model appeared sign reversed with respect to the transformed data,
I reversed only the forcing sign and the correlation coefficient maintained a value close to 0.8. See the curve below. I am using essentially the same forcing parameters of QBO, Chandler wobble, TSI, and tidal periods as in the previous comments but calculating using a spreadsheet instead of Mathematica.
I mentioned this before (comment #127), but if a second-order DiffEq shows an inflection point at amplitudes close to zero, the direction that the curve will take may be very sensitive to a forcing disturbance. Just like in a cavity, the standing wave's initial direction behaves like a metastable point. This could explain much as the disturbance may be very subtle and therefore a highly plausible effect. For example, It could be a peak in the TSI right around 1980.
Experimenting with the ENSO model and the transition that I detect and Astudillo also observe (see comment #125), it may be that a phase reversal is happening. What I did was fit a model between 1880 and 1980 with a correlation coefficient approaching 0.8. I then extended the model as an out-of-band forecast beyond 1880 and noticed that the model appeared sign reversed with respect to the transformed data,  I reversed only the forcing sign and the correlation coefficient maintained a value close to 0.8. See the curve below. I am using essentially the same forcing parameters of QBO, Chandler wobble, TSI, and tidal periods as in the previous comments but calculating using a spreadsheet instead of Mathematica.  I mentioned this before (comment #127), but if a second-order DiffEq shows an inflection point at amplitudes close to zero, the direction that the curve will take may be very sensitive to a forcing disturbance. Just like in a cavity, the standing wave's initial direction behaves like a metastable point. This could explain much as the disturbance may be very subtle and therefore a highly plausible effect. For example, It could be a peak in the TSI right around 1980.This thread has been dormant for a while because the activity has been on the QBO recently.
https://forum.azimuthproject.org/discussion/1640/predictability-of-the-quasi-biennial-oscillation#latest
The striking commonality of the forcing factors between the QBO and ENSO models is the most critical piece in solidifying the overall concept. In other words, putting effort into either model improves both, at least that is what I am seeing.
This thread has been dormant for a while because the activity has been on the QBO recently. https://forum.azimuthproject.org/discussion/1640/predictability-of-the-quasi-biennial-oscillation#latest The striking commonality of the forcing factors between the QBO and ENSO models is the most critical piece in solidifying the overall concept. In other words, putting effort into either model improves both, at least that is what I am seeing.This is cool. I regressed (i.e. trained) the ENSO model on the UEP coral proxy data from 1880 to 1977 and then used that solution to back extrapolate to 1650.
The correlation coefficient over the 100-year training interval is 0.74 and over the entire 300+ year region it is 0.54. The waveform looks jagged because there is only one data point per year.
This uses the same parameters as the ENSO SOI model and the QBO model.
The multiple regression takes less than a second to compute.
This is cool. I regressed (i.e. trained) the ENSO model on the UEP coral proxy data from 1880 to 1977 and then used that solution to back extrapolate to 1650.  The correlation coefficient over the 100-year training interval is 0.74 and over the entire 300+ year region it is 0.54. The waveform looks jagged because there is only one data point per year. This uses the same parameters as the ENSO SOI model and the QBO model. The multiple regression takes less than a second to compute.http://ContextEarth.com/2015/10/03/raising-the-bar-on-enso-model-validation
Over the validation interval, the sign of the excursion matches 143 out of 230 times (i.e. for each year from 1650 to 1880). Using the binomial theorem with P=0.5, this number of matches (or greater) should occur with probability 0.00008 strictly due to chance.
http://stattrek.com/online-calculator/binomial.aspx
On the training side (which doesn't count), the sign of the excursion matches 70 out of 97 times (each year from 1880 to 1977). This number of matches could occur by chance with probability 0.000007. Even though the number is less than 0.00008, it is meaningless since the training side is prone to overfitting .
However, on the validation side, the probability is highly significant because it is an out-of-band blind test. The pattern in the underlying oscillations is very likely deterministic as also shown by Astudillo, but now extended over 3 centuries plus some.
[1]H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015.
http://ContextEarth.com/2015/10/03/raising-the-bar-on-enso-model-validation  Over the validation interval, the sign of the excursion matches 143 out of 230 times (i.e. for each year from 1650 to 1880). Using the binomial theorem with P=0.5, this number of matches (or greater) should occur with probability 0.00008 strictly due to chance. http://stattrek.com/online-calculator/binomial.aspx On the training side (which doesn't count), the sign of the excursion matches 70 out of 97 times (each year from 1880 to 1977). This number of matches could occur by chance with probability 0.000007. Even though the number is less than 0.00008, it is meaningless since the training side is prone to overfitting . However, on the validation side, the probability is highly significant because it is an out-of-band blind test. The pattern in the underlying oscillations is very likely deterministic as also shown by Astudillo, but now extended over 3 centuries plus some. [1]H. Astudillo, R. Abarca-del-Rio, and F. Borotto, “Long-term non-linear predictability of ENSO events over the 20th century,” arXiv preprint arXiv:1506.04066, 2015.Wild training interval for only 5 coefficients over a 15 year span
ENSO shows very stationary time-series traits if the data is transformed via the wave equation. http://contextearth.com/2015/07/11/enso-transformation/
Wild training interval for only 5 coefficients over a 15 year span  ENSO shows very stationary time-series traits if the data is transformed via the wave equation. http://contextearth.com/2015/07/11/enso-transformation/I figured out an intriguing ENSO pattern, based on precisely defining sidebands +/- on each side of the exact biennial [1] period. The sidebands appear to follow the period of the three triaxial wobbles in the earth's angular momentum [2]. It is able to extrapolate most of the wave-equation transformed curve by fitting to a short interval. This is simply a consequence of the interval containing enough information to reconstruct the rest of the stationary time series.
Its not a lot different from what the sloshing formulation I started with, but the symmetry and canonical form is now much more readily apparent. The three sideband periods are 6.5 year, 14.3 year, and 18.6 year, which you can understand from reading the fractured English in reference [2].
[1] S.-R. Yeo and K.-Y. Kim, “Global warming, low-frequency variability, and biennial oscillation: an attempt to understand the physical mechanisms driving major ENSO events,” Climate Dynamics, vol. 43, no. 3–4, pp. 771–786, 2014. http://link.springer.com/article/10.1007/s00382-013-1862-1/fulltext.html
[2] Wang, Wen-Jun, W.-B. Shen, and H.-W. Zhang, “Verifications for Multiple Solutions of Triaxial Earth Rotation,” IERS Workshop on Conventions Bureau International des Poids et Mesures (BIPM), Sep. 2007. http://www1.bipm.org/utils/en/events/iers/Wang.pdf
I figured out an intriguing ENSO pattern, based on precisely defining sidebands +/- on each side of the exact biennial [1] period. The sidebands appear to follow the period of the three triaxial wobbles in the earth's angular momentum [2]. It is able to extrapolate most of the [wave-equation transformed curve](http://contextearth.com/2015/07/11/enso-transformation/) by fitting to a short interval. This is simply a consequence of the interval containing enough information to reconstruct the rest of the stationary time series.     Its not a lot different from what the sloshing formulation I started with, but the symmetry and canonical form is now much more readily apparent. The three sideband periods are 6.5 year, 14.3 year, and 18.6 year, which you can understand from reading the fractured English in reference [2]. [1] S.-R. Yeo and K.-Y. Kim, “Global warming, low-frequency variability, and biennial oscillation: an attempt to understand the physical mechanisms driving major ENSO events,” Climate Dynamics, vol. 43, no. 3–4, pp. 771–786, 2014. http://link.springer.com/article/10.1007/s00382-013-1862-1/fulltext.html [2] Wang, Wen-Jun, W.-B. Shen, and H.-W. Zhang, “Verifications for Multiple Solutions of Triaxial Earth Rotation,” IERS Workshop on Conventions Bureau International des Poids et Mesures (BIPM), Sep. 2007. http://www1.bipm.org/utils/en/events/iers/Wang.pdfHere is my narrative to elaborate on the previous comment: http://contextearth.com/2015/12/12/biennial-connection-from-qbo-to-enso/
Figured some of this out by trying to decipher ref [2] in prior comment.
and conclusion
They were arguing that the ENSO period would come about by bifurcating the 14-year period down to the known ENSO range of 2-7 years. While I think the key is that the 14-year period is multiplied against the 2-year biennial period, generating the time-series profile that matches the ENSO data.
Here is my narrative to elaborate on the previous comment: http://contextearth.com/2015/12/12/biennial-connection-from-qbo-to-enso/ Figured some of this out by trying to decipher [ref [2] in prior comment](http://www1.bipm.org/utils/en/events/iers/Wang.pdf). >"Interestingly, the El Nino events are aware of reason for Earth accelerations while La Nina events for Earth decelerations. El Nino events have discovered of quasi-period of 2~7 yr. Notice that bifurcation may take place in the process of Earth’s free acceleration for LOD, maybe the El Nino events are caused by the free accelerations of periods 7.3 yr with bifurcations. In 7.3 years, the ocean water near the equator may be translated about 0.402 m as huge matter redistribution for water fluid persisting in period of intra annual to one side of the ocean. So the water flows to pile up in one side of Pacific Ocean eastward or reflux westward. The pile-up of the equatorial water brings warmth eastward or westward and changes the inherent equilibrium of the equatorial atmosphere so that causes the weather to change yielding El Nino or La Nina. Hence this may be the dynamical force causing ENSO. Evidently, there are some terms of forced fluctuations in LOD. But as known, forced fluctuations may not cause accelerations. Forced fluctuations are made from the variations of atmosphere and ocean flood while free fluctuations yield centrifugal accelerations. Only free fluctuations may cause accelerations and further centrifugal forces that yield water flowing periodically one way of a direction." > "However, the periods are difficult to explain the ENSO occurring near Equator in Pacific Ocean. We deduce here the free fluctuations for LOD may have inherent period of 14.6 yr from the triaxiality of real Earth. As known, the ENSO events have no inherent periods but from 1.5 to 7 yr as. We make some discussions here for the phenomenon. As discussed in the text, the model of rotation equations has been seen evidently as nonlinear and we solve the model with linearized approach of direct product decomposition. The real result of the solution must be treated in product formulation. So the linearized solution of period 14.6 yr must be affected by product of other fluctuations in the process. Thus the inherent period of 14.6 may become instable and may yield bifurcation to split into series of semi-periods such as 7.3 yr, 3.65 yr, 1.825 yr and such on. This may be the different periods for ENSO to occur as pseudo-periods 7 yr, 4 yr and 2 yr. On the other hand, the centrifugal force of acceleration amount 7.55 mm/yr2 also may not stable in the nonlinear process. Some times the centrifugal force is strengthened and other times decreased so that the ENSO events may burst strong some times and weak other times. Generally, the ENSO events occur in this nonlinear process unstably caused by the free fluctuations of LOD. Here we discover the dynamical force of free accelerations of rotation velocity. This model solves the resources of dynamical force for ENSO but is to study further and deeper. In order to test the two free acceleration variations deduced in theoretically in this study of data series for El Nino and Southern Oscillation (ENSO) events, subharmonic beating and synchronic need to be noticed. The appearance quasi-period of ENSO may not be identical to the prediction here. The main quasi-period of ENSO may be correlated to the free oscillation periods but with variations of subharmonic beating and nonlinear coupling. Anyway, the free acceleration of rotation velocity may be clue to discuss quasi-period of ENSO." and conclusion > "Nonlinear dynamics makes the Earth rotation breaking a revolution so that updated the whole polar motion science. We must humor the conformance of the advancing history without against the tidal current. Professor E Grafarend [2005] published a course about nonlinearity in geodesy as well as in geophysics. Therefore, Earth rotation theory must be reformed to a new Convention in which the two free wobbles should be stated clearly and the similar accompanied consequences as those of the Chandler wobble. Simultaneously, the whole theory of Earth rotation must face to new challenges of two stable and an unstable component solutions with chaotic final state." They were arguing that the ENSO period would come about by bifurcating the 14-year period down to the known ENSO range of 2-7 years. While I think the key is that the 14-year period is multiplied against the 2-year biennial period, generating the time-series profile that matches the ENSO data.Integrating the wave equation transform approach, the biennial modulation isn't 100% and longer term factors are unmodulated to a degree.
Integrating the wave equation transform approach, the biennial modulation isn't 100% and longer term factors are unmodulated to a degree. Here is a training interval starting in 1953, adding the unmodulated long terms
I went back to an older paper [1] that was discussed here before. In retrospect, these guys were tantalizingly close to figuring out ENSO. They had the biennial factor in their Eq(1) but then didn't seem to pursue it.
[1]K.-Y. Kim, J. J. O’Brien, and A. I. Barcilon, “The principal physical modes of variability over the tropical Pacific,” Earth Interactions, vol. 7, no. 3, pp. 1–32, 2003.
PDF
Here is a training interval starting in 1953, adding the unmodulated long terms  I went back to an older paper [1] that was [discussed here before](https://forum.azimuthproject.org/discussion/comment/12639/#Comment_12639). In retrospect, these guys were tantalizingly close to figuring out ENSO. They had the biennial factor in their Eq(1) but then didn't seem to pursue it. [1]K.-Y. Kim, J. J. O’Brien, and A. I. Barcilon, “The principal physical modes of variability over the tropical Pacific,” Earth Interactions, vol. 7, no. 3, pp. 1–32, 2003. [PDF](http://journals.ametsoc.org/doi/pdf/10.1175/1087-3562(2003)007%3C0001%3ATPPMOV%3E2.0.CO%3B2)This Chinese research team predicted a plausible mechanism for the forcing of ENSO.
Wang, Wenjun. “Free acceleration of Earth rotation and energy source of ENSO.” 37th COSPAR Scientific Assembly. Vol. 37. 2008.
I modeled a 7 month periodic forcing as Wang suggested, and sure enough, it showed up as a sharp factor centered at 0.5791 years = 6.95 months (albeit within the noise of ENSO). This is a 2nd derivative of the time-series, so it exaggerates the higher frequencies: Most of the filtered signal is in the indicated biennial area, which contains the observed ~14.6 year modulated signal.
Tie this into Meehl's ongoing research on the biennial component to ENSO and it can shore up some loose ends in our understanding:
Meehl, Gerald A., and Julie M. Arblaster. "Relating the strength of the tropospheric biennial oscillation (TBO) to the phase of the Interdecadal Pacific Oscillation (IPO)." Geophysical Research Letters 39.20 (2012).
Yet there are strong voices against this kind of approach in climate science. I read through the following paper, and they claim just what the title says -- picking out biennial oscillations in the time series could be just the result of a random pattern in the white noise.
Stuecker, M. F., Timmermann, A., Yoon, J., & Jin, F. F. (2015). Tropospheric Biennial Oscillation (TBO) indistinguishable from white noise. Geophysical Research Letters, 42(18), 7785-7791.
Note that one of the authors is Jin, who has one of the delayed action oscillator models for ENSO
Jin, Fei-Fei. "An equatorial ocean recharge paradigm for ENSO. Part I: Conceptual model." Journal of the Atmospheric Sciences 54.7 (1997): 811-829.
This Chinese research team predicted a plausible mechanism for the forcing of ENSO. Wang, Wenjun. “Free acceleration of Earth rotation and energy source of ENSO.” 37th COSPAR Scientific Assembly. Vol. 37. 2008. > “Earth is a triaxial body. Triaxial body rotation has Euler dynamic model of nonlinearity with A¡B¡C. We develop a decomposition theorem for obtaining three true solutions for the rotation with two stable wobbles of Chandler and another decadal of period 14.6 a. The other may be an unstable inverted pendulum of one way sway. The nonlinear coupling of the two free wobbles provide free accelerations for the rotation or LOD with a period of 7 month and another 14.6 a. The fluctuation of period 14.6 a may cause bifurcation cascade in nonlinear coupling and a series of periods 7.3, 3.65 and 1.825 a may be observed with free accelerations and decelerations so that cause ocean water near the equator flow eastward and westward. This makes the energy source of ENSO in eigen periods of 7, 4, 2 a. Therefore in this study we verify that the energy source of ENSO is the free acceleration of Earth rotation. ” I modeled a 7 month periodic forcing as Wang suggested, and sure enough, it showed up as a sharp factor centered at 0.5791 years = 6.95 months (albeit within the noise of ENSO). This is a 2nd derivative of the time-series, so it exaggerates the higher frequencies: Most of the filtered signal is in the indicated biennial area, which contains the observed ~14.6 year modulated signal.  Tie this into Meehl's ongoing research on the biennial component to ENSO and it can shore up some loose ends in our understanding: Meehl, Gerald A., and Julie M. Arblaster. "Relating the strength of the tropospheric biennial oscillation (TBO) to the phase of the Interdecadal Pacific Oscillation (IPO)." Geophysical Research Letters 39.20 (2012). Yet there are strong voices against this kind of approach in climate science. I read through the following paper, and they claim just what the title says -- picking out biennial oscillations in the time series could be just the result of a random pattern in the white noise. Stuecker, M. F., Timmermann, A., Yoon, J., & Jin, F. F. (2015). Tropospheric Biennial Oscillation (TBO) indistinguishable from white noise. Geophysical Research Letters, 42(18), 7785-7791. Note that one of the authors is Jin, who has one of the delayed action oscillator models for ENSO Jin, Fei-Fei. "An equatorial ocean recharge paradigm for ENSO. Part I: Conceptual model." Journal of the Atmospheric Sciences 54.7 (1997): 811-829.I know everyone likes to see symmetry in nature. Here is the ENSO biennial signal modulated with longer-term periods described in the preceding comments
This is essentially a periodogram of the sidebands, where I transformed the x-axis to reflect the period instead of the frequency.
Note the nice symmetry about a 2.0077 biennial year period, which was chosen to align the discrete 2048 point FFT with the 1 month per sample data series. Keeping track of the 15-year phase reversal starting in 1981 is important, as without that, it would likely not be observable.
As a reminder, the + and - sidebands come about from this modulation:
$ sin(\pi t) \cdot sin(\omega_m t) = \frac{1}{2} ( cos(\pi t - \omega_m t) - cos(\pi t + \omega_m t) ) $
I know everyone likes to see symmetry in nature. Here is the ENSO biennial signal modulated with longer-term periods described in the preceding comments  This is essentially a periodogram of the sidebands, where I transformed the x-axis to reflect the period instead of the frequency. Note the nice symmetry about a 2.0077 biennial year period, which was chosen to align the discrete 2048 point FFT with the 1 month per sample data series. Keeping track of the 15-year phase reversal starting in 1981 is important, as without that, it would likely not be observable. As a reminder, the + and - sidebands come about from this modulation: $ sin(\pi t) \cdot sin(\omega_m t) = \frac{1}{2} ( cos(\pi t - \omega_m t) - cos(\pi t + \omega_m t) ) $Here is a very short training interval, from 1940 to 1960, using the 7 bolded frequency factors from the previous chart. Evaluate how well it extrapolates outside of the interval.
Areas highlighted in yellow are where the predicted phase is reversed. This is on the wave equation transformed data, with the ENSO signal inverted from 1981 to 1996.
Here is a very short training interval, from 1940 to 1960, using the 7 bolded frequency factors from the previous chart. Evaluate how well it extrapolates outside of the interval.  Areas highlighted in yellow are where the predicted phase is reversed. This is on the wave equation transformed data, with the ENSO signal inverted from 1981 to 1996.@WebHubTel,
I'm sure it is documented someplace on Azimuth, but might you post a short description of the machine learning ("ML") algorithm used with these, and possibly a pointer to a longer description?
I have another application involving time series where I might want to try an ML algorithm.
Thank you!
@WebHubTel, I'm sure it is documented someplace on Azimuth, but might you post a short description of the machine learning ("ML") algorithm used with these, and possibly a pointer to a longer description? I have another application involving time series where I might want to try an ML algorithm. Thank you!HNY Paul and Jan, My "haven't done list" has hoping I could get Paul and anybody else to add to my cut and pasted abstracts on ML: Deep learning eg. about Eureqa and symbolic regression robustness in general.
HNY Paul and Jan, My "haven't done list" has hoping I could get Paul and anybody else to add to my cut and pasted abstracts on ML: [[Deep learning]] eg. about Eureqa and symbolic regression robustness in general.Jan, The main machine learning algorithm I use is embedded in a tool called Eureqa by Nutonian.
Everything that I know about Eureqa is contained in the User Guide. This includes all the details of how to search for embedded differential equations in the data.
Jan, The main machine learning algorithm I use is embedded in a tool called [Eureqa](http://www.nutonian.com/products/eureqa/) by Nutonian. Everything that I know about Eureqa is contained in the [User Guide](http://formulize.nutonian.com/eureqa-user-guide.html). This includes all the details of how to search for embedded differential equations in the data.BTW, the results from comments such as #141 above are not from any sophisticated machine learning. That is simply multiple linear regression where I "train" the factor set by focusing on an sub-interval, and then check to see how it works outside that interval.
Eureqa does this as well with their symbolic regression algorithm but how they mix the "in-band" and "out-of-bad" intervals is a bit of a mystery to me. They somehow use only the "in-band" interval for training, yet still peek at how well it works on the "out-of-band" data during their exhaustive ML search. It may be that they only allow the error to drop slightly when computing "out-of-band" fit, otherwise they do not continue with the search on the "in-band" interval. That's understandable if they want to prove that the fit is stationary over a sub-interval as well as over a longer interval, otherwise it is cheating according to the strict precepts of no look-ahead for out-of-band validation.
I am just guessing here because the guts of the algorithm are proprietary. They make money off this tool so I imagine that whatever algorithm they use is OK as long as they have satisfied customers.
BTW, the results from comments such as [#141](#Comment_15092) above are not from any sophisticated machine learning. That is simply multiple linear regression where I "train" the factor set by focusing on an sub-interval, and then check to see how it works outside that interval. Eureqa does this as well with their symbolic regression algorithm but how they mix the "in-band" and "out-of-bad" intervals is a bit of a mystery to me. They somehow use only the "in-band" interval for training, yet still peek at how well it works on the "out-of-band" data during their exhaustive ML search. It may be that they only allow the error to drop slightly when computing "out-of-band" fit, otherwise they do not continue with the search on the "in-band" interval. That's understandable if they want to prove that the fit is stationary over a sub-interval as well as over a longer interval, otherwise it is cheating according to the strict precepts of no look-ahead for out-of-band validation. I am just guessing here because the guts of the algorithm are proprietary. They make money off this tool so I imagine that whatever algorithm they use is OK as long as they have satisfied customers.Thanks, Jim. I'm very much interested in developing and coding up ML algorithms for these kinds of series independently of any product. I've seen some remarkable success in my own work from boosting, and hope to adapt it to series. Thanks.
Thanks, Jim. I'm very much interested in developing and coding up ML algorithms for these kinds of series independently of any product. I've seen some remarkable success in my own work from boosting, and hope to adapt it to series. Thanks.The two examples of machine learning that I have, one for QBO and one for ENSO, show the strengths and weaknesses of the basic approach. The generic machine learning of Eureqa was able to set one on a trail by doing a heavy-lifting search of possibilities, but wan't able to close the book, IMO. It all depends on what rules go in to the machine learning algorithm. With respect to QBO machine learning, Eureqa hinted at the possibility of signal aliasing, but was unable to use that information to further refine a search. For example, it had no idea as to the aliased vs unaliased value matching sets and it actually seemed accidental that the aliased signals were even found. And with ENSO, Eureqa likely could have detected a frequency modulated signal but I was able to spot it from the set of individual factors that Eureqa supplied.
Also I could point out that one could even see the aliasing and modulation from a frequency power spectra, especially the modulation, whereas Eureqa was actually violating the Nyquist criteria when it suggested the higher frequency aliased values, which sets it apart from a Nyquist-limited power spectra.
Like all things AI, the power is dependent on the rules that go in to the inference engine. If the rules don't exist then it won't find the deeper meaning of any result the algorithm supplies. In other words, there is still much interpretation that needs to occur.
The two examples of machine learning that I have, one for QBO and one for ENSO, show the strengths and weaknesses of the basic approach. The generic machine learning of Eureqa was able to set one on a trail by doing a heavy-lifting search of possibilities, but wan't able to close the book, IMO. It all depends on what rules go in to the machine learning algorithm. With respect to QBO machine learning, Eureqa hinted at the possibility of signal aliasing, but was unable to use that information to further refine a search. For example, it had no idea as to the aliased vs unaliased value matching sets and it actually seemed accidental that the aliased signals were even found. And with ENSO, Eureqa likely could have detected a frequency modulated signal but I was able to spot it from the set of individual factors that Eureqa supplied. Also I could point out that one could even see the aliasing and modulation from a frequency power spectra, especially the modulation, whereas Eureqa was actually violating the Nyquist criteria when it suggested the higher frequency aliased values, which sets it apart from a Nyquist-limited power spectra. Like all things AI, the power is dependent on the rules that go in to the inference engine. If the rules don't exist then it won't find the deeper meaning of any result the algorithm supplies. In other words, there is still much interpretation that needs to occur.In fitting to ENSO, I have never looked at data beyond the date September 2013. I did this intentionally because I wanted to leave some out-of-band data for validation.
The following should be taken with a grain of salt but this is the SOI model extrapolation for years beyond 2013.
It does not prove anything because the accuracy of the model is <80% in getting the sign of the excursion right, but it does nail the current El Ni~no, which some say has just peaked. See the green up-arrow. It is a strong peak in terms of width but doesn't match the deep excursion of the 1998 peak. (Recall that for SOI, negative means higher temperatures)
In fitting to ENSO, I have never looked at data beyond the date September 2013. I did this intentionally because I wanted to leave some out-of-band data for validation. The following should be taken with a grain of salt but this is the SOI model extrapolation for years beyond 2013.  It does not prove anything because the accuracy of the model is <80% in getting the sign of the excursion right, but it does nail the current El Ni~no, which [some say has just peaked](http://www.wunderground.com/news/el-nino-noaa-january-2016-update). See the green up-arrow. It is a strong peak in terms of width but doesn't match the deep excursion of the 1998 peak. (Recall that for SOI, negative means higher temperatures)So, if I were digging into this, I'd like to know how the "accuracy of the model" is assessed. In particular I wonder if that's Mean Integrated Square Error ("MISE") or something else. It's not at all clear that MISE is the appropriate measure for a non-stationary series and, especially, for a series which has chaotic contributions. The score, in the latter two instances, should penalize for the result not being smooth, since it is the estimator, at any point, of all possible futures. Chasing a particular future, since it is unattainable, is not a useful goal.
So, if I were digging into this, I'd like to know how the "accuracy of the model" is assessed. In particular I wonder if that's Mean Integrated Square Error ("MISE") or something else. It's not at all clear that MISE is the appropriate measure for a non-stationary series and, especially, for a series which has chaotic contributions. The score, in the latter two instances, should penalize for the result not being smooth, since it is the estimator, at any point, of all possible futures. Chasing a particular future, since it is unattainable, is not a useful goal.So would I. I've been looking at the ENSO and QBO in tandem, and since I discovered that the QBO is stationary and predictable according to the lunisolar tidal parameters, my working assumption is that ENSO may be as well.
Yet, ENSO is a tougher nut to crack, largely because the sub-yearly behavior is so sensitive to localized weather conditions, such as cyclonic activity. In comparison, QBO does show stationary patterns in the fine sub-yearly structure -- I can rationalize this as the QBO data is taken from the stratosphere, which is removed from the lower troposphere instabilities.
So QBO is largely stationary at both sub-yearly and longer time-scales, while ENSO likely only on time scales greater than a year.
So would I. I've been looking at the ENSO and QBO in tandem, and since I discovered that the QBO is stationary and predictable according to the lunisolar tidal parameters, my working assumption is that ENSO may be as well. Yet, ENSO is a tougher nut to crack, largely because the sub-yearly behavior is so sensitive to localized weather conditions, such as cyclonic activity. In comparison, QBO does show stationary patterns in the fine sub-yearly structure -- I can rationalize this as the QBO data is taken from the stratosphere, which is removed from the lower troposphere instabilities. So QBO is largely stationary at both sub-yearly and longer time-scales, while ENSO likely only on time scales greater than a year.