ENSO means zero in Japanese and also has a Zen interpretation
“The ensō symbolizes absolute enlightenment, strength, elegance, the universe, and mu (the void).”
Comment Source:Couple of more blog posts on nailing down the mechanism for ENSO:
---
---
**(1)** This one contextualizes the ENSO behavior in terms of a common forcing governing ENSO, QBO, and the Chandler wobble
http://contextearth.com/2017/05/21/the-lunar-geophysical-connection/
Here are a few charts from the post:
What are the odds that the fundamental frequencies of all these behaviors are the same to that precision?
As it turns out NASA JPL were on this lunar-forcing path several years ago, but elected not to fund the proposed research and so the progress stalled.
---
---
**(2)** Yesterday's post provides a historical context. Essentially all the geophysics applicable to the model was known by 1920.
http://contextearth.com/2017/05/30/ocean-dynamics-history/
Could have done a decent job in predicting ENSO evolution with Pacific Ocean SST data up to 1920
---
---
**(3)** Posted on the Azimuth blog.
https://johncarlosbaez.wordpress.com/2017/04/24/complexity-theory-and-evolution-in-economics/#comment-92343
ENSO means zero in Japanese and also has a Zen interpretation
> “The ensō symbolizes absolute enlightenment, strength, elegance, the universe, and mu (the void).”
If we actually had better knowledge of climate behavior and more emphasis on how AGW interacts with Peak Oil, we would likely have a different discourse.
Just by having a real understanding of ENSO, we could compensate out the natural variability in the global temperature signal. The almost monotonic increase in temperature would be much more obvious and we wouldn't have to contend with the uncertainty players such as Curry controlling the political discussion.
The common criticism of these kinds of models is that they overfit and can not be tested with historical data, as any contamination of a model with available data will then taint the model and give a false impression that it actually works. This model is different and works with minimal set of parameters, just the 2 tidal cycles and a seasonal cycle.
Comment Source:Trump pulled US out of the Paris Accord :(
If we actually had better knowledge of climate behavior and more emphasis on how AGW interacts with Peak Oil, we would likely have a different discourse.
Just by having a real understanding of ENSO, we could compensate out the natural variability in the global temperature signal. The almost monotonic increase in temperature would be much more obvious and we wouldn't have to contend with the uncertainty players such as Curry controlling the political discussion.
This ENSO model is looking more solid. With virtually no free parameters, any automated fit to an ENSO interval does a good job of recreating the unfitted intervals.
http://contextearth.com/2017/05/31/enso-model-fit-1880-1980/
The common criticism of these kinds of models is that they overfit and can not be tested with historical data, as any contamination of a model with available data will then taint the model and give a false impression that it actually works. This model is different and works with minimal set of parameters, just the 2 tidal cycles and a seasonal cycle.
Have to look at Fig.6 in the post to see the match closely.
This is untainted because the LOD is completely out-of-band with the fitting data and so becomes a strong validation test for the model.
Comment Source:Couple more blog posts that demonstrate how well the ENSO model works in describing the observations and the geophysics ...
The first is evidence as to how such a simple model can produce such a rich Fourier spectra, contrary to people's preconceived notions:
http://contextearth.com/2017/06/02/enso-and-fourier-analysis/
The second is an untainted match between the lunar forcing used to model ENSO and the lunar forcing obtained from the earth's Length-of-day (LOD) data
http://contextearth.com/2017/06/03/enso-forcing-validation-via-lod-data/
Have to look at Fig.6 in the post to see the match closely.
This is untainted because the LOD is completely [out-of-band](https://en.wikipedia.org/wiki/Out-of-band) with the fitting data and so becomes a strong validation test for the model.
ENSO forcing match against digitized LOD variation
I took the correlation coefficient of this and its above 0.7. For the cycle factors applied, the fit doesn't get much better if the phases and amplitude are allowed to vary -- the correlation coefficient go up by a slight 0.02, and it reduces the ENSO fit only slightly.
Comment Source:ENSO forcing match against digitized LOD variation
I took the correlation coefficient of this and its above 0.7. For the cycle factors applied, the fit doesn't get much better if the phases and amplitude are allowed to vary -- the correlation coefficient go up by a slight 0.02, and it reduces the ENSO fit only slightly.
Comment Source:From the above agreement in forcing stimulii.
(A)The forcing for QBO is mainly Draconic
(B)(C) ENSO and LOD have the same tidal forcing
Difficult to believe that behaviors such as ENSO and QBO are not related to external forcing. I can't think of one large scale cyclic behavior that can't be pinned to some other regular cycle. Even the cycles of sunspots are known to be intimately tied to the sun's rotation. So even though they haven't quite nailed the predictability of sunspots yet, they know it isn't some spontaneous oscillation as the purveyors of the wind-only mechanism for ENSO seem to think.
Thus, much like sunspots, ENSO is likely sensitive to variations in the Earth's rotation speed. As the moon is known to cause cyclic variations in the speed, these same variations should be able to be picked up in an ENSO wave equation model. And what do we find but that the two most critical lunar periods, the Draconic 27.2122 days and Anomalistic 27.5545 days feed into a best-fit model to within 1 minute each.
Got a reply tweet from Andrew Dessler concerning this and he said "Climate is a physics problem, not a statistics one. Looking at correlations is interesting, but not sufficient. Must have physical basis."
Some of these guys do not realize that science deals with this situation automatically. They should be able to eventually reject the lunar forcing by coming up with evidence that rejects it. It shouldn't be hard, as all they have to do is show that the ENSO cycles are incommensurate with the lunar cycles. And show how there is not enough energy supplied by the lunisolar cycles to move volumes of water in a reduced effective gravity environment. If they can't, however, then the lunar model will remain as a potential ENSO driver.
Comment Source:Difficult to believe that behaviors such as ENSO and QBO are not related to external forcing. I can't think of one large scale cyclic behavior that can't be pinned to some other regular cycle. Even the cycles of sunspots are known to be intimately tied to the sun's rotation. So even though they haven't quite nailed the predictability of sunspots yet, they know it isn't some spontaneous oscillation as the purveyors of the wind-only mechanism for ENSO seem to think.
Thus, much like sunspots, ENSO is likely sensitive to variations in the Earth's rotation speed. As the moon is known to cause cyclic variations in the speed, these same variations should be able to be picked up in an ENSO wave equation model. And what do we find but that the two most critical lunar periods, the Draconic 27.2122 days and Anomalistic 27.5545 days feed into a best-fit model to within 1 minute each.
http://contextearth.com/2017/06/08/scaling-el-nino/
Got a reply tweet from Andrew Dessler concerning this and he said *"Climate is a physics problem, not a statistics one. Looking at correlations is interesting, but not sufficient. Must have physical basis."*
Some of these guys do not realize that science deals with this situation automatically. They should be able to eventually reject the lunar forcing by coming up with evidence that rejects it. It shouldn't be hard, as all they have to do is show that the ENSO cycles are incommensurate with the lunar cycles. And show how there is not enough energy supplied by the lunisolar cycles to move volumes of water in a reduced effective gravity environment. If they can't, however, then the lunar model will remain as a potential ENSO driver.
This is a magnification of the fitting contour around the best forcing period values for ENSO. These pair of peak values are each found to be less than a minute apart from the known values of the Draconic cycle (27.2122 days) and Anomalistic cycle (27.5545 days).
The forcing comes directly from the angular momentum variations in the Earth's rotation. The comparison between what the ENSO model uses (from the Draconic and Anomalistic terms above) and what is measured via monitoring the length-of-day (LOD) is shown below
The lower LOD pane is a fit over 3 years, which is about 40 lunar months. These essentially get aliased in the upper ENSO pane, which only responds to the peak tidal forces at a specific time of the year -- around Nov/Dec.
So many numbers have to align perfectly for this model to work out, and it looks like it does.
Comment Source:This is a magnification of the fitting contour around the best forcing period values for ENSO. These pair of peak values are each found to be less than a minute apart from the known values of the Draconic cycle (27.2122 days) and Anomalistic cycle (27.5545 days).
The forcing comes directly from the angular momentum variations in the Earth's rotation. The comparison between what the ENSO model uses (from the Draconic and Anomalistic terms above) and what is measured via monitoring the length-of-day (LOD) is shown below
The lower LOD pane is a fit over 3 years, which is about 40 lunar months. These essentially get aliased in the upper ENSO pane, which only responds to the peak tidal forces at a specific time of the year -- around Nov/Dec.
So many numbers have to align perfectly for this model to work out, and it looks like it does.
This is the physics of the tidal forcing -- imparting a 1 millisecond slowdown (or speedup) on the rotation of the earth with a surface velocity of almost 500 meters/second over the course of a couple of weeks (a fortnight) will result in an inertial lateral movement of ~ 1/2 a meter in the volume of the Pacific ocean due to Newton's first law.
This does not seem like a big deal until you realize that the thermocline can absorb this inertial impulse as a vertical sloshing, since the effective gravity is reduced by orders of magnitude due to the slight density differences above and below the thermocline. This is reflected as an Atwood number and shows up in Rayleigh-Taylor instability experiments, e.g. SEE THIS PAPER
With an Atwood number less than 0.001 which is ~0.1% density differences in a stratified fluid, the 0.5 meter displacement that occurs over two weeks now occurs effectively over half an hour. That's just an elementary scaling exercise.
So intuitively, one has to ask the question of what would happen if the ocean was translated laterally by 1/2 a meter over the course of a 1/2 an hour? We know what happens with earthquakes in something as simple as a swimming pool
or as threatening as a tsunami. But this is much more subtle because we can't obviously see it, and why it has likely been overlooked as a driver of ENSO.
All that math modeling of ENSO described here works backwards to this point. The actual forcing working on the earth's rotation can lead to the response shown here, both in the dynamic sense of tracing the measured path and now in terms of a physical order-of-magnitude justification.
Comment Source:This is the physics of the tidal forcing -- imparting a 1 millisecond slowdown (or speedup) on the rotation of the earth with a surface velocity of almost 500 meters/second over the course of a couple of weeks (a fortnight) will result in an inertial lateral movement of ~ 1/2 a meter in the volume of the Pacific ocean due to Newton's first law.
This does not seem like a big deal until you realize that the thermocline can absorb this inertial impulse as a vertical sloshing, since the effective gravity is reduced by orders of magnitude due to the slight density differences above and below the thermocline. This is reflected as an Atwood number and shows up in Rayleigh-Taylor instability experiments, e.g. [SEE THIS PAPER](http://rsta.royalsocietypublishing.org/content/roypta/368/1916/1663.full.pdf)
With an Atwood number less than 0.001 which is ~0.1% density differences in a stratified fluid, the 0.5 meter displacement that occurs over two weeks now occurs effectively over half an hour. That's just an elementary scaling exercise.
So intuitively, one has to ask the question of what would happen if the ocean was translated laterally by 1/2 a meter over the course of a 1/2 an hour? We know what happens with earthquakes in something as simple as a swimming pool
https://youtu.be/27GMnYEWL0M
or as threatening as a tsunami. But this is much more subtle because we can't obviously see it, and why it has likely been overlooked as a driver of ENSO.
All that math modeling of ENSO described here works backwards to this point. The *actual forcing* working on the earth's rotation can lead to the response shown here, both in the dynamic sense of tracing the measured path and now in terms of a physical order-of-magnitude justification.
The supposedly simplest "toy" models of ENSO that we describe on the Azimuth Project wiki page here http://www.azimuthproject.org/azimuth/show/ENSO are the ones that remarkably work the best to describe the actual dynamics. If the delayed action oscillator (minus the cubic term) is combined with a seasonally-modulated lunar forcing that's essentially all that is needed to train the model.
Comment Source:The supposedly simplest "toy" models of ENSO that we describe on the Azimuth Project wiki page here http://www.azimuthproject.org/azimuth/show/ENSO are the ones that remarkably work the best to describe the actual dynamics. If the delayed action oscillator (minus the cubic term) is combined with a seasonally-modulated lunar forcing that's essentially all that is needed to train the model.
"The supposedly simplest "toy" models of ENSO that we describe on the Azimuth Project wiki page here http://www.azimuthproject.org/azimuth/show/ENSO are the ones that remarkably work the best to describe the actual dynamics. If the delayed action oscillator (minus the cubic term) is combined with a seasonally-modulated lunar forcing that's essentially all that is needed to train the model."
Comment Source:My last comment:
> "The supposedly simplest "toy" models of ENSO that we describe on the Azimuth Project wiki page here http://www.azimuthproject.org/azimuth/show/ENSO are the ones that remarkably work the best to describe the actual dynamics. If the delayed action oscillator (minus the cubic term) is combined with a seasonally-modulated lunar forcing that's essentially all that is needed to train the model."
Elaborated further here:
http://contextearth.com/2017/06/23/ensoqbo-elevator-pitch/
Thought to comment on what a long strange trip it's been. The journey to modeling ENSO and QBO has been circuitous and then essentially doubled back to the most basic kind of forcing and the simplest toy differential equations.
The ENSO behavior is modeled as 2 lunar tidal signals and an annual forcing impulse applied to a delay differential equation of 1 year delay. Could have started with this premise from day one, but nothing in the research literature indicated lunar forcing had any effect on ENSO.
Same goes for QBO except that it is essentially a single lunar tidal signal and a bi-annual seasonal forcing signal - one impulse per nodal crossing. Lindzen had considered lunar forcing early but apparently couldn't find any correlation and that's why no one followed up there.
Looking back my first blog post on this topic was early 2014, so it's been almost 4 years of spare-time effort. And even though this was anticipated to be a software coding project, the model is simple enough to express on a spreadsheet without the need for any macros or scripts except for a standard Solver plugin. It's essentially a little more complex than a basic tidal analysis program.
Comment Source:Thought to comment on what a long strange trip it's been. The journey to modeling ENSO and QBO has been circuitous and then essentially doubled back to the most basic kind of forcing and the simplest toy differential equations.
The ENSO behavior is modeled as 2 lunar tidal signals and an annual forcing impulse applied to a delay differential equation of 1 year delay. Could have started with this premise from day one, but nothing in the research literature indicated lunar forcing had any effect on ENSO.
Same goes for QBO except that it is essentially a single lunar tidal signal and a bi-annual seasonal forcing signal - one impulse per nodal crossing. Lindzen had considered lunar forcing early but apparently couldn't find any correlation and that's why no one followed up there.
Looking back my first blog post on this topic was early 2014, so it's been almost 4 years of spare-time effort. And even though this was anticipated to be a software coding project, the model is simple enough to express on a spreadsheet without the need for any macros or scripts except for a standard Solver plugin. It's essentially a little more complex than a basic tidal analysis program.
http://ContextEarth.com
For awhile, I was using a Mathematica solver but it doesn't allow a correlation coefficient as a goal, only least squares. I think there is some property of using a correlation coefficient that allows a solver to avoid getting stuck in local minima. It may have to do with not having to worry about scaling at every step. The rescaling can always get done at the end.
There is also the Eureqa solver that I used early on, but that tool got bought out by another company and not sure of it's status. Eureqa allowed correlation coefficient and a whole range of optimization targets, including an interesting hybrid cc+leastSquares target. The architects of that tool understood the importance of providing different targets for optimization.
Probably should try using R and one of the solvers there, but I am in a mode of staying with what works for now
Comment Source:Jim asked:
> "Btw what solver did you use?"
I used the builtin [Excel Solver](https://support.office.com/en-us/article/Define-and-solve-a-problem-by-using-Solver-9ed03c9f-7caf-4d99-bb6d-078f96d1652c).
There is also an open Solver that I have yet to try: http://opensolver.org/
For awhile, I was using a Mathematica solver but it doesn't allow a correlation coefficient as a goal, only least squares. I think there is some property of using a correlation coefficient that allows a solver to avoid getting stuck in local minima. It may have to do with not having to worry about scaling at every step. The rescaling can always get done at the end.
There is also the Eureqa solver that I used early on, but that tool got bought out by another company and not sure of it's status. Eureqa allowed correlation coefficient and a whole range of optimization targets, including an interesting hybrid cc+leastSquares target. The architects of that tool understood the importance of providing different targets for optimization.
Probably should try using R and one of the solvers there, but I am in a mode of staying with what works for now
Comment Source:Jim, If you use it, let me know how it works. I am also going for minimizing the barrier to usage.
Latest validation here http://contextearth.com/2017/09/27/enso-tidal-forcing-validated-by-lod-data/
So far I've failed to load an .xlsx file of NOAA sesimic data into the Google sheets opensolver plugin version. I'll have to try with whatever the openoffice spreadsheet is called. Opensolver needs chromium which won't run on my currently, semi-borked setup.
Comment Source:So far I've failed to load an .xlsx file of NOAA sesimic data into the Google sheets opensolver plugin version. I'll have to try with whatever the openoffice spreadsheet is called. Opensolver needs chromium which won't run on my currently, semi-borked setup.
Comment Source:After spending all that time with the ENSO model, this is the progress after a day of working the Atlantic version of ENSO, the AMO
http://contextearth.com/2017/10/03/amo/
Naw, I don't understand with any depth whatsoever, but I think it's very cool.
Comment Source:[Quantum and ENSO connects](http://www.sciencemag.org/news/2017/10/waves-drive-global-weather-patterns-finally-explained-thanks-inspiration-bagel-shaped)!
[Full paper: _Topological_ _origins_ _of_ _equatorial_ _waves_](http://science.sciencemag.org/content/early/2017/10/04/science.aan8819.full)
Oh, JohnB's gonna be writing about this for _months_!
There's also [a supplement](http://science.sciencemag.org/highwire/filestream/700023/field_highwire_adjunct_files/0/aan8819_Delplace_SM.pdf) and two movies, [S1](http://science.sciencemag.org/highwire/filestream/700023/field_highwire_adjunct_files/1/aan8819s1.mov) and [S2](http://science.sciencemag.org/highwire/filestream/700023/field_highwire_adjunct_files/2/aan8819s2.mov).
_Naw_, I don't understand with any depth whatsoever, but I think it's very cool.
Comment Source:Thanks Jan, that's looks a lot like what I solved for last year -- reducing the Coriolis forces at the equator.
http://contextEarth.com/2016/09/23/compact-qbo-derviation/
I simplified much more than what they did.
"These curl equations are fascinating and are of course endemic in applications from electromagnetics to fluid dynamics. Perhaps there is some overlap with the model of the QBO equatorial winds that we are working on at John’s Azimuth Forum (see sidebar) and at my blog. Have some notes here: http://contextearth.com/2016/09/23/compact-qbo-derviation/#comment-199906 "
A few years ago, I referenced this paper by Marston, which was a call-to-arms to solving climate problems:
Comment Source:Jan, Perhaps I can try that.
Is the QBO a Berry monopole?
Or is it related to a Weyl point across a Lifshitz transitiion? https://inspirehep.net/record/1441222/plots
Yes, John will be interested in this stuff because it looks like the anti-vortex stuff that he had written about last year
https://johncarlosbaez.wordpress.com/2016/10/07/kosterlitz-thouless-transition/
I made a comment at the time:
> "These curl equations are fascinating and are of course endemic in applications from electromagnetics to fluid dynamics. Perhaps there is some overlap with the model of the QBO equatorial winds that we are working on at John’s Azimuth Forum (see sidebar) and at my blog. Have some notes here: http://contextearth.com/2016/09/23/compact-qbo-derviation/#comment-199906 "
---
A few years ago, I referenced this paper by Marston, which was a call-to-arms to solving climate problems:
https://physics.aps.org/articles/v4/20
And this paper by Vallis is a good inspiration to look at simplifying the physics before doing CFD
http://contextearth.com/2016/09/03/geophysical-fluid-dynamics-first-and-then-cfd/
---
https://youtu.be/JBY5iIfPgd0
Comment Source:This is probably the best preliminary paper on the topic
ELASTIC WAVE EQUATION, Yves Colin de Verdière,
Séminaire de théorie spectrale et géométrie, Grenoble Volume25 (2006-2007) 55-69
http://tsg.cedram.org/cedram-bin/article/TSG_2006-2007__25__55_0.pdf
Comment Source:Re: Topological origins of equatorial waves
Blog post here on this paper: http://contextEarth.com/2017/10/13/interface-inflection-geophysics/
Comment Source:Couple of recent posts where the harmonic series approximation for the ENSO forcing is reduced to a closed-form expression:
http://contextearth.com/2017/10/27/reverse-engineering-the-moons-orbit-from-enso-behavior/
http://contextearth.com/2017/11/03/approximating-the-enso-forcing-potential/
This simplification was also applied to QBO
These are timely findings, as the presentation was accepted to the AGU next month
[GC41B-1022: Biennial-Aligned Lunisolar-Forcing of ENSO: Implications for Simplified Climate Models](https://agu.confex.com/agu/fm17/meetingapp.cgi/Paper/221914) 
Check the sub-title :)
Nothing truly impressive, the paper is reporting on short-term predictions
Comment Source:A rare paper on machine learning for El Nino
Using Network Theory and Machine Learning to predict El Nino, Peter Nooteboom
https://dspace.library.uu.nl/bitstream/handle/1874/353201/Thesis_Peter_Nooteboom.pdf
Nothing truly impressive, the paper is reporting on short-term predictions
In the last month, two of the great citizen scientists that I will be forever personally grateful for have passed away. If anyone has followed climate science discussions on blogs and social media, you probably have seen their contributions.
Keith Pickering was an expert on computer science, astrophysics, energy, and history from my neck of the woods in Minnesota. He helped me so much in working out orbital calculations when I was first looking at lunar correlations. He provided source code that he developed and it was a great help to get up to speed. He was always there to tweet any progress made. Thanks Keith
Kevin O'Neill was a metrologist and an analysis whiz from Wisconsin. In the weeks before he passed, he told me that he had extra free time to help out with ENSO analysis. He wanted to use his remaining time to help out with the solver computations. I could not believe the effort he put in to his spreadsheet, and it really motivated me to spending more time in validating the model. He was up all the time working on it because he was unable to lay down. Kevin was also there to promote the research on other blogs, right to the end. Thanks Kevin.
There really aren't too many people willing to spend time working analysis on a scientific forum, and these two exemplified what it takes to really contribute to the advancement of ideas. Like us, they were not climate science insiders and so will only get credit if we remember them.
Comment Source:In the last month, two of the great citizen scientists that I will be forever personally grateful for have passed away. If anyone has followed climate science discussions on blogs and social media, you probably have seen their contributions.
Keith Pickering was an expert on computer science, astrophysics, energy, and history from my neck of the woods in Minnesota. He helped me so much in working out orbital calculations when I was first looking at lunar correlations. He provided source code that he developed and it was a great help to get up to speed. He was always there to tweet any progress made. Thanks Keith
Kevin O'Neill was a metrologist and an analysis whiz from Wisconsin. In the weeks before he passed, he told me that he had extra free time to help out with ENSO analysis. He wanted to use his remaining time to help out with the solver computations. I could not believe the effort he put in to his spreadsheet, and it really motivated me to spending more time in validating the model. He was up all the time working on it because he was unable to lay down. Kevin was also there to promote the research [on other blogs](https://andthentheresphysics.wordpress.com/2017/10/23/watt-about-breaking-the-pal-review-glass-ceiling/#comment-105271), right to the end. Thanks Kevin.
There really aren't too many people willing to spend time working analysis on a scientific forum, and these two exemplified what it takes to really contribute to the advancement of ideas. Like us, they were not climate science insiders and so will only get credit if we remember them.
This is how conventional tidal prediction is done:
This is an ENSO model fit to SOI data. Same tidal analysis algorithm is used but applying the annual solar cycle and monthly/fortnightly lunar cycles instead of the diurnal and semi-diurnal cycle.
This is an expanded view, with the corelation coefficient of 0.73:
This is a fit trained on the 1880-1950 interval (CC=0.76) and cross-validated on the post-1950 data
This is a fit trained on the post-1950 interval (CC=0.77) and cross-validated on the 1880-1950 data
Like conventional tidal prediction, very little overfitting is observed. Most of what is considered noise in the SOI data is actually the tidal forcing signal.
Comment Source:This is the last of the ENSO charts.
This is how conventional tidal prediction is done:
This is an ENSO model fit to SOI data. Same tidal analysis algorithm is used but applying the annual solar cycle and monthly/fortnightly lunar cycles instead of the diurnal and semi-diurnal cycle.
This is an expanded view, with the corelation coefficient of 0.73:
This is a fit trained on the 1880-1950 interval (CC=0.76) and cross-validated on the post-1950 data
This is a fit trained on the post-1950 interval (CC=0.77) and cross-validated on the 1880-1950 data
Like conventional tidal prediction, very little overfitting is observed. Most of what is considered noise in the SOI data is actually the tidal forcing signal.
Slightly different approach that I should have tried long ago for the ENSO model. Instead of training on the time-series, train on the complex Fourier series in frequency space.
So, this is trained on intervals of the ENSO SOI spectrum (in yellow), and it fills in the rest of the spectrum. Even though there are only 3 known lunar periods + 1 solar, because of the nonlinear Navier-Stokes doubling with multiple interactions due to the precision required of the orbits, it generates a very busy spectral waveform (much more variable than a conventional tidal spectrum)
In real space, the inverse fit captures the time-series with a high >0.8 correlation coefficient.
Comment Source:Slightly different approach that I should have tried long ago for the ENSO model. Instead of training on the time-series, train on the complex Fourier series in frequency space.
So, this is trained on intervals of the ENSO SOI spectrum (in yellow), and it fills in the rest of the spectrum. Even though there are only 3 known lunar periods + 1 solar, because of the nonlinear Navier-Stokes doubling with multiple interactions due to the precision required of the orbits, it generates a very busy spectral waveform (much more variable than a conventional tidal spectrum)
In real space, the inverse fit captures the time-series with a high >0.8 correlation coefficient.
The underlying issue is how does one confirm a hypothetical scientific model without having a controlled experiment to test against?
The best you can do is use portions of the data to align/calibrate the model and then use other orthogonal parts of the data to verify. Otherwise one can always predict future behavior and then wait 20 to 50 years to verify -- which unfortunately is detrimental to progress :(
Comment Source:An example of verification in pattern recognition and signal processing
http://contextearth.com/2018/04/29/enso-model-verification-via-fourier-analysis-infill/
The underlying issue is how does one confirm a hypothetical scientific model without having a controlled experiment to test against?
The best you can do is use portions of the data to align/calibrate the model and then use other orthogonal parts of the data to verify. Otherwise one can always predict future behavior and then wait 20 to 50 years to verify -- which unfortunately is detrimental to progress :(
The residual of the fit across the entire spectrum (Nyquist T = 1/6 y) appears to be flat white noise
No use trying to improve the fit beyond this point --
"Bottom line: when the residuals fail to be white noise, a different model should be tried. Short answer regarding time series regression: If they are not white noise (i.e. they are not normal, not have zero mean or serially autocorrelated), then your model is not fully adequate. Therefore, you should revise your model."
Comment Source:A full model fit to shortest component 0.4y.
The residual of the fit across the entire spectrum (Nyquist T = 1/6 y) appears to be flat white noise
No use trying to improve the fit beyond this point --
>>>>>"Bottom line: when the residuals fail to be white noise, a different model should be tried. Short answer regarding time series regression: If they are not white noise (i.e. they are not normal, not have zero mean or serially autocorrelated), then your model is not fully adequate. Therefore, you should revise your model."
Congratulations seem in order for all your ENSO work. It's given me lots of techniques to consider trying to use in other contexts. I hope the final paper gets published in GRL at least. So thanks.
Comment Source:Congratulations seem in order for all your ENSO work. It's given me lots of techniques to consider trying to use in other contexts. I hope the final paper gets published in GRL at least. So thanks.
I hope the final paper gets published in GRL at least.
Jim, Right now the research is going through peer-review and the outcome is still pending as I recently responded to the critical review.
I don't want to copy&paste the written criticisms here out of courtesy, but the reviewer said essentially that it's unproven research. Of course, that's not a great position to be put in, but I responded by saying that there are really no controlled experiments available to verify the model -- all one can do is self-consistency and sampled-data tests, which is the focus of the last year of effort on the model.
The criticism also cited stochastic resonance as more of a consensus model for ENSO. The evidence that it's not stochastic is that the cyclic forcing periods have to be set precisely to the known orbital periods. The lunar months have to be precise, otherwise the errors accumulate over the 130+ years of the data. If I let these parameters vary during the fit, they converge to the lunar months being within a second in precision and the tropical year within a minute of the known parameters. They also have to be shaped according to the non-linear interactions according to the known orbital ephemeris data and the \(1/R^3\) tidal forcing law.
Because of this finding and the fact that I can't reject that lunar and solar tidal forcing based on the excellent agreement found, I have to stand my ground. In other words, if the periods didn't match then I would have abandoned the effort long ago. So I ended up not changing anything but added a few more citations on re-submittal.
Comment Source:> I hope the final paper gets published in GRL at least.
Jim, Right now the research is going through peer-review and the outcome is still pending as I recently responded to the critical review.
I don't want to copy&paste the written criticisms here out of courtesy, but the reviewer said essentially that it's unproven research. Of course, that's not a great position to be put in, but I responded by saying that there are really no controlled experiments available to verify the model -- all one can do is self-consistency and sampled-data tests, which is the focus of the last year of effort on the model.
The criticism also cited stochastic resonance as more of a consensus model for ENSO. The evidence that it's not stochastic is that the cyclic forcing periods have to be set precisely to the known orbital periods. The lunar months have to be precise, otherwise the errors accumulate over the 130+ years of the data. If I let these parameters vary during the fit, they converge to the lunar months being within a second in precision and the tropical year within a minute of the known parameters. They also have to be shaped according to the non-linear interactions according to the known orbital ephemeris data and the \\(1/R^3\\) tidal forcing law.
Because of this finding and the fact that I can't reject that lunar and solar tidal forcing based on the excellent agreement found, I have to stand my ground. In other words, if the periods didn't match then I would have abandoned the effort long ago. So I ended up not changing anything but added a few more citations on re-submittal.
Yes. Yet as far as I can tell there is no real consensus for ENSO. The split is between stochastic models and chaotic models and both can only give a statistical agreement. The non-chaotic deterministic model is the odd-man out because no one is looking at that approach. Somehow it was rejected long ago, even though the repeat cycle for a full lunisolar forcing is 372 years! With that long a period, hard to tell the difference between a cyclic behavior and one that is chaotic or random.
IMO, a deterministic model was dismissed too soon and it became conventional wisdom not to look at it again.
Interesting that in regard to the 372 year repeat period, which is the same as the recurrence of eclipses on the same day of the tropical year:
Stockwell, John N. "On the law of recurrence of eclipses on the same day of the tropical year." The Astronomical Journal 15 (1895): 73-75.
Sure enough, the model calculation I use follows this cycle fairly closely. Consider that the graph below is a combination of the 3 lunar sine waves plotted on precisely 1 year intervals (which provides the same day of the tropical year)
You can detect the general 372 year repeat period from comparing the two blue arrow intervals, and there is another repeat period of 1230 years which you can see by shifting and aligning two intervals. Both of these repeat periods are subtle and it's not surprising that it's not picked up in any natural phenomena apart from something as striking as a eclipse, where the 372 year cycle has been long known.
When the \(1/R^3\) forcing and the ephemeris corrections to the sine waves is invoked, we can compare to the data and how it appears when it is shifted by 372 years.
It's not perfect because the 372 year period is not perfect, but it's enough to understand how the repeat period is preserved even after the perfect sine waves are individually distorted by the same base 4 frequencies.
Comment Source:Yes. Yet as far as I can tell there is no real consensus for ENSO. The split is between stochastic models and chaotic models and both can only give a statistical agreement. The non-chaotic deterministic model is the odd-man out because no one is looking at that approach. Somehow it was rejected long ago, even though the repeat cycle for a full lunisolar forcing is 372 years! With that long a period, hard to tell the difference between a cyclic behavior and one that is chaotic or random.
IMO, a deterministic model was dismissed too soon and it became conventional wisdom not to look at it again.
Interesting that in regard to the 372 year repeat period, which is the same as the recurrence of eclipses on the same day of the tropical year:
>Stockwell, John N. "On the law of recurrence of eclipses on the same day of the tropical year." The Astronomical Journal 15 (1895): 73-75.
Sure enough, the model calculation I use follows this cycle fairly closely. Consider that the graph below is a combination of the 3 lunar sine waves plotted on precisely 1 year intervals (which provides the same day of the tropical year)
You can detect the general 372 year repeat period from comparing the two blue arrow intervals, and there is another repeat period of 1230 years which you can see by shifting and aligning two intervals. Both of these repeat periods are subtle and it's not surprising that it's not picked up in any natural phenomena apart from something as striking as a eclipse, where the 372 year cycle has been long known.
When the \\(1/R^3\\) forcing and the ephemeris corrections to the sine waves is invoked, we can compare to the data and how it appears when it is shifted by 372 years.
It's not perfect because the 372 year period is not perfect, but it's enough to understand how the repeat period is preserved even after the perfect sine waves are individually distorted by the same base 4 frequencies.
The fit does a back-extrapolation over a 230 year interval prior to 1880 using coral proxy records with a resolution of 1 year. The temporal forcing pattern is very close to that used in the modern record but the scaling has to be modified, as it appears that the magnitude of ENSO excursions were measurably reduced prior to 1880.
The salient point is that trying to fit a curve as chaotic-appearing as an ENSO time-series record by applying only 4 interacting sinusoidal waves is difficult enough. But then to extend that to almost double the length -- from 135 years to an additional 230 years with what are essentially minor tweaks to the boundary conditions -- is next to impossible if it was a truly chaotic behavior.
Comment Source:The following analysis is an excellent substantiation of the ENSO model
http://contextearth.com/2018/05/08/unified-enso-proxy/
The fit does a back-extrapolation over a 230 year interval prior to 1880 using coral proxy records with a resolution of 1 year. The temporal forcing pattern is very close to that used in the modern record but the scaling has to be modified, as it appears that the magnitude of ENSO excursions were measurably reduced prior to 1880.
The salient point is that trying to fit a curve as chaotic-*appearing* as an ENSO time-series record by applying only 4 interacting sinusoidal waves is difficult enough. But then to extend that to almost double the length -- from 135 years to an additional 230 years with what are essentially minor tweaks to the boundary conditions -- is next to impossible if it was a truly chaotic behavior.
The solution to Laplace's Tidal Equation along the equator is
$$ I(t) = \Sigma^n_{i=0} A_i \sin (k_i f(t) + \theta_i) $$
where f(t) is the equatorial gravitational forcing, and the index terms are the separated standing wave components.
Since f(t) is cyclic as well, we want to determine what harmonics are generated by the sum. First, we isolate the phase factor and expand.
$$ I(t) = \Sigma A_i (\cos(\theta_i) \sin (k_i f(t)) + \sin(\theta_i) \cos(k_i f(t)) ) $$
The harmonics come about predominately from the Taylor's-series expansion of the sinusoidal factors.
$$ I(t) = \Sigma A_i (\cos(\theta_i) ( k_i f(t) - (k_i f(t))^3/3! + (k_i f(t))^5/5! + ...) + \sin(\theta_i) ( 1- (k_i f(t))^2/2! + (k_i f(t))^4/4! + ...) )$$
Notice that all the f(t) terms become power factors. This will essentially explode the power spectrum with multiplicative harmonics when there is significant saturation of the bounding sinusoidal terms. If the values are not close to saturated, only the first-order term remains, equivalent to the approximation
$$ sin(k f(t)) \sim k f(t) $$
The k pre-factors are related to the spatial standing-wave modes in the original derivation. The higher the value of k the greater the spatial wavenumber of the standing wave. Empirically, we observe one value of k that stands out, and that is likely the primary ENSO standing wave. The progressively higher values of k likely relate in some way to Tropical Instability Waves, which are thought to be related to ENSO according to a recent review article [1].
Suffice to say that this is very different math than we are used to applying for spectral analysis. How to optimize is a challenge. We can try one of these approaches:
Applying a nonlinear search of the k values, while also modifying f(t), to best correlate to the ENSO data
Group the terms of the same power factor and attempt a multi-linear regression to optimize values of k for a given f(t).
The first grinds away forever as it uses a gradient search. The second may be faster but it doesn't automatically adjust f(t). A combination of these may work best.
[1] Timmermann, Axel, Soon-Il An, Jong-Seong Kug, Fei-Fei Jin, Wenju Cai, Antonietta Capotondi, Kim Cobb, et al. “El Niño–Southern Oscillation Complexity.” Nature 559, no. 7715 (July 2018): 535. https://doi.org/10.1038/s41586-018-0252-6.
Comment Source:The solution to Laplace's Tidal Equation along the equator is
$$ I(t) = \Sigma^n_{i=0} A_i \sin (k_i f(t) + \theta_i) $$
where f(t) is the equatorial gravitational forcing, and the index terms are the separated standing wave components.
Since f(t) is cyclic as well, we want to determine what harmonics are generated by the sum. First, we isolate the phase factor and expand.
$$ I(t) = \Sigma A_i (\cos(\theta_i) \sin (k_i f(t)) + \sin(\theta_i) \cos(k_i f(t)) ) $$
The harmonics come about predominately from the Taylor's-series expansion of the sinusoidal factors.
$$ I(t) = \Sigma A_i (\cos(\theta_i) ( k_i f(t) - (k_i f(t))^3/3! + (k_i f(t))^5/5! + ...) + \sin(\theta_i) ( 1- (k_i f(t))^2/2! + (k_i f(t))^4/4! + ...) )$$
Notice that all the f(t) terms become power factors. This will essentially explode the power spectrum with multiplicative harmonics when there is significant saturation of the bounding sinusoidal terms. If the values are not close to saturated, only the first-order term remains, equivalent to the approximation
$$ sin(k f(t)) \sim k f(t) $$
The k pre-factors are related to the spatial standing-wave modes in the original derivation. The higher the value of k the greater the spatial wavenumber of the standing wave. Empirically, we observe one value of k that stands out, and that is likely the primary ENSO standing wave. The progressively higher values of k likely relate in some way to Tropical Instability Waves, which are thought to be related to ENSO according to a recent review article [1].
Suffice to say that this is very different math than we are used to applying for spectral analysis. How to optimize is a challenge. We can try one of these approaches:
* Applying a nonlinear search of the k values, while also modifying f(t), to best correlate to the ENSO data
* Group the terms of the same power factor and attempt a multi-linear regression to optimize values of k for a given f(t).
The first grinds away forever as it uses a gradient search. The second may be faster but it doesn't automatically adjust f(t). A combination of these may work best.
---
[1] Timmermann, Axel, Soon-Il An, Jong-Seong Kug, Fei-Fei Jin, Wenju Cai, Antonietta Capotondi, Kim Cobb, et al. “El Niño–Southern Oscillation Complexity.” Nature 559, no. 7715 (July 2018): 535. https://doi.org/10.1038/s41586-018-0252-6.
Comment Source:The key to analytically solving Laplace's tidal equations (Navier-Stokes) along the equator is to use the ansatz of a non-fixed equator. Solved [here](http://contextearth.com/compact-qbo-derivation/)
This is the rationale:
from "A Chorus of the Winds—On Saturn!" P. L. Read
https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2018JE005625
The basis of the ENSO model is the forcing derived from the long-period cyclic lunisolar gravitational pull of the moon and sun. There is some thought that ENSO shows teleconnections to other oceanic behaviors. The primary oceanic dipoles are ENSO and AMO for the Pacific and Atlantic. There is also the PDO for the mid-northern-latitude of the Pacific, which has a pattern distinct from ENSO. So the question is: Are these connected through interactions or do they possibly share a common-mode mechanism through the same lunisolar forcing mechanism?
Based on tidal behaviors, it is known that the gravitational pull varies geographically, so it would be understandable that ENSO, AMO, and PDO would demonstrate distinct time-series signatures. In checking this, you will find that the correlation coefficient between any two of these series is essentially zero, regardless of applied leads or lags. Yet the underlying component factors (the lunar Draconic, lunar Anomalistic, and solar modified terms) may potentially emerge with only slight variations in shape, with differences only in relative amplitude. This is straightforward to test by fitting the basic ENSO model to AMO and PDO by allowing the parameters to vary.
The following figure is the result of fitting the model to ENSO, AMO, and PDO and then comparing the constituent factors.
First, note that the same parametric model fits each of the time series arguably well. The Draconic factor underling both the ENSO and AMO model is almost perfectly aligned, indicated by the red starred graph, with excursions showing a CC above 0.99. All of the rest of the CC's in fact are above 0.6.
The upshot of this analysis is two-fold. First to consider how difficult it is to fit any one of these time series to a minimal set of periodically-forced signals. Secondly that the underlying signals are not that different in character, only that the combination in terms of a Laplace's tidal equation weighting are what couples them together via a common-mode mechanism. Thus, the teleconnection between these oceanic indices is likely an underlying common lunisolar tidal forcing, just as one would suspect from conventional tidal analysis.
Comment Source:The basis of the ENSO model is the forcing derived from the long-period cyclic lunisolar gravitational pull of the moon and sun. There is some thought that ENSO shows [teleconnections](https://en.wikipedia.org/wiki/Teleconnection) to other oceanic behaviors. The primary oceanic dipoles are ENSO and AMO for the Pacific and Atlantic. There is also the PDO for the mid-northern-latitude of the Pacific, which has a pattern distinct from ENSO. So the question is: Are these connected through interactions or do they possibly share a common-mode mechanism through the same lunisolar forcing mechanism?
Based on tidal behaviors, it is known that the gravitational pull varies geographically, so it would be understandable that ENSO, AMO, and PDO would demonstrate distinct time-series signatures. In checking this, you will find that the correlation coefficient between any two of these series is essentially zero, regardless of applied leads or lags. Yet the underlying component factors (the lunar Draconic, lunar Anomalistic, and solar modified terms) may *potentially* emerge with only slight variations in shape, with differences only in relative amplitude. This is straightforward to test by fitting the basic ENSO model to AMO and PDO by allowing the parameters to vary.
The following figure is the result of fitting the model to ENSO, AMO, and PDO and then comparing the constituent factors.
First, note that the same parametric model fits each of the time series arguably well. The Draconic factor underling both the ENSO and AMO model is almost perfectly aligned, indicated by the red starred graph, with excursions showing a CC above 0.99. All of the rest of the CC's in fact are above 0.6.
The upshot of this analysis is two-fold. First to consider how difficult it is to fit any one of these time series to a minimal set of periodically-forced signals. Secondly that the underlying signals are not that different in character, only that the combination in terms of a [Laplace's tidal equation weighting](https://agu.confex.com/agu/fm18/preliminaryview.cgi/Paper415478.html) are what couples them together via a common-mode mechanism. Thus, the teleconnection between these oceanic indices is likely an underlying common lunisolar tidal forcing, just as one would suspect from conventional tidal analysis.
Many of the recent advances in fundamental climate science have been made by researchers with a condensed matter physics background.
Brad Marston of Brown U and colleagues are making progress with their topologically-constrained climate models. Marston's presentations are very familiar to those of us that understand the energy band diagrams used in solid-state theory. Equatorial waves have similar dispersion relations to a typical gapped band structure. The topological confinement is very similar to the assumption I am using in solving Laplace's tidal equations (i.e. reduced Navier-Stokes) along the equator.
-- from Wikipedia entry on Topological insulators
John Wettlaufer of Yale U is also applying a similar approach
Wettlaufer: “There is a vast gulf, both conceptually and in terms of space and time scales, between simulations and idealized models. Attempts to reconcile them will have to focus on the problem of scales, a task well suited to physicists: The challenge of scale separation in both condensed matter and particle physics led to the development of the renormalization group, unifying concepts in previously disparate fields [5]. Renormalization group concepts and methods have been successfully applied to fluid dynamics problems [6,7], which are central to climate dynamics.”
This is an intriguing model fit feature in the paper
Michael Mann of Penn State also has a background in condensed matter physics.
Comment Source:Many of the recent advances in fundamental climate science have been made by researchers with a condensed matter physics background.
Brad Marston of Brown U and colleagues are making progress with their [topologically-constrained climate models](http://science.sciencemag.org/content/358/6366/1075). Marston's presentations are very familiar to those of us that understand the energy band diagrams used in solid-state theory. Equatorial waves have similar dispersion relations to a typical gapped band structure. The topological confinement is very similar to the assumption I am using in solving Laplace's tidal equations (i.e. reduced Navier-Stokes) along the equator.
-- from Wikipedia entry on Topological insulators
John Wettlaufer of Yale U is also applying a similar approach
> [Wettlaufer](https://journals.aps.org/prl/edannounce/10.1103/PhysRevLett.116.150002): “There is a vast gulf, both conceptually and in terms of space and time scales, between simulations and idealized models. Attempts to reconcile them will have to focus on the problem of scales, a task well suited to physicists: The challenge of scale separation in both condensed matter and particle physics led to the development of the renormalization group, unifying concepts in previously disparate fields [5]. Renormalization group concepts and methods have been successfully applied to fluid dynamics problems [6,7], which are central to climate dynamics.”
Wettlaufer has a [recent paper in PRL](https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.121.108701), but prior to that he published an interesting paper that clearly shows the impact of a strong annual impulse on the behavior of ENSO.
[A unified nonlinear stochastic time series analysis for climate science](https://www.nature.com/articles/srep44228)
This is an intriguing model fit feature in the paper
Michael Mann of Penn State also has a background in condensed matter physics.
"In the frequency domain cross validation is easily performed by dividing the frequency measurements in two disjoint sets : estimation data and validation data. [...] A model is then estimated using the estimation data only. The quality of the model is assessed by comparing the estimated transfer function with the validation data set and a proper model order can then be inferred."
If the sets are non-overlapping and the model fit to one set is able to predict the amplitude and phase spectrum of other set, there must be a fundamental process linking the profiles of the two frequency intervals. This could happen if the frequencies are correlated as is the case with harmonics. Consider modeling an unknown spectrum using square-wave components, but all the estimation data contained was high-harmonic frequency data. In this case, if the actual data was a square-wave, then the high-harmonic fit would naturally include the lower-frequency fundamental components. This would then generate a high-quality cross-validation check as the root model identifies the necessary fundamental implicitly.
That's a trivial and perhaps contrived example, but it also illustrates how this technique won't work on behaviors with independent signal components, but remains effective only if modulation occurs. For example, it would be impossible to predict that a 60 Hz hum existed in a signal if all you could process was frequency components of 1 kHz and above. That would be possible only if some behavior in the model forced it to simultaneously occur both in the low and high bandwidth through some cooperative interaction. For example, it might happen through some sort of nonlinear interaction or frequency modulation, but the model would have to account for that. In fact, this is why FM or AM demodulation works -- even though the signals measured are embedded, i.e. non-linearly mixed within a high-frequency carrier, the model is able to extract the low-frequency signal. The "validation" that the model works is that a high-fidelity tune is recognizable when played on a radio. Our brain is sophisticated enough that it is able to recognize the tune, otherwise something like Shazam could do the identification.
With ENSO, the nonlinear mixing is due to the alternately-annual (biennial) modulation of the cyclic lunar gravitational tidal pull. A biennial modulation is the mixing factor that allows a frequency-domain cross-validation to work. We simply need to tune the tidal factors to fit to the frequency spectrum of one interval (the estimation part), and then check to see if they also pop out on the orthogonal part of the frequency spectrum while showing a high correlation with the modeled results (the validation part).
The only way this approach would fail as a validation is if the extra degree of freedom (DOF) afforded by the biennial modulation somehow improved the correlation better than any other modulation we pick (such as triennial, biannual, etc). However, this is wildly implausible, as it is well-known that a strong biennial modulation is operational in the Pacific ocean's dynamics. In other words, this is a very weak DOF and functions more as a constraint than a free parameter. Like the tidal pull (which certainly exists with strength debateable), the biennial modulation is highly likely to exist. It's just a matter of pulling the signal out from any other noise that might exist.
Two other elements are necessary to do the FDCV, (1) the solution to Laplace's tidal equations and (2) taking the derivative of the ENSO data to equalize the ENSO frequency spectrum, thus allowing for a balanced orthogonal interval comparison.
The first training interval uses only frequency components between 0.5 /year and 1 /year in amplitude. The fit is extremely aggressive, reaching a correlation coefficient of 0.99, yet the validation interval appears to match.
The second training interval works the complement.
These are the real-space fits corresponding to the frequency-space fits
Comment Source:There is a neat technique that can validate a certain class of nonlinear models called [frequency-domain cross-validation](http://www.diva-portal.org/smash/get/diva2:315809/FULLTEXT02) (FDCV)
> "In the frequency domain cross validation is easily performed by dividing the frequency measurements in two disjoint sets : estimation data and validation data. [...] A model is then estimated using the estimation data only. The quality of the model is assessed by comparing the estimated transfer function with the validation data set and a proper model order can then be inferred."
If the sets are non-overlapping and the model fit to one set is able to predict the amplitude and phase spectrum of other set, there must be a fundamental process linking the profiles of the two frequency intervals. This could happen if the frequencies are correlated as is the case with harmonics. Consider modeling an unknown spectrum using square-wave components, but all the estimation data contained was high-harmonic frequency data. In this case, if the *actual* data was a square-wave, then the high-harmonic fit would naturally include the lower-frequency fundamental components. This would then generate a high-quality cross-validation check as the root model identifies the necessary fundamental implicitly.
That's a trivial and perhaps contrived example, but it also illustrates how this technique won't work on behaviors with independent signal components, but remains effective only if modulation occurs. For example, it would be impossible to predict that a 60 Hz hum existed in a signal if all you could process was frequency components of 1 kHz and above. That would be possible only if some behavior in the model forced it to simultaneously occur both in the low and high bandwidth through some cooperative interaction. For example, it might happen through some sort of nonlinear interaction or frequency modulation, but the model would have to account for that. In fact, this is why FM or AM demodulation works -- even though the signals measured are embedded, i.e. non-linearly mixed within a high-frequency carrier, the model is able to extract the low-frequency signal. The "validation" that the model works is that a high-fidelity tune is recognizable when played on a radio. Our brain is sophisticated enough that it is able to recognize the tune, otherwise something like Shazam could do the identification.
With ENSO, the nonlinear mixing is due to the alternately-annual (biennial) modulation of the cyclic lunar gravitational tidal pull. A biennial modulation is the mixing factor that allows a frequency-domain cross-validation to work. We simply need to tune the tidal factors to fit to the frequency spectrum of one interval (the estimation part), and then check to see if they also pop out on the orthogonal part of the frequency spectrum while showing a high correlation with the modeled results (the validation part).
The only way this approach would fail as a validation is if the extra degree of freedom (DOF) afforded by the biennial modulation somehow improved the correlation better than any other modulation we pick (such as triennial, biannual, etc). However, this is wildly implausible, as it is well-known that a strong biennial modulation is operational in the Pacific ocean's dynamics. In other words, this is a very weak DOF and functions more as a constraint than a free parameter. Like the tidal pull (which **certainly** exists with strength debateable), the biennial modulation is highly likely to exist. It's just a matter of pulling the signal out from any other noise that might exist.
Two other elements are necessary to do the FDCV, (1) the solution to Laplace's tidal equations and (2) taking the derivative of the ENSO data to equalize the ENSO frequency spectrum, thus allowing for a balanced orthogonal interval comparison.
The first training interval uses only frequency components between 0.5 /year and 1 /year in amplitude. The fit is extremely aggressive, reaching a correlation coefficient of 0.99, yet the validation interval appears to match.
The second training interval works the complement.
These are the real-space fits corresponding to the frequency-space fits
I had been using monthly data and it wasn't clear whether the high wave-number (high-K) amplitudes used to fit the monthly time-series would also be apparent on a daily basis. Keeping the filtering to a minimum, the upper panel shows the monthly SOI time-series, applying a fit from 1992 onward. The bottom panel shows what it looks like on a magnified daily time scale.
The recent paper by Jajcay (cite below) claims that there is synchronization across time scales for ENSO, which is implicit for the ENSO model used above. For our ENSO model, the low -K solutions (low frequency, corresponding to the main Tahiti-Darwin standing wave dipole) are automatically synchronized with the high-K solutions (high frequency, which are likely related to Tropical Instability Waves and/or Madden-Julian Oscillations). The main dipole requires both the low-K and high-K solutions to achieve the sharp jagged profile -- constructed through the superposition of the non-linear harmonics -- which is clearly apparent when one expands to the daily time-scale. For example, the strong 1998 El-Nino is actually composed of the superposition of a longer-time-period signal superposed with multiple higher-frequency jagged peaks.
Comment Source:There is SOI data on a daily time scale since 1992 on this site https://data.longpaddock.qld.gov.au/SeasonalClimateOutlook/SouthernOscillationIndex/SOIDataFiles/DailySOI1887-1989Base.txt
I had been using monthly data and it wasn't clear whether the high wave-number (high-K) amplitudes used to fit the monthly time-series would also be apparent on a daily basis. Keeping the filtering to a minimum, the upper panel shows the monthly SOI time-series, applying a fit from 1992 onward. The bottom panel shows what it looks like on a magnified daily time scale.
The recent paper by Jajcay (cite below) claims that there is synchronization across time scales for ENSO, which is implicit for the ENSO model used above. For our ENSO model, the low -K solutions (low frequency, corresponding to the main Tahiti-Darwin standing wave dipole) are automatically synchronized with the high-K solutions (high frequency, which are likely related to Tropical Instability Waves and/or Madden-Julian Oscillations). The main dipole requires both the low-K and high-K solutions to achieve the sharp jagged profile -- constructed through the superposition of the non-linear harmonics -- which is clearly apparent when one expands to the daily time-scale. For example, the strong 1998 El-Nino is actually composed of the superposition of a longer-time-period signal superposed with multiple higher-frequency jagged peaks.
> Jajcay, Nikola, et al. ["Synchronization and causality across time scales in El Niño Southern Oscillation."](https://www.nature.com/articles/s41612-018-0043-7) npj Climate and Atmospheric Science 1.1 (2018): 33.
Fitting the model simultaneously to the historical monthly data (top panel) and high-resolution daily data (middle panel) using the high-frequency spectrum correlation (bottom panel) to weight the high-K parameters.
Comment Source:
Fitting the model simultaneously to the historical monthly data (top panel) and high-resolution daily data (middle panel) using the high-frequency spectrum correlation (bottom panel) to weight the high-K parameters.
Comments
Couple of more blog posts on nailing down the mechanism for ENSO:
(1) This one contextualizes the ENSO behavior in terms of a common forcing governing ENSO, QBO, and the Chandler wobble
http://contextearth.com/2017/05/21/the-lunar-geophysical-connection/
Here are a few charts from the post:
What are the odds that the fundamental frequencies of all these behaviors are the same to that precision?
As it turns out NASA JPL were on this lunar-forcing path several years ago, but elected not to fund the proposed research and so the progress stalled.
(2) Yesterday's post provides a historical context. Essentially all the geophysics applicable to the model was known by 1920.
http://contextearth.com/2017/05/30/ocean-dynamics-history/
Could have done a decent job in predicting ENSO evolution with Pacific Ocean SST data up to 1920
(3) Posted on the Azimuth blog.
https://johncarlosbaez.wordpress.com/2017/04/24/complexity-theory-and-evolution-in-economics/#comment-92343
ENSO means zero in Japanese and also has a Zen interpretation
Couple of more blog posts on nailing down the mechanism for ENSO: --- --- **(1)** This one contextualizes the ENSO behavior in terms of a common forcing governing ENSO, QBO, and the Chandler wobble http://contextearth.com/2017/05/21/the-lunar-geophysical-connection/ Here are a few charts from the post:    What are the odds that the fundamental frequencies of all these behaviors are the same to that precision?  As it turns out NASA JPL were on this lunar-forcing path several years ago, but elected not to fund the proposed research and so the progress stalled. --- --- **(2)** Yesterday's post provides a historical context. Essentially all the geophysics applicable to the model was known by 1920. http://contextearth.com/2017/05/30/ocean-dynamics-history/ Could have done a decent job in predicting ENSO evolution with Pacific Ocean SST data up to 1920  --- --- **(3)** Posted on the Azimuth blog. https://johncarlosbaez.wordpress.com/2017/04/24/complexity-theory-and-evolution-in-economics/#comment-92343 ENSO means zero in Japanese and also has a Zen interpretation > “The ensō symbolizes absolute enlightenment, strength, elegance, the universe, and mu (the void).”Trump pulled US out of the Paris Accord :(
If we actually had better knowledge of climate behavior and more emphasis on how AGW interacts with Peak Oil, we would likely have a different discourse. Just by having a real understanding of ENSO, we could compensate out the natural variability in the global temperature signal. The almost monotonic increase in temperature would be much more obvious and we wouldn't have to contend with the uncertainty players such as Curry controlling the political discussion.
This ENSO model is looking more solid. With virtually no free parameters, any automated fit to an ENSO interval does a good job of recreating the unfitted intervals. http://contextearth.com/2017/05/31/enso-model-fit-1880-1980/
The common criticism of these kinds of models is that they overfit and can not be tested with historical data, as any contamination of a model with available data will then taint the model and give a false impression that it actually works. This model is different and works with minimal set of parameters, just the 2 tidal cycles and a seasonal cycle.
Trump pulled US out of the Paris Accord :( If we actually had better knowledge of climate behavior and more emphasis on how AGW interacts with Peak Oil, we would likely have a different discourse. Just by having a real understanding of ENSO, we could compensate out the natural variability in the global temperature signal. The almost monotonic increase in temperature would be much more obvious and we wouldn't have to contend with the uncertainty players such as Curry controlling the political discussion. This ENSO model is looking more solid. With virtually no free parameters, any automated fit to an ENSO interval does a good job of recreating the unfitted intervals. http://contextearth.com/2017/05/31/enso-model-fit-1880-1980/  The common criticism of these kinds of models is that they overfit and can not be tested with historical data, as any contamination of a model with available data will then taint the model and give a false impression that it actually works. This model is different and works with minimal set of parameters, just the 2 tidal cycles and a seasonal cycle.Couple more blog posts that demonstrate how well the ENSO model works in describing the observations and the geophysics ...
The first is evidence as to how such a simple model can produce such a rich Fourier spectra, contrary to people's preconceived notions: http://contextearth.com/2017/06/02/enso-and-fourier-analysis/
The second is an untainted match between the lunar forcing used to model ENSO and the lunar forcing obtained from the earth's Length-of-day (LOD) data http://contextearth.com/2017/06/03/enso-forcing-validation-via-lod-data/
Have to look at Fig.6 in the post to see the match closely.
This is untainted because the LOD is completely out-of-band with the fitting data and so becomes a strong validation test for the model.
Couple more blog posts that demonstrate how well the ENSO model works in describing the observations and the geophysics ... The first is evidence as to how such a simple model can produce such a rich Fourier spectra, contrary to people's preconceived notions: http://contextearth.com/2017/06/02/enso-and-fourier-analysis/ The second is an untainted match between the lunar forcing used to model ENSO and the lunar forcing obtained from the earth's Length-of-day (LOD) data http://contextearth.com/2017/06/03/enso-forcing-validation-via-lod-data/ Have to look at Fig.6 in the post to see the match closely.  This is untainted because the LOD is completely [out-of-band](https://en.wikipedia.org/wiki/Out-of-band) with the fitting data and so becomes a strong validation test for the model.ENSO forcing match against digitized LOD variation
I took the correlation coefficient of this and its above 0.7. For the cycle factors applied, the fit doesn't get much better if the phases and amplitude are allowed to vary -- the correlation coefficient go up by a slight 0.02, and it reduces the ENSO fit only slightly.
ENSO forcing match against digitized LOD variation  I took the correlation coefficient of this and its above 0.7. For the cycle factors applied, the fit doesn't get much better if the phases and amplitude are allowed to vary -- the correlation coefficient go up by a slight 0.02, and it reduces the ENSO fit only slightly.From the above agreement in forcing stimulii.
(A)The forcing for QBO is mainly Draconic
(B)(C) ENSO and LOD have the same tidal forcing
From the above agreement in forcing stimulii. (A)The forcing for QBO is mainly Draconic (B)(C) ENSO and LOD have the same tidal forcing Difficult to believe that behaviors such as ENSO and QBO are not related to external forcing. I can't think of one large scale cyclic behavior that can't be pinned to some other regular cycle. Even the cycles of sunspots are known to be intimately tied to the sun's rotation. So even though they haven't quite nailed the predictability of sunspots yet, they know it isn't some spontaneous oscillation as the purveyors of the wind-only mechanism for ENSO seem to think.
Thus, much like sunspots, ENSO is likely sensitive to variations in the Earth's rotation speed. As the moon is known to cause cyclic variations in the speed, these same variations should be able to be picked up in an ENSO wave equation model. And what do we find but that the two most critical lunar periods, the Draconic 27.2122 days and Anomalistic 27.5545 days feed into a best-fit model to within 1 minute each.
http://contextearth.com/2017/06/08/scaling-el-nino/
Got a reply tweet from Andrew Dessler concerning this and he said "Climate is a physics problem, not a statistics one. Looking at correlations is interesting, but not sufficient. Must have physical basis."
Some of these guys do not realize that science deals with this situation automatically. They should be able to eventually reject the lunar forcing by coming up with evidence that rejects it. It shouldn't be hard, as all they have to do is show that the ENSO cycles are incommensurate with the lunar cycles. And show how there is not enough energy supplied by the lunisolar cycles to move volumes of water in a reduced effective gravity environment. If they can't, however, then the lunar model will remain as a potential ENSO driver.
Difficult to believe that behaviors such as ENSO and QBO are not related to external forcing. I can't think of one large scale cyclic behavior that can't be pinned to some other regular cycle. Even the cycles of sunspots are known to be intimately tied to the sun's rotation. So even though they haven't quite nailed the predictability of sunspots yet, they know it isn't some spontaneous oscillation as the purveyors of the wind-only mechanism for ENSO seem to think. Thus, much like sunspots, ENSO is likely sensitive to variations in the Earth's rotation speed. As the moon is known to cause cyclic variations in the speed, these same variations should be able to be picked up in an ENSO wave equation model. And what do we find but that the two most critical lunar periods, the Draconic 27.2122 days and Anomalistic 27.5545 days feed into a best-fit model to within 1 minute each. http://contextearth.com/2017/06/08/scaling-el-nino/ Got a reply tweet from Andrew Dessler concerning this and he said *"Climate is a physics problem, not a statistics one. Looking at correlations is interesting, but not sufficient. Must have physical basis."* Some of these guys do not realize that science deals with this situation automatically. They should be able to eventually reject the lunar forcing by coming up with evidence that rejects it. It shouldn't be hard, as all they have to do is show that the ENSO cycles are incommensurate with the lunar cycles. And show how there is not enough energy supplied by the lunisolar cycles to move volumes of water in a reduced effective gravity environment. If they can't, however, then the lunar model will remain as a potential ENSO driver.This is a magnification of the fitting contour around the best forcing period values for ENSO. These pair of peak values are each found to be less than a minute apart from the known values of the Draconic cycle (27.2122 days) and Anomalistic cycle (27.5545 days).
The forcing comes directly from the angular momentum variations in the Earth's rotation. The comparison between what the ENSO model uses (from the Draconic and Anomalistic terms above) and what is measured via monitoring the length-of-day (LOD) is shown below
The lower LOD pane is a fit over 3 years, which is about 40 lunar months. These essentially get aliased in the upper ENSO pane, which only responds to the peak tidal forces at a specific time of the year -- around Nov/Dec.
So many numbers have to align perfectly for this model to work out, and it looks like it does.
This is a magnification of the fitting contour around the best forcing period values for ENSO. These pair of peak values are each found to be less than a minute apart from the known values of the Draconic cycle (27.2122 days) and Anomalistic cycle (27.5545 days).  The forcing comes directly from the angular momentum variations in the Earth's rotation. The comparison between what the ENSO model uses (from the Draconic and Anomalistic terms above) and what is measured via monitoring the length-of-day (LOD) is shown below  The lower LOD pane is a fit over 3 years, which is about 40 lunar months. These essentially get aliased in the upper ENSO pane, which only responds to the peak tidal forces at a specific time of the year -- around Nov/Dec. So many numbers have to align perfectly for this model to work out, and it looks like it does.This is the physics of the tidal forcing -- imparting a 1 millisecond slowdown (or speedup) on the rotation of the earth with a surface velocity of almost 500 meters/second over the course of a couple of weeks (a fortnight) will result in an inertial lateral movement of ~ 1/2 a meter in the volume of the Pacific ocean due to Newton's first law.
This does not seem like a big deal until you realize that the thermocline can absorb this inertial impulse as a vertical sloshing, since the effective gravity is reduced by orders of magnitude due to the slight density differences above and below the thermocline. This is reflected as an Atwood number and shows up in Rayleigh-Taylor instability experiments, e.g. SEE THIS PAPER
With an Atwood number less than 0.001 which is ~0.1% density differences in a stratified fluid, the 0.5 meter displacement that occurs over two weeks now occurs effectively over half an hour. That's just an elementary scaling exercise.
So intuitively, one has to ask the question of what would happen if the ocean was translated laterally by 1/2 a meter over the course of a 1/2 an hour? We know what happens with earthquakes in something as simple as a swimming pool
https://youtu.be/27GMnYEWL0M
or as threatening as a tsunami. But this is much more subtle because we can't obviously see it, and why it has likely been overlooked as a driver of ENSO.
All that math modeling of ENSO described here works backwards to this point. The actual forcing working on the earth's rotation can lead to the response shown here, both in the dynamic sense of tracing the measured path and now in terms of a physical order-of-magnitude justification.
This is the physics of the tidal forcing -- imparting a 1 millisecond slowdown (or speedup) on the rotation of the earth with a surface velocity of almost 500 meters/second over the course of a couple of weeks (a fortnight) will result in an inertial lateral movement of ~ 1/2 a meter in the volume of the Pacific ocean due to Newton's first law. This does not seem like a big deal until you realize that the thermocline can absorb this inertial impulse as a vertical sloshing, since the effective gravity is reduced by orders of magnitude due to the slight density differences above and below the thermocline. This is reflected as an Atwood number and shows up in Rayleigh-Taylor instability experiments, e.g. [SEE THIS PAPER](http://rsta.royalsocietypublishing.org/content/roypta/368/1916/1663.full.pdf) With an Atwood number less than 0.001 which is ~0.1% density differences in a stratified fluid, the 0.5 meter displacement that occurs over two weeks now occurs effectively over half an hour. That's just an elementary scaling exercise. So intuitively, one has to ask the question of what would happen if the ocean was translated laterally by 1/2 a meter over the course of a 1/2 an hour? We know what happens with earthquakes in something as simple as a swimming pool https://youtu.be/27GMnYEWL0M or as threatening as a tsunami. But this is much more subtle because we can't obviously see it, and why it has likely been overlooked as a driver of ENSO. All that math modeling of ENSO described here works backwards to this point. The *actual forcing* working on the earth's rotation can lead to the response shown here, both in the dynamic sense of tracing the measured path and now in terms of a physical order-of-magnitude justification.The supposedly simplest "toy" models of ENSO that we describe on the Azimuth Project wiki page here http://www.azimuthproject.org/azimuth/show/ENSO are the ones that remarkably work the best to describe the actual dynamics. If the delayed action oscillator (minus the cubic term) is combined with a seasonally-modulated lunar forcing that's essentially all that is needed to train the model.
The supposedly simplest "toy" models of ENSO that we describe on the Azimuth Project wiki page here http://www.azimuthproject.org/azimuth/show/ENSO are the ones that remarkably work the best to describe the actual dynamics. If the delayed action oscillator (minus the cubic term) is combined with a seasonally-modulated lunar forcing that's essentially all that is needed to train the model.My last comment:
Elaborated further here: http://contextearth.com/2017/06/23/ensoqbo-elevator-pitch/
My last comment: > "The supposedly simplest "toy" models of ENSO that we describe on the Azimuth Project wiki page here http://www.azimuthproject.org/azimuth/show/ENSO are the ones that remarkably work the best to describe the actual dynamics. If the delayed action oscillator (minus the cubic term) is combined with a seasonally-modulated lunar forcing that's essentially all that is needed to train the model." Elaborated further here: http://contextearth.com/2017/06/23/ensoqbo-elevator-pitch/tweet
http://contextearth.com/2017/08/14/solar-eclipse-2017-what-else/
tweet <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Because lunar & solar cycles so accurately known, we can predict <a href="https://twitter.com/hashtag/SolarEclipse2017?src=hash">#SolarEclipse2017</a> precisely. Same for <a href="https://twitter.com/hashtag/ENSO?src=hash">#ENSO</a> <a href="https://twitter.com/hashtag/ElNino?src=hash">#ElNino</a> <br> <a href="https://t.co/M8xJ3DwOso">https://t.co/M8xJ3DwOso</a></p>— Paul Pukite (@WHUT) <a href="https://twitter.com/WHUT/status/896857059366494208">August 13, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> http://contextearth.com/2017/08/14/solar-eclipse-2017-what-else/Thought to comment on what a long strange trip it's been. The journey to modeling ENSO and QBO has been circuitous and then essentially doubled back to the most basic kind of forcing and the simplest toy differential equations.
The ENSO behavior is modeled as 2 lunar tidal signals and an annual forcing impulse applied to a delay differential equation of 1 year delay. Could have started with this premise from day one, but nothing in the research literature indicated lunar forcing had any effect on ENSO.
Same goes for QBO except that it is essentially a single lunar tidal signal and a bi-annual seasonal forcing signal - one impulse per nodal crossing. Lindzen had considered lunar forcing early but apparently couldn't find any correlation and that's why no one followed up there.
Looking back my first blog post on this topic was early 2014, so it's been almost 4 years of spare-time effort. And even though this was anticipated to be a software coding project, the model is simple enough to express on a spreadsheet without the need for any macros or scripts except for a standard Solver plugin. It's essentially a little more complex than a basic tidal analysis program.
http://ContextEarth.com
Thought to comment on what a long strange trip it's been. The journey to modeling ENSO and QBO has been circuitous and then essentially doubled back to the most basic kind of forcing and the simplest toy differential equations. The ENSO behavior is modeled as 2 lunar tidal signals and an annual forcing impulse applied to a delay differential equation of 1 year delay. Could have started with this premise from day one, but nothing in the research literature indicated lunar forcing had any effect on ENSO. Same goes for QBO except that it is essentially a single lunar tidal signal and a bi-annual seasonal forcing signal - one impulse per nodal crossing. Lindzen had considered lunar forcing early but apparently couldn't find any correlation and that's why no one followed up there. Looking back my first blog post on this topic was early 2014, so it's been almost 4 years of spare-time effort. And even though this was anticipated to be a software coding project, the model is simple enough to express on a spreadsheet without the need for any macros or scripts except for a standard Solver plugin. It's essentially a little more complex than a basic tidal analysis program. http://ContextEarth.comCan we look forward to a publication submission or arXiv addition, @WebHubTel?
Can we look forward to a publication submission or arXiv addition, @WebHubTel?Have submitted a presentation to the AGU this December.
Already have an arXiv paper in place but need to update it.
Have submitted a presentation to the AGU this December. Already have an arXiv paper in place but need to update it.+1. Congratulations. I've much enjoyed the epic and learned a lot. Btw what solver did you use?
+1. Congratulations. I've much enjoyed the epic and learned a lot. Btw what solver did you use?Jim asked:
I used the builtin Excel Solver.
There is also an open Solver that I have yet to try: http://opensolver.org/
For awhile, I was using a Mathematica solver but it doesn't allow a correlation coefficient as a goal, only least squares. I think there is some property of using a correlation coefficient that allows a solver to avoid getting stuck in local minima. It may have to do with not having to worry about scaling at every step. The rescaling can always get done at the end.
There is also the Eureqa solver that I used early on, but that tool got bought out by another company and not sure of it's status. Eureqa allowed correlation coefficient and a whole range of optimization targets, including an interesting hybrid cc+leastSquares target. The architects of that tool understood the importance of providing different targets for optimization.
Probably should try using R and one of the solvers there, but I am in a mode of staying with what works for now
Jim asked: > "Btw what solver did you use?" I used the builtin [Excel Solver](https://support.office.com/en-us/article/Define-and-solve-a-problem-by-using-Solver-9ed03c9f-7caf-4d99-bb6d-078f96d1652c). There is also an open Solver that I have yet to try: http://opensolver.org/ For awhile, I was using a Mathematica solver but it doesn't allow a correlation coefficient as a goal, only least squares. I think there is some property of using a correlation coefficient that allows a solver to avoid getting stuck in local minima. It may have to do with not having to worry about scaling at every step. The rescaling can always get done at the end. There is also the Eureqa solver that I used early on, but that tool got bought out by another company and not sure of it's status. Eureqa allowed correlation coefficient and a whole range of optimization targets, including an interesting hybrid cc+leastSquares target. The architects of that tool understood the importance of providing different targets for optimization. Probably should try using R and one of the solvers there, but I am in a mode of staying with what works for nowThanks for the opensolver.org link.
Thanks for the opensolver.org link.Jim, If you use it, let me know how it works. I am also going for minimizing the barrier to usage.
Latest validation here http://contextearth.com/2017/09/27/enso-tidal-forcing-validated-by-lod-data/
Jim, If you use it, let me know how it works. I am also going for minimizing the barrier to usage. Latest validation here http://contextearth.com/2017/09/27/enso-tidal-forcing-validated-by-lod-data/ So far I've failed to load an .xlsx file of NOAA sesimic data into the Google sheets opensolver plugin version. I'll have to try with whatever the openoffice spreadsheet is called. Opensolver needs chromium which won't run on my currently, semi-borked setup.
So far I've failed to load an .xlsx file of NOAA sesimic data into the Google sheets opensolver plugin version. I'll have to try with whatever the openoffice spreadsheet is called. Opensolver needs chromium which won't run on my currently, semi-borked setup.After spending all that time with the ENSO model, this is the progress after a day of working the Atlantic version of ENSO, the AMO
http://contextearth.com/2017/10/03/amo/
After spending all that time with the ENSO model, this is the progress after a day of working the Atlantic version of ENSO, the AMO http://contextearth.com/2017/10/03/amo/Quantum and ENSO connects!
Full paper: Topological origins of equatorial waves
Oh, JohnB's gonna be writing about this for months!
There's also a supplement and two movies, S1 and S2.
Naw, I don't understand with any depth whatsoever, but I think it's very cool.
[Quantum and ENSO connects](http://www.sciencemag.org/news/2017/10/waves-drive-global-weather-patterns-finally-explained-thanks-inspiration-bagel-shaped)! [Full paper: _Topological_ _origins_ _of_ _equatorial_ _waves_](http://science.sciencemag.org/content/early/2017/10/04/science.aan8819.full) Oh, JohnB's gonna be writing about this for _months_! There's also [a supplement](http://science.sciencemag.org/highwire/filestream/700023/field_highwire_adjunct_files/0/aan8819_Delplace_SM.pdf) and two movies, [S1](http://science.sciencemag.org/highwire/filestream/700023/field_highwire_adjunct_files/1/aan8819s1.mov) and [S2](http://science.sciencemag.org/highwire/filestream/700023/field_highwire_adjunct_files/2/aan8819s2.mov). _Naw_, I don't understand with any depth whatsoever, but I think it's very cool.Thanks Jan, that's looks a lot like what I solved for last year -- reducing the Coriolis forces at the equator.
http://contextEarth.com/2016/09/23/compact-qbo-derviation/
I simplified much more than what they did.
Thanks Jan, that's looks a lot like what I solved for last year -- reducing the Coriolis forces at the equator. http://contextEarth.com/2016/09/23/compact-qbo-derviation/ I simplified much more than what they did. @WebHubTel, a letter to Science is in order!
@WebHubTel, a letter to _Science_ is in order!Jan, Perhaps I can try that.
Is the QBO a Berry monopole?
Or is it related to a Weyl point across a Lifshitz transitiion? https://inspirehep.net/record/1441222/plots
Yes, John will be interested in this stuff because it looks like the anti-vortex stuff that he had written about last year
https://johncarlosbaez.wordpress.com/2016/10/07/kosterlitz-thouless-transition/
I made a comment at the time:
A few years ago, I referenced this paper by Marston, which was a call-to-arms to solving climate problems:
https://physics.aps.org/articles/v4/20
And this paper by Vallis is a good inspiration to look at simplifying the physics before doing CFD
http://contextearth.com/2016/09/03/geophysical-fluid-dynamics-first-and-then-cfd/
https://youtu.be/JBY5iIfPgd0
Jan, Perhaps I can try that. Is the QBO a Berry monopole? Or is it related to a Weyl point across a Lifshitz transitiion? https://inspirehep.net/record/1441222/plots  Yes, John will be interested in this stuff because it looks like the anti-vortex stuff that he had written about last year https://johncarlosbaez.wordpress.com/2016/10/07/kosterlitz-thouless-transition/ I made a comment at the time: > "These curl equations are fascinating and are of course endemic in applications from electromagnetics to fluid dynamics. Perhaps there is some overlap with the model of the QBO equatorial winds that we are working on at John’s Azimuth Forum (see sidebar) and at my blog. Have some notes here: http://contextearth.com/2016/09/23/compact-qbo-derviation/#comment-199906 " --- A few years ago, I referenced this paper by Marston, which was a call-to-arms to solving climate problems: https://physics.aps.org/articles/v4/20 And this paper by Vallis is a good inspiration to look at simplifying the physics before doing CFD http://contextearth.com/2016/09/03/geophysical-fluid-dynamics-first-and-then-cfd/ --- https://youtu.be/JBY5iIfPgd0This is probably the best preliminary paper on the topic
ELASTIC WAVE EQUATION, Yves Colin de Verdière, Séminaire de théorie spectrale et géométrie, Grenoble Volume25 (2006-2007) 55-69
http://tsg.cedram.org/cedram-bin/article/TSG_2006-2007__25__55_0.pdf
This is probably the best preliminary paper on the topic ELASTIC WAVE EQUATION, Yves Colin de Verdière, Séminaire de théorie spectrale et géométrie, Grenoble Volume25 (2006-2007) 55-69 http://tsg.cedram.org/cedram-bin/article/TSG_2006-2007__25__55_0.pdfRe: Topological origins of equatorial waves
Blog post here on this paper: http://contextEarth.com/2017/10/13/interface-inflection-geophysics/
Re: Topological origins of equatorial waves Blog post here on this paper: http://contextEarth.com/2017/10/13/interface-inflection-geophysics/Couple of recent posts where the harmonic series approximation for the ENSO forcing is reduced to a closed-form expression:
http://contextearth.com/2017/10/27/reverse-engineering-the-moons-orbit-from-enso-behavior/
http://contextearth.com/2017/11/03/approximating-the-enso-forcing-potential/
This simplification was also applied to QBO
These are timely findings, as the presentation was accepted to the AGU next month
GC41B-1022: Biennial-Aligned Lunisolar-Forcing of ENSO: Implications for Simplified Climate Models
Check the sub-title :)
Couple of recent posts where the harmonic series approximation for the ENSO forcing is reduced to a closed-form expression: http://contextearth.com/2017/10/27/reverse-engineering-the-moons-orbit-from-enso-behavior/ http://contextearth.com/2017/11/03/approximating-the-enso-forcing-potential/ This simplification was also applied to QBO  These are timely findings, as the presentation was accepted to the AGU next month [GC41B-1022: Biennial-Aligned Lunisolar-Forcing of ENSO: Implications for Simplified Climate Models](https://agu.confex.com/agu/fm17/meetingapp.cgi/Paper/221914)  Check the sub-title :)Presentations at this week's AGU meeting on ENSO, QBO, and other stuff
http://contextearth.com/2017/12/11/agu-2017-posters/
Presentations at this week's AGU meeting on ENSO, QBO, and other stuff http://contextearth.com/2017/12/11/agu-2017-posters/A rare paper on machine learning for El Nino Using Network Theory and Machine Learning to predict El Nino, Peter Nooteboom https://dspace.library.uu.nl/bitstream/handle/1874/353201/Thesis_Peter_Nooteboom.pdf
Nothing truly impressive, the paper is reporting on short-term predictions
A rare paper on machine learning for El Nino Using Network Theory and Machine Learning to predict El Nino, Peter Nooteboom https://dspace.library.uu.nl/bitstream/handle/1874/353201/Thesis_Peter_Nooteboom.pdf Nothing truly impressive, the paper is reporting on short-term predictionsapplying chiral amplitude mapping
applying chiral amplitude mapping In the last month, two of the great citizen scientists that I will be forever personally grateful for have passed away. If anyone has followed climate science discussions on blogs and social media, you probably have seen their contributions.
Keith Pickering was an expert on computer science, astrophysics, energy, and history from my neck of the woods in Minnesota. He helped me so much in working out orbital calculations when I was first looking at lunar correlations. He provided source code that he developed and it was a great help to get up to speed. He was always there to tweet any progress made. Thanks Keith
Kevin O'Neill was a metrologist and an analysis whiz from Wisconsin. In the weeks before he passed, he told me that he had extra free time to help out with ENSO analysis. He wanted to use his remaining time to help out with the solver computations. I could not believe the effort he put in to his spreadsheet, and it really motivated me to spending more time in validating the model. He was up all the time working on it because he was unable to lay down. Kevin was also there to promote the research on other blogs, right to the end. Thanks Kevin.
There really aren't too many people willing to spend time working analysis on a scientific forum, and these two exemplified what it takes to really contribute to the advancement of ideas. Like us, they were not climate science insiders and so will only get credit if we remember them.
In the last month, two of the great citizen scientists that I will be forever personally grateful for have passed away. If anyone has followed climate science discussions on blogs and social media, you probably have seen their contributions. Keith Pickering was an expert on computer science, astrophysics, energy, and history from my neck of the woods in Minnesota. He helped me so much in working out orbital calculations when I was first looking at lunar correlations. He provided source code that he developed and it was a great help to get up to speed. He was always there to tweet any progress made. Thanks Keith  Kevin O'Neill was a metrologist and an analysis whiz from Wisconsin. In the weeks before he passed, he told me that he had extra free time to help out with ENSO analysis. He wanted to use his remaining time to help out with the solver computations. I could not believe the effort he put in to his spreadsheet, and it really motivated me to spending more time in validating the model. He was up all the time working on it because he was unable to lay down. Kevin was also there to promote the research [on other blogs](https://andthentheresphysics.wordpress.com/2017/10/23/watt-about-breaking-the-pal-review-glass-ceiling/#comment-105271), right to the end. Thanks Kevin.  There really aren't too many people willing to spend time working analysis on a scientific forum, and these two exemplified what it takes to really contribute to the advancement of ideas. Like us, they were not climate science insiders and so will only get credit if we remember them.This is the last of the ENSO charts.
This is how conventional tidal prediction is done:
This is an ENSO model fit to SOI data. Same tidal analysis algorithm is used but applying the annual solar cycle and monthly/fortnightly lunar cycles instead of the diurnal and semi-diurnal cycle.
This is an expanded view, with the corelation coefficient of 0.73:
This is a fit trained on the 1880-1950 interval (CC=0.76) and cross-validated on the post-1950 data
This is a fit trained on the post-1950 interval (CC=0.77) and cross-validated on the 1880-1950 data
Like conventional tidal prediction, very little overfitting is observed. Most of what is considered noise in the SOI data is actually the tidal forcing signal.
This is the last of the ENSO charts. This is how conventional tidal prediction is done:  This is an ENSO model fit to SOI data. Same tidal analysis algorithm is used but applying the annual solar cycle and monthly/fortnightly lunar cycles instead of the diurnal and semi-diurnal cycle.  This is an expanded view, with the corelation coefficient of 0.73:  This is a fit trained on the 1880-1950 interval (CC=0.76) and cross-validated on the post-1950 data  This is a fit trained on the post-1950 interval (CC=0.77) and cross-validated on the 1880-1950 data  Like conventional tidal prediction, very little overfitting is observed. Most of what is considered noise in the SOI data is actually the tidal forcing signal.Slightly different approach that I should have tried long ago for the ENSO model. Instead of training on the time-series, train on the complex Fourier series in frequency space.
So, this is trained on intervals of the ENSO SOI spectrum (in yellow), and it fills in the rest of the spectrum. Even though there are only 3 known lunar periods + 1 solar, because of the nonlinear Navier-Stokes doubling with multiple interactions due to the precision required of the orbits, it generates a very busy spectral waveform (much more variable than a conventional tidal spectrum)
In real space, the inverse fit captures the time-series with a high >0.8 correlation coefficient.
Slightly different approach that I should have tried long ago for the ENSO model. Instead of training on the time-series, train on the complex Fourier series in frequency space. So, this is trained on intervals of the ENSO SOI spectrum (in yellow), and it fills in the rest of the spectrum. Even though there are only 3 known lunar periods + 1 solar, because of the nonlinear Navier-Stokes doubling with multiple interactions due to the precision required of the orbits, it generates a very busy spectral waveform (much more variable than a conventional tidal spectrum)  In real space, the inverse fit captures the time-series with a high >0.8 correlation coefficient. An example of verification in pattern recognition and signal processing
http://contextearth.com/2018/04/29/enso-model-verification-via-fourier-analysis-infill/
The underlying issue is how does one confirm a hypothetical scientific model without having a controlled experiment to test against?
The best you can do is use portions of the data to align/calibrate the model and then use other orthogonal parts of the data to verify. Otherwise one can always predict future behavior and then wait 20 to 50 years to verify -- which unfortunately is detrimental to progress :(
An example of verification in pattern recognition and signal processing http://contextearth.com/2018/04/29/enso-model-verification-via-fourier-analysis-infill/ The underlying issue is how does one confirm a hypothetical scientific model without having a controlled experiment to test against? The best you can do is use portions of the data to align/calibrate the model and then use other orthogonal parts of the data to verify. Otherwise one can always predict future behavior and then wait 20 to 50 years to verify -- which unfortunately is detrimental to progress :(Interesting ..congrats
Interesting ..congratsA full model fit to shortest component 0.4y.
The residual of the fit across the entire spectrum (Nyquist T = 1/6 y) appears to be flat white noise
No use trying to improve the fit beyond this point --
A full model fit to shortest component 0.4y.  The residual of the fit across the entire spectrum (Nyquist T = 1/6 y) appears to be flat white noise  No use trying to improve the fit beyond this point -- >>>>>"Bottom line: when the residuals fail to be white noise, a different model should be tried. Short answer regarding time series regression: If they are not white noise (i.e. they are not normal, not have zero mean or serially autocorrelated), then your model is not fully adequate. Therefore, you should revise your model."Congratulations seem in order for all your ENSO work. It's given me lots of techniques to consider trying to use in other contexts. I hope the final paper gets published in GRL at least. So thanks.
Congratulations seem in order for all your ENSO work. It's given me lots of techniques to consider trying to use in other contexts. I hope the final paper gets published in GRL at least. So thanks.Thanks Jim. Noticed you are working on spatial aspects, which is the next step for me
Thanks Jim. Noticed you are working on spatial aspects, which is the next step for meJim, Right now the research is going through peer-review and the outcome is still pending as I recently responded to the critical review.
I don't want to copy&paste the written criticisms here out of courtesy, but the reviewer said essentially that it's unproven research. Of course, that's not a great position to be put in, but I responded by saying that there are really no controlled experiments available to verify the model -- all one can do is self-consistency and sampled-data tests, which is the focus of the last year of effort on the model.
The criticism also cited stochastic resonance as more of a consensus model for ENSO. The evidence that it's not stochastic is that the cyclic forcing periods have to be set precisely to the known orbital periods. The lunar months have to be precise, otherwise the errors accumulate over the 130+ years of the data. If I let these parameters vary during the fit, they converge to the lunar months being within a second in precision and the tropical year within a minute of the known parameters. They also have to be shaped according to the non-linear interactions according to the known orbital ephemeris data and the \(1/R^3\) tidal forcing law.
Because of this finding and the fact that I can't reject that lunar and solar tidal forcing based on the excellent agreement found, I have to stand my ground. In other words, if the periods didn't match then I would have abandoned the effort long ago. So I ended up not changing anything but added a few more citations on re-submittal.
> I hope the final paper gets published in GRL at least. Jim, Right now the research is going through peer-review and the outcome is still pending as I recently responded to the critical review. I don't want to copy&paste the written criticisms here out of courtesy, but the reviewer said essentially that it's unproven research. Of course, that's not a great position to be put in, but I responded by saying that there are really no controlled experiments available to verify the model -- all one can do is self-consistency and sampled-data tests, which is the focus of the last year of effort on the model. The criticism also cited stochastic resonance as more of a consensus model for ENSO. The evidence that it's not stochastic is that the cyclic forcing periods have to be set precisely to the known orbital periods. The lunar months have to be precise, otherwise the errors accumulate over the 130+ years of the data. If I let these parameters vary during the fit, they converge to the lunar months being within a second in precision and the tropical year within a minute of the known parameters. They also have to be shaped according to the non-linear interactions according to the known orbital ephemeris data and the \\(1/R^3\\) tidal forcing law. Because of this finding and the fact that I can't reject that lunar and solar tidal forcing based on the excellent agreement found, I have to stand my ground. In other words, if the periods didn't match then I would have abandoned the effort long ago. So I ended up not changing anything but added a few more citations on re-submittal.I hope the reviewer says what they think constitutes proof and cites the papers with a better stochastic explanation of the evidence.
I hope the reviewer says what they think constitutes proof and cites the papers with a better stochastic explanation of the evidence.Yes. Yet as far as I can tell there is no real consensus for ENSO. The split is between stochastic models and chaotic models and both can only give a statistical agreement. The non-chaotic deterministic model is the odd-man out because no one is looking at that approach. Somehow it was rejected long ago, even though the repeat cycle for a full lunisolar forcing is 372 years! With that long a period, hard to tell the difference between a cyclic behavior and one that is chaotic or random.
IMO, a deterministic model was dismissed too soon and it became conventional wisdom not to look at it again.
Interesting that in regard to the 372 year repeat period, which is the same as the recurrence of eclipses on the same day of the tropical year:
Sure enough, the model calculation I use follows this cycle fairly closely. Consider that the graph below is a combination of the 3 lunar sine waves plotted on precisely 1 year intervals (which provides the same day of the tropical year)
You can detect the general 372 year repeat period from comparing the two blue arrow intervals, and there is another repeat period of 1230 years which you can see by shifting and aligning two intervals. Both of these repeat periods are subtle and it's not surprising that it's not picked up in any natural phenomena apart from something as striking as a eclipse, where the 372 year cycle has been long known.
When the \(1/R^3\) forcing and the ephemeris corrections to the sine waves is invoked, we can compare to the data and how it appears when it is shifted by 372 years.
It's not perfect because the 372 year period is not perfect, but it's enough to understand how the repeat period is preserved even after the perfect sine waves are individually distorted by the same base 4 frequencies.
Yes. Yet as far as I can tell there is no real consensus for ENSO. The split is between stochastic models and chaotic models and both can only give a statistical agreement. The non-chaotic deterministic model is the odd-man out because no one is looking at that approach. Somehow it was rejected long ago, even though the repeat cycle for a full lunisolar forcing is 372 years! With that long a period, hard to tell the difference between a cyclic behavior and one that is chaotic or random. IMO, a deterministic model was dismissed too soon and it became conventional wisdom not to look at it again. Interesting that in regard to the 372 year repeat period, which is the same as the recurrence of eclipses on the same day of the tropical year: >Stockwell, John N. "On the law of recurrence of eclipses on the same day of the tropical year." The Astronomical Journal 15 (1895): 73-75. Sure enough, the model calculation I use follows this cycle fairly closely. Consider that the graph below is a combination of the 3 lunar sine waves plotted on precisely 1 year intervals (which provides the same day of the tropical year)  You can detect the general 372 year repeat period from comparing the two blue arrow intervals, and there is another repeat period of 1230 years which you can see by shifting and aligning two intervals. Both of these repeat periods are subtle and it's not surprising that it's not picked up in any natural phenomena apart from something as striking as a eclipse, where the 372 year cycle has been long known. When the \\(1/R^3\\) forcing and the ephemeris corrections to the sine waves is invoked, we can compare to the data and how it appears when it is shifted by 372 years.  It's not perfect because the 372 year period is not perfect, but it's enough to understand how the repeat period is preserved even after the perfect sine waves are individually distorted by the same base 4 frequencies.The following analysis is an excellent substantiation of the ENSO model http://contextearth.com/2018/05/08/unified-enso-proxy/
The fit does a back-extrapolation over a 230 year interval prior to 1880 using coral proxy records with a resolution of 1 year. The temporal forcing pattern is very close to that used in the modern record but the scaling has to be modified, as it appears that the magnitude of ENSO excursions were measurably reduced prior to 1880.
The salient point is that trying to fit a curve as chaotic-appearing as an ENSO time-series record by applying only 4 interacting sinusoidal waves is difficult enough. But then to extend that to almost double the length -- from 135 years to an additional 230 years with what are essentially minor tweaks to the boundary conditions -- is next to impossible if it was a truly chaotic behavior.
The following analysis is an excellent substantiation of the ENSO model http://contextearth.com/2018/05/08/unified-enso-proxy/ The fit does a back-extrapolation over a 230 year interval prior to 1880 using coral proxy records with a resolution of 1 year. The temporal forcing pattern is very close to that used in the modern record but the scaling has to be modified, as it appears that the magnitude of ENSO excursions were measurably reduced prior to 1880. The salient point is that trying to fit a curve as chaotic-*appearing* as an ENSO time-series record by applying only 4 interacting sinusoidal waves is difficult enough. But then to extend that to almost double the length -- from 135 years to an additional 230 years with what are essentially minor tweaks to the boundary conditions -- is next to impossible if it was a truly chaotic behavior. The solution to Laplace's Tidal Equation along the equator is
$$ I(t) = \Sigma^n_{i=0} A_i \sin (k_i f(t) + \theta_i) $$ where f(t) is the equatorial gravitational forcing, and the index terms are the separated standing wave components.
Since f(t) is cyclic as well, we want to determine what harmonics are generated by the sum. First, we isolate the phase factor and expand.
$$ I(t) = \Sigma A_i (\cos(\theta_i) \sin (k_i f(t)) + \sin(\theta_i) \cos(k_i f(t)) ) $$ The harmonics come about predominately from the Taylor's-series expansion of the sinusoidal factors.
$$ I(t) = \Sigma A_i (\cos(\theta_i) ( k_i f(t) - (k_i f(t))^3/3! + (k_i f(t))^5/5! + ...) + \sin(\theta_i) ( 1- (k_i f(t))^2/2! + (k_i f(t))^4/4! + ...) )$$ Notice that all the f(t) terms become power factors. This will essentially explode the power spectrum with multiplicative harmonics when there is significant saturation of the bounding sinusoidal terms. If the values are not close to saturated, only the first-order term remains, equivalent to the approximation
$$ sin(k f(t)) \sim k f(t) $$ The k pre-factors are related to the spatial standing-wave modes in the original derivation. The higher the value of k the greater the spatial wavenumber of the standing wave. Empirically, we observe one value of k that stands out, and that is likely the primary ENSO standing wave. The progressively higher values of k likely relate in some way to Tropical Instability Waves, which are thought to be related to ENSO according to a recent review article [1].
Suffice to say that this is very different math than we are used to applying for spectral analysis. How to optimize is a challenge. We can try one of these approaches:
The first grinds away forever as it uses a gradient search. The second may be faster but it doesn't automatically adjust f(t). A combination of these may work best.
[1] Timmermann, Axel, Soon-Il An, Jong-Seong Kug, Fei-Fei Jin, Wenju Cai, Antonietta Capotondi, Kim Cobb, et al. “El Niño–Southern Oscillation Complexity.” Nature 559, no. 7715 (July 2018): 535. https://doi.org/10.1038/s41586-018-0252-6.
The solution to Laplace's Tidal Equation along the equator is $$ I(t) = \Sigma^n_{i=0} A_i \sin (k_i f(t) + \theta_i) $$ where f(t) is the equatorial gravitational forcing, and the index terms are the separated standing wave components. Since f(t) is cyclic as well, we want to determine what harmonics are generated by the sum. First, we isolate the phase factor and expand. $$ I(t) = \Sigma A_i (\cos(\theta_i) \sin (k_i f(t)) + \sin(\theta_i) \cos(k_i f(t)) ) $$ The harmonics come about predominately from the Taylor's-series expansion of the sinusoidal factors. $$ I(t) = \Sigma A_i (\cos(\theta_i) ( k_i f(t) - (k_i f(t))^3/3! + (k_i f(t))^5/5! + ...) + \sin(\theta_i) ( 1- (k_i f(t))^2/2! + (k_i f(t))^4/4! + ...) )$$ Notice that all the f(t) terms become power factors. This will essentially explode the power spectrum with multiplicative harmonics when there is significant saturation of the bounding sinusoidal terms. If the values are not close to saturated, only the first-order term remains, equivalent to the approximation $$ sin(k f(t)) \sim k f(t) $$ The k pre-factors are related to the spatial standing-wave modes in the original derivation. The higher the value of k the greater the spatial wavenumber of the standing wave. Empirically, we observe one value of k that stands out, and that is likely the primary ENSO standing wave. The progressively higher values of k likely relate in some way to Tropical Instability Waves, which are thought to be related to ENSO according to a recent review article [1]. Suffice to say that this is very different math than we are used to applying for spectral analysis. How to optimize is a challenge. We can try one of these approaches: * Applying a nonlinear search of the k values, while also modifying f(t), to best correlate to the ENSO data * Group the terms of the same power factor and attempt a multi-linear regression to optimize values of k for a given f(t). The first grinds away forever as it uses a gradient search. The second may be faster but it doesn't automatically adjust f(t). A combination of these may work best. --- [1] Timmermann, Axel, Soon-Il An, Jong-Seong Kug, Fei-Fei Jin, Wenju Cai, Antonietta Capotondi, Kim Cobb, et al. “El Niño–Southern Oscillation Complexity.” Nature 559, no. 7715 (July 2018): 535. https://doi.org/10.1038/s41586-018-0252-6.The key to analytically solving Laplace's tidal equations (Navier-Stokes) along the equator is to use the ansatz of a non-fixed equator. Solved here
This is the rationale:
from "A Chorus of the Winds—On Saturn!" P. L. Read
https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2018JE005625
The key to analytically solving Laplace's tidal equations (Navier-Stokes) along the equator is to use the ansatz of a non-fixed equator. Solved [here](http://contextearth.com/compact-qbo-derivation/) This is the rationale:  from "A Chorus of the Winds—On Saturn!" P. L. Read https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2018JE005625The basis of the ENSO model is the forcing derived from the long-period cyclic lunisolar gravitational pull of the moon and sun. There is some thought that ENSO shows teleconnections to other oceanic behaviors. The primary oceanic dipoles are ENSO and AMO for the Pacific and Atlantic. There is also the PDO for the mid-northern-latitude of the Pacific, which has a pattern distinct from ENSO. So the question is: Are these connected through interactions or do they possibly share a common-mode mechanism through the same lunisolar forcing mechanism?
Based on tidal behaviors, it is known that the gravitational pull varies geographically, so it would be understandable that ENSO, AMO, and PDO would demonstrate distinct time-series signatures. In checking this, you will find that the correlation coefficient between any two of these series is essentially zero, regardless of applied leads or lags. Yet the underlying component factors (the lunar Draconic, lunar Anomalistic, and solar modified terms) may potentially emerge with only slight variations in shape, with differences only in relative amplitude. This is straightforward to test by fitting the basic ENSO model to AMO and PDO by allowing the parameters to vary.
The following figure is the result of fitting the model to ENSO, AMO, and PDO and then comparing the constituent factors.
First, note that the same parametric model fits each of the time series arguably well. The Draconic factor underling both the ENSO and AMO model is almost perfectly aligned, indicated by the red starred graph, with excursions showing a CC above 0.99. All of the rest of the CC's in fact are above 0.6.
The upshot of this analysis is two-fold. First to consider how difficult it is to fit any one of these time series to a minimal set of periodically-forced signals. Secondly that the underlying signals are not that different in character, only that the combination in terms of a Laplace's tidal equation weighting are what couples them together via a common-mode mechanism. Thus, the teleconnection between these oceanic indices is likely an underlying common lunisolar tidal forcing, just as one would suspect from conventional tidal analysis.
The basis of the ENSO model is the forcing derived from the long-period cyclic lunisolar gravitational pull of the moon and sun. There is some thought that ENSO shows [teleconnections](https://en.wikipedia.org/wiki/Teleconnection) to other oceanic behaviors. The primary oceanic dipoles are ENSO and AMO for the Pacific and Atlantic. There is also the PDO for the mid-northern-latitude of the Pacific, which has a pattern distinct from ENSO. So the question is: Are these connected through interactions or do they possibly share a common-mode mechanism through the same lunisolar forcing mechanism? Based on tidal behaviors, it is known that the gravitational pull varies geographically, so it would be understandable that ENSO, AMO, and PDO would demonstrate distinct time-series signatures. In checking this, you will find that the correlation coefficient between any two of these series is essentially zero, regardless of applied leads or lags. Yet the underlying component factors (the lunar Draconic, lunar Anomalistic, and solar modified terms) may *potentially* emerge with only slight variations in shape, with differences only in relative amplitude. This is straightforward to test by fitting the basic ENSO model to AMO and PDO by allowing the parameters to vary. The following figure is the result of fitting the model to ENSO, AMO, and PDO and then comparing the constituent factors.  First, note that the same parametric model fits each of the time series arguably well. The Draconic factor underling both the ENSO and AMO model is almost perfectly aligned, indicated by the red starred graph, with excursions showing a CC above 0.99. All of the rest of the CC's in fact are above 0.6. The upshot of this analysis is two-fold. First to consider how difficult it is to fit any one of these time series to a minimal set of periodically-forced signals. Secondly that the underlying signals are not that different in character, only that the combination in terms of a [Laplace's tidal equation weighting](https://agu.confex.com/agu/fm18/preliminaryview.cgi/Paper415478.html) are what couples them together via a common-mode mechanism. Thus, the teleconnection between these oceanic indices is likely an underlying common lunisolar tidal forcing, just as one would suspect from conventional tidal analysis.Many of the recent advances in fundamental climate science have been made by researchers with a condensed matter physics background.
Brad Marston of Brown U and colleagues are making progress with their topologically-constrained climate models. Marston's presentations are very familiar to those of us that understand the energy band diagrams used in solid-state theory. Equatorial waves have similar dispersion relations to a typical gapped band structure. The topological confinement is very similar to the assumption I am using in solving Laplace's tidal equations (i.e. reduced Navier-Stokes) along the equator.
-- from Wikipedia entry on Topological insulators
John Wettlaufer of Yale U is also applying a similar approach
Wettlaufer has a recent paper in PRL, but prior to that he published an interesting paper that clearly shows the impact of a strong annual impulse on the behavior of ENSO. A unified nonlinear stochastic time series analysis for climate science
This is an intriguing model fit feature in the paper
Michael Mann of Penn State also has a background in condensed matter physics.
Many of the recent advances in fundamental climate science have been made by researchers with a condensed matter physics background. Brad Marston of Brown U and colleagues are making progress with their [topologically-constrained climate models](http://science.sciencemag.org/content/358/6366/1075). Marston's presentations are very familiar to those of us that understand the energy band diagrams used in solid-state theory. Equatorial waves have similar dispersion relations to a typical gapped band structure. The topological confinement is very similar to the assumption I am using in solving Laplace's tidal equations (i.e. reduced Navier-Stokes) along the equator. -- from Wikipedia entry on Topological insulators  John Wettlaufer of Yale U is also applying a similar approach > [Wettlaufer](https://journals.aps.org/prl/edannounce/10.1103/PhysRevLett.116.150002): “There is a vast gulf, both conceptually and in terms of space and time scales, between simulations and idealized models. Attempts to reconcile them will have to focus on the problem of scales, a task well suited to physicists: The challenge of scale separation in both condensed matter and particle physics led to the development of the renormalization group, unifying concepts in previously disparate fields [5]. Renormalization group concepts and methods have been successfully applied to fluid dynamics problems [6,7], which are central to climate dynamics.” Wettlaufer has a [recent paper in PRL](https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.121.108701), but prior to that he published an interesting paper that clearly shows the impact of a strong annual impulse on the behavior of ENSO. [A unified nonlinear stochastic time series analysis for climate science](https://www.nature.com/articles/srep44228) This is an intriguing model fit feature in the paper  Michael Mann of Penn State also has a background in condensed matter physics.https://contextearth.com/2018/10/16/asymptotic-qbo-period/
[https://contextearth.com/2018/10/16/asymptotic-qbo-period/](https://contextearth.com/2018/10/16/asymptotic-qbo-period/)There is a neat technique that can validate a certain class of nonlinear models called frequency-domain cross-validation (FDCV)
If the sets are non-overlapping and the model fit to one set is able to predict the amplitude and phase spectrum of other set, there must be a fundamental process linking the profiles of the two frequency intervals. This could happen if the frequencies are correlated as is the case with harmonics. Consider modeling an unknown spectrum using square-wave components, but all the estimation data contained was high-harmonic frequency data. In this case, if the actual data was a square-wave, then the high-harmonic fit would naturally include the lower-frequency fundamental components. This would then generate a high-quality cross-validation check as the root model identifies the necessary fundamental implicitly.
That's a trivial and perhaps contrived example, but it also illustrates how this technique won't work on behaviors with independent signal components, but remains effective only if modulation occurs. For example, it would be impossible to predict that a 60 Hz hum existed in a signal if all you could process was frequency components of 1 kHz and above. That would be possible only if some behavior in the model forced it to simultaneously occur both in the low and high bandwidth through some cooperative interaction. For example, it might happen through some sort of nonlinear interaction or frequency modulation, but the model would have to account for that. In fact, this is why FM or AM demodulation works -- even though the signals measured are embedded, i.e. non-linearly mixed within a high-frequency carrier, the model is able to extract the low-frequency signal. The "validation" that the model works is that a high-fidelity tune is recognizable when played on a radio. Our brain is sophisticated enough that it is able to recognize the tune, otherwise something like Shazam could do the identification.
With ENSO, the nonlinear mixing is due to the alternately-annual (biennial) modulation of the cyclic lunar gravitational tidal pull. A biennial modulation is the mixing factor that allows a frequency-domain cross-validation to work. We simply need to tune the tidal factors to fit to the frequency spectrum of one interval (the estimation part), and then check to see if they also pop out on the orthogonal part of the frequency spectrum while showing a high correlation with the modeled results (the validation part).
The only way this approach would fail as a validation is if the extra degree of freedom (DOF) afforded by the biennial modulation somehow improved the correlation better than any other modulation we pick (such as triennial, biannual, etc). However, this is wildly implausible, as it is well-known that a strong biennial modulation is operational in the Pacific ocean's dynamics. In other words, this is a very weak DOF and functions more as a constraint than a free parameter. Like the tidal pull (which certainly exists with strength debateable), the biennial modulation is highly likely to exist. It's just a matter of pulling the signal out from any other noise that might exist.
Two other elements are necessary to do the FDCV, (1) the solution to Laplace's tidal equations and (2) taking the derivative of the ENSO data to equalize the ENSO frequency spectrum, thus allowing for a balanced orthogonal interval comparison.
The first training interval uses only frequency components between 0.5 /year and 1 /year in amplitude. The fit is extremely aggressive, reaching a correlation coefficient of 0.99, yet the validation interval appears to match.
The second training interval works the complement.
These are the real-space fits corresponding to the frequency-space fits
There is a neat technique that can validate a certain class of nonlinear models called [frequency-domain cross-validation](http://www.diva-portal.org/smash/get/diva2:315809/FULLTEXT02) (FDCV) > "In the frequency domain cross validation is easily performed by dividing the frequency measurements in two disjoint sets : estimation data and validation data. [...] A model is then estimated using the estimation data only. The quality of the model is assessed by comparing the estimated transfer function with the validation data set and a proper model order can then be inferred." If the sets are non-overlapping and the model fit to one set is able to predict the amplitude and phase spectrum of other set, there must be a fundamental process linking the profiles of the two frequency intervals. This could happen if the frequencies are correlated as is the case with harmonics. Consider modeling an unknown spectrum using square-wave components, but all the estimation data contained was high-harmonic frequency data. In this case, if the *actual* data was a square-wave, then the high-harmonic fit would naturally include the lower-frequency fundamental components. This would then generate a high-quality cross-validation check as the root model identifies the necessary fundamental implicitly. That's a trivial and perhaps contrived example, but it also illustrates how this technique won't work on behaviors with independent signal components, but remains effective only if modulation occurs. For example, it would be impossible to predict that a 60 Hz hum existed in a signal if all you could process was frequency components of 1 kHz and above. That would be possible only if some behavior in the model forced it to simultaneously occur both in the low and high bandwidth through some cooperative interaction. For example, it might happen through some sort of nonlinear interaction or frequency modulation, but the model would have to account for that. In fact, this is why FM or AM demodulation works -- even though the signals measured are embedded, i.e. non-linearly mixed within a high-frequency carrier, the model is able to extract the low-frequency signal. The "validation" that the model works is that a high-fidelity tune is recognizable when played on a radio. Our brain is sophisticated enough that it is able to recognize the tune, otherwise something like Shazam could do the identification. With ENSO, the nonlinear mixing is due to the alternately-annual (biennial) modulation of the cyclic lunar gravitational tidal pull. A biennial modulation is the mixing factor that allows a frequency-domain cross-validation to work. We simply need to tune the tidal factors to fit to the frequency spectrum of one interval (the estimation part), and then check to see if they also pop out on the orthogonal part of the frequency spectrum while showing a high correlation with the modeled results (the validation part). The only way this approach would fail as a validation is if the extra degree of freedom (DOF) afforded by the biennial modulation somehow improved the correlation better than any other modulation we pick (such as triennial, biannual, etc). However, this is wildly implausible, as it is well-known that a strong biennial modulation is operational in the Pacific ocean's dynamics. In other words, this is a very weak DOF and functions more as a constraint than a free parameter. Like the tidal pull (which **certainly** exists with strength debateable), the biennial modulation is highly likely to exist. It's just a matter of pulling the signal out from any other noise that might exist. Two other elements are necessary to do the FDCV, (1) the solution to Laplace's tidal equations and (2) taking the derivative of the ENSO data to equalize the ENSO frequency spectrum, thus allowing for a balanced orthogonal interval comparison. The first training interval uses only frequency components between 0.5 /year and 1 /year in amplitude. The fit is extremely aggressive, reaching a correlation coefficient of 0.99, yet the validation interval appears to match. The second training interval works the complement.  These are the real-space fits corresponding to the frequency-space fits There is SOI data on a daily time scale since 1992 on this site https://data.longpaddock.qld.gov.au/SeasonalClimateOutlook/SouthernOscillationIndex/SOIDataFiles/DailySOI1887-1989Base.txt
I had been using monthly data and it wasn't clear whether the high wave-number (high-K) amplitudes used to fit the monthly time-series would also be apparent on a daily basis. Keeping the filtering to a minimum, the upper panel shows the monthly SOI time-series, applying a fit from 1992 onward. The bottom panel shows what it looks like on a magnified daily time scale.
The recent paper by Jajcay (cite below) claims that there is synchronization across time scales for ENSO, which is implicit for the ENSO model used above. For our ENSO model, the low -K solutions (low frequency, corresponding to the main Tahiti-Darwin standing wave dipole) are automatically synchronized with the high-K solutions (high frequency, which are likely related to Tropical Instability Waves and/or Madden-Julian Oscillations). The main dipole requires both the low-K and high-K solutions to achieve the sharp jagged profile -- constructed through the superposition of the non-linear harmonics -- which is clearly apparent when one expands to the daily time-scale. For example, the strong 1998 El-Nino is actually composed of the superposition of a longer-time-period signal superposed with multiple higher-frequency jagged peaks.
There is SOI data on a daily time scale since 1992 on this site https://data.longpaddock.qld.gov.au/SeasonalClimateOutlook/SouthernOscillationIndex/SOIDataFiles/DailySOI1887-1989Base.txt I had been using monthly data and it wasn't clear whether the high wave-number (high-K) amplitudes used to fit the monthly time-series would also be apparent on a daily basis. Keeping the filtering to a minimum, the upper panel shows the monthly SOI time-series, applying a fit from 1992 onward. The bottom panel shows what it looks like on a magnified daily time scale.  The recent paper by Jajcay (cite below) claims that there is synchronization across time scales for ENSO, which is implicit for the ENSO model used above. For our ENSO model, the low -K solutions (low frequency, corresponding to the main Tahiti-Darwin standing wave dipole) are automatically synchronized with the high-K solutions (high frequency, which are likely related to Tropical Instability Waves and/or Madden-Julian Oscillations). The main dipole requires both the low-K and high-K solutions to achieve the sharp jagged profile -- constructed through the superposition of the non-linear harmonics -- which is clearly apparent when one expands to the daily time-scale. For example, the strong 1998 El-Nino is actually composed of the superposition of a longer-time-period signal superposed with multiple higher-frequency jagged peaks. > Jajcay, Nikola, et al. ["Synchronization and causality across time scales in El Niño Southern Oscillation."](https://www.nature.com/articles/s41612-018-0043-7) npj Climate and Atmospheric Science 1.1 (2018): 33.Fitting the model simultaneously to the historical monthly data (top panel) and high-resolution daily data (middle panel) using the high-frequency spectrum correlation (bottom panel) to weight the high-K parameters.
 Fitting the model simultaneously to the historical monthly data (top panel) and high-resolution daily data (middle panel) using the high-frequency spectrum correlation (bottom panel) to weight the high-K parameters.