Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Categories
- All Categories 2.4K
- Chat 505
- Study Groups 21
- Petri Nets 9
- Epidemiology 4
- Leaf Modeling 2
- Review Sections 9
- MIT 2020: Programming with Categories 51
- MIT 2020: Lectures 20
- MIT 2020: Exercises 25
- Baez ACT 2019: Online Course 339
- Baez ACT 2019: Lectures 79
- Baez ACT 2019: Exercises 149
- Baez ACT 2019: Chat 50
- UCR ACT Seminar 4
- General 75
- Azimuth Code Project 111
- Statistical methods 4
- Drafts 10
- Math Syntax Demos 15
- Wiki - Latest Changes 3
- Strategy 113
- Azimuth Project 1.1K
- - Spam 1
- News and Information 148
- Azimuth Blog 149
- - Conventions and Policies 21
- - Questions 43
- Azimuth Wiki 719
Options
Tidal records and ENSO
I think this is a possible direction to go, based on some very promising results that I worked on over the past day.
http://contextearth.com/2014/09/16/using-tidal-gauges-to-estimate-enso
For all I know, this may be a well known correlation, but I discovered that the tidal gauge data in Sydney Harbor aligns with the SOI behavior remarkably well, especially if a delay differencing of about 2 years is applied.
The tidal data itself has a significant annual and bi-annual periodicity modulated by an erratic envelope, potentially modeled by a DiffEq as shown below.
Another interesting teleconnection to consider !
Paul Pukite
Comments
Paul I got the same, redid you calcs, this is terrific
Dara
Paul I got the same, redid you calcs, this is terrific DaraDara, Thanks for the sanity check.
I am trying to understand the significance of the two-year differencing.
A two year delay differencing can act as a very granular derivative, so it may be a measure of the speed at which the sea-level is rising in the harbor.
Or it could be an indicator of a previous value that has reflected back.
Timely that Blake Pollard wrote the blog post with the animated GIF of SST, which is a result of the water volume instability.
Contrast that with a dynamic simulation of sloshing -- be patient with this one as it builds up over time. This is a solution to a forced Mathieu equation [1]
[1] S. S. Kolukula and P. Chellapandi, “Finite Element Simulation of Dynamic Stability of plane free-surface of a liquid under vertical excitation.”
Dara, Thanks for the sanity check. I am trying to understand the significance of the two-year differencing. A two year delay differencing can act as a very granular derivative, so it may be a measure of the speed at which the sea-level is rising in the harbor. Or it could be an indicator of a previous value that has reflected back. Timely that Blake Pollard wrote [the blog post](http://johncarlosbaez.wordpress.com/2014/09/16/exploring-climate-data-part-2/) with the animated GIF of SST, which is a result of the water volume instability.  Contrast that with a dynamic simulation of sloshing -- be patient with this one as it builds up over time. This is a solution to a forced Mathieu equation [1]  [1] S. S. Kolukula and P. Chellapandi, “Finite Element Simulation of Dynamic Stability of plane free-surface of a liquid under vertical excitation.”Why don't you cook up a time series so I could wavelet transform it and see this 2 year periodicity.
I have a question: Why do you think these periodicals need to have a cognitive or rational explanation e.g. sloshing. Maybe the dynamical system has 'beats' like waves canceling some parts of other waves.
>A two year delay differencing can act as a very granular derivative, so it may be a measure of the speed at which the sea-level is rising in the harbor. Why don't you cook up a time series so I could wavelet transform it and see this 2 year periodicity. I have a question: Why do you think these periodicals need to have a cognitive or rational explanation e.g. sloshing. Maybe the dynamical system has 'beats' like waves canceling some parts of other waves.Dara, I always need to work against a physical explanation. I have learned from experience that it is common to be accused of numerology or "trendology" if one doesn't have a physical basis.
Yet that is essentially what constitutes Bloch waves -- a set of waveforms that are allowed based on the constraints of the system. Within the math abstraction of sloshing, standing waves develop which interfere with each other.
Adding delay differencing adds another layer of complexity. I am not even certain that the 2-year delay manifests as a periodicity.
In digital signal processing a delay difference is a $1 - 1/Z$ term, which is a building block for a filter. And unless I am mistaken, this should eliminate 2-year periodicities because it acts as a notch filter. Think about it, if one samples a sin wave of 2 year period and takes the difference at any two points separated by two years, one gets a zero result !
Dara, I always need to work against a physical explanation. I have learned from experience that it is common to be accused of numerology or "trendology" if one doesn't have a physical basis. "Maybe the dynamical system has ’beats’ like waves canceling some parts of other waves." Yet that is essentially what constitutes Bloch waves -- a set of waveforms that are allowed based on the constraints of the system. Within the math abstraction of sloshing, standing waves develop which interfere with each other. Adding delay differencing adds another layer of complexity. I am not even certain that the 2-year delay manifests as a periodicity. In digital signal processing a delay difference is a $1 - 1/Z$ term, which is a building block for a filter. And unless I am mistaken, this should eliminate 2-year periodicities because it acts as a notch filter. Think about it, if one samples a sin wave of 2 year period and takes the difference at any two points separated by two years, one gets a zero result !SOI is atmospheric pressure Tahiti-Darwin, so -SOI correlates with atmospheric pressure at Darwin, not all that far from Sydney. I wonder how atmospheric pressure at Sydney would fit in?
WHT, SOI is atmospheric pressure Tahiti-Darwin, so -SOI correlates with atmospheric pressure at Darwin, not all that far from Sydney. I wonder how atmospheric pressure at Sydney would fit in?Thanks Nick, I will look that up. Pressure does effect sea-level, via the inverted barometer effect decribed here.
I will post an interesting fit to the tidal variation shortly, which involves an exact two-year sinusoidal forcing function. This is starting to make some sense.
Thanks Nick, I will look that up. Pressure does effect sea-level, via the *inverted barometer effect* decribed [here](http://www.bom.gov.au/oceanography/tides/met_effects.shtml). "In addition, oceanographic effects such as Southern Ocean Oscillation (El Niño) can produce large scale variations in mean sea level of up to 0.5m with corresponding changes in rate and direction of tidal streams." I will post an interesting fit to the tidal variation shortly, which involves an exact two-year sinusoidal forcing function. This is starting to make some sense.Paul could I challenge you on this statement?
Dara
>Dara, I always need to work against a physical explanation. Paul could I challenge you on this statement? DaraPaul I think our ideas on PHYSICAL EXPLANATION are antiquated and primitive, we could have much more sophisticated explanations but machine-generated. I think we have reached the boudaries of human explanation for physical phenomenon, we need to think of a new machine-aided paradigm.
Best example is what we are seeing with atmospherics and oceanics, there is no reasonable human explanation at all.
Dara
Paul I think our ideas on PHYSICAL EXPLANATION are antiquated and primitive, we could have much more sophisticated explanations but machine-generated. I think we have reached the boudaries of human explanation for physical phenomenon, we need to think of a new machine-aided paradigm. Best example is what we are seeing with atmospherics and oceanics, there is no reasonable human explanation at all. DaraIn the tidal data there are strong frequency components at 1 year and 1/2 year. These are understandable from seasonal variations caused by the earth's rotation.
So I placed a simple 1-year box-filter on the Sydney tidal data and I also filtered out the low-freq components and overall trend.
Next I added a forcing function of exactly a 2 year period on the RHS of a Mathieu-type DiffEq and optimized over an interval that I used for the validation fit earlier:
This is a promising path because the formulation is very simple yet seems to be able to capture the erratic nature of the waveform.
The need for the 2-year delay differencing may now make more sense. It is possible that ENSO is not sensitive to the 2-year tidal cycle so that to extract the ENSO factor from the tidal data, it is important to filter out that periodicity. As I said before the delay difference of 2 years acts as a notch filter to selectively remove that period. It may be that what remains after this filtering is an erratic gradient that tracks the ENSO behavior.
This is like untying a knot ! As one is never sure if one is making things more knotty or less knotty.
BTW, I have no good idea as to what physical mechanism drives the 2-year period, unless some nonlinear behavior is causing a period-doubling of the seasonal behavior -- perhaps a bifurcation?
In the tidal data there are strong frequency components at 1 year and 1/2 year. These are understandable from seasonal variations caused by the earth's rotation. So I placed a simple 1-year box-filter on the Sydney tidal data and I also filtered out the low-freq components and overall trend. Next I added a forcing function of exactly a 2 year period on the RHS of a Mathieu-type DiffEq and optimized over an interval that I used for the validation fit earlier:  This is a promising path because the formulation is very simple yet seems to be able to capture the erratic nature of the waveform. The need for the 2-year delay differencing may now make more sense. It is possible that ENSO is *not* sensitive to the 2-year tidal cycle so that to extract the ENSO factor from the tidal data, it is important to filter out that periodicity. As I said before the delay difference of 2 years acts as a notch filter to selectively remove that period. It may be that what remains after this filtering is an erratic gradient that tracks the ENSO behavior. This is like untying a knot ! As one is never sure if one is making things more knotty or less knotty. BTW, I have no good idea as to what physical mechanism drives the 2-year period, unless some nonlinear behavior is causing a period-doubling of the seasonal behavior -- perhaps a bifurcation?Dara said:
I am going on the assumption that the climate science establishment has overlooked something simple. I realize that machine learning may be able to deduce new connections, but the combination of machine learning with guided insight is what can prune unproductive paths -- and it is fun and challenging to work this way.
BTW, I have heard it said there is also no reasonable human explanation for quantum mechanics, you just take it as a given. Yet one can scientifically reason based on applying the physical explanation of wave interference, e.g. in doing electron diffraction.
Dara said: "Best example is what we are seeing with atmospherics and oceanics, there is no reasonable human explanation at all." I am going on the assumption that the climate science establishment has overlooked something simple. I realize that machine learning may be able to deduce new connections, but the combination of machine learning with guided insight is what can prune unproductive paths -- and it is fun and challenging to work this way. BTW, I have heard it said there is also no reasonable human explanation for quantum mechanics, you just take it as a given. Yet one can scientifically reason based on applying the physical explanation of wave interference, e.g. in doing electron diffraction.This is true if you were quoting Feynman, may he rest in peace. The proposed algebras that model quantum behaviour are human cognizable but the results obtained from them are not e.g. quantum tunneling or time reversal.
So the written code that does certain algorithm is human cognizable but the results it produces are not! This is almost true for any application of Machine Learning and Machine Vision as well, that humans could not explain the output of these code/algorithms. Yet the results are useful and model some dynamical systems well.
If we remove the code and the algorithm and try to explain how to navigate a UAV in a storm or how to balance a 1-legged robot or how to fly an airplane with a broken damaged wing, we cannot.
Almost the same is true for solutions to a system of differential equations.
So my comment to you was: DO USE SLOSHING like arguments and computations, but we do not have to be stuck with known formulas or patterns of behaviour to model a dynamical system.
Dara
>BTW, I have heard it said there is also no reasonable human explanation for quantum mechanics, This is true if you were quoting Feynman, may he rest in peace. The proposed algebras that model quantum behaviour are human cognizable but the results obtained from them are not e.g. quantum tunneling or time reversal. So the written code that does certain algorithm is human cognizable but the results it produces are not! This is almost true for any application of Machine Learning and Machine Vision as well, that humans could not explain the output of these code/algorithms. Yet the results are useful and model some dynamical systems well. If we remove the code and the algorithm and try to explain how to navigate a UAV in a storm or how to balance a 1-legged robot or how to fly an airplane with a broken damaged wing, we cannot. Almost the same is true for solutions to a system of differential equations. So my comment to you was: DO USE SLOSHING like arguments and computations, but we do not have to be stuck with known formulas or patterns of behaviour to model a dynamical system. DaraOne thing to remember about 2nd-order DiffEq formulations is that the results are attenuated based on the frequency of the RHS forcing. So that if a natural resonant frequency of $\omega_0$ =1.5 rads/year is excited with equal magnitudes of frequencies $\omega$ = 1, $\pi$, $2\pi$, and $4\pi$ rads\year, the amount of attenuation will increase in the order, according to this fraction :
$ \frac{1}{1+(\omega/\omega_0)^2} $
And a Mathieu-type non-linearity in the DiffEq can further selectively amplify or attenuate frequencies.
This helps explain how lower frequencies of relative small magnitudes can emerge, even though they aren't the strongest perturbation. And why the higher frequencies don't become erratic as well, since they are not as close to the threshold of resonance as the lower frequencies are.
The 2-year delay differential is a convenient way of notching each of the $\pi$, $2\pi$, and $4\pi$ forcing factors from contributing to the lower frequency ENSO dynamics. They may still be there but whenever anyone plots SOI, typically a yearly smoothing filter is applied beforehand so they don't appear in the chart. Otherwise , the SOI will get obscured by this seasonal noise.
I can stand to be corrected on this interpretation. The main objective I have is to understand the rationale for being able to isolate the ENSO features in the tidal gauge data. I am concerned that the $\pi$, $2\pi$, and $4\pi$ forcing factors may all have an impact and that by throwing this data away, and in particular, when we potentially have a non-linear dynamic occurring, may not be wise.
One thing to remember about 2nd-order DiffEq formulations is that the results are attenuated based on the frequency of the RHS forcing. So that if a natural resonant frequency of $\omega_0$ =1.5 rads/year is excited with equal magnitudes of frequencies $\omega$ = 1, $\pi$, $2\pi$, and $4\pi$ rads\year, the amount of attenuation will increase in the order, according to this fraction : $ \frac{1}{1+(\omega/\omega_0)^2} $ And a Mathieu-type non-linearity in the DiffEq can further selectively amplify or attenuate frequencies. This helps explain how lower frequencies of relative small magnitudes can emerge, even though they aren't the strongest perturbation. And why the higher frequencies don't become erratic as well, since they are not as close to the threshold of resonance as the lower frequencies are. The 2-year delay differential is a convenient way of notching each of the $\pi$, $2\pi$, and $4\pi$ forcing factors from contributing to the lower frequency ENSO dynamics. They may still be there but whenever anyone plots SOI, typically a yearly smoothing filter is applied beforehand so they don't appear in the chart. Otherwise , the SOI will get obscured by this seasonal noise. I can stand to be corrected on this interpretation. The main objective I have is to understand the rationale for being able to isolate the ENSO features in the tidal gauge data. I am concerned that the $\pi$, $2\pi$, and $4\pi$ forcing factors may all have an impact and that by throwing this data away, and in particular, when we potentially have a non-linear dynamic occurring, may not be wise.I am freeing up, so I will fire off more solutions and track your work
I am freeing up, so I will fire off more solutions and track your workI created another variant of the chart in #10. Besides the biennial forcing function on the RHS of the model, I added annual and biannual forcing factors to give the model fine structure. This doesn't improve the correlation coefficient significantly but it indicates where the details in the signal may be coming from.
The fit is qualitatively good and it appears on the verge of matching exactly, except for areas highlighted in yellow. These appear to flip as mirror images around the zero level, perhaps indicating a phase reversal on a zero crossing. If the forcing indeed is a biennial function, the phase on an even and odd year is metastable, as neither is physically preferred.
Post year 2000, the phase reverses completely, and the forcing waveform needs to be shifted to get the model in synch with the tidal data:
This is uncharted territory AFAIAC and so it will be interesting what it leads to.
I created another variant of the chart in #10. Besides the biennial forcing function on the RHS of the model, I added annual and biannual forcing factors to give the model fine structure. This doesn't improve the correlation coefficient significantly but it indicates where the details in the signal may be coming from.  The fit is qualitatively good and it appears on the verge of matching exactly, except for areas highlighted in yellow. These appear to flip as mirror images around the zero level, perhaps indicating a phase reversal on a zero crossing. If the forcing indeed is a biennial function, the phase on an even and odd year is metastable, as neither is physically preferred. Post year 2000, the phase reverses completely, and the forcing waveform needs to be shifted to get the model in synch with the tidal data:  This is uncharted territory AFAIAC and so it will be interesting what it leads to.So far the only feedback that I have received on the tidalGauge/ENSO correlation concerns applying a 3 month offset-delay to the tidal data data before doing the correlation. Two others elsewhere mentioned this, and Nick's suggestion was to use Sydney atmospheric pressure data instead, which I inferred meant that a time shift $\delta T$ would be unnecessary. I am still looking for the Sydney pressure data, but in the interim I decided to show how subtle this 3-month effect is.
This was what was fed into the optimizer:
$$ SOI(t) = k (Tide(t - \delta T) - Tide(t - \delta T - \Delta T)) $$ In the figure below, the top chart has the delay and the bottom has no offset delay, $\delta T =0$, with $\Delta T$ unchanged at 23 months. The difference in correlation coefficient is 0.65 vs 0.62. The correlation coefficient was used as the measure to minimize against via the optimizer and the 3-month offset and the 23-month delay difference is what the machine learning determined as the best correlation.
I don't think this offset delay is the most interesting feature in the correlation, but since three people made essentially the same observation I figured I would dive a little deeper into the optimization approach. Bottom-line is that I am not putting a finger on the scale, and this is what the machine learning discovered.
So far the only feedback that I have received on the tidalGauge/ENSO correlation concerns applying a 3 month offset-delay to the tidal data data before doing the correlation. Two others elsewhere mentioned this, and Nick's suggestion was to use Sydney atmospheric pressure data instead, which I inferred meant that a time shift $\delta T$ would be unnecessary. I am still looking for the Sydney pressure data, but in the interim I decided to show how subtle this 3-month effect is. This was what was fed into the optimizer: $$ SOI(t) = k (Tide(t - \delta T) - Tide(t - \delta T - \Delta T)) $$ In the figure below, the top chart has the delay and the bottom has no offset delay, $\delta T =0$, with $\Delta T$ unchanged at 23 months. The difference in correlation coefficient is 0.65 vs 0.62. The correlation coefficient was used as the measure to minimize against via the optimizer and the 3-month offset and the 23-month delay difference is what the machine learning determined as the best correlation.  I don't think this offset delay is the most interesting feature in the correlation, but since three people made essentially the same observation I figured I would dive a little deeper into the optimization approach. Bottom-line is that I am not putting a finger on the scale, and this is what the machine learning discovered.I was searching for any references to a strict biennial == 2-year periodicity in the climate science literature and found [1] by Kim in 2003. This doesn't have many cites other than by the author to his own work, but it may hold the key.
From the abstract:
[1] Kim, Kwang-Yul, James J. O'Brien, and Albert I. Barcilon. "The principal physical modes of variability over the tropical Pacific." Earth Interactions 7.3 (2003): 1-32. PDF
Here is a selected passage where they describe the even and odd year effects
This seems like a compelling story, but why isn't there more on this topic of even versus odd year ENSO effects? It is possible that since this was published in 2003, that something happened to cause the even/odd years to switch places and so scientists reading the paper later would think it was not applicable. It is difficult for me to see how an odd year is special over an even year, and so one should be open to the idea that it could reverse.
Notice also the reference to the "about 3 months for the Kelvin waves to reach the eastern boundary", which happens to be the delay from Sydney harbor tidal readings to the corresponding SOI readings that I discovered in the correlation.
There are many reference to biennial variations in Pacific Ocean salmon catch -- google Salmon "Odd Year" "Even Year".
In recent times , the salmon catch has been higher in odd years, but this has flipped in the past.
[2] Irvine, J. R., et al. "Increasing Dominance of Odd-Year Returning Pink Salmon." Transactions of the American Fisheries Society 143.4 (2014): 939-956. PDF here
I was searching for any references to a strict biennial == 2-year periodicity in the climate science literature and found [1] by Kim in 2003. This doesn't have many cites other than by the author to his own work, but it may hold the key. From the abstract: "The first CSEOF is stochastic in nature and represents a standing mode of SST variability associated with a basinwide change in the surface wind. The second CSEOF exhibits a strong deterministic component describing a biennial oscillation between a warm phase and a cold phase. The surface wind directional change in the far-western Pacific appears to be instrumental for the oscillation between the two phases. Because of the distinct nature of evolution, dynamical and thermodynamical responses of the two modes are different. Further, the predictability of the two modes is different. Specifically, the biennial mode is more predictable because of the strong deterministic component associated with its evolution. The distinction of the two modes, therefore, may be important for predicting ENSO. The irregular interplay of the two modes seems to explain some inter-ENSO variability, namely, variable duration of ENSO events, approximate phase-locking property, and irregular onset and termination times." [1] Kim, Kwang-Yul, James J. O'Brien, and Albert I. Barcilon. "The principal physical modes of variability over the tropical Pacific." Earth Interactions 7.3 (2003): 1-32. [PDF](http://journals.ametsoc.org/doi/pdf/10.1175/1087-3562%282003%29007%3C0001%3ATPPMOV%3E2.0.CO%3B2) Here is a selected passage where they describe the even and odd year effects "As they propagate eastward, the upwelling (negative) Kelvin waves erode the thermocline structure favorable for the maintenance of the positive sea surface temperature anomaly condition in the central and eastern Pacific. This terminates the warm phase of the biennial oscillation. It takes about 3 months for the Kelvin waves to reach the eastern boundary; then, the colder SST condition is observed in the eastern Pacific. In the winters of even years, the situation reverses. An easterly surface wind anomaly is observed in the far-western Pacific, and henceforth the thermocline deepens. The warmer subsurface water at the thermocline depth propagates eastward as downwelling Kelvin waves as the direction of the surface wind anomaly changes in the western Pacific by the end of the even year. Then, a warmer surface condition is observed in the following spring" This seems like a compelling story, but why isn't there more on this topic of even versus odd year ENSO effects? It is possible that since this was published in 2003, that something happened to cause the even/odd years to switch places and so scientists reading the paper later would think it was not applicable. It is difficult for me to see how an odd year is special over an even year, and so one should be open to the idea that it could reverse. Notice also the reference to the *"about 3 months for the Kelvin waves to reach the eastern boundary"*, which happens to be the delay from Sydney harbor tidal readings to the corresponding SOI readings that I discovered in the correlation. --- There are many reference to biennial variations in Pacific Ocean salmon catch -- google Salmon "Odd Year" "Even Year". In recent times , the salmon catch has been higher in odd years, but this has flipped in the past. [2] Irvine, J. R., et al. "Increasing Dominance of Odd-Year Returning Pink Salmon." Transactions of the American Fisheries Society 143.4 (2014): 939-956. [PDF here](http://www.tandfonline.com/doi/full/10.1080/00028487.2014.889747#.VBsx07RtR8w)Hello Paul, not referring to your work, these papers are too verbose for me. I have no idea what this fellow is saying in such ambiguous verbiage e.g. KELVIN WAVES ERODE THE THERMOCLINE STRUCTURE... I have no idea what is ERODE nor STRUCTURE.
This is why I read these papers and while I understand every line and most of the math, I have no idea what in the world they are saying as an engineer.
These English sentences reek with ambiguity and double meanings. Say I was coding some model for these, I will imagine several versions of the code completely different.
I think these paper are written so that if someone critic them, they could always say: O that is not what I meant I meant this... and there is much room for interpretation.
I prefer what you do, there is data, math and code and a bit of English to explain the model and let the CPU chug along
Dara
Hello Paul, not referring to your work, these papers are too verbose for me. I have no idea what this fellow is saying in such ambiguous verbiage e.g. KELVIN WAVES ERODE THE THERMOCLINE STRUCTURE... I have no idea what is ERODE nor STRUCTURE. This is why I read these papers and while I understand every line and most of the math, I have no idea what in the world they are saying as an engineer. These English sentences reek with ambiguity and double meanings. Say I was coding some model for these, I will imagine several versions of the code completely different. I think these paper are written so that if someone critic them, they could always say: O that is not what I meant I meant this... and there is much room for interpretation. I prefer what you do, there is data, math and code and a bit of English to explain the model and let the CPU chug along DaraSame here, Dara. These wordy narratives are often referred to as "just so" stories. They are built on a sequence of premises "just so" they start to make plausible sense. Whether they are actually correct is another matter. I include these articles so that we can use them to cite previous work should we make definite progress.
For now, this is my toe-hold to try to explain why the tidal data has the magic 2-year number, both in the biennial oscillations and in the delay differencing factor.
As my own variation of a "just so" story, I could hypothesize that the surface of the ocean is sloshing with a certain behavior, while the thermocline layer below is sloshing with a similar behavior, but delayed or lagged by two years. The difference between the two could be the maximally exposed sub-surface temperature leading to SST and atmospheric pressure changes. Whether this working hypothesis is valid, who knows? but it gives us some other ideas for data to look at.
Same here, Dara. These wordy narratives are often referred to as "just so" stories. They are built on a sequence of premises "just so" they start to make plausible sense. Whether they are actually correct is another matter. I include these articles so that we can use them to cite previous work should we make definite progress. For now, this is my toe-hold to try to explain why the tidal data has the magic 2-year number, both in the biennial oscillations and in the delay differencing factor. As my own variation of a "just so" story, I could hypothesize that the surface of the ocean is sloshing with a certain behavior, while the thermocline layer below is sloshing with a similar behavior, but delayed or lagged by two years. The difference between the two could be the maximally exposed sub-surface temperature leading to SST and atmospheric pressure changes. Whether this working hypothesis is valid, who knows? but it gives us some other ideas for data to look at.Dear Paul let me know if I am getting on your nerves.
I buy the magic 2-year number form your code or some wavelet code or something similar which shows the periodicity as such, the rest of JUST SO English I do not buy.
Also on the sloshing: The sloshing might occur between warmer less dense waters sloshing against much colder denser water! I wonder why we should only think that sloshing happened against the shorelines?
Dear Paul let me know if I am getting on your nerves. I buy the magic 2-year number form your code or some wavelet code or something similar which shows the periodicity as such, the rest of JUST SO English I do not buy. Also on the sloshing: The sloshing might occur between warmer less dense waters sloshing against much colder denser water! I wonder why we should only think that sloshing happened against the shorelines?That's it exactly. We will see how far the computational math can take us and have fun speculating on what is happening as we go along.
That's it exactly. We will see how far the computational math can take us and have fun speculating on what is happening as we go along.Ok count me in
> We will see how far the computational math can take us and have fun speculating on what is happening as we go along. Ok count me inLatest information on the Tide Gauge / ENSO connection is here:
http://contextearth.com/2014/09/21/an-enso-predictor-based-on-a-tide-gauge-data-model/
The model is a Mathieu DiffEq with a RHS forcing that includes annual, biannual, and biennial sinusoidal factors. The biennial is most critical, while the annual and biannual factors add detail.
I timed posting this to coincide with the People's Climate March and am getting some feedback.
Bottomline I think the reason the model is working so well is that the Tide Gauge data is more directly sensitive to sloshing of the ocean, which makes sense since sloshing is a height differential phenomena. Whereas ENSO indices such as SOI are sensitive but only though an affine transformation to the sloshing motion.
This is the latest model fit:
The correlation coefficient is "only" 0.53 (highlighted in blue) but that is one of those measures that is dependent on the structure of the data. It also depends on not encountering bad stretches such as that highlighted in yellow. These stretches essentially negate and then detract because they show almost perfect anti-correlation. The task now is to find out if there is any way that this quality of fit is merely fortuitous and coincidental. I am looking for a cookbook approach to verifying that is not the case.
Latest information on the Tide Gauge / ENSO connection is here: <http://contextearth.com/2014/09/21/an-enso-predictor-based-on-a-tide-gauge-data-model/> The model is a Mathieu DiffEq with a RHS forcing that includes annual, biannual, and biennial sinusoidal factors. The biennial is most critical, while the annual and biannual factors add detail. I timed posting this to coincide with the People's Climate March and am getting some feedback. Bottomline I think the reason the model is working so well is that the Tide Gauge data is more directly sensitive to sloshing of the ocean, which makes sense since sloshing is a height differential phenomena. Whereas ENSO indices such as SOI are sensitive but only though an affine transformation to the sloshing motion. This is the latest model fit:  The correlation coefficient is "only" 0.53 (highlighted in blue) but that is one of those measures that is dependent on the structure of the data. It also depends on not encountering bad stretches such as that highlighted in yellow. These stretches essentially negate and then detract because they show almost perfect anti-correlation. The task now is to find out if there is any way that this quality of fit is merely fortuitous and coincidental. I am looking for a cookbook approach to verifying that is not the case.Early on there was a discussion of how Ludescher et al were comparing peaks and classifying success in terms of how many of the peaks matched.
I am thinking that something similar can be done with the tidal sloshing model. In the figure below, I highlighted peaks that matched in yellow and circled in red those that appeared out of phase.
What is interesting about the red regions is that they appear in anti-phase pairs, perhaps indicating that a metastable phase reversal lasts for a full cycle before it flips back. .
Early on there was a discussion of how Ludescher et al were comparing peaks and classifying success in terms of how many of the peaks matched. I am thinking that something similar can be done with the tidal sloshing model. In the figure below, I highlighted peaks that matched in yellow and circled in red those that appeared out of phase.  What is interesting about the red regions is that they appear in anti-phase pairs, perhaps indicating that a metastable phase reversal lasts for a full cycle before it flips back. .One thing to notice on the phase reversals in #24 is that they occur near the origin. Recall the Mathieu differential equation for sloshing:
$ f''(t)+[a-2q\cos(2\omega t)]f(t)=F(t) $
where $F(t)$ is $ A sin(\pi t ) + B sin(2\pi t) + C sin(4\pi t ) $ ignoring all the phase constants for illustrative purposes
As $f(t)$ crosses zero, $f''(t)$ also nears an inflection point of zero if the waveform is near sinusoidal and has a zero mean value. This means that the forcing RHS, $F(t)$, has a greater influence on the direction that the waveform will take as it crosses the axis.
The premise is that the biennial forcing factor, $ A sin(\pi t ) $, has to be metastable to begin with since the earth favors neither odd years nor even years. And so the question becomes -- is this zero-crossing the point where the biennial oscillation could change from odd/even pairing in years to even/odd (or vice versa) ? And if there is a bias in the metastability, that it could change back on the next cycle, because one pairing is slightly favored ?
One thing to notice on the phase reversals in #24 is that they occur near the origin. Recall the Mathieu differential equation for sloshing: $ f''(t)+[a-2q\cos(2\omega t)]f(t)=F(t) $ where $F(t)$ is $ A sin(\pi t ) + B sin(2\pi t) + C sin(4\pi t ) $ ignoring all the phase constants for illustrative purposes As $f(t)$ crosses zero, $f''(t)$ also nears an inflection point of zero if the waveform is near sinusoidal and has a zero mean value. This means that the forcing RHS, $F(t)$, has a greater influence on the direction that the waveform will take as it crosses the axis. The premise is that the biennial forcing factor, $ A sin(\pi t ) $, has to be metastable to begin with since the earth favors neither odd years nor even years. And so the question becomes -- is this zero-crossing the point where the biennial oscillation could change from odd/even pairing in years to even/odd (or vice versa) ? And if there is a bias in the metastability, that it could change back on the next cycle, because one pairing is slightly favored ?Paul we need John's input on these equations, I know you are dang smart but John's opinions are vastly important. In particular I am interested, per John's earlier write up, to have non-smooth F(t).
We are having a CDF problem with version 10, so I am not releasing the code above for you.
D
Paul we need John's input on these equations, I know you are dang smart but John's opinions are vastly important. In particular I am interested, per John's earlier write up, to have non-smooth F(t). We are having a CDF problem with version 10, so I am not releasing the code above for you. DHere is a post on finding a second mode in the ENSO time series, in addition to the biennial forcing mode
http://contextearth.com/2014/09/23/two-modes-to-enso-variability/
The paper that the analysis aligns well with is
Kim, Kwang-Yul, James J O’Brien, and Albert I Barcilon. “The Principal Physical Modes of Variability over the Tropical Pacific.” Earth Interactions 7, no. 3 (2003): 1–32. -- PDF
After having read this paper, I was amazed that it only has 8 citations and those were mainly self-directed. Kim et al are one of the few papers where one finds references to the biennial periodicity of ENSO.
Here is a post on finding a second mode in the ENSO time series, in addition to the biennial forcing mode <http://contextearth.com/2014/09/23/two-modes-to-enso-variability/> The paper that the analysis aligns well with is Kim, Kwang-Yul, James J O’Brien, and Albert I Barcilon. “The Principal Physical Modes of Variability over the Tropical Pacific.” Earth Interactions 7, no. 3 (2003): 1–32. -- [PDF](http://journals.ametsoc.org/doi/pdf/10.1175/1087-3562%282003%29007%3C0001%3ATPPMOV%3E2.0.CO%3B2) After having read this paper, I was amazed that it only has 8 citations and those were mainly self-directed. Kim et al are one of the few papers where one finds references to the biennial periodicity of ENSO.For anybody who want to get nearer the road than any of us are in this algorithm chucking game (which I think possibly everybody is forced to play, given the complexity), there is this CSEOF Fortran code (apparently derived from that used by Kim et al.).
I don't have the mileage to download the datasets so I'm not trying it :).
Anyway I just ground my way through the just so, convoluted text of this paper trying to extract any strong statements I could. I wanted to compare its description of El Nino and techniques with Ian Ross's description which is among the clearest I've come across. I always start trying to substitute the greek squiggles with v. bad pseudocode due to bad maths fu.
My notes should be here
For anybody who want to get nearer the road than any of us are in this algorithm chucking game (which I think possibly everybody is forced to play, given the complexity), there is this [CSEOF Fortran](http://www.cgd.ucar.edu/cas/software/SUBR_CYCLOSTATIONARY_WD.f90.html) code (apparently derived from that used by Kim et al.). I don't have the mileage to download the datasets so I'm not trying it :). Anyway I just ground my way through the just so, convoluted text of this paper trying to extract any strong statements I could. I wanted to compare its description of El Nino and techniques with Ian Ross's description which is among the clearest I've come across. I always start trying to substitute the greek squiggles with v. bad pseudocode due to bad maths fu. My notes should be [here](https://dl.dropboxusercontent.com/u/61621163/Docs/QBOKim.md)Jim, Thanks for that note summary. I agree that the Kim paper is a tough slog, with a perhaps too facile "just so" narrative, but I have learned that this is quite typical for climate science papers.
Yet the authors did somehow pull this biennial odd/even periodicity out of the muck, which is what intrigued me since I am seeing this effect as well.
Part of what sets my approach apart is that I am not using any of the conventional climate science methods -- I am not comfortable at all with EOFs for instance. I base my entire premise on the fact that we have this humongous body of water in the Pacific and trying to understand what happens when it starts to slosh back and forth. There is very little viscosity in this motion from what I understand, so that it should oscillate once it gets set in motion. And the simplest first-order inviscid perturbation to the oscillation has the form of the Mathieu differential equation.
There could be two major modes of the oscillation, separable I am guessing in orthogonal directions aligned with the geographic axis.
In the past I had noticed that one of the authors, O'Brien a highly regarded emeritus climatologist, had written several papers (again low-cited) on solving the hydrodynamics problem with a co-author D.Muller.
D. Müller and J. J. O’Brien, Shallow water waves on the rotating sphere, Phys. Rev. E 51, 4418 , 1995
As I recall, they had some references to a Mathieu equation formulation, but did not pursue it. I will dig this paper out in my PDF stash. EDIT: PDF here
Amazing how these guys can keep any of this straight in their minds.
---------------- EDIT ADDED BELOW ----------------------------
This is an extract from the Muller & O'Brien paper referenced above
Because of the spheroid nature of the earth, they punted on the problem. They may be formally correct that one can not apply a full 2D treatment to a spheroidal problem, but that does not mean we should avoid a first-order perturbation approach to at least capture the quasi-periodidicity.
There is a gaping hole in the research on this topic. Nothing below the horribly complex.
Jim, Thanks for that note summary. I agree that the Kim paper is a tough slog, with a perhaps too facile "just so" narrative, but I have learned that this is quite typical for climate science papers. Yet the authors did somehow pull this biennial odd/even periodicity out of the muck, which is what intrigued me since I am seeing this effect as well. Part of what sets my approach apart is that I am not using any of the conventional climate science methods -- I am not comfortable at all with EOFs for instance. I base my entire premise on the fact that we have this humongous body of water in the Pacific and trying to understand what happens when it starts to slosh back and forth. There is very little viscosity in this motion from what I understand, so that it should oscillate once it gets set in motion. And the simplest first-order inviscid perturbation to the oscillation has the form of the Mathieu differential equation. There could be two major modes of the oscillation, separable I am guessing in orthogonal directions aligned with the geographic axis. In the past I had noticed that one of the authors, O'Brien a highly regarded emeritus climatologist, had written several papers (again low-cited) on solving the hydrodynamics problem with a co-author D.Muller. D. Müller and J. J. O’Brien, Shallow water waves on the rotating sphere, Phys. Rev. E 51, 4418 , 1995 As I recall, they had some references to a Mathieu equation formulation, but did not pursue it. I will dig this paper out in my PDF stash. EDIT: [PDF here](http://coaps.fsu.edu/bios/95-4.pdf) Amazing how these guys can keep any of this straight in their minds. ---------------- EDIT ADDED BELOW ---------------------------- This is an extract from the Muller & O'Brien paper referenced above  Because of the spheroid nature of the earth, they punted on the problem. They may be formally correct that one can not apply a full 2D treatment to a spheroidal problem, but that does not mean we should avoid a first-order perturbation approach to at least capture the quasi-periodidicity. There is a gaping hole in the research on this topic. Nothing below the horribly complex.Hello Paul
I think the problem is the Hairy Ball Theorem:
Hairy Ball Theorem
And this comes from the standard formulation of any GLOBAL waves on 2-Sphere, however if you used wavelets with compact support or vanishing support (as t->infinity) then you have local waves (not global) and you might be able to find localized solutions.
That was why I was asking you to include e^-r^2 in your wave-forms.
Hello Paul I think the problem is the Hairy Ball Theorem: [Hairy Ball Theorem](http://en.wikipedia.org/wiki/Hairy_ball_theorem) And this comes from the standard formulation of any GLOBAL waves on 2-Sphere, however if you used wavelets with compact support or vanishing support (as t->infinity) then you have local waves (not global) and you might be able to find localized solutions. That was why I was asking you to include e^-r^2 in your wave-forms.If you use the 3D volumetric data for the planet or 3-sphere, then Hairy Ball Theorem does not hold, you should be able to have global waves in the volumetric data, I really really really need John's comments on these since I am at undergrad level with manifold theory.
Dara
If you use the 3D volumetric data for the planet or 3-sphere, then Hairy Ball Theorem does not hold, you should be able to have global waves in the volumetric data, I really really really need John's comments on these since I am at undergrad level with manifold theory. DaraHere are some wavelet formulations to expand our imagination, I hope this is not crackpot math or physics:
Making electromagnetic wavelets
Eigenwavelets of the Wave Equation
Wavelet Method for Numerical Solution of Wave Equation with Neumann Boundary Conditions
Here are some wavelet formulations to expand our imagination, I hope this is not crackpot math or physics: [Making electromagnetic wavelets](http://arxiv.org/pdf/math-ph/0402006v3.pdf) [Eigenwavelets of the Wave Equation](http://arxiv.org/pdf/math-ph/0408044v3.pdf) [Wavelet Method for Numerical Solution of Wave Equation with Neumann Boundary Conditions](http://www.iaeng.org/publication/IMECS2011/IMECS2011_pp1513-1516.pdf)Thanks for load of reading guys,
I can't keep all this in my head so I'm having to summarise the El Nino paper refs. I cited above before I could possibly get a straight and up-to-date story.
Paul wrote:
Have you commented on this somewhere?
I agree that a volumetric model of the Pacific warm pool and its shearing of teleconnections seems like a necessary condition of a physical explanation. Do you know what arguments might be or are used to justify using the shallow water equations for changes in thermocline depth.
I took it as an implication of the Ludescher et al. paper that meridional effects are important in convergence of Rossby waves.
Wrt. neural networks I've only got as far as a list of the problems associated with trying to use them on 3D data.
Cheers
Thanks for load of reading guys, I can't keep all this in my head so I'm having to summarise the El Nino paper refs. I cited above before I could possibly get a straight and up-to-date story. Paul wrote: > I am not comfortable at all with EOFs for instance. Have you commented on this somewhere? I agree that a volumetric model of the Pacific warm pool and its shearing of teleconnections seems like a necessary condition of a physical explanation. Do you know what arguments might be or are used to justify using the shallow water equations for changes in thermocline depth. I took it as an implication of the Ludescher et al. paper that meridional effects are important in convergence of Rossby waves. Wrt. neural networks I've only got as far as a list of the problems associated with trying to use them on 3D data. CheersBe careful about pontifications on neural networks. Neural Networks are non-linear adaptive approximators to any smooth function from R^n -----> R^m, I try to post the paper which does the math technicalities for this claim.
So many problems with NN are actually people's lack of education and ability and proper software dev acumen to deliver, so instead of saying "hey! I am inept to do NN" they claim troubles with the algorithm itself.
Any approximator or regressor has limitation otherwise it won't be an approximation, but bear in mind that people who write these papers are not always most competent.
There are also such people as CRACKPOT COMPUTER SCIENTISTS :) and lots of them
Dara
>Wrt. neural networks I’ve only got as far as a list of the problems associated with trying to use them on 3D data. Be careful about pontifications on neural networks. Neural Networks are non-linear adaptive approximators to any smooth function from R^n -----> R^m, I try to post the paper which does the math technicalities for this claim. So many problems with NN are actually people's lack of education and ability and proper software dev acumen to deliver, so instead of saying "hey! I am inept to do NN" they claim troubles with the algorithm itself. Any approximator or regressor has limitation otherwise it won't be an approximation, but bear in mind that people who write these papers are not always most competent. There are also such people as CRACKPOT COMPUTER SCIENTISTS :) and lots of them DaraJim I give you an example. In Toronto area 10 years ago some PhD guy in a bank tried to use NN for forecasting stocks or some app like that. He failed, because he was not good at computing nor math and got fired perhaps for other reasons. Now the entire banking industry in Toronto quotes that experience as fault with all machine learning algorithms.
D
Jim I give you an example. In Toronto area 10 years ago some PhD guy in a bank tried to use NN for forecasting stocks or some app like that. He failed, because he was not good at computing nor math and got fired perhaps for other reasons. Now the entire banking industry in Toronto quotes that experience as fault with all machine learning algorithms. DDara,
Thanks for the offer to dig out a maths explanation, especially as the hessian-free deep learning papers I've been reading apply only to 2D image recognition.
Dara, Thanks for the offer to dig out a maths explanation, especially as the hessian-free deep learning papers I've been reading apply only to 2D image recognition.This result is stronger than what I quoted, it is for Borel Measurable Functions:
Multilayer Feeforward Networks are Universal Approximators
He does not use the word Neural for one reason that the feedforward networks could do the job for classical neural network definitions and some neural network variations might not satisfy the conditions for his theorem.
But for the same what we do his theorem is applicable.
Dara
This result is stronger than what I quoted, it is for Borel Measurable Functions: [Multilayer Feeforward Networks are Universal Approximators ](http://files.lossofgenerality.com/approx_hornik.pdf) He does not use the word Neural for one reason that the feedforward networks could do the job for classical neural network definitions and some neural network variations might not satisfy the conditions for his theorem. But for the same what we do his theorem is applicable. DaraDara said:
I don't have any spatial dependence in my analysis. I treat the oscillations as standing waves, with the spatial component assumed to separate out (of which I proceed to ignore for the moment).
This may or may not be correct but what I am looking for is physics-based heuristics -- remember that my goal is to be able to project the SOI for the CSALT model and also to help predict El Ninos, not necessarily reproduce the 3D dynamical flow. I am pragmatic in that regard -- simple physics and first-order approximations is my game.
This is the Mathieu differential equation I am dealing with
$ f''(t) + [a + q cos(\omega t)] f(t) = F(t) $
The perturbation is the q term on the LHS. If it wasn't for this, the result would be a sinusoid convolved with whatever the F(t) is, which is likely a sinusoidal forcing as well. As q gets bigger, the waveform starts to become more erratic and this is I am presuming represents the tide gauge fluctuations to first-order. This also leads to ENSO, as sloshing sea-level changes cause shoaling down into the thermocline, which can bring cold water to the surface.
That's the basic premise, no spatial coverage for now. I am trying to see how far I can carry this formulation, and based on the results of comment #23 , to me it looks very promising.
Cheers as always, because where else can we get this kind of discussion going?
Jim asked
I really never said outright that I wasn't working the EOF approach, except on my blog -- http://contextearth.com/2014/09/13/azimuth-project-on-el-ninos/
I guess what I don't understand is how deconstructing one unknown into two or more unknowns makes the problem any easier. Some of the EOFs that I have seen are more complex than the original waveform. So we still have to understand the elements of the deconstruction to make predictions! Or am I missing something?
Yet perhaps that is what I am doing by adding a second mode in addition to the dominant tide gauge model. And from my assuming that these are standing waves, I sweep under the rug any spatial dependence, which means I don't have to officially call these EOFs.
Dara said: "That was why I was asking you to include e^-r^2 in your wave-forms." I don't have any spatial dependence in my analysis. I treat the oscillations as standing waves, with the spatial component assumed to separate out (of which I proceed to ignore for the moment). This may or may not be correct but what I am looking for is physics-based heuristics -- remember that my goal is to be able to project the SOI for the CSALT model and also to help predict El Ninos, not necessarily reproduce the 3D dynamical flow. I am pragmatic in that regard -- simple physics and first-order approximations is my game. This is the Mathieu differential equation I am dealing with $ f''(t) + [a + q cos(\omega t)] f(t) = F(t) $ The perturbation is the *q* term on the LHS. If it wasn't for this, the result would be a sinusoid convolved with whatever the *F(t)* is, which is likely a sinusoidal forcing as well. As *q* gets bigger, the waveform starts to become more erratic and this is I am presuming represents the tide gauge fluctuations to first-order. This also leads to ENSO, as sloshing sea-level changes cause shoaling down into the thermocline, which can bring cold water to the surface. That's the basic premise, no spatial coverage for now. I am trying to see how far I can carry this formulation, and based on the results of comment #23 , to me it looks very promising. Cheers as always, because where else can we get this kind of discussion going? --- Jim asked "Have you commented on this somewhere?" I really never said outright that I wasn't working the EOF approach, except on my blog -- <http://contextearth.com/2014/09/13/azimuth-project-on-el-ninos/> I guess what I don't understand is how deconstructing one unknown into two or more unknowns makes the problem any easier. Some of the EOFs that I have seen are more complex than the original waveform. So we still have to understand the elements of the deconstruction to make predictions! Or am I missing something? Yet perhaps that is what I am doing by adding a second mode in addition to the dominant tide gauge model. And from my assuming that these are standing waves, I sweep under the rug any spatial dependence, which means I don't have to officially call these EOFs.Dungarvan Bee Keepers association, waterford county Ireland, :)
>Cheers as always, because where else can we get this kind of discussion going? Dungarvan Bee Keepers association, waterford county Ireland, :)Here is another view of how the second mode can improve the fit. This graph is mapping Sydney Harbor tide data to SOI (blue dots). The Eureqa optimizer definitely wants to smooth over 12 months (to get rid of annual and biannual fluctuations in tide gauge readings). It also subtracts out a background of 37 months via a smoothing average (sma) to remove trends and longer scale fluctuations. That gives a fit of size complexity = 15. But to fill in details it adds in a modulated sine wave to increase the size complexity to 60.
Here is another view of how the second mode can improve the fit. This graph is mapping Sydney Harbor tide data to SOI (blue dots). The Eureqa optimizer definitely wants to smooth over 12 months (to get rid of annual and biannual fluctuations in tide gauge readings). It also subtracts out a background of 37 months via a smoothing average (sma) to remove trends and longer scale fluctuations. That gives a fit of size complexity = 15. But to fill in details it adds in a modulated sine wave to increase the size complexity to 60. Hello Paul
I sent you the .nb files for the code, the CDF breaks in both v9 and v10 of Mathematica and techsupport was able to partially fix some problems today
Dara
Hello Paul I sent you the .nb files for the code, the CDF breaks in both v9 and v10 of Mathematica and techsupport was able to partially fix some problems today DaraThanks, see what I can do with it tonight.
Thanks, see what I can do with it tonight.Timely paper on the pure mathematics of sloshing. This is a pre-print of a recent submission.
Dubois, François, and Dimitri Stoliaroff. "Coupling Linear Sloshing with Six Degrees of Freedom Rigid Body Dynamics." arXiv preprint arXiv:1407.1829 (2014).
Imagine a volume with an equilibrium surface and then agitating it:
They consider a global vector, $q(t)$ for the sloshing representing elongation, rotation, and center of gravity and then elegantly describe it as this 2nd-order DiffEq (in the text as Eq. 84)
$ M \cdot \frac{d^2q}{dt^2} + K \cdot q =F(t) $
Looks familiar from the Mathieu DiffEq that I use above in #38, but is a vector representation and so $K$ is a matrix. If $K$ has a time dependence, then the quasi-periodic Mathieu/Hill dynamics emerge.
Timely paper on the pure mathematics of sloshing. This is a pre-print of a recent submission. Dubois, François, and Dimitri Stoliaroff. *"Coupling Linear Sloshing with Six Degrees of Freedom Rigid Body Dynamics."* arXiv preprint [arXiv:1407.1829](http://arxiv.org/pdf/1407.1829) (2014). Imagine a volume with an equilibrium surface and then agitating it:  They consider a global vector, $q(t)$ for the sloshing representing elongation, rotation, and center of gravity and then elegantly describe it as this 2nd-order DiffEq (in the text as Eq. 84) $ M \cdot \frac{d^2q}{dt^2} + K \cdot q =F(t) $ Looks familiar from the Mathieu DiffEq that I use above in #38, but is a vector representation and so $K$ is a matrix. If $K$ has a time dependence, then the quasi-periodic Mathieu/Hill dynamics emerge.This is another synopsis of what the sloshing modeling results are showing.
The tide gauge numbers are revealing a forced sloshing oscillation with a primary forcing period of two years. This is panel A.
The SOI is showing the same forced oscillation in addition to an unforced sloshing oscillation with no preferred period. This is panel B.
So the tide gauge and SOI share a common forcing (Panel C overlaid curves) but the SOI has another component (Panel C green) based on perhaps a deeper oscillation not emergent as a sea-level change. This could be an oscillation in the thermocline?
All this data is simulated based on applying a Mathieu DiffEq
This is another synopsis of what the sloshing modeling results are showing. The tide gauge numbers are revealing a forced sloshing oscillation with a primary forcing period of two years. This is panel A. The SOI is showing the same forced oscillation in addition to an unforced sloshing oscillation with no preferred period. This is panel B. So the tide gauge and SOI share a common forcing (Panel C overlaid curves) but the SOI has another component (Panel C green) based on perhaps a deeper oscillation not emergent as a sea-level change. This could be an oscillation in the thermocline? All this data is simulated based on applying a Mathieu DiffEq This is another round of error minimization. Instead of using the full SOI measure, I decided to compare the Sydney tide gauge data (panel A) to just the Darwin atmospheric pressure anomaly (panel B).
The underlying model forcing between the two is identical, apart from a slight time-scale shift. But the Darwin model has an additional unforced component as shown in panel C of #44.
I am pulling out all the stops on the analysis with this teleconnection. The correlation coefficient between Darwin and model is almost 61% which makes it higher than the correlation between Darwin and -Tahiti. This you can see below
The nature of the characterized behavior is that one will never whether a feature detail is part of the underlying phenomena or some error or disturbance. This will always knock the correlation coefficient down and I am guessing will prevent the correlation coefficient from ever reaching unity.
This is another round of error minimization. Instead of using the full SOI measure, I decided to compare the Sydney tide gauge data (panel A) to just the Darwin atmospheric pressure anomaly (panel B). The underlying model forcing between the two is identical, apart from a slight time-scale shift. But the Darwin model has an additional unforced component as shown in panel C of #44.  I am pulling out all the stops on the analysis with this teleconnection. The correlation coefficient between Darwin and model is almost 61% which makes it higher than the correlation between Darwin and -Tahiti. This you can see below  The nature of the characterized behavior is that one will never whether a feature detail is part of the underlying phenomena or some error or disturbance. This will always knock the correlation coefficient down and I am guessing will prevent the correlation coefficient from ever reaching unity.I am starting to lean hard on the Mathieu DiffEq sloshing model to see how it can break.
The top graph (and error curve) is an almost 100 year fit of the sloshing model to the Darwin portion of the SOI. Then I decided to push it a little more and make it at least 100 years by getting the last several months of data, up to July of this year. Notice how in adding a few more months, the sudden downward trend was predicted.
I admit that this is perilously close to overfitting, but the indications are that this model is getting perilously close to describing the sloshing of the equatorial Pacific waters.
That downward spike is also an indicator of El Nino, btw. The last big ones were 1982, 1987, and 1998.
I am starting to lean hard on the Mathieu DiffEq sloshing model to see how it can break. The top graph (and error curve) is an almost 100 year fit of the sloshing model to the Darwin portion of the SOI. Then I decided to push it a little more and make it at least 100 years by getting the last several months of data, up to July of this year. Notice how in adding a few more months, the sudden downward trend was predicted.  I admit that this is perilously close to overfitting, but the indications are that this model is getting perilously close to describing the sloshing of the equatorial Pacific waters. That downward spike is also an indicator of El Nino, btw. The last big ones were 1982, 1987, and 1998.Wonderful!
CDF I wrote for this has failed in botn v9 and v10 of Mathematica just got an email from Wolfram techsupport. The inputfield where the equation is defined has a bug.
So I am looking into a workaround
Dara
Hello Paul Wonderful! CDF I wrote for this has failed in botn v9 and v10 of Mathematica just got an email from Wolfram techsupport. The inputfield where the equation is defined has a bug. So I am looking into a workaround DaraWhat I think I am up against is determining what the "true" data is. For SOI, I can use Darwin, or Tahiti, or the difference between the two. But neither of these is likely right, since the correlation between Darwin and Tahiti is only -0.55. So unless Darwin is more correct than Tahiti, or that doing the difference is a noise reducer, what the optimization will try to fit to is an incorrect view of the true ENSO process, and any predictions will tend to disperse from the actual result the further out that they are made. In other words, getting a 0.9 model correlation with the Darwin data, which has only a correlation of -0.55 with Tahiti has to be an example of over-fitting, unless the small chance of Darwin being a perfect proxy for the ENSO sloshing.
In this case the model correlation to Darwin is 0.67, which is better than 0.55, which makes it a decent fit if Darwin happens to be more correct that Tahiti as an ENSO measure.
That brings up another question:
Is Nino3.4 better than SOI in extracting the true measure of what we should be fitting the model to? Or is MEI better?
Yet that said, if this approach can only predict for a few years, it will still be quite an achievement in terms of laying to rest the idea that ENSO is a purely chaotic process.
Dara, good luck on getting the bugs fixed. I haven't run into anything so far, except for an occasional wild ride that Mathematica takes the computer on -- using up all available memory.
What I think I am up against is determining what the "true" data is. For SOI, I can use Darwin, or Tahiti, or the difference between the two. But neither of these is likely right, since the correlation between Darwin and Tahiti is only -0.55. So unless Darwin is more correct than Tahiti, or that doing the difference is a noise reducer, what the optimization will try to fit to is an incorrect view of the true ENSO process, and any predictions will tend to disperse from the actual result the further out that they are made. In other words, getting a 0.9 model correlation with the Darwin data, which has only a correlation of -0.55 with Tahiti has to be an example of over-fitting, unless the small chance of Darwin being a perfect proxy for the ENSO sloshing. In this case the model correlation to Darwin is 0.67, which is better than 0.55, which makes it a decent fit if Darwin happens to be more correct that Tahiti as an ENSO measure. That brings up another question: Is Nino3.4 better than SOI in extracting the true measure of what we should be fitting the model to? Or is MEI better? Yet that said, if this approach can only predict for a few years, it will still be quite an achievement in terms of laying to rest the idea that ENSO is a purely chaotic process. --- Dara, good luck on getting the bugs fixed. I haven't run into anything so far, except for an occasional wild ride that Mathematica takes the computer on -- using up all available memory.Look at the latest SST numbers coming out
Record warmth according to the ocean surface temperature readings. June, July, August are the 3 highest ever as you can see from the circles.
Data http://www.metoffice.gov.uk/hadobs/hadsst3/data/HadSST.3.1.1.0/diagnostics/HadSST.3.1.1.0_monthly_globe_ts.txt
Look at the latest SST numbers coming out  Record warmth according to the ocean surface temperature readings. June, July, August are the 3 highest ever as you can see from the circles. Data <http://www.metoffice.gov.uk/hadobs/hadsst3/data/HadSST.3.1.1.0/diagnostics/HadSST.3.1.1.0_monthly_globe_ts.txt>Paul I am on the road, should be at my machine tonight
Dara
Paul I am on the road, should be at my machine tonight Dara