Options

I've cracked the ENSO code and an introduction on me, Per Strandberg

edited May 2015 in Azimuth Forum

I got the advice to come over to this forum from Reference Frame And I have something to tell! I have cracked the ENSO code. I now know about its underlying drivers. I know this sounds crazy, but that is what I have done!

I’ve used an ANN or Artificial Neural Network program I’ve created, which I used to play around with, with solar cycles, tidal effects and sea current indexes. I discovered the connection with tidal and solar forcing on ENSO a while back, which I have presented to others at a seminar. At that time I used this graph. image

I was preparing to write a paper on this and to upgrade the data up to the end of 2014 when, last month I used a new approach. The result I then got was extraordinary. Look at this! image

Wow!

The training period is from 1979 until 2005, the testing period is from 2005 until 2012. The rest is forecast. The data is based on the MEI ENSO index. The only inputs I use are from tidal gravitational anomalies and from Kp, Ap and variations in solar wind parameters. Most so called skeptics, refer to the Svensmark’s effect when it comes to the connection between global temperature variations and the Sun. Critics from the other camp often point that this effect is too weak. I think they are right on that. How much influence from ENSO variations which contribute directly to variations in the global mean temperature anomaly is up to others to figure out, but I think it should be somewhere between 30 and up to 80 % of the effect, making any input from human caused temperature impact much smaller that is assumed by the IPCC.

I can show with empirical evidence that one of the main causes of recent temperature changes, directly can be attributed to electromagnetic solar driven ENSO variations. If one take into account this ENSO effect and quality questions related to the main surface temperature station by NASA, NOAA and from CRU Hadley Centre, then there is not much space left for any AGW influence on recent temperature changes, irrespective if such effect exists or not. The temperature variation on the global temperature which is generated by ENSO is caused by variations in sun’s electromagnetic activity which is then blurred somewhat by the tidal gravitational effect. When for example the Ap index is weak there is a tendency for fewer and weaker El Niños which leads to cooling. The opposite happens when the Ap index is higher.

I realize that what I have, is an atomic bomb set to explode in the face of climate scientists, ENSO researchers and of course ultimately this is going to be the (beginning) of the end of the CAGW hysteria. I would also point out that the mechanisms I found is relative simple and should therefore be easy for others to confirm and to replicate. And I hope, also by you.

Here is my forecast from now up to 2020. image

As you can see according to this forecast the current ENSO value should peek with an El Niño at the end of the current NH summer. It is going to drop after that and continue to drop into a deep La Niña at the NH winter of 2016/17. Time will tell if I’m right.

Here is the detail of the test period up to 2012 and forecast up to 2015. image

As you can see there are some anomalies at the end of the period which I attribute to the recent period with Modoki El Niño. The MEI index from NOAA is not well defined for that.

I haven’t released the detail of the mechanism by which the Sun and the Moon drives ENSO yet, but I am going to this publicly. But, because of the magnitude of what I have found I need step back and do some brainstorming and to think through how to proceed.

Should I just send a description to important blogs and websites and then maybe because I need to continue to work with this by investigate with my ANN NINO1+2. NINO3+4, QBO; SOI, LOD, trade wind index and so on? Maybe I should use crowdsouring to finance my future work? Should I try to publish in a scientific paper, where? Tips on anyone I can co-operate with? Of course my claim to have solved the driving mechanisms of ENSO may seem rather extreme and it is OK to be skeptical. I mean I would if I were you. So let it be hypothetical. What would you do if you have discovered what the drivers are of ENSO and you have the data and mechanism to back it up?

My name is Per Strandberg and I have an M.Sc degree in Physics and electronics. I became interested in the climate question a while back and because I have experience with ANN and climate data is freely available I started to play around with this data. Never in my wildest dream did I ever think that I would solve the ENSO mystery, when so many others have failed.

Comments

  • 1.

    Per, welcome and glad that you took up the invitation.

    As a suggestion, taake a look at what Dara was doing in this forum with regards to NN :

    http://forum.azimuthproject.org/discussion/1534/k-nn-regression-el-nino-3-4-anomalies-index

    There is some similarity between what you have done and what Dara tried to accomplish. Unfortunately, he could not get much accuracy, even a few months in advance.

    If what you have found is a useful predictor that you can use in a NN feed-forrward forcasting scheme, that would be very interesting to understand.

    Comment Source:Per, welcome and glad that you took up the invitation. As a suggestion, taake a look at what Dara was doing in this forum with regards to NN : http://forum.azimuthproject.org/discussion/1534/k-nn-regression-el-nino-3-4-anomalies-index There is some similarity between what you have done and what Dara tried to accomplish. Unfortunately, he could not get much accuracy, even a few months in advance. If what you have found is a useful predictor that you can use in a NN feed-forrward forcasting scheme, that would be very interesting to understand.
  • 2.

    Per said:

    "Maybe I should use crowdsouring to finance my future work?"

    If it is as straightforward as you say, I don't think you will need funding.

    As a suggestion, if you are indicating that the solar and tidal parameters are really what is driving the behavior of ENSO, then you should be able to generate a time series profile starting at, say 1900, and follow it through to the current time. This should rely only on one set of initial conditions, in this case at 1900. The boundary conditions of the roughly periodic nature of the sun and tides should be all it takes to keep the model inline with the data. That's the nature of a deterministic solution.

    My concern is that these neural network type approaches require constant updating of the initial conditions throughout the time-series, otherwise they start to lose coherence. So you are assuming that some stochastic or other unknown factor is causing the behavior to get out-of-phase and so you need the continuous update to force a regular re-alignment.

    So think in terms of tidal charts. Those are very deterministic and only require a single point to get in sync and will be good for quite a long while. So my question is: can you do something like that? I am just trying to get a feel of what you have accomplished.

    Comment Source:Per said: > "Maybe I should use crowdsouring to finance my future work?" If it is as straightforward as you say, I don't think you will need funding. As a suggestion, if you are indicating that the solar and tidal parameters are really what is driving the behavior of ENSO, then you should be able to generate a time series profile starting at, say 1900, and follow it through to the current time. This should rely only on one set of initial conditions, in this case at 1900. The boundary conditions of the roughly periodic nature of the sun and tides should be all it takes to keep the model inline with the data. That's the nature of a deterministic solution. My concern is that these neural network type approaches require constant updating of the initial conditions throughout the time-series, otherwise they start to lose coherence. So you are assuming that some stochastic or other unknown factor is causing the behavior to get out-of-phase and so you need the continuous update to force a regular re-alignment. So think in terms of tidal charts. Those are very deterministic and only require a single point to get in sync and will be good for quite a long while. So my question is: can you do something like that? I am just trying to get a feel of what you have accomplished.
  • 3.
    edited May 2015

    Hi Per,

    The two most immediate ways I see to get some traction for your work

    1) Put your code & data on github or similar. This make it easier to reproduce your results and how the results vary with tweaks. It also makes it way more clear what your methods an inputs are than any verbal description can.

    2) Try extending the period if you can get the data. Currently you test set is 10 years which is relatively short for observing persistent climate change. If your training data started a few decades earlier you could get several decades of test data, while keeping the same amount of training data. As is your prediction data seems to start to diverge from observation after 2011. It would be interesting to see if it diverges completely or remains at least partially correlated. Even if it diverges completely, a near perfect prediction over the first 6 years is extremely impressive, but one needs to see the code and data to really understand what is really being predicted from what, in order to be certain that it is really as good as it looks.

    Comment Source:Hi Per, The two most immediate ways I see to get some traction for your work 1) Put your code & data on github or similar. This make it easier to reproduce your results and how the results vary with tweaks. It also makes it way more clear what your methods an inputs are than any verbal description can. 2) Try extending the period if you can get the data. Currently you test set is 10 years which is relatively short for observing persistent climate change. If your training data started a few decades earlier you could get several decades of test data, while keeping the same amount of training data. As is your prediction data seems to start to diverge from observation after 2011. It would be interesting to see if it diverges completely or remains at least partially correlated. Even if it diverges completely, a near perfect prediction over the first 6 years is extremely impressive, but one needs to see the code and data to really understand what is really being predicted from what, in order to be certain that it is really as good as it looks.
  • 4.

    Good point Daniel,

    There is nothing particularly special about the MEI ENSO index other than it rolls in more measures, yet it also has the disadvantage is a much shorter record than the others available. It is easy to find ENSO data that goes back to 1880 and earlier -- e.g. the SOI.

    And then if you want to go for proxy records, try out the coral proxy data, which goes back hundreds of years: http://forum.azimuthproject.org/discussion/1451/enso-proxy-records#latest

    If as you say that the ENSO is governed by tides and solar cycles then you should be able to use the log-term historical records; the tidal movements are well-characterized over time and enough detail on solar cycles is available from sunspot data. This should be more than enough to test the model against different time intervals.

    Comment Source:Good point Daniel, There is nothing particularly special about the MEI ENSO index other than it rolls in more measures, yet it also has the disadvantage is a much shorter record than the others available. It is easy to find ENSO data that goes back to 1880 and earlier -- e.g. the SOI. And then if you want to go for proxy records, try out the coral proxy data, which goes back hundreds of years: http://forum.azimuthproject.org/discussion/1451/enso-proxy-records#latest If as you say that the ENSO is governed by tides and solar cycles then you should be able to use the log-term historical records; the tidal movements are well-characterized over time and enough detail on solar cycles is available from sunspot data. This should be more than enough to test the model against different time intervals.
  • 5.

    As I said you have to wait until I publish my work to learn in more detail of what I found, need to think this through.

    At that time it should be easy for others to collaborate my result, either by building an ANN or simply by doing some statistical analysis. Sure, it is a good idea to go back in time. I can’t use sunspot data as input because there is no good correlation between that and ENSO. The reason I don’t go back beyond 1979 is that I use solar wind data which don’t go back before that and there are also issues with gaps in the data. I also use Ap and Kp. I think Ap goes back to about 1930. By using Ap and tidal data from back then, the result I would expect should be overwhelming in supporting my claims. I think the modoki or near modoki state has a lot to do with the divergence in the last years. The El Niño we are going in to right now is a real El Niño with warming alongside the South American coast. So let’s see if it now gets into phase with my prediction. I’m not working in academia, but rather I have had this as a hobby as I became interested in the climate question a while back. I reacted to the bizarre level of hysteria that is going on in this field. With my background in physics I became quite appalled at what I saw. Whenever I looked at real data I found close correlation with variations in temperature and changes in solar activity. I became aware of a curious correlation between variations in LOD, Length of day and global temperature after listen to this video.

    I then found out that it was quite easy to find related data on the Internet, which I put into an ANN I built. From that I could make estimations of the strengths of different in-signal’s impact on the global temperature. I quickly found out that variation in LOD was almost as strong as ENSO which was then followed by solar data. I later found out that LOD and ENSO are closely related, so in fact changes in the LOD signal affects the temperature through ENSO.

    I then turn my attention to ENSO and just followed where the data led me and eventually I arrived to where I am now with my data. No criticism of you guys, but when I look on the work you are doing on ENSO it looks for me similar to a group of medieval scientists equipped with computer and with access to advanced mathematical techniques trying to put in yet another circles into the Ptolemaic explanation of the movements of the planets. Of course as Copernicus realized, the Earth and the planets are circling the Sun. What I have discovered is the equivalent of Newton’s laws for planetary movements for ENSO. Well, it is not a new law, but rather it is the explanations for the mechanisms for how the solar and tidal drives ENSO variability. In fact the mechanisms for ENSO are rather simple and therefore I know my explanation is going to be accepted by most, even by those at the currently highest pecking order of climate science, even if they are doing this kicking and screaming. Eventually they have no other alternative. This bizarre history of CAGW in climate science is soon coming to and end. I think.

    Comment Source:As I said you have to wait until I publish my work to learn in more detail of what I found, need to think this through. At that time it should be easy for others to collaborate my result, either by building an ANN or simply by doing some statistical analysis. Sure, it is a good idea to go back in time. I can’t use sunspot data as input because there is no good correlation between that and ENSO. The reason I don’t go back beyond 1979 is that I use solar wind data which don’t go back before that and there are also issues with gaps in the data. I also use Ap and Kp. I think Ap goes back to about 1930. By using Ap and tidal data from back then, the result I would expect should be overwhelming in supporting my claims. I think the modoki or near modoki state has a lot to do with the divergence in the last years. The El Niño we are going in to right now is a real El Niño with warming alongside the South American coast. So let’s see if it now gets into phase with my prediction. I’m not working in academia, but rather I have had this as a hobby as I became interested in the climate question a while back. I reacted to the bizarre level of hysteria that is going on in this field. With my background in physics I became quite appalled at what I saw. Whenever I looked at real data I found close correlation with variations in temperature and changes in solar activity. I became aware of a curious correlation between variations in LOD, Length of day and global temperature after listen to this video. https://youtu.be/IG_7zK8ODGA I then found out that it was quite easy to find related data on the Internet, which I put into an ANN I built. From that I could make estimations of the strengths of different in-signal’s impact on the global temperature. I quickly found out that variation in LOD was almost as strong as ENSO which was then followed by solar data. I later found out that LOD and ENSO are closely related, so in fact changes in the LOD signal affects the temperature through ENSO. I then turn my attention to ENSO and just followed where the data led me and eventually I arrived to where I am now with my data. No criticism of you guys, but when I look on the work you are doing on ENSO it looks for me similar to a group of medieval scientists equipped with computer and with access to advanced mathematical techniques trying to put in yet another circles into the Ptolemaic explanation of the movements of the planets. Of course as Copernicus realized, the Earth and the planets are circling the Sun. What I have discovered is the equivalent of Newton’s laws for planetary movements for ENSO. Well, it is not a new law, but rather it is the explanations for the mechanisms for how the solar and tidal drives ENSO variability. In fact the mechanisms for ENSO are rather simple and therefore I know my explanation is going to be accepted by most, even by those at the currently highest pecking order of climate science, even if they are doing this kicking and screaming. Eventually they have no other alternative. This bizarre history of CAGW in climate science is soon coming to and end. I think.
  • 6.

    Per, The LOD variations are linked to ENSO for known reasons. The average tropospheric wind speed correlates with ENSO indices and so the changes in angular momentum in moving volumes of air need to be compensated with changes in angular momentum of the earth. And since the LOD deviation is a sensitive measure of angular momentum changes, that all makes consistent sense.

    So, if what you are doing is using LOD to estimate ENSO, this is equivalent to plotting X=X. Yet that would cease to become predictive because nothing is predicting X.

    You might want to clarify at least that aspect of your argument.

    Another point you make is this:

    "At that time it should be easy for others to collaborate my result, either by building an ANN or simply by doing some statistical analysis. "

    What do you mean by this? Can you not take the results from the neural net and specify the fit as a set of parametrically valued factors? I would suggest that this needs to be done otherwise no one has a feel for whether your ANN is massively overfitting. Your R^2 appears very high but the AIC may be very low if too many factors contribute to the fit. The latter would be the statistical analysis you will be faced with. Low AIC scores indicate less than parsimonious explanations. I am faced with this challenge as well and so strive to come up with the simplest model that will track ENSO.

    Comment Source:Per, The LOD variations are linked to ENSO for known reasons. The average tropospheric wind speed correlates with ENSO indices and so the changes in angular momentum in moving volumes of air need to be compensated with changes in angular momentum of the earth. And since the LOD deviation is a sensitive measure of angular momentum changes, that all makes consistent sense. So, if what you are doing is using LOD to estimate ENSO, this is equivalent to plotting X=X. Yet that would cease to become predictive because nothing is predicting X. You might want to clarify at least that aspect of your argument. Another point you make is this: > "At that time it should be easy for others to collaborate my result, either by building an ANN or simply by doing some statistical analysis. " What do you mean by this? Can you not take the results from the neural net and specify the fit as a set of parametrically valued factors? I would suggest that this needs to be done otherwise no one has a feel for whether your ANN is massively overfitting. Your R^2 appears very high but the AIC may be very low if too many factors contribute to the fit. The latter would be the statistical analysis you will be faced with. Low AIC scores indicate less than parsimonious explanations. I am faced with this challenge as well and so strive to come up with the simplest model that will track ENSO.
  • 7.
    edited May 2015

    I must stress that I’m not using LOD in my ANN. I only use non terrestrial input into the ANN and in that I also include Ap and Kp data. While being terrestrial data those parameters are mainly driven by the sun. The reason I mentioned LOD was that this was what draw my attention to the tidal effect. “the changes in angular momentum in moving volumes of air need to be compensated with changes in angular momentum of the earth.” No, it doesn’t!

    The atmosphere is floating above the Earth’s ground and crust and the friction between the atmosphere and the ground is very small. Of course, changes in atmospheric pressure have some affect on sea currents, but that is not coupled to LOD.

    Instead LOD is correlated with ENSO via the tidal effect either through movements of Earth’s crust or/and because the energy is going into the currents of the sea. "At that time it should be easy for others to collaborate my result, either by building an ANN or simply by doing some statistical analysis." What I mean is because I have found out the physical connection between ENSO and the tidal effect, it should therefore be easy to verify this in a simple way from another angel by using linear regression. Not all researchers have a good feel for what an ANN is.

    Much of your questions are going to be resolved when I publish my result.

    Comment Source:I must stress that I’m not using LOD in my ANN. I only use non terrestrial input into the ANN and in that I also include Ap and Kp data. While being terrestrial data those parameters are mainly driven by the sun. The reason I mentioned LOD was that this was what draw my attention to the tidal effect. “the changes in angular momentum in moving volumes of air need to be compensated with changes in angular momentum of the earth.” No, it doesn’t! The atmosphere is floating above the Earth’s ground and crust and the friction between the atmosphere and the ground is very small. Of course, changes in atmospheric pressure have some affect on sea currents, but that is not coupled to LOD. Instead LOD is correlated with ENSO via the tidal effect either through movements of Earth’s crust or/and because the energy is going into the currents of the sea. "At that time it should be easy for others to collaborate my result, either by building an ANN or simply by doing some statistical analysis." What I mean is because I have found out the physical connection between ENSO and the tidal effect, it should therefore be easy to verify this in a simple way from another angel by using linear regression. Not all researchers have a good feel for what an ANN is. Much of your questions are going to be resolved when I publish my result.
  • 8.
    " No, it doesn’t! "

    So you don't believe in conservation of angular momentum?

    Also I have a couple of thoughts:

    (1) Do you use any of the ENSO MEI measure as a feedback in your fitting procedure?

    (2) Further, I assume that you believe that there is no noise in the MEI measure since your fit seems to track all the excursions -- no matter how small they are.

    These two go together, because I can see how you can create a good fit if you apply X=k*X+model and thereby self-correlate.

    Otherwise, if you are applying tidal periods, and since tides have relatively short periods, I can also see how a linear combination of different tides could possibly replicate the fine structure. Is that what you are doing?

    Comment Source:<blockquote>" No, it doesn’t! "</blockquote> So you don't believe in conservation of angular momentum? Also I have a couple of thoughts: (1) Do you use any of the ENSO MEI measure as a feedback in your fitting procedure? (2) Further, I assume that you believe that there is no noise in the MEI measure since your fit seems to track all the excursions -- no matter how small they are. These two go together, because I can see how you can create a good fit if you apply X=k*X+model and thereby self-correlate. Otherwise, if you are applying tidal periods, and since tides have relatively short periods, I can also see how a linear combination of different tides *could* possibly replicate the fine structure. Is that what you are doing?
  • 9.
    nad
    edited May 2015

    The video in 5. is interesting. I haven't done a regional analysis of temperature data, but somehow I would have expected that the bold temperature average I took here would have mostly reflected an average of north american and european temperatures, since finally it seems most stations of the HADCRUT collections are there (and I took the average of all stations). But in the video it is explained that in Europe there is a temperature jump around 1986, moreover the US temperature curve shows also a rise at that time, so this jump should be visible in my data. However it isn't. But then maybe the US and european stations are not that much moreover I don't know what that three year running mean means (header in the curve diagram) and I am not a group of 5 researchers.

    Comment Source:The video in 5. is interesting. I haven't done a regional analysis of temperature data, but somehow I would have expected that the bold temperature average I took <a href="http://en.wikipedia.org/wiki/Witch_trials_in_the_early_modern_period#Sporadic_witch-hunts_after_1750">here</a> would have mostly reflected an average of north american and european temperatures, since finally it seems most stations of the HADCRUT collections are there (and I took the average of all stations). But in the video it is explained that in Europe there is a temperature jump around 1986, moreover the US temperature curve shows also a rise at that time, so this jump should be visible in my data. However it isn't. But then maybe the US and european stations are not that much moreover I don't know what that three year running mean means (header in the curve diagram) and I am not a group of 5 researchers.
  • 10.

    Per said

    "The atmosphere is floating above the Earth’s ground and crust and the friction between the atmosphere and the ground is very small. Of course, changes in atmospheric pressure have some affect on sea currents, but that is not coupled to LOD."

    That is contrary to the current literature. As an example, the relief of the Himalayas provides a significant drag on the atmospheric wind which clearly reveals itself as LOD changes.

    " Not all researchers have a good feel for what an ANN is."

    I admit that I haven't done much with that, but Dara has done NN analysis on this forum, see http://forum.azimuthproject.org/discussion/1534/k-nn-regression-el-nino-3-4-anomalies-index

    I infer that what is needed is inputs from the tidal gravitational anomalies?

    Comment Source:Per said > "The atmosphere is floating above the Earth’s ground and crust and the friction between the atmosphere and the ground is very small. Of course, changes in atmospheric pressure have some affect on sea currents, but that is not coupled to LOD." That is contrary to the current literature. As an example, the relief of the Himalayas provides a significant drag on the atmospheric wind which clearly reveals itself as LOD changes. > " Not all researchers have a good feel for what an ANN is." I admit that I haven't done much with that, but Dara has done NN analysis on this forum, see [http://forum.azimuthproject.org/discussion/1534/k-nn-regression-el-nino-3-4-anomalies-index](http://forum.azimuthproject.org/discussion/1534/k-nn-regression-el-nino-3-4-anomalies-index) I infer that what is needed is inputs from the tidal gravitational anomalies?
  • 11.

    Regarding my comment in 9. I now wanted to find out how much the european temp stations dominate the global average so I now took the average over all stations between:

    40<Latitude <61

    -32<Longitude<+32

    Those are altogether around 400 stations, depending on time. The black and blue curves are threeyearrunningmean temperature anomaly curves where black is the global and blue is the european average. I interpreted the threeyearrunningmean as: $threeyearrunningmean(f(j))= 1/36*\sum_{i=-17}^{19} f(j+i)$. Slightly visible is also the oneyearrunningmean (annual average) curve behind the black threeyearrunningmean curve. The jagged grey curve is the european average temp anomaly curve.

    The CRUTEM4 temperature collection starts in 1701 with Berlin, then in 1706 De Bilt, Netherlands joins in and then you see in 1722 that the "average european temperature" is going quite down because Uppsala joins in. In 1743 St. Peterburg joins in, which also doesn't raise average temperatures much a.s.o. In 1758 finally the US joins in with Philadelphia and one sees how the blue curve splits up into a global black curve and a european blue curve.

    In the blue european curve one can know now see (as was said in the video) that around 1986 there is a visible temperature raise, however it doesn't look as dramatic as in the video, but then in the video the average was over 44 special stations and not over 400 random european stations. To me the blue curve looks still raising even after 1988, but it's not very visible. It would now be interesting to see wether eventually a lot of northern european stations switched of in around 1988, I am not sure I thought the big switch off of (in particular northern) russian stations was I thought a little later, but maybe not.

    Comment Source:<img src="http://www.randform.org/blog/wp-content/2015/05/averagethree1710-1810.jpg" alt="" title="averagethree1710-1810" width="1384" height="390" class="aligncenter size-full wp-image-5841" /> <img src="http://www.randform.org/blog/wp-content/2015/05/averagethree1810-1910.jpg" alt="" title="averagethree1810-1910" width="1384" height="390" class="aligncenter size-full wp-image-5840" /> <img src="http://www.randform.org/blog/wp-content/2015/05/averagethree1910-2010.jpg" alt="" title="averagethree1910-2010" width="1384" height="390" class="aligncenter size-full wp-image-5839" /> Regarding my comment in 9. I now wanted to find out how much the european temp stations dominate the global average so I now took the average over all stations between: 40<Latitude <61 -32<Longitude<+32 Those are altogether around 400 stations, depending on time. The black and blue curves are threeyearrunningmean temperature anomaly curves where black is the global and blue is the european average. I interpreted the threeyearrunningmean as: $threeyearrunningmean(f(j))= 1/36*\sum_{i=-17}^{19} f(j+i)$. Slightly visible is also the oneyearrunningmean (annual average) curve behind the black threeyearrunningmean curve. The jagged grey curve is the european average temp anomaly curve. The CRUTEM4 temperature collection starts in 1701 with Berlin, then in 1706 De Bilt, Netherlands joins in and then you see in 1722 that the "average european temperature" is going quite down because Uppsala joins in. In 1743 St. Peterburg joins in, which also doesn't raise average temperatures much a.s.o. In 1758 finally the US joins in with Philadelphia and one sees how the blue curve splits up into a global black curve and a european blue curve. In the blue european curve one can know now see (as was said in the video) that around 1986 there is a visible temperature raise, however it doesn't look as dramatic as in the video, but then in the video the average was over 44 special stations and not over 400 random european stations. To me the blue curve looks still raising even after 1988, but it's not very visible. It would now be interesting to see wether eventually a lot of northern european stations switched of in around 1988, I am not sure I thought the big switch off of (in particular northern) russian stations was I thought a little later, but maybe not.
  • 12.
    edited May 2015

    I like looking at graphs. For the global mean it's interesting to see leaps in ~1782-1788, ~1835-1855, ~1948-1951 and 2005-2008. I can only see a 'leap' in the european mean between ~1984 and ~1998 when it peaks. It seems smaller than any of the global leaps? BTW It's fun to read the French (and therefore involving politics and intrigue) account of Courtillot's role in publishing papers in Elsevier's Earth and Planetary Science Letters in 2003 and 2005. You shouldn't be surprised to know he's a notorious AGW denier and has zero track record in peer-reviewed climate science (iow another bloody geologist!).

    Comment Source:I like looking at graphs. For the global mean it's interesting to see leaps in ~1782-1788, ~1835-1855, ~1948-1951 and 2005-2008. I can only see a 'leap' in the european mean between ~1984 and ~1998 when it peaks. It seems smaller than any of the global leaps? BTW It's fun to read the French (and therefore involving politics and intrigue) account of Courtillot's role in publishing papers in Elsevier's Earth and Planetary Science Letters in 2003 and 2005. You shouldn't be surprised to know he's a notorious AGW denier and has zero track record in peer-reviewed climate science (iow another bloody geologist!).
  • 13.
    nad
    edited May 2015

    I like looking at graphs. For the global mean it's interesting to see leaps in ~1782-1788, ~1835-1855, ~1948-1951 and 2005-2008. I can only see a 'leap' in the european mean between ~1984 and ~1998 when it peaks.

    Yes it is sometimes quite interesting and here in those graphs you see especially well how the graph is quite "overshadowed" by historical and not geological events. That is a big part of the leaps are probably of a similar nature like the "Uppsala down leap" in 1722. So I suspect that the big rise in the global temperatures after 1860 may be at least partially connected with the reconstruction era the rise 1948-1951 could be mostly due to the end of the second world war and the enlargement of the collection (like by african states). The 2005-2008 rise seems to be mostly due to the fact that a lot of station data wasn't yet reported/included into the 2011 record, so actually I wouldn't trust the CRUTEM4 much after 2005. The big ditch in 1785 could be at least partially due to the american revolutionary war. The 1835-1855 ditch seems however eventually of a mostly geological origin that is I can't fastly find a historical reason, which could have caused that ditch.


    comment added on May 17:

    John has mentioned two geological events which you can see in the above curves:

    The downward spikes are explained nicely by volcanic activity. For example, you can see the 1815 eruption of Tambora in Indonesia, which blanketed the atmosphere with ash. 1816 was called The Year Without a Summer: frost and snow were reported in June and July in both New England and Northern Europe! Average global temperatures dropped 0.4–0.7 °C, resulting in major food shortages across the Northern Hemisphere. Similarly, the dip in 1783-1785 seems to be to due to Grímsvötn in Iceland.

    The 1815 event is however less pronounced visible in the curves, which is probably also due to the fact that there are probably not so many active asian stations in the collection at that time.


    As said for the after 1986 rise in Europe I would have guessed an influence of the russian stations but I guess V. Courtillot chose the 44 temperature stations, because they provided stable and more or less equally distributed reporting (It seemed though to me that there are more stations in Europe who could provide stable reporting at least over the approximate time period he documents, but maybe I'm wrong). So maybe there is a geological origin for that 1986 pecularity.

    It's fun to read the French (and therefore involving politics and intrigue) account of Courtillot's role in publishing papers in Elsevier's Earth and Planetary Science Letters in 2003 and 2005.

    The Elsevier Scandal looks not nice.

    You shouldn't be surprised to know he's a notorious AGW denier and has zero track record in peer-reviewed climate science (iow another bloody geologist!).

    Well I wouldn't call myself an AGW denier, but by what I have seen in the last couple of years about the state of temperature and other records and some reasonings I acquired strong doubts in quite a lot of the current main stream science paradigms. That needn't be apriori disconcerting if I could exclude that things could be way worse then that what the fiercest climate change alerters say. But I can't.

    Comment Source:>I like looking at graphs. For the global mean it's interesting to see leaps in ~1782-1788, ~1835-1855, ~1948-1951 and 2005-2008. I can only see a 'leap' in the european mean between ~1984 and ~1998 when it peaks. Yes it is sometimes quite interesting and here in those <a href="https://forum.azimuthproject.org/discussion/comment/14637/#Comment_14637">graphs</a> you see especially well how the graph is quite "overshadowed" by historical and not geological events. That is a big part of the leaps are probably of a similar nature like the "Uppsala down leap" in 1722. So I suspect that the big rise in the global temperatures after 1860 may be at least partially connected with the <a href="http://en.wikipedia.org/wiki/Reconstruction_Era">reconstruction era</a> the rise 1948-1951 could be mostly due to the end of the second world war and the enlargement of the collection (like by african states). The 2005-2008 rise seems to be mostly due to the fact that a lot of station data wasn't yet reported/included into the 2011 record, so actually I wouldn't trust the CRUTEM4 much after 2005. The big ditch in 1785 could be at least partially due to the <a href="http://en.wikipedia.org/wiki/American_Revolutionary_War">american revolutionary war</a>. The 1835-1855 ditch seems however eventually of a mostly geological origin that is I can't fastly find a historical reason, which could have caused that ditch. ********************************** comment added on May 17: <a href="https://johncarlosbaez.wordpress.com/2015/05/15/carbon-emissions-stop-growing/#comment-66588">John has mentioned</a> two geological events which you can see in the above curves: >The downward spikes are explained nicely by volcanic activity. For example, you can see the 1815 eruption of Tambora in Indonesia, which blanketed the atmosphere with ash. 1816 was called The Year Without a Summer: frost and snow were reported in June and July in both New England and Northern Europe! Average global temperatures dropped 0.4–0.7 °C, resulting in major food shortages across the Northern Hemisphere. Similarly, the dip in 1783-1785 seems to be to due to Grímsvötn in Iceland. The 1815 event is however less pronounced visible in the curves, which is probably also due to the fact that there are probably not so many active asian stations in the collection at that time. ********************************** As said for the after 1986 rise in Europe I would have guessed an influence of the russian stations but I guess V. Courtillot chose the 44 temperature stations, because they provided stable and more or less equally distributed reporting (It seemed though to me that there are more stations in Europe who could provide stable reporting at least over the approximate time period he documents, but maybe I'm wrong). So maybe there is a geological origin for that 1986 pecularity. >It's fun to read the French (and therefore involving politics and intrigue) account of Courtillot's role in publishing papers in Elsevier's Earth and Planetary Science Letters in 2003 and 2005. The Elsevier Scandal looks not nice. >You shouldn't be surprised to know he's a notorious AGW denier and has zero track record in peer-reviewed climate science (iow another bloody geologist!). Well I wouldn't call myself an AGW denier, but by what I have seen in the last couple of years about the state of temperature and other records and some reasonings I acquired strong doubts in quite a lot of the current main stream science paradigms. That needn't be apriori disconcerting if I could exclude that things could be way worse then that what the fiercest climate change alerters say. But I can't.
  • 14.

    Per what Per is asserting, there may be something to it, as the tidal connection to ENSO and QBO is worth considering. There is a recent paper by Li et al [1], that I discuss on my blog that finds some significant correlations between tidal periods and cyclic atmospheric measures http://contextearth.com/2014/08/15/change-of-tide-in-thought/

    I did a few other things here trying to compare the oscillations in the oceanic tidal records with ENSO measures such as SOI http://contextearth.com/2014/09/21/an-enso-predictor-based-on-a-tide-gauge-data-model/

    There is definitely something relating the two. If we can consolidate (1) tidal records, (2) ENSO, and (3) QBO, that may help our understanding significantly.

    [1] Li, G., Zong, H., & Zhang, Q. (2011). 27.3-day and average 13.6-day periodic oscillations in the Earth’s rotation rate and atmospheric pressure fields due to celestial gravitation forcing. Advances in Atmospheric Sciences, 28, 45-58.

    Comment Source:Per what Per is asserting, there may be something to it, as the tidal connection to ENSO and QBO is worth considering. There is a recent paper by Li et al [1], that I discuss on my blog that finds some significant correlations between tidal periods and cyclic atmospheric measures http://contextearth.com/2014/08/15/change-of-tide-in-thought/ I did a few other things here trying to compare the oscillations in the oceanic tidal records with ENSO measures such as SOI http://contextearth.com/2014/09/21/an-enso-predictor-based-on-a-tide-gauge-data-model/ There is definitely something relating the two. If we can consolidate (1) tidal records, (2) ENSO, and (3) QBO, that may help our understanding significantly. [1] Li, G., Zong, H., & Zhang, Q. (2011). 27.3-day and average 13.6-day periodic oscillations in the Earth’s rotation rate and atmospheric pressure fields due to celestial gravitation forcing. Advances in Atmospheric Sciences, 28, 45-58.
  • 15.
    edited May 2015

    Yes, I do believe in the conservation of angular momentum, but not between two systems which have no direct physical contact with each other. Whether the friction between the atmosphere has an large enough effect in order to link the atmosphere and the Earth’s crust in short timespans is questinable. I personally think the correlation with ENSO and LOD has much much more to do with changes in the crust and in the Oceans.

    No I’m not using MEI data as input or as in some feedback link. The only linkage is in the training part where I use the variance of the difference between the calculated output and MEI, which is a standard procedure in any ANN.

    How accurate is the MEI and how large is its error bar? I don’t know, but MEI is the sum of a collection of different signals as defined by NOAA. I am going to do the same calculations with NINO1+2 and NINO3+4, which is the temperature anomalies from those regions, which therefore are more well defined. They should give a better result.

    In fact, on the first graph which I created about year ago I used a feedback loop in order to catch the underlining oscillation behavior of ENSO.

    In the next improved graph where I used a new approach I removed the feedback and found a way to greatly reduce noise and over fitting. I can’t go into more detail of what I have until I have published my paper in which this is going to be explained.

    Yes, I was myself very surprised as the close correlation in the training part as well as in the testing part. When it comes to explain the drivers of ENSO this is the real deal. This clearly shows that tidal and solar influences dominate absolutely as drivers of ENSO. Of course there are other sources, such as noise. Popping up of hurricanes for example. But a lot of this chaotic weather noise has its origin in solar impact. Remember, it is necessary with strong forces to drive changes of ENSO. Yes the Walker circulation seems to be a very instable mechanism, but the flapping of some butterfly in the Amazon is not the source of an El Niño 10 years later. The reason for that is that climate system is not living in a world isolated from space impacts and other external forcing.

    One thing I find amazing is the ignorance between the close correlation between weather and climate and variations in the sun’s electromagnetic activity among climate scientists. It’s not that hard to find. Of course it is hard to find if nobody is looks for it.

    This whole tribal debate between two camps is childish. It should be about empirical evidence. I think it is time for people in this field to start grow up.

    BTW: A week from now I’m going to the University at the nearby town and there I’m going to listen to a lecture by Richard Lindzen. He has been invited there by a skeptical association of which I’m a member. 100 meters away is a climate research institute with about 100 climate researchers. I wonder if any of those people are going to show up at the lecture or if he is going to tour their premises? I don’t know if there has been any public announcement of his lecture. Maybe greens are going to show up protesting as he is a representative of the Devil himself for some of them.

    Comment Source:Yes, I do believe in the conservation of angular momentum, but not between two systems which have no direct physical contact with each other. Whether the friction between the atmosphere has an large enough effect in order to link the atmosphere and the Earth’s crust in short timespans is questinable. I personally think the correlation with ENSO and LOD has much much more to do with changes in the crust and in the Oceans. No I’m not using MEI data as input or as in some feedback link. The only linkage is in the training part where I use the variance of the difference between the calculated output and MEI, which is a standard procedure in any ANN. How accurate is the MEI and how large is its error bar? I don’t know, but MEI is the sum of a collection of different signals as defined by NOAA. I am going to do the same calculations with NINO1+2 and NINO3+4, which is the temperature anomalies from those regions, which therefore are more well defined. They should give a better result. In fact, on the first graph which I created about year ago I used a feedback loop in order to catch the underlining oscillation behavior of ENSO. In the next improved graph where I used a new approach I removed the feedback and found a way to greatly reduce noise and over fitting. I can’t go into more detail of what I have until I have published my paper in which this is going to be explained. Yes, I was myself very surprised as the close correlation in the training part as well as in the testing part. When it comes to explain the drivers of ENSO this is the real deal. This clearly shows that tidal and solar influences dominate absolutely as drivers of ENSO. Of course there are other sources, such as noise. Popping up of hurricanes for example. But a lot of this chaotic weather noise has its origin in solar impact. Remember, it is necessary with strong forces to drive changes of ENSO. Yes the Walker circulation seems to be a very instable mechanism, but the flapping of some butterfly in the Amazon is not the source of an El Niño 10 years later. The reason for that is that climate system is not living in a world isolated from space impacts and other external forcing. One thing I find amazing is the ignorance between the close correlation between weather and climate and variations in the sun’s electromagnetic activity among climate scientists. It’s not that hard to find. Of course it is hard to find if nobody is looks for it. This whole tribal debate between two camps is childish. It should be about empirical evidence. I think it is time for people in this field to start grow up. BTW: A week from now I’m going to the University at the nearby town and there I’m going to listen to a lecture by Richard Lindzen. He has been invited there by a skeptical association of which I’m a member. 100 meters away is a climate research institute with about 100 climate researchers. I wonder if any of those people are going to show up at the lecture or if he is going to tour their premises? I don’t know if there has been any public announcement of his lecture. Maybe greens are going to show up protesting as he is a representative of the Devil himself for some of them.
  • 16.
    edited May 2015

    ENSO is the result of a standing wave dipole set in the equatorial Pacific. To first-order this dipole can be modeled as a second-order differential equation. I will be curious if you have any of that physics in your model. As it stands the model that we are working here on this forum provides a good fit on a much longer time scale than your mysterious ANN does on such a short time-scale. I urge you to try it on a longer time scale.

    One thing I don't understand is how you can predict as in your first comment if you require the solar data, and the AP and KP indices and solar wind which you mention as well. The TSI/sun spot solar data itself can not be predicted with any certainty, and same for the AP and KP indices. The latter look very erratic in particular. Do you not use those for predictions, and that is why your results diverge?

    Good luck.

    Comment Source:ENSO is the result of a standing wave dipole set in the equatorial Pacific. To first-order this dipole can be modeled as a second-order differential equation. I will be curious if you have any of that physics in your model. As it stands the model that we are working here on this forum provides a good fit on a much longer time scale than your mysterious ANN does on such a short time-scale. I urge you to try it on a longer time scale. One thing I don't understand is how you can predict as in your first comment if you require the solar data, and the AP and KP indices and solar wind which you mention as well. The TSI/sun spot solar data itself can not be predicted with any certainty, and same for the AP and KP indices. The latter look very erratic in particular. Do you not use those for predictions, and that is why your results diverge? Good luck.
  • 17.

    I wrote:

    It would now be interesting to see wether eventually a lot of northern european stations switched of in around 1988, I am not sure I thought the big switch off of (in particular northern) russian stations was I thought a little later, but maybe not. and As said for the after 1986 rise in Europe I would have guessed an influence of the russian stations but I guess V. Courtillot chose the 44 temperature stations, because they provided stable and more or less equally distributed reporting..

    I would actually like to point out that I chose the longitudes on purpose rather small so that a possible influence of russian and other former soviet union and other eastern states (which were affected by the changes at that time) stations wouldn't get too big in the european data. In particular Moscow is at longitude 37.37 so it does not contribute to the above blue curve.

    Comment Source:I wrote: >It would now be interesting to see wether eventually a lot of northern european stations switched of in around 1988, I am not sure I thought the big switch off of (in particular northern) russian stations was I thought a little later, but maybe not. and >As said for the after 1986 rise in Europe I would have guessed an influence of the russian stations but I guess V. Courtillot chose the 44 temperature stations, because they provided stable and more or less equally distributed reporting.. I would actually like to point out that I chose the longitudes on purpose rather small so that a possible influence of russian and other former soviet union and other eastern states (which were affected by the changes at that time) stations wouldn't get too big in the european data. In particular Moscow is at longitude 37.37 so it does not contribute to the above blue curve.
  • 18.

    Here is an example of a longer range fit. This is to a tidal gauge record in Sydney Harbor

    tidal

    This uses forcing functions with biannual, annual, and biennial(2-yr) periods applied to a wave-equation with a natural resonance of ~4 years and a sloshing modulation. This is not that difficult to do, so once again urge you Per to try a longer set of data to make sure that you are not overfitting. If in fact you are using tidal gravitational anomalies, these should be stationary over long periods so that a fit over the last 25 years should also apply to 100 years ago.

    I do think that there is a periodic element buried in the behavior, but if your NN contains dozens of factors it may be impossible to pin down the primary mechanism.

    Comment Source:Here is an example of a longer range fit. This is to a tidal gauge record in Sydney Harbor ![tidal](http://imagizer.imageshack.us/a/img903/7049/yxkLF2.gif) This uses forcing functions with biannual, annual, and biennial(2-yr) periods applied to a wave-equation with a natural resonance of ~4 years and a sloshing modulation. This is not that difficult to do, so once again urge you Per to try a longer set of data to make sure that you are not overfitting. If in fact you are using tidal gravitational anomalies, these should be stationary over long periods so that a fit over the last 25 years should also apply to 100 years ago. I do think that there is a periodic element buried in the behavior, but if your NN contains dozens of factors it may be impossible to pin down the primary mechanism.
  • 19.

    I use the measured value of Ap, Kp and solar wind data up to the end of 2014. After that I can not use correct value of those parameters. But I can make predictions. I predict that in the coming years these indexes are in a declining mode as we now are past the last solar peak of the current solar cycle. I expect next solar minimum to occur in 2022. +/- 2 years, because this is a weak and long solar cycle. I use an input file in which I define the slope for each year up to 2020. On those signals I put random noise which I modify in such a way that those signals look like they are expected to look. I can make several variations of those signals so as to create an ensemble of the end results, each with unique random noise values. In the coming years the tidal forcing is dominating so the variation based on solar impact is not so great.

    TSI and sunspot data are not useful. No correlation. But Kp and Ap are. I would concentrate on Ap. The thing with these solar data is that they impact the weather and the jetsream, but probably not imidietly, but after a period. This time lag effect is not something I can see, but here is where the benefit of an ANN comes in. Something happens, but part of the impact appearch 3 months later. Another impact 4 months later and so on. The result looks noisy, but the ANN can untangle the real signals from all this noise.

    Comment Source:I use the measured value of Ap, Kp and solar wind data up to the end of 2014. After that I can not use correct value of those parameters. But I can make predictions. I predict that in the coming years these indexes are in a declining mode as we now are past the last solar peak of the current solar cycle. I expect next solar minimum to occur in 2022. +/- 2 years, because this is a weak and long solar cycle. I use an input file in which I define the slope for each year up to 2020. On those signals I put random noise which I modify in such a way that those signals look like they are expected to look. I can make several variations of those signals so as to create an ensemble of the end results, each with unique random noise values. In the coming years the tidal forcing is dominating so the variation based on solar impact is not so great. TSI and sunspot data are not useful. No correlation. But Kp and Ap are. I would concentrate on Ap. The thing with these solar data is that they impact the weather and the jetsream, but probably not imidietly, but after a period. This time lag effect is not something I can see, but here is where the benefit of an ANN comes in. Something happens, but part of the impact appearch 3 months later. Another impact 4 months later and so on. The result looks noisy, but the ANN can untangle the real signals from all this noise.
  • 20.

    TSI and sunspot data are not useful. No correlation. But Kp and Ap are.

    huh? I'd suspected that TSI and sunspot data are ultimately related to Kp and Ap.

    Comment Source:>TSI and sunspot data are not useful. No correlation. But Kp and Ap are. huh? I'd suspected that TSI and sunspot data are ultimately related to Kp and Ap.
  • 21.

    Here is an example of an Ap time-series record, courtesy of Svalgaard

    ap

    So what you are saying is that this data shifted in a particular way and added to tidal gravitational anomalies will reflect the MEI profile?

    I do understand how some sort of machine learning would be the only hope for finding a pattern here, as I don't see it myself.


    You really ought to use the entire interval of modern day instrumental records. ENSO data is available from way back. Same with tidal data. And with AP data going back before 1900, it should be easy for you to substantiate your findings. If you try to publish, I am certain the reviewers will ask for this, as they will be even more skeptical than me with respect to your findings.

    Comment Source:Here is an example of an Ap time-series record, courtesy of Svalgaard ![ap](http://www.leif.org/research/Ap-1844-now.png) So what you are saying is that this data shifted in a particular way and added to tidal gravitational anomalies will reflect the MEI profile? I do understand how some sort of machine learning would be the only hope for finding a pattern here, as I don't see it myself. --- You really ought to use the entire interval of modern day instrumental records. ENSO data is available from way back. Same with tidal data. And with AP data going back before 1900, it should be easy for you to substantiate your findings. If you try to publish, I am certain the reviewers will ask for this, as they will be even more skeptical than me with respect to your findings.
  • 22.

    That looks impressive. What is the M stand for in SOIM acronym? In fact I working wiith Nils-Axel Mörner and he is an expert on tides and sea level changes as you may know. Of course he knows of the mechanism I discovered and is therefore very supportive.

    He have sent me a number of research paper, which I havn’t read, partly because I know what I doing and partly I have spent much of my time with the software. I better look at to them.

    Comment Source:That looks impressive. What is the M stand for in SOIM acronym? In fact I working wiith Nils-Axel Mörner and he is an expert on tides and sea level changes as you may know. Of course he knows of the mechanism I discovered and is therefore very supportive. He have sent me a number of research paper, which I havn’t read, partly because I know what I doing and partly I have spent much of my time with the software. I better look at to them.
  • 23.
    edited June 2015

    They're all in the same pack http://www.desmogblog.com/nils-axel-morner these geologists. He's an emeritus, barking crackpot who believes in dowsing and denies there is any sea level rise. Apparently even James Randi has had to deal with him.

    Desmog reports him as claiming:

    In the last 2000 years, sea level has oscillated with 5 peaks reaching 0.6 to 1.2 m above the present sea level. From 1790 to 1970 sea level was about 20 cm higher than today In the 1970s, sea level fell by about 20 cm to its present level Sea level has remained stable for the last 30 years, implying that there are no traces of any alarming on-going sea level rise. Therefore, we are able to free the Maldives (and the rest of low-lying coasts and island around the globe) from the condemnation of becoming flooded in the near future.

    Comment Source:They're all in the same pack http://www.desmogblog.com/nils-axel-morner these geologists. He's an emeritus, barking crackpot who believes in dowsing and denies there is any sea level rise. Apparently even James Randi has had to deal with him. Desmog reports him as claiming: > In the last 2000 years, sea level has oscillated with 5 peaks reaching 0.6 to 1.2 m above the present sea level. From 1790 to 1970 sea level was about 20 cm higher than today In the 1970s, sea level fell by about 20 cm to its present level Sea level has remained stable for the last 30 years, implying that there are no traces of any alarming on-going sea level rise. Therefore, we are able to free the Maldives (and the rest of low-lying coasts and island around the globe) from the condemnation of becoming flooded in the near future.
  • 24.

    nad said

    "huh? I'd suspected that TSI and sunspot data are ultimately related to Kp and Ap."

    Very good point. The time-series for the AP index I showed in #21 reflects the same underlying periodicity as the TSI/sunspot data. The 11-year cycle is fainter in the AP data but it is clearly there.

    So it doesn't make a lot of sense when you say "TSI and sunspot data are not useful. No correlation. But Kp and Ap are."

    In this case, one way that AP would work is if your NN were to cherry-pick sections of the time-series and use those intervals to match the ENSO profile. I know that other machine-learning tools such as Eureqa have the capability to do this via the application of time conditionals, but this has the effect of penalizing for complexity.

    The reason I think that you may be doing this is that the AP data does illustrate "pulse-like" behavior, and this might be effective in reproducing the peaks in the MEI data, particularly the squared peak during the 1998 EN.

    Incidentally, in my model, I generate the steeper peaks and valleys via the application of Mathieu modulation to the wave equation. This is a common feature of physical hydrodynamic sloshing mechanisms.

    You could also do it by generating higher harmonics in a Fourier series. For example, a square wave time series would be generated by a fundamental frequency and a number of harmonics of this frequency. A NN may be able to reproduce this by taking the fundamental frequency and "squaring" this to double and quadruple the frequency and then composing from those harmonics.

    As I am reminded, the sky is the limit as far as machine learning is concerned -- you just have to be aware of the extra complexity that each of the transforms has on the outcome

    Comment Source:nad said > "huh? I'd suspected that TSI and sunspot data are ultimately related to Kp and Ap." Very good point. The time-series for the AP index I showed in #21 reflects the same underlying periodicity as the TSI/sunspot data. The 11-year cycle is fainter in the AP data but it is clearly there. So it doesn't make a lot of sense when you say *"TSI and sunspot data are not useful. No correlation. But Kp and Ap are."* In this case, one way that AP would work is if your NN were to cherry-pick sections of the time-series and use those intervals to match the ENSO profile. I know that other machine-learning tools such as [Eureqa](http://www.nutonian.com/products/eureqa/) have the capability to do this via the application of time conditionals, but this has the effect of penalizing for complexity. The reason I think that you may be doing this is that the AP data does illustrate "pulse-like" behavior, and this might be effective in reproducing the peaks in the MEI data, particularly the squared peak during the 1998 EN. Incidentally, in my model, I generate the steeper peaks and valleys via the application of Mathieu modulation to the wave equation. This is a common feature of physical hydrodynamic sloshing mechanisms. You could also do it by generating higher harmonics in a Fourier series. For example, a square wave time series would be generated by a fundamental frequency and a number of harmonics of this frequency. A NN may be able to reproduce this by taking the fundamental frequency and "squaring" this to double and quadruple the frequency and then composing from those harmonics. As I am reminded, the sky is the limit as far as machine learning is concerned -- you just have to be aware of the extra complexity that each of the transforms has on the outcome
  • 25.

    Here is an example of an Ap time-series record, courtesy of Svalgaard

    Interesting. The Ap or Kp index is an average. Do you or Leif and/or Vera Svalgaard know wether there exist somewhere magnetometer measurements of single stations? Like in the arctic? That is although there currently seem bigger efforts in arctic data management taking place:

    The INTERACT network of research and monitoring stations are engaged in the provision of high-quality data for purposes such as assessing environmental change across the polar region. Overall assessment across stations (across the Arctic) is, however, problematic due to partially incompatible data archives. This imposes a risk that information gaps and redundancies remain undetected, that synergies across activities remain unexploited, and that new experiments are being performed on grounds that are affected by previous activities. In addition, public outreach is hampered by the lack of centralised information management, and data communication unnecessarily complicated.

    I unfortunately couldn't yet find any such measurement set.

    Comment Source:>Here is an example of an Ap time-series record, courtesy of Svalgaard Interesting. The Ap or Kp index is an average. Do you or <a href="http://www.leif.org/">Leif and/or Vera Svalgaard</a> know wether there exist somewhere magnetometer measurements of single stations? Like in the arctic? That is although there currently seem bigger efforts in arctic <a href="http://www.eu-interact.org/about-interact/joint-research-activities/data-management/">data management</a> taking place: >The INTERACT network of research and monitoring stations are engaged in the provision of high-quality data for purposes such as assessing environmental change across the polar region. Overall assessment across stations (across the Arctic) is, however, problematic due to partially incompatible data archives. This imposes a risk that information gaps and redundancies remain undetected, that synergies across activities remain unexploited, and that new experiments are being performed on grounds that are affected by previous activities. In addition, public outreach is hampered by the lack of centralised information management, and data communication unnecessarily complicated. I unfortunately couldn't yet find any such measurement set.
  • 26.

    What is amazing and perhaps amazing to the point of incredulousness is how well that Per can model every little wiggle in the MEI time-series.

    I highlighted two small peaks that are almost within the noise margin below:

    mei

    Yet, Per's machine learning technique actually fits to these points, and many more besides those two!

    Those are very high frequency components that have a huge complexity factor according to machine learning rules. That is especially true if they are composed of simusoidal waveforms. And tidal gravitational anomalies are close to sinusoidal, so you have to wonder on the complexity involved.

    But if some other factor that Per is using, such as the AP index contains these sharp peaks, then I can understand how those can lock in as the AP signal emerges from the MEI measure.

    Whatever comes out of this, it will be a learning experience. And we will likely find out either (1) how useful neural networks are or (2) how misleading they are if allowed to run unchecked.

    Comment Source:What is amazing and perhaps amazing to the point of incredulousness is how well that Per can model every little wiggle in the MEI time-series. I highlighted two small peaks that are almost within the noise margin below: ![mei](http://imageshack.com/a/img913/254/66m934.gif) Yet, Per's machine learning technique actually fits to these points, and many more besides those two! Those are very high frequency components that have a huge complexity factor according to machine learning rules. That is especially true if they are composed of simusoidal waveforms. And tidal gravitational anomalies are close to sinusoidal, so you have to wonder on the complexity involved. But if some other factor that Per is using, such as the AP index contains these sharp peaks, then I can understand how those can lock in as the AP signal emerges from the MEI measure. Whatever comes out of this, it will be a learning experience. And we will likely find out either (1) how useful neural networks are or (2) how misleading they are if allowed to run unchecked.
  • 27.
    edited May 2015

    When it comes to the connection between Ap, TSI and sunspots, I would say this. Of course I examined various potential parameters individually. In this case I have established that both TSI and the sunspots number don’t give any measurable correlation which is more than noise and as such would only add extra noise in my ANN calculations.

    One reason can be that I don’t include low frequency data. I only go 3 years back in time. Further, sunspots activities are as a response to the solar magnetic activity and are not closely related on shorter timescale. TSI variations is very small. The assumed TSI values before 1980 are only based on speculation and not on real measurements.

    When it comes to Svalgaard’s reconstruction, I would be skeptical of the values in the 19th century. I have to examine the accuracy of those values. Interesting drop around 1880.

    I would also add that I have tripled each input value into 3 in a PID pattern and I therefore get a response with enhanced dynamic in the ANN calculations. My guess is that solar and tidal forcing represent somewhere between 94 and 98 % of all the driving forces behind ENSO variation on a monthly basis, which is going to have huge impact on the AGW debate.

    Something about the smear site desmog run by pr-consultant Jim Hoggan who has no scientific credential at all, which is financed by David Suzuki foundation that is own by David Suzuki the now famous apocalyptic environmentalist who has a background in fruit flies studies and is profiteering on people’s scientific ignorance and emotional response.

    Get out of the sandbox, this is supposed to be a scientific discussion, please.

    Comment Source:When it comes to the connection between Ap, TSI and sunspots, I would say this. Of course I examined various potential parameters individually. In this case I have established that both TSI and the sunspots number don’t give any measurable correlation which is more than noise and as such would only add extra noise in my ANN calculations. One reason can be that I don’t include low frequency data. I only go 3 years back in time. Further, sunspots activities are as a response to the solar magnetic activity and are not closely related on shorter timescale. TSI variations is very small. The assumed TSI values before 1980 are only based on speculation and not on real measurements. When it comes to Svalgaard’s reconstruction, I would be skeptical of the values in the 19th century. I have to examine the accuracy of those values. Interesting drop around 1880. I would also add that I have tripled each input value into 3 in a PID pattern and I therefore get a response with enhanced dynamic in the ANN calculations. My guess is that solar and tidal forcing represent somewhere between 94 and 98 % of all the driving forces behind ENSO variation on a monthly basis, which is going to have huge impact on the AGW debate. Something about the smear site desmog run by pr-consultant Jim Hoggan who has no scientific credential at all, which is financed by David Suzuki foundation that is own by David Suzuki the now famous apocalyptic environmentalist who has a background in fruit flies studies and is profiteering on people’s scientific ignorance and emotional response. Get out of the sandbox, this is supposed to be a scientific discussion, please.
  • 28.

    Per said:

    "I would also add that I have tripled each input value into 3 in a PID pattern"

    Oh, now I see. So you are pulling derivatives out of the signal. That makes it much easier to achieve a good fit. The second derivative in a PID extract will reflect relatively closely the mirror of a peak or valley, so that is enough to model the fine features that you show in your fit.

    I have evaluated many models that use derivatives of the signal and I have found fits that equal the quality that you show. However, I don't consider these of true significance since information is being recycled during the fitting process.

    The caveat is that I am misinterpreting what you mean by a PID pattern. So the three values that you extract are proportional, integral, and derivative (i.e. each input value is tripled). This can be extended to a proportional, derivative, and second derivative triple as well to match a physical model such as a wave equation. You said you are a Physics/Electronics grad, so I am assuming your definition of PID matches mine.

    Comment Source:Per said: > "I would also add that I have tripled each input value into 3 in a PID pattern" Oh, now I see. So you are pulling derivatives out of the signal. That makes it much easier to achieve a good fit. The second derivative in a PID extract will reflect relatively closely the mirror of a peak or valley, so that is enough to model the fine features that you show in your fit. I have evaluated many models that use derivatives of the signal and I have found fits that equal the quality that you show. However, I don't consider these of true significance since information is being recycled during the fitting process. The caveat is that I am misinterpreting what you mean by a PID pattern. So the three values that you extract are proportional, integral, and derivative (i.e. each input value is tripled). This can be extended to a proportional, derivative, and second derivative triple as well to match a physical model such as a wave equation. You said you are a Physics/Electronics grad, so I am assuming your definition of PID matches mine.
  • 29.
    edited May 2015

    WebHubTel, you might find what I just added to the wiki relevant. Neural nets, mathematics section.

    Comment Source:WebHubTel, you might find what I just added to the wiki relevant. [Neural nets](http://www.azimuthproject.org/azimuth/show/Neural+networks), mathematics section.
  • 30.

    Thanks Graham,

    This is really a game of 20 questions, trying to extract the significance out of what Per is showing us.

    Here is an example of a machine learning exercise where I provided one tidal gravitational anomaly period (18.6 years = lunar standstill) and then applied it to a finite interval of the MEI ENSO time series: eur

    This is in months and the peak at 230 months is the 1998 hot spike.

    Note the D(mei,t,2) term, which is the second derivative of the signal. This part is able to track the fine detail in the time series.

    Of course this is a short series so the machine learning goes crazy in finding a good fit, which it composes as a complicated set of harmonics of the 18.6 year fundamental cycle.

    That is why I am urging Per to look at a longer interval. It will become much more difficult to line up all of the excursions of an ENSO time series if it extends for over 120 years.

    Comment Source:Thanks Graham, This is really a game of 20 questions, trying to extract the significance out of what Per is showing us. Here is an example of a machine learning exercise where I provided one tidal gravitational anomaly period (18.6 years = [lunar standstill](http://www.nature.com/ngeo/journal/v1/n3/full/ngeo127.html)) and then applied it to a finite interval of the MEI ENSO time series: ![eur](http://imageshack.com/a/img540/2231/qocR4E.gif) This is in months and the peak at 230 months is the 1998 hot spike. Note the D(mei,t,2) term, which is the second derivative of the signal. This part is able to track the fine detail in the time series. Of course this is a short series so the machine learning goes crazy in finding a good fit, which it composes as a complicated set of harmonics of the 18.6 year fundamental cycle. That is why I am urging Per to look at a longer interval. It will become much more difficult to line up all of the excursions of an ENSO time series if it extends for over 120 years.
  • 31.

    Machine learning algorithms are marvellous tools for self-deception.

    Models which can be “tuned” in many different ways give researchers more scope to perceive a pattern where none exists. According to some estimates, three-quarters of published scientific papers in the field of machine learning are bunk because of this “overfitting”, says Sandy Pentland, a computer scientist at the Massachusetts Institute of Technology.

    http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

    Comment Source:Machine learning algorithms are marvellous tools for self-deception. > Models which can be “tuned” in many different ways give researchers more scope to perceive a pattern where none exists. According to some estimates, three-quarters of published scientific papers in the field of machine learning are bunk because of this “overfitting”, says Sandy Pentland, a computer scientist at the Massachusetts Institute of Technology. http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble
  • 32.

    Your definition of PID is the same as mine. You can look at the ENSO’s dynamic as a huge control theory problem and that it therefore makes sense to catch the signals’ full dynamic nature. Ultimately it is the forecast that counts, while the solar data of the future is based of estimations, the tidal forcing into the future is based on exact tidal forcing data.

    I introduced PID data quite early in my research effort into climate analysis. I first collected every type of relevant input signals I could find and I built an ANN and then I tested this against the global temperature as measured from satellite by using input signals from all these inputs which of course led to much noise and overfitting.

    I then realized it would be much better if I tested each signal separately. Which I did.

    I then tried to estimate the strength of the impacted relative to the impact of ENSO’s impact on the global temperature. I defined ENSO’s impact on the global temperature to 100%. Here is the result against the derivate of the global temperature:

    SST 110%

    ENSO 100%

    Variation in Earth's rotation 68.5%

    Solar wind speed 49.5%

    Solar wind temperature 26.3%

    Solar wind density 9.7%

    Kp Magnetic Index 27.4%

    Ap Geomagnetic index 13.3%

    Sunspot number 4.8%

    F10.7 radio flux. 4.0%

    SST is the global sea surface temperature anomaly and it makes sense that this has a greater impact than ENSO.

    One thing I had to consider was if sensors that is used in the temperature measuring from satellite itself were impacted by solar and gravitational variations.

    After that I continued and started to work with ENSO, which I found more interesting. Sure you can see there is some correlation to sunspots, but with ENSO it is no good.

    When it comes to machine learning I would like to point out that I’m not using time serie analysis, but use as input real solar and tidal forcing.

    Comment Source:Your definition of PID is the same as mine. You can look at the ENSO’s dynamic as a huge control theory problem and that it therefore makes sense to catch the signals’ full dynamic nature. Ultimately it is the forecast that counts, while the solar data of the future is based of estimations, the tidal forcing into the future is based on exact tidal forcing data. I introduced PID data quite early in my research effort into climate analysis. I first collected every type of relevant input signals I could find and I built an ANN and then I tested this against the global temperature as measured from satellite by using input signals from all these inputs which of course led to much noise and overfitting. I then realized it would be much better if I tested each signal separately. Which I did. I then tried to estimate the strength of the impacted relative to the impact of ENSO’s impact on the global temperature. I defined ENSO’s impact on the global temperature to 100%. Here is the result against the derivate of the global temperature: SST 110% ENSO 100% Variation in Earth's rotation 68.5% Solar wind speed 49.5% Solar wind temperature 26.3% Solar wind density 9.7% Kp Magnetic Index 27.4% Ap Geomagnetic index 13.3% Sunspot number 4.8% F10.7 radio flux. 4.0% SST is the global sea surface temperature anomaly and it makes sense that this has a greater impact than ENSO. One thing I had to consider was if sensors that is used in the temperature measuring from satellite itself were impacted by solar and gravitational variations. After that I continued and started to work with ENSO, which I found more interesting. Sure you can see there is some correlation to sunspots, but with ENSO it is no good. When it comes to machine learning I would like to point out that I’m not using time serie analysis, but use as input real solar and tidal forcing.
  • 33.

    Yes, we have to be very cognizant of over-fitting. In these multi-parameter fits, high P-values often give misleading indications. What I often see is that a couple of factors can work together to increase each others P-value, showing increased significance when there more than likely isn't any. For example, two sine waves of slightly different frequency can create a linear trend when one sinusoid is subracted from another. This linear trend can then generate an improved correlation, where in reality this is just an artifact of linear superposition.

    Yet, there is also the reality of no-pain means no-gain. Unless we go down these dead-ends and try out different combinations, we may not strike on a combination that is highly plausible and can not be chalked up to just coincidence.

    Let's see what Per has to say about justifying his PID approach. If he thinks he will get a different reception to his article on submission to a peer-reviewed journal than he will get here, that may or may not happen. I think it might still be useful in projecting out short term, especially if he can do tests across many different historical time intervals.


    As a companion chart to what I showed in #30, this is what happens when I remove the second-derivative factor from the fit. no-second

    This removes the fine structure but keeps the longer term modulation. I think that this is what his " PID pattern" is providing to improve the fit.

    Comment Source:Yes, we have to be very cognizant of over-fitting. In these multi-parameter fits, high P-values often give misleading indications. What I often see is that a couple of factors can work together to increase each others P-value, showing increased significance when there more than likely isn't any. For example, two sine waves of slightly different frequency can create a linear trend when one sinusoid is subracted from another. This linear trend can then generate an improved correlation, where in reality this is just an artifact of linear superposition. Yet, there is also the reality of no-pain means no-gain. Unless we go down these dead-ends and try out different combinations, we may not strike on a combination that is highly plausible and can not be chalked up to just coincidence. Let's see what Per has to say about justifying his PID approach. If he thinks he will get a different reception to his article on submission to a peer-reviewed journal than he will get here, that may or may not happen. I think it might still be useful in projecting out short term, especially if he can do tests across many different historical time intervals. --- As a companion chart to what I showed in #30, this is what happens when I remove the second-derivative factor from the fit. ![no-second](http://imageshack.com/a/img911/9421/Z13d6f.gif) This removes the fine structure but keeps the longer term modulation. I think that this is what his " PID pattern" is providing to improve the fit.
  • 34.

    Per said:

    " Your definition of PID is the same as mine. You can look at the ENSO’s dynamic as a huge control theory problem and that it therefore makes sense to catch the signals’ full dynamic nature. "

    I am not convinced that it is a control theory problem. Or that control theory somehow supersedes the essential physics.

    Here is the issue. If you are using a PID to track the behavior of a wave equation

    $ D f''(t) + f(t) = 0 $

    then if we expand the second derivative as a difference equation:

    $ f''(t) \sim (f(t+dt) - f(t)) - (f(t) - f(t-dt)) $

    or

    $ f''(t) \sim f(t+dt) + f(t-dt) - 2 f(t) $

    what you see is that the second derivative is taking on the character of the function itself, but with opposite sign. Therefore an autocorrelation occurs and the fit will automatically improve.

    I am not going to address the other points in your comment until we find out that this is what you are doing. The quality of the fit really hinges on this specific concern that we have with your work.

    If this is what you are actually doing, then your fit has to be treated with some circumspection.

    Comment Source:Per said: > " Your definition of PID is the same as mine. You can look at the ENSO’s dynamic as a huge control theory problem and that it therefore makes sense to catch the signals’ full dynamic nature. " I am not convinced that it is a control theory problem. Or that control theory somehow supersedes the essential physics. Here is the issue. If you are using a PID to track the behavior of a wave equation $ D f''(t) + f(t) = 0 $ then if we expand the second derivative as a difference equation: $ f''(t) \sim (f(t+dt) - f(t)) - (f(t) - f(t-dt)) $ or $ f''(t) \sim f(t+dt) + f(t-dt) - 2 f(t) $ what you see is that the second derivative is taking on the character of the function itself, but with opposite sign. Therefore an autocorrelation occurs and the fit will automatically improve. I am not going to address the other points in your comment until we find out that this is what you are doing. The quality of the fit really hinges on this specific concern that we have with your work. If this is what you are actually doing, then your fit has to be treated with some circumspection.
  • 35.

    I have to point out that the PID signals are only used as inputs. No derivate calculations are made inside deeper layers.

    Much of the influences on ENSO are arriving in the form of tidal and solar pulses. One can of course have a quasi theological discussion of the benefit of using ENSO model using thermodynamic and fluid dynamic similar to GSM model versus using ANN pattern identification as a techniques to make forecasts and a way to understand ENSO.

    I don’t work in a vacuum and I know based on the analysis I made that tidal gravitational and electromagnetic solar forces are the main drivers of ENSO. Ultimately these drivers should be incorporated in traditional ENSO models. If this is done, they would see impressive improvements.

    I work together with a prominent solar professor and I’m going to ask him about the quality of available Ap index data. I’m somewhat skeptical about Leif Svalgaard’s Ap reconstruction.

    Anyway, I’m going to use Ap data going back longer in time and while doing this I’m going to use long time spans for the learning part, the training part and for the forecast part. By doing this limiting potential for overfitting problems.

    Comment Source:I have to point out that the PID signals are only used as inputs. No derivate calculations are made inside deeper layers. Much of the influences on ENSO are arriving in the form of tidal and solar pulses. One can of course have a quasi theological discussion of the benefit of using ENSO model using thermodynamic and fluid dynamic similar to GSM model versus using ANN pattern identification as a techniques to make forecasts and a way to understand ENSO. I don’t work in a vacuum and I know based on the analysis I made that tidal gravitational and electromagnetic solar forces are the main drivers of ENSO. Ultimately these drivers should be incorporated in traditional ENSO models. If this is done, they would see impressive improvements. I work together with a prominent solar professor and I’m going to ask him about the quality of available Ap index data. I’m somewhat skeptical about Leif Svalgaard’s Ap reconstruction. Anyway, I’m going to use Ap data going back longer in time and while doing this I’m going to use long time spans for the learning part, the training part and for the forecast part. By doing this limiting potential for overfitting problems.
  • 36.
    edited May 2015

    Per said:

    "I have to point out that the PID signals are only used as inputs. No derivate calculations are made inside deeper layers."

    Well, I am only using PID signals as inputs in my "model" of comment #30 and it does a better job fitting the MEI than your formulation does. You could perhaps respond to that as a challenge, and show why my argument of comment #34 is not operable.

    "Much of the influences on ENSO are arriving in the form of tidal and solar pulses. One can of course have a quasi theological discussion of the benefit of using ENSO model using thermodynamic and fluid dynamic similar to GSM model versus using ANN pattern identification as a techniques to make forecasts and a way to understand ENSO."

    Solar may in fact be a big driver because of the annual and biannual forcing, which is likely to influence QBO, and therefore ENSO.

    I don't know what "quasi theological discussion" means in our context. We do math and physics here, not religious pontificating. You may want to modify that statement out of respect for what the goals of the Azimuth project are.

    I know that there are enough people here that will support you if you are receptive to accepting some feedback. But if you are here to just jerk us around, that is not so good :)

    Comment Source:Per said: > "I have to point out that the PID signals are only used as inputs. No derivate calculations are made inside deeper layers." Well, I am only using PID signals as inputs in my "model" of comment #30 and it does a better job fitting the MEI than your formulation does. You could perhaps respond to that as a challenge, and show why my argument of comment #34 is not operable. > "Much of the influences on ENSO are arriving in the form of tidal and solar pulses. One can of course have a quasi theological discussion of the benefit of using ENSO model using thermodynamic and fluid dynamic similar to GSM model versus using ANN pattern identification as a techniques to make forecasts and a way to understand ENSO." Solar may in fact be a big driver because of the annual and biannual forcing, which is likely to influence QBO, and therefore ENSO. I don't know what "quasi theological discussion" means in our context. We do math and physics here, not religious pontificating. You may want to modify that statement out of respect for what the goals of the Azimuth project are. I know that there are enough people here that will support you if you are receptive to accepting some feedback. But if you are here to just jerk us around, that is not so good :)
  • 37.

    I have a general observation and question regarding Per's work.

    Is it easier to (1) falsify a model such as Per's that shows remarkable agreement with the data, than it is to (2) support a model that shows only reasonable agreement with the data?

    I think (1) is easier, because all you have to do is find the hidden piece where the researcher is making a substitution of X=X somewhere in his model (see comment #34) And once that X=X is exposed, then the agreement becomes run-of-the-mill.

    If Per does submit his paper for peer review, he is going to have to carefully consider what Graham said in comment #31.

    The way I am doing the ENSO modeling is to use machine learning to gain some insight and winnow the search of plausible candidates. But in the end, I make the differential equation solution explicit so someone else can reproduce it.

    Comment Source:I have a general observation and question regarding Per's work. Is it easier to (1) falsify a model such as Per's that shows remarkable agreement with the data, than it is to (2) support a model that shows only reasonable agreement with the data? I think (1) is easier, because all you have to do is find the hidden piece where the researcher is making a substitution of X=X somewhere in his model (see comment #34) And once that X=X is exposed, then the agreement becomes run-of-the-mill. If Per does submit his paper for peer review, he is going to have to carefully consider what Graham said in comment #31. The way I am doing the ENSO modeling is to use machine learning to gain some insight and winnow the search of plausible candidates. But in the end, I make the differential equation solution explicit so someone else can reproduce it.
  • 38.

    It has been a couple of weeks since Per has defended his ENSO model on the forum. Ruminating some more on his admission that he is using derivatives of the signal as input, I am convinced that this will lead to an excellent fit over a short interval but at the expense of long-range coherence.

    I am of mixed-mind whether I want to pursue that approach with my sloshing wave equation analysis. I have been using mainly a correlation coefficient to evaluate the quality of fit between model and data, but this eventually runs into a wall of diminishing returns.

    Consider the evaluation of the SOI time-series against the (inverted) NINO3.4 data. Using a filter of 7 months, the CC between the two reaches 0.86. The visual comparison is shown below. There is little doubt that the two sets of data are intimately connected, even though the CC hasn't reached a 0.99 level -- which is often considered necessary for much less complex waveforms.

    soi_nino34

    In contrast, the SOI sloshing model can reach a correlation coefficient of 0.68 with the assumed forcings, shown below:

    soi_soim

    but it takes much more effort to achieve a fit close to 0.8, obtained mainly by tuning the forcings slightly.

    However, with the advantage of applying a second-derivative of the data to the fit, I can easily achieve a fit as good as the SOI to NINO3.4 comparison, which is ultimately close to the noise and uncertainty limit. The question is whether this approach is kosher, and the same consideration I believe applies to Per's work. The advantages of this approach are definitely that it may improve the short-range forecasting ability, but is this what we want? Or do we want to understand the entire time history of ENSO, and leave the forecasting for others?

    BTW, there is another group independent of Per that is crowing about cracking the ENSO puzzle with their own model. Similar to Per, they are acting very secretive about their approach, but claim they predicted the current ElNino conditions based on a fit from a couple of years ago. I have been having a twitter war with them, and they just love to toss out the stink-bomb that I am not performing forecasts -- as if that is somehow "proof" of the validity of a model.

    Comment Source:It has been a couple of weeks since Per has defended his ENSO model on the forum. Ruminating some more on his admission that he is using derivatives of the signal as input, I am convinced that this will lead to an excellent fit over a short interval but at the expense of long-range coherence. I am of mixed-mind whether I want to pursue that approach with my sloshing wave equation analysis. I have been using mainly a correlation coefficient to evaluate the quality of fit between model and data, but this eventually runs into a wall of diminishing returns. Consider the evaluation of the SOI time-series against the (inverted) NINO3.4 data. Using a filter of 7 months, the CC between the two reaches 0.86. The visual comparison is shown below. There is little doubt that the two sets of data are intimately connected, even though the CC hasn't reached a 0.99 level -- which is often considered necessary for much less complex waveforms. ![soi_nino34](http://imageshack.com/a/img661/1446/Pe2MyC.gif ) In contrast, the SOI sloshing model can reach a correlation coefficient of 0.68 with the assumed forcings, shown below: ![soi_soim](http://imageshack.com/a/img913/7829/IOef7S.gif) but it takes much more effort to achieve a fit close to 0.8, obtained mainly by tuning the forcings slightly. However, with the advantage of applying a second-derivative of the data to the fit, I can easily achieve a fit as good as the SOI to NINO3.4 comparison, which is ultimately close to the noise and uncertainty limit. The question is whether this approach is kosher, and the same consideration I believe applies to Per's work. The advantages of this approach are definitely that it may improve the short-range forecasting ability, but is this what we want? Or do we want to understand the entire time history of ENSO, and leave the forecasting for others? BTW, there is another group independent of Per that is crowing about cracking the ENSO puzzle with their own model. Similar to Per, they are acting very secretive about their approach, but claim they predicted the current ElNino conditions based on a fit from a couple of years ago. I have been having a twitter war with them, and they just love to toss out the stink-bomb that I am not performing forecasts -- as if that is somehow "proof" of the validity of a model.
  • 39.

    I haven't been reading this thread lately, but it's interesting to catch up on it.

    WebHubTel writes:

    The advantages of this approach are definitely that it may improve the short-range forecasting ability, but is this what we want? Or do we want to understand the entire time history of ENSO, and leave the forecasting for others?

    I think those are both, separately, interesting questions. Right now it seems people really want predictions that work well for further than 6 months into the future, because these could have big payoffs for agriculture. But when it comes to science, that's not the only interesting question.

    Comment Source:I haven't been reading this thread lately, but it's interesting to catch up on it. WebHubTel writes: > The advantages of this approach are definitely that it may improve the short-range forecasting ability, but is this what we want? Or do we want to understand the entire time history of ENSO, and leave the forecasting for others? I think those are both, separately, interesting questions. Right now it seems people really want predictions that work well for further than 6 months into the future, because these could have big payoffs for agriculture. But when it comes to science, that's not the only interesting question.
  • 40.

    Since Per Strandberg had claimed that he had "solved" ENSO, I have been tracking what he has been up to. Recall that he won't explain the details of his model when called on it because he has overriding Intellectual Property interests in his findings.

    This is what he wrote recently: https://tallbloke.wordpress.com/2015/10/31/ian-wilson-are-lunar-tides-responsible-for-historical-temperature-anomalies/comment-page-1/#comment-109278

    " As some of you may know I’m using an ANN to analyze and make predictions of ENSO, based on the tidal-lunar-solar cycles, on changes in Earth’s magnetic field and on changes in the solar wind. My result is something I’m now preparing for publishing. Result from this should interest most of you.

    What I have studied is the MEI index. I also tried to use it to analyze NINO3+4 and NINO1+2 temperature anomaly. With the NINO3+4 anomaly I didn’t get any correlation from the ANN. This doesn’t mean there are correlations between the MEI index and NINO3+4, there are. But there is no direct coupling between tidal forcing and changes in the solar electromagnetic activity with NINO3+4 as it is with MEI. With the NINO1+2 anomaly I got a weak sinusoidal coupling that has a period of about 9 years. I assume this is a tidal cycle effect which affects the speed of the Humboldt Current. This should be visible by anyone who analyze NINO1+2 with FFT analyses."

    This is odd that he can't get a correlation of his model with NINO3+4 but he can with MEI. Yet, in fact NINO3+4 is correlated with MEI and so he is violating a transitive law here. These indices are all measuring essentially the same ENSO behavior (give or take geographical differences), so I am not at all sure what he is doing, wrong or right..

    wuwt

    Lots of smoke-blowing going on right now.

    Comment Source:Since Per Strandberg had claimed that he had "solved" ENSO, I have been tracking what he has been up to. Recall that he won't explain the details of his model when called on it because he has overriding Intellectual Property interests in his findings. This is what he wrote recently: https://tallbloke.wordpress.com/2015/10/31/ian-wilson-are-lunar-tides-responsible-for-historical-temperature-anomalies/comment-page-1/#comment-109278 >" As some of you may know I’m using an ANN to analyze and make predictions of ENSO, based on the tidal-lunar-solar cycles, on changes in Earth’s magnetic field and on changes in the solar wind. My result is something I’m now preparing for publishing. Result from this should interest most of you. > What I have studied is the MEI index. I also tried to use it to analyze NINO3+4 and NINO1+2 temperature anomaly. With the NINO3+4 anomaly I didn’t get any correlation from the ANN. This doesn’t mean there are correlations between the MEI index and NINO3+4, there are. But there is no direct coupling between tidal forcing and changes in the solar electromagnetic activity with NINO3+4 as it is with MEI. With the NINO1+2 anomaly I got a weak sinusoidal coupling that has a period of about 9 years. I assume this is a tidal cycle effect which affects the speed of the Humboldt Current. This should be visible by anyone who analyze NINO1+2 with FFT analyses." This is odd that he can't get a correlation of his model with NINO3+4 but he can with MEI. Yet, in fact NINO3+4 is correlated with MEI and so he is violating a transitive law here. These indices are all measuring essentially the same ENSO behavior (give or take geographical differences), so I am not at all sure what he is doing, wrong or right.. ![wuwt](http://i56.tinypic.com/8yzwv7.jpg) Lots of smoke-blowing going on right now.
  • 41.
    edited November 2015

    This is odd that he can't get a correlation of his model with NINO3+4 but he can with MEI. Yet, in fact NINO3+4 is correlated with MEI and so he is violating a transitive law here.

    While this is certainly peculiar, transitivity of correlation is not a law. Consider independent random variables $B, C$ and define $A = B + C$ then $B$ is correlated with $A$ and $A$ is correlated with $C$ but $B$ and $C$ are independent by definition. If $B$ and $C$ have equal variance then $R_{AB} = R_{AC} > 0.7$. Some degree of transitivity is implied when the correlations are very high though. In the extreme $R_{XY} = 1$ and $R_{YZ} = 1$ implies $R_{XZ} = 1$

    Comment Source:> This is odd that he can't get a correlation of his model with NINO3+4 but he can with MEI. Yet, in fact NINO3+4 is correlated with MEI and so he is violating a transitive law here. While this is certainly peculiar, transitivity of correlation is not a law. Consider independent random variables $B, C$ and define $A = B + C$ then $B$ is correlated with $A$ and $A$ is correlated with $C$ but $B$ and $C$ are independent by definition. If $B$ and $C$ have equal variance then $R_{AB} = R_{AC} > 0.7$. Some degree of transitivity is implied when the correlations are very high though. In the extreme $R_{XY} = 1$ and $R_{YZ} = 1$ implies $R_{XZ} = 1$
  • 42.

    yup, but come on ... these two indices are identical as far as the eye can tell, yet Per says he can correlate to one but not the other? this is very fishy. two

    The issue with Per is that he keeps everything under an "Intellectual Property" cloak, so all we have to go on is what he says. And when he says stuff like the above, you have to wonder if his secret stuff is just as ill-conceived.

    Comment Source:yup, but come on ... these two indices are identical as far as the eye can tell, yet Per says he can correlate to one but not the other? this is very fishy. ![two](http://i56.tinypic.com/8yzwv7.jpg) The issue with Per is that he keeps everything under an "Intellectual Property" cloak, so all we have to go on is what he says. And when he says stuff like the above, you have to wonder if his secret stuff is just as ill-conceived.
  • 43.

    yup, but come on ... these two indices are identical as far as the eye can tell, yet Per says he can correlate to one but not the other? this is very fishy.

    I agree something looks strange. But then correlation is a rather coarse grain measure for correlation. What's the correlation between the MEI and the Nino 3.4 above?

    Comment Source:>yup, but come on ... these two indices are identical as far as the eye can tell, yet Per says he can correlate to one but not the other? this is very fishy. I agree something looks strange. But then correlation is a rather coarse grain measure for correlation. What's the correlation between the MEI and the Nino 3.4 above?
  • 44.

    "I agree something looks strange. But then correlation is a rather coarse grain measure for correlation. What's the correlation between the MEI and the Nino 3.4 above?"

    I'm not going to chase that number down, but it is obviously very high. Since ENSO is a standing wave phenomena, what's also important is to look at the correlation of the inflection points. Those will show a highly detailed match at the sub-annual level because many of the peaks and valleys align.

    Moreover (and this really makes you wonder if Per has any scientific intuition at all), MEI is an index composed of several factors, one of which is the sea-surface temperature -- and that is exactly what Nino 3.4 represents!

    So by definition, there will be a high correlation! One is composed of the other! That's what correlation and regression is often used for -- to discover "composed-of" relations. And if you do this blindly, ignorant of the fact that one is an artificial composition of another, you will get corrected for it. What Per is doing is a small step away from plotting X vs X and denying that there is a correlation.

    I don't know what Per's deal is, but I do know that he is a global warming denier, likely intent on messing with people's head as he spews all this. Come on, his twitter handle is @LittleIceAge and his profile says that he is predicting global cooling and in the middle of an ice age. Whereas mine is @Whut, as in whut the [bleep] is going on with these deniers?

    Comment Source:> "I agree something looks strange. But then correlation is a rather coarse grain measure for correlation. What's the correlation between the MEI and the Nino 3.4 above?" I'm not going to chase that number down, but it is obviously very high. Since ENSO is a standing wave phenomena, what's also important is to look at the correlation of the inflection points. Those will show a highly detailed match at the sub-annual level because many of the peaks and valleys align. Moreover (and this really makes you wonder if Per has any scientific intuition at all), MEI is an index composed of several factors, one of which is the sea-surface temperature -- and that is exactly what Nino 3.4 represents! So by definition, there will be a high correlation! One is composed of the other! That's what correlation and regression is often used for -- to discover "composed-of" relations. And if you do this blindly, ignorant of the fact that one is an artificial composition of another, you will get corrected for it. What Per is doing is a small step away from plotting X vs X and denying that there is a correlation. I don't know what Per's deal is, but I do know that he is a global warming denier, likely intent on messing with people's head as he spews all this. Come on, his twitter handle is @LittleIceAge and his profile says that he is predicting global cooling and in the middle of an ice age. Whereas mine is @Whut, as in whut the [bleep] is going on with these deniers?
  • 45.

    Pearson correlation can be related to a metric http://math.stackexchange.com/questions/296292/pearson-correlation-and-metric-properties. So not transitive, but there is a triangle inequalty lurking nearby.

    Comment Source:Pearson correlation can be related to a metric [http://math.stackexchange.com/questions/296292/pearson-correlation-and-metric-properties](http://math.stackexchange.com/questions/296292/pearson-correlation-and-metric-properties). So not transitive, but there is a triangle inequalty lurking nearby.
Sign In or Register to comment.