Options

Odd patterns in temperature data

I have been running component analysis algorithm on the Pacific region using the full 65 years data. I am seeing what look like diffraction patterns in the basis images, most prominently between the coast of Chile and the Central Pacific, but also around the cost of Australia. The most natural explanation would be that they are just numerical artifacts. I would not expect this kind of pattern in temperature data, but I am seeing them across different algorithms and different settings.

Does anybody have thoughts on the plausibility of these patterns having a real physical basis or suggestions on to determine if they are real? The methods I would use in my day job do not seem applicable in this case.

nmf10

Comments

  • 1.
    edited July 2014

    It would be great if you could say a bit about how this principal component analysis works. I get the basic idea, but there are some variations, right? If I can describe it sort of precisely, I can do a blog post on this.

    We'll have to think about these patterns one at a time. You may be seeing signs of the Humboldt current off Chile and Peru, which has cold upwelling water:

    The Humboldt Current is a cold, low-salinity ocean current that flows north along the west coast of South America from the southern tip of Chile to northern Peru. Also called the Peru Current, it is an eastern boundary current flowing in the direction of the equator, and can extend 1,000 kilometers offshore. The Humboldt Current Large Marine Ecosystem (LME), named after the Prussian naturalist Alexander von Humboldt, is one of the major upwelling systems of the world, supporting an extraordinary abundance of marine life. Upwelling occurs off Peru year-round but off Chile only during the spring and summer, because of the displacement of the subtropical center of high pressure during the summer.

    The Humboldt Current LME is considered a Class I, highly productive (>300 gC/$m^2$-yr), ecosystem. It is the most productive marine ecosystem in the world, as well as the largest upwelling system.

    Comment Source:It would be great if you could say a bit about how this principal component analysis works. I get the basic idea, but there are some variations, right? If I can describe it sort of precisely, I can do a blog post on this. We'll have to think about these patterns one at a time. You may be seeing signs of the [Humboldt current](https://en.wikipedia.org/wiki/Humboldt_Current) off Chile and Peru, which has cold upwelling water: > The Humboldt Current is a cold, low-salinity ocean current that flows north along the west coast of South America from the southern tip of Chile to northern Peru. Also called the Peru Current, it is an eastern boundary current flowing in the direction of the equator, and can extend 1,000 kilometers offshore. The Humboldt Current Large Marine Ecosystem (LME), named after the Prussian naturalist Alexander von Humboldt, is one of the major upwelling systems of the world, supporting an extraordinary abundance of marine life. Upwelling occurs off Peru year-round but off Chile only during the spring and summer, because of the displacement of the subtropical center of high pressure during the summer. > The Humboldt Current LME is considered a Class I, highly productive (>300 gC/$m^2$-yr), ecosystem. It is the most productive marine ecosystem in the world, as well as the largest upwelling system. <img src = "https://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Humboldt_current.jpg/300px-Humboldt_current.jpg" alt = ""/>
  • 2.

    Interesting possibility. Would one expect such a current to produce bands (waves?) of alternating temperature?

    Comment Source:Interesting possibility. Would one expect such a current to produce bands (waves?) of alternating temperature?
  • 3.

    I got something possibly similar in Wavelet decompositions:

    Odd Patterns

    They are not artifacts, they are actually fundamental parts of the signal architecture e.g. they could be used to forecast the signal much better or they could be used to explain certain behaviour of signal.

    In wavelets they are uncovered by changing the resolution of how the data is discreet-ized, you might think of them as simpler pencil drawn caricature of a landscape (raw signal).

    Comment Source:I got something possibly similar in Wavelet decompositions: [Odd Patterns](http://files.lossofgenerality.com/odd_temp_patterns.pdf) They are not artifacts, they are actually fundamental parts of the signal architecture e.g. they could be used to forecast the signal much better or they could be used to explain certain behaviour of signal. In wavelets they are uncovered by changing the resolution of how the data is discreet-ized, you might think of them as simpler pencil drawn caricature of a landscape (raw signal).
  • 4.
    edited July 2014

    The "diffraction" is even visible (top image) by just averaging the data at each point over the 65 years. It then seems to extend over most ocean areas. I wonder what could be causing that.

    The second image is the standard deviation over entire history at each point. It looks like the equatorial currents reduce the variation as expected, but there seem to be islands high variation in the middle of the stream.

    Both images are light=low dark=high.

    mean-std

    Comment Source:The "diffraction" is even visible (top image) by just averaging the data at each point over the 65 years. It then seems to extend over most ocean areas. I wonder what could be causing that. The second image is the standard deviation over entire history at each point. It looks like the equatorial currents reduce the variation as expected, but there seem to be islands high variation in the middle of the stream. Both images are light=low dark=high. <a href="http://imgur.com/Cb16gAG"><img src="http://i.imgur.com/Cb16gAG.png" title="mean-std" alt="mean-std" /></a>
  • 5.

    Daniel

    I took your two images and 'Edge Detect ' filtered them:

    Odd Patterns: Daniel

    You need to get the CDF to work and play with the sliders so you could investigate the images.

    On the top image I see some eddies in the dark region, the bottom standard deviation looks like biased towards horizontal direction, I suspect an artifact of computation

    Dara

    Comment Source:Daniel I took your two images and 'Edge Detect ' filtered them: [Odd Patterns: Daniel](http://mathematica.lossofgenerality.com/2014/07/06/odd-patterns-1/) You need to get the CDF to work and play with the sliders so you could investigate the images. On the top image I see some eddies in the dark region, the bottom standard deviation looks like biased towards horizontal direction, I suspect an artifact of computation Dara
  • 6.

    Definitely have to distinguish them from artifacts such as moire patterns.

    Otherwise, one of the most interesting forms of waves are seiche waves. These are either formed by two waves traveling in opposite directions or by nonlinear resonance forming a standing wave pattern. Also cnoidal waves can form periodic patterns.

    I think standing waves and dipoles are worth pursuing. The SOI used to characterize ENSO is really a standing wave dipole. The value at Tahiti is almost always the opposite in sign to that at Darwin. I can imagine that this pattern might continue across the Pacific, but the problem is that islands with historic atmospheric records are rather sparse. But with the advent of satellite readings these are likely to become more obvious.

    Comment Source:Definitely have to distinguish them from artifacts such as moire patterns. Otherwise, one of the most interesting forms of waves are seiche waves. These are either formed by two waves traveling in opposite directions or by nonlinear resonance forming a standing wave pattern. Also cnoidal waves can form periodic patterns. I think standing waves and dipoles are worth pursuing. The SOI used to characterize ENSO is really a standing wave dipole. The value at Tahiti is almost always the opposite in sign to that at Darwin. I can imagine that this pattern might continue across the Pacific, but the problem is that islands with historic atmospheric records are rather sparse. But with the advent of satellite readings these are likely to become more obvious.
  • 7.

    Dara, can you please also post links directly to the cdf files. It turns out that CDFPlayer is not supported as browser plugin on linux. If I have links directly to the files I can dowload them and run them with Mathematica.

    Comment Source:Dara, can you please also post links directly to the cdf files. It turns out that CDFPlayer is not supported as browser plugin on linux. If I have links directly to the files I can dowload them and run them with Mathematica.
  • 8.

    I think the horizontal direction of the standard deviation image is physical. The light band along the equator is a region of very low variability. I think it is the result of the equatorial current and trade winds keeping the temperatures stable over time and smoothed out in space. I am curious about the dark patches in the middle of this region.

    Comment Source:I think the horizontal direction of the standard deviation image is physical. The light band along the equator is a region of very low variability. I think it is the result of the equatorial current and trade winds keeping the temperatures stable over time and smoothed out in space. I am curious about the dark patches in the middle of this region.
  • 9.
    edited July 2014

    Daniel

    This is standalone CDF, let me know if you could view it: odd patterns STANDALONE CDF

    This is Matehmatica file odd patterns .nb

    Comment Source:Daniel This is standalone CDF, let me know if you could view it: [odd patterns STANDALONE CDF](http://files.lossofgenerality.com/odd_pattern_danCDF_standalone.cdf) This is Matehmatica file [odd patterns .nb](http://files.lossofgenerality.com/odd_pattern_dan.nb)
  • 10.

    Daniel let me know if you could view the STANDALONE CDF files. Because we will generate them automatically in our servers and I wanna make sure the are viewable by all. D

    Comment Source:Daniel let me know if you could view the STANDALONE CDF files. Because we will generate them automatically in our servers and I wanna make sure the are viewable by all. D
  • 11.

    Hi Dara, Thanks! I can play the CDF.

    Comment Source:Hi Dara, Thanks! I can play the CDF.
  • 12.

    If you move the bottom slide to left and the top to left as well, you see square patterns on the top plot which was the top plot in your post earlier. I suspect the squares are caused by the original grid and averaging. Move to slider to 3.82 you see eddies in your original DARK region left middle to top region.

    Standard deviation shows horizontal edges perhaps you know what they are.

    Dara

    Comment Source:If you move the bottom slide to left and the top to left as well, you see square patterns on the top plot which was the top plot in your post earlier. I suspect the squares are caused by the original grid and averaging. Move to slider to 3.82 you see eddies in your original DARK region left middle to top region. Standard deviation shows horizontal edges perhaps you know what they are. Dara
  • 13.
    edited July 2014

    Dara, there is a possibility that converting the raw data to png and back to a matrix introduced some artifacts.

    Comment Source:Dara, there is a possibility that converting the raw data to png and back to a matrix introduced some artifacts.
  • 14.
    edited July 2014

    I have taken the data for the whole world with the full 65 year history, computed the historical median for each location (left image) & passed the result through a local rank filter (right image). It seem looks like the effect I am seeing is primarily localized to near equatorial waters. There is a band from Central America to North Africa/Spain that could be associated with the Gulf Stream. I still can not think of a reason why temperatures associated with ocean currents should show relatively short wavelength spatial oscillations.

    Also, since these are long time averaged temperatures, it seems these oscillations would need to be static since if the were moving the time averaging should cancel them out. This should happen even if they were standing waves. They could only be dynamic if they were completely synchronized with the sampling frenquency, ie the frequency of the oscilation would need to be an integer multiple of 24hrs and the velocitity would need to be an integer number of wavelength per 24hrs.

    median local rank

    Comment Source:I have taken the data for the whole world with the full 65 year history, computed the historical median for each location (left image) & passed the result through a local rank filter (right image). It seem looks like the effect I am seeing is primarily localized to near equatorial waters. There is a band from Central America to North Africa/Spain that could be associated with the Gulf Stream. I still can not think of a reason why temperatures associated with ocean currents should show relatively short wavelength spatial oscillations. Also, since these are long time averaged temperatures, it seems these oscillations would need to be static since if the were moving the time averaging should cancel them out. This should happen even if they were standing waves. They could only be dynamic if they were completely synchronized with the sampling frenquency, ie the frequency of the oscilation would need to be an integer multiple of 24hrs and the velocitity would need to be an integer number of wavelength per 24hrs. <img src="http://i.imgur.com/OT7P5x1.png" title="median local rank" alt="median local rank"/>
  • 15.
    edited July 2014

    Dara, there is a possibility that converting the raw data to png and back to a matrix introduced some artifacts.

    Possibly, unless you zoomed in or out, I did not zoom

    Dara

    Comment Source:> Dara, there is a possibility that converting the raw data to png and back to a matrix introduced some artifacts. Possibly, unless you zoomed in or out, I did not zoom Dara
  • 16.

    why don't you send me your output in csv or image format of some kind

    Comment Source:why don't you send me your output in csv or image format of some kind
  • 17.

    And I cut & paste from my screen did not use png

    Comment Source:And I cut & paste from my screen did not use png
  • 18.

    I am sure the patterns you see are for real, at least some of them, I see them in the wavelet decompositions.

    Comment Source:I am sure the patterns you see are for real, at least some of them, I see them in the wavelet decompositions.
  • 19.

    Hi Dara, I have sent you a csv of the median image by email.

    Comment Source:Hi Dara, I have sent you a csv of the median image by email.
  • 20.
    edited July 2014

    Daniel

    Entropy Filter applied to historical temperature data

    I am almost sure that you first image above in post 15 is smoothed and it is not your raw data you sent me in a csv file.

    So to be sure I plotted 3 versions of the data:

    1. your csv file RAW without any interpolation
    2. ListStremPlot which smoothes image from pixel to pixel
    3. First day of 2010 raw temperature (no averaging)

    It turns out that the Entropy Filter clearly shows that all three have dark equator but there are bands of high information content. The more image is smoothed or interpolated the more these bands show up. Why? Smoothing is addition of information to get rid of sudden random jumps of intensity/colour, therefore more detectable change in information.

    References:

    Entropy Filter

    Density Plot

    Comment Source:Daniel [Entropy Filter applied to historical temperature data](http://files.lossofgenerality.com/odd_patterns_dan2.pdf) I am almost sure that you first image above in post 15 is smoothed and it is not your raw data you sent me in a csv file. So to be sure I plotted 3 versions of the data: 1. your csv file RAW without any interpolation 2. ListStremPlot which smoothes image from pixel to pixel 3. First day of 2010 raw temperature (no averaging) It turns out that the Entropy Filter clearly shows that all three have dark equator but there are bands of high information content. The more image is smoothed or interpolated the more these bands show up. Why? Smoothing is addition of information to get rid of sudden random jumps of intensity/colour, therefore more detectable change in information. References: [Entropy Filter](http://reference.wolfram.com/mathematica/ref/EntropyFilter.html) [Density Plot](http://reference.wolfram.com/mathematica/ref/ListDensityPlot.html)
  • 21.

    Daniel said:

    I think the horizontal direction of the standard deviation image is physical. The light band along the equator is a region of very low variability. I think it is the result of the equatorial current and trade winds keeping the temperatures stable over time and smoothed out in space. I am curious about the dark patches in the middle of this region.

    Are you doing any sort of seasonal adjustment? People usually do something like subtracting the mean temperature for all 7th of Julys, from the value for each 7th of July. If not you may be seeing variations in seasonal variability, which would obscure things to do with currents and wind.

    Comment Source:Daniel said: > I think the horizontal direction of the standard deviation image is physical. The light band along the equator is a region of very low variability. I think it is the result of the equatorial current and trade winds keeping the temperatures stable over time and smoothed out in space. I am curious about the dark patches in the middle of this region. Are you doing any sort of seasonal adjustment? People usually do something like subtracting the mean temperature for all 7th of Julys, from the value for each 7th of July. If not you may be seeing variations in seasonal variability, which would obscure things to do with currents and wind.
  • 22.

    Graham, I did not do any seasonal adjustments. I thought that averaging over the entire history would smooth out seasonal effects. For the ICA analysis which used the individual historical values, I standardized the histories to have zero mea and unit variance.

    Comment Source:Graham, I did not do any seasonal adjustments. I thought that averaging over the entire history would smooth out seasonal effects. For the ICA analysis which used the individual historical values, I standardized the histories to have zero mea and unit variance.
  • 23.

    I wonder if the way the Pacific equatorial band that is visible in a number the images forks towards the East is an effect of the ITZC that Graham mentioned here

    Comment Source:I wonder if the way the Pacific equatorial band that is visible in a number the images forks towards the East is an effect of the ITZC that Graham mentioned [here](http://forum.azimuthproject.org/discussion/1386/boring-weather-interesting-climate/?Focus=11469#Comment_11469)
  • 24.
    edited July 2014

    Dara, I did not smooth the image, but I am fairly sure the image is enlarged. The underlying matrix is only 25 by 79 points for the Pacific and the world is 73 by 144 points. The images are too large to be 1 pixel per point.

    Comment Source:Dara, I did not smooth the image, but I am fairly sure the image is enlarged. The underlying matrix is only 25 by 79 points for the Pacific and the world is 73 by 144 points. The images are too large to be 1 pixel per point.
  • 25.

    Ok Daniel if you enlarge then you introduce BLOCKs into the data and any filtering is altered accordingly.

    I posted a pdf to show you how smoothing and interpolation exaggarated your patterns, but they are there from the getgo

    D

    Comment Source:Ok Daniel if you enlarge then you introduce BLOCKs into the data and any filtering is altered accordingly. I posted a pdf to show you how smoothing and interpolation exaggarated your patterns, but they are there from the getgo D
  • 26.

    Entropy Filter is Sigma -pLog(p) for the surrounding pixels of a 2D matrix/image.

    This is information content from shannon theory

    Comment Source:Entropy Filter is Sigma -pLog(p) for the surrounding pixels of a 2D matrix/image. This is information content from shannon theory
  • 27.

    Daniel (#23), averaging over the entire history would smooth out out seasonal effects for means, but not sds.

    Comment Source:Daniel (#23), averaging over the entire history would smooth out out seasonal effects for means, but not sds.
  • 28.
    edited July 2014

    Graham, the wave train and interefence pattern like features are in the averaged image. The sd mainly measure the seasonal variation which mostly dominates the long term variation, except close to the equator. The ICA, NMF and SVD components try to explain the total variation in terms of a small number of latent basis patterns. The components also pick up these odd features, but they seem to decompose the interference like patterns into a superposition of wave train patterns, which is to be expected from these algorithms. Note that I am using the terms "wave" and "interference" to describe the visual appearance of the patterns; I have no data about how these patterns are created or their dynamic behaviour. That is what I would like to figure out.

    Comment Source:Graham, the wave train and interefence pattern like features are in the averaged image. The sd mainly measure the seasonal variation which mostly dominates the long term variation, except close to the equator. The ICA, NMF and SVD components try to explain the total variation in terms of a small number of latent basis patterns. The components also pick up these odd features, but they seem to decompose the interference like patterns into a superposition of wave train patterns, which is to be expected from these algorithms. Note that I am using the terms "wave" and "interference" to describe the visual appearance of the patterns; I have no data about how these patterns are created or their dynamic behaviour. That is what I would like to figure out.
  • 29.

    Daniel, I understand you're mainly talking about the interference-type patterns. I was just concerned you might be misinterpreting the sd images (eg in comment 9).

    Comment Source:Daniel, I understand you're mainly talking about the interference-type patterns. I was just concerned you might be misinterpreting the sd images (eg in comment 9).
  • 30.
    edited July 2014

    Graham, in comment 9 I was thinking of both the seasonal variation and long term variation. I was hypothesizing that the streak of low sd (low variation, seasonal and long term) along the equator (comment 5 lower image, light colour = low sd) was due to to the stabilizing effect of the ocean currents and/or the trade winds. I was curious about why there were darker regions of higher sd in the very middle of this. It seems like the ITZC you pointed out on another tread could help explain this.

    Comment Source:Graham, in comment 9 I was thinking of both the seasonal variation and long term variation. I was hypothesizing that the streak of low sd (low variation, seasonal and long term) along the equator (comment 5 lower image, light colour = low sd) was due to to the stabilizing effect of the ocean currents and/or the trade winds. I was curious about why there were darker regions of higher sd in the very middle of this. It seems like the ITZC you pointed out on another tread could help explain this.
  • 31.
    edited July 2014

    A rough estimate of the wavelength of these patterns.

    The wavelength seems to be about 2 grid points of the data. There are 144 horizontal grid points on the map. Circumference of the earth is ~40000 km.

    2*40000/144=555 km

    Since the grid is close to what the grid can resolve the relative error of this estimate is probably around +/- 50%.

    If these patterns are really some kind of moving wave, then they have to be moving in sync with the measurement process, since otherwise they would get averaged away in the mean images. This would imply a speed of that is an integral multiple of around 555 km/day = 23km/hour = 6.4 m/s.

    Comment Source:A rough estimate of the wavelength of these patterns. The wavelength seems to be about 2 grid points of the data. There are 144 horizontal grid points on the map. Circumference of the earth is ~40000 km. 2*40000/144=555 km Since the grid is close to what the grid can resolve the relative error of this estimate is probably around +/- 50%. If these patterns are really some kind of moving wave, then they have to be moving in sync with the measurement process, since otherwise they would get averaged away in the mean images. This would imply a speed of that is an integral multiple of around 555 km/day = 23km/hour = 6.4 m/s.
  • 32.

    Hello Daniel

    This is really good approach to measure some aspects of these patterns like wavelength or speed.

    Comment Source:Hello Daniel This is really good approach to measure some aspects of these patterns like wavelength or speed.
  • 33.

    555 km/day = 23km/hour = 6.4 m/s.

    Hmm. I was think these patterns must be an artefact, but here's a quote about trade winds:

    their constancy of direction and speed (about 7m/s) is remarkable.

    Comment Source:> 555 km/day = 23km/hour = 6.4 m/s. Hmm. I was think these patterns must be an artefact, but here's a quote about trade winds: > their constancy of direction and speed (about 7m/s) is remarkable.
  • 34.

    I have posted a question regarding these patterns on the earth science stack exchange site.

    Comment Source:I have posted a [question](http://earthscience.stackexchange.com/questions/2263/odd-structures-in-66-year-daily-global-temperature-data) regarding these patterns on the earth science stack exchange site.
  • 35.

    Hello Daniel

    This pattern issue you have raised is paramount in our understanding of the data. I am looking at the volumetric patterns from the 4D data of the grid sensors and see there are definitely patterns.

    However I rely on two levels of researching the patterns:

    1. Wavelet decomposition
    2. Image processing as you are doing

    The data is deceptive, one looking at them might think "Oh well another time-series", actually it is quite complex in nature and clearly deals with a sophisticate dynamical system.

    My worry is that we are not looking at all the needed data or we are not look at high enough resolution.

    Dara

    Comment Source:Hello Daniel This pattern issue you have raised is paramount in our understanding of the data. I am looking at the volumetric patterns from the 4D data of the grid sensors and see there are definitely patterns. However I rely on two levels of researching the patterns: 1. Wavelet decomposition 2. Image processing as you are doing The data is deceptive, one looking at them might think "Oh well another time-series", actually it is quite complex in nature and clearly deals with a sophisticate dynamical system. My worry is that we are not looking at all the needed data or we are not look at high enough resolution. Dara
  • 36.
    nad
    edited July 2014

    They could be aliasing effects, but it doesn't seem so to me at the moment. At least I find one could see ripples or at least rather stable cold spots in front of South America already by just looking at the Comment 11553 animation John linked to. But may be the animation is based on the same images and so it can be that some strange aliasing subsampling would also show up here. Do you know whether those are the same images? Sofar the ripples look to me rather as some kind of "heat transfer ripples", just as one has "mechanical ripples in windy water. In particular right at the equator one can see meander like patterns, just as in fluid dynamics, which must be some interplay of water and wind currents. I have however no feeling for the speed of heat transfer from air to water.

    Following Grahams comment:

    Hmm. I was think these patterns must be an artefact, but here’s a quote about trade winds:

    which is in accordance with this chart, that would suggest that it could eventually make sense to look for heat oscillations within the trade winds.

    Comment Source:They could be aliasing effects, but it doesn't seem so to me at the moment. At least I find one could see ripples or at least rather stable cold spots in front of South America already by just looking at the <a href="http://forum.azimuthproject.org/discussion/169/el-nino-southern-oscillation-enso/?Focus=11553#Comment_11553">Comment 11553</a> animation John linked to. But may be the animation is based on the same images and so it can be that some strange aliasing subsampling would also show up here. Do you know whether those are the same images? Sofar the ripples look to me rather as some kind of "heat transfer ripples", just as one has "mechanical ripples in windy water. In particular right at the equator one can see meander like patterns, just as in fluid dynamics, which must be some interplay of water and wind currents. I have however no feeling for the speed of heat transfer from air to water. Following Grahams comment: >Hmm. I was think these patterns must be an artefact, but here’s a quote about trade winds: which is in accordance with <a href="https://en.wikipedia.org/wiki/File:Global_Wave_Height_Speed.jpg">this chart,</a> that would suggest that it could eventually make sense to look for heat oscillations within the trade winds.
Sign In or Register to comment.