Options

Creating animations and other visualizations

Hi everyone,

I created an animation of temperature over time. Here is the video and code. I am working on creating more animations. I would like to plot the so called temperature anomalies . If people have any ideas of what would be cool to visualize, I would to love talk about it.

I think there is a lot room for interactive visualizations. Maybe we can expose and visualize the internals of Ludescher et al's model?

The tools I was thinking of using are Python, d3, and P5.

«1

Comments

  • 1.

    Here is a similar work, you need not average the data you could animate the time-slices:

    The links are on the first discussion:

    youtube rendering

    So on each frame a machine learning algorithm and a machine vision algorithm were applied, the results then combined and rendered into a video for youtube.

    D

    Comment Source:Here is a similar work, you need not average the data you could animate the time-slices: The links are on the first discussion: [youtube rendering](http://forum.azimuthproject.org/discussion/1450/cluster-vision-surface-temperatures-2010/#Item_11) So on each frame a machine learning algorithm and a machine vision algorithm were applied, the results then combined and rendered into a video for youtube. D
  • 2.
    nad
    edited September 2014

    I created an animation of temperature over time. Here is the video and code. I am working on creating more animations. I would like to plot the so called temperature anomalies . If people have any ideas of what would be cool to visualize, I would to love talk about it.

    I think there is a lot room for interactive visualizations.

    Hello Rafael

    Nice visualization. The image jumps in the video are though quite fast, so I think an interactive application, where one could scroll through could be indeed very helpful (something like the time line in Blender).

    We had looked at the sea surface temperatures in this thread and wondered about the red spot in the Barents sea.

    My husband had helped me to plot the global average (sea and surface) temperature anomalies from HadCrut together with methane and CO2 here

    But then when I started to scrutinize the underlying data, and in particular the air temperatures per station it turned out that this data was in a rather bad shape, especially in certain regions, like northern Siberia, central Africa and parts of Canada. That is I even wonder how one can make grid data with this.

    Nathan Urban said (I hope I cite correctly) that unfortunately also other air temperature sets are to a great extent based on these station data.

    Nevertheless one should check wether the other sets which are used by IPCC are equally rotten.

    I hope the sea temperatures are in a better shape, but I guess one should check these as well.

    I think without such a check a further reasoning is problematic, that is how useful is all this machine learning machinery if the base data is rotten?

    Comment Source:>I created an animation of temperature over time. Here is the video and code. I am working on creating more animations. I would like to plot the so called temperature anomalies . If people have any ideas of what would be cool to visualize, I would to love talk about it. >I think there is a lot room for interactive visualizations. Hello Rafael Nice visualization. The image jumps in the video are though quite fast, so I think an interactive application, where one could scroll through could be indeed very helpful (something like the time line in Blender). We had looked at the <a href="http://forum.azimuthproject.org/discussion/169/el-nino-southern-oscillation-enso/?Focus=11553#Comment_11553">sea surface temperatures in this thread </a> and <a href="http://forum.azimuthproject.org/discussion/169/el-nino-southern-oscillation-enso/?Focus=11703#Comment_11703">wondered about the red spot in the Barents sea.</a> My husband had helped me to plot the global average (sea and surface) temperature anomalies from HadCrut together with methane and CO2 <a href="http://www.daytar.de/art/co2ch4TempViz/index.html">here</a> But then when I started to scrutinize the underlying data, and in particular the air temperatures per station it turned out that this data was <a href="http://www.randform.org/blog/?p=5642">in a rather bad shape</a>, especially in certain regions, like northern Siberia, central Africa and parts of Canada. That is I even wonder how one can make grid data with this. Nathan Urban said (I hope I cite correctly) that unfortunately also other air temperature sets are to a great extent based on these station data. Nevertheless one should check wether the <a href="http://www.ipcc-data.org/observ/clim/ar4_global.html">other sets</a> which are used by IPCC are equally rotten. I hope the sea temperatures are in a better shape, but I guess one should check these as well. I think without such a check a further reasoning is problematic, that is how useful is all this machine learning machinery if the base data is rotten?
  • 3.

    I do movies of SST, using Hi Res (1/4°) AVHRR from NOAA. There's a page here, exmple of recent ENSO here. The underlying WebGL dataset is here.

    Here is a WebGL plot of temperature anomalies. It's done directly from GHCN and SST station data. The idea is that the shading is correct at the stations which are nodes of a triangular mesh, and linearly shaded in between. The notion of anomaly that I use is deviation from a weighrted linear regression, current value. The result depends quite a lot on what you use, and that is better for spatial fidelity. It wouldn't do for calculating a global trend.

    I make the movies using ffmpeg. Recommended.

    Comment Source:I do movies of SST, using Hi Res (1/4&deg;) AVHRR from NOAA. There's a page <a href="http://www.moyhu.blogspot.com.au/p/sst-regional-movies-as-described-here-i.html">here</a>, exmple of recent ENSO <a href="?WxK=0">here</a>. The underlying WebGL dataset is <a href="http://www.moyhu.blogspot.com.au/p/blog-page.html">here</a>. <a href="http://www.moyhu.blogspot.com.au/p/blog-page_24.html">Here</a> is a WebGL plot of temperature anomalies. It's done directly from GHCN and SST station data. The idea is that the shading is correct at the stations which are nodes of a triangular mesh, and linearly shaded in between. The notion of anomaly that I use is deviation from a weighrted linear regression, current value. The result depends quite a lot on what you use, and that is better for spatial fidelity. It wouldn't do for calculating a global trend. I make the movies using ffmpeg. Recommended.
  • 4.

    I do movies of SST, using Hi Res (1/4°) AVHRR from NOAA. There’s a page here, exmple of recent ENSO here. The underlying WebGL dataset is here.

    the webGl looks good. I had though problems with the tabs on the side, I didn't fully understand their function in particular it seems you can't switch years and cycle without reload.

    Here is a WebGL plot of temperature anomalies.

    Yes Nathan Urban was mentioning it already here in the forum and I had linked to it from our blog. My problem with the HADCRUT data is that I think it is problematic to average/gloss over the missing data. But you used the NOAA set. May be this is better (see also my comment above)

    Comment Source:>I do movies of SST, using Hi Res (1/4°) AVHRR from NOAA. There’s a page here, exmple of recent ENSO here. The underlying WebGL dataset is here. the webGl looks good. I had though problems with the tabs on the side, I didn't fully understand their function in particular it seems you can't switch years and cycle without reload. >Here is a WebGL plot of temperature anomalies. Yes Nathan Urban was mentioning it already here in the forum and I had linked to it from our blog. My problem with the HADCRUT data is that I think it is problematic to average/gloss over the missing data. But you used the NOAA set. May be this is better (see also my <a href="http://forum.azimuthproject.org/discussion/1489/creating-animations-and-other-vizualizations/?Focus=12729#Comment_12729">comment above</a>)
  • 5.

    "I had though problems with the tabs on the side" You need to make a choice of year, month, day and then press "New Plot". Then it downloads new data using XMLHttprequest and draws the new plot.

    "I think it is problematic to average/gloss over the missing data" The idea of the mesh is that you don't have empty cells. It interpolates - shading has to. It's using station data, not gridded.

    Comment Source:<i>"I had though problems with the tabs on the side"</i> You need to make a choice of year, month, day and then press "New Plot". Then it downloads new data using XMLHttprequest and draws the new plot. <i>"I think it is problematic to average/gloss over the missing data"</i> The idea of the mesh is that you don't have empty cells. It interpolates - shading has to. It's using station data, not gridded.
  • 6.
    nad
    edited October 2014

    You need to make a choice of year, month, day and then press “New Plot”. Then it downloads new data using XMLHttprequest and draws the new plot.

    I did that and then pressed cycle, but the cycle went over the previous year and only I think one month of the new chosen year. The new plot worked only after reloading.

    The idea of the mesh is that you don’t have empty cells. It interpolates - shading has to. It’s using station data, not gridded.

    The problem with gridded data or any other form of interpolation is that it may seclude essential features. If there are essential features which may pose problems, then not seeing them means usually that the problems won't be tackled. However often it is better to tackle problems at an early stage, at least one should know about them and keep an eye on them.

    For the case of the Hadley/CRUTEM station data I think that the interpolation pretends that there is a well covered data set, which there isn't. And especially in northern Siberia and northern Canada, it looks to me that something possibly worrisome is going on there. I mentioned above the red spot in the Barent sea. In your nice sea surface temp webGl animation it looks as if a large part of the SST anomalies around the north pole are shooting through the ceiling. But then I don't know how good the data set is. And the CRUTEM station data doesn't help here much, since there are (meanwhile) too few stations there.

    By the way you can blockquote with a >sign before the text here in Markdown (below is a help button)

    Comment Source:>You need to make a choice of year, month, day and then press “New Plot”. Then it downloads new data using XMLHttprequest and draws the new plot. I did that and then pressed cycle, but the cycle went over the previous year and only I think one month of the new chosen year. The new plot worked only after reloading. >The idea of the mesh is that you don’t have empty cells. It interpolates - shading has to. It’s using station data, not gridded. The problem with gridded data or any other form of interpolation is that it may seclude essential features. If there are essential features which may pose problems, then not seeing them means usually that the problems won't be tackled. However often it is better to tackle problems at an early stage, at least one should know about them and keep an eye on them. For the case of the Hadley/CRUTEM station data I think that the interpolation pretends that there is a well covered data set, which there isn't. And especially in northern Siberia and northern Canada, it looks to me that something possibly worrisome is going on there. I mentioned above the red spot in the Barent sea. In your nice sea surface temp webGl animation it looks as if a large part of the SST anomalies around the north pole are shooting through the ceiling. But then I don't know how good the data set is. And the CRUTEM station data doesn't help here much, since there are (meanwhile) too few stations there. By the way you can blockquote with a >sign before the text here in Markdown (below is a help button)
  • 7.

    nad, "but the cycle went over the previous year"

    Ah yes, the cycling system has become a bit flaky and needs attention. Sorry about that. But I think the date menus should be OK.

    Comment Source:nad, "but the cycle went over the previous year" Ah yes, the cycling system has become a bit flaky and needs attention. Sorry about that. But I think the date menus should be OK.
  • 8.
    edited October 2014

    Nick Stokes and Rafael Turner! Hi! I'd like to post some of your images, or links to them, on the blog series "Visualizing Climate Data".

    The easiest first thing would be for me to post a copy of this blog article to Azimuth:

    Would that be okay, Nick? I would include a link back to your blog.

    If you wrote a paragraph or two introducing yourself, I could start the article with that. E.g. it could say what you do, what you do on your blog... and then we could segue over to the blog article, as a kind of "sample".

    Comment Source:Nick Stokes and Rafael Turner! Hi! I'd like to post some of your images, or links to them, on the blog series "Visualizing Climate Data". The easiest first thing would be for me to post a copy of this blog article to Azimuth: * Nick Stokes, [WebGL map of past GHCN/SST station temperatures](http://www.moyhu.blogspot.com.au/p/blog-page_24.html). Would that be okay, Nick? I would include a link back to your blog. If you wrote a paragraph or two introducing yourself, I could start the article with that. E.g. it could say what you do, what you do on your blog... and then we could segue over to the blog article, as a kind of "sample".
  • 9.
    nad
    edited October 2014

    Nick Stokes and Rafael Turner! Hi! I’d like to post some of your images, or links to them, on the blog series “Visualizing Climate Data”.

    What's the reason for not showing our visualization about the eroding CRUTEM temperature file ? Are you afraid that this might fuel some climate sceptics? But it is a fact.

    LIttle warning:Unfortunately it might be that the interactive application behind the images might not be working, since it turned out that the source code of WebGL earth was not self-contained, that is there are links in the program and they once in a while shuffle around or alter the corresponding files. I had written to the person who seems responsible for this, namely Petr Pidal of Klokan technologies, but I have sofar gotten only the response that they intend to look at this. But the choice of images is here anyways the more important point. The application takes very long to load, since the CRUTEM data file is so big.

    Comment Source:> Nick Stokes and Rafael Turner! Hi! I’d like to post some of your images, or links to them, on the blog series “Visualizing Climate Data”. What's the reason for not showing [our visualization](http://www.randform.org/blog/?p=5642) about the eroding CRUTEM temperature file ? Are you afraid that this might fuel some climate sceptics? But it is a fact. LIttle warning:Unfortunately it might be that the interactive application behind the images might not be working, since it turned out that the source code of WebGL earth was not self-contained, that is there are links in the program and they once in a while shuffle around or alter the corresponding files. I had written to the person who seems responsible for this, namely [Petr Pidal of Klokan technologies](http://www.klokantech.com/company/), but I have sofar gotten only the response that they intend to look at this. But the choice of images is here anyways the more important point. The application takes very long to load, since the CRUTEM data file is so big.
  • 10.
    edited October 2014

    What’s the reason for not showing our visualization about the eroding CRUTEM temperature file ? Are you afraid that this might fuel some climate sceptics?

    Did you ask me to show this visualization on the Azimuth blog? If so, did I read your request? I don't think so. I'm way behind on reading posts on this forum. I don't remember you ever asking me to do a blog article on this. So, the main reason I haven't done it is that I didn't know you wanted me to. I'm not "afraid" to do it. I never saw this visualization until 10 minutes ago.

    Are you asking me to put a copy of this on the Azimuth blog? If so, it would help a lot create the blog article you desire on the Azimuth Wiki, following the directions here:

    These directions are a bit outdated because now you can use either HTML or Markdown on the Azimuth blog.

    If the interactive application is not working, you can still include links to the images in your own blog article, like this:

    Also, it would be helpful if you explain a bit about what you mean here:

    Black rectangles are invalid data.

    Who says the data is invalid, and what are the most common causes of invalid data?

    By the way, the post says:

    That is already visible is that the temperature data is quite deteriorating (please see also the previous randform post on the topic of deteriorization of data).

    However, the link does not go to a post on the deterioration of data.

    Comment Source:> What’s the reason for not showing our visualization about the eroding CRUTEM temperature file ? Are you afraid that this might fuel some climate sceptics? Did you ask me to show this visualization on the Azimuth blog? If so, did I read your request? I don't think so. I'm way behind on reading posts on this forum. I don't remember you ever asking me to do a blog article on this. So, the main reason I haven't done it is that I didn't know you wanted me to. I'm not "afraid" to do it. I never saw this visualization until 10 minutes ago. _Are_ you asking me to put a copy of this on the Azimuth blog? If so, it would help a lot create the blog article you desire on the Azimuth Wiki, following the directions here: * [How to blog](http://www.azimuthproject.org/azimuth/show/How+to#blog). These directions are a bit outdated because now you can use either HTML or Markdown on the Azimuth blog. If the interactive application is not working, you can still include links to the images in your own blog article, like this: <img src = "http://www.randform.org/blog/wp-content/2014/08/Africa2009Jan450.jpg" alt = ""/> Also, it would be helpful if you explain a bit about what you mean here: > Black rectangles are invalid data. Who says the data is invalid, and what are the most common causes of invalid data? By the way, the post says: > That is already visible is that the temperature data is quite deteriorating (please see also the [previous randform post](http://www.randform.org/blog/?p=5626) on the topic of deteriorization of data). However, the link does not go to a post on the deterioration of data.
  • 11.
    nad
    edited October 2014

    Did you ask me to show this visualization on the Azimuth blog? If so, did I read your request? I don’t think so. I’m way behind on reading posts on this forum.

    Did Nick Stokes and Rafael Turner ask?

    Yes it has become more difficult to keep track with the posts on this forum and the fact that the posts with minor technical remarks look the same as e.g. important announcements doesn't make the reading easier. I had made similar remarks earlier, but since I don't want to care about the software infrastructure of Azimuth I don't want to further complain, the forum itself is of course already an improvement. Anyways I am asking you now hereby officially ---in case you should read this---- to post this, I think it might be important.

    Are you asking me to put a copy of this on the Azimuth blog? If so, it would help a lot create the blog article you desire on the Azimuth Wiki, following the directions here:

    How to blog.

    These directions are a bit outdated because now you can use either HTML or Markdown on the Azimuth blog.

    I used HTML (I don't really like Markdown) on our blog and I would copy most part over, so it makes more sense to paste that into Wordpress directly.

    Who says the data is invalid, and what are the most common causes of invalid data?

    I said this. That is CRUTEM calls it missing data, it is a -999 in the datafile. But missing data is also invalid data and I found the more general term clearer, since I find that the ceased data of closed stations (which is indicated by the closing date of a station and not by -999) is also missing.

    I don't know what the most common causes of invalid data are, I haven't found any information on this. It could be on the side of the stations as well as on the side of CRUTEM. I took the most recent file that was offered by CRUTEM. It could be that they are updating only every 5 years or it it could be that they just don't get any info.

    By the way, the post says:

    That is already visible is that the temperature data is quite deteriorating (please see also the previous randform post on the topic of deteriorization of data).

    Sure it does I wrote in that post:

    I am still collecting data on global employment in order to better investigate the replacement of human work by machines. Unfortunately it turned out that the International Labour Organisation (ILO), which holds most of the original data restructured their IT-sector. This means in particular that some older data can’t be reproduced any more. Above you can see that the worldwide employment went down on average since the nineties. I keep the data now here locally on our account as a copy from ILO in order to keep the findings reproducible. The data source as well as the source code for extracting it (GPL) are here. As always: if you spot some mistakes please let me know.

    If data can't be reproduced anymore then that is some sort of deterioration.

    Comment Source:>Did you ask me to show this visualization on the Azimuth blog? If so, did I read your request? I don’t think so. I’m way behind on reading posts on this forum. Did Nick Stokes and Rafael Turner ask? Yes it has become more difficult to keep track with the posts on this forum and the fact that the posts with minor technical remarks look the same as e.g. important announcements doesn't make the reading easier. I had made similar remarks earlier, but since I don't want to care about the software infrastructure of Azimuth I don't want to further complain, the forum itself is of course already an improvement. Anyways I am asking you now hereby officially ---in case you should read this---- to post this, I think it might be important. >Are you asking me to put a copy of this on the Azimuth blog? If so, it would help a lot create the blog article you desire on the Azimuth Wiki, following the directions here: >> How to blog. >These directions are a bit outdated because now you can use either HTML or Markdown on the Azimuth blog. I used HTML (I don't really like Markdown) on our blog and I would copy most part over, so it makes more sense to paste that into Wordpress directly. >Who says the data is invalid, and what are the most common causes of invalid data? I said this. That is CRUTEM [calls it](http://www.cru.uea.ac.uk/cru/data/temperature/crutem4/station-data.htm) missing data, it is a -999 in the datafile. But missing data is also invalid data and I found the more general term clearer, since I find that the ceased data of closed stations (which is indicated by the closing date of a station and not by -999) is also missing. I don't know what the most common causes of invalid data are, I haven't found any information on this. It could be on the side of the stations as well as on the side of CRUTEM. I took the most recent file that was offered by CRUTEM. It could be that they are updating only every 5 years or it it could be that they just don't get any info. >By the way, the post says: > >That is already visible is that the temperature data is quite deteriorating (please see also the previous randform post on the topic of deteriorization of data). Sure it does I wrote in that post: >_I am still collecting data on global employment in order to better investigate the replacement of human work by machines. Unfortunately it turned out that the International Labour Organisation (ILO), which holds most of the original data restructured their IT-sector. This means in particular that some older data can’t be reproduced any more. Above you can see that the worldwide employment went down on average since the nineties. I keep the data now here locally on our account as a copy from ILO in order to keep the findings reproducible. The data source as well as the source code for extracting it (GPL) are here. As always: if you spot some mistakes please let me know._ If data can't be reproduced anymore then that is some sort of deterioration.
  • 12.

    John, Sorry I've been missing for a couple of days. Yes, I'd be very happy for you to post images in an article, with links or whatever. I'll set about writing something. Nick

    Comment Source:John, Sorry I've been missing for a couple of days. Yes, I'd be very happy for you to post images in an article, with links or whatever. I'll set about writing something. Nick
  • 13.

    John, You asked for some bio information. I have been for my working life a scientist/mathematician with CSIRO in Australia, and I'm now a kind of emeritus there. I started out in ordinary differential equations, with an interest in complex analysis, but became more numerical, moving to pde and then computational fluid dynamics. I was the principal writer of a FEM-based PDE solver (Fastflo).

    After retiring, I followed more actively the climate debate, and started a blog Moyhu. I generally support the regular scientific view of climate change, which I am happy to debate in sceptical environments. I have taught myself various new programming tools - R, Javascript, HTML 5,WebGL. The blog monitors climate data regularly, and I use these tools for various active presentations, to make large data more digestible. I particularly like WebGL based spherical mapping.

    There is a gallery of Moyhu graphics arranged by method here.

    Comment Source:John, You asked for some bio information. I have been for my working life a scientist/mathematician with CSIRO in Australia, and I'm now a kind of emeritus there. I started out in ordinary differential equations, with an interest in complex analysis, but became more numerical, moving to pde and then computational fluid dynamics. I was the principal writer of a FEM-based PDE solver (Fastflo). After retiring, I followed more actively the climate debate, and started a blog <a href="http://moyhu.blogspot.com.au/">Moyhu</a>. I generally support the regular scientific view of climate change, which I am happy to debate in sceptical environments. I have taught myself various new programming tools - R, Javascript, HTML 5,WebGL. The blog monitors climate data regularly, and I use these tools for various active presentations, to make large data more digestible. I particularly like WebGL based spherical mapping. There is a gallery of Moyhu graphics arranged by method <a href="http://www.moyhu.blogspot.com.au/p/gallery-of-javascript-enhanced-graphics.html">here</a>.
  • 14.

    Nick, I didn't realize your specialty was fluid dynamics; I just figured you knew everything -- science, math, programming, etc.

    Welcome aboard!

    Comment Source:Nick, I didn't realize your specialty was fluid dynamics; I just figured you knew everything -- science, math, programming, etc. Welcome aboard!
  • 15.

    Thanks, Web. Glad to be able to join.

    Comment Source:Thanks, Web. Glad to be able to join.
  • 16.
    edited October 2014

    Nad wrote:

    [...] it has become more difficult to keep track with the posts on this forum and the fact that the posts with minor technical remarks look the same as e.g. important announcements doesn’t make the reading easier.

    Yes, it can be difficult, especially when people don't post articles with clear subject headers.

    Ideally, anyone who wants to post a blog article will follow the instructions on How to blog, which include:

    1) putting a draft blog article on the wiki,

    2) writing a short notification here with title "Blog: ....", linking to the blog article on the wiki.

    People here will see the notification here and begin the process of editing it, and then I can post it.

    But new members don't know this procedure, so I'm inviting some articles from them.

    I used HTML (I don’t really like Markdown) on our blog and I would copy most part over, so it makes more sense to paste that into Wordpress directly.

    Please copy it onto the wiki. I'll need to edit the English grammar and spelling, and other people may have comments too. I think we need to add a bit information, namely:

    1) a link to where CRUTEM discusses this missing data.

    2) if possible, a quick explanation of why there's missing data - everyone will want to know this.

    I can add this information if that helps.

    Comment Source:Nad wrote: > [...] it has become more difficult to keep track with the posts on this forum and the fact that the posts with minor technical remarks look the same as e.g. important announcements doesn’t make the reading easier. Yes, it can be difficult, especially when people don't post articles with clear subject headers. Ideally, anyone who wants to post a blog article will follow the instructions on [How to blog](http://www.azimuthproject.org/azimuth/show/How+to#blog), which include: 1) putting a draft blog article on the wiki, 2) writing a short notification here with title "Blog: ....", linking to the blog article on the wiki. People here will see the notification here and begin the process of editing it, and then I can post it. But new members don't know this procedure, so I'm inviting some articles from them. > I used HTML (I don’t really like Markdown) on our blog and I would copy most part over, so it makes more sense to paste that into Wordpress directly. Please copy it onto the wiki. I'll need to edit the English grammar and spelling, and other people may have comments too. I think we need to add a bit information, namely: 1) a link to [where CRUTEM discusses this missing data](http://www.cru.uea.ac.uk/cru/data/temperature/crutem4/station-data.htm). 2) if possible, a quick explanation of why there's missing data - everyone will want to know this. I can add this information if that helps.
  • 17.
    edited October 2014

    Nick wrote:

    Yes, I’d be very happy for you to post images in an article, with links or whatever. I’ll set about writing something.

    Great! But I'm pretty happy with your article, so I will start by copying it over to the Azimuth Wiki and adding your bio information. You can then fix it up a bit.

    Hmm. The WebGL doesn't seem to work on Wordpress blog. So, the article will have to mainly be a link to your blog.

    Comment Source:Nick wrote: > Yes, I’d be very happy for you to post images in an article, with links or whatever. I’ll set about writing something. Great! But I'm pretty happy with your article, so I will start by copying it over to the Azimuth Wiki and adding your bio information. You can then fix it up a bit. Hmm. The WebGL doesn't seem to work on Wordpress blog. So, the article will have to mainly be a link to your blog.
  • 18.
    edited October 2014
    John,
    The WebGL should be a function of the browser rather than the blog platform. If you paste into the HTML this incantation, it might work:













    I've noted at the source that there is a current problem with GHCN data for August and September - it seems to have little but US. Odd, because August had been OK. Earlier months are OK.
    Comment Source:John, The WebGL should be a function of the browser rather than the blog platform. If you paste into the HTML this incantation, it might work: <img src="https://sites.google.com/site/moyhudocs/pics/dec/white.png" style="height: 880px; left: 0px; position: absolute; width: 1460px;" /> <br /> <iframe src="https://s3-us-west-1.amazonaws.com/www.moyhu.org/GHCN/allghcn.html" style="height: 850px; position: absolute; width: 1420px;"></iframe><br /> <br /> <img src="https://sites.google.com/site/moyhudocs/pics/dec/white.png" style="height: 880px; width: 1460px;" /> <br /> I've noted at the source that there is a current problem with GHCN data for August and September - it seems to have little but US. Odd, because August had been OK. Earlier months are OK.
  • 19.

    John wrote:

    1) a link to where CRUTEM discusses this missing data.

    I linked to that explanation of the format already in the randform post, but there are no "discussions" about the missing data in it, apart from the small remark about the polish stations.

    2) if possible, a quick explanation of why there’s missing data - everyone will want to know this.

    As said above I don't know why it is missing -

    I can add this information if that helps.

    so surely that would be good. You can say this also right here and then I paste this into the blog post.

    Comment Source:John wrote: >1) a link to where CRUTEM discusses this missing data. I linked to that explanation of the format already in the randform post, but there are no "discussions" about the missing data in it, apart from the small remark about the polish stations. >2) if possible, a quick explanation of why there’s missing data - everyone will want to know this. As said [above](http://forum.azimuthproject.org/discussion/1489/creating-animations-and-other-visualizations/?Focus=12831#Comment_12831) I don't know why it is missing - >I can add this information if that helps. so surely that would be good. You can say this also right here and then I paste this into the blog post.
  • 20.

    Nick wrote:

    John, The WebGL should be a function of the browser rather than the blog platform. If you paste into the HTML this incantation, it might work...

    As with the previous HTML, this creates a large white region, wider than the usual width of a blog article. (On my blog, articles are 450 pixels wide.)

    Comment Source:Nick wrote: > John, The WebGL should be a function of the browser rather than the blog platform. If you paste into the HTML this incantation, it might work... As with the previous HTML, this creates a large white region, wider than the usual width of a blog article. (On my blog, articles are 450 pixels wide.)
  • 21.
    edited October 2014

    Nad wrote:

    John wrote:

    1) a link to where CRUTEM discusses this missing data.

    I linked to that explanation of the format already in the randform post, but there are no “discussions” about the missing data in it, apart from the small remark about the polish stations.

    Okay.

    2) if possible, a quick explanation of why there’s missing data - everyone will want to know this.

    As said above I don’t know why it is missing.

    Okay.

    I can add this information if that helps.

    so surely that would be good.

    Okay. This is a pretty famous issue: I'll try to find some decent information about it...

    Comment Source:Nad wrote: > John wrote: > > 1) a link to where CRUTEM discusses this missing data. > I linked to that explanation of the format already in the randform post, but there are no “discussions” about the missing data in it, apart from the small remark about the polish stations. Okay. > > 2) if possible, a quick explanation of why there’s missing data - everyone will want to know this. > As said above I don’t know why it is missing. Okay. > > I can add this information if that helps. > so surely that would be good. Okay. This is a pretty famous issue: I'll try to find some decent information about it...
  • 22.
    edited October 2014

    Nad: please take a look at this blog article based on yours, and improve it if you want. Let's discuss it there, not here.

    Comment Source:Nad: please take a look at [this blog article based on yours](http://forum.azimuthproject.org/discussion/1507/blog-exploring-climate-data-part-3/#Item_1), and improve it if you want. Let's discuss it there, not here.
  • 23.
    nad
    edited October 2014

    Nad: please take a look at this blog article based on yours, and improve it if you want. Let’s discuss it there, not here.

    If I follow the link I get the following message:

    Some problems were encountered

    You do not have sufficient privileges to edit the requested comment.

    Comment Source:>Nad: please take a look at this blog article based on yours, and improve it if you want. Let’s discuss it there, not here. If I follow the link I get the following message: >Some problems were encountered >You do not have sufficient privileges to edit the requested comment.
  • 24.
    edited October 2014

    Here's one possible explanation of the "increase of missing data": when new weather stations appear, they just appear. But when old stations disappear (or stop producing data) there's a mark for "missing data".

    This could be wrong, but it would explain the appearance of an increase in missing data without an actual decline in the number of stations.

    Comment Source:Here's one possible explanation of the &quot;increase of missing data&quot;: when new weather stations appear, they just appear. But when old stations disappear (or stop producing data) there's a mark for &quot;missing data&quot;. This could be wrong, but it would explain the appearance of an increase in missing data without an actual decline in the number of stations.
  • 25.

    Nad wrote:

    If I follow the link I get the following message...

    Sorry, typo! Try again please.

    Comment Source:Nad wrote: > If I follow the link I get the following message... Sorry, typo! Try again please.
  • 26.

    John, "As with the previous HTML, this creates a large white region" I create a backing white space, basically to override everything else. They are the elements with white.png.

    However, 450 pixels probably won't work. Then a link is probably better.

    Comment Source:John, <i>"As with the previous HTML, this creates a large white region"</i> I create a backing white space, basically to override everything else. They are the elements with white.png. However, 450 pixels probably won't work. Then a link is probably better.
  • 27.

    "a link to where CRUTEM discusses this missing data."

    If this refers to the empty cells in HADCRUT, it may well not be missing data. It's just a cell with no data; there were no stations reporting at the time (maybe ever).

    The reference to Cowtan and Way is not exactly what they did. The problem they drew attention to was that HADCRUT simply leaves out cells with missing data. This has the effect of attributing global (or hemisphere) average data to those cells. But Arctic isn't behaving as average recently. There are many better ways of estimating those missing cells.

    " It looks as if the deterioration was bigger in the years 2000-2009 than in the years 1980-2000. " There is a lot of knowledge to catch up on here. The big data resource is GHCN. This was a project of the '90's, grant funded, which collected all the long temperature records into a database. They put in everything they could find. Sometime after it finished, NOAA decided to systematically update. But they updated only a selected subset, chosen for good coverage etc, and using the CLIMAT monthly reporting system. I have described the transition here.

    Here is a Google Earth visualisation of the stations reporting at various times. Here is a Google Maps app where you can see what stations were reporting over any time period.

    But as I said earlier, it doesn't make much sense to fuss about missing data without trying to estimate what you actually need. When NOAA decided that maintaining 2000 stations was enough, they would have done that calculation. You need to understand it.

    Comment Source:<i>"a link to where CRUTEM discusses this missing data."</i> If this refers to the empty cells in HADCRUT, it may well not be missing data. It's just a cell with no data; there were no stations reporting at the time (maybe ever). The reference to Cowtan and Way is not exactly what they did. The problem they drew attention to was that HADCRUT simply leaves out cells with missing data. This has the effect of attributing global (or hemisphere) average data to those cells. But Arctic isn't behaving as average recently. There are many better ways of estimating those missing cells. <i>" It looks as if the deterioration was bigger in the years 2000-2009 than in the years 1980-2000. "</i> There is a lot of knowledge to catch up on here. The big data resource is GHCN. This was a project of the '90's, grant funded, which collected all the long temperature records into a database. They put in everything they could find. Sometime after it finished, NOAA decided to systematically update. But they updated only a selected subset, chosen for good coverage etc, and using the CLIMAT monthly reporting system. I have described the transition <a href="http://moyhu.blogspot.com.au/2010/08/reduction-of-station-numbers-in-ghcn.html">here</a>. <a href="http://moyhu.blogspot.com.au/2010/08/ge-visualisation-of-changes-to-ghcn.html">Here</a> is a Google Earth visualisation of the stations reporting at various times. <a href="http://www.moyhu.blogspot.com.au/2011/12/google-maps-display-of-ghcn-stations.html">Here</a> is a Google Maps app where you can see what stations were reporting over any time period. But as I said earlier, it doesn't make much sense to fuss about missing data without trying to estimate what you actually need. When NOAA decided that maintaining 2000 stations was enough, they would have done that calculation. You need to understand it.
  • 28.

    Thanks for the references.

    ” It looks as if the deterioration was bigger in the years 2000-2009 than in the years 1980-2000. “ There is a lot of knowledge to catch up on here. The big data resource is GHCN. This was a project of the ’90’s, grant funded, which collected all the long temperature records into a database.

    What is GHCN ? Which database are you talking about?

    Comment Source:Thanks for the references. >” It looks as if the deterioration was bigger in the years 2000-2009 than in the years 1980-2000. “ There is a lot of knowledge to catch up on here. The big data resource is GHCN. This was a project of the ’90’s, grant funded, which collected all the long temperature records into a database. What is GHCN ? Which database are you talking about?
  • 29.
    Comment Source:[Global historical climate network (GHCN)](http://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets/global-historical-climatology-network-ghcn)
  • 30.
    nad
    edited October 2014

    Global historical climate network (GHCN)

    thanks Jim. Nick I will include your references in the blog post. Do you know how different the GHCN and the CRUTEM4 stations datasets are? The article on WUWT was interesting they write:

    All three major gridded global temperature anomaly products rely exclusively or nearly exclusively on the GHCN archive.

    However unfortunately their link to the original paper doens't work. A search on [http://www.uoguelph.ca/~rmckitri/research/papers.html](Ross McKitricks homepage) neither displayed this article but it revealed the following paper, which could be another version:

    McIntyre, Stephen and Ross McKitrick (2011) Discussion of: A Statistical Analysis of Multiple Temperature Proxies: Are Reconstructions of Surface Temperatures Over the Last 1000 Years Reliable? Annals of Applied Statistics Vol. 5, No. 1, 56-6 DOI: 10.1214/10-AOAS398L.

    Unfortunately I couldn't find access to Annals of applied statistics.

    Irritatingly on the http://arxiv.org/pdf/1104.4002.pdf there is an article with the title:

    A STATISTICAL ANALYSIS OF MULTIPLE TEMPERATURE PROXIES: ARE RECONSTRUCTIONS OF SURFACE TEMPERATURES OVER THE LAST 1000 YEARS RELIABLE?

    listing the authors: Blakeley B. McShane and Abraham J. Wyner and carrying the header:

    arXiv:1104.4002v1 [stat.AP] 20 Apr 2011 The Annals of Applied Statistics 2011, Vol. 5, No. 1, 5–44 DOI: 10.1214/10-AOAS398

    copyright Institute of Mathematical Statistics, 2011 Northwestern University and the University of Pennsylvania

    There is no further literature reference in Mc Kitrick's listing for his discussion. Do you have any idea what's going on here? What is this page numbering meaning?

    Comment Source:>Global historical climate network (GHCN) thanks Jim. Nick I will include your references in the blog post. Do you know how different the GHCN and the CRUTEM4 stations datasets are? The article on WUWT was interesting they write: >All three major gridded global temperature anomaly products rely exclusively or nearly exclusively on the GHCN archive. However unfortunately their link to the original paper doens't work. A search on [http://www.uoguelph.ca/~rmckitri/research/papers.html](Ross McKitricks homepage) neither displayed this article but it revealed the following paper, which could be another version: >McIntyre, Stephen and Ross McKitrick (2011) Discussion of: A Statistical Analysis of Multiple Temperature Proxies: Are Reconstructions of Surface Temperatures Over the Last 1000 Years Reliable? Annals of Applied Statistics Vol. 5, No. 1, 56-6 DOI: 10.1214/10-AOAS398L. Unfortunately I couldn't find access to Annals of applied statistics. Irritatingly on the [http://arxiv.org/pdf/1104.4002.pdf](arXiv) there is an article with the title: A STATISTICAL ANALYSIS OF MULTIPLE TEMPERATURE PROXIES: ARE RECONSTRUCTIONS OF SURFACE TEMPERATURES OVER THE LAST 1000 YEARS RELIABLE? listing the authors: Blakeley B. McShane and Abraham J. Wyner and carrying the header: arXiv:1104.4002v1 [stat.AP] 20 Apr 2011 The Annals of Applied Statistics 2011, Vol. 5, No. 1, 5–44 DOI: 10.1214/10-AOAS398 copyright Institute of Mathematical Statistics, 2011 Northwestern University and the University of Pennsylvania There is no further literature reference in Mc Kitrick's listing for his discussion. Do you have any idea what's going on here? What is this page numbering meaning?
  • 31.
    edited October 2014

    A trawl over breakfast reveals that M&M are discussing the McShane and Wyner paper.

    Attempts to use the Blakeley paper to justify the Medieval Warm Period supposed issue are, as usual, dealt with on Real Climate which refers to a big bust up over some other McShane and Wyner paper which might be worth reading as I think they all seem to be about statistically illiterate denunciations of sensor data.

    Apparently there were 13 posts and an editorial on Blakeley but I keep getting sent to projecteuclid.org/pdfview/euclid_aoaos which lists 2 articles but not discussion comments. A rejoinder by Blakely and Wyner to McShane and presumably others is here.

    PS Nad, your link to the version on arXiv is broken.

    Comment Source:A trawl over breakfast reveals that M&M are [discussing](http://projecteuclid.org/download/pdfview_1/euclid.aoas/1300715174) the McShane and Wyner [paper](http://projecteuclid.org/download/pdfview_1/euclid.aoas/1300715170). Attempts to use the Blakeley paper to justify the Medieval Warm Period supposed issue are, as usual, dealt with on [Real Climate](http://www.realclimate.org/index.php/archives/2011/07/arctic-sea-ice-discussions/comment-page-5/) which refers to a big bust up over some other McShane and Wyner [paper](http://www.realclimate.org/index.php/archives/2010/12/responses-to-mcshane-and-wyner/) which might be worth reading as I think they all seem to be about statistically illiterate denunciations of sensor data. Apparently there were 13 posts and an editorial on Blakeley but I keep getting sent to projecteuclid.org/pdfview/euclid_aoaos which lists 2 articles but not discussion comments. A rejoinder by Blakely and Wyner to McShane and presumably others is [here](http://projecteuclid.org/download/pdfview_1/euclid.aoas/1300715184). PS Nad, your link to the version on arXiv is broken.
  • 32.

    A trawl over breakfast reveals that M&M are discussing the McShane and Wyner paper.

    yes thanks. I was already suspecting that something strange with the page numbering was going on.

    Sorry i haven't found the link...it seems my search engine cloud is not as climate/math science adapted as yours.

    Do you know wether this discussion is similar to the review Nick listed?

    Comment Source:>A trawl over breakfast reveals that M&M are discussing the McShane and Wyner paper. yes thanks. I was already suspecting that something strange with the page numbering was going on. Sorry i haven't found the link...it seems my search engine cloud is not as climate/math science adapted as yours. Do you know wether this discussion is similar to the review Nick listed?
  • 33.
    nad
    edited October 2014

    PS Nad, your link to the version on arXiv is broken.

    I am sorry, I can't get acquainted with this markdown language. Here it is: http://arxiv.org/pdf/1104.4002.pdf

    Comment Source:>PS Nad, your link to the version on arXiv is broken. I am sorry, I can't get acquainted with this markdown language. Here it is: [http://arxiv.org/pdf/1104.4002.pdf](http://arxiv.org/pdf/1104.4002.pdf)
  • 34.

    Do you know wether this discussion is similar to the review Nick listed?

    No it doesn't seem so. In particular there are no images in that discussion, like that Figure 1-7: GHCN mean latitude of monitoring stations.

    Comment Source:>Do you know wether this discussion is similar to the review Nick listed? No it doesn't seem so. In particular there are no images in that discussion, like that Figure 1-7: GHCN mean latitude of monitoring stations.
  • 35.

    Attempts to use the Blakeley paper to justify the Medieval Warm Period supposed issue are, as usual, dealt with on Real Climate which refers to a big bust up over some other McShane and Wyner paper which might be worth reading as I think they all seem to be about statistically illiterate denunciations of sensor data.

    haha it is sometimes really funny to read this articles:

    The discussions raise some serious general issues with MW’s work – with respect to how they use the data, the methodologies they introduce (specifically the ‘Lasso’ method), the conclusions they draw, whether there are objective methods to decide whether one method of reconstruction is better than another and whether the Bayesian approach outlined in the last part of the paper is really what it is claimed. - See more at: http://www.realclimate.org/index.php/archives/2010/12/responses-to-mcshane-and-wyner/#sthash.3I8Nz5O0.dpuf

    Lasso as in Lasso?

    Comment Source:>Attempts to use the Blakeley paper to justify the Medieval Warm Period supposed issue are, as usual, dealt with on Real Climate which refers to a big bust up over some other McShane and Wyner paper which might be worth reading as I think they all seem to be about statistically illiterate denunciations of sensor data. haha it is sometimes really funny to read this articles: >The discussions raise some serious general issues with MW’s work – with respect to how they use the data, the methodologies they introduce (specifically the ‘Lasso’ method), the conclusions they draw, whether there are objective methods to decide whether one method of reconstruction is better than another and whether the Bayesian approach outlined in the last part of the paper is really what it is claimed. - See more at: http://www.realclimate.org/index.php/archives/2010/12/responses-to-mcshane-and-wyner/#sthash.3I8Nz5O0.dpuf Lasso as in [Lasso](http://www.spiegel.de/fotostrecke/rodeo-auf-mexikanisch-locker-mit-lasso-fotostrecke-120151.html)?
  • 36.
    edited October 2014

    Real Climate is really entertaining. It deals in (no uncertain) terms with eristic sciolists.

    I didn't click your link when I saw it contained rodeo fotostrecke!

    Lasso is a statistical methodology I want to learn about. There is a discussion about whether it is being used correctly by McShane and Wyner or possibly about whether it can be. GIYF. :).

    Comment Source:Real Climate is really entertaining. It deals in (no uncertain) terms with eristic sciolists. I didn't click your link when I saw it contained rodeo fotostrecke! Lasso is a statistical methodology I want to learn about. There is a discussion about whether it is being used correctly by McShane and Wyner or possibly about whether it can be. GIYF. :).
  • 37.

    nad, Here are some KMZ files which show the stations. You can download the files, maybe unzip, click on them and they will pop up in Google Earth. You can restrict to sets of decades using folders. There is a post here describing the combined version.

    Here is CRUTEM 3, and here GHCN V3. Each is 300-400 Kb. There is a version here, described in the link above, which shows GHCN, CRUTEM, GSOD and BEST all on one KMZ. It's about 2.7Mb, and a bit cluttered, but has everything.

    Comment Source:nad, Here are some KMZ files which show the stations. You can download the files, maybe unzip, click on them and they will pop up in Google Earth. You can restrict to sets of decades using folders. There is a post <a href="http://moyhu.blogspot.com.au/2011/10/combined-kmz-file-for-best-ghcn-gsod.html">here</a> describing the combined version. Here is <a href="https://sites.google.com/site/moyhudocs/documents/crutem_kml.zip">CRUTEM 3</a>, and here <a href="https://sites.google.com/site/moyhudocs/documents/GHCN3_kml.zip">GHCN V3</a>. Each is 300-400 Kb. There is a version <a href="https://sites.google.com/site/moyhudocs/documents/ALL4.kmz">here</a>, described in the link above, which shows GHCN, CRUTEM, GSOD and BEST all on one KMZ. It's about 2.7Mb, and a bit cluttered, but has everything.
  • 38.

    I didn’t click your link when I saw it contained rodeo fotostrecke!

    Oh Jim - it is no dangerous link! It is just a travel recommendation in the well-known german online magazine "Der Spiegel" (the mirror).

    Comment Source:>I didn’t click your link when I saw it contained rodeo fotostrecke! Oh Jim - it is no dangerous link! It is just a travel recommendation in the well-known german online magazine "Der Spiegel" (the mirror).
  • 39.
    nad
    edited October 2014

    nad, Here are some KMZ files which show the stations. You can download the files, maybe unzip, click on them and they will pop up in Google Earth. You can restrict to sets of decades using folders. There is a post here describing the combined version.

    thanks I might try this. can you provide me with a direct link to all the combined station data so that I can directly see it in google earth via onclick?

    I clicked on visualising stations in:

    KMLfiles.zip KML files for visualising stations in Google Earth

    at this link and just got a blog post.

    Comment Source:>nad, Here are some KMZ files which show the stations. You can download the files, maybe unzip, click on them and they will pop up in Google Earth. You can restrict to sets of decades using folders. There is a post here describing the combined version. thanks I might try this. can you provide me with a direct link to all the combined station data so that I can directly see it in google earth via onclick? I clicked on visualising stations in: >KMLfiles.zip KML files for visualising stations in Google Earth at this <a href="https://sites.google.com/site/moyhudocs/documents">link</a> and just got a blog post.
  • 40.
    edited October 2014

    it is no dangerous link!

    I know, but I've got a bad back and can't ride a horse! No, I saw the link and guessed it wasn't a photo of a statistical lasso (although I think the horse-swaggling lasso might be where it derives from; otherwise it's somebody's name) :).

    Comment Source:> it is no dangerous link! I know, but I've got a bad back and can't ride a horse! No, I saw the link and guessed it wasn't a photo of a statistical lasso (although I think the horse-swaggling lasso might be where it derives from; otherwise it's somebody's name) :).
  • 41.

    Incidentally, I co-authored a comment on McShane and Wyner, although we didn't get into as much detail on the methods as some other discussions. I don't think it was a truly bad analysis, but it could have been improved in a number of ways.

    Comment Source:Incidentally, I co-authored [a comment](http://public.lanl.gov/nurban/pubs/millennial-temperature-reconstruction-discussion.pdf) on McShane and Wyner, although we didn't get into as much detail on the methods as some other discussions. I don't think it was a truly bad analysis, but it could have been improved in a number of ways.
  • 42.

    nad, "at this link and just got a blog post"

    It has a table of links with descriptions. ALL4.zip, for example, has the the inventory of all stations, which I think is what you want. The various KMZ and KML file links are there.

    Comment Source:nad, <i>"at this link and just got a blog post"</i> It has a table of links with descriptions. ALL4.zip, for example, has the the inventory of all stations, which I think is what you want. The various KMZ and KML file links are there.
  • 43.

    Sorry Nick - I still can't find the file. On the list there is no ALL4.zip

    and the blog post contains a sentence "The files are on the docs repository." But docs repository is a broken link

    Comment Source:Sorry Nick - I still can't find the file. On the [list](https://sites.google.com/site/moyhudocs/documents) there is no ALL4.zip and the blog post contains a sentence "The files are on the docs repository." But docs repository is a [broken link](http://drop.io/erfitad)
  • 44.

    My mistake - it's ALL4inv.zip. Top of the list.

    Comment Source:My mistake - it's ALL4inv.zip. Top of the list.
  • 45.
    nad
    edited October 2014

    My mistake - it’s ALL4inv.zip. Top of the list.

    No I wrote: can you provide me with a direct link to all the combined station data so that I can directly see it in google earth via onclick?

    I wanted first to see the file ALL4.kmz which I have now looked at. If I understand correctly then BEST contains more or less all the other data files, which are GHCN, GSOD and CRUTEM3 In particular GSOD looks at a first glance as if it would be different but if one looks into sparse regions then one sees that GSOD and BEST are if they are both present ALMOST overlapping, which I guess could be due to that GSOD eventually does/n't use decimals behind the comma.

    Nick so you basically did the same as me and Tim did with CRUTEM 4 only. Where I guess CRUTEM 4 is also included in BEST, since it has only 4634 stations and BEST has 36736 stations, as you wrote here. The only difference (apart from from the overall look) is that in our application the temperature is visible and that the station description shows the numerical value of the temperature and latitude and longitude, which is not so important for understanding that the temperature data is rotten. The deterioration of the BEST collection is also quite visible, if one clicks through your folders.

    I wish you would have been earlier here in this forum because I asked around which temperature collection is most exhaustive and wether there are visualizations which display how good the temperatures are and I didn't get BEST as an answer, neither did I get your kmz visualizations as an answer, where to be fair I might have also overlooked something in this messy forum.

    If you would have been here that would have saved me from programming at the application, which was a lot of partially very nerving, mostly in vain work for no money and with strenous interactions with the company Klokan. Anyways the only positive thing about my (and Tims) application is that one is allowed to use the screenshots (the maps (...if they appear...) are openstreetmap. I don't know what happens if one shows screenshots of google earth. It is good that one has the screenshots, because I think most people will take (if at all) only a quick glance over the images and not even bother to go to the javascript webpage or download googleearth kmz files.

    Anyways I guess it should be clear why I am -at least as far as I can tell now - not interested in the csv files.

    Comment Source:>My mistake - it’s ALL4inv.zip. Top of the list. No I wrote: can you provide me with a direct link to all the combined station data so that I can directly see it in google earth via onclick? I wanted first to see the file [ALL4.kmz](https://sites.google.com/site/moyhudocs/documents/ALL4.kmz?attredirects=0&d=1) which I have now looked at. If I understand correctly then BEST contains more or less all the other data files, which are GHCN, GSOD and CRUTEM3 In particular GSOD looks at a first glance as if it would be different but if one looks into sparse regions then one sees that GSOD and BEST are if they are both present ALMOST overlapping, which I guess could be due to that GSOD eventually does/n't use decimals behind the comma. Nick so you basically did the same as me and Tim did with CRUTEM 4 only. Where I guess CRUTEM 4 is also included in BEST, since it has only 4634 stations and BEST has 36736 stations, as you wrote [here](http://moyhu.blogspot.de/2011/10/kmz-file-for-best-stations.html). The only difference (apart from from the overall look) is that in our application the temperature is visible and that the station description shows the numerical value of the temperature and latitude and longitude, which is not so important for understanding that the temperature data is rotten. The deterioration of the BEST collection is also quite visible, if one clicks through your folders. I wish you would have been earlier here in this forum because I asked around which temperature collection is most exhaustive and wether there are visualizations which display how good the temperatures are and I didn't get BEST as an answer, neither did I get your kmz visualizations as an answer, where to be fair I might have also overlooked something in this messy forum. If you would have been here that would have saved me from programming at the application, which was a lot of partially very nerving, mostly in vain work for no money and with strenous interactions with the company Klokan. Anyways the only positive thing about my (and Tims) application is that one is allowed to use the screenshots (the maps (...if they appear...) are openstreetmap. I don't know what happens if one shows screenshots of google earth. It is good that one has the screenshots, because I think most people will take (if at all) only a quick glance over the images and not even bother to go to the javascript webpage or download googleearth kmz files. Anyways I guess it should be clear why I am -at least as far as I can tell now - not interested in the csv files.
  • 46.
    edited October 2014

    nad,

    "can you provide me with a direct link to all the combined station data so that I can directly see it in google earth via onclick?"

    The data files are huge. BEST is 100's Mb. I doubt if anyone has tried to combine them. They weren't used for the KMZ files of station location.

    As far as displaying the station data - I use the shaded style here. You can click a button to show the stations, and you can shift-click near a station to show its value for that month. The colour corresponds to temp at each actual station, and shades between.

    Comment Source:nad, <i>"can you provide me with a direct link to all the combined station data so that I can directly see it in google earth via onclick?"</i> The data files are huge. BEST is 100's Mb. I doubt if anyone has tried to combine them. They weren't used for the KMZ files of station location. As far as displaying the station data - I use the shaded style <a href="http://www.moyhu.blogspot.com.au/p/blog-page_24.html">here</a>. You can click a button to show the stations, and you can shift-click near a station to show its value for that month. The colour corresponds to temp at each actual station, and shades between.
  • 47.
    nad
    edited October 2014

    As far as displaying the station data - I use the shaded style here. You can click a button to show the stations, and you can shift-click near a station to show its value for that month. The colour corresponds to temp at each actual station, and shades between.

    Yes but in this visualizaiton you don't see the indvidual stations anymore and this is important for seeing how rotten the data is.

    The data files are huge. BEST is 100’s Mb. I doubt if anyone has tried to combine them. They weren’t used for the KMZ files of station location.

    ?????

    So which original data file did you use for the kmz's? - could you give me the original data location for those data files, at least this which had been used for the BEST.kmz data?

    That is as far as I had understood those TAVG files at BESTdata might eventually provide the original station data for BEST. Is this the data you used for BEST.kmz? Which one ? the quality controlled?

    I doubt if anyone has tried to combine them.

    I am still investigating this.

    Actually you are right it seems at least not for all years.

    Comment Source:As far as displaying the station data - I use the shaded style here. You can click a button to show the stations, and you can shift-click near a station to show its value for that month. The colour corresponds to temp at each actual station, and shades between. Yes but in this visualizaiton you don't see the indvidual stations anymore and this is important for seeing how rotten the data is. >The data files are huge. BEST is 100’s Mb. I doubt if anyone has tried to combine them. They weren’t used for the KMZ files of station location. ????? So which original data file did you use for the kmz's? - could you give me the original data location for those data files, at least this which had been used for the BEST.kmz data? That is as far as I had understood those TAVG files at [BESTdata](http://berkeleyearth.org/data) might eventually provide the original station data for BEST. Is this the data you used for BEST.kmz? Which one ? the quality controlled? >I doubt if anyone has tried to combine them. I am still investigating this. Actually you are right it seems at least not for all years.
  • 48.

    By the way - Nick did you change something at the kmz files within the last tow days and today?

    That is I am pretty sure that I clicked yesterday on some stations (however not on all of course) and no stations data window appeared, now there seem to be one at least for some stations.

    According to the google earth visualization GSOD has a few more stations in Russia than BEST in the last ten years, like SALEKHARD at 66.58, 66.6 or PERM 58.2,56.3 moreover most of the in GSOD and BEST contained stations are not contained in CRUTEM3 and GHCN.

    CRUTEM 4 is however better with respect to this.

    Comment Source:By the way - Nick did you change something at the kmz files within the last tow days and today? That is I am pretty sure that I clicked yesterday on some stations (however not on all of course) and no stations data window appeared, now there seem to be one at least for some stations. According to the google earth visualization GSOD has a few more stations in Russia than BEST in the last ten years, like SALEKHARD at 66.58, 66.6 or PERM 58.2,56.3 moreover most of the in GSOD and BEST contained stations are not contained in CRUTEM3 and GHCN. CRUTEM 4 is however better with respect to this.
  • 49.
    nad
    edited October 2014

    By the way - Nick did you change something at the kmz files within the last tow days and today?

    I should also add that I had the impression that yesterday the BEST collection displayed way more data in Russia in the last ten years. Currently there is almost nothing. But yesterday I browsed through the collection rather fast, so my memory may betray me here.

    Nick please see also this comment

    Comment Source:>By the way - Nick did you change something at the kmz files within the last tow days and today? I should also add that I had the impression that yesterday the BEST collection displayed way more data in Russia in the last ten years. Currently there is almost nothing. But yesterday I browsed through the collection rather fast, so my memory may betray me here. Nick please see also this <a href="http://forum.azimuthproject.org/discussion/1501/how-good-is-climate-science-temperature-data/?Focus=13014#Comment_13014">comment</a>
  • 50.
    edited February 2015

    For folks who haven't seen the brilliant rotatable, animated 3D model from earth.nullschool.net I've scraped a video of ocean currents and winds and put it on you tube. It needs to be watched full screen.

    Comment Source:For folks who haven't seen the brilliant rotatable, animated 3D model from [earth.nullschool.net](http://earth.nullschool].net/) I've scraped a video of ocean currents and winds and put it on [you tube](https://www.youtube.com/edit?o=U&video_id=Mmbhu3iENPY). It needs to be watched full screen.
Sign In or Register to comment.