Options

R programming language

I've updated the R page.

I couldn't find a previous discussion of this page, though it may exist. 'R' is not a good search term.

«1

Comments

  • 1.

    Thanks! I may start asking you how to write some very simple R programs.

    Comment Source:Thanks! I may start asking you how to write some very simple R programs.
  • 2.

    I'm going to download R from here:

    Comment Source:I'm going to download R from here: * [Download R 3.1.0 for Windows](http://cran.r-project.org/bin/windows/base/) at one of the [Comprehensive R Archive Network mirrors](http://cran.rstudio.com/).
  • 3.

    Great. It'll be fun watching you struggle! I think - for what imagine is your intended use - it is similar to learning Latex. So there is:

    • a syntax, which is irritating and frustating until you get used to it, and results in cryptic error messages when you get it wrong.

    • an ever-expanding vocabulary of key words, but you can have fun with a few.

    • ways of working with the grain rather than against it (and knowing when R is not the best tool for the job).

    I recommend you also install R studio. It will help with the syntax.

    Some things to try:

    2^(1:8)
    
    exp(pi * 1i)
    
    plot(rnorm(20))
    
    Comment Source:Great. It'll be fun watching you struggle! I think - for what imagine is your intended use - it is similar to learning Latex. So there is: * a syntax, which is irritating and frustating until you get used to it, and results in cryptic error messages when you get it wrong. * an ever-expanding vocabulary of key words, but you can have fun with a few. * ways of working with the grain rather than against it (and knowing when R is not the best tool for the job). I recommend you also install [R studio](http://www.rstudio.com/). It will help with the syntax. Some things to try: ~~~~ 2^(1:8) ~~~~ ~~~~ exp(pi * 1i) ~~~~ ~~~~ plot(rnorm(20)) ~~~~
  • 4.
    edited June 2014

    It’ll be fun watching you struggle!

    With learning to program, or with installing the software?

    • a syntax, which is irritating and frustating until you get used to it, and results in cryptic error messages when you get it wrong.

    Ah yes! Sounds familiar! I'm by now quite an expert at LaTeX, so the more recent example of this sort of frustration is my attempt to learn TikZ, the fancy drawing package in LaTeX, by taking examples and trying to modify them. Syntax is always cryptic if you try to learn it by looking at examples instead of mastering it from the ground up. But that's how I like to get started learning stuff.

    Thanks for the easy examples. After I overcome the basic hurdles I will try to take some programs you wrote, run them, then tinker with them. For example, if you computed something in the El Niño basin I should be able to do something analogous elsewhere in the world.... if I can figure out how to download the data, which still seems scary.

    I hope that some of my struggles will make for educational or at least entertaining blog articles in the El Niño Project series.

    Comment Source:> It’ll be fun watching you struggle! With learning to program, or with installing the software? <img src = "http://math.ucr.edu/home/baez/emoticons/sm_rolleyes.gif" alt = ""/> > * a syntax, which is irritating and frustating until you get used to it, and results in cryptic error messages when you get it wrong. Ah yes! Sounds familiar! I'm by now quite an expert at LaTeX, so the more recent example of this sort of frustration is my attempt to learn TikZ, the fancy drawing package in LaTeX, by taking examples and trying to modify them. Syntax is always cryptic if you try to learn it by looking at examples instead of mastering it from the ground up. But that's how I like to get started learning stuff. Thanks for the easy examples. After I overcome the basic hurdles I will try to take some programs you wrote, run them, then tinker with them. For example, if you computed something in the El Ni&ntilde;o basin I should be able to do something analogous elsewhere in the world.... if I can figure out how to download the data, which still seems scary. I hope that some of my struggles will make for educational or at least entertaining blog articles in the El Ni&ntilde;o Project series.
  • 5.

    Okay, installing R on a Windows machine was a snap, and it was fun to evaluate the expressions Graham suggested, and also

    mean(2^(1:8))

    Now I need to figure out how to input data.

    Comment Source:Okay, installing R on a Windows machine was a snap, and it was fun to evaluate the expressions Graham suggested, and also `mean(2^(1:8))` Now I need to figure out how to input data.
  • 6.

    I'll write some comments that may get folded into a blog article about "R for dummies".

    To download R to my Windows PC I cleverly typed "download R" into Google and went to the top website it recommended:

    http://cran.r-project.org/bin/windows/base/

    I clicked the big fat thing on top saying

    Download R 3.1.0 for Windows

    and got asked to save a file R-3.1.0-win.exe. I saved it in my Downloads folder - it took a while to download since it was 57 megabytes. I clicked on it there and followed the easy default installation instructions. I now have a little icon that says R on my screen.

    Clicking this I get a little interface where I can type commands after a red

    >

    symbol. I started by trying

    > 2^(1:8)

    which generated a list of powers of 2 from $2^1$ to $2^8$, and

    > mean(2^(1:8))

    which gave the arithmetic mean of this list. Somewhat more fun was

    > plot(rnorm(20))

    which plotted a bunch of points, apparently 20 random numbers chosen from a Gaussian distribution of mean zero and variance 1. (I'm just guessing.)

    Comment Source:I'll write some comments that may get folded into a blog article about "R for dummies". To download R to my Windows PC I cleverly typed "download R" into Google and went to the top website it recommended: [http://cran.r-project.org/bin/windows/base/](http://cran.r-project.org/bin/windows/base/) I clicked the big fat thing on top saying > Download R 3.1.0 for Windows and got asked to save a file R-3.1.0-win.exe. I saved it in my Downloads folder - it took a while to download since it was 57 megabytes. I clicked on it there and followed the easy default installation instructions. I now have a little icon that says R on my screen. Clicking this I get a little interface where I can type commands after a red `>` symbol. I started by trying `> 2^(1:8)` which generated a list of powers of 2 from $2^1$ to $2^8$, and `> mean(2^(1:8))` which gave the arithmetic mean of this list. Somewhat more fun was `> plot(rnorm(20))` which plotted a bunch of points, apparently 20 random numbers chosen from a Gaussian distribution of mean zero and variance 1. (I'm just guessing.)
  • 7.
    edited June 2014

    To do something more interesting, I want to input data.

    The papers by Ludescher et al use surface air temperatures in a patch of the Pacific, so I want to get ahold of those. They're here:

    More precisely, there's a bunch of files here containing worldwide daily average temperatures on a 2.5 degree latitude × 2.5 degree longitude grid (144 × 73 grid points), from 1948 to 2010. If you go here the website will help you get data from within a chosen rectangle in a grid, for a chosen time interval.

    These are NetCDF files, where NetCDF stands for Network Common Data Form:

    NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

    Graham Jones said that R has a "package" called RNetCDF for using NetCDF files. So, I need to get ahold of this package, figure out how you use "packages" in R, and then I guess download some files in the CDF format and somehow get R to eat those files, with the help of this package.

    I'm utterly clueless...

    Comment Source:To do something more interesting, I want to input _data_. The papers by Ludescher _et al_ use surface air temperatures in a patch of the Pacific, so I want to get ahold of those. They're here: * [NCEP/NCAR Reanalysis 1: Surface](http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.surface.html). More precisely, there's a bunch of files [here](http://www.esrl.noaa.gov/psd/cgi-bin/db_search/DBSearch.pl?Dataset=NCEP+Reanalysis+Daily+Averages+Surface+Level&Variable=Air+Temperature&group=0&submit=Search) containing worldwide daily average temperatures on a 2.5 degree latitude &times; 2.5 degree longitude grid (144 &times; 73 grid points), from 1948 to 2010. If you go [here](http://www.esrl.noaa.gov/psd/cgi-bin/DataAccess.pl?DB_dataset=NCEP+Reanalysis+Daily+Averages+Surface+Level&DB_variable=Air+temperature&DB_statistic=Mean&DB_tid=41710&DB_did=33&DB_vid=668) the website will help you get data from within a chosen rectangle in a grid, for a chosen time interval. These are NetCDF files, where NetCDF stands for [Network Common Data Form](http://www.unidata.ucar.edu/software/netcdf/): > NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. Graham Jones said that R has a "package" called <a href = "http://cran.r-project.org/web/packages/RNetCDF/index.html">RNetCDF</a> for using NetCDF files. So, I need to get ahold of this package, figure out how you use "packages" in R, and then I guess download some files in the CDF format and somehow get R to eat those files, with the help of this package. I'm utterly clueless...
  • 8.
    edited June 2014

    After a bit of messing around, I notice that right on top of the R interface there's a menu item called "Packages". I boldly click on this and choose "Install Package(s)".

    I'm rewarded with a truly huge alphabetically ordered list of packages... obviously statisticians have lots of stuff they like to do over and over. I find "RNetCDF", click on that and click something like "OK".

    I'm asked if I want to use a "personal library". After some reading I click "no", and get an error message:

    Warning in install.packages(NULL, .libPaths()[1L], dependencies = NA, type = type) : 'lib = "C:/Program Files/R/R-3.1.0/library"' is not writable Error in install.packages(NULL, .libPaths()[1L], dependencies = NA, type = type) : unable to install packages

    I cleverly read something Graham wrote, and try using the command line interface:

    > install.packages("RNetCDF")

    Nope:

    Warning in install.packages("RNetCDF") : 'lib = "C:/Program Files/R/R-3.1.0/library"' is not writable Error in install.packages("RNetCDF") : unable to install packages

    So, I try the other method again and say "yes" when asked if I want to use a "personal library".

    Yes! The computer barfs out some promising text:

    utils:::menuInstallPkgs() trying URL 'http://cran.stat.nus.edu.sg/bin/windows/contrib/3.1/RNetCDF_1.6.2-3.zip' Content type 'application/zip' length 548584 bytes (535 Kb) opened URL downloaded 535 Kb

    package ‘RNetCDF’ successfully unpacked and MD5 sums checked

    The downloaded binary packages are in C:\Users\JOHN\AppData\Local\Temp\Rtmp4qJ2h8\downloaded_packages

    But now I need to figure out how to download a file from the NCAR website and get R to eat it and digest it.

    Comment Source:After a bit of messing around, I notice that right on top of the R interface there's a menu item called "Packages". I boldly click on this and choose "Install Package(s)". I'm rewarded with a truly huge alphabetically ordered list of packages... obviously statisticians have lots of stuff they like to do over and over. I find "RNetCDF", click on that and click something like "OK". I'm asked if I want to use a "personal library". After some reading I click "no", and get an error message: `Warning in install.packages(NULL, .libPaths()[1L], dependencies = NA, type = type) : 'lib = "C:/Program Files/R/R-3.1.0/library"' is not writable Error in install.packages(NULL, .libPaths()[1L], dependencies = NA, type = type) : unable to install packages` I cleverly read something Graham wrote, and try using the command line interface: `> install.packages("RNetCDF")` Nope: `Warning in install.packages("RNetCDF") : 'lib = "C:/Program Files/R/R-3.1.0/library"' is not writable Error in install.packages("RNetCDF") : unable to install packages` So, I try the other method again and say "yes" when asked if I want to use a "personal library". Yes! The computer barfs out some promising text: `utils:::menuInstallPkgs() trying URL 'http://cran.stat.nus.edu.sg/bin/windows/contrib/3.1/RNetCDF_1.6.2-3.zip' Content type 'application/zip' length 548584 bytes (535 Kb) opened URL downloaded 535 Kb` `package ‘RNetCDF’ successfully unpacked and MD5 sums checked` `The downloaded binary packages are in C:\Users\JOHN\AppData\Local\Temp\Rtmp4qJ2h8\downloaded_packages` But now I need to figure out how to download a file from the NCAR website and get R to eat it and digest it.
  • 9.

    But now I need to figure out how to download a file from the NCAR website and get R to eat it and digest it.

    You can download the files from your browser. It is probably easiest to do that for starters. Put ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/ into the browser, then right-click a file and Save link as...

    This code wil download a bunch of them:

    for (year in 1950:1979) { 
      download.file(url=paste0("ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/air.sig995.", year, ".nc"), destfile=paste0("air.sig995.", year, ".nc"), mode="wb") 
      }
    

    It will put them into the "working directory", probably C:\Users\JOHN\Documents. You can find the working directory using getwd(), and change it with setwd(). But you must use / not \ in the filepath.

    and get R to eat it and digest it.

    The R script netcdf-convertor.R from https://github.com/azimuth-project/el-nino/tree/master/R will eat it, digest it, and spit it out again (but not make any maps). It contains instructions.

    Comment Source:> But now I need to figure out how to download a file from the NCAR website and get R to eat it and digest it. You can download the files from your browser. It is probably easiest to do that for starters. Put `ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/` into the browser, then right-click a file and Save link as... This code wil download a bunch of them: ~~~~ for (year in 1950:1979) { download.file(url=paste0("ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/air.sig995.", year, ".nc"), destfile=paste0("air.sig995.", year, ".nc"), mode="wb") } ~~~~ It will put them into the "working directory", probably C:\Users\JOHN\Documents. You can find the working directory using `getwd()`, and change it with `setwd()`. But you must use / not \ in the filepath. > and get R to eat it and digest it. The R script `netcdf-convertor.R` from `https://github.com/azimuth-project/el-nino/tree/master/R` will eat it, digest it, and spit it out again (but not make any maps). It contains instructions.
  • 10.
    edited June 2014

    Graham wrote:

    You can find the working directory using getwd(), and change it with setwd(). But you must use / not \ in the filepath.

    Hmm... here's what happens:

    > getwd()

    [1] "C:/Users/JOHN/Documents"

    Okay, just what you said. Then:

    > setwd(C:\Users\JOHN\Documents\My Backups\azimuth\el nino)

    Error: unexpected input in "setwd(C:\"

    My mistake: I used backslashes. Then:

    > setwd(C:/Users/JOHN/Documents/My Backups/azimuth/el nino)

    Error: unexpected '/' in "setwd(C:/"

    Unexpected forwards slash??? Then:

    > setwd(C:/Users/JOHN/Documents)

    Error: unexpected '/' in "setwd(C:/"

    So even if I try to set the working directory to what R says it already is, it complains! And if it doesn't like that /, it also doesn't like not seeing it:

    setwd(C:Users/John/Documents)

    Error in setwd(C:Users/John/Documents) : object 'Users' not found

    Comment Source:Graham wrote: > You can find the working directory using getwd(), and change it with setwd(). But you must use / not \ in the filepath. Hmm... here's what happens: `> getwd()` `[1] "C:/Users/JOHN/Documents"` Okay, just what you said. Then: `> setwd(C:\Users\JOHN\Documents\My Backups\azimuth\el nino)` `Error: unexpected input in "setwd(C:\"` My mistake: I used backslashes. Then: `> setwd(C:/Users/JOHN/Documents/My Backups/azimuth/el nino)` `Error: unexpected '/' in "setwd(C:/"` Unexpected <i>forwards</i> slash??? Then: `> setwd(C:/Users/JOHN/Documents)` `Error: unexpected '/' in "setwd(C:/"` So even if I try to set the working directory to what R says it already is, it complains! And if it doesn't like that /, it also doesn't like <i>not</i> seeing it: `setwd(C:Users/John/Documents)` `Error in setwd(C:Users/John/Documents) : object 'Users' not found`
  • 11.
    edited June 2014

    Then I look at a file of Graham's and cleverly realize quotes are needed:

    > setwd("C:Users/John/Documents")

    I'm rewarded with:

    Error in setwd("C:Users/John/Documents") : cannot change working directory

    which seems like progress... of a sort.

    It can't change the working directory to what it already is? Is it just peeved that I'm doing something so silly? I'll boldly try to change the directory to what I actually want:

    > setwd("C:/Users/JOHN/Documents/My Backups/azimuth/el nino")

    I'm now granted this laconic response:

    >

    Eloquent in its complaints, it's darn tight-lipped when it comes to providing positive feedback. However:

    > getwd()

    [1] "C:/Users/JOHN/Documents/My Backups/azimuth/el nino"

    Yay!

    I could have proved a fairly interesting theorem in this amount of time, but I guess I'm learning something. I've never done any programming that involves manipulating files that are stored in directories...

    Comment Source:Then I look at a file of Graham's and cleverly realize quotes are needed: `> setwd("C:Users/John/Documents")` I'm rewarded with: `Error in setwd("C:Users/John/Documents") : cannot change working directory` which seems like progress... of a sort. It can't change the working directory to what it already is? Is it just peeved that I'm doing something so silly? I'll boldly try to change the directory to what I actually want: `> setwd("C:/Users/JOHN/Documents/My Backups/azimuth/el nino")` I'm now granted this laconic response: `>` Eloquent in its complaints, it's darn tight-lipped when it comes to providing positive feedback. However: `> getwd()` `[1] "C:/Users/JOHN/Documents/My Backups/azimuth/el nino"` Yay! <img src = "http://math.ucr.edu/home/baez/emoticons/celebrating.gif" alt = ""/> I could have proved a fairly interesting theorem in this amount of time, but I guess I'm learning something. I've never done any programming that involves manipulating files that are stored in directories...
  • 12.
    edited June 2014

    Next: I decided to download wads of surface air temperature data from NCAR. Following Graham's advice, I entered this into the R interface:

    for (year in 1950:1979) { 
      download.file(url=paste0("ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/air.sig995.", year, ".nc"), destfile=paste0("air.sig995.", year, ".nc"), mode="wb") 
      }
    

    It seems to be working! It's chugging away, taking about a minute for each year's worth of data.

    Comment Source:Next: I decided to download wads of surface air temperature data from NCAR. Following Graham's advice, I entered this into the R interface: ~~~~ for (year in 1950:1979) { download.file(url=paste0("ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/air.sig995.", year, ".nc"), destfile=paste0("air.sig995.", year, ".nc"), mode="wb") } ~~~~ It seems to be working! It's chugging away, taking about a minute for each year's worth of data.
  • 13.

    Okay, now I've got all the worldwide daily average temperatures on a 2.5 degree latitude × 2.5 degree longitude grid (144 × 73 grid points), from 1950 to 1970. Bwahaha, the world is MINE!

    Now what do I do with it? Graham wrote:

    The R script netcdf-convertor.R from https://github.com/azimuth-project/el-nino/tree/master/R will eat it, digest it, and spit it out again (but not make any maps). It contains instructions.

    The instructions are admirably detailed concerning what I should do (I'm not being sarcastic, I'm serious!) but they say nothing about what will happen when I do it. This may be typical of software documentation? Anyway, I guess I should just try it.

    Comment Source:Okay, now I've got all the worldwide daily average temperatures on a 2.5 degree latitude × 2.5 degree longitude grid (144 × 73 grid points), from 1950 to 1970. _Bwahaha, the world is MINE!_ Now what do I do with it? Graham wrote: > The R script netcdf-convertor.R from [https://github.com/azimuth-project/el-nino/tree/master/R](https://github.com/azimuth-project/el-nino/tree/master/R) will eat it, digest it, and spit it out again (but not make any maps). It contains instructions. The instructions are admirably detailed concerning what I should do (I'm not being sarcastic, I'm serious!) but they say nothing about what will happen when I do it. This may be typical of software documentation? Anyway, I guess I should just try it.
  • 14.
    edited June 2014

    Unfortunately, I also don't know how you take a "script" and enter it in R. I want to edit it a bit beforehand, too.

    I guess I'll try to download it.

    Embarrassingly, I don't even know how to download something from GitHub, where this script is stored. I could cut-and-paste it, but that can't be how you're supposed to do it. There's no button that says "download" - that would be too easy, I guess.

    So, I click on the next best thing, "Open". When I mouse over that button it says "Open this file in Github for Windows". When I click on it, I get something that says Using GitHub on Windows has never been this easy. I'm glad I didn't try it sooner, then.

    It suggests that I "Download GitHub for Windows 2.0". That seems potentially useful, so I go ahead and download something called GitHubSetup.exe into my Downloads folder, then click on that. It asks me if I want to install it, and I say yes.

    Comment Source:Unfortunately, I also don't know how you take a "script" and enter it in R. I want to edit it a bit beforehand, too. I guess I'll try to download it. Embarrassingly, I don't even know how to download something from GitHub, where this script is stored. I could cut-and-paste it, but that can't be how you're supposed to do it. There's no button that says "download" - that would be too easy, I guess. So, I click on the next best thing, "Open". When I mouse over that button it says "Open this file in Github for Windows". When I click on it, I get something that says *Using GitHub on Windows has never been this easy*. I'm glad I didn't try it sooner, then. It suggests that I "Download GitHub for Windows 2.0". That seems potentially useful, so I go ahead and download something called GitHubSetup.exe into my Downloads folder, then click on that. It asks me if I want to install it, and I say yes.
  • 15.
    edited June 2014

    While I'm waiting for that to do its thing, I see a button I'd overlooked, called "Raw". I suddenly realized this is a way to download the file in "raw", uncooked form. I click on it and yes - I get the file netcdf-convertor.R. I save it in my "working directory" for R, namely C:\Users\JOHN\Documents\My Backups\azimuth\el nino.

    Why didn't I think of that sooner? Obviously if you want to download a file you should click on a button that says "Raw".

    Now the challenge is to feed this "raw" script into R's mouth somehow.

    Comment Source:While I'm waiting for that to do its thing, I see a button I'd overlooked, called "Raw". I suddenly realized this is a way to download the file in "raw", uncooked form. I click on it and _yes_ - I get the file `netcdf-convertor.R`. I save it in my "working directory" for R, namely `C:\Users\JOHN\Documents\My Backups\azimuth\el nino`. Why didn't I think of that sooner? _Obviously_ if you want to download a file you should click on a button that says "Raw". Now the challenge is to feed this "raw" script into R's mouth somehow.
  • 16.
    edited June 2014

    The left-hand menu item is always the one that has the most important stuff, so on the R interface I click that one, which happens to be called "File" for some reason... and get a dropdown menu containing promising candidates such as "New Script" and "Open Script".

    Clicking on the latter, I see the script I just downloaded to my working directory... and, presumably, all the scripts (.R files) in that directory... but so far I only have one.

    I realize now that I wanted to edit this script a little before testing it out.

    Comment Source:The left-hand menu item is always the one that has the most important stuff, so on the R interface I click that one, which happens to be called "File" for some reason... and get a dropdown menu containing promising candidates such as "New Script" and "Open Script". Clicking on the latter, I see the script I just downloaded to my working directory... and, presumably, _all_ the scripts (.R files) in that directory... but so far I only have one. I realize now that I wanted to edit this script a little before testing it out.
  • 17.

    Graham has helpfully put this commented-out stuff at the top of the script:

    ###############################################################################
    ###############################################################################
    
    # You should be able to use this by editing this section only.
    
    setwd("C:/Users/Work/AAA/Programming/ProgramOutput/Nino")
    
    
    lat.range <- 13:14
    lon.range <- 142:143
    
    firstyear <- 1957
    lastyear <- 1958
    
    outputfilename <- paste0("Scotland-", firstyear, "-", lastyear, ".txt")
    
    
    ###############################################################################
    ###############################################################################
    
    #                  Explanation
    
    
    # 1. Use setwd() to set the working directory to the one 
    # containing the .nc files such as air.sig995.1951.nc.
    # Example:
    # setwd("C:/Users/Work/AAA/Programming/ProgramOutput/Nino")
    
    # 2. Supply the latitude and longitude range.  The NOAA data is 
    # every 2.5 degrees. The ranges are supplied as the number of steps of this 
    # size. For latitude, 1 means North Pole, 73 means South Pole. For longitude,
    # 1 means 0 degrees East, 37 is 90E, 73 is 180, 109 is 90W or 270E, 144 is 2.5W.
    
    # These roughly cover Scotland.
    # lat.range <- 13:14
    # lon.range <- 142:143
    
    # These are the area used by Ludescher et al, 2013. It is 27x69 points
    # which then subsampled to 9 by 23.
    #lat.range <- 24:50
    #lon.range <- 48:116 
    
    # 3. Supply the years
    # firstyear <- 1950
    # lastyear <- 1952
    
    # 4. Supply the output name as a text string. paste0() concatenates strings
    # which you may find handy:
    # outputfilename <- paste0("Pacific-", firstyear, "-", lastyear, ".txt")
    
    
    ###############################################################################
    ###############################################################################
    
    
    #                      Example of output
    
    
    #  S013E142 S013E143 S014E142 S014E143
    #  Y1950P001 281.60000272654 281.570002727211 281.60000272654 280.970002740622
    #  Y1950P002 280.740002745762 280.270002756268 281.070002738386 280.49000275135
    #  Y1950P003 280.100002760068 278.820002788678 281.120002737269 280.070002760738
    #  Y1950P004 281.070002738386 279.420002775267 281.620002726093 280.640002747998
    #  ...
    #  Y1950P193 285.450002640486 285.290002644062 285.720002634451 285.75000263378
    #  Y1950P194 285.570002637804 285.640002636239 286.070002626628 286.570002615452
    #  Y1950P195 285.92000262998 286.220002623275 286.200002623722 286.620002614334
    #  ...
    #  Y1950P364 276.100002849475 275.350002866238 276.37000284344 275.200002869591
    #  Y1950P365 276.990002829581 275.820002855733 276.020002851263 274.72000288032
    #  Y1951P001 278.220002802089 277.470002818853 276.700002836064 275.870002854615
    #  Y1951P002 277.750002812594 276.890002831817 276.650002837181 275.520002862439
    #  ...
    #  Y1952P365 280.35000275448 280.120002759621 280.370002754033 279.390002775937
    
    # There is one row for each day, and 365 days in each year (leap days are 
    # omitted). In each row, you have temperatures in Kelvin for each grid 
    # point in a rectangle.
    
    # S13E142 means 13 steps South from the North Pole and 142 steps East from 
    # Greenwich. The points are in reading order, starting at the top-left 
    # (Northmost, Westmost) and going along the top row first.
    
    # Y1950P001 means year 1950, day 1. (P because longer periods might be used 
    # later.)
    
    ###############################################################################
    ###############################################################################
    
    
    Comment Source:Graham has helpfully put this commented-out stuff at the top of the script: ~~~~ ############################################################################### ############################################################################### # You should be able to use this by editing this section only. setwd("C:/Users/Work/AAA/Programming/ProgramOutput/Nino") lat.range <- 13:14 lon.range <- 142:143 firstyear <- 1957 lastyear <- 1958 outputfilename <- paste0("Scotland-", firstyear, "-", lastyear, ".txt") ############################################################################### ############################################################################### # Explanation # 1. Use setwd() to set the working directory to the one # containing the .nc files such as air.sig995.1951.nc. # Example: # setwd("C:/Users/Work/AAA/Programming/ProgramOutput/Nino") # 2. Supply the latitude and longitude range. The NOAA data is # every 2.5 degrees. The ranges are supplied as the number of steps of this # size. For latitude, 1 means North Pole, 73 means South Pole. For longitude, # 1 means 0 degrees East, 37 is 90E, 73 is 180, 109 is 90W or 270E, 144 is 2.5W. # These roughly cover Scotland. # lat.range <- 13:14 # lon.range <- 142:143 # These are the area used by Ludescher et al, 2013. It is 27x69 points # which then subsampled to 9 by 23. #lat.range <- 24:50 #lon.range <- 48:116 # 3. Supply the years # firstyear <- 1950 # lastyear <- 1952 # 4. Supply the output name as a text string. paste0() concatenates strings # which you may find handy: # outputfilename <- paste0("Pacific-", firstyear, "-", lastyear, ".txt") ############################################################################### ############################################################################### # Example of output # S013E142 S013E143 S014E142 S014E143 # Y1950P001 281.60000272654 281.570002727211 281.60000272654 280.970002740622 # Y1950P002 280.740002745762 280.270002756268 281.070002738386 280.49000275135 # Y1950P003 280.100002760068 278.820002788678 281.120002737269 280.070002760738 # Y1950P004 281.070002738386 279.420002775267 281.620002726093 280.640002747998 # ... # Y1950P193 285.450002640486 285.290002644062 285.720002634451 285.75000263378 # Y1950P194 285.570002637804 285.640002636239 286.070002626628 286.570002615452 # Y1950P195 285.92000262998 286.220002623275 286.200002623722 286.620002614334 # ... # Y1950P364 276.100002849475 275.350002866238 276.37000284344 275.200002869591 # Y1950P365 276.990002829581 275.820002855733 276.020002851263 274.72000288032 # Y1951P001 278.220002802089 277.470002818853 276.700002836064 275.870002854615 # Y1951P002 277.750002812594 276.890002831817 276.650002837181 275.520002862439 # ... # Y1952P365 280.35000275448 280.120002759621 280.370002754033 279.390002775937 # There is one row for each day, and 365 days in each year (leap days are # omitted). In each row, you have temperatures in Kelvin for each grid # point in a rectangle. # S13E142 means 13 steps South from the North Pole and 142 steps East from # Greenwich. The points are in reading order, starting at the top-left # (Northmost, Westmost) and going along the top row first. # Y1950P001 means year 1950, day 1. (P because longer periods might be used # later.) ############################################################################### ############################################################################### ~~~~
  • 18.
    edited June 2014

    I'm still quite unsure of what will happen when I actually run the script: a bunch of data as shown in the example will spew forth, but where will it go? I would like it to go into some array, or maybe some file, so I can then do something with it. But I should just see what happens.

    Now that I look at his example, I see that it has a very small latitude range, longitude range, and range of years. That's just what I want... so I don't actually need to edit his script. In fact I should leave it exactly the same so I can compare my output with his.

    I also see that this line is a huge clue:

    outputfilename <- paste0("Scotland-", firstyear, "-", lastyear, ".txt")

    The output is going to a file, apparently. Something to do with Scotland...

    This helps a bit too:

    4. Supply the output name as a text string. paste0() concatenates strings 
    which you may find handy:
    outputfilename <- paste0("Pacific-", firstyear, "-", lastyear, ".txt")
    

    Somehow by adjusting these parameters I'll get to choose a name for the file where the output goes. I guess.

    Okay, time to grit my teeth and just try it.

    Comment Source:I'm still quite unsure of what will happen when I actually run the script: a bunch of data as shown in the example will spew forth, but _where will it go?_ I would like it to go into some array, or maybe some file, so I can then do something with it. But I should just see what happens. Now that I look at his example, I see that it has a very small latitude range, longitude range, and range of years. That's just what I want... so I don't actually need to edit his script. In fact I should leave it exactly the same so I can compare my output with his. I also see that this line is a huge clue: `outputfilename <- paste0("Scotland-", firstyear, "-", lastyear, ".txt")` The output is going to a file, apparently. Something to do with Scotland... This helps a bit too: ~~~~ 4. Supply the output name as a text string. paste0() concatenates strings which you may find handy: outputfilename <- paste0("Pacific-", firstyear, "-", lastyear, ".txt") ~~~~ Somehow by adjusting these parameters I'll get to choose a name for the file where the output goes. I guess. Okay, time to grit my teeth and just try it.
  • 19.

    Well, to my surprise the script doesn't actually run: it appears in a little window called "R Editor" which is awkwardly confined to move around only inside the window I've been calling the R interface. Presumably I could now edit the script. But I don't want to edit it. I want to run it!

    There's no menu on top of the R Editor, so it's not instantly obvious what to do. But when in doubt about what your options are in a given situation, click the right-hand mouse button.

    I then get some options including

    Run line or selection: Ctrl+R

    How do I run the whole damned script? Clicking control R just does this:

    >

    so presumably I ran nothing at all. Then I mouse over the whole script, "selecting" it and making it turn blue, and hit control R again. Seems like a weird way to run a program, but a bunch of stuff happens...

    Mainly, however, it prints the whole program in my R interface, in red, with this error message in blue:

    Error in setwd("C:/Users/Work/AAA/Programming/ProgramOutput/Nino") : 
      cannot change working directory
    

    Duh! I forgot to edit the line where Graham had set his working directory, namely this:

    setwd("C:/Users/Work/AAA/Programming/ProgramOutput/Nino")

    I don't want to reset the working directory.

    So, I'll delete that line in the R Editor, and select the whole script again, and hit control R again.

    Now it prints out the whole program again, with no error message in blue. So I should look in my working directory and see if R has deposited a file there...

    Comment Source:Well, to my surprise the script doesn't actually run: it appears in a little window called "R Editor" which is awkwardly confined to move around only inside the window I've been calling the R interface. Presumably I could now edit the script. But I don't want to edit it. I want to run it! There's no menu on top of the R Editor, so it's not instantly obvious what to do. But _when in doubt about what your options are in a given situation, click the right-hand mouse button_. I then get some options including `Run line or selection: Ctrl+R` How do I run the whole damned script? Clicking control R just does this: `>` so presumably I ran nothing at all. Then I mouse over the whole script, "selecting" it and making it turn blue, and hit control R again. Seems like a weird way to run a program, but a bunch of stuff happens... Mainly, however, it prints the whole program in my R interface, in red, with this error message in blue: ~~~~ Error in setwd("C:/Users/Work/AAA/Programming/ProgramOutput/Nino") : cannot change working directory ~~~~ Duh! I forgot to edit the line where Graham had set his working directory, namely this: ` setwd("C:/Users/Work/AAA/Programming/ProgramOutput/Nino")` I don't want to reset the working directory. So, I'll delete that line in the R Editor, and select the whole script again, and hit control R again. Now it prints out the whole program again, with no error message in blue. So I should look in my working directory and see if R has deposited a file there...
  • 20.

    Hmm, I don't see a file there... no sign of any output.

    So, I think I'll beg Graham for help now. Where does the output go, when you run netcdf-convertor.R? Into a file somewhere?

    Comment Source:Hmm, I don't see a file there... no sign of any output. So, I think I'll beg Graham for help now. Where does the output go, when you run `netcdf-convertor.R`? Into a file somewhere?
  • 21.

    Re comment 17, there is Source R code... That will do the same job as Open script... followed by selecting everything followed by Ctrl+R, but it won't give you a chance to edit it. I think it is better to edit and save the file, then use Source R code... Then you can show us your file exactly as you sent it to R.

    Here are instructions for doing what you already did:

    It suggests that I “Download GitHub for Windows 2.0”. That seems potentially useful, so I go ahead and download something called GitHubSetup.exe into my Downloads folder, then click on that. It asks me if I want to install it, and I say yes.

    It is potentially useful, but ignore it for now.

    Comment Source:Re comment 17, there is `Source R code...` That will do the same job as `Open script...` followed by selecting everything followed by `Ctrl+R`, but it won't give you a chance to edit it. I think it is better to edit **and save** the file, then use `Source R code...` Then you can show us your file exactly as you sent it to R. Here are instructions for doing what you already did: * Go here `https://github.com/azimuth-project/el-nino/tree/master/R` * Right-click `netcdf-convertor.R` and use `Save link as...` to save it on your computer. * Edit it (in the R editor or another text editor like Notepad) and save it * Start R and use `File - Source R code...` > It suggests that I “Download GitHub for Windows 2.0”. That seems potentially useful, so I go ahead and download something called GitHubSetup.exe into my Downloads folder, then click on that. It asks me if I want to install it, and I say yes. It is potentially useful, but ignore it for now.
  • 22.
    edited June 2014

    Where does the output go, when you run netcdf-convertor.R? Into a file somewhere?

    You should get a file in your working directory called Scotland-1950-1952.txt about 41Kb in size. I don't know what's wrong. This sounds promising:

    Now it prints out the whole program again, with no error message in blue.

    So perhaps it has worked but there's some confusion about where the output is. Can you see the 30 files air.sig995.1950.nc ... air.sig995.1979.nc with sizes about 7.5Mb? And your R script?

    After it has run, you can try this

    plot(setof.Kvals[,1])
    

    You should see a plot of daily temperatures in Kelvin for Scotland over three years.

    Comment Source:> Where does the output go, when you run netcdf-convertor.R? Into a file somewhere? You should get a file in your working directory called `Scotland-1950-1952.txt` about 41Kb in size. I don't know what's wrong. This sounds promising: > Now it prints out the whole program again, with no error message in blue. So perhaps it has worked but there's some confusion about where the output is. Can you see the 30 files `air.sig995.1950.nc` ... `air.sig995.1979.nc` with sizes about 7.5Mb? And your R script? After it has run, you can try this ~~~~ plot(setof.Kvals[,1]) ~~~~ You should see a plot of daily temperatures in Kelvin for Scotland over three years.
  • 23.

    From 20:

    There’s no menu on top of the R Editor,

    That's odd. I see a menu.

    Comment Source:From 20: > There’s no menu on top of the R Editor, That's odd. I see a menu.
  • 24.

    Graham wrote:

    So perhaps it has worked but there’s some confusion about where the output is. Can you see the 30 files air.sig995.1950.nc … air.sig995.1979.nc with sizes about 7.5Mb? And your R script?

    Yes, I see all those, but not anything like Scotland-1950-1952.txt. It's my bedtime, so I'll try again tomorrow.

    (Since the script says

    firstyear <- 1957
    lastyear <- 1958
    

    I think I should get a file Scotland-1957-1958.txt... but anyway, I see nothing remotely resembling this.)

    Comment Source:Graham wrote: > So perhaps it has worked but there’s some confusion about where the output is. Can you see the 30 files air.sig995.1950.nc … air.sig995.1979.nc with sizes about 7.5Mb? And your R script? Yes, I see all those, but not anything like `Scotland-1950-1952.txt`. It's my bedtime, so I'll try again tomorrow. (Since the script says ~~~~ firstyear <- 1957 lastyear <- 1958 ~~~~ I think I should get a file `Scotland-1957-1958.txt`... but anyway, I see nothing remotely resembling this.)
  • 25.

    What's a good minimal example of an R program that creates a file and puts it in the working directory?

    Comment Source:What's a good minimal example of an R program that creates a file and puts it in the working directory?
  • 26.
    I just wanted to comment that the method of downloading a bunch of raw data via 'ftp' (file transfer protocol) is a great one to become familiar with.
    If you poke around on ftp://ftp.cdc.noaa.gov/Datasets or some other ftp servers maintained by government agencies you will find all the data you could ever want.
    Examples of things you can download for free: raw multispectral satellite images, processed data products, 're-analysis' data (which is some way of combining analysis/simulation to assimilate data), sea surface temperature anomalies at resolutions much higher than 2.5 degrees (although you pay for that in file size). Also believe it or not people actually use NetCDF files quite widely, so once you know how to play around with those you'll find the world quite literally at your fingertips!
    Comment Source:I just wanted to comment that the method of downloading a bunch of raw data via 'ftp' (file transfer protocol) is a great one to become familiar with. If you poke around on ftp://ftp.cdc.noaa.gov/Datasets or some other ftp servers maintained by government agencies you will find all the data you could ever want. Examples of things you can download for free: raw multispectral satellite images, processed data products, 're-analysis' data (which is some way of combining analysis/simulation to assimilate data), sea surface temperature anomalies at resolutions much higher than 2.5 degrees (although you pay for that in file size). Also believe it or not people actually use NetCDF files quite widely, so once you know how to play around with those you'll find the world quite literally at your fingertips!
  • 27.

    What’s a good minimal example of an R program that creates a file and puts it in the working directory?

    write.table(1,"x.txt")

    Comment Source:> What’s a good minimal example of an R program that creates a file and puts it in the working directory? `write.table(1,"x.txt")`
  • 28.
    edited June 2014
    Here is some R code that creates a vector and a matrix and saves them as comma-separated-value files in your working directory. It also writes a text-file with the words "Hello, world!" inside.

    sets the working directory, where files will be saved

    setwd("C:/Users/Blake/Desktop/Elnino")

    makes a vector

    A
    Comment Source:Here is some R code that creates a vector and a matrix and saves them as comma-separated-value files in your working directory. It also writes a text-file with the words "Hello, world!" inside. # sets the working directory, where files will be saved setwd("C:/Users/Blake/Desktop/Elnino") # makes a vector A<-c(0,1,0,1) # makes a matrix AB<-matrix(A,nrow=2,ncol=2) # saves the vector A, as a comma-seperated-value file firstvector.csv write.csv(A,file="firstvector.csv") # saves the matrix A, as a comma-seperated-value file firstmatrix.csv write.csv(AB, file="firstmatrix.csv") # creates a file output.txt containing the text: Hello, World! writeLines("Hello, world!","output.txt") # can also save using commands like write.table() or save() for tables or R data frames
  • 29.

    Thanks, Graham and Blake!

    This "minimal example" worked fine:

    write.table(1,"x.txt")

    Going to my working directory, I found a file x.txt containing this:

    "x"
    "1" 1
    

    Not quite sure what the second line means, but it's working! Next, I'll try

    writeLines("Hello, world!","output.txt")

    Yes! I get a file containing

    Hello, world!
    

    I'll also run Blake's other examples, and then try to debug what I was originally trying to do.

    Comment Source:Thanks, Graham and Blake! This "minimal example" worked fine: `write.table(1,"x.txt")` Going to my working directory, I found a file `x.txt` containing this: ~~~~ "x" "1" 1 ~~~~ Not quite sure what the second line means, but it's working! Next, I'll try `writeLines("Hello, world!","output.txt")` Yes! I get a file containing ~~~~ Hello, world! ~~~~ I'll also run Blake's other examples, and then try to debug what I was originally trying to do.
  • 30.

    Blake's examples worked fine too.

    > A<-c(0,1,0,1)
    > A
    [1] 0 1 0 1
    > AB<-matrix(A,nrow=2,ncol=2)
    > 
    > AB
         [,1] [,2]
    [1,]    0    0
    [2,]    1    1
    

    Not sure why a vector is labelled just [1] while a 2d matrix has labels for rows and columns...

    Anyway,

    > write.csv(AB,file="firstmatrix.csv")

    produces a "comma-separated-value" file which (for me) opens by default with Microsoft Excel. I'd rather set the default to OpenOffice .Calc, since I prefer freeware... but when I do I get gunk, even after choosing settings that seem to correspond to comma-separated values. Anyway, a small complaint. Now on to the big issue.

    Comment Source:Blake's examples worked fine too. ~~~~ > A<-c(0,1,0,1) > A [1] 0 1 0 1 > AB<-matrix(A,nrow=2,ncol=2) > > AB [,1] [,2] [1,] 0 0 [2,] 1 1 ~~~~ Not sure why a vector is labelled just `[1]` while a 2d matrix has labels for rows and columns... Anyway, `> write.csv(AB,file="firstmatrix.csv")` produces a "comma-separated-value" file which (for me) opens by default with Microsoft Excel. I'd rather set the default to OpenOffice .Calc, since I prefer freeware... but when I do I get gunk, even after choosing settings that seem to correspond to comma-separated values. Anyway, a small complaint. Now on to the big issue.
  • 31.
    edited June 2014

    WOW, IT WORKED!

    I bet last time I'd somehow set the working directory to Graham's favorite directory and not succeeded in resetting it to mine... this time I changed the relevant line in my copy of netcdf-convertor.R

    setwd("C:/Users/JOHN/Documents/My Backups/Azimuth/el nino")

    before running this script. A file called Scotland-1957-1958.txt was created, which contains just the stuff Graham said it should.

    Now as an exercise I would like to create a file containing the temperature near Riverside, California over a long time period... and then plot that as a graph. I have only the foggiest idea how to do the latter step, but I'll give it a try.

    Comment Source:WOW, IT WORKED! <img src = "http://math.ucr.edu/home/baez/emoticons/surprised.gif" alt = ""/> I bet last time I'd somehow set the working directory to Graham's favorite directory and not succeeded in resetting it to mine... this time I changed the relevant line in my copy of `netcdf-convertor.R` `setwd("C:/Users/JOHN/Documents/My Backups/Azimuth/el nino")` before running this script. A file called `Scotland-1957-1958.txt` was created, which contains just the stuff Graham said it should. Now as an exercise I would like to create a file containing the temperature near Riverside, California over a long time period... and then plot that as a graph. I have only the foggiest idea how to do the latter step, but I'll give it a try.
  • 32.

    Graham said:

    # 2. Supply the latitude and longitude range.  The NOAA data is 
    # every 2.5 degrees. The ranges are supplied as the number of steps of this 
    # size. For latitude, 1 means North Pole, 73 means South Pole. For longitude,
    # 1 means 0 degrees East, 37 is 90E, 73 is 180, 109 is 90W or 270E, 144 is 2.5W.
    

    Googling the info, I see Riverside California is at 33.9481° N, 117.3961° W.

    That's about 50 degrees south of the North Pole, so 20 steps of size 2.5°. But since NOAA are a bunch of idiots (or because some idiot decided everyone should count starting at 1 instead of 0 even in contexts like this), the North Pole itself is step 1, not step 0... so Riverside is latitude step 21. So I do:

    lat.range <- 21:21

    For longitude, 117.3961° W is 242.61° E, which is about 97 steps of size 2.5°, which counts as step 98 according to this braindead system. So I do:

    long.range <- 98:98

    I also do

    firstyear <- 1948
    lastyear <- 1979
    

    For this purpose I'd really like to see what's going on in more recent years, when I've lived in Riverside, but I haven't downloaded those more recent years.

    Comment Source:Graham said: ~~~~ # 2. Supply the latitude and longitude range. The NOAA data is # every 2.5 degrees. The ranges are supplied as the number of steps of this # size. For latitude, 1 means North Pole, 73 means South Pole. For longitude, # 1 means 0 degrees East, 37 is 90E, 73 is 180, 109 is 90W or 270E, 144 is 2.5W. ~~~~ Googling the info, I see Riverside California is at 33.9481° N, 117.3961° W. That's about 50 degrees south of the North Pole, so 20 steps of size 2.5°. But since NOAA are a bunch of idiots (or because some idiot decided everyone should count starting at 1 instead of 0 even in contexts like this), the North Pole itself is step 1, not step 0... so Riverside is latitude step 21. So I do: `lat.range <- 21:21` For longitude, 117.3961° W is 242.61° E, which is about 97 steps of size 2.5°, which counts as step 98 according to this braindead system. So I do: `long.range <- 98:98` I also do ~~~~ firstyear <- 1948 lastyear <- 1979 ~~~~ For this purpose I'd really like to see what's going on in more recent years, when I've lived in Riverside, but I haven't downloaded those more recent years.
  • 33.
    edited June 2014

    Hmm, I get an error message around here:

    > setof.Kvals <- NULL
    > for (i in firstyear:lastyear) {
    +   setof.Kvals <- rbind(setof.Kvals, make.Kvals.for.year(i))
    + }
    Error: No such file or directory
    

    Not sure why...

    The output seems to consist of Riverside temperatures for 1948; then it stops. So something about moving on to the next year doesn't work.

    Maybe Graham did something wrong and his program freaks out if the latitude and longitude ranges consist of just a single step?

    > lat.range <- 21:21
    > lon.range <- 98:98
    

    I change them to

    > lat.range <- 21:22
    > lon.range <- 98:99
    

    but the problem persists: I get temperature data for 1948 but no subsequent years.

    Comment Source:Hmm, I get an error message around here: ~~~~ > setof.Kvals <- NULL > for (i in firstyear:lastyear) { + setof.Kvals <- rbind(setof.Kvals, make.Kvals.for.year(i)) + } Error: No such file or directory ~~~~ Not sure why... The output seems to consist of Riverside temperatures for 1948; then it stops. So something about moving on to the next year doesn't work. Maybe Graham did something wrong and his program freaks out if the latitude and longitude ranges consist of just a single step? ~~~~ > lat.range <- 21:21 > lon.range <- 98:98 ~~~~ I change them to ~~~~ > lat.range <- 21:22 > lon.range <- 98:99 ~~~~ but the problem persists: I get temperature data for 1948 but no subsequent years.
  • 34.

    As a sanity check I close R and run Graham's original script again. It works: no bugs.

    I change his script so the latitude and longitude ranges consist of a single grid point. It still works.

    I change it to my preferred range for Riverside:

    > lat.range <- 21:21
    > lon.range <- 98:98
    

    It still works... apparently... but I notice the output file says "S013E142" on top, which corresponds to Graham's original choice of 13 steps south, 142 steps east. Maybe when I keep running R scripts in the same window certain variables aren't getting updated?

    I'll close that window and start over.

    Comment Source:As a sanity check I close R and run Graham's original script again. It works: no bugs. I change his script so the latitude and longitude ranges consist of a single grid point. It still works. I change it to my preferred range for Riverside: ~~~~ > lat.range <- 21:21 > lon.range <- 98:98 ~~~~ It still works... apparently... but I notice the output file says "S013E142" on top, which corresponds to Graham's original choice of 13 steps south, 142 steps east. Maybe when I keep running R scripts in the same window certain variables aren't getting updated? I'll close that window and start over.
  • 35.

    In the process, I change the script so it saves to a file called "Riverside" instead of "Scotland".

    Now I get the same error message:

    > setof.Kvals <- NULL
    > for (i in firstyear:lastyear) {
    +   setof.Kvals <- rbind(setof.Kvals, make.Kvals.for.year(i))
    + }
    Error: No such file or directory
    

    I change "Riverside" back to "Scotland", but the error message persists. Clearly I don't understand the source of this bug. This is why I hate programming compared to math, where it's only my own brain that I'm fighting against.

    Comment Source:In the process, I change the script so it saves to a file called "Riverside" instead of "Scotland". Now I get the same error message: ~~~~ > setof.Kvals <- NULL > for (i in firstyear:lastyear) { + setof.Kvals <- rbind(setof.Kvals, make.Kvals.for.year(i)) + } Error: No such file or directory ~~~~ I change "Riverside" back to "Scotland", but the error message persists. Clearly I don't understand the source of this bug. This is why I hate programming compared to math, where it's only my own brain that I'm fighting against.
  • 36.

    In a text editor I edit Graham's script netcdf-convertor.R so it has my preferred range of latitudes and longitudes and filename for output. I forget to change the range of years. I start up R from scratch, run the script, and it works with no error message, outputting a file that says

    S021E098

    on top, as it should.

    I close down R, edit the script again so it has 1948-1979 as its range of dates, and repeat the process.

    Now I get that error message again! So maybe something about this range of dates is the problem?

    Comment Source:In a text editor I edit Graham's script `netcdf-convertor.R` so it has my preferred range of latitudes and longitudes and filename for output. I forget to change the range of years. I start up R from scratch, run the script, and it works with no error message, outputting a file that says `S021E098` on top, as it should. I close down R, edit the script again so it has 1948-1979 as its range of dates, and repeat the process. Now I get that error message again! So maybe something about this range of dates is the problem?
  • 37.
    edited June 2014

    I close down R, edit the script again so it has 1950-1960 as its range of dates, start up R from scratch, and run the script.

    Now it works with no error messages!

    Now I finally see what the problem is: for some reason the data from the year 1949 hadn't downloaded, even though all other years from 1948 to 1979 did.

    If the error message had said the name of the file it was unable to find, I would have known this right away.

    On the other hand, if I'd guessed that 'missing file' meant that an input file was missing, I would have saved a lot of time. It didn't occur to me that out of all the files I downloaded, one file would somehow not work. And not knowing how things work, I too easily guessed that "missing file" referred to some file that was created in the inner workings of Graham's script... i.e., some bug that was beyond my comprehension, given my limited understanding of R. So I suppose I've learned a lesson here.

    Comment Source:I close down R, edit the script again so it has 1950-1960 as its range of dates, start up R from scratch, and run the script. Now it works with no error messages! Now I finally see what the problem is: for some reason the data from the year 1949 hadn't downloaded, even though all other years from 1948 to 1979 did. If the error message had said the _name_ of the file it was unable to find, I would have known this right away. On the other hand, if I'd guessed that 'missing file' meant that an input file was missing, I would have saved a lot of time. It didn't occur to me that out of all the files I downloaded, one file would somehow not work. And not knowing how things work, I too easily guessed that "missing file" referred to some file that was created in the inner workings of Graham's script... i.e., some bug that was beyond my comprehension, given my limited understanding of R. So I suppose I've learned a lesson here.
  • 38.
    edited June 2014

    I download the missing file, air.sig995.1949 from here: ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/.

    Now everything works fine!

    Comment Source:I download the missing file, `air.sig995.1949` from here: [ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/](ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.dailyavgs/surface/). Now everything works fine!
  • 39.
    edited June 2014

    Now I have this huge list of temperatures near Riverside, one per day, in kelvin, from 1948 to 1979:

    S021E098
    Y1948P001 268.57
    Y1948P002 272.87
    Y1948P003 274.37
    Y1948P004 276.4
    Y1948P005 276.17
    

    etc...

    Y1979P362 263.23
    Y1979P363 266.13
    Y1979P364 268.48
    Y1979P365 276.16
    

    Now let's see if I'm smart enough to plot them! How can I remove the Y1979P365 type crap and just plot the list of numbers?

    It seems like a difficult challenge, since I don't know how to get R to grab a .txt file, read the second entry in each row, and make a list of those numbers. Worse, I've never done anything like this in any programming language.

    My first impulse is to wish those Y1979P365 things weren't there... and thus, to rewrite Graham's script so they don't get there in the first place. That would at least simplify the problem.

    I look at Graham's script and see:

     # extract the data as a vector (not a matrix) at each time
    > # These functions do the translation
    > index.from.latlon <- function(i, j) { (i-lat.range[1]) * n.lon + (j-lon.range[1]) + 1 }
    > lon.from.index <- function(idx) { (idx-1) %% n.lon  +  lon.range[1] }
    > lat.from.index <- function(idx) { floor((idx-1) / n.lon)  +  lat.range[1] }
    > 
    > index.table <- matrix(0, nrow=biggest.lat, ncol=biggest.lon)
    > for (lat in lat.range) {
    +   for (lon in lon.range) {
    +     index.table[lat, lon] <- index.from.latlon(lat, lon)
    +   }
    + }
    > 
    > 
    > point.names <- function() {
    +   pointnames <- rep("", n.points)
    +   for (idx in 1:n.points) {
    +     lon <- lon.from.index(idx)
    +     lat <- lat.from.index(idx)
    +     pointnames[idx] <- paste0("S", sprintf("%03d", lat), "E", sprintf("%03d", lon))
    +     stopifnot(idx == index.from.latlon(lat, lon))
    +   }
    +   pointnames
    + }
    > 
    

    Almost incomprehensible stuff! Luckily he said something in English: this stuff is producing one vector of temperatures for each time, in general... which for my degenerate case happens to be a single temperature for each time. I wish I could get a single vector of temperatures, with one temperature for each time.

    Nearing the pit of desperation, I console myself by typing

    plot(1:8)
    

    and get a graph of the numbers from 1 to 8. So presumably this is how R plots vectors. (I hope R knows a vector is just a list of numbers.) I try

    plot(1,2,3,4,5)
    

    and get an error message. So I don't even know what R calls the vector (1,2,3,4,5).

    Comment Source:Now I have this huge list of temperatures near Riverside, one per day, in kelvin, from 1948 to 1979: ~~~~ S021E098 Y1948P001 268.57 Y1948P002 272.87 Y1948P003 274.37 Y1948P004 276.4 Y1948P005 276.17 ~~~~ etc... ~~~~ Y1979P362 263.23 Y1979P363 266.13 Y1979P364 268.48 Y1979P365 276.16 ~~~~ Now let's see if I'm smart enough to plot them! How can I remove the `Y1979P365` type crap and just plot the list of numbers? It seems like a difficult challenge, since I don't know how to get R to grab a .txt file, read the second entry in each row, and make a list of those numbers. Worse, I've never done anything like this in <i>any</i> programming language. My first impulse is to wish those `Y1979P365` things weren't there... and thus, to rewrite Graham's script so they don't get there in the first place. That would at least simplify the problem. I look at Graham's script and see: ~~~~ # extract the data as a vector (not a matrix) at each time > # These functions do the translation > index.from.latlon <- function(i, j) { (i-lat.range[1]) * n.lon + (j-lon.range[1]) + 1 } > lon.from.index <- function(idx) { (idx-1) %% n.lon + lon.range[1] } > lat.from.index <- function(idx) { floor((idx-1) / n.lon) + lat.range[1] } > > index.table <- matrix(0, nrow=biggest.lat, ncol=biggest.lon) > for (lat in lat.range) { + for (lon in lon.range) { + index.table[lat, lon] <- index.from.latlon(lat, lon) + } + } > > > point.names <- function() { + pointnames <- rep("", n.points) + for (idx in 1:n.points) { + lon <- lon.from.index(idx) + lat <- lat.from.index(idx) + pointnames[idx] <- paste0("S", sprintf("%03d", lat), "E", sprintf("%03d", lon)) + stopifnot(idx == index.from.latlon(lat, lon)) + } + pointnames + } > ~~~~ Almost incomprehensible stuff! Luckily he said something in English: this stuff is producing one _vector_ of temperatures for each time, in general... which for my degenerate case happens to be a _single_ temperature for each time. I wish I could get a single vector of temperatures, with one temperature for each time. Nearing the pit of desperation, I console myself by typing ~~~~ plot(1:8) ~~~~ and get a graph of the numbers from 1 to 8. So presumably this is how R plots vectors. (I hope R knows a vector is just a list of numbers.) I try ~~~~ plot(1,2,3,4,5) ~~~~ and get an error message. So I don't even know what R calls the vector (1,2,3,4,5).
  • 40.

    I figure it's time to look up some information, but before doing that I cleverly change the last line in Graham's script from

    > write.table(x=round(setof.Kvals, digits=2), file=outputfilename, quote=FALSE)

    to

    > write.csv(x=round(setof.Kvals, digits=2), file=outputfilename, quote=FALSE)

    It works! It goes into a file called .txt (for reasons I soon realize), but it's a comma-separated-value file, so I could now mess with it using Microsoft Excel. Pretty pathetic, I know... but I feel an urge to actually accomplish something, however small, not just messing with files and running programs Graham wrote.

    So, I open the file in Excel and delete the unwanted first column listing dates. It's not as if I know how to do anything in Excel, but you'd hope any idiot could take a file listing lots of numbers and graph it, without weeks (or even minutes) of training.

    Comment Source:I figure it's time to look up some information, but before doing that I cleverly change the last line in Graham's script from `> write.table(x=round(setof.Kvals, digits=2), file=outputfilename, quote=FALSE)` to `> write.csv(x=round(setof.Kvals, digits=2), file=outputfilename, quote=FALSE)` It works! It goes into a file called `.txt` (for reasons I soon realize), but it's a comma-separated-value file, so I could now mess with it using Microsoft Excel. Pretty pathetic, I know... but I feel an urge to actually _accomplish_ something, however small, not just messing with files and running programs Graham wrote. So, I open the file in Excel and delete the unwanted first column listing dates. It's not as if I know how to do anything in Excel, but you'd _hope_ any idiot could take a file listing lots of numbers and graph it, without weeks (or even minutes) of training.
  • 41.
    edited June 2014

    I mess around for a minute, click "File" (the left menu item, always useful when you're completely clueless) and then "Design" (since that might be related to graphs)... and get a choice of graph formats! I pick the most obvious one and get this:

    Yay! I admit I was hoping to see some global warming or even just an "urban heat island effect", but no, nothing obvious like that betwen 1948 and 1979.

    300 kelvin is about 80 °F, which looks roughly right; I may have screwed up the latitude and longitude, but at least this is more like Riverside than Scotland.

    I notice a big downward spike in the winter of 1963. As a sanity check I google "Riverside California cold snap 1963". I don't get much confirmation, except perhaps for this map of Los Angeles temperatures in 1963:

    It's often warmer in Los Angeles than Riverside in winter nights... and 33°F is unusually cold. I check January 13th on my climate data from NCAR and see that it was 254 kelvin that day, and 251 kelvin on the 12th... that's -7°F, indeed very unusually cold for Riverside.

    So the data looks roughly right.

    Comment Source:I mess around for a minute, click "File" (the left menu item, always useful when you're completely clueless) and then "Design" (since that might be related to graphs)... and get a choice of graph formats! I pick the most obvious one and get this: <img src = "http://math.ucr.edu/home/baez/ecological/riverside_temperature_1948-1979.jpg" alt = ""/> Yay! I admit I was hoping to see some global warming or even just an "urban heat island effect", but no, nothing obvious like that betwen 1948 and 1979. 300 kelvin is about 80 &deg;F, which looks roughly right; I may have screwed up the latitude and longitude, but at least this is more like Riverside than Scotland. I notice a big downward spike in the winter of 1963. As a sanity check I google "Riverside California cold snap 1963". I don't get much confirmation, except perhaps for this map of Los Angeles temperatures in 1963: <a href = "http://weatherspark.com/history/30699/1963/Los-Angeles-California-United-States"> <img src = "http://fs.weatherspark.com.s3.amazonaws.com/production/reports/history/year/000/030/699/1963/temperature_temperature_f.png" alt = ""/></a> It's often warmer in Los Angeles than Riverside in winter nights... and 33&deg;F is unusually cold. I check January 13th on my climate data from NCAR and see that it was 254 kelvin that day, and 251 kelvin on the 12th... that's -7&deg;F, indeed _very unusually_ cold for Riverside. So the data looks roughly right.
  • 42.
    edited June 2014

    I wish I could get a single vector of temperatures, with one temperature for each time.

    I told you back in comment 23:

    plot(setof.Kvals[,1])
    

    This assumes that the program has just been successfully run (and you haven't quit R yet). The temperature values are then hanging around in a matrix called setof.Kvals. If M is a matrix, you can look at the values using M[r,c] where r and c are the row and column numbers, counting from 1. To get column c, use M[,c]. So plot(setof.Kvals[,1]) says: plot column 1 of the matrix setof.Kvals.

    If you have made a file, and want to read the numbers back in, you can use something like:

    setwd("C:/Users/Work/AAA/Programming/ProgramOutput/Nino")
    Kvals <- as.matrix(read.table(file="Pacific-1950-1979.txt", header=TRUE))
    
    Comment Source:> I wish I could get a single vector of temperatures, with one temperature for each time. I told you back in comment 23: ~~~~ plot(setof.Kvals[,1]) ~~~~ This assumes that the program has just been successfully run (and you haven't quit R yet). The temperature values are then hanging around in a matrix called `setof.Kvals`. If `M` is a matrix, you can look at the values using `M[r,c]` where `r` and `c` are the row and column numbers, counting from 1. To get column `c`, use `M[,c]`. So `plot(setof.Kvals[,1])` says: plot column 1 of the matrix `setof.Kvals`. If you have made a file, and want to read the numbers back in, you can use something like: ~~~~ setwd("C:/Users/Work/AAA/Programming/ProgramOutput/Nino") Kvals <- as.matrix(read.table(file="Pacific-1950-1979.txt", header=TRUE)) ~~~~
  • 43.

    Maybe when I keep running R scripts in the same window certain variables aren’t getting updated?

    What typically happens is you run a program successfully, and variables are set. Then you change it so it doesn't run, but you don't notice this. R will go on past the error and use variables set in the successful run.I restart R when I suspect this has happened.

    Comment Source:> Maybe when I keep running R scripts in the same window certain variables aren’t getting updated? What typically happens is you run a program successfully, and variables are set. Then you change it so it doesn't run, but you don't notice this. R will go on past the error and use variables set in the successful run.I restart R when I suspect this has happened.
  • 44.

    97 steps of size 2.5°, which counts as step 98 according to this braindead system.

    Unlike many programming languages, R counts from 1. I wish it counted from 0.

    Comment Source:> 97 steps of size 2.5°, which counts as step 98 according to this braindead system. Unlike many programming languages, R counts from 1. I wish it counted from 0.
  • 45.
    edited June 2014

    I wrote:

    Googling the info, I see Riverside California is at 33.9481° N, 117.3961° W.

    That’s about 50 degrees south of the North Pole, so 20 steps of size 2.5°.

    No, it's more like 56 degrees south of the North pole, or 22 steps of size 2.5°. So, all my temperatures for Riverside were off... I was looking at someplace colder. I'll redo this stuff...

    I noticed this by realizing that a heat wave in Los Angeles in 1963 was not visible in my (fake) Riverside data. When it's hot in LA, it's almost always hotter in Riverside!

    Comment Source:I wrote: > Googling the info, I see Riverside California is at 33.9481° N, 117.3961° W. > That’s about 50 degrees south of the North Pole, so 20 steps of size 2.5°. No, it's more like 56 degrees south of the North pole, or 22 steps of size 2.5°. So, all my temperatures for Riverside were off... I was looking at someplace colder. I'll redo this stuff... I noticed this by realizing that a heat wave in Los Angeles in 1963 was not visible in my (fake) Riverside data. When it's hot in LA, it's almost always hotter in Riverside!
  • 46.

    I downloaded some GHCN station data for Riverside (i.e., from actual thermometers), and can see the global warming trend after doing some post-processing to pull it out of the noise. I'll try to get this uploaded in a day or two; I'm having computer problems at the moment. I don't see a big September 1963 heat wave, though ... and you say Riverside is even hotter than LA.

    Comment Source:I downloaded some GHCN station data for Riverside (i.e., from actual thermometers), and can see the global warming trend after doing some post-processing to pull it out of the noise. I'll try to get this uploaded in a day or two; I'm having computer problems at the moment. I don't see a big September 1963 heat wave, though ... and you say Riverside is even hotter than LA.
  • 47.
    edited June 2014

    Interesting! I was just using data from 1948 to 1979, for no particularly good reason. (Graham is avoiding looking at more recent temperature data from the Pacific, so he can formulate hypotheses about El Niño and then test them on more recent data.)

    Good luck on your computer problems.

    Maybe September 26th 1963 was a weird fluke. 109°F in LA is very rare, and I'd expect it to be about 113°F or more in Riverside on such a day. (The hottest I've experienced there is about 110°F - that's 43°C. - though I see it got up to 117°F, or 47°C, in 2006.)

    Comment Source:Interesting! I was just using data from 1948 to 1979, for no particularly good reason. (Graham is avoiding looking at more recent temperature data from the Pacific, so he can formulate hypotheses about El Ni&ntilde;o and then test them on more recent data.) Good luck on your computer problems. Maybe September 26th 1963 was a weird fluke. 109&deg;F in LA is very rare, and I'd expect it to be about 113&deg;F or more in Riverside on such a day. (The hottest I've experienced there is about 110&deg;F - that's 43&deg;C. - though I see it got up to 117&deg;F, or 47&deg;C, in 2006.)
  • 48.

    This surprised me:

    > x <- c(1,2)
    > x
    [1] 1 2
    >  mean(x^2) 
    [1] 2.5
    > mean(x)^2
    [1] 2.25
    > mean(x^2) - mean(x)^2
    [1] 0.25
    > var(x)
    [1] 0.5
    

    I would have guessed that var(x) = mean(x^2) - mean(x)^2. Are they multiplying the latter by 2 to estimate a population variance?

    Comment Source:This surprised me: ~~~~ > x <- c(1,2) > x [1] 1 2 > mean(x^2) [1] 2.5 > mean(x)^2 [1] 2.25 > mean(x^2) - mean(x)^2 [1] 0.25 > var(x) [1] 0.5 ~~~~ I would have guessed that var(x) = mean(x^2) - mean(x)^2. Are they multiplying the latter by 2 to estimate a [population variance](http://mathworld.wolfram.com/SampleVariance.html)?
  • 49.

    var() gives the unbiased sample variance, not the population variance.

    Comment Source:var() gives the unbiased sample variance, not the population variance.
  • 50.

    I think that's what I was trying to say: I guess what you're calling "unbiased sample variance" is what the Mathworld article I linked to calls the "unbiased estimator of the population variance". The point being that it's

    $$ \frac{1}{N-1} \sum_{i=1}^N (x_i - \langle x\rangle)^2 $$ with an $N-1$ instead of an $N$.

    As more of a math type, I'd call

    $$ \frac{1}{N} \sum_{i=1}^N (x_i - \langle x\rangle)^2 = \langle x^2 \rangle - \langle x\rangle^2 $$ simply the "variance", so I was initially shocked when var() didn't compute that.

    Comment Source:I think that's what I was trying to say: I guess what you're calling "unbiased sample variance" is what the Mathworld article I linked to calls the "unbiased estimator of the population variance". The point being that it's $$ \frac{1}{N-1} \sum_{i=1}^N (x_i - \langle x\rangle)^2 $$ with an $N-1$ instead of an $N$. As more of a math type, I'd call $$ \frac{1}{N} \sum_{i=1}^N (x_i - \langle x\rangle)^2 = \langle x^2 \rangle - \langle x\rangle^2 $$ simply the "variance", so I was initially shocked when var() didn't compute that.
Sign In or Register to comment.