Exploring Data Frames

How can I manipulate a dataframe?

• At this point, you’ve see it all - in the last lesson, we toured all the basic data types and data structures in R. Everything you do will be a manipulation of those tools.
• star of show == data.frame
• we learned that data.frames were vectors, so that we have consistent data down a column
• if we want a new column we start by making a vector

cats <- read.csv(file="data/feline-data.csv")

## Warning in read.table(file = file, header = header, sep = sep, quote =
## quote, : incomplete final line found by readTableHeader on 'data/feline-
## data.csv'

You can add a vector to the data frame

##     coat weight likes_string
## 1 calico    2.1            1
## 2  black    5.0            0
## 3  tabby    3.2            1

• we can then add a column via:

cats <- cbind(cats, age)

• what happened?
• Of course, R wants to see one element in our new column for every row in the table:

cats

##     coat weight likes_string
## 1 calico    2.1            1
## 2  black    5.0            0
## 3  tabby    3.2            1

age <- c(4,5,8)
cats <- cbind(cats, age)
cats

##     coat weight likes_string age
## 1 calico    2.1            1   4
## 2  black    5.0            0   5
## 3  tabby    3.2            1   8

Now how about adding rows - in this case, we saw last time that the rows of a data.frame are made of lists:

newRow <- list("tortoiseshell", 3.3, TRUE, 9)
cats <- rbind(cats, newRow)

## Warning in `[<-.factor`(`*tmp*`, ri, value = structure(c(2L, 1L, 3L,
## NA), .Label = c("black", : invalid factor level, NA generated

• Another thing - when R creates a factor, it only allows whatever is originally there when our data was loaded (‘black’, ‘calico’, ‘tabby’)
• anything that doesn’t fit into one of its categories is rejected
• we need to explicitly add it as a level

levels(cats$coat)

## [1] "black"  "calico" "tabby"

levels(cats$coat) <- c(levels(cats$coat), 'tortoiseshell')
cats <- rbind(cats, list("tortoiseshell", 3.3, TRUE, 9))

• Alternatively we can change a factor column to a character vector
• but we lose handy categories of the factor but can subseqently add any word we want to the column

str(cats)

## 'data.frame':    5 obs. of  4 variables:
##  $ coat        : Factor w/ 4 levels "black","calico",..: 2 1 3 NA 4
##  $ weight      : num  2.1 5 3.2 3.3 3.3
##  $ likes_string: int  1 0 1 1 1
##  $ age         : num  4 5 8 9 9

cats$coat <- as.character(cats$coat)
str(cats)

## 'data.frame':    5 obs. of  4 variables:
##  $ coat        : chr  "calico" "black" "tabby" NA ...
##  $ weight      : num  2.1 5 3.2 3.3 3.3
##  $ likes_string: int  1 0 1 1 1
##  $ age         : num  4 5 8 9 9

• we now know how to add rows and columns to our data.fram in R
• but in our work we’ve accidentlly added a garbage row

cats

##            coat weight likes_string age
## 1        calico    2.1            1   4
## 2         black    5.0            0   5
## 3         tabby    3.2            1   8
## 4          <NA>    3.3            1   9
## 5 tortoiseshell    3.3            1   9

• we can ask for a data.frame minus this offending row:

cats[-4,]

##            coat weight likes_string age
## 1        calico    2.1            1   4
## 2         black    5.0            0   5
## 3         tabby    3.2            1   8
## 5 tortoiseshell    3.3            1   9

• notice the comma with nothing after it to indicate we want to drop that row
• Note we can remove both new rows at once by putting the row numbers inside of a vector: `cats[c(-4,-5)]
• we can also drop all rows with NA values

na.omit(cats)

##            coat weight likes_string age
## 1        calico    2.1            1   4
## 2         black    5.0            0   5
## 3         tabby    3.2            1   8
## 5 tortoiseshell    3.3            1   9

• let’s reassign the output to cats, so our canges are made permanent

cats <- na.omit(cats)

• the key to remember when adding data to a data.frame is that columns are vectors or factors, and rows are lists
• we can also glue two data.frames together with rbind:

cats <- rbind(cats, cats)
cats

##             coat weight likes_string age
## 1         calico    2.1            1   4
## 2          black    5.0            0   5
## 3          tabby    3.2            1   8
## 5  tortoiseshell    3.3            1   9
## 11        calico    2.1            1   4
## 21         black    5.0            0   5
## 31         tabby    3.2            1   8
## 51 tortoiseshell    3.3            1   9

• but now row names are complicated
• we can remove the rownames and R will auto re-name them sequentially

rownames(cats) <- NULL
cats

##            coat weight likes_string age
## 1        calico    2.1            1   4
## 2         black    5.0            0   5
## 3         tabby    3.2            1   8
## 4 tortoiseshell    3.3            1   9
## 5        calico    2.1            1   4
## 6         black    5.0            0   5
## 7         tabby    3.2            1   8
## 8 tortoiseshell    3.3            1   9

Challenge 1

http://swcarpentry.github.io/r-novice-gapminder/05-data-structures-part2/#challenge-1

So far:

• basics of manipulating data.frames with our cat data
• let’s work on a real data set

gapminder <- read.csv("data/gapminder-FiveYearData.csv")

• mention tab-separated values files (.tsv)
• to specify a tab separator use “\tab” or read.delim()
• files can be downloading via the internet using the download.file and the read.csv can be executed to read the downloaded file such as:

download.file("https://raw.githubusercontent.com/swcarpentry/r-novice-gapminder/gh-pages/_episodes_rmd/data/gapminder-FiveYearData.csv", destfile = "data/gapminder-FiveYearData.csv")
gapminder <- read.csv("data/gapminder-FiveYearData.csv")

• alternately you can read files directly into R from the internet by replacing file paths with a web address in read.csv
• note when you do this there is not local copy saved

gapminder <- read.csv("https://raw.githubusercontent.com/swcarpentry/r-novice-gapminder/gh-pages/_episodes_rmd/data/gapminder-FiveYearData.csv")

• You can read directly from excel spreadsheets without converting them to plain text by using the readxl package
• let’s investigate gapminder

str(gapminder)

## 'data.frame':    1704 obs. of  6 variables:
##  $ country  : Factor w/ 142 levels "Afghanistan",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ year     : int  1952 1957 1962 1967 1972 1977 1982 1987 1992 1997 ...
##  $ pop      : num  8425333 9240934 10267083 11537966 13079460 ...
##  $ continent: Factor w/ 5 levels "Africa","Americas",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ lifeExp  : num  28.8 30.3 32 34 36.1 ...
##  $ gdpPercap: num  779 821 853 836 740 ...

typeof(gapminder$year)

## [1] "integer"

typeof(gapminder$lifeExp)

## [1] "double"

typeof(gapminder$country)

## [1] "integer"

str(gapminder$country)

##  Factor w/ 142 levels "Afghanistan",..: 1 1 1 1 1 1 1 1 1 1 ...

• we can also interrogate the data.frame for info about itself
• remember that str(gapminder) said there were 1704 observations of 6 variables in gapminder
• what do you think the following will produce?

length(gapminder)

## [1] 6

• a fair guess would say that the length of a data.frame would be the number of rows it has (1704)
• not the case, remember, a data.frame is a list of vectors and factors:

typeof(gapminder)

## [1] "list"

• when length gave us 6 it’s because gapminder is built out of a list of 6 columns
• to get number of rows and columns in our data set try:

nrow(gapminder)

## [1] 1704

ncol(gapminder)

## [1] 6

• or both at once

dim(gapminder)

## [1] 1704    6

• we also would want to know what the titles of all the columns are, so we can ask for them by name:

colnames(gapminder)

## [1] "country"   "year"      "pop"       "continent" "lifeExp"   "gdpPercap"

• it is important to ask if the structure R is reporting matches our intuition or expectations
• do the basic data a types reported make sense?

• if not we need to sort out problems now before they turn into negative surprises down the road

• once we are happy that the data types and structures seem reasonable, it’s time to start digging into our data properly

head(gapminder)

##       country year      pop continent lifeExp gdpPercap
## 1 Afghanistan 1952  8425333      Asia  28.801  779.4453
## 2 Afghanistan 1957  9240934      Asia  30.332  820.8530
## 3 Afghanistan 1962 10267083      Asia  31.997  853.1007
## 4 Afghanistan 1967 11537966      Asia  34.020  836.1971
## 5 Afghanistan 1972 13079460      Asia  36.088  739.9811
## 6 Afghanistan 1977 14880372      Asia  38.438  786.1134

• to make sure out analysis is reproducible we should put the code into a script file so we can come back later

Challenge 2

http://swcarpentry.github.io/r-novice-gapminder/05-data-structures-part2/#challenge-2

Challege 3

http://swcarpentry.github.io/r-novice-gapminder/05-data-structures-part2/#challenge-3