12 adv-manip

Stat405
Advanced data manipulation 2

Hadley Wickham
Thursday, 30 September 2010

1. Colloquium Monday
2. String basics
3. Group-wise transformations
4. Practice challenges


Colloquium on
Summer research experiences

When: Monday, 4:00 - 5:00
Where: DH 1070

Coffee and Cookies will be
served ahead of time


Speakers

1. James Rigby -summer institute in bioinformatics
2. Gabi Quart - internship at Deutche Bank
3. Liz Jackson - Survey methodology summer program
4. Ollie McDonald - internship at Novartis in
Switzerland
5. Christine Peterson - research in the Med center
6. Joseph Egbulefu - research at Rice


String
basics

install.packages("stringr")
library(stringr)

str_length("Hadley")
str_c(letters, LETTERS)
str_c(letters, LETTERS, sep = " ")
str_c("H", "a", "d", "l", "e", "y", collapse = "")

tolower("Hadley")
toupper("Hadley")

str_sub("Hadley", 1, 3)
str_sub("Hadley", -1)


Your turn

Add new columns that give:
The length of each name
The ﬁrst letter of the name
The last letter of the name


library(stringr)

bnames <- read.csv("baby-names2.csv.bz2",
stringsAsFactors = FALSE)

bnames <- transform(bnames,
length = str_length(name),
first = str_sub(name, 1, 1),
last = str_sub(name, -1, -1))


Your turn

Explore how the average length of names
has changed over time (for each sex)


library(plyr)
sy <- ddply(bnames, c("sex", "year"), summarise,
avg_length = weighted.mean(length, prop))

library(ggplot2)
qplot(year, avg_length, data = sy, colour = sex,
geom = "line")


# Another approach
syl <- ddply(bnames, c("sex", "length", "year"),
summarise, prop = sum(prop))
qplot(year, prop, data = syl, colour = sex,
geom = "line") + facet_wrap(~ length)


Transformations


Transformations

What about group-wise
transformations? e.g. what if we want to
compute the rank of a name within a sex
and year?
This task is easy if we have a single year
& sex, but hard otherwise.


Transformations

What about group-wise
transformations? e.g. what if we want to
compute the rank of a name within a sex
and year?
This task is easy if we have a single year
& sex, but hard otherwise.

How would you do it for a single group?

one <- subset(bnames, sex == "boy" & year == 2008)
one$rank <- rank(-one$prop,
ties.method = "first")

# or
one <- transform(one,
rank = rank(-prop, ties.method = "first"))
head(one)

What if we want to transform
every sex and year?

Workﬂow

1. Extract a single group
2. Figure out how to solve it for just that
group
3. Use ddply to solve it for all groups


Workﬂow

1. Extract a single group
2. Figure out how to solve it for just that
group
3. Use ddply to solve it for all groups

How would you use ddply to calculate all ranks?

bnames <- ddply(bnames, c("sex", "year"), transform,
rank = rank(-prop, ties.method = "first"))


ddply + transform =
group-wise transformation

ddply + summarise =
per-group summaries

ddply + subset =
per-group subsets


Tools

You know have all the tools to solve 95%
of data manipulation problems in R. It’s
just a matter of ﬁguring out which tools to
use, and how to combine them.
The following challenges will give you
some practice.


Challenges


Warmups

Which names were most popular in 1999?
Work out the average proportion for each
name.
List the 10 names with the highest
average proportions.


# Which names were most popular in 1999?
subset(bnames, year == 1999 & rank < 10)
subset(bnames, year == 1999 & prop == max(prop))

# Average usage
overall <- ddply(bnames, "name", summarise,
prop = mean(prop))

# Top 10 names
head(arrange(overall, desc(prop)), 10)


Challenge 1

How has the total proportion of babies
with names in the top 1000 changed over
time?
How has the popularity of different initials
changed over time?


sy <- ddply(bnames, c("year","sex"), summarise,
prop = sum(prop),
npop = sum(prop > 1/1000))

qplot(year, prop, data = sy, colour = sex,
geom = "line")
qplot(year, npop, data = sy, colour = sex,
geom = "line")


Challenge 2

For each name, ﬁnd the year in which it
was most popular, and the rank in that
year. (Hint: you might ﬁnd which.max
useful).
Print all names that have been the most
popular name at least once.


most_pop <- ddply(bnames, "name", summarise,
year = year[which.max(prop)],
rank = min(rank))
most_pop <- ddply(bnames, "name", subset,
prop == max(prop))

subset(most_pop, rank == 1)

# Double challenge: Why is this last one wrong?


Challenge 3

What name has been in the top 10 most
often?
(Hint: you'll have to do this in three steps.
Think about what they are before starting)


top10 <- subset(bnames, rank <= 10)
counts <- count(top10, c("sex", "name"))

ddply(counts, "sex", subset, freq == max(freq))
head(arrange(counts, desc(freq)), 10)


Homework

No homework this week.
Use what you’ve learned to make your
projects even better!


12 adv-manip

More Related Content

Similar to 12 adv-manip

More from Hadley Wickham

12 adv-manip