• Session 5, 1/24: E62-223
• Session 6, 1/26: E51-325
• Session 7, 1/31: E62-223
• Session 8, 2/2: E51-325

"Don't Write For Loops"

"Be Lazy"

"Stupid Data Frame Tricks"

"Advanced Topics in Data Science"

1. Look left
2. Look right
3. Pick a partner (groups of three are fine)

You are going to need them soon.

We'd like to fit a LOESS smoother to the data to capture long-term trend. We can fit a single one like this:

model_data <- prices %>%       # extract a single listing's 
    filter(listing_id == 5506) # worth of data

# fit a loess model, the span is a hyperparameter, a bit 
# like lambda in LASSO
single_model <-  loess(price_per ~ as.numeric(date),  
                       data = model_data, 
                       span = .5)

model_preds <- broom::augment(single_model, model_data) 
model_preds %>% head()

##   listing_id       date price_per  .fitted  .se.fit     .resid
## 1       5506 2017-09-05      72.5 73.73994 2.042531 -1.2399385
## 2       5506 2017-09-04      72.5 73.66160 1.984215 -1.1615956
## 3       5506 2017-09-03      72.5 73.58551 1.927151 -1.0855140
## 4       5506 2017-09-02      72.5 73.51158 1.871376 -1.0115812
## 5       5506 2017-09-01      72.5 73.43968 1.816931 -0.9396847
## 6       5506 2017-08-31      72.5 73.36971 1.763855 -0.8697121

model_container <- ????
for(id in unique(prices$listing_id)){
    model <- prices %>% 
        filter(listing_id == id) %>% 
        loess(price_per ~ as.numeric(date),
              data = .,
              span = .25)
    
    model_container %>% update(model) # ?????
    .
    .
    .
}

my_list <- list('Computing', 'in', 'Optimization', 'and' ,'Statistics')

length_list <- list()
i           <- 1
for(word in my_list){
    length_list[i] <- nchar(word)
    i              <- i + 1
}

length_list %>% unlist() # just for display

## [1]  9  2 12  3 10

my_list <- list('Computing', 'in', 'Optimization', 'and' ,'Statistics')

length_list <- map(my_list, nchar) # or my_list %>% map(nchar)

length_list %>% unlist() # just for display

## [1]  9  2 12  3 10

purrr::map() applies a function (nchar) to each entry of the original list.

Use lists of data frames (and data frames of lists) to organize your work.

Data frames are the fundamental unit of data science.

Usually their columns are atomic vectors of integers, doubles, dates, characters, or booleans. E.g.

## # A tibble: 6 × 3
##   listing_id       date price_per
##        <int>     <date>     <dbl>
## 1    3075044 2017-08-22      32.5
## 2    3075044 2017-08-21      32.5
## 3    3075044 2017-08-20      32.5
## 4    3075044 2017-08-19      37.5
## 5    3075044 2017-08-18      37.5
## 6    3075044 2017-08-17      32.5

But this is somewhat inflexible. What about more complex objects? Lists can hold anything…

prices_nested <- prices %>% 
    tidyr::nest(-listing_id)

# view the data types of the columns
map(prices_nested, class) %>% unlist()

## listing_id       data 
##  "integer"     "list"

prices_nested %>% head()

## # A tibble: 6 × 2
##   listing_id               data
##        <int>             <list>
## 1    3075044 <tibble [359 × 2]>
## 2       6976 <tibble [319 × 2]>
## 3    7651065 <tibble [334 × 2]>
## 4    5706985 <tibble [344 × 2]>
## 5    2843445 <tibble [365 × 2]>
## 6     753446 <tibble [347 × 2]>

prices_nested$data[[1]] # get the first item of the list

## # A tibble: 359 × 2
##          date price_per
##        <date>     <dbl>
## 1  2017-08-22      32.5
## 2  2017-08-21      32.5
## 3  2017-08-20      32.5
## 4  2017-08-19      37.5
## 5  2017-08-18      37.5
## 6  2017-08-17      32.5
## 7  2017-08-16      32.5
## 8  2017-08-15      32.5
## 9  2017-08-14      32.5
## 10 2017-08-13      32.5
## # ... with 349 more rows

Now do the same thing, but assign the result to a new list column of prices_nested. Don't forget what you learned in Session 2!

prices_nested %>% 
    mutate(highest_price = map_dbl(data, get_biggest_price))

## # A tibble: 5 × 3
##   listing_id               data highest_price
##        <int>             <list>         <dbl>
## 1    3075044 <tibble [359 × 2]>      37.50000
## 2       6976 <tibble [319 × 2]>      32.50000
## 3    7651065 <tibble [334 × 2]>      39.50000
## 4    5706985 <tibble [344 × 2]>      66.66667
## 5    2843445 <tibble [365 × 2]>      37.50000

We know how to:

1. Nest a data frame, creating a list of data frames.
2. Use map to apply a function to every element of a list.
3. Apply a model function to an individual data frame (from Session 3).

Looks like it's time for…

my_loess <- function(data, span){
    loess(price_per ~ as.numeric(date),
    data = data, 
    span = span)
}

prices_nested <- prices %>% 
    nest(-listing_id) 

prices_modeled <- prices_nested %>% 
    mutate(model = map(data, my_loess, span = .25))

prices_with_preds <- prices_modeled %>% 
    mutate(preds = map2(model, data, augment))

prices_unnested <- prices_with_preds %>% 
    unnest(preds)

my_loess <- function(data, span){
    loess(price_per ~ as.numeric(date),
    data = data, 
    span = span)
}

prices_unnested <- prices %>% 
    nest(-listing_id) %>% 
    mutate(model = map(data, my_loess, span = .25),
           preds = map2(model, data, augment)) %>% 
    unnest(preds)

• Illustrative vignettes for purrr and broom.
• Jenny Bryan's Stat 545 course website has excellent coverage of all our R topics, including purrr.
• Our K-means example borrowed a lot from this broom vignette.

• R For Data Science is a book by Garrett Grolemund and Hadley Wickham that goes deep into a complete suite of tools for the data science pipeline.

• Automating long data science pipelines with GNU make.
• Literate programming with RMarkdown. These slides, as well as the "pretty" lecture notes, were made in R!
• Interactive data applications with Shiny
• Develop your own R packages.
• Structure of the R language – check out Advanced R.