Lecture 3: Tidying and Wrangling Data

Worksheets/Tutorials - Week 03

• No need to submit anything on Canvas. Server will submit your work for you at the deadline!

• Week 3:

  • worksheet_wrangling and tutorial_wrangling: Due on Saturday, Sep 26

Reminder

Where are we? Where are we going?

image source: R for Data Science by Grolemund & Wickham

Data Wrangling!

The cartoon illustrations in this slide deck are all created by Allison Horst

• In the real world, when you get data, it’s usually very messy
  • inconsistent format (commas, tabs, semicolons, missing data, extra empty lines)
  • split into multiple files (e.g. yearly recorded data over many years)
  • corrupted files, custom formats
• when you read it successfully into R, it will often still be very messy

What is Tidy Data?

Illustrations from the Openscapes blog Tidy Data for reproducibility, efficiency, and collaboration by Julia Lowndes and Allison Horst”

What is Tidy Data

Example of Tidy Data

Examples of Tidy(?) Data

Tuberculosis data

Let’s assume the goal is to analyze the rates of tuberculosis, and perhaps consider population size in the analysis.

True (A) or False (B): This data is tidy.

Examples of Tidy(?) Data

True(A) or False(B): This data is tidy.

Examples of Tidy(?) Data

True(A) or False(B): This data is tidy.

Examples of Tidy(?) Data

True(A) or False(B): This data is tidy.

Examples of Tidy(?) Data

True(A) or False(B): This data is tidy.

Tools for tidying and wrangling data

• tidyverse package functions from:

  • dplyr package (select, filter, mutate, group_by, summarize)

  • tidyr package (pivot_longer, pivot_wider)

  • purrr package (map_dfr)

Demo Time!

Demo

The penguins data set comtains data on 344 penguins. There are 3 different species of penguins in this data set, collected from 3 islands in the Antarctica.

select One Column

The select function is used to select a subset of columns (variables) from a dataframe.

select Many Columns

select, and Assign to an Object

select All But Specific Columns

Bonus: select All Columns That Start With “bill_”

filter with One Condition

The filter function is used to choose a subset of rows (observations) in a dataframe.

Example: filter for only penguins with flippers longer than 190 mm

filter with Two Conditions

Example: filter for Chinstrap penguins with flippers longer than 190 mm

filter with Two Conditions, Assign to Object

We can also assign the filtered data set to a new object to reference later!

mutate

The mutate function transforms old columns to add new columns.

Example: convert body mass from grams to pounds

mutate and Assign to Object

The mutate function does not automatically overwrite the original data frame. So, we can save it as a new one. Note that the body_mass_g column still exists! We could drop it later using select.

1. Save intermediate objects

Disadvantages:

• The reader may be tricked into thinking the named penguins_1 and penguins_2 objects are important for some reason, while they are just temporary intermediate computations.
• Further, the reader has to look through and find where penguins_1 and penguins_2 are used in each subsequent line.
• Creating variables that we don’t need, could lead to memory issues if they are big and take up a lot of space.

2. Composing functions

Disadvantage:

• Difficult to read since you the innermost function is exectued first
  • You have to start reading from the middle of the code block (mutate above)

3. Piping

You can also pipe with the |> symbol: passes the output of a function (or a data set) to the 1st argument of another.

Notes:

• If R sees a |> at the end of a line, it keeps reading the next line of code before evaluating!
• When a data set is piped into the function, you no longer need to input the data set as the first argument

3. Piping

Advantages

• Pipes make code much more readable when you need to do a long sequence of operations on data.
• No intermediate variables created

Try It Yourself!

Try to fill in the code below that looks at the mtcars data set and:

• Creates a new variable called wt_lbs (hint: the variable wt is currently written in thousands of lbs. Simply multiply wt by 1000)

• Remove cars that have less than 20 mpg

• Keeps only the mpg, cyl, wt_lbs, and hp columns.

Grouping and Summarizing via group_by + summarize

• Grouping is when you split data into groups based on the value of a column.
• Sumarizing is when you combine data into fewer summary values.

For example computing the average particle pollution per city (source https://info201.github.io/dplyr.html):

Grouping and Summarizing via group_by + summarize

Another example: splitting the penguins data into one group per species, and then summarizing the values within each group e.g. reporting the average body_mass per species.

To do this we use group_by + summarize to iterate over species, calculating average body mass.

Iteration

• Iteration is when you need to do something repeatedly (e.g., ringing in and bagging groceries at the till)

• In data science, it’s doing the same thing over and over, too

  • important to reduce duplication of code

  • easier to see the intent of your code

  • likely have fewer bugs

map_dfr

map_dfr iterates over columns, e.g. to calculate the average for each numeric column in the penguins data

Be careful: There are a lot of map_... functions that you could use (map, map_lgl, map_chr, etc). Usually in this course we’ll only use map_dfr, but you should investigate the others yourself!

Let’s calculate the average of numeric column in penguins data set

Questions?

If not, let’s work together on this week’s worksheet.

Want to save these slides?

• Right click > Print > Save to PDF.
• Your code and notes will only save if you save as a PDF!