When we write select(penguins, species), species is not an object in our environment, yet the code works. Normally, R evaluates every argument we give a function: it looks up species, and stops with an error if it cannot find it. Functions like select() instead capture the code we typed and evaluate it somewhere else, such as inside a data frame. This is called non-standard evaluation (NSE). The tidyverse way of doing it is called tidy evaluation, and it is built on three ideas:
Quoting and unquoting: capturing code without running it, and inserting values into captured code.
Quosures: captured code together with the environment it was written in.
The data mask: evaluating code so that the columns of a data frame are found before other objects.
This chapter builds up these ideas by writing our own simple version of select(), step by step. It is adapted from the talk Learning Tidy Evaluation by Reimplementing dplyr by Daniel Chen (slides and code), updated for the current versions of the tidyverse. If you are looking for which pattern to use in your own code, rather than how it works, see the “Programming with the tidyverse functions” section of the Defining functions in R appendix (Section 65.3.2).
We use the rlang package, which implements tidy evaluation, and the penguins data set from the palmerpenguins package:
library(dplyr)
Attaching package: 'dplyr'
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
library(rlang)library(palmerpenguins)
Attaching package: 'palmerpenguins'
The following objects are masked from 'package:datasets':
penguins, penguins_raw
penguins
# A tibble: 344 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 40.3 18 195 3250
4 Adelie Torgersen NA NA NA NA
5 Adelie Torgersen 36.7 19.3 193 3450
6 Adelie Torgersen 39.3 20.6 190 3650
7 Adelie Torgersen 38.9 17.8 181 3625
8 Adelie Torgersen 39.2 19.6 195 4675
9 Adelie Torgersen 34.1 18.1 193 3475
10 Adelie Torgersen 42 20.2 190 4250
# ℹ 334 more rows
# ℹ 2 more variables: sex <fct>, year <int>
66.1 Many ways to select columns
R has many ways of selecting columns from a data frame. Most of them take the column name as a string, or the column’s position:
So non-standard evaluation is not just a tidyverse thing: base R’s subset() uses it too. But if we try to use a bare column name with [, R evaluates species as an object in our environment, and there is no such object:
penguins[, species]
Error:
! object 'species' not found
66.2 Quoting code: expressions
When we type 3 + 3, R evaluates it straight away:
3+3
[1] 6
quote() captures the code instead of running it. The result is called an expression:
ex <-quote(3+3)ex
3 + 3
We can run the expression later with eval():
eval(ex)
[1] 6
rlang::expr() does the same thing as quote():
expr(3+3)
3 + 3
You can think of an expression as the code before it runs. It is not a string, but that is a reasonable first approximation. A bare name like species is a special kind of expression, called a symbol, and 3 + 3, which calls the function +, is called a call:
str(quote(species))
symbol species
str(quote(3+3))
language 3 + 3
The rlang documentation calls quoting defusing: the code is captured like a defused bomb, so that it does not go off until we decide.
66.3 Rebuilding select()
66.3.1 Try 1: strings
Our first my_select() passes the column name straight to [:
my_select <-function(data, col) { data[, col, drop =FALSE]}
but not with a bare column name, because R evaluates species as soon as [ needs the value of col:
my_select(penguins, species)
Error:
! object 'species' not found
So we need to quote what the user typed, instead of evaluating it.
66.3.2 Try 2: quoting inside the function
expr() quotes exactly what we type, so inside a function, expr(col) gives us the symbol col: the name of the argument, not what the user passed to it:
f_expr <-function(x) {expr(x)}f_expr(hello)
x
To capture what the user typed, we need the “enriched” version, enexpr():
f_enexpr <-function(x) {enexpr(x)}f_enexpr(hello)
hello
The en prefix means “capture the code given to this argument”.
66.3.3 Try 3: capturing what the user typed
Now we can capture the user’s column names with enexprs() (the plural of enexpr(), for ...), turn them into strings, find their positions with match(), and subset the data on those positions with regular base R:
my_select <-function(data, ...) { cols <-enexprs(...) # quote what the user typed cols_chr <-as.vector(cols, mode ="character") # turn the expressions into strings idx <-match(cols_chr, names(data)) # find the column positions data[, idx, drop =FALSE] # regular base R}
Error in `data[, idx, drop = FALSE]`:
! Can't subset columns with `idx`.
✖ Subscript `idx` can't contain missing values.
✖ It has a missing value at location 1.
Our function quoted col_name as the symbol col_name, and there is no column with that name. We need a way to tell a quoting function “use the value of this variable, not its name”. That is what !! (pronounced “bang-bang”) does: it unquotes (rlang calls it injects) a value into the code before the function captures it:
!!! (“bang-bang-bang”) does the same for a list of values, inserting each one as a separate argument (similar to *args in Python). rlang calls this splicing:
cols <-exprs(species, island, year)my_select(penguins, !!!cols)
!! and !!! only work inside functions that quote their arguments with enexpr(), enquo(), { } and friends (and in rlang::inject()). Anywhere else, !! is just two ! (not) operators in a row:
!!TRUE
[1] TRUE
66.5 Quosures: code plus its environment
66.5.1 Environments
An expression on its own does not say where to find the objects it uses. eval() looks them up in an environment: a collection of named objects, like the global environment you work in.
e <-new.env()e$z <-3eval(quote(3+ z))
Error:
! object 'z' not found
eval(quote(3+ z), envir = e)
[1] 6
66.5.2 Quosures
A quosure (quote + closure) is an expression bundled with the environment it was created in, so it always knows where to look up its objects. enquo() and enquos() capture arguments as quosures (quo() and new_quosure() also exist, but you rarely need them). rlang::eval_tidy() evaluates a quosure in its own environment:
q <-new_quosure(expr(3+ z), env = e)q
<quosure>
expr: ^3 + z
env: 0x8c1da6740
eval_tidy(q)
[1] 6
Why does this matter? The code a user passes to your function should be evaluated where the user wrote it, so that it finds their objects, not the objects inside your function. Here, both functions create their own x, but only the quosure version uses the user’s x:
x <-100times_two_expr <-function(arg) { x <-1eval(enexpr(arg))}times_two_quo <-function(arg) { x <-1eval_tidy(enquo(arg))}
times_two_expr(x *2)
[1] 2
times_two_quo(x *2)
[1] 200
Formulas (~) work the same way: a formula is an expression plus the environment it was created in, which is why model formulas can find the variables you use in them.
66.6 The data mask
eval_tidy() can also take a data frame (or a list) as its data argument. The columns of the data are then found before the objects in the quosure’s environment. This is the data mask: the data “masks” the environment.
y was found in the data, and x in the quosure’s environment. This is how filter(penguins, species == "Gentoo") finds the species column.
66.6.1 Try 4: quosures and a data mask
Now we can make my_select() work like select() does: we evaluate each argument in a data mask where every column name stands for its position. A bare column name evaluates to its position, a number stays a number, and anything else (like a variable holding a string) is evaluated in the user’s environment:
my_select <-function(data, ...) { cols <-enquos(...) # quote what the user typed positions <-as.list(set_names(seq_along(data), names(data))) # column name = position idx <-vapply(cols, function(col) { value <-eval_tidy(col, data = positions) # evaluate in the data maskas.integer(if (is.character(value)) positions[[value]] else value) }, integer(1)) data[, idx, drop =FALSE] # regular base R}
The column wins, and we get the wrong column, with no error. This is called the data mask ambiguity: when a column and an object in our environment have the same name, the data mask picks the column.
This is exactly the quirk of pull(all_of(col)) described in Section 65.3.4.3. pull() is built on tidyselect::vars_pull(), which evaluates its argument in a data mask of column positions, just like our my_select():
body(tidyselect::vars_pull)
{
expr <- enquo(var)
if (quo_is_missing(expr)) {
cli::cli_abort("{.arg var} is absent but must be supplied.",
call = error_call)
}
local_vars(vars)
n <- length(vars)
with_chained_errors(loc <- eval_tidy(expr, set_names(seq_along(vars),
vars)), call = error_call, eval_expr = expr)
loc <- pull_as_location2(loc, n, vars, error_arg = error_arg,
error_call = error_call)
if (loc < 0L) {
loc <- n + 1L + loc
}
vars[[loc]]
}
So inside pull(all_of(col)), col is looked up in the data first, and if the data has a column called col, all_of() receives that column’s position instead of our string.
66.6.3 The .data and .env pronouns
To remove the ambiguity, eval_tidy() provides two pronouns in every data mask: .data, which only looks in the data, and .env, which only looks in the environment. They work in our my_select() too:
and in every tidyverse function that uses data masking:
df <-data.frame(x =NA, y =2)x <-100mutate(df, y = y / x)
x y
1 NA NA
mutate(df, y = y / .env$x)
x y
1 NA 0.02
.data[[var]] evaluates var in the environment, and then looks up the column with that name only in the data, so it cannot collide with a column called var. That is why .data[[col]] is the safe way to use a column name stored in a string, including in pull().
66.7 Back to the tidyverse
With these pieces, the patterns from the Defining functions in R appendix are less mysterious.
{ col } is shorthand for capturing an argument as a quosure and injecting it: !!enquo(col). These two functions are the same:
filter_penguins <-function(col, val) { col <-enquo(col)filter(penguins, !!col == val)}filter_penguins(species, "Chinstrap")
.data[[col]] uses the .data pronoun from the data mask, and all_of(cols) is how tidy selection takes a vector of names from the environment.
The talk this chapter is based on ended with a table of four cases: does the user pass a string or a bare name, and does the function you are calling need a string or a bare name? Here they are with today’s recommended tools:
The user passes
The function needs
Use
a string
a string
the string, as is: penguins[[col]]
a string
a bare name
.data[[col]] in data masking, all_of(col) in tidy selection
a bare name
a string
as_name(enquo(col)), or englue("{{ col }}") to build a longer string
a bare name
a bare name
{ col }
For example, a bare name can be turned into a string for a title or a file name:
mean_label <-function(col) {englue("Mean of {{ col }}")}mean_label(body_mass_g)
[1] "Mean of body_mass_g"
Some patterns from 2020 are no longer recommended:
aes_string() in ggplot2 is deprecated (since ggplot2 3.0.0). Use aes(x = .data[[col]]) for strings and aes(x = {{ col }}) for bare names.
Turning a string into code with rlang::parse_expr() and then injecting it with !! runs whatever the string contains as code, and fails on column names that are not valid R names:
parse_expr("body mass")
Error in `parse()`:
! <text>:1:6: unexpected symbol
1: body mass
^
Use .data[[col]] instead, which treats the string as a name and nothing else.