An R/Python script is simply a plain text file containing (almost) the same commands that you would enter into R/Python’s console or in cells/chunks in the RStudio/Juptyer notebooks. We often run these from top to bottom from the command line/unix shell.
It also makes your report files a lot cleaner!!!
You can find the code repository for this lesson here: https://github.com/DSCI-310/2024-02-13-scripts
Let’s start with a small, simple example to demonstrate how we write and run scripts in R (it is very similar in Python and we will get to this later in the lesson).
Our script will be called print_mean_hp.R, and it will calculate the mean horsepower of the cars from the built-in R data frame mtcars.
We will develop this script inside RStudio, make sure it works, and then run it from the command line/terminal/Git bash.
# author: Tiffany Timbers
# date: 2020-01-15
#
# This script calculates the mean horsepower of the cars from the built-in
# R data frame `mtcars`. This script takes no arguments.
#
# Usage: Rscript print_mean_hp.R
mean_hp <- mean(mtcars$hp)
print(mean_hp)To run our R script, we need to open the command line/terminal/Git bash, and either navigate to the directory that houses the script OR point to it when we call it. We will do the former.
Then to run the R script, we use the Rscript command, followed by the name/path to the file:
Rscript print_mean_hp.RThe output should be:
[1] 146.6875Although not necessary in R or Python, it is still good practice and advised to organize the code in your script into related sections. This practice keeps your code readable and organized. Below we outline how we typically organize R scripts:
# documentation comments
# import libraries/packages
# parse/define command line arguments here
# code for other functions
# define main function
main <- function(){
# code for "guts" of script goes here
}
# call main function
main() # pass any command line args to main hereHere we write a script called quick_titanic_fare_mean.R which reads in the titanic dataset (original source: https://biostat.app.vumc.org/wiki/Main/DataSets) and calculates the mean for the fare (ticket price) variable.
Our script has two functions, a function we defined to calculate the standard error of the mean (such a function does not exist in R) and a main function which runs the “body” of our code.
# author: Tiffany Timbers
# date: 2020-01-15
#
# This script calculates the mean for the fare (ticket price)
# from titanic.csv. This script takes no arguments.
#
# Usage: quick_titanic_fare_mean.R
library(tidyverse)
main <- function() {
data <- read_csv('data/titanic.csv')
out <- data %>%
pull(fare) %>%
mean(na.rm = TRUE)
print(out)
}
main()Above we just printed the mean to the terminal. That is was done because the purpose of that script was to have a very simple illustration of how to create and run scripts in R. However, in practice, we typically want to save our analysis artifacts (figures, tables, values, etc) to disc so that we can load them into other files (e.g., our final reports to communicate our analysis findings).
Below we show an example of how we would use readr::write_csv to save the mean value we calculated to a .csv file:
# author: Tiffany Timbers
# date: 2020-01-15
#
# This script calculates the mean horsepower of the cars from the built-in
# R data frame `mtcars` and saves it to `results/mean_hp_col.csv`.
# This script takes no arguments.
#
# Usage: Rscript print_mean_hp.R
library(readr)
main <- function() {
mean_hp <- mean(mtcars$hp)
mean_hp <- data.frame(value = mean_hp)
write_csv(mean_hp, "results/mean_hp_col.csv")
}
main()In this script we are saving the file to the results directory. There needs to be a results directory created before this script would work.
Let’s make our script more flexible, and let us specify what column variable we want to calculate the mean for when we call the script.
To do this, we use the docopt R package. This will allow us to collect the text we enter at the command line when we call the script, and make it available to us when we run the script.
When we run docopt it takes the text we entered at the command line and gives it to us as a named list of the text provided after the script name. The names of the items in the list come from the documentation. Whitespace at the command line is what is used to parse the text into separate items in the list. The list is a bit funny, in that the arguments are duplicated in the list. In the first occurrence of the arguments in the list, the names include the syntax of how you specify the argument in the docopt docs (so it can include characters like < and - and [). These make the names for the arguments non-syntactic, and thus difficult to program with in R. Thus, they are also given without those special docopt characters.
It might be easier to understand if we see an example. Consider this script named demo-docopt.R:
# author: Tiffany Timbers
# date: 2024-11-29
"This script prints out docopt args and the type of object they are stored as.
Usage: demo-docopt.R <arg1> <arg2> <arg3>
Options:
<arg1> Takes any value
<arg2> Takes any value
<arg3> Takes any value
" -> doc
library(docopt)
opt <- docopt(doc)
print(opt)
print(typeof(opt))If we run this via Rscript demo-docopt.R I LOVE MDS (so our three arguments were I, LOVE and MDS), we get this output:
List of 6
$ <arg1>: chr "I"
$ <arg2>: chr "LOVE"
$ <arg3>: chr "MDS"
$ arg1 : chr "I"
$ arg2 : chr "LOVE"
$ arg3 : chr "MDS"
NULL
[1] "list"You can see that opt (the list of command line arguments) is of length 6, even though there are only 3 command line arguments expected by the script and there were only 3 command line arguments passed to the script at the command line. The argument values are repeated, with only the names being different. The second occurence of the argument name is easier to work with because it doesn’t have the special docopt command line characters. So to reference a command line argument value, let’s say the second command line argument value in the example above, we would type opt$arg2.
We could also use numerical indexing to access the second command line argument value in the example above. To do that we could type either opt[[2]] or opt[[5]]. But this is far less readable, and so we tend to reference the command line arguments using the $name approach.
Let’s look at a more realistic example:
# author: Tiffany Timbers
# date: 2020-01-15
"This script calculates the mean for a specified column
from titanic.csv.
Usage: quick_titanic_col_mean.R <var>
" -> doc
library(tidyverse)
library(docopt)
opt <- docopt(doc)
main <- function(var) {
# read in data
data <- read_csv('data/titanic.csv')
# print out statistic of variable of interest
out <- data |>
pull(.data[[var]]) |>
mean(na.rm = TRUE)
print(out)
}
main(opt$var).data[[var]]?
docopt gives us every command line argument as a string, so var holds the string "fare", not the column fare itself. .data[[var]] means “the column of the data whose name is stored in var”. .data is a special object (called a pronoun) that tidyverse functions provide, and it stands for the columns of the data the function is working on.
The same pattern works in the tidyverse functions that compute with columns, like filter(), mutate(), summarise(), group_by() and aes() in ggplot2. Here we run it on the first five rows of titanic.csv:
library(dplyr)
data <- tibble(
sex = c("male", "female", "female", "female", "male"),
fare = c(7.25, 71.2833, 7.925, 53.1, 8.05)
)
var <- "fare"
group_var <- "sex"
data |>
group_by(.data[[group_var]]) |>
summarise(mean = mean(.data[[var]], na.rm = TRUE))# A tibble: 2 × 2
sex mean
<chr> <dbl>
1 female 44.1
2 male 7.65You may also have seen { var } in the Defining functions in R appendix. That is for bare column names passed into a function you wrote, like my_function(data, fare), and it does not work with strings from the command line: it would use the string "fare" itself rather than the column, often without an error.
all_of()
select(), rename() and relocate() choose columns rather than compute with them, and in those functions using .data is deprecated (since tidyselect 1.2.0). Use all_of() instead, as we do in the configuration file example later in this chapter:
data |>
select(all_of(c(var, "sex")))# A tibble: 5 × 2
fare sex
<dbl> <chr>
1 7.25 male
2 71.3 female
3 7.92 female
4 53.1 female
5 8.05 male The tidyverse documentation suggests all_of() for pull() too, but pull(all_of(var)) has a quirk: if the data happens to have a column with the same name as your variable (here, a column called var), pull() returns that column instead of the column named in var, without an error or a warning (tidyselect issue #376):
var <- "fare"
data_with_var <- data |>
mutate(var = 0)
# returns the column called var: the mean is 0
data_with_var |>
pull(all_of(var)) |>
mean(na.rm = TRUE)[1] 0# returns the fare column, as intended
data_with_var |>
pull(.data[[var]]) |>
mean(na.rm = TRUE)[1] 29.52166pull(.data[[var]]) does not have this problem, which is why we use it in this chapter. See Section 65.3.4 for all of these patterns with examples you can run, and the Non-standard evaluation in R appendix for why they work the way they do.
And we would run a script like this from the command line as follows:
Rscript src/quick_titanic_col_mean.R fareLet’s make our script even more flexible, and let us specify that dataset as well (we could then use it more generally on other files, such as the Gapminder .csv’s).
# author: Tiffany Timbers
# date: 2020-01-15
"This script calculates the mean for a specified column
from titanic.csv.
Usage: quick_csv_col_mean.R <file_path> <var>
" -> doc
library(tidyverse)
library(docopt)
opt <- docopt(doc)
main <- function(file_path, var) {
# read in data
data <- read_csv(file_path)
# print out statistic of variable of interest
out <- data |>
pull(.data[[var]]) |>
mean(na.rm = TRUE)
print(out)
}
main(opt$file_path, opt$var)Now we would run a script like this from the command line as follows:
Rscript src/quick_csv_col_mean.R data/titanic.csv fareIn the examples above, we used docopt to specify positional arguments. This means that the order matters! If we change the order of the values of the arguments at the command line, our script will likely throw an error, because it will try to perform the wrong operations on the wrong values.
Another downside to positional arguments, is that without good documentation, they can be less readable. And certainly the call to the script to is less readable. We can instead give the arguments names using --ARGUMENT_NAME syntax. We call these “options”. Below is the same script but specified using options as opposed to positional arguments:
# author: Tiffany Timbers
# date: 2020-01-15
"This script calculates the mean for a specified column
from titanic.csv.
Usage: quick_csv_col_mean.R --file_path=<file_path> --var=<var>
Options:
--file_path=<file_path> Path to the data file
--var=<var> Unquoted column name of the numerical vector for which to calculate the mean
" -> doc
library(tidyverse)
library(docopt)
opt <- docopt(doc)
main <- function(file_path, var) {
# read in data
data <- read_csv(file_path)
# print out statistic of variable of interest
out <- data |>
pull(.data[[var]]) |>
mean(na.rm = TRUE)
print(out)
}
main(opt$file_path, opt$var)And we would run a script like this that uses options like this:
Rscript src/quick_csv_col_mean.R --file_path=data/titanic.csv --var=fareor like this:
Rscript src/quick_csv_col_mean.R --var=fare --file_path=data/titanic.csvbecause we gave the arguments names, and thus their position no longer matters!
# documentation comments
# import libraries/packages
# parse/define command line arguments here
# code for other functions
# define main function
def main():
# code for "guts" of script goes here
# call main function
if __name__ == "__main__":
main() # pass any command line args to main hereYou can see that R and Python scripts should have roughly the same style. There is the difference of if __name__ == "__main__": in Python scripts, and R does not really have an equivalent. The benefit of some control flow around main, as is done in Python, is so that you could import or source the other functions in the script without running the main function.
Another important note is that if you are using the click package for parsing command line arguments in Python, this packages uses decorators on the main function to do this. As a consequence, these decorators need to be positioned directly above the main function definition. So, in this case the organization rules for “# parse/define command line arguments here” above is a little different.
It is still worthwhile however to have a main function in your R scripts, as it helps with organization and readability.
Although docopt for Python exists, it is not currently being supported by an active development community. Thus, we will use the click Python package instead. It is widely used, has a healthy and active development community, and excellent functionality.
Below is an example of using click for a simple Python script:
import click
@click.command()
@click.argument('num1', type=int)
@click.argument('num2', type=int)
def main(num1, num2):
"""Simple program that adds two numbers."""
result = num1 + num2
click.echo(f"The sum of {num1} and {num2} is {result}")
if __name__ == '__main__':
main()Running this script via:
python sum.py 5 7Would result in:
The sum of 5 and 7 is 12We do not need to pass the variables into main(). The click decorators take care of that for us! How nice!!!
If we instead wanted to use options in the script above, we swap the argument method for the option method and add -- to the prefix of our options:
import click
@click.command()
@click.option('--num1', type=int)
@click.option('--num2', type=int)
def main(num1, num2):
"""Simple program that adds two numbers."""
result = num1 + num2
click.echo(f"The sum of {num1} and {num2} is {result}")
if __name__ == '__main__':
main()Running this script, we now add the names of the options as shown below via:
python sum.py --num1=5 --num2=7Would result in:
The sum of 5 and 7 is 12As mentioned already, it is important you save results from your scripts so that you can import them into your reports (or other other data products). For data frame objects that will be presented as tables, writing the objects to a .csv file through readr (in R) or pandas (in Python) is great.
For figures, saving images as .png is also a good choice. Although the downstream usage of the figure can sometimes change this recommendation. For a brief but more thorough discussion of this topic, see the “Saving the visualization” chapter from Data Science: An Introduction by Timbers, Campbell & Lee (2020).
Model objects that are trained/fit in one script, and then need to be used again later in another script can and should be saved as binary files. In R, the format is .RDS and we use the functions saveRDS() and readRDS() to do this. In python, the format is .pickle and we use the functions pickle.dump() and pickle.load() from the pickle package.
saveRDS()saveRDS(final_knn_model, "final_knn_model.rds")readRDS()final_knn_model <- readRDS("final_knn_model.rds")pickle.dump()import pickle
pickle.dump(knn_preprocessor, open("knn_preprocessor.pickle", "wb"))import pickle
with open("knn_fit.pickle", 'wb') as f:
pickle.dump(knn_fit, f)pickle.load()import pickle
knn_preprocessor = pickle.load(open("knn_preprocessor.pickle", "rb"))import pickle
with open("knn_fit.pickle", 'rb') as f:
knn_fit = pickle.load(f)As your analysis grows in complexity, you may find that the number of command line arguments you need to pass to your scripts increases. If you are passing 5, 10, or even more arguments (e.g., hyperparameters for a machine learning model, multiple file paths, specific variable names), your command line calls can become extremely long, messy, and prone to typing errors.
Furthermore, from a reproducibility standpoint, it is difficult to track exactly which parameters were used for a specific analysis run if they were just typed into a terminal session that has since been closed.
This is where configuration files come in. Instead of passing many individual arguments, we pass a single configuration file containing all the necessary parameters.
| Property | Command Line Arguments | Configuration Files |
|---|---|---|
| Complexity | Best for simple scripts with few inputs (1-3) | Best for complex pipelines with many parameters |
| Frequency of Change | Good for values that change every time you run the script | Good for values that remain static across many runs |
| Reproducibility | Harder to document exactly what was run later | The file serves as a record of settings |
| Data Types | mostly limited to strings and numbers | Can handle lists, nested structures, and key-value pairs |
A common format for configuration files in data science is YAML (.yaml or .yml). It is human-readable and works well with both R and Python.
Here is an example of what a config.yaml file might look like for a script that cleans data and fits a model:
input_data: "data/raw/titanic.csv"
output_data: "data/processed/titanic_clean.csv"
model_settings:
type: "random_forest"
n_estimators: 100
max_depth: 5
keep_cols:
- "fare"
- "age"
- "pclass"To read YAML files in R, we use the yaml package. The read_yaml function reads the file and parses it into a named list, which we can then access using the $ operator, similar to how we accessed docopt arguments.
Here is how you would modify a script to accept a configuration file using docopt and yaml:
"This script cleans data based on parameters in a config file.
Usage: clean_data.R --config=<config>
Options:
--config=<config> Path to the YAML configuration file
" -> doc
library(tidyverse)
library(docopt)
library(yaml)
opt <- docopt(doc)
main <- function(config_path) {
# read in the config file
config <- read_yaml(config_path)
# Access values from the config list
raw_data <- read_csv(config$input_data)
# Example: selecting columns defined in the YAML file
clean_data <- raw_data |>
select(all_of(config$keep_cols))
print(paste("Data cleaned. Keeping columns:", paste(config$keep_cols, collapse = ", ")))
# Save the cleaned data to the path specified in the config file
write_csv(clean_data, config$output_data)
}
main(opt$config)You would run this from the command line like this:
Rscript clean_data.R --config=config.yamlIn Python, we use the PyYAML package (imported as yaml). We typically use yaml.safe_load() to read the file into a Python dictionary.
Here is how you would structure the Python equivalent using click:
import click
import yaml
import pandas as pd
@click.command()
@click.option('--config', type=click.Path(exists=True))
def main(config):
"""
Main function that loads config and runs analysis.
"""
# Load the config file
with open(config, "r") as f:
config_dict = yaml.safe_load(f)
# Access values from the dictionary
input_path = config_dict['input_data']
cols_to_keep = config_dict['keep_cols']
# Read data
raw_data = pd.read_csv(input_path)
# Example usage of config variables
clean_data = raw_data[cols_to_keep]
print(f"Data loaded from {input_path}")
print(f"Model type: {config_dict['model_settings']['type']}")
# Save the cleaned data to the path specified in the config file
clean_data.to_csv(config_dict['output_data'], index=False)
if __name__ == "__main__":
main()You would run this from the command line like this:
python clean_data.py --config=config.yamlWhen using configuration files, it is a great practice to associate a config.yaml file with the result it generated, such as by copying it into your results directory alongside your output figures and tables. This way, you have a permanent snapshot of exactly which parameters generated those specific results!