rgtlab — Ronald G. Thomas

Functional Plot Generation with purrr

Ronald G. Thomas — Mon, 08 Jun 2026 07:00:00 GMT

Functional programming tools bring order to repetitive plotting tasks.

Generating publication-quality plot grids without writing the same code three times.

Introduction

I did not really know how to programmatically generate multiple plots from grouped data until I started working with the Palmer Penguins dataset. The task seemed straightforward: produce scatter plots for every pairwise combination of numeric variables, separately for each penguin species, and then assemble them into a single coherent grid.

My first instinct was to copy-paste code for each species and each variable pair. That approach collapsed quickly once I counted the combinations: three species, four numeric variables, six pairwise combinations per species – eighteen plots total. Copy-pasting eighteen ggplot calls is tedious, brittle, and nearly impossible to maintain.

The purrr package solved this problem cleanly. By splitting the data by species and mapping a plotting function over every variable combination, I could generate all eighteen plots with a single pipeline and assemble them into a structured grid using patchwork. This post documents that workflow.

Motivations

I was tired of manually duplicating ggplot code for each subgroup in a dataset.
I wanted to see all pairwise relationships between numeric features at a glance, stratified by species.
I needed a workflow that could scale: if a fourth species appeared in the data, the code should handle it without modification.
I was curious whether purrr::pmap could handle the three-argument case (x variable, y variable, grouping variable) cleanly.
I wanted practice assembling complex multi-panel layouts with patchwork beyond simple + and / operators.

Objectives

Split a data frame by a categorical variable (species) and iterate over the resulting list with map2.
Use pmap to map a plotting function across all pairwise combinations of numeric columns.
Assemble the resulting list of plots into a structured grid using patchwork::wrap_plots with a custom design layout string.
Produce a final composite figure with species labels, shared legends, and collected axis titles.

I am documenting my learning process here. Errors and better approaches are welcome; see the contact section below.

Functional programming and visualisation.

Prerequisites and Setup

This post assumes familiarity with ggplot2 and basic tidyverse operations. The key packages are:

purrr for functional iteration (map2, pmap)
patchwork for plot composition (wrap_plots, plot_layout, plot_spacer)
palmerpenguins for the dataset
rlang for tidy evaluation (.data[[var]] pronoun)

library(palmerpenguins)
library(tidyverse)
library(purrr)
library(patchwork)
library(rlang)
library(grid)

The Palmer Penguins dataset contains measurements for 344 penguins across three species (Adelie, Chinstrap, Gentoo) from three islands in the Palmer Archipelago, Antarctica.

penguins |>
  drop_na() |>
  count(species) |>
  knitr::kable(
    caption = "Observations per species after
    removing missing values."
  )

Observations per species after removing missing values.
species	n
Adelie	146
Chinstrap	68
Gentoo	119

What is Functional Plot Generation?

Functional plot generation means treating plot creation as a function and then applying that function systematically across combinations of inputs. Instead of writing:

# Do not do this for 18 combinations
plot_adelie_bill_flipper <- ggplot(...)
plot_adelie_bill_body   <- ggplot(...)
plot_adelie_bill_depth  <- ggplot(...)
# ... 15 more times

one writes a single plotting function and lets purrr call it for every combination of species, x-variable, and y-variable. The output is a list of ggplot objects that can be arranged into any layout.

Think of it like a mail merge: the template is defined once and the data fills in the blanks.

Getting Started: Preparing the Data

The first step is to sample from the penguins data and split it into a named list by species. We also compute all pairwise combinations of the four numeric columns (bill_length_mm, bill_depth_mm, flipper_length_mm, body_mass_g).

set.seed(42)
df_sampled <- penguins |>
  sample_n(50) |>
  drop_na()

df_by_species <- split(df_sampled, df_sampled$species)

numeric_cols <- names(df_sampled)[3:6]
pairs <- t(combn(numeric_cols, 2))
colnames(pairs) <- c("x_var", "y_var")
pair_grid <- as.data.frame(pairs) |>
  mutate(group_var = "sex")

The pair_grid data frame now holds six rows, one for each pairwise combination. Each row names an x-variable, a y-variable, and a grouping variable (sex) for colour coding within each species panel.

pair_grid |>
  knitr::kable(
    caption = "All pairwise variable combinations
    to plot."
  )

All pairwise variable combinations to plot.
x_var	y_var	group_var
bill_length_mm	bill_depth_mm	sex
bill_length_mm	flipper_length_mm	sex
bill_length_mm	body_mass_g	sex
bill_depth_mm	flipper_length_mm	sex
bill_depth_mm	body_mass_g	sex
flipper_length_mm	body_mass_g	sex

Building the Plotting Function

The core of the workflow is a single function that accepts variable names as strings and a data frame, then returns a ggplot object. The .data[[var]] pronoun from rlang allows column selection by string name inside aes().

make_scatter <- function(x_var, y_var,
                         group_var, species_name,
                         df) {
  df |>
    ggplot(aes(
      x = .data[[x_var]],
      y = .data[[y_var]]
    )) +
    geom_point(
      aes(color = .data[[group_var]]),
      alpha = 0.5
    ) +
    geom_smooth(
      method = "loess",
      se = TRUE,
      linewidth = 0.7
    ) +
    scale_color_manual(
      values = c("purple", "green", "red")
    ) +
    theme_bw() +
    theme(text = element_text(size = 8))
}

Two design choices deserve explanation. First, geom_smooth(method = "loess") adds a local regression line so we can see nonlinear trends. Second, the function does not set titles (those come from the grid layout labels). Keeping the function minimal makes it reusable across different layouts.

Iterating with purpose across data strata.

Mapping Across Species and Variable Pairs

This is the heart of the approach. We use map2 to iterate over the list of species data frames and their names simultaneously. Inside that outer loop, pmap iterates over every row of pair_grid, calling make_scatter with the appropriate variable names.

all_plots <- df_by_species |>
  map2(names(df_by_species), function(df, spc) {
    pair_grid |>
      pmap(function(x_var, y_var, group_var) {
        make_scatter(
          x_var, y_var, group_var, spc, df
        )
      })
  })

The result all_plots is a nested list: the outer level has three elements (one per species), and each inner level has six elements (one per variable pair). That gives us eighteen ggplot objects total, organized by species.

cat(
  "Species:", length(all_plots), "\n",
  "Plots per species:", length(all_plots[[1]]), "\n",
  "Total plots:", sum(lengths(all_plots)), "\n"
)

Species: 3 
 Plots per species: 6 
 Total plots: 18

Assembling the Grid with Patchwork

The final step is arranging all eighteen plots into a structured grid. I want each species to occupy one row group, with a text label identifying the species. The patchwork package supports custom layout strings where each letter maps to a named plot element.

Extracting Individual Plots

First, we pull every plot out of the nested list and assign readable names.

p1  <- all_plots[[1]][[1]]
p2  <- all_plots[[1]][[2]]
p3  <- all_plots[[1]][[3]]
p4  <- all_plots[[1]][[4]]
p5  <- all_plots[[1]][[5]]
p6  <- all_plots[[1]][[6]]
p7  <- all_plots[[2]][[1]]
p8  <- all_plots[[2]][[2]]
p9  <- all_plots[[2]][[3]]
p10 <- all_plots[[2]][[4]]
p11 <- all_plots[[2]][[5]]
p12 <- all_plots[[2]][[6]]
p13 <- all_plots[[3]][[1]]
p14 <- all_plots[[3]][[2]]
p15 <- all_plots[[3]][[3]]
p16 <- all_plots[[3]][[4]]
p17 <- all_plots[[3]][[5]]
p18 <- all_plots[[3]][[6]]

Defining the Layout

The layout string below arranges plots in an upper triangular pattern for each species, with text labels (X, Y, Z) marking each species group. Each letter in the string maps to one named plot element in wrap_plots.

layout_string <- "
X##
ABC
#DE
##F
Y##
GHI
#JK
##L
Z##
MNO
#PQ
##R
"

The # characters represent empty cells. This produces a staircase pattern: three plots in the first row, two in the second, one in the third (mirroring the upper triangle of a pairwise comparison matrix).

Composing the Final Figure

We create text labels for each species using grid::textGrob and pass everything to wrap_plots.

label_sp1 <- grid::textGrob(
  names(df_by_species)[1],
  gp = gpar(fontsize = 10, fontface = "bold")
)
label_sp2 <- grid::textGrob(
  names(df_by_species)[2],
  gp = gpar(fontsize = 10, fontface = "bold")
)
label_sp3 <- grid::textGrob(
  names(df_by_species)[3],
  gp = gpar(fontsize = 10, fontface = "bold")
)

composite <- wrap_plots(
  X = label_sp1,
  A = p1,  B = p2,  C = p3,
  D = p4,  E = p5,  F = p6,
  Y = label_sp2,
  G = p7,  H = p8,  I = p9,
  J = p10, K = p11, L = p12,
  Z = label_sp3,
  M = p13, N = p14, O = p15,
  P = p16, Q = p17, R = p18,
  design = layout_string
) +
  plot_layout(
    guides = "collect",
    axis_titles = "collect"
  ) +
  theme(
    legend.position = "bottom",
    legend.direction = "horizontal",
    text = element_text(size = 8)
  )

composite

Pairwise scatter plots for four numeric penguin measurements, stratified by species. Each row group shows one species with a LOESS smooth and points coloured by sex.

The guides = "collect" argument consolidates duplicate legends into a single shared legend at the bottom. The axis_titles = "collect" argument prevents repeated axis labels across panels. Together, these two settings produce a clean, readable composite figure.

Things to Watch Out For

Variable types matter. The .data[[var]] pronoun works only when the column name is a string. Passing a symbol instead produces cryptic subscription errors.
Missing data propagates. Omitting drop_na() before splitting leaves NA rows in some species subsets, causing geom_smooth to warn about removed observations.
Layout string alignment. Every row in the patchwork layout string must have the same number of characters. A misaligned row silently produces an incorrect layout.
Global environment side effects. The original draft used assign(..., envir = .GlobalEnv) inside the plotting function. This is fragile and pollutes the workspace. Returning the plot object and storing it in a list is cleaner and more predictable.
Sample size per species. When sampling 50 rows from the full dataset, some species may end up with very few observations. For production analyses, use the full dataset or stratified sampling to ensure adequate representation.

Assembling the pieces into a coherent whole.

What Did We Learn?

Lessons Learnt

Conceptual Understanding:

Pairwise scatter plot matrices provide a rapid visual summary of relationships between continuous features, but they scale quadratically: four variables produce six pairs, five produce ten.
Stratification by species reveals within-group patterns that pooled analyses obscure (Simpson’s paradox).
LOESS smoothers are useful for initial exploration but carry no inferential interpretation without further modeling.
The upper-triangular layout avoids redundancy by showing each pair exactly once.

Technical Skills:

purrr::map2 iterates over two lists in parallel – ideal for pairing data frames with their names.
purrr::pmap generalises to arbitrary numbers of arguments by iterating over rows of a data frame.
rlang::.data[[var]] enables column selection by string name inside aes(), which is essential for programmatic ggplot construction.
patchwork::wrap_plots with a custom design string provides fine-grained control over multi-panel layouts.

Gotchas and Pitfalls:

Forgetting drop_na() before split() produces species-level data frames with missing rows that cause warnings in geom_smooth.
The plot_spacer() function is useful for empty cells, but the # character in the design string is more readable for fixed layouts.
assign() to .GlobalEnv inside mapped functions creates hard-to-debug side effects. Always return values from functions and collect them in lists.
scale_color_manual requires the correct number of colour values matching the levels of the grouping variable.

Limitations

The sample of 50 penguins is small. Some species may have fewer than 10 observations, making LOESS fits unreliable.
The grouping variable (sex) is hard-coded. A more general approach would accept the grouping variable as a parameter.
The layout string is manually crafted for exactly three species and six variable pairs. Adding a fourth species or fifth numeric variable requires rewriting the layout.
No statistical tests accompany the visual exploration. The plots show associations but do not quantify significance or effect size.
The colour palette (purple, green, red) is not colourblind-friendly. Production code should use a palette validated for accessibility.

Opportunities for Improvement

Replace the hard-coded layout string with a function that generates the design string dynamically based on the number of species and variable pairs.
Add correlation coefficients as text annotations inside each scatter plot panel.
Use GGally::ggpairs as a comparison point – it handles pairwise plots natively, though with less layout flexibility.
Implement stratified sampling with dplyr::slice_sample(n, by = species) to ensure balanced representation.
Replace the manual plot extraction (p1 through p18) with a programmatic approach using list_flatten and named assignment.
Switch to a colourblind-friendly palette such as viridis or the Okabe-Ito palette.

Wrapping Up

This post demonstrated how purrr::map2 and purrr::pmap can replace copy-pasted ggplot code with a single functional pipeline. The key insight is that plots are objects that can be stored in lists, passed to functions, and assembled programmatically.

The approach worked well for the Palmer Penguins data. Starting from a single plotting function and a table of variable combinations, we generated eighteen scatter plots and arranged them into a structured grid that reveals within-species patterns. The total code is compact enough to fit in a single script, yet flexible enough to adapt to new datasets.

When working with grouped data and finding repetitive ggplot code, this pattern is worth trying:

Split the data by your grouping variable.
Define a single plotting function that accepts column names as strings.
Map the function across all combinations using pmap.
Assemble the resulting list with patchwork.

Reproducibility

This analysis uses the palmerpenguins built-in dataset and requires no external data files. To reproduce:

quarto render index.qmd

Session information:

sessionInfo()

R version 4.5.3 (2026-03-11)
Platform: aarch64-apple-darwin25.3.0
Running under: macOS Tahoe 26.5

Matrix products: default
BLAS:   /opt/homebrew/Cellar/openblas/0.3.32/lib/libopenblasp-r0.3.32.dylib 
LAPACK: /opt/homebrew/Cellar/r/4.5.3/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.1

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Los_Angeles
tzcode source: internal

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
 [1] rlang_1.2.0          patchwork_1.3.2      lubridate_1.9.5     
 [4] forcats_1.0.0        stringr_1.5.1        dplyr_1.2.1         
 [7] purrr_1.1.0          readr_2.2.0          tidyr_1.3.1         
[10] tibble_3.3.1         ggplot2_4.0.0        tidyverse_2.0.0     
[13] palmerpenguins_0.1.1

loaded via a namespace (and not attached):
 [1] Matrix_1.7-4       gtable_0.3.6       jsonlite_2.0.0     compiler_4.5.3    
 [5] tidyselect_1.2.1   parallel_4.5.3     splines_4.5.3      scales_1.4.0      
 [9] yaml_2.3.10        fastmap_1.2.0      lattice_0.22-9     R6_2.6.1          
[13] labeling_0.4.3     generics_0.1.4     knitr_1.50         htmlwidgets_1.6.4 
[17] pillar_1.11.1      RColorBrewer_1.1-3 tzdb_0.5.0         stringi_1.8.7     
[21] xfun_0.56          S7_0.2.0           timechange_0.4.0   cli_3.6.6         
[25] mgcv_1.9-4         withr_3.0.2        magrittr_2.0.5     digest_0.6.37     
[29] hms_1.1.4          nlme_3.1-168       lifecycle_1.0.5    vctrs_0.7.3       
[33] evaluate_1.0.5     glue_1.8.0         farver_2.1.2       rmarkdown_2.29    
[37] tools_4.5.3        pkgconfig_2.0.3    htmltools_0.5.8.1

Let’s Connect

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: rgtlab.org/contact

I would enjoy hearing from you if:

An error or a better approach to any of the code in this post comes to mind.
There are suggestions for topics to see covered.
The interest is in discussing R programming, data science, or reproducible research.
There are questions about anything in this tutorial.
The goal is simply to say hello and connect.

Setting Up Multi-Language Quarto Documents on macOS

Ronald G. Thomas — Mon, 08 Jun 2026 07:00:00 GMT

Quarto logo: the open-source publishing system that ties multiple languages together in a single document.

Quarto provides a unified authoring framework for scientific and technical publishing.

Introduction

The plumbing behind a multi-language Quarto document is more extensive than the Quarto website suggests. The documentation makes it look effortless: drop an R chunk here, a Python chunk there, maybe some Julia for good measure, and hit Render. The reality involves an afternoon of chasing PATH variables, fixing reticulate configurations, and troubleshooting why Julia refuses to find its own packages.

The problem is not that any single language is hard to install. Each one has a clean installer and decent documentation. The difficulty is making them all coexist inside a single rendering pipeline where R orchestrates Python through reticulate, Julia through JuliaCall, and Observable JS runs natively in the browser. Small misconfigurations in any layer can cascade into cryptic errors.

We document here the lessons learned getting all four languages to cooperate in a single Quarto document on macOS, written from the perspective of working through the process rather than having mastered it.

More formally, this post documents the polyglot capability of the Document layer of the Workflow Construct described in post 52. Quarto is the construct’s recommended document layer, subsuming the roles previously divided between R Markdown, Jupyter, and standalone LaTeX. This post addresses the specific case in which a single document spans R, Python, Julia, and Observable, which is the configuration that surfaces the most layer-interaction subtle failures.

Motivations

The following considerations motivated this exploration:

Comparing plotting libraries across languages side by side, using the same dataset, in a single rendered document.
Collaborators who work in Python need a way to view analyses that mix R and Python code without installing R themselves.
Julia’s speed for numerical computation makes it worth exploring, particularly when called from inside an R session.
Observable JS examples in Quarto documentation suggest useful integration patterns worth understanding.
Documenting the setup process forces understanding of each component rather than blindly following a tutorial.
A reproducible multi-language environment can be reused across future projects.

Objectives

Install and configure R, Python, Julia, and Observable JS to work within a single Quarto document on macOS.
Verify each language integration independently before combining them.
Render a minimal multi-language Quarto document that executes code in all four languages.
Document common setup pitfalls and their solutions so future attempts take minutes rather than hours.

Errors and better approaches are welcome; see the Feedback section at the end.

A workspace ready for multi-language exploration.

Prerequisites and Setup

Before starting, the following should be available on the macOS system:

macOS (tested on Monterey and later)
At least 8 GB RAM (each language runtime consumes memory; running all four simultaneously is demanding)
Roughly 5 GB free disk space for all installations
Admin privileges for Homebrew and installer packages
Internet connection for downloading packages

Background. This guide assumes familiarity with the macOS Terminal and basic comfort installing software from the command line. Prior experience with all four languages is not required; that is the whole point.

What is Multi-Language Quarto?

Quarto is an open-source scientific publishing system that extends the idea behind R Markdown to multiple languages. A single .qmd file can contain code chunks written in R, Python, Julia, and Observable JS. When the document is rendered, Quarto routes each chunk to the appropriate language engine, collects the output, and weaves everything into a unified HTML, PDF, or Word document.

The key insight is that R acts as the orchestration layer. Python chunks are executed through the reticulate package, Julia chunks through JuliaCall, and Observable JS chunks run natively in the browser when producing HTML output. This means the R installation needs to know where Python and Julia live on the system, and each language needs its own packages installed separately.

Getting Started: Basic Setup

The setup proceeds in four stages. Completing and verifying each stage before moving to the next is recommended. Debugging a four-language environment is much harder than debugging a single-language problem.

Stage 1: R and RStudio

Install R from CRAN
Install RStudio
Install Quarto

Then install the R packages that bridge to other languages:

install.packages(c(
  "dplyr",
  "ggplot2",
  "scales",
  "reticulate",
  "JuliaCall"
))

The first three packages are for data work. The last two are the bridge packages: reticulate connects R to Python, and JuliaCall connects R to Julia.

Stage 2: Python

Install Anaconda (recommended) or Miniconda
Install required Python packages:

conda install pandas seaborn matplotlib

Verify Python configuration in R:

library(reticulate)
py_config()

The output of py_config() should show your Anaconda Python path. If it does not, you need to tell reticulate where to look.

Common Python Issues

If py_config() does not show your Anaconda installation, specify it explicitly:
```
use_python("/path/to/anaconda/bin/python")
```
Seaborn styling requires specific syntax:
```
sns.set_theme(style="whitegrid")
```
Do not use plt.style.use() for Seaborn themes.

Stage 3: Julia

Install Julia from julialang.org
Add Julia to PATH (usually automatic with the installer)
Install required Julia packages:

using Pkg
Pkg.add([
  "UnicodePlots",
  "DataFrames",
  "CSV",
  "Statistics"
])

Set up the Julia-R connection:

library(JuliaCall)
julia_setup()

Julia Gotchas

Some Julia plotting backends have system dependencies that may not be present on macOS by default.
Use UnicodePlots for the most reliable terminal-based results.
Ensure the Julia working directory has write permissions.

Stage 4: Observable JS

No separate installation is needed. Observable JS runs in the browser when Quarto produces HTML output. Your Quarto document must include the dependency declaration in the YAML header:

dependencies:
  - name: "@observablehq/plot"
    version: latest

That is it. Observable chunks will render automatically when you build to HTML.

Python logo: the second language engine in the multi-language Quarto setup.

Python integration through reticulate is often the first multi-language feature Quarto users encounter.

Verifying Each Language

Before combining all four languages in a single document, verify that each integration works independently. This saves considerable debugging time.

R Environment Check

sessionInfo()

This confirms your R version and loaded packages. Look for the version number and ensure the base packages are listed.

Python Environment Check

library(reticulate)
py_config()

Confirm that the Python path points to your Anaconda installation, not the system Python. The output should show the version number and the path to the Python binary.

Julia Environment Check

library(JuliaCall)
julia_eval("versioninfo()")

This should print the Julia version and system information. If JuliaCall cannot find Julia, check that the Julia binary is on your PATH.

Testing a Minimal Example

Once all four verifications pass, test a minimal multi-language document. Create a new .qmd file with these chunks:

Code

Rapid Conversion of Draft R Scripts to Formal Rmd Reports

Ronald G. Thomas — Mon, 08 Jun 2026 07:00:00 GMT

A structured workflow transforms messy R scripts into polished analytical reports

Introduction

I did not really know how to quickly convert a working R script into a presentable report until I ran into a situation where a collaborator needed results the same afternoon. The analysis was complete, the code was tested, but the output was a tangled mix of console printouts and saved PNG files scattered across a directory. Assembling those fragments into a coherent document took longer than the analysis itself.

Three things many R developers are often reluctant to tackle: code documentation, testing, and report preparation. Of these, report preparation tends to be the most urgent. When a supervisor or reviewer asks for results, they rarely want a raw R script. They want tables, figures, narrative context, and a document they can read without opening RStudio.

We walk through several approaches to bridging the gap between a working R script and a finished R Markdown report. The goal is practical: reduce the friction between “my code works” and “here is a document someone can read.”

More formally, this post documents the script-to-document promotion path in the Document layer of the Workflow Construct described in post 52. The Document layer holds the artefact a reader sees (a rendered HTML or PDF report); the script-to-document promotion is the operation that lifts a working R script onto that layer without rewriting the analysis. This is the most frequent project-tier transition for an applied biostatistician and is the specific concern the post addresses.

Motivations

I had a folder of working R scripts that produced correct results but existed only as console output and disconnected figure files.
Collaborators asked for formatted reports repeatedly, and each time I rebuilt the narrative from scratch rather than converting what already existed.
I wanted a systematic approach that would take minutes, not hours, to move from script to document.
I needed to preserve the exact code that generated the results, not rewrite it for presentation purposes.
I was curious whether knitr::spin() could genuinely replace manual chunk insertion for simple analytical scripts.

Objectives

Understand the knitr::spin() function and how it converts annotated R scripts directly into R Markdown documents.
Compare the spin() approach against manual conversion (adding YAML headers and wrapping code in chunks by hand).
Use RStudio’s built-in “Compile Report” feature to generate quick reports without modifying the original script.
Develop a reusable template workflow for consistent formatting across multiple script-to-report conversions.

I am documenting my learning process here. Errors and better approaches are welcome; see the contact section.

Settling in for a focused coding session.

Prerequisites and Setup

The techniques in this post require a standard R installation with the knitr and rmarkdown packages. RStudio is recommended for the “Compile Report” feature but is not strictly necessary for the command-line workflows.

Background: This post assumes familiarity with writing R scripts and a basic understanding of what R Markdown documents look like. No prior experience with knitr::spin() is needed.

install.packages(c("knitr", "rmarkdown"))

library(knitr)
library(rmarkdown)

What is Script-to-Report Conversion?

Script-to-report conversion is the process of transforming a plain R script (a .R file containing code, comments, and possibly some inline output) into a formatted document that weaves together code, results, and narrative text. The output is typically an HTML page, PDF, or Word document that a non-programmer can read without needing to run any code.

Think of it as the reverse of the usual R Markdown workflow. Instead of starting with a .qmd or .Rmd file and embedding code chunks, one starts with a working .R file and adds just enough structure to produce a readable report. The key insight is that existing comments can serve as the narrative, and existing code already produces the figures and tables needed.

Getting Started: The Problem with Raw Scripts

Consider a typical analytical R script. It loads data, fits a model, prints a summary, and saves a plot. The code works perfectly, but the output is fragmented: summary statistics appear in the console, the plot is saved to disk, and there is no unified document tying them together.

library(ggplot2)

data(mtcars)
mtcars$cyl <- as.factor(mtcars$cyl)

summary(mtcars[, c("mpg", "wt", "hp")])

model <- lm(mpg ~ wt + hp, data = mtcars)
summary(model)

ggplot(mtcars, aes(x = wt, y = mpg, color = cyl)) +
  geom_point(size = 3) +
  geom_smooth(method = "lm", se = FALSE) +
  theme_minimal() +
  labs(
    title = "Fuel Efficiency by Weight",
    x = "Weight (1000 lbs)",
    y = "Miles per Gallon"
  )
ggsave("mpg_by_weight.png", width = 8, height = 5)

This script runs correctly, but sharing its results requires manually assembling the console output, the saved figure, and some written context into a separate document. The three approaches described below solve this problem at different levels of effort and control.

Transitioning from raw scripts to structured reports

Approach 1: `knitr::spin()` (The Fastest Path)

The knitr::spin() function converts a specially annotated R script directly into an R Markdown document (and optionally renders it in one step). The key is a lightweight annotation syntax that uses R comments to embed narrative text and chunk options without restructuring the script.

How spin() Annotation Works

In a spin-compatible R script, three comment styles carry special meaning:

#' (hash-apostrophe) marks lines as narrative Markdown text.
#- (hash-hyphen) starts a new unnamed code chunk.
#+ label, option=value (hash-plus) starts a named chunk with explicit options.

Everything else is treated as R code inside the current chunk. This means an existing script can be annotated with minimal changes.

Annotating an Existing Script

Here is the same analysis script rewritten with spin annotations. The original code is unchanged; only comments have been modified.

#' ---
#' title: "Fuel Efficiency Analysis"
#' author: "Ronald G. Thomas"
#' date: today
#' output: html_document
#' ---
#'
#' # Overview
#'
#' This report examines the relationship between
#' vehicle weight, horsepower, and fuel efficiency
#' using the mtcars dataset.

#+ setup, message=FALSE
library(ggplot2)

#' # Data Preparation

data(mtcars)
mtcars$cyl <- as.factor(mtcars$cyl)

#' # Summary Statistics
#'
#' The dataset contains 32 observations across
#' three engine configurations.

summary(mtcars[, c("mpg", "wt", "hp")])

#' # Linear Model
#'
#' We fit a multiple regression predicting miles
#' per gallon from weight and horsepower.

model <- lm(mpg ~ wt + hp, data = mtcars)
summary(model)

#' # Visualization
#'
#' The scatterplot below shows the relationship
#' between weight and fuel efficiency, colored
#' by cylinder count.

#+ fig-mpg, fig.width=8, fig.height=5
ggplot(mtcars, aes(x = wt, y = mpg, color = cyl)) +
  geom_point(size = 3) +
  geom_smooth(method = "lm", se = FALSE) +
  theme_minimal() +
  labs(
    title = "Fuel Efficiency by Weight",
    x = "Weight (1000 lbs)",
    y = "Miles per Gallon"
  )

Running spin()

To convert and render in one step:

knitr::spin("analysis_script.R", knit = FALSE)

Setting knit = FALSE produces the .Rmd file without rendering it, which is useful for inspecting or editing the intermediate Markdown before producing the final document. To generate the report directly:

knitr::spin("analysis_script.R", knit = TRUE)

The function also accepts a format argument for producing different output types:

rmarkdown::render(
  knitr::spin("analysis_script.R", knit = FALSE),
  output_format = "pdf_document"
)

Approach 2: Manual Conversion

Manual conversion involves creating a new .Rmd file and migrating code from the R script into fenced code chunks. This approach provides the most control over document structure but requires the most effort.

The Conversion Steps

The process follows a predictable sequence:

Create a new .Rmd file with a YAML header.
Copy code from the .R file into fenced code chunks (```{r} ... ```).
Convert existing comments into narrative Markdown text outside the chunks.
Add chunk options for figure sizing, echo control, and caching.
Render the document with rmarkdown::render() or the RStudio Knit button.

A Minimal Template for Manual Conversion

The following template provides a starting point that covers the most common needs:

Rmd template structure

Install Linux Mint on a MacBook Air

Ronald G. Thomas — Mon, 08 Jun 2026 07:00:00 GMT

A 13-inch MacBook Air running Linux Mint, with an installer USB drive alongside

Linux Mint: a polished desktop experience on repurposed Apple hardware.

Introduction

Many users assume that installing Linux on Apple hardware requires fighting obscure driver issues: wifi cards that refuse to connect, trackpads that stutter, and suspend modes that brick the machine. This assumption, while once valid, has become largely outdated. Modern distributions like Linux Mint have matured to the point where installation on older MacBooks is surprisingly painless.

The specific goal of this project is to refurbish a 2016 13-inch MacBook Air with a modern Linux operating system and configure it as a functional data science workstation. The distribution we focus on is Linux Mint 22 (“Wilma”), chosen primarily because it ships with the hardware drivers needed for this particular MacBook, making the installation far more straightforward than most alternatives.

We walk through every step of the process: from downloading the ISO image and burning it onto a USB drive, through the installation itself, to post-install configuration of the shell, keyboard, display, and essential software. Common stumbling blocks are documented throughout so that readers can avoid them.

More formally, this post documents the Hardware and Operating System layers (Layers 1 and 2) of the Workflow Construct described in post 52. The ThinkPad-with-Mint pairing is the construct’s Linux side; it mirrors the macOS-with-MacBook side and provides the cross-platform substrate on which the higher layers (shell, editor, container runtime) are identical. The installation walked through here is the bootstrapping that brings a new ThinkPad to the point where the dotfiles repository’s install.sh (see post 24) can run.

Motivations

The following considerations motivated this project:

A functional MacBook Air sitting unused because macOS updates had slowed it to a crawl represented wasted hardware that could be given a second life.
Data science workflows that rely on R, Python, and Docker benefit from a lightweight machine that can run all three without the overhead of a modern macOS installation.
Curiosity about whether Linux Mint could genuinely replace macOS for users accustomed to Apple’s trackpad gestures and keyboard shortcuts.
Setting up a reproducible development environment from scratch is a useful exercise in understanding what dependencies a workflow actually requires.
A portable secondary machine for teaching demonstrations, where the full software stack is visible and inspectable rather than hidden behind proprietary layers.

Objectives

Download, verify, and burn the Linux Mint 22 ISO image onto a bootable USB drive.
Install Mint on a 2016 MacBook Air, including handling wifi, trackpad, and display drivers.
Configure the desktop environment, keyboard shortcuts, and shell (zsh) for a data science workflow.
Install and validate the core software stack: R, vim, Docker, Dropbox, and essential command-line utilities.

Errors and alternative approaches are welcome; see the Let’s Connect section at the end.

A workspace ready for a fresh OS installation.

Prerequisites and Setup

Before beginning, gather the following:

Target machine: A 2016 13-inch MacBook Air (Early 2015 hardware, MacBookAir7,2; one Thunderbolt 2 port, two USB-A ports, MagSafe 2 power). Other MacBook models may work but driver support varies.
Working machine: Any computer with internet access and a USB port, used to download the ISO and burn the USB drive.
USB flash drive: At least 4 GB capacity (the ISO is approximately 3 GB).
External wifi adapter (if needed): A Panda Wireless USB adapter is a reliable fallback. Mint 22 ships with Ralink RT5372 drivers, which support this adapter out of the box.
Ethernet cable (optional but recommended): Wired connectivity during initial setup avoids wifi driver issues entirely.

What is Linux Mint?

Linux Mint is a community-driven Linux distribution built on top of Ubuntu (which is itself built on Debian). It emphasises ease of use and ships with the Cinnamon desktop environment, which provides a traditional desktop metaphor: a taskbar, a start menu, and a system tray. For users migrating from macOS or Windows, Cinnamon feels immediately familiar.

The distinguishing feature of Mint, relative to Ubuntu or Fedora, is its focus on hardware compatibility and out-of-the-box usability. The Mint team bundles proprietary multimedia codecs, common drivers, and a curated set of default applications so that the system is functional immediately after installation. Since the beginning of the Linux era (circa 1991), the central challenge of installing a Linux distribution on consumer hardware has been wrestling with video, audio, trackpad, and power management drivers. Mint addresses this directly.

Getting Started: Installation

Download and Verify the ISO

Download the torrent file for Linux Mint 22 Wilma Cinnamon edition from the official Mint website:

linuxmint-22-cinnamon-64bit.iso.torrent

Install the macOS app Transmission and add the torrent file. Also download the associated sha256sum.txt file.

To verify the integrity of the downloaded ISO, generate its SHA256 checksum and compare it to the published hash:

sha256sum -b linuxmint-22-cinnamon-64bit.iso

Compare the output against the contents of sha256sum.txt. If the hashes match, the download is intact.

Burn the ISO to USB

Transfer the ISO file to a USB flash drive using balenaEtcher, a cross-platform tool that writes disk images reliably. Insert the USB drive, select the ISO in balenaEtcher, and click “Flash.”

Boot from USB and Install

Insert the bootable USB flash drive into the target MacBook and reboot. Hold the ALT key during startup to access the boot drive selection screen. Select the icon for the USB drive.

A GRUB menu will appear.¹

¹ GNU GRand Unified Bootloader (GRUB): When a Linux operating system starts, GRUB is the first program that runs. It loads the kernel, which in turn loads the shell, the desktop environment, and other system services. codecademy.com

From the GRUB menu, choose Start Linux Mint 22 Cinnamon 64-bit. A Mint desktop will appear, allowing you to test-drive the system or proceed with installation by double-clicking the “Install Linux Mint” icon.

Installation Dialog

The setup dialog walks through the following screens in sequence:

Language: English (or your preference).
Network: If ethernet is available, Mint connects automatically. Wifi can be configured later; skip this step if only wireless is available.
Multimedia codecs: Check the box to install multimedia codecs.
Installation type: Choose “Erase disk and install Linux Mint.”
Location: Select your timezone (e.g., Los Angeles).
User account: Create an administrator username, assign a password, and set a hostname.

The installation proceeds without further input.

Post-Install Connectivity

When the installation completes, connect the target machine to the internet. If you have ethernet, plug the cable directly into the MacBook; Mint will connect automatically. For wireless access, Mint may or may not recognise the internal wifi hardware. If it does not, use a supported external adapter such as the Panda Wireless USB modem (supported via the Ralink RT5372 drivers bundled with Mint 22).

Optionally, connect a second monitor through a Mini DisplayPort adapter into the MacBook’s Thunderbolt 2 port.

Reboot and log in with the administrator credentials you created during installation. The base system is ready.

A container-based workflow is one of the first things to configure on a fresh Linux installation

Building the software stack: from bare metal to a configured development environment.

Configuring the Desktop

Keyboard and Trackpad

Open Mouse and Touchpad in system settings and enable Reverse scroll (natural scrolling for macOS converts).

Open Keyboard > Layouts > Options > Caps Lock behavior and select Swap Esc and Caps-Lock. This is an important setting for vim users.

Open Shortcuts > Windows and configure:

Maximize window: Super-f
Unmaximize window: Super-g
Close window: Super-q
Move window to other monitor: Shift-Super-UpArrow

Display Settings

On a two-monitor system, open the Display settings (press the Super key and search for “display”). Set the external monitor as the primary display at its native resolution (e.g., 1920x1200 on a 24-inch Dell UltraSharp). Set Monitor scale to 150% to increase the default font size in applications. The MacBook’s internal panel is 1440x900 native and serves as the secondary display. A 4K external monitor at 3840x2160 can be substituted for the 24-inch primary.

Power Management

Switch the “suspend on timeout” behaviour to “shut down immediately” in power settings. This avoids system lock issues on lid close or idle timeout, which can be problematic on older MacBook hardware.

Installing the Software Stack

Copy Configuration Files from Host

Connect to the new Mint machine via SSH from an existing workstation:

# On the Mint machine
sudo apt install openssh-server
ip route

Note the IP address from the ip route output (e.g., 10.0.1.196).

From the host machine, copy essential dotfiles:

# From the Mac
scp ~/Dropbox/dotfiles/kickstart/* z@10.0.1.196:~

Key configuration files to transfer:

.vimrc: vim configuration
.zshrc: zsh shell configuration
.vim/ directory: contains plug and ultisnips

Optionally, copy a test workspace to verify the rendering pipeline later:

scp ~/work/teaching/fmph243b/project1 z@10.0.1.196:~

Install Core Utilities

Update the package listings and install the essential tool chain:

sudo apt update
sudo apt upgrade -y
sudo apt install \
  r-base-core terminator eza tree zsh git vim \
  fzf ripgrep autojump zsh-autosuggestions \
  zathura -y

Set Up the Shell

Make zsh the default shell:

chsh -s $(which zsh)

Configure zsh following the companion post on shell setup (see the “See Also” section below for the link).

Set Up Symbolic Links

Run the dotfile linking script to connect the home directory to configurations stored in Dropbox:

~/Dropbox/dotfiles/set_up_links.sh

The script performs the following:

#!/bin/zsh

# Move the install-created .config temporarily
mv ~/.config ~/.config.tmp

# Link dotfiles from Dropbox to Home
ff=(".zshrc" ".viminfo" ".vimrc" ".local" ".vim" \
    ".vimplugins" ".config" ".Rprofile")
for P in "${ff[@]}"
do
echo "Linking Dropbox version of $P to Home"
    ln -v -s "$HOME/Dropbox/dotfiles/$P" "$HOME/$P"
done

# Restore original .config contents into the linked dir
cp -R ~/.config.tmp/* ~/.config
rm -rf ~/.config.tmp

# Link working directories from Dropbox
dd=("sandbox" "bin" "docs" "prj" "work" "ssh" "shr")
for P in "${dd[@]}"
do
    echo "Linking $P from Dropbox to Home"
    ln -v -s "$HOME/Dropbox/$P" "$HOME/$P"
done

Alternative shortcut: Install Dropbox first (see below), then run install_app.sh and set_up_links.sh from ~/Dropbox/dotfiles/ while Dropbox syncs in the background.

Install Additional Applications

Install Zotero using the Software Manager. Set up Zotero syncing with the relevant account credentials.

Add the vimium extension to Firefox for keyboard-driven browsing.

Install Dropbox

sudo apt install nautilus-dropbox
dropbox autostart y
dropbox start -i

The Dropbox startup process launches a browser-based sign-in page. Log in with the appropriate Dropbox credentials.

Install Docker

Installing Docker on Mint requires adding Docker’s GPG key and repository to the Apt sources, because the Docker packages in the default Mint repositories are outdated:

sudo apt update
sudo apt install \
  apt-transport-https ca-certificates curl gnupg
curl -fsSL \
  https://download.docker.com/linux/ubuntu/gpg \
  | sudo gpg --dearmor \
  -o /usr/share/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) \
  signed-by=/usr/share/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  noble stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list \
  > /dev/null
sudo apt update
sudo apt install \
  docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin
sudo systemctl is-active docker

Reference: How to Install Docker on Linux Mint 21 (linuxiac.com). The linked guide targets Mint 21 (Ubuntu 22.04 “jammy”); for Mint 22 on Ubuntu 24.04, substitute the noble codename in the repository line as shown above.

Checking Our Work

The system should now be able to render both .Rmd and .qmd files. A good test is to render a known-working document:

cd ~/prj/setupmint
quarto render index.qmd --to pdf

If this fails, walk through the dependency chain systematically. The troubleshooting sequence below captures common errors encountered and how to resolve each one.

Troubleshooting the Rendering Pipeline

Starting with a sample peng1.Rmd file, the rendering was attempted from the command line. Each attempt surfaced the next missing dependency.

Attempt 1: R -e "render('peng1.Rmd')" Error: Command R not found. Fix: sudo apt install r-base-core

Attempt 2: R -e "render('peng1.Rmd')" Error: could not find function "render". Fix: install.packages("rmarkdown") in R.

Attempt 3: R -e "library(rmarkdown); render('peng1.Rmd')" Error: pandoc version 1.12.3 or higher required. Fix: Install pandoc via apt or Quarto (which bundles its own pandoc).

Attempt 4: R -e "library(rmarkdown); render('peng1.Rmd')" Error: there is no package called 'pacman'. Fix: install.packages("pacman") in R.

Attempt 5: R -e "library(rmarkdown); render('peng1.Rmd')" Error: could not find preamble.tex. Fix: The file path used a macOS-style path (/Users/zenn/shr/preamble.tex). Transfer the file and update the path to the Linux equivalent.

Attempt 6: R -e "library(rmarkdown); render('peng1.Rmd')" Error: pdflatex not found. Fix: Install TinyTeX from within R:

install.packages("tinytex")
tinytex::install_tinytex()

Attempt 7: R -e "library(rmarkdown); render('peng1.Rmd')" Error: file sudoku_apple.pdf not found. Fix: Transfer the missing logo file from the host machine.

Attempt 8: R -e "library(rmarkdown); render('peng1.Rmd')" Error: no bibliography file found. Fix: Transfer the .bib file and update the path in the YAML header.

Attempt 9: R -e "library(rmarkdown); render('peng1.Rmd')" Minor errors from packages that failed to load via pacman. Removed janitor, kableExtra, tidyverse, readxl from the preamble and added ggplot2 directly.

Attempt 10: Success. The document rendered to PDF without errors.

We note that every rendering failure pointed to a specific missing dependency. Addressing them one at a time is tedious but educational; each fix clarifies what a workflow actually requires.

Things to Watch Out For

Wifi drivers are the first hurdle. If the internal wifi card is not recognised, do not waste time debugging; plug in ethernet or use a Panda Wireless USB adapter and move on.
macOS file paths break on Linux. Any configuration file, .Rmd header, or script that references /Users/ will fail. Search and replace all paths after migrating files.
Suspend-on-lid-close can lock the system. On older MacBooks, the safest power management setting is to shut down on lid close rather than suspend.
Docker on Mint is not straightforward. The default Mint repositories do not include current Docker packages. Docker’s official GPG key and repository must be added manually.
Package installation in R is slower on Linux. R packages compile from source on Linux (unlike macOS and Windows, which use pre-built binaries). Expect the first install.packages() call to take significantly longer.

Daily Workflow

Once the system is configured, these commands become routine:

Command	Action
`quarto render doc.qmd --to pdf`	Render a Quarto document to PDF
`Rscript -e "rmarkdown::render('file.Rmd')"`	Render an R Markdown report
`docker compose up -d`	Start a containerised analysis environment
`make r`	Enter zzcollab Docker container
`dropbox status`	Check Dropbox sync progress
`sudo apt update && sudo apt upgrade`	System update

Uninstall / Rollback

To restore macOS on the MacBook:

Create a macOS bootable USB installer from another Mac using createinstallmedia.
Boot from the USB (hold Option at startup).
In Disk Utility, erase the entire disk (APFS format).
Install macOS from the USB installer.

This is a full disk wipe. Back up any Linux-side files (e.g., ~/prj/, configuration files) to external storage or Dropbox before proceeding.

A well-organised workspace makes the difference between a system you use and a system you abandon

From installation to a configured, productive environment.

What Did We Learn?

Lessons Learnt

Conceptual Understanding:

Linux Mint’s value proposition is not novelty; it is that the hard work of driver integration has already been done for common hardware configurations.
The Debian/Ubuntu package ecosystem (apt) is remarkably comprehensive. Nearly every tool required for data science workflows is a single apt install away.
A fresh Linux installation reveals every implicit dependency in your workflow that macOS had been hiding behind pre-installed frameworks.
Hardware compatibility remains the single most important factor when choosing a Linux distribution for physical (non-virtual) installation.

Technical Skills:

Verifying ISO integrity using SHA256 checksums before installation is a practice often skipped but worth adopting.
The scp and ssh workflow for bootstrapping a new machine from an existing one is efficient and repeatable.
Symbolic linking dotfiles from a cloud-synced directory (Dropbox) to the home directory creates a portable configuration that survives reinstallation.
Managing Docker on Mint requires understanding GPG keys and APT repository configuration at a level that macOS package managers abstract away.

Gotchas and Pitfalls:

The order of operations matters: install Dropbox early so that dotfile linking scripts can reference ~/Dropbox/ paths immediately.
R packages compile from source on Linux, which means system-level dependencies (libcurl4-openssl-dev, libxml2-dev) must be installed before certain R packages will build.
The chsh command requires a logout/login cycle to take effect; do not assume zsh is active immediately after running it.
TinyTeX installation from within R does not always add pdflatex to the system PATH. Verify with which pdflatex after installation.

Limitations

This guide is specific to the 2016 13-inch MacBook Air (Early 2015 hardware, Thunderbolt 2 port). Other MacBook models, especially those with the T2 security chip (2018 and later), may require additional steps or may not work at all.
Wifi driver support depends on the specific wireless chipset. The Panda Wireless workaround is reliable but adds an external dependency.
The guide assumes a complete disk erasure. Dual-boot configurations (macOS alongside Mint) are not covered and involve additional partitioning complexity.
Battery life on Linux is typically shorter than on macOS for the same hardware, because Apple’s power management optimisations are proprietary and not available to Linux kernels.
The Docker installation instructions reference the Ubuntu “noble” repository (Ubuntu 24.04), which matches the Mint 22.x base. Future Mint releases built on a newer Ubuntu base will require substituting the new codename.
Software Manager availability varies. Some applications (e.g., Zotero) may require manual installation from the vendor’s website rather than through the Mint Software Manager.

Opportunities for Improvement

Automate the entire post-install setup with a single shell script that installs all packages, configures the shell, sets up symbolic links, and installs Docker in one pass.
Use Ansible or a similar configuration management tool to make the setup reproducible across multiple machines without manual intervention.
Investigate the ubuntu-drivers utility for automated hardware driver detection and installation, which may simplify wifi and display driver setup.
Benchmark battery life under Linux versus macOS on the same hardware and document power management tweaks (TLP, powertop) that close the gap.
Create a minimal renv-based project as the verification test instead of relying on a colleague’s .Rmd file with unknown dependencies.
Document the T2 chip workaround for newer MacBooks, which requires disabling Secure Boot via macOS Recovery before Linux can be installed.

Wrapping Up

Repurposing an aging MacBook Air with Linux Mint proves to be a practical project. A machine that had become too slow for macOS updates can run a complete data science environment (R, Python, Docker, vim, and Quarto) without noticeable difficulty.

The most valuable outcome is often not the end result but the process itself. Each rendering failure, each missing driver, each broken file path teaches something specific about what a workflow actually depends on. When everything runs without incident on macOS, that understanding never develops.

For those with an old MacBook collecting dust, this project is worth attempting. The worst case is an afternoon spent learning about one’s own toolchain; the best case is a fully functional secondary workstation.

In conclusion, four points merit emphasis. First, Linux Mint 22 installs cleanly on a 2016 MacBook Air with minimal driver friction, a result that would have been implausible on the same hardware a decade earlier. Second, the full software stack (R, Docker, vim, zsh, and Quarto) can be configured in a single sitting once the base system is running. Third, symbolic linking dotfiles from Dropbox creates a portable, reinstallation-proof configuration that transfers across machines with a single script. Fourth, the troubleshooting sequence for document rendering is genuinely educational: each error reveals a specific dependency that a working macOS environment had previously concealed.

Reproducibility

Hardware

Target machine: 2016 13-inch MacBook Air (Early 2015 hardware, model identifier MacBookAir7,2)
Ports: 1 Thunderbolt 2, 2 USB-A 3.0, MagSafe 2, SDXC, 3.5 mm audio
Native panel: 1440x900, 13.3-inch
RAM: 8 GB
Storage: 256 GB SSD

Software Versions

Linux Mint: 22 “Wilma” (Cinnamon edition)
Kernel: Based on Ubuntu 24.04 LTS (“Noble Numbat”)
R: Version installed via r-base-core from Ubuntu repositories
Docker: Latest stable from Docker’s official repository
Quarto: Latest stable release

Verification Commands

cat /etc/os-release

uname -r

R --version | head -1

sudo systemctl is-active docker

echo $SHELL

Let’s Connect

Have questions, suggestions, or spot an error? Let me know.

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: Contact form

I would enjoy hearing from you if:

You spot an error or a better approach to any of the code in this post.
You have suggestions for topics you would like to see covered.
You want to discuss R programming, data science, or reproducible research.
You have questions about anything in this tutorial.
You just want to say hello and connect.

Provisioning AWS EC2 Instances: Console and CLI Methods

Ronald G. Thomas — Sun, 17 May 2026 07:00:00 GMT

2026-05-17 17:08 PDT

Automating cloud infrastructure removes repetitive manual steps and frees attention for the work that matters.

Introduction

I did not really know how virtual servers worked until I had a Shiny app running locally and needed to share it with collaborators over the web. The app was finished, the code was clean, and everything worked on my laptop. ‘Works on my machine’ is not a deployment strategy.

The challenge had two parts: first, create and launch a virtual server in the cloud; second, configure that server to host the application securely. We cover the first part in full.

There are two well-defined entry points to AWS EC2 provisioning: the interactive web console and the aws CLI. The console is a good way to understand what each component does; the CLI is a better choice once the same configuration has been launched more than twice. We document both paths. They reach the same terminal state (a running Ubuntu instance with a security group, key pair, Elastic IP, and Docker installed). The choice is a matter of how often one expects to repeat the operation.

We cover Layer 11 (Cloud) of the workflow architecture described in post 52.

Motivations

A working Shiny application had no deployment path for remote collaborators.
Managed platforms such as shinyapps.io impose constraints on the software stack; a self-managed server does not.
Clicking through the EC2 console is error-prone and slow after the first few repetitions; scripted provisioning is reproducible.
The four bash scripts presented here can be version controlled alongside application code, making the infrastructure part of the project record.

Objectives

Understand the four pre-launch components required for any EC2 deployment (key pair, firewall, static IP, domain name).
Walk through the console path step by step, including Route 53 domain configuration.
Install and configure the AWS CLI, and document the eight environment variables that drive the automation scripts.
Write four bash scripts (security group, key pair, instance launch, Docker bootstrap) that replace the manual console workflow.
Document connecting to the server, verification, and teardown.

A clean workstation poised at the start of a focused provisioning session.

What is AWS EC2?

Amazon Elastic Compute Cloud (EC2) is a service that provides resizable virtual servers in the cloud. Think of it as renting a computer that lives in an Amazon data centre. One chooses the operating system, the amount of memory and storage, and the network configuration. Once launched, one connects over SSH just as one would connect to any remote Linux machine.

The key distinction from a managed hosting platform is control: the operator manages everything on the server, from the operating system packages to the web server configuration. This requires more effort, but it provides complete flexibility over the software stack.

There are several cloud server hosting options: Microsoft Azure, Oracle, Google Cloud, Amazon AWS, Digital Ocean, and Hetzner. Each has its own approach to provisioning, and several offer free or low-cost tiers. AWS is a reasonable choice for a small custom server: the system is well documented and, in my experience, reliable. The AWS free tier includes 750 hours per month of t2.micro instance usage for the first 12 months.

The Four Pre-Launch Components

Regardless of whether the console or the CLI is used, four components must be in place before launching an instance:

SSH key pair: allows encrypted remote login to the server.
Security group (firewall): restricts incoming traffic to named ports only.
Static IP address (Elastic IP): maintains the link between the domain name and the server across reboots. Without it, a new IP is assigned each time the instance restarts.
Domain name: provides a human-readable URL rather than a raw IP address.

These components are independent of any specific server instance. You can define multiple instances of each and reuse them across projects.

After launch, post-launch tasks include installing a web server, obtaining an SSL certificate, and configuring a reverse proxy to translate HTTPS (port 443) requests to Shiny (port 3838). Those steps are covered in the companion post on Dockerizing a Shiny application.

Prerequisites

Operating system: macOS 13+ or a Linux distribution with bash 5+.
AWS account: an active account with permission to create IAM users.
Already installed (for CLI path): Homebrew (macOS) or apt (Debian/Ubuntu), plus jq for JSON parsing.
Background knowledge: comfort editing dotfiles and running shell commands; basic familiarity with the AWS console.
Time required: approximately 30 minutes for either path on a first run.

Method A: Console Walkthrough

This section walks through the EC2 web console. It is instructive for understanding the components and their relationships. After two or three manual launches, the CLI path below will be more efficient.

Open the EC2 console at https://aws.amazon.com/console, choose the regional service closest to the operator’s location (e.g., N. California), and navigate to the EC2 dashboard.

I recommend configuring the four pre-launch components before launching the instance, as it saves back-and-forth with the console.

SSH Key Pair

EC2 supports two approaches: generate the key pair locally and upload the public key, or have EC2 generate the pair and download the private key.

Local generation approach:

cd ~/.ssh
ssh-keygen -m PEM

Name the key prefix power1_app.pem when prompted. The dialog asks for a passphrase; entering one adds additional security but is not required. ssh-keygen generates two files: power1_app.pem and power1_app.pem.pub.

Return to the EC2 dashboard and select Network & Security > Key Pairs > Actions > Import key pair. Enter the name power1_app, browse to ~/.ssh/power1_app.pem.pub, and select Import key pair.

EC2-generated approach:

Select Create key pair in the upper right of the console. Enter a name (e.g., power1_app), select RSA key type and .pem file format. The private key power1_app.pem is offered for download. Place it in ~/.ssh/ and restrict permissions:

chmod 400 ~/.ssh/power1_app.pem

Warning

If the private key file has permissions broader than 400, SSH will refuse to use it with a ‘permissions too open’ error.

Firewall (Security Group)

Select Security Groups under Network & Security in the left panel. Choose Create security group and name it power1_app. Under Inbound Rules, add SSH (port 22) and HTTPS (port 443), each with source Anywhere IPv4 0.0.0.0/0.

This creates a firewall that accepts only SSH and HTTPS inbound traffic.

Static IP Address (Elastic IP)

Navigate to Network and Security > Elastic IPs > Allocate Elastic IP address. An IP from the EC2 pool is assigned (e.g., 13.57.139.31).

Warning

Elastic IPs incur charges when allocated but not associated with a running instance. Release unused Elastic IPs immediately to avoid unexpected costs.

Domain Name (Route 53)

From the AWS console, navigate to the Route 53 dashboard (Services > Route 53). Once a domain name is acquired (e.g., rgtlab.org), associate it with the static IP:

Click Hosted zones in the side panel.
Click on rgtlab.org in the centre panel.
Check the rgtlab.org Type A record.
Click Edit record in the right panel.
Change the IP address in the Value field to the allocated Elastic IP.

Selecting and Launching the Instance

From Instances in the EC2 dashboard, click Launch Instances and follow these steps:

Name the server: enter power1_app.
Select the OS: choose Ubuntu (a mature Linux distribution).
Choose an instance type: select t2.micro (1 CPU, 1 GiB memory).
Choose a key pair: select power1_app from the dropdown.
Select a security group: choose the power1_app group.
Choose storage: enter 30 GB of EBS General Purpose SSD (GP2). Thirty gigabytes is the maximum allowed under the free tier. Smaller disk sizes can cause problems during Docker image builds.
Advanced options: scroll to the bottom and upload the startup script from the Docker Bootstrap section below.
Click Launch Instance.

After the instance launches, open the Elastic IP dialog under Network & Security, select Actions > Associate Elastic IP address, and associate the IP with the new instance.

Method B: CLI Scripts

This section documents the CLI approach. Four bash scripts replace the manual console workflow entirely.

Installation

On macOS, Homebrew provides the simplest installation path:

brew install awscli jq
aws configure

Note

Instructions for installing Homebrew can be found at brew.sh. IAM credential setup is covered in Appendix A below.

The aws configure command prompts for four values:

AWS Access Key ID and AWS Secret Access Key: from your IAM credentials CSV (see Appendix A).
Default region: e.g., us-west-1.
Default output format: json is recommended.

Configuration

Nine parameters are required for automated instance generation. Eight are stored as environment variables in ~/.zshrc (or equivalent) so every script can reference them without hardcoded values.

VPC and Subnet (found on the EC2 dashboard under Your VPCs):

export vpc_id="vpc-14814b73"
export subnet_id="subnet-f02c90ab"

AMI, instance type, and storage (OS image, server capabilities):

export ami_id="ami-014d05e6b24240371"
export instance_type="t2.micro"
export storage_size="30"

Key pair name and security group:

export key_name="power1_app"
export security_grp="sg-0fef542d93849669c"

Static IP (the Elastic IP allocated above):

export static_ip="13.57.139.31"

A ninth parameter, proj_name, can be supplied at call time via the -p flag on each script.

Verify after sourcing:

aws --version
aws configure list
echo "vpc_id=$vpc_id"
aws ec2 describe-instances \
  --query 'Reservations[].Instances[].InstanceId'

If step 3 returns a JSON array (empty or populated), the credentials and region are configured correctly.

Script 1: Create Security Group

Creates a firewall and opens specified ports. The -n flag sets the group name; -p adds a port. Default behaviour opens ports 22 and 443 only.

aws_create_security_group.sh -n power1_app -p 22 -p 80 -p 443

#!/usr/bin/env bash
Help()
{
echo "The script generates a new security group."
echo "The group name is given with the -n flag."
echo "Ports are specified with the -p flag."
echo "Anticipated incoming ports: 22 ssh, 80 http,"
echo "  3838 shiny, 443 https."
echo "Script will fail if group name already exists."
echo "Reads vpc_id from environment variables."
echo "Example:"
echo "  aws_create_security_group.sh \\"
echo "    -n power1_app -p 22 -p 80 -p 443"
}
sg_grp_name=$(basename "$PWD")
while getopts ":hp:n:" opt; do
    case $opt in
        p ) ports+=("$OPTARG") ;;
        n ) sg_grp_name=$OPTARG ;;
        h ) Help
            exit ;;
        * ) echo \
            'error in command line parsing' >&2
            exit 1
    esac
done
echo "sg group name = $sg_grp_name"

aws ec2 create-security-group \
    --group-name "$sg_grp_name" \
    --description "security group" \
    --tag-specifications \
    "ResourceType=security-group,\
Tags=[{Key=Name,Value=$sg_grp_name}]" \
    --vpc-id "$vpc_id" > temp.txt
wait
security_grp=$(jq -r .GroupId temp.txt)
wait
echo "security group ID = $security_grp"

for i in "${ports[@]}"
do
  aws ec2 authorize-security-group-ingress \
    --group-id "$security_grp" \
    --protocol tcp \
    --port "${i}" \
    --cidr "0.0.0.0/0" > /dev/null
done

Script 2: Create Key Pair

Generates an SSH key pair and stores the private key in ~/.ssh/. The -k flag sets the key pair name; if omitted, the current directory name is used.

aws_create_keypair.sh -k power1_app

#!/usr/bin/env bash
Help()
{
echo "The script generates a new key pair."
echo "The key pair name is given with the -k flag."
echo "Script will fail if name already exists."
echo "Example:"
echo "  aws_create_keypair.sh -k power1_app"
}
while getopts 'hk:' flag; do
  case "${flag}" in
    h) Help
      exit;;
    k) key_pair_name=${OPTARG};;
  esac
done
base=$(basename "$PWD")
if [ -z "$key_pair_name" ]
then
  key_pair_name=$base
fi
echo "key_pair_name is $key_pair_name"

cd ~/.ssh
rm -f "$HOME/.ssh/$key_pair_name.pem"
aws ec2 create-key-pair \
  --key-name "$key_pair_name" \
  --query 'KeyMaterial' \
  --output text > "$HOME/.ssh/$key_pair_name.pem"

wait
chmod 400 "$HOME/.ssh/$key_pair_name.pem"

Server racks in a data centre conveying the physical infrastructure behind virtual cloud instances.

Script 3: Launch the EC2 Instance

This is the core script. It reads all eight environment variables, launches the instance, and associates the Elastic IP. The -p flag sets the project name used for tagging.

aws_create_instance.sh -p power1_app

#!/usr/bin/env bash
Help()
{
echo "Notes on current parameters:"
echo "Security group should already exist."
echo "  If not, run aws_create_security_group.sh."
echo "Key pair should already exist."
echo "  If not, run aws_create_keypair.sh."
echo "AMI ID is for Ubuntu Linux 22.04 LTS."
echo "Check static IP: nslookup "
echo ""
echo "Usage:"
echo "  aws_create_instance.sh -p power1_app"
}
while getopts 'hp:' flag; do
  case "${flag}" in
    h) Help
      exit;;
    p) proj_name=${OPTARG};;
  esac
done
base=$(basename "$PWD")
if [ -z "$proj_name" ]
then
  proj_name=$base
fi

aws ec2 run-instances \
  --image-id "$ami_id" \
  --count 1 \
  --instance-type "$instance_type" \
  --key-name "$keypair_name" \
  --security-group-ids "$security_grp" \
  --subnet-id "$subnet_id" \
  --block-device-mappings \
  "[{\"DeviceName\":\"/dev/sda1\",\
\"Ebs\":{\"VolumeSize\":$storage_size}}]" \
  --tag-specifications \
  "ResourceType=instance,\
Tags=[{Key=Name,Value=$proj_name}]" \
  --user-data \
  file://~/Dropbox/prj/c060/aws_startup_code.sh

iid0=$(aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=$proj_name" | \
  jq -r \
  '.Reservations[].Instances[].InstanceId')
echo "$iid0"
read -p "enter instance id:" iid
echo "instance id: $iid"
aws ec2 associate-address \
  --public-ip "$static_ip" \
  --instance-id "$iid"

Script 4: Docker Bootstrap

This user-data script runs automatically on first boot. It installs Docker and Docker Compose on the new Ubuntu instance, then adds the default ubuntu user to the docker group. Upload it in the Advanced options section of the console launch dialog, or pass it via --user-data in Script 3.

#!/bin/bash
apt update
apt-get install curl -y
apt-get install gnupg -y
apt-get install ca-certificates -y
apt-get install lsb-release -y
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL \
  https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor \
  -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch="$(dpkg --print-architecture)" \
  signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release \
  && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list \
  > /dev/null
apt-get update
apt-get install docker-ce docker-ce-cli \
  containerd.io docker-compose-plugin -y
su ubuntu -
usermod -aG docker ubuntu

Note

The startup script runs as root on first boot only. If it fails, check /var/log/cloud-init-output.log on the instance for debugging output.

Connecting to the Server

Construct a config file in ~/.ssh/ to connect with a simple hostname rather than remembering the IP address and key path:

Host rgtlab.org
  HostName 13.57.139.31
  User ubuntu
  Port 22
  IdentityFile ~/.ssh/power1_app.pem

Then connect with:

ssh rgtlab.org

Tip

Set the private key permissions to 400 if not already done: chmod 400 ~/.ssh/power1_app.pem

Verification

After launching the instance and associating the Elastic IP, confirm the server is accessible and Docker is running.

CLI path:

aws --version
aws configure list
echo "vpc_id=$vpc_id"
aws ec2 describe-instances \
  --query 'Reservations[].Instances[].InstanceId'

Server access:

ssh rgtlab.org "echo 'SSH connection successful'"
ssh rgtlab.org "docker --version"
ssh rgtlab.org "docker compose version"

If all three server checks return expected output, the instance is correctly configured.

Daily Workflow

Command	Action
`aws_create_security_group.sh -n NAME -p P`	Create firewall, open port `P`
`aws_create_keypair.sh -k NAME`	Generate `.pem` key under `~/.ssh/`
`aws_create_instance.sh -p NAME`	Launch tagged EC2 instance
`ssh rgtlab.org`	Connect to the running instance
`ssh rgtlab.org "docker ps"`	View running containers
`ssh rgtlab.org "df -h"`	Check disk usage
`aws ec2 describe-instances`	List all instances in current region
`aws ec2 terminate-instances --instance-ids ID`	Terminate by instance ID
AWS Console > EC2 > Instances	Check instance status via GUI

Things to Watch Out For

Security group names must be unique within a VPC. If a group with the same name already exists, the create script will fail. Check with aws ec2 describe-security-groups before running the script.
Key pair names must also be unique. A duplicate name causes a CLI error. Delete the old pair first if regeneration is needed.
The Elastic IP must be allocated before association. If a static IP has not yet been allocated, the associate command in Script 3 will fail silently.
The startup script runs as root on first boot only. If the bootstrap script has an error, the instance must be terminated and a new one launched. There is no way to re-run user-data on an existing instance.
Environment variables are session-scoped. Opening a new terminal without sourcing .zshrc means the scripts will fail because the variables are not set. Verify with echo $vpc_id before running any script.
AWS CLI commands are region-specific. The aws configure region must match the region where the VPC and Elastic IP were allocated.
The free tier storage limit is 30 GB. Choosing less than 30 GB may seem sufficient initially, but Docker images and system packages accumulate quickly.
The --cidr "0.0.0.0/0" rule opens a port to the entire internet. Restrict this to the operator’s IP in production environments.

Uninstall / Rollback

To remove all AWS resources associated with a project, follow these steps in order:

Terminate the instance. EC2 Console > Instances > select instance > Actions > Terminate instance.
Release the Elastic IP. EC2 Console > Elastic IPs > select IP > Actions > Release Elastic IP address.
Delete the security group. EC2 Console > Security Groups > select group > Actions > Delete security groups.
Delete the SSH key pair. EC2 Console > Key Pairs > select pair > Actions > Delete.
Remove local SSH config. Delete the entry from ~/.ssh/config and remove the key file: rm ~/.ssh/power1_app.pem

To remove the AWS CLI from the workstation:

rm -i ~/.aws/credentials ~/.aws/config
brew uninstall awscli   # macOS

Warning

Orphaned Elastic IPs and unused EBS volumes continue to accrue charges. Always run the full teardown when a project is complete.

A calm workspace with a laptop and notebook, representing the planning that goes into infrastructure automation.

What Did We Learn?

Lessons Learnt

Conceptual Understanding:

The EC2 console and the AWS CLI are two interfaces to the same API. Anything clickable in the browser has a corresponding aws ec2 subcommand.
A security group is a stateful firewall at the instance level, not the subnet level. Each rule opens a single port to a specified CIDR range.
Elastic IPs are free while associated with a running instance but incur charges when allocated but unused.
User-data scripts provide a one-shot bootstrap mechanism. For ongoing configuration management, tools such as Ansible or cloud-init are more appropriate.
Cloud servers are ordinary Linux machines running in someone else’s data centre. The mental model of ‘remote laptop’ is surprisingly accurate.

Technical Skills:

Writing bash scripts with getopts for flag parsing produces reusable, self-documenting CLI tools.
The jq utility is essential for extracting fields from the JSON responses returned by AWS CLI commands.
Storing infrastructure parameters as environment variables decouples configuration from code, following the twelve-factor app methodology.
SSH config files simplify repeated connections and reduce typing errors.

Gotchas and Pitfalls:

Forgetting to chmod 400 the PEM file will cause SSH to reject the connection with a permissions error.
Deleting a security group while an instance references it will cause the deletion to fail. Terminate the instance first.
Running out of disk space during Docker builds is a common issue on instances with less than 30 GB of storage.
The EC2 console interface changes periodically. Screenshots in tutorials may not match the current layout exactly.

Limitations

These scripts assume a single-instance deployment and do not address auto-scaling groups, load balancers, or multi-AZ redundancy.
The security group rules open ports to all IPv4 addresses, which is acceptable for development but inappropriate for production.
The bootstrap script installs Docker but does not configure TLS certificates, reverse proxies, or application-level security.
IAM credentials stored in ~/.aws/credentials are long-lived. Rotating to short-lived credentials via IAM roles would be more secure.
The scripts do not include error recovery or rollback logic. A failure mid-sequence can leave orphaned resources.

Opportunities for Improvement

Replace long-lived IAM credentials with IAM roles attached to the EC2 instance profile.
Add set -euo pipefail to each script for stricter error handling.
Migrate the bootstrap script to a cloud-init configuration file for better logging and idempotency.
Restrict security group ingress to the operator’s current public IP rather than 0.0.0.0/0.
Wrap all four scripts in a single orchestration script with a --teardown flag to reverse the entire setup.
Explore AWS CloudFormation or Terraform for declarative infrastructure that can be version controlled and reviewed.
Use AWS Systems Manager Session Manager for SSH access without opening port 22, improving the security posture.

Wrapping Up

Provisioning an AWS EC2 instance requires the same four components regardless of method: a key pair, a security group, an Elastic IP, and an optional domain name. The console walkthrough reveals what each component does; the CLI scripts make the process repeatable in under two minutes.

The most transferable lesson is that the AWS console and the CLI are two windows into the same API. Once that relationship becomes clear, every console action suggests a corresponding scriptable command, and the scope for automation expands considerably.

In conclusion, four points merit emphasis. First, four bash scripts replace the entire EC2 console provisioning workflow, reducing a multi-step manual process to a single command sequence. Second, eight environment variables parameterise the scripts, making them reusable across projects without hardcoded values. Third, the teardown procedure is equally important: orphaned Elastic IPs and unused volumes accumulate charges that are easy to overlook. Fourth, the console path is valuable for learning the component relationships, while the CLI path is more efficient for any repeated provisioning.

Reproducibility

Tested configuration:

Component	Version
Operating system	macOS 13.x
AWS CLI	2.x
Shell	zsh 5.9
jq	1.6+
Homebrew	4.x
Last verified	2026-05-17

The scripts in this post do not require R or Quarto. They require macOS or Linux with bash, Homebrew (macOS) for AWS CLI installation, an active AWS account with IAM credentials, and jq installed (brew install jq).

To reproduce the full provisioning workflow:

aws configure
aws_create_security_group.sh -n power1_app -p 22 -p 80 -p 443
aws_create_keypair.sh -k power1_app
aws_create_instance.sh -p power1_app

Appendix A: IAM Credential Setup

Log into the AWS console.
Search for the IAM service and navigate to the IAM dashboard.
Select User groups and create a group based on the Power User profile. Name it admin and include your IAM user in the group.
Select Users in the left panel, then click Create User.
Enter a username (e.g. zenn) and click Next, then Create User.
Click on the new username. Select the Security Credentials tab.
Under Access Keys, click Create access key.
Select Command Line Interface (CLI) and check the acknowledgement box.
Click Create access key and then Download .csv file.
Save the CSV to ~/.aws/.

Now configure the CLI:

aws configure

Paste the Access Key ID and Secret Access Key from the CSV. Enter your region (e.g. us-west-1) and output format (json).

Note

Never commit AWS credentials to version control. Treat them as passwords.

Appendix B: Sample Work Session

A complete CLI provisioning session starting from scratch, assuming aws configure has been run and VPC/subnet IDs are set as environment variables.

Step 1. Create a security group:

aws_create_security_group.sh -n power1_app -p 22 -p 80 -p 443

Step 2. Create the key pair:

aws_create_keypair.sh -k power1_app

Step 3. Allocate a new Elastic IP via the console and update shell configuration. If the new IP is 204.236.167.50:

Add to ~/.config/zsh/.zsh_export:

export static_ip='204.236.167.50'
export security_grp='sg-0fda72c2879d6b2ad'

Step 4. Launch the instance:

aws_create_instance.sh -p power1_app

Step 5. Update SSH config with the new IP:

# Edit ~/.ssh/config and update the HostName line
# to the new Elastic IP address.

Let’s Connect

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: rgtlab.org/contact

I would enjoy hearing from you if:

You spot an error or a better approach to any of the code in this post.
You have suggestions for topics you would like to see covered.
You want to discuss R programming, data science, or reproducible research.
You have questions about anything in this tutorial.
You just want to say hello and connect.

Rendered on 2026-05-17 at 17:08 PDT.
Source: ~/prj/qblog/posts/22-serversetupawscli/serversetupawscli/analysis/report/index.qmd

Research Backup Architecture: Ongoing System and GitHub Archival

Ronald G. Thomas — Sun, 17 May 2026 07:00:00 GMT

2026-05-17 16:55 PDT

Version control is the starting point; a backup architecture is the surrounding structure that makes it durable.

Introduction

GitHub alone is not a backup. That claim sounds counterintuitive when you have been committing and pushing faithfully for years, but it describes a real architectural gap. GitHub is a remote source-of-truth tier, not a backup tier: it holds one copy, operated by one company, under one pricing model, with no local fallback. A suspended account, a platform outage, or a credential compromise can sever access immediately.

The same gap appears at the local level. Manual git push commands get skipped during busy stretches. An external drive’s Time Machine backup silently stops when the drive is not connected. Cloud sync replicates files but does not preserve commit history. Each tool covers one slice of the risk surface; none covers it entirely.

This post brings together two complementary approaches to research backup:

An ongoing three-tier system that runs automatically every 15 minutes, pushing all dirty Git repositories to GitHub, synchronising files to cloud storage, and relying on Time Machine for system-wide safety.
A bulk GitHub archival procedure that creates verified local mirrors of 400+ private repositories, exports all GitHub-side metadata (issues, PRs, releases, wikis), and supports selective deletion after confirmed verification.

Together they form a complete backup architecture: the ongoing system keeps the day-to-day workflow protected; the archival procedure handles the periodic task of mirroring GitHub itself, so that the source- of-truth tier survives the loss of the account or platform.

More formally, both components document the backup layer of the Workflow Construct described in post 52. Post 52 names backup as a load-bearing layer under the principle ‘two tiers or it is not backup’; the configuration here implements three active tiers for daily use, plus a fourth archival leaf that mirrors GitHub off-platform.

Motivations

The following considerations motivated this architecture:

Having 400+ private repositories on GitHub with no local mirrors creates a single point of failure for years of accumulated work.
Running git push manually and forgetting for days at a time leaves important work vulnerable to local disk failure.
A plain git clone captures code history but misses issues, pull requests, releases, and wiki content that can be more valuable than the code itself.
A colleague’s hard-drive failure, which erased months of analytical work, demonstrated that a single backup tier is never sufficient.
Cloud synchronisation provides file-level replication but does not preserve Git commit history or GitHub metadata.
An automated solution must handle hundreds of repositories without manual intervention for each one.
The archival script requires a dry-run mode so that every action can be previewed before anything destructive is attempted.

Objectives

Configure Time Machine as a system-wide safety net for files that live outside Git.
Write and schedule a production-grade script that commits and pushes all dirty Git repositories automatically every 15 minutes.
Build a bulk archival script that creates a full mirror, portable bundle, wiki clone, and metadata export for every private repository on a GitHub account.
Implement a verification phase that checks every bundle with git bundle verify before any deletion is permitted.
Add a selective preservation mechanism so active or shared repos remain on GitHub while dormant ones are archived and removed.

Corrections and alternative approaches are welcome.

A workspace ready for a focused archival session.

Prerequisites and Setup

For the Ongoing Backup System

This guide assumes a macOS environment with the following tools available:

macOS 12 (Monterey) or later
Homebrew Bash (/opt/homebrew/bin/bash) – macOS ships with Bash 3.2, but the scripts here use Bash 4+ features
Git 2.30 or later, configured with SSH keys for GitHub
A USB external drive (1 TB recommended) for Time Machine
A cloud sync service (Google Drive, Dropbox, or iCloud) for the third tier

The research directory structure assumed throughout is ~/prj/, containing all Git repositories. Adjust paths as needed for your own layout.

For the GitHub Archival Script

Before running the archival script, three tools must be installed and authenticated:

gh auth status

git --version

df -h ~

The disk space check is worth doing early: 400 repositories can require 20-40 GB depending on release asset history.

You also need a GitHub personal access token with repo and delete_repo scopes if you plan to use the deletion phase. The gh auth login flow can configure this interactively. For backup-only runs, the standard repo scope is sufficient.

What is a Research Backup Architecture?

A research backup architecture layers multiple independent protection mechanisms so that no single point of failure can result in data loss. The architecture in this post has four components:

Automated Git commits and pushes (every 15 minutes) – protects against uncommitted work and local corruption by pushing changes to GitHub.
Cloud synchronisation (real-time via Google Drive or Dropbox) – provides continuous file-level replication across devices, useful for non-Git files and immediate access from other machines.
Time Machine backups (hourly, system-wide) – captures the entire filesystem including system settings, application data, and files not covered by the other two tiers.
Periodic GitHub archival (quarterly or before any account change) – creates verified local mirrors of every private repository, exporting all GitHub-side metadata that a plain git clone would miss: issue threads, pull request discussions, release binaries, labels, milestones, and wikis.

Each tier compensates for a weakness in the others. Git does not capture large binary files well; cloud sync does not preserve commit history; Time Machine does not push data off-site; GitHub holds only one copy of the data it hosts. Together, the four components cover the full risk surface.

Section 1: The Three-Tier Ongoing Backup System

Configuring Time Machine

Time Machine provides system-wide backup protection and serves as the safety net for everything beyond Git repositories.

Connect and Format the USB Drive

Connect your USB drive to the MacBook.
When prompted, do not use it for Time Machine yet; configure it properly first.
Open Disk Utility (Applications > Utilities > Disk Utility).
Select the USB drive from the sidebar.
Click Erase.
Choose format: APFS (recommended for modern Macs) or Mac OS Extended (Journaled).
Name it something recognisable, such as ‘Research Backup’.
Click Erase.

Configure Time Machine

Open System Preferences > Time Machine.
Click Select Backup Disk.
Choose your USB drive.
Click Use Disk.
If prompted about encryption, choose Encrypt Backup for security.

Customise Exclusions

Click Options in Time Machine preferences.
Add folders to exclude: Downloads, Trash, virtual machines, and similar high-churn directories.
Do not exclude ~/prj – that directory should be covered as a secondary layer behind Git.
Enable ‘Back up while on battery power’ if you work unplugged frequently.

Time Machine will now back up the entire system (including ~/prj) every hour when the USB drive is connected.

The Minimal Backup Script

Before building the full production script, it helps to see the core logic in its simplest form. This minimal version does three things: finds every Git repository under ~/prj, checks whether it has uncommitted changes, and pushes those changes to the remote.

#!/opt/homebrew/bin/bash

find "$HOME/prj" -name ".git" -type d \
    | while read git_dir; do
    cd "$(dirname "$git_dir")" || continue
    [[ -n $(git status --porcelain) ]] || continue
    git add -A
    git commit -m \
        "Auto-backup: $(date '+%Y-%m-%d %H:%M:%S')"
    git push origin main 2>/dev/null \
        || git push origin master 2>/dev/null
done

This works, but it lacks error handling, logging, user filtering, and any mechanism for diagnosing failures. The full script below addresses each of these gaps.

The Full Backup Script

The production script extends the minimal version with comprehensive features. The sections below walk through it in logical segments.

Configuration and Argument Parsing

#!/opt/homebrew/bin/bash

RESEARCH_DIR="$HOME/prj/"
LOG_FILE="$HOME/Library/Logs/research_backup.log"
MAX_LOG_SIZE=10485760
VERBOSE=false

while [[ $# -gt 0 ]]; do
    case $1 in
        -v|--verbose)
            VERBOSE=true
            shift
            ;;
        -h|--help)
            echo "Usage: $0 [-v|--verbose]" \
                 "[-h|--help]"
            echo "  -v, --verbose" \
                 "Enable verbose output"
            echo "  -h, --help" \
                 "Show this help message"
            exit 0
            ;;
        *)
            echo "Unknown option: $1"
            echo "Use -h or --help for usage"
            exit 1
            ;;
    esac
done

Logging Infrastructure

Log rotation prevents the log file from growing without bound. The log_message function writes every event to disk and optionally echoes colour-coded output to the console when verbose mode is active.

mkdir -p "$(dirname "$LOG_FILE")"

if [[ -f "$LOG_FILE" \
      && $(stat -f%z "$LOG_FILE") \
         -gt $MAX_LOG_SIZE ]]; then
    mv "$LOG_FILE" "${LOG_FILE}.old"
    if [[ "$VERBOSE" == true ]]; then
        echo "INFO: Rotated log file" \
             "(exceeded ${MAX_LOG_SIZE} bytes)"
    fi
fi

log_message() {
    local level="$1"
    local message="$2"
    local timestamp
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    local log_entry
    log_entry="$timestamp: [$level] $message"

    echo "$log_entry" >> "$LOG_FILE"

    if [[ "$VERBOSE" == true ]]; then
        case "$level" in
            ERROR)
                echo -e \
                    "\033[31m$log_entry\033[0m"
                ;;
            WARNING)
                echo -e \
                    "\033[33m$log_entry\033[0m"
                ;;
            SUCCESS)
                echo -e \
                    "\033[32m$log_entry\033[0m"
                ;;
            INFO)
                echo -e \
                    "\033[34m$log_entry\033[0m"
                ;;
            *)
                echo "$log_entry"
                ;;
        esac
    fi
}

Repository Validation Functions

Four helper functions handle the filtering logic. check_remote verifies that the repository has an origin remote. check_user_association ensures that only repositories belonging to the ‘rgt47’ account are processed, preventing the script from pushing to collaborator remotes. should_exclude_directory skips archive and backup folders. get_current_branch and branch_exists_on_remote support intelligent push behaviour.

check_remote() {
    local repo_dir="$1"
    cd "$repo_dir" || return 1
    local remote_url
    remote_url=$(git remote get-url origin \
        2>/dev/null)
    [[ -n "$remote_url" ]]
}

check_user_association() {
    local repo_dir="$1"
    cd "$repo_dir" || return 1

    local remote_url
    remote_url=$(git remote get-url origin \
        2>/dev/null)
    if [[ "$remote_url" == *"rgt47"* ]]; then
        return 0
    fi

    local git_user git_email
    git_user=$(git config user.name 2>/dev/null)
    git_email=$(git config user.email 2>/dev/null)

    if [[ "$git_user" == *"rgt47"* ]] \
       || [[ "$git_email" == *"rgt47"* ]]; then
        return 0
    fi

    if [[ -z "$git_user" ]]; then
        git_user=$(git config --global \
            user.name 2>/dev/null)
    fi
    if [[ -z "$git_email" ]]; then
        git_email=$(git config --global \
            user.email 2>/dev/null)
    fi

    if [[ "$git_user" == *"rgt47"* ]] \
       || [[ "$git_email" == *"rgt47"* ]]; then
        return 0
    fi

    return 1
}

should_exclude_directory() {
    local repo_name="$1"
    local repo_path="$2"

    local lower_name lower_path
    lower_name=$(echo "$repo_name" \
        | tr '[:upper:]' '[:lower:]')
    lower_path=$(echo "$repo_path" \
        | tr '[:upper:]' '[:lower:]')

    if [[ "$lower_name" == *"archive"* ]] \
       || [[ "$lower_name" == *"backup"* ]]; then
        return 0
    fi

    if [[ "$lower_path" == *"archive"* ]] \
       || [[ "$lower_path" == *"backup"* ]]; then
        return 0
    fi

    return 1
}

get_current_branch() {
    git symbolic-ref --short HEAD 2>/dev/null \
        || git rev-parse --short HEAD 2>/dev/null
}

branch_exists_on_remote() {
    local branch="$1"
    git ls-remote --heads origin "$branch" \
        2>/dev/null | grep -q "$branch"
}

Main Loop: Discovery and Processing

The main loop uses find with null-delimited output to safely handle repository paths that contain spaces. Each repository passes through a series of checks before any Git operations are attempted.

log_message "INFO" \
    "Starting research backup scan" \
    " with verbose=$VERBOSE"

if [[ ! -d "$RESEARCH_DIR" ]]; then
    log_message "ERROR" \
        "Research directory $RESEARCH_DIR" \
        " does not exist"
    exit 1
fi

log_message "INFO" \
    "Scanning: $RESEARCH_DIR"

repo_count=0
backup_count=0
error_count=0
warning_count=0
skipped_count=0
excluded_count=0

while IFS= read -r -d '' git_dir; do
    repo_dir=$(dirname "$git_dir")
    repo_name=$(basename "$repo_dir")
    relative_path="${repo_dir#$RESEARCH_DIR}"

    if should_exclude_directory \
       "$repo_name" "$relative_path"; then
        log_message "INFO" \
            "Excluding (archive/backup):" \
            " $relative_path"
        ((excluded_count++))
        continue
    fi

    log_message "INFO" \
        "Processing: $relative_path"

    if ! cd "$repo_dir"; then
        log_message "ERROR" \
            "Cannot access: $repo_dir"
        ((error_count++))
        continue
    fi

    ((repo_count++))

    if ! git rev-parse --git-dir \
         >/dev/null 2>&1; then
        log_message "ERROR" \
            "Not a valid git repo:" \
            " $relative_path"
        ((error_count++))
        continue
    fi

    if ! check_user_association "$repo_dir"; then
        log_message "INFO" \
            "Skipping (not rgt47):" \
            " $relative_path"
        ((skipped_count++))
        continue
    fi

    log_message "INFO" \
        "$relative_path associated with rgt47"

    if ! check_remote "$repo_dir"; then
        log_message "WARNING" \
            "No remote configured:" \
            " $relative_path"
        ((warning_count++))
        ((skipped_count++))
        continue
    fi

    current_branch=$(get_current_branch)
    if [[ -z "$current_branch" ]]; then
        log_message "ERROR" \
            "Cannot determine branch:" \
            " $relative_path"
        ((error_count++))
        continue
    fi

    log_message "INFO" \
        "$relative_path on branch:" \
        " $current_branch"

    git_status=$(git status --porcelain \
        2>/dev/null)

    if [[ -z "$git_status" ]]; then
        log_message "INFO" \
            "$relative_path is clean"
        continue
    fi

    untracked=$(echo "$git_status" \
        | grep -c "^??" || echo 0)
    modified=$(echo "$git_status" \
        | grep -c "^ M" || echo 0)
    added=$(echo "$git_status" \
        | grep -c "^A " || echo 0)
    deleted=$(echo "$git_status" \
        | grep -c "^D " || echo 0)

    log_message "INFO" \
        "$relative_path: $untracked new," \
        " $modified modified, $added added," \
        " $deleted deleted"

    if ! git add -A 2>/dev/null; then
        log_message "ERROR" \
            "Failed to stage: $relative_path"
        ((error_count++))
        continue
    fi

    log_message "INFO" \
        "Staged changes: $relative_path"

    commit_message="Auto-backup:" \
        " $(date '+%Y-%m-%d %H:%M:%S')"

    if git commit -m "$commit_message" \
       >/dev/null 2>&1; then
        log_message "SUCCESS" \
            "Committed: $relative_path"

        if ! branch_exists_on_remote \
             "$current_branch"; then
            log_message "WARNING" \
                "'$current_branch' not on" \
                " remote: $relative_path"

            if git push --set-upstream origin \
               "$current_branch" 2>/dev/null
            then
                log_message "SUCCESS" \
                    "Created and pushed" \
                    " '$current_branch':" \
                    " $relative_path"
                ((backup_count++))
            else
                log_message "ERROR" \
                    "Failed to push new" \
                    " branch: $relative_path"
                ((error_count++))
            fi
        else
            if git push origin \
               "$current_branch" 2>/dev/null
            then
                log_message "SUCCESS" \
                    "Pushed '$current_branch':" \
                    " $relative_path"
                ((backup_count++))
            else
                log_message "ERROR" \
                    "Push failed:" \
                    " $relative_path" \
                    " (check network/auth)"
                ((error_count++))
            fi
        fi
    else
        if git diff --cached --quiet; then
            log_message "INFO" \
                "No changes to commit:" \
                " $relative_path"
        else
            log_message "ERROR" \
                "Commit failed:" \
                " $relative_path"
            ((error_count++))
        fi
    fi

done < <(find "$RESEARCH_DIR" \
    -name ".git" -type d -print0)

Summary Report

After processing every repository, the script logs aggregate statistics and, in verbose mode, prints a human-readable summary to the console.

log_message "INFO" "Backup scan complete"
log_message "INFO" \
    "Summary: $repo_count processed," \
    " $backup_count backed up"
log_message "INFO" \
    "Excluded: $excluded_count," \
    " Skipped: $skipped_count," \
    " Errors: $error_count," \
    " Warnings: $warning_count"

if [[ "$VERBOSE" == true ]]; then
    echo ""
    echo "=== BACKUP SUMMARY ==="
    echo "Repositories found:" \
         "$((repo_count + excluded_count" \
         " + skipped_count))"
    echo "Excluded: $excluded_count" \
         "(archive/backup)"
    echo "Skipped: $skipped_count (not rgt47)"
    echo "Processed: $repo_count"
    echo "Backed up: $backup_count"
    echo "Errors: $error_count"
    echo "Warnings: $warning_count"
    echo ""
    echo "Log file: $LOG_FILE"

    if [[ $error_count -gt 0 ]]; then
        echo ""
        echo "WARNING: There were errors" \
             "during backup. Check the log" \
             "file for details."
        exit 1
    elif [[ $warning_count -gt 0 ]]; then
        echo ""
        echo "NOTE: Backup completed with" \
             "warnings. Check the log file" \
             "for details."
    else
        echo ""
        echo "Backup completed successfully."
    fi
fi

exit 0

Scheduling with Cron

The final step is to make the script run automatically. A cron job at 15-minute intervals provides a good balance between backup frequency and system resource usage.

crontab -e

Add the following entry:

*/15 * * * * /Users/$(whoami)/scripts/backup-research.sh

Save and exit (Ctrl+X, then Y, then Enter in nano; or Esc, :wq, Enter in vim). Verify:

crontab -l

Wait 15 minutes, then confirm execution by inspecting the log:

tail -20 ~/Library/Logs/research_backup.log

Terminal windows and backup logs – the machinery behind automated research protection.

Section 2: Bulk GitHub Account Archival

The ongoing backup system protects daily work, but it does not address the accumulation of private repositories that exist only on GitHub. A GitHub archive, in this context, is a local copy of everything GitHub stores for a repository: not just the code and commit history, but also the metadata that lives only on GitHub’s servers (issue threads, pull request discussions, release binaries, labels, milestones, and wiki pages).

A regular git clone captures the commit graph but misses all of that surrounding context. A mirror clone (git clone --mirror) captures every ref, including remote-tracking branches and tags. A git bundle packages that mirror into a single portable file. And the GitHub API exports capture the metadata that git itself does not track.

The archival script combines all three approaches: mirror clone for completeness, bundle for portability, and API exports for metadata. The result is a self-contained backup directory per repository that can survive even if GitHub itself becomes unavailable.

The Three-Phase Approach

The archive script follows a strict three-phase process. Understanding this structure makes the full script easier to follow.

Phase 1: Backup Everything. For each repository, the script creates:

A full git mirror with every branch and tag
A portable bundle file for easy transfer
Wiki content (if the repo has one)
Metadata exports via the GitHub API (issues, PRs, releases, labels, milestones, workflows)
Downloaded release assets (binaries, artifacts)

Phase 2: Verify Backups. Before any deletion, the script runs git bundle verify on every bundle. If any verification fails, the entire deletion phase is aborted. This step is essential: without it, the deletion phase cannot be trusted.

Phase 3: Selective Deletion. Only repos not in the keep-list get deleted, and only after an explicit typed confirmation (DELETE). Repos in KEEP_ON_GITHUB are still backed up but are not removed from GitHub.

The Complete Working Script

Save the full script as github-archive.sh and make it executable with chmod +x github-archive.sh.

Script Header and Configuration

#!/bin/bash

set -e

OWNER="your-username"
BACKUP_DIR="$HOME/github-archive"
DATE=$(date +%Y%m%d)
LOG_FILE="$BACKUP_DIR/archive_$DATE.log"
DRY_RUN=false

KEEP_ON_GITHUB=(
    "important-project"
    "active-work"
    "shared-with-team"
)

usage() {
    echo "Usage: $0 [OPTIONS]"
    echo ""
    echo "Options:"
    echo "  -n, --dry-run     Show what would happen"
    echo "  -o, --owner NAME  GitHub username/org"
    echo "  -d, --dir PATH    Backup directory"
    echo "  -h, --help        Show this help message"
    exit 0
}

while [[ $# -gt 0 ]]; do
    case $1 in
        -n|--dry-run)
            DRY_RUN=true
            shift
            ;;
        -o|--owner)
            OWNER="$2"
            shift 2
            ;;
        -d|--dir)
            BACKUP_DIR="$2"
            shift 2
            ;;
        -h|--help)
            usage
            ;;
        *)
            echo "Unknown option: $1"
            usage
            ;;
    esac
done

mkdir -p "$BACKUP_DIR"

The KEEP_ON_GITHUB array is the most important configuration. Reviewing the repository list before running the script ensures the right ones are included.

Utility Functions

log() {
    local prefix=""
    if [ "$DRY_RUN" = true ]; then
        prefix="[DRY-RUN] "
    fi
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] \
${prefix}$1" | tee -a "$LOG_FILE"
}

is_kept() {
    local repo=$1
    for kept in "${KEEP_ON_GITHUB[@]}"; do
        if [ "$repo" = "$kept" ]; then
            return 0
        fi
    done
    return 1
}

Phase 1: The Backup Function

backup_repo() {
    local repo=$1
    local repo_dir="$BACKUP_DIR/$repo"

    log "=== Backing up $repo ==="

    if [ "$DRY_RUN" = true ]; then
        log "Would create: $repo_dir"
        log "Would clone: $OWNER/$repo"
        log "Would export: issues, PRs, releases"
        return
    fi

    mkdir -p "$repo_dir"
    cd "$repo_dir"

    if [ ! -d "repo.git" ]; then
        log "Cloning repository..."
        gh repo clone "$OWNER/$repo" \
            repo.git -- --mirror
    else
        log "Updating existing clone..."
        cd repo.git && git fetch --all && cd ..
    fi

    log "Creating git bundle..."
    cd repo.git
    git bundle create ../repo.bundle --all
    cd ..

    log "Checking for wiki..."
    if gh api "repos/$OWNER/$repo" \
        --jq '.has_wiki' 2>/dev/null \
        | grep -q true; then
        git clone \
            "https://github.com/$OWNER/$repo.wiki.git" \
            wiki.git 2>/dev/null \
            || log "No wiki content"
    fi

    log "Exporting metadata..."
    gh api "repos/$OWNER/$repo" \
        > repo-info.json 2>/dev/null || true
    gh api "repos/$OWNER/$repo/issues?state=all" \
        --paginate > issues.json 2>/dev/null || true
    gh api "repos/$OWNER/$repo/pulls?state=all" \
        --paginate \
        > pull-requests.json 2>/dev/null || true
    gh api "repos/$OWNER/$repo/releases" \
        --paginate > releases.json 2>/dev/null || true
    gh api "repos/$OWNER/$repo/labels" \
        --paginate > labels.json 2>/dev/null || true
    gh api "repos/$OWNER/$repo/milestones?state=all" \
        --paginate \
        > milestones.json 2>/dev/null || true

    if [ -s releases.json ] \
        && [ "$(cat releases.json)" != "[]" ]; then
        log "Downloading release assets..."
        mkdir -p release-assets
        gh release list -R "$OWNER/$repo" \
            --limit 100 2>/dev/null \
            | while read -r tag rest; do
            gh release download "$tag" \
                -R "$OWNER/$repo" \
                -D "release-assets/$tag" \
                2>/dev/null || true
        done
    fi

    log "Completed backup of $repo"
    cd "$BACKUP_DIR"
}

The --paginate flag on gh api calls is essential. Without it, GitHub’s API returns only the first 30 items per endpoint, which means issues and PRs on larger repos are silently lost.

Phase 2: The Verification Function

verify_backup() {
    local repo=$1
    local repo_dir="$BACKUP_DIR/$repo"

    log "Verifying backup of $repo..."

    if [ "$DRY_RUN" = true ]; then
        log "Would verify: $repo_dir/repo.bundle"
        return 0
    fi

    if [ -f "$repo_dir/repo.bundle" ]; then
        cd "$repo_dir"
        if git bundle verify repo.bundle \
            > /dev/null 2>&1; then
            log "PASS: Bundle verified"
            return 0
        else
            log "FAIL: Bundle verification failed!"
            return 1
        fi
    else
        log "FAIL: Bundle not found!"
        return 1
    fi
}

Phase 3: The Deletion Function

delete_repo() {
    local repo=$1

    if [ "$DRY_RUN" = true ]; then
        log "Would delete: $OWNER/$repo"
        return
    fi

    log "Deleting $repo from GitHub..."
    gh repo delete "$OWNER/$repo" --yes
    log "Deleted $repo"
}

Main Execution Flow

if [ "$DRY_RUN" = true ]; then
    echo "======================================="
    echo " DRY-RUN MODE - No changes will be made"
    echo "======================================="
    echo ""
fi

log "Starting GitHub archive process"
log "Owner: $OWNER"
log "Backup directory: $BACKUP_DIR"

log "Fetching list of private repositories..."
repos=$(gh repo list "$OWNER" \
    --limit 500 --private \
    --json name -q '.[].name')
repo_count=$(echo "$repos" | wc -l | tr -d ' ')
log "Found $repo_count private repositories"

repos_to_delete=""
repos_to_keep=""
delete_count=0
keep_count=0

for repo in $repos; do
    if is_kept "$repo"; then
        repos_to_keep="$repos_to_keep $repo"
        ((keep_count++)) || true
    else
        repos_to_delete="$repos_to_delete $repo"
        ((delete_count++)) || true
    fi
done

log "Repos to archive and DELETE: $delete_count"
log "Repos to archive and KEEP: $keep_count"

echo ""
echo "=== REPO CATEGORIZATION ==="
echo ""
echo "Will be DELETED after backup ($delete_count):"
for repo in $repos_to_delete; do
    echo "  x $repo"
done
echo ""
echo "Will be KEPT on GitHub ($keep_count):"
for repo in $repos_to_keep; do
    echo "  + $repo"
done
echo ""

if [ "$DRY_RUN" = true ]; then
    echo "=== DRY-RUN: PHASE 1 (BACKUP) ==="
    for repo in $repos; do
        backup_repo "$repo"
    done

    echo ""
    echo "=== DRY-RUN: PHASE 2 (VERIFICATION) ==="
    for repo in $repos; do
        verify_backup "$repo"
    done

    echo ""
    echo "=== DRY-RUN: PHASE 3 (DELETION) ==="
    for repo in $repos_to_delete; do
        delete_repo "$repo"
    done

    echo ""
    echo "======================================="
    echo "   DRY-RUN COMPLETE"
    echo "======================================="
    echo ""
    echo "Run without --dry-run to execute."
    exit 0
fi

log "=== PHASE 1: BACKUP ==="
for repo in $repos; do
    backup_repo "$repo"
done

log "=== PHASE 2: VERIFICATION ==="
failed_repos=""
for repo in $repos; do
    if ! verify_backup "$repo"; then
        failed_repos="$failed_repos $repo"
    fi
done

if [ -n "$failed_repos" ]; then
    log "WARNING: Verification failed:$failed_repos"
    log "Aborting deletion phase"
    exit 1
fi

log "All backups verified successfully!"

log "=== PHASE 3: DELETION ==="
echo ""
echo "Backup complete and verified!"
echo ""
read -p "Delete $delete_count repos? \
(type 'DELETE'): " confirm

if [ "$confirm" = "DELETE" ]; then
    for repo in $repos_to_delete; do
        delete_repo "$repo"
    done
    log "Deleted $delete_count repositories"
else
    log "Deletion cancelled"
fi

log "Archive process complete"

Using the Archival Script

./github-archive.sh --dry-run

./github-archive.sh

./github-archive.sh \
    --owner myorg --dir /external/drive/backup

Running the dry-run first is not merely useful; it is essential. The categorisation output can catch repos that were forgotten in the keep-list.

Configure the KEEP_ON_GITHUB array to match your situation:

KEEP_ON_GITHUB=(
    "active-projects"
    "shared-with-team"
    "client-work"
    "portfolio"
)

Backup Directory Structure

After running the script, the backup directory looks like this:

~/github-archive/
+-- important-project/      # KEPT on GitHub
|   +-- repo.git/
|   +-- repo.bundle
|   +-- repo-info.json
|
+-- old-project-1/          # DELETED from GitHub
|   +-- repo.git/
|   +-- repo.bundle
|   +-- wiki.git/
|   +-- repo-info.json
|   +-- issues.json
|   +-- pull-requests.json
|   +-- releases.json
|   +-- labels.json
|   +-- milestones.json
|   +-- release-assets/
|
+-- archive_20260517.log

Content	Format	Use Case
Git history	`repo.git/` + bundle	Full reproducibility
Wiki	`wiki.git/`	Documentation
Issues	`issues.json`	Discussion archive
Pull requests	`pull-requests.json`	Code review history
Releases	`releases.json`	Version history
Release assets	`release-assets/`	Binaries, artifacts
Metadata	`repo-info.json`	Repository config
Labels	`labels.json`	Issue classification
Milestones	`milestones.json`	Project tracking

Restoring from Archive

git clone \
    ~/github-archive/repo-name/repo.bundle \
    restored-repo

gh repo create new-repo-name --private
cd ~/github-archive/repo-name/repo.git
git push --mirror \
    git@github.com:you/new-repo-name.git

The bundle approach is more convenient for quick local inspection; the mirror push is better for actually recreating a repository on GitHub.

Section 3: Verification and Testing

Ongoing Backup Verification

To confirm that the daily backup script is running correctly:

tail -20 ~/Library/Logs/research_backup.log

~/scripts/backup-research.sh --verbose

Check that recent log entries show [SUCCESS] for pushed repositories and that the timestamp reflects the expected cron interval.

GitHub Archive Verification

Before trusting any archival backup, run these checks:

cd ~/github-archive/repo-name
git bundle verify repo.bundle

ls -la *.json

git bundle list-heads repo.bundle

python3 -m json.tool issues.json | head -50

These checks confirm that the bundle is structurally valid, that metadata files are non-empty, that all branches are present, and that the issue export is well-formed JSON.

Organised workspace: the goal after archiving 400 repos.

After the archive, what remains on GitHub is clean and intentional.

Daily Workflow

Command	Action
`~/scripts/backup-research.sh --verbose`	Run ongoing backup with console output
`tail -20 ~/Library/Logs/research_backup.log`	Review recent backup log entries
`bash github-archive.sh --dry-run`	Preview what would be backed up or deleted
`bash github-archive.sh`	Run full archival: backup, verify, confirm deletion
`bash github-archive.sh --owner ORG`	Archive a different owner’s repositories
`git bundle verify repo.bundle`	Spot-check a specific archival bundle
`gh repo list --json name`	Confirm which repositories remain on GitHub
`crontab -l`	Verify that the 15-minute cron job is registered

Run the archival script quarterly or before any GitHub plan or account change. The ongoing backup script runs automatically once scheduled.

Things to Watch Out For

Homebrew Bash path. macOS ships with Bash 3.2, which lacks features the backup script depends on. Ensure the shebang points to /opt/homebrew/bin/bash (Apple Silicon) or /usr/local/bin/bash (Intel). Version mismatches can cause silent failures that are difficult to diagnose.
SSH key agent in cron. Cron jobs do not inherit your shell environment. If Git remotes use SSH, the cron job may fail silently because ssh-agent is not available. Add eval "$(ssh-agent -s)" to the script or use macOS Keychain integration.
The --limit 500 cap on gh repo list. The command defaults to 30 results. The archival script sets it to 500, but if you have more repositories than that, you need to increase the limit or paginate manually.
Disk space surprises. An initial estimate of 20 GB for 400 repos may prove insufficient; closer to 35 GB may be needed if several repos have large binary assets in their release history. Check with df -h before starting.
Network interruptions during archival. If a clone fails midway, the repo.git directory exists but is incomplete. The verification phase catches this, but you must delete the partial clone and rerun.
API rate limits. GitHub’s API allows 5,000 requests per hour for authenticated users. With 400 repos and 6 API calls each, that is 2,400 requests – within the limit but close. If the limit is reached, the script pauses without automatic retry.
Time Machine drive connection. Time Machine requires the USB drive to be physically connected. When travelling without the drive, this tier is inactive. The Git push tier continues to operate but cloud sync becomes the only off-site copy.

Uninstall / Rollback

Removing the Ongoing Backup System

crontab -e

Delete the line referencing backup-research.sh. Then remove the script:

rm -i ~/scripts/backup-research.sh

The log file can be removed independently:

rm -i ~/Library/Logs/research_backup.log

Restoring from the GitHub Archive

cd ~/github-archive/repo-name
git clone repo.bundle restored-repo

gh repo create owner/repo-name --private
cd restored-repo
git remote add origin git@github.com:owner/repo-name.git
git push --mirror origin

Removing the Archive Script and Local Copies

rm -i github-archive.sh
rm -ri ~/github-archive/

Confirm before removing backups: if GitHub has already been cleaned up, the local archive is the only copy.

What Did We Learn?

Lessons Learnt

Conceptual Understanding:

GitHub is a source-of-truth tier, not a backup tier. A suspended account, a pricing change, or a credential compromise can sever access immediately, regardless of how many commits are in the history.
A single backup mechanism covers only one slice of the risk surface. The combination of automated Git pushes, cloud sync, Time Machine, and periodic archival covers the full surface more reliably than any one mechanism alone.
A plain git clone misses metadata that can be more valuable than the code itself: issue discussions, PR review threads, and release notes.
The three-phase archival pattern (backup, verify, delete) is a general discipline applicable to any destructive batch operation, not just GitHub archiving.

Technical Skills:

Bash find with -print0 and read -d '' safely handles directory names containing spaces and special characters, which is common in research project naming.
The gh CLI is more powerful than often expected. Combining gh repo list, gh api --paginate, and gh release download covers nearly every GitHub operation without touching the web interface.
git bundle verify is a built-in safety net that may be unfamiliar. It confirms the bundle is a valid, complete repository.
Log rotation using file size checks (stat -f%z) prevents unbounded log growth in long-running automated scripts.

Gotchas and Pitfalls:

Cron does not source .bashrc or .zshrc, so environment variables, SSH keys, and PATH modifications are not available unless explicitly set within the script or the crontab.
The --paginate flag on gh api is critical. Forgetting it silently loses issues beyond the first 30 on larger repos.
GitHub wikis are technically separate git repositories. They must be cloned independently, and they silently fail if the wiki was enabled but never populated.
set -e causes the archival script to exit on the first error, which is good for safety but requires || true on commands that are expected to fail (such as cloning an empty wiki).

Limitations

The ongoing backup script only pushes to the origin remote. Repositories using multiple remotes receive backup to only one of them.
Auto-generated commit messages (‘Auto-backup: timestamp’) lack descriptive content. This serves the backup purpose but pollutes Git history for active development branches.
The archival script does not handle GitHub Actions workflow run history or Codespaces configurations, which are not available through the standard API.
Repository settings (branch protection rules, webhook configurations, deploy keys) are not exported. Recreating those requires manual setup or additional API calls.
Git LFS objects are not included in the mirror clone by default. Repositories using LFS need git lfs fetch --all added to the backup function.
The archival script processes repositories sequentially. For 400+ repos, this can take several hours. Parallelisation would speed things up but adds complexity.
Time Machine requires the USB drive to be physically connected. When travelling without the drive, this tier is inactive.

Opportunities for Improvement

Replace fixed commit messages in the ongoing backup script with a brief summary of changed file names, providing more informative Git history.
Add Git LFS support to the archival script by inserting git lfs fetch --all into the backup function for repositories that use large file storage.
Parallelise the archival backup phase using xargs -P 4 or GNU parallel to clone multiple repositories concurrently, reducing total runtime.
Export repository settings (branch protection rules, webhooks, deploy keys) so that restoration is truly complete.
Migrate from cron to launchd for better macOS integration, including wake-from-sleep triggers and retry logic.
Add a companion github-restore.sh script that reads a backup directory and recreates repositories on GitHub, including metadata re-import.
Add compression: after archival verification, compress each repository directory with tar -czf to reduce storage requirements by roughly 50-70%.

Wrapping Up

The two components in this post address different timescales of the same underlying concern: ensuring that research work is recoverable regardless of what fails. The ongoing three-tier system handles the daily rhythm, committing and pushing every 15 minutes while Time Machine and cloud sync provide complementary coverage. The archival procedure handles the periodic task of verifying that GitHub itself – the source-of-truth tier – has a local mirror that would survive the loss of the account.

The most important lesson is not about any specific tool but about the architecture: no single backup mechanism is sufficient, verification must be part of the process rather than assumed, and automation is the only reliable way to maintain discipline across hundreds of repositories over months and years.

Main takeaways:

GitHub is not a backup. It is one copy, held by one company. The backup architecture is everything surrounding it.
Three independent ongoing tiers (Git push, cloud sync, Time Machine) cover the full spectrum of daily failure modes.
The three-phase archival pattern (backup, verify, delete) prevents data loss from incomplete archives and ensures deletions are irreversible only after confirmation.
A 400-repo archive takes roughly 2-4 hours and 20-40 GB of disk space depending on repo sizes.

Reproducibility

Environment requirements:

macOS 12 (Monterey) or later; tested on macOS 14 Sonoma
Bash 5.x via Homebrew (macOS ships 3.2; install with brew install bash)
Git 2.39 or later
gh CLI 2.40 or later

Version checks:

git --version
gh --version
bash --version

Script files:

File	Description
`backup-research.sh`	Ongoing 15-minute backup
`github-archive.sh`	Bulk archival script
`archive_YYYYMMDD.log`	Archival execution log
`research_backup.log`	Ongoing backup log

This post does not use R or Docker. The entire workflow is pure Bash with the GitHub CLI and macOS system tools.

Let’s Connect

Have questions, suggestions, or spot an error? Let me know.

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: Contact form

I would enjoy hearing from you if:

You spot an error or a better approach to any of the code in this post.
You have suggestions for topics you would like to see covered.
You want to discuss R programming, data science, or reproducible research.
You have questions about anything in this tutorial.
You just want to say hello and connect.

Rendered on 2026-05-17 at 17:08 PDT.
Source: ~/prj/qblog/posts/02-githubarchive/githubarchive/analysis/report/index.qmd

Extending the R-Vim Workflow: LaTeX Integration and Dynamic Snippets

Ronald G. Thomas — Sun, 17 May 2026 07:00:00 GMT

2026-05-17 16:55 PDT

Setting the stage for a unified editing environment.

Introduction

I did not really know how to connect Vim, R, and LaTeX into a single productive workflow until I spent a weekend configuring vimtex and UltiSnips from scratch. Before that weekend, my workflow was fragmented: I wrote R code in one editor, switched to another for LaTeX, and lost time every time I moved between tools.

The frustration peaked when I realised I was spending more time switching contexts than actually writing code or prose. Every tool had its own shortcuts, its own conventions, and its own mental model. I needed a single environment that could handle R scripts, R Markdown documents, and LaTeX files without forcing me to leave Vim.

We document the complete configuration arrived at after several rounds of trial and error, then extend it with a technique discovered later: embedding Python code directly inside UltiSnips snippets. Static snippets cover predictable boilerplate. Python-interpolated snippets cover cases where the expansion must be computed at runtime (for instance, generating a variable number of repeated elements based on a value typed into the first tabstop).

More formally, we document the Vim-plugins layer (Layer 6) of the Workflow Construct described in post 52, specifically the R-plus-LaTeX subset. The plugins documented here (Nvim-R or zzvim-r, vimtex, UltiSnips) compose to give Vim the same edit-run-inspect cycle that RStudio provides for R and Skim provides for LaTeX, without leaving the editor. They are the operative artefacts that make the Editor layer (post 26) usable for the applied biostatistician’s daily R-and-LaTeX work.

Motivations

I was tired of context-switching between RStudio, a LaTeX editor, and the terminal every time I worked on a statistical report.
I wanted a single editor that could send R code to a live REPL, compile LaTeX documents, and expand code snippets without leaving the keyboard.
My existing Vim setup had no awareness of R or LaTeX filetypes, so I had to configure everything from scratch.
I needed ALE linting and auto-fixing for R code style consistency across projects.
I wanted UltiSnips templates for R Markdown YAML headers so I could scaffold new analysis files in seconds.
I had never configured ftplugin files before and wanted to understand how Vim dispatches settings by filetype.
Static snippets proved too rigid for cases that require a variable number of output elements; I needed a way to compute expansion content at the moment of triggering.

Objectives

Install and configure vimtex, UltiSnips, ALE, and supporting plugins in a single .vimrc file.
Create filetype-specific configurations for R and R Markdown that open an R terminal and define useful keybindings.
Set up UltiSnips snippet expansion and popup menu navigation without conflicts between Tab mappings.
Walk through a practical example of editing an R Markdown file with the Palmer Penguins dataset inside this configured Vim environment.
Extend UltiSnips with Python interpolation to generate dynamic, parametric snippet expansions.

I am documenting my learning process here. If you spot errors or have better approaches, please let me know.

The kind of focused environment where configuration work happens best.

Prerequisites and Setup

Before proceeding, the following software must be installed on the macOS system:

Vim 8.1+ compiled with Python 3 support or Neovim 0.5+ with terminal support. To verify Python 3 support, run :echo has('python3') inside Vim; a return value of 1 confirms it is available.
R 4.4+ from CRAN
MacTeX from TUG
vim-plug plugin manager, installed with:

curl -fLo ~/.vim/autoload/plug.vim \
  --create-dirs \
  https://raw.githubusercontent.com/\
junegunn/vim-plug/master/plug.vim

A working terminal emulator (iTerm2 or Kitty) and basic familiarity with Vim normal and insert modes are also assumed. See the companion post ‘Setting up a minimal neovim…’ for details on installing plugins with Neovim. Python 3 support is required for UltiSnips to function at all; without it, the plugin loads but snippet expansion silently fails.

What is a Vim-Based R and LaTeX Workflow?

A Vim-based R and LaTeX workflow is a configuration that supports writing R code, R Markdown documents, and LaTeX files inside a single Vim session. Instead of switching between RStudio for data analysis and a dedicated LaTeX editor for typesetting, Vim serves as the unified interface.

Think of it as turning Vim into a lightweight IDE. The .vimrc file loads plugins for LaTeX compilation (vimtex), code snippet expansion (UltiSnips), and syntax linting (ALE). Filetype plugins detect whether the open file is an R script or an R Markdown document and automatically open an R terminal in a split pane. Keybindings allow sending code to the REPL, inspecting data objects, and rendering documents without leaving the editor.

The practical benefit is speed. Once configured, one can move from opening a file to running a complete analysis and compiling a PDF report without touching the mouse or switching applications.

Getting Started

The first step is to create or edit the ~/.vimrc file with the full plugin configuration. The configuration below is a complete, working .vimrc that I use daily. It includes plugin declarations, global settings, and keybindings.

After saving the .vimrc, open Vim and run :PlugInstall to download and install all declared plugins.

A close-up of a keyboard and terminal screen showing configuration files being edited.

The iterative process of refining a configuration file one setting at a time.

The Complete .vimrc Configuration

The following is the full ~/.vimrc file. It is organised into sections: plugin declarations with vim-plug, plugin-specific settings inline with each Plug call, and global editor settings at the end.

set runtimepath^=~/vimplugins
syntax enable
filetype plugin indent on
let mapleader = ","
let maplocalleader = " "

call plug#begin('~/vimplugins')

" -- Colour schemes --
Plug 'rakr/vim-one'
Plug 'mhartington/oceanic-next'
Plug 'rafi/awesome-vim-colorschemes'
Plug 'tpope/vim-vividchalk'

" -- Clipboard --
Plug 'jasonccox/vim-wayland-clipboard'

" -- Alignment --
Plug 'junegunn/vim-easy-align'

" -- Undo tree --
Plug 'mbbill/undotree'

" -- Terminal REPL integration --
Plug 'jpalardy/vim-slime'

" -- LaTeX --
Plug 'lervag/vimtex'
let g:vimtex_complete_close_braces=1
let g:vimtex_quickfix_mode=0

" -- Snippets --
Plug 'sirver/ultisnips'
let g:UltiSnipsExpandTrigger=""
let g:UltiSnipsJumpForwardTrigger=""
let g:UltiSnipsJumpBackwardTrigger=""
nnoremap U \
  call UltiSnips#RefreshSnippets()
nnoremap u :UltiSnipsEdit

" -- Linting and fixing --
Plug 'dense-analysis/ale'
let g:ale_virtualtext_cursor = 'disabled'
let g:ale_set_balloons = 1
highlight clear ALEErrorSign
highlight clear ALEWarningSign
let g:ale_sign_error = '●'
let g:ale_sign_warning = '.'
let g:ale_linters = {
\   'python': ['pylsp']
\ }
let g:ale_fix_on_save = 1
let g:ale_fixers = {
\   '*': ['remove_trailing_lines',
\         'trim_whitespace'],
\   'r': ['styler'],
\   'rmd': ['styler'],
\   'quarto': ['styler'],
\   'python': ['black','isort'],
\   'javascript': ['eslint'],
\ }
nnoremap n :ALENext

" -- Fuzzy finder --
Plug 'junegunn/fzf',
  \ { 'do': { -> fzf#install() } }
Plug 'junegunn/fzf.vim'
nnoremap z :Files
nnoremap ' :Marks
nnoremap / :BLines
nnoremap b :Buffers
nnoremap r :Rg
nnoremap s :Snippets
tnoremap b :Buffers
tnoremap r :Rg
tnoremap z :Files

" -- Navigation and editing --
Plug 'junegunn/vim-peekaboo'
Plug 'tpope/vim-unimpaired'
Plug 'tpope/vim-obsession'
Plug 'tpope/vim-repeat'
Plug 'tpope/vim-commentary'
autocmd FileType quarto
  \ setlocal commentstring=#\ %s
Plug 'tpope/vim-surround'
Plug 'justinmk/vim-sneak'
let g:sneak#label = 1
let g:sneak#s_next = 1
let g:sneak#use_ic_scs = 1
Plug 'machakann/vim-highlightedyank'

" -- Status line --
Plug 'vim-airline/vim-airline'
let g:airline#extensions#ale#enabled = 1
let g:airline#extensions#tabline#enabled = 1
let g:airline#extensions#fzf#enabled = 1

" -- R integration --
Plug 'rgt47/rgt-R'

" -- Completion --
Plug 'girishji/vimcomplete'

call plug#end()

" -- Editor settings --
if $COLORTERM == 'truecolor'
  set termguicolors
endif
set scrolloff=7
set iskeyword-=.
set completeopt=menu,menuone,popup,
  \noinsert,noselect
set complete+=k
set dictionary=/usr/share/dict/words
highlight Pmenu guifg=Black guibg=cyan
  \ gui=bold
highlight PmenuSel gui=bold guifg=White
  \ guibg=blue
set gfn=Monaco:h14
set encoding=utf-8
set lazyredraw
set autochdir
set number relativenumber
set clipboard=unnamed
set textwidth=80
set colorcolumn=80
set cursorline
set hlsearch
set incsearch
set ignorecase
set smartcase
set showmatch
set noswapfile
set hidden
set gdefault
set splitright
set wildmenu
set wildignorecase
set wildmode=list:full

" -- Popup menu navigation with Tab --
inoremap  
  \ pumvisible() ? "\" : "\"
inoremap  
  \ pumvisible() ? "\" : "\"

" -- Buffer and window navigation --
nnoremap o :b1
nnoremap t :b2
nnoremap h :b3
nnoremap  w
nnoremap a ggVG
nnoremap m vipgq
nnoremap f :tab split
nnoremap v :edit ~/.vimrc
nnoremap  
nnoremap  
noremap - $
noremap : ;
noremap ; :
inoremap  
inoremap  
inoremap   `^

" -- Terminal mode navigation --
tnoremap  
tnoremap o :b1
tnoremap t :b2
tnoremap h :b3
tnoremap  w
tnoremap lf ls()

" -- Auto-save and formatting --
au FocusGained * :let @z=@*
set updatetime=1000
autocmd CursorHold,CursorHoldI * update
autocmd FileType rmd
  \ setlocal commentstring=#\ %s
let $FZF_DEFAULT_OPTS =
  \ '--bind "ctrl-j:down,ctrl-k:up,'
  \ . 'j:preview-down,k:preview-up"'
set formatoptions-=c
  \ formatoptions-=r formatoptions-=o

" -- EasyAlign --
xmap ga (EasyAlign)
nmap ga (EasyAlign)

colorscheme one
set background=dark

After saving this file, open Vim and run:

:PlugInstall

Vim-plug will download and install every plugin declared between plug#begin() and plug#end().

Filetype Plugin for R Markdown

Create the file ~/.vim/ftplugin/rmd.vim. This file is loaded automatically whenever Vim opens a file with the .Rmd or .rmd extension. It defines terminal-mode shortcuts for rendering and sourcing, and normal-mode shortcuts for inspecting R objects under the cursor.

" ~/.vim/ftplugin/rmd.vim

" -- Terminal shortcuts for R Markdown --
tnoremap ZD quarto::quarto_render(
  \output_format = "pdf")
tnoremap ZO source(""%")
tnoremap ZQ q('no'):q!
tnoremap ZR render(""%")
tnoremap ZT :!R -e 'render("%",
  \ output_format="pdf_document")'
tnoremap ZS style_dir()
tnoremap ZX exit
tnoremap ZZ q('no'):q!

" -- Object inspection under cursor --
nnoremap d
  \ :let @c=expand("") \|
  \ :let @d="dim(".@c.")"."\n" \|
  \ :call term_sendkeys(
  \   term_list()[0], @d)
nnoremap h
  \ :let @c=expand("") \|
  \ :let @d="head(".@c.")"."\n" \|
  \ :call term_sendkeys(
  \   term_list()[0], @d)
nnoremap s
  \ :let @c=expand("") \|
  \ :let @d="str(".@c.")"."\n" \|
  \ :call term_sendkeys(
  \   term_list()[0], @d)
nnoremap p
  \ :let @c=expand("") \|
  \ :let @d="print(".@c.")"."\n" \|
  \ :call term_sendkeys(
  \   term_list()[0], @d)
nnoremap n
  \ :let @c=expand("") \|
  \ :let @d="names(".@c.")"."\n" \|
  \ :call term_sendkeys(
  \   term_list()[0], @d)

" -- Auto-open R terminal --
autocmd BufEnter *
  \ if &ft ==# 'rmd' && !exists('b:entered')
  \ | execute(
  \   'let b:entered = 1 | :ter ++rows=5 R')
  \ | endif

The object inspection mappings deserve explanation. When the cursor is on a variable name, pressing d extracts the word under the cursor, wraps it in dim(), and sends the command to the R terminal. This allows checking dimensions, printing heads, inspecting structure, and viewing names of any R object without leaving normal mode.

Filetype Plugin for R Scripts

Create the file ~/.vim/ftplugin/r.vim. This file is nearly identical to the R Markdown ftplugin but uses the alternate file register (# instead of %) for sourcing commands, which is appropriate when the R script is not the current buffer.

" ~/.vim/ftplugin/r.vim

" -- Terminal shortcuts for R scripts --
tnoremap ZD quarto::quarto_render(
  \output_format = "pdf")
tnoremap ZO source(""#")
tnoremap ZQ q('no'):q!
tnoremap ZR render(""#")
tnoremap ZS style_dir()
tnoremap ZX exit
tnoremap ZZ q('no'):q!

" -- Object inspection under cursor --
nnoremap d
  \ :let @c=expand("") \|
  \ :let @d="dim(".@c.")"."\n" \|
  \ :call term_sendkeys(
  \   term_list()[0], @d)
nnoremap h
  \ :let @c=expand("") \|
  \ :let @d="head(".@c.")"."\n" \|
  \ :call term_sendkeys(
  \   term_list()[0], @d)
nnoremap s
  \ :let @c=expand("") \|
  \ :let @d="str(".@c.")"."\n" \|
  \ :call term_sendkeys(
  \   term_list()[0], @d)
nnoremap p
  \ :let @c=expand("") \|
  \ :let @d="print(".@c.")"."\n" \|
  \ :call term_sendkeys(
  \   term_list()[0], @d)
nnoremap n
  \ :let @c=expand("") \|
  \ :let @d="names(".@c.")"."\n" \|
  \ :call term_sendkeys(
  \   term_list()[0], @d)

" -- Auto-open R terminal --
autocmd BufEnter *
  \ if &ft ==# 'r' && !exists('b:entered')
  \ | execute(
  \   'let b:entered = 1 | :ter ++rows=5 R')
  \ | endif

UltiSnips: Static Snippets

UltiSnips is a snippet engine for Vim that expands trigger words into multi-line text templates. Each snippet definition lives in a filetype-specific file under ~/.vim/UltiSnips/, for example ~/.vim/UltiSnips/rmd.snippets for R Markdown files. A snippet consists of a trigger keyword, one or more tabstops (marked $1, $2, …) where the cursor will land after expansion, and the body text placed between those tabstops.

The critical configuration detail for this setup is avoiding conflicts between UltiSnips and popup menu navigation, since both want to use the Tab key. The solution is to assign Ctrl-Tab as the expand trigger and reserve plain Tab and Shift-Tab for jumping between tabstops. Separately, the popup menu mappings in .vimrc use Tab and Shift-Tab only when a popup is visible.

let g:UltiSnipsExpandTrigger=""
let g:UltiSnipsJumpForwardTrigger=""
let g:UltiSnipsJumpBackwardTrigger=""

inoremap  
  \ pumvisible() ? "\" : "\"
inoremap  
  \ pumvisible() ? "\" : "\"

With this configuration, the workflow is:

Type a snippet trigger word (e.g., rheader).
Press Ctrl-Tab to expand the snippet.
Press Tab to jump to the next tabstop.
When a popup menu appears during typing, Tab navigates forward through completions instead.

A minimal R Markdown YAML header snippet looks like this in the snippets file:

snippet rheader "R Markdown YAML header"
---
title: "$1"
author: "${2:R.G. Thomas}"
date: today
output:
  pdf_document:
    keep_tex: true
header-includes:
  - \usepackage{lipsum, fancyhdr,
      titling, currfile}
  - \usepackage[export]{adjustbox}
  - \pagestyle{fancy}
---
$0
endsnippet

When rheader is typed and Ctrl-Tab is pressed, the template expands and the cursor lands at $1 (the title field). Pressing Tab moves to $2 (the author field, which has a default value), and then to $0 (the final resting position after all tabstops are filled).

UltiSnips: Python-Interpolated Dynamic Snippets

Static snippets cover predictable boilerplate. There is a class of problem, however, where the expansion content must be computed at the moment of triggering based on something the user has typed. UltiSnips supports three interpolation mechanisms beyond static text: shell execution (backtick), Vimscript (!v), and Python (!p). Python interpolation is the most capable: it executes an arbitrary Python expression inside the snippet body and inserts the result into the expanded text.

The !p marker tells UltiSnips to treat the enclosed block as Python. Inside that block, the special variable snip.rv (rv standing for ‘return value’) holds whatever the snippet should insert at that position. Any valid Python expression can compute snip.rv, including imports, loops, and string formatting.

A Concrete Motivating Problem

A question on Stack Overflow (https://stackoverflow.com/questions/78636197) asked how to write a snippet that generates a variable number of chord placeholders. The number of placeholders depends on a value the user types into the first tabstop. This is exactly the kind of problem that static snippets cannot solve: the body cannot be known at the time the snippet file is written because it depends on runtime input.

The Python function that solves this problem is straightforward:

def make_chords(num_reps):
    text = ""
    for i in range(num_reps):
        text += "< > "
    return text

To embed this inside an UltiSnips snippet definition:

snippet chords "Generate chord placeholders"
`!p
def make_chords(num_reps):
    text = ""
    for i in range(num_reps):
        text += "< > "
    return text

snip.rv = make_chords(int(t[1]))
`
endsnippet

The backtick delimiters with !p open and close the Python block. Inside the block, t[1] refers to the current value of tabstop 1. When the user types a number into the first tabstop and moves focus, UltiSnips re-evaluates the Python block and replaces the output accordingly. The int() conversion is required because t[1] is always a string.

Inserting the Current Date

A simpler but commonly useful dynamic snippet inserts the current date at expansion time:

snippet today "Insert today's date"
`!p
import datetime
snip.rv = datetime.date.today().isoformat()
`
endsnippet

This snippet takes no tabstops. When triggered, it imports the datetime module and assigns the ISO-formatted date string to snip.rv. The result is a plain text date that does not change after insertion, unlike a live template that re-evaluates on every keystroke.

A Numbered List Generator

A more elaborate pattern uses Python to generate a numbered list of a length specified by the user:

snippet numlist "Numbered list of N items"
${1:5}
`!p
n = int(t[1])
snip.rv = "\n".join(
    f"{i+1}. " for i in range(n)
)
`
endsnippet

When 5 is the value of tabstop 1, the expansion becomes five numbered lines. Changing the number before moving off the tabstop regenerates the list.

Debugging Python Snippets

When a Python interpolation block fails silently, the most reliable diagnostic is to open Vim’s message log with :messages immediately after a failed expansion. Python exceptions inside UltiSnips are caught and displayed as Vim error messages. A second approach is to test the Python logic independently in a Python interpreter before embedding it, since the UltiSnips execution environment is standard CPython with no special restrictions beyond access to the snip object.

" Force UltiSnips to re-read all snippet files
:call UltiSnips#RefreshSnippets()

" Open the snippet file for the current filetype
:UltiSnipsEdit

If the expansion trigger fires but produces no output, confirm Python 3 support with :echo has('python3'). A return value of 0 means the Vim binary was compiled without Python and the entire !p mechanism is unavailable.

Practical Application

This section walks through a concrete example of using the configured environment. The goal is to perform a logistic regression on the Palmer Penguins dataset, predicting gender, using only Vim and the R terminal.

Step 1: Create a working directory and open an empty R Markdown file.

cd ~/prj/penguins
vim -u ~/.vimrc p.Rmd

Step 2: Enter insert mode and type the first R command to load the data.

i
library(palmerpenguins)

Step 3: Exit insert mode with Ctrl-C, then yank the line into the unnamed register.

yy

Step 4: The ftplugin for R Markdown automatically opens an R terminal in a split pane. If it has not opened yet, use :ter R to start one manually.

Step 5: In the terminal pane, paste the yanked line from the register:

""

This sends library(palmerpenguins) to the R REPL and executes it.

Step 6: Use UltiSnips to scaffold the R Markdown header. Type rheader on the first line and press Ctrl-Tab. The snippet expands into a YAML header template with tabstops for the project name, title, author, and bibliography path. Press Tab to navigate between tabstops. Do not leave insert mode while navigating or the snippet session will end.

R Markdown Template Structure

For reference, a minimal R Markdown YAML header for PDF output with LaTeX customisation looks like this:

---
title: "Penguins data analysis"
author: "R.G. Thomas"
date: today
output:
  pdf_document:
    keep_tex: true
header-includes:
  - \usepackage{lipsum, fancyhdr,
      titling, currfile}
  - \usepackage[export]{adjustbox}
  - \pagestyle{fancy}
---

A useful tip for file completion in Vim when editing R Markdown or Quarto files: temporarily set the filetype to tex with :set filetype=tex, then type \includegraphics{ or \input{ followed by Ctrl-X Ctrl-O for omni-completion of file paths.

Things to Watch Out For

Tab key conflicts. UltiSnips and popup menu completion both use Tab. If snippets stop expanding, check that g:UltiSnipsExpandTrigger is set to rather than .
ftplugin not loading. If the R terminal does not open automatically, verify that filetype plugin indent on appears in the .vimrc before any Plug calls, and that the ftplugin files are in ~/.vim/ftplugin/ (not ~/.vim/plugin/).
ALE styler dependency. The ALE fixer for R uses the styler package. If ALE reports errors on save, install styler in R first: install.packages("styler").
Colour scheme not found. If Vim reports E185: Cannot find color scheme, run :PlugInstall to ensure the colour scheme plugin has been downloaded.
Terminal register paste. The "" paste in terminal mode pastes from the unnamed register. If the text was yanked into a named register, use "a (for register a).
Python interpolation requires Python 3 support. Vim must be compiled with +python3. The default macOS system Vim often lacks this. If :echo has('python3') returns 0, install Vim via Homebrew (brew install vim) or use Neovim with a Lua snippet engine such as LuaSnip.
Leaving insert mode terminates the snippet session. During tabstop navigation, pressing Escape ends the session and leaves the remaining tabstop markers as literal text. If this happens, undo the expansion with u and re-trigger the snippet.

A calm library reading room with natural light filtering through tall windows.

Stepping back to reflect on what was learnt.

Lessons Learnt

Conceptual Understanding

A Vim-based R and LaTeX workflow replaces application switching with buffer switching, which is measurably faster once muscle memory develops.
Filetype plugins are the correct mechanism for language-specific settings in Vim; global .vimrc settings should remain language-agnostic.
The separation between snippet expansion (Ctrl-Tab) and popup navigation (Tab) is essential for avoiding conflicts in a multi-plugin environment.
UltiSnips tabstops provide a structured way to scaffold repetitive code patterns, reducing boilerplate errors in YAML headers.
Static snippets and Python-interpolated snippets occupy different parts of the solution space: use static snippets for fixed structure and Python snippets when the expansion depends on runtime input.

Technical Skills

Configuring vim-plug with inline let statements keeps plugin settings co-located with their declarations, improving maintainability.
The term_sendkeys() function in Vim 8+ enables programmatic communication with terminal buffers, which is the foundation for REPL integration.
ALE fixers configured per filetype (r, rmd, quarto) apply consistent code style without manual intervention.
The autocmd BufEnter pattern with a guard variable (b:entered) prevents multiple R terminals from spawning when switching buffers.
The !p marker in UltiSnips gives access to a full Python environment; snip.rv is the single output variable that carries the computed text back into the expansion.
t[1] inside a Python block reads the current value of tabstop 1, enabling parametric expansions where the output depends on what the user has typed.

Gotchas and Pitfalls

Leaving insert mode during UltiSnips tabstop navigation silently terminates the snippet session, leaving partially expanded text.
The autochdir setting can interfere with relative file paths in R Markdown documents if the working directory changes unexpectedly.
ALE’s ale_fix_on_save may conflict with files that are intentionally unformatted (e.g., raw data files opened in Vim).
The set iskeyword-=. setting breaks word boundaries at dots, which is helpful for R (where data.frame is two words) but can confuse navigation in other filetypes.
Python exceptions inside UltiSnips are swallowed silently unless :messages is checked; a snippet that produces no output is often a Python runtime error rather than a configuration problem.

Limitations

This configuration is specific to Vim 8+ with terminal support; Neovim requires different terminal API calls (jobsend instead of term_sendkeys).
The ftplugin auto-opens R in a 5-row terminal split, which may be too small on low-resolution displays.
ALE’s styler fixer requires the styler R package to be installed in the active R library; if missing, ALE silently fails to fix on save.
UltiSnips requires Vim compiled with Python 3 support; the default macOS system Vim may lack this. Neovim users should use LuaSnip, which provides equivalent Python-like interpolation through Lua functions.
Python code inside UltiSnips snippets runs synchronously on the main thread. Expensive computations (file reads, network requests) will block the editor for the duration.
The configuration does not include debugging support; stepping through R code requires a separate tool or plugin.
Vimtex’s forward and inverse search with a PDF viewer (e.g., Zathura or Skim) requires additional configuration not covered here.
Python snippet code runs in a shared interpreter state; variables defined in one snippet block persist into subsequent expansions in the same session, which can produce unexpected results if variable names collide.

Opportunities for Improvement

Add Neovim-compatible terminal integration using vim.fn.jobsend() for cross-editor portability.
Configure vimtex forward search with Skim or Zathura for real-time PDF preview during LaTeX editing.
Create project-specific .vimrc files that override the terminal split size based on the display resolution.
Develop additional UltiSnips templates for common R analysis patterns (ggplot scaffolds, model fitting boilerplate, knitr chunk options).
Port the Python interpolation patterns to LuaSnip for Neovim users, using Lua functions in place of Python !p blocks.
Integrate vim-slime as an alternative to the built-in terminal for sending code to tmux sessions.
Add a Quarto-specific ftplugin file for .qmd files with render commands that target both HTML and PDF output.

Wrapping Up

The configuration described in this post transforms Vim into a unified environment for R programming and LaTeX document preparation, then extends it with programmable snippet expansion through Python interpolation. The core insight is that filetype plugins and a carefully managed Tab key mapping are the two pillars of the static setup, while snip.rv and the t[] tabstop array are the two primitives needed for dynamic expansions.

What I learnt most from this process is that configuration is iterative. The .vimrc I use today looks nothing like the one I started with. Each problem encountered (a Tab conflict, a missing colour scheme, a terminal that would not open) taught me something about how Vim dispatches events and how plugins interact. The Python interpolation extension followed the same pattern: a concrete problem (a Stack Overflow question about chord placeholders) revealed a capability I had not known existed.

For anyone attempting a similar setup, start with just vimtex and one ftplugin file. Get the R terminal auto-opening reliably before adding UltiSnips or ALE. Add static snippets next. Reach for Python interpolation only after encountering a problem that static snippets genuinely cannot solve. Layering complexity gradually makes debugging much simpler.

In conclusion, five points merit emphasis. First, use Ctrl-Tab for snippet expansion to avoid Tab key conflicts with popup menu navigation. Second, place filetype-specific settings in ~/.vim/ftplugin/ rather than cluttering the global .vimrc. Third, ALE with styler provides automatic R code formatting on save with no manual intervention. Fourth, the term_sendkeys() function enables sending arbitrary R commands from normal mode to a running R REPL. Fifth, the !p marker in UltiSnips enables Python interpolation: assign the result to snip.rv and read tabstop values from t[1], t[2], etc.

Reproducibility

This configuration was developed and tested on macOS with Vim 9.0, Python 3.11, and R 4.4. The following files constitute the complete configuration:

File	Purpose
`~/.vimrc`	Global plugin and editor settings
`~/.vim/ftplugin/rmd.vim`	R Markdown keybindings
`~/.vim/ftplugin/r.vim`	R script keybindings
`~/.vim/UltiSnips/rmd.snippets`	R Markdown snippet library
`~/.vim/UltiSnips/all.snippets`	Filetype-agnostic snippets

To reproduce the environment:

curl -fLo ~/.vim/autoload/plug.vim \
  --create-dirs \
  https://raw.githubusercontent.com/\
junegunn/vim-plug/master/plug.vim

cp vimrc ~/.vimrc
cp ftplugin/rmd.vim ~/.vim/ftplugin/rmd.vim
cp ftplugin/r.vim ~/.vim/ftplugin/r.vim

vim +PlugInstall +qall

Rscript -e "install.packages('styler')"

To verify Python 3 support after installation:

:echo has('python3')
:echo g:UltiSnipsExpandTrigger

Let’s Connect

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: rgtlab.org/contact

I would enjoy hearing from you if:

You spot an error or a better approach to any of the code in this post.
You have suggestions for topics you would like to see covered.
You want to discuss R programming, data science, or reproducible research.
You have questions about anything in this tutorial.
You just want to say hello and connect.

Rendered on 2026-05-17 at 17:08 PDT.
Source: ~/prj/qblog/posts/30-setupRvimtex/setupRvimtex/analysis/report/index.qmd

Sharing R Code via Docker: R Markdown Reports and Shiny Applications

Ronald G. Thomas — Sun, 17 May 2026 07:00:00 GMT

2026-05-17 16:55 PDT

Just as shipping containers standardised global freight, Docker containers standardise software delivery: for static analyses and interactive applications alike.

Introduction

I did not really know how fragile sharing R code could be until my colleague (let us call him Joe) spent an entire afternoon debugging missing packages, incompatible R versions, and absent files, all because I sent him a single .Rmd file by email. The code ran perfectly on my machine. Joe’s Linux Mint workstation disagreed on every count.

The Shiny case is worse. A static R Markdown report requires R, packages, and possibly LaTeX. A Shiny application requires all of that plus a running web server, reactive dependencies, and often JavaScript libraries or database connections. Each additional dependency is another opportunity for something to break on a collaborator’s machine, and the failure mode is a blank browser tab with no useful error message.

We cover both cases. The core Docker machinery (Dockerfile authoring, base image selection, dependency installation) is explained once and applies to both. The post then presents two parallel case studies:

Case A: Dockerizing an R Markdown report for static PDF output.
Case B: Dockerizing a Shiny application for interactive browser use.

More formally, we document the Container layer of the Workflow Construct described in post 52. The Container layer is replaceable across runtimes (Docker on the laptop, Apptainer on HPC, Podman as a daemonless alternative); its core machinery is the same regardless of what the container ultimately serves.

Motivations

I had sent an R Markdown analysis to a colleague and watched him spend hours debugging environment issues that had nothing to do with the analysis itself.
I had also sent a Shiny application as a zip file and observed the same pattern: hours resolving package version conflicts and missing system libraries before the app would launch.
Every error my colleagues encountered was entirely preventable with proper packaging.
I wanted a reproducible method for sharing R analyses that would work regardless of the recipient’s operating system, R version, or installed packages.
The experience convinced me that ‘works on my machine’ is not a valid standard for collaborative data science, and that Docker addresses both the static and interactive output cases.

Objectives

Explain the core Docker concepts (image, container, Dockerfile, layer caching, volume mounts, port mapping) that apply to all R containerisation workflows.
Demonstrate the failure modes of naive sharing for both R Markdown and Shiny by walking through the sequence of errors a collaborator encounters.
Write complete Dockerfiles for each case: one for an R Markdown report, one for a Shiny application.
Document the complete workflow from authoring to collaboration, including image sharing options and Shiny-specific concerns (process supervision, port binding, host parameter).

I am documenting my learning process here. If you spot errors or have better approaches, please let me know.

The analysis begins on one machine, but it needs to run on many.

Prerequisites and Setup

Before proceeding, the following are required:

Docker Desktop installed on the workstation (macOS, Windows, or Linux)
R 4.0+ with rmarkdown and/or shiny installed locally (for authoring)
A Docker Hub account (free) for pushing and sharing images
Basic familiarity with the terminal and R Markdown or Shiny application structure

What is Docker for R Users?

Docker is a tool that packages an application and its entire computing environment (operating system libraries, programming language, installed packages, configuration files, data) into a single portable unit called an image. When someone runs that image, they get a container that behaves identically to the environment in which the analysis was developed.

Think of it like shipping a complete laboratory instead of just a lab notebook. The naive approach to sharing code is like sending someone a lab notebook and hoping they have the same equipment, reagents, and room temperature. Docker is like shipping the entire lab bench, pre-loaded with everything needed to replicate the experiment.

For R users specifically, Docker solves the problem of environment divergence. The R version, installed packages, LaTeX distribution, auxiliary files, and data are all captured in the image. The recipient does not need to install anything except Docker itself.

Core Concepts

Image: A snapshot of a complete computing environment. Images are built from a Dockerfile and stored as a series of read-only layers.

Container: A running instance of an image. Containers are ephemeral by default; data written inside a container is lost when it stops unless a volume mount is used.

Dockerfile: A text file that defines how to build an image. Each instruction (FROM, RUN, COPY, CMD) creates a new layer. Docker caches layers, so unchanged instructions do not re-execute on subsequent builds.

Volume mount: A link between a directory on the host machine and a path inside the container. Volume mounts allow the container to read host files and write output that persists after the container stops.

Port mapping: For network services (Shiny applications), the container’s internal port must be mapped to a host port with the -p flag, or the host machine cannot connect to the running service.

The Rocker Project

The Rocker Project maintains a family of Docker images for R. The relevant images for this post are:

Image	Contents	Use case
`rocker/r-ver`	Base R only	Lightweight scripts
`rocker/verse`	R + tidyverse + LaTeX	R Markdown PDF
`rocker/shiny`	R + Shiny Server	Shiny applications

All images accept a version tag (e.g., rocker/verse:4) to pin the R version.

Case A: Dockerizing an R Markdown Report

The Analysis

Assume you have a simple R Markdown file called peng.Rmd that analyses the Palmer Penguins dataset. The file uses pacman for package management, tidyverse for data manipulation and plotting, and knitr for output formatting. It also references a custom LaTeX preamble file (preamble.tex) and a logo image (sudoku.png) for the PDF header.


---
title: "Penguins analysis"
author: "R.G. Thomas"
date: "`r format(Sys.time(), '%B %d, %Y')`"
fontsize: 11pt
geometry: "left=3cm,right=5cm,top=2cm,bottom=2cm"
output:
  pdf_document:
    keep_tex: true
    includes:
      in_header: "preamble.tex"
---

```{r include=F, echo=F}
library(pacman)
p_load(palmerpenguins, tidyverse, knitr)

opts_chunk$set(
  warning = FALSE,
  message = FALSE,
  echo = FALSE,
  results = "asis",
  dev = "pdf"
)
```

# Introduction

We can work with the dataset `penguins`
included in the package `palmerpenguins`.

```{r }
library(palmerpenguins)
```

One naive approach is to split the dataset
and do three separate analyses:

```{r }
df1 <- split(penguins, penguins$species)

foo <- function(df, z) {
  df |>
    ggplot(
      aes(
        x = bill_length_mm,
        y = flipper_length_mm
      )
    ) +
    geom_point(
      aes(color = island), alpha = .5
    ) +
    geom_smooth() +
    scale_color_manual(
      values = c("purple", "green", "red")
    ) +
    theme_bw() +
    labs(
      title = paste(
        z, " Penguin Anatomy Comparison"
      ),
      x = "Flipper length",
      y = "Bill length",
      color = "Island"
    )
  plotfile_name <- paste0(z, ".pdf")
  ggsave(plotfile_name)
  cat(
    paste0(
      "\\includegraphics[height=3cm]{",
      plotfile_name, "}"
    ),
    "\n"
  )
  cat("\\vspace{1cm}", "\n")
}

bar <- df1 |> map2(names(df1), foo)
```

The file runs cleanly on the author’s machine and produces a PDF report with three species-specific scatter plots comparing bill length and flipper length.

The Naive Approach and Its Errors

The simplest approach is to email peng.Rmd to Joe and ask him to render it. The following sequence of errors occurred in the order presented.

Error 1: R Not Found. Linux cannot find R. Joe installs r-base-core via apt.

Error 2: Function render Not Found. R loads but rmarkdown is missing. The package installation fails due to missing system libraries (libssl-dev, libcurl4-openssl-dev, libxml2-dev, and others).

Error 3: Pandoc Version Too Old. The system pandoc is below 1.12.3. Joe installs a newer version.

Error 4: Package pacman Not Found. After resolving pandoc, pacman is missing.

Error 5: Missing preamble.tex. The author forgot to send the LaTeX preamble file. The path convention also differs between macOS and Linux.

Error 6: pdflatex Not Found. LaTeX is absent. Joe installs tinytex.

Error 7: Missing Logo File. pandoc error: file sudoku.png not found. The author forgot to send the logo image referenced in the preamble.

Error 8: Missing Bibliography File. Another file the author forgot to include.

After eight errors and several hours, Joe renders the report. Every error was environmental.

The Dockerfile

The following Dockerfile starts from rocker/verse:4 (which includes R, tidyverse, and LaTeX), installs additional R packages, updates the LaTeX distribution, creates a non-root user, and copies all necessary files into the image.

FROM rocker/verse:4
RUN apt update
RUN apt install vim -y
RUN R -e "install.packages('pacman')"
RUN R -e "install.packages('palmerpenguins')"
RUN R -e "install.packages('tidyverse')"
RUN R -e "install.packages('knitr')"
RUN R -e "install.packages('rmarkdown')"
RUN tlmgr init-usertree
RUN tlmgr update --self --all
RUN tlmgr install fancyhdr adjustbox \
  geometry titling

RUN addgroup --system joe && \
  adduser --system --ingroup joe joe
RUN chmod -R 0777 \
  '/usr/local/lib/R/site-library'
RUN chown joe:joe -R /home/joe
USER joe
WORKDIR /home/joe
RUN mkdir -p /home/joe/shr
RUN mkdir -p /home/joe/output
COPY /preamble.tex /home/joe/shr
COPY sudoku.png /home/joe/shr
COPY peng.Rmd /home/joe/shr
CMD ["/bin/bash"]

Key decisions:

FROM rocker/verse:4 provides R 4.x with tidyverse and LaTeX pre-installed.
Each RUN installs a specific package, making layer caching effective. If only the Rmd file changes, only the COPY layer rebuilds.
tlmgr install adds the exact LaTeX packages referenced by preamble.tex.
A non-root user (joe) runs the analysis as a security best practice.
COPY commands place the preamble, logo, and Rmd file inside the image, so nothing is missing.

Building and Sharing the Image

Build with a tag and platform flag for cross-architecture compatibility:

docker build -t rgt47/penguin_review \
  --platform=linux/amd64 .

Option 1: Docker Hub (recommended for remote collaboration)

docker push rgt47/penguin_review

Joe pulls the image on his machine:

docker pull rgt47/penguin_review

Option 2: File transfer (useful for air-gapped networks)

docker save rgt47/penguin_review \
  | gzip > penguin_review.tgz

Joe loads the image from the archive:

docker load -i penguin_review.tgz

Running the Container

Joe runs the container with a volume mount that connects a local output directory to the container’s output directory. This is how rendered files are extracted from the container.

droot="$PWD/output"
docker run -it --rm \
  --platform linux/x86_64 \
  -v "$droot":/home/joe/output \
  rgt47/penguin_review

Inside the container, Joe renders the report:

cd output
R -e "library(rmarkdown); \
  render('../shr/peng.Rmd')"

The rendered PDF appears in the local output/ directory. No errors. No missing packages. No missing files.

The LaTeX Preamble

For reference, the preamble.tex file that caused Error 5 in the naive approach:

\usepackage[export]{adjustbox}
\usepackage{fancyhdr}
\usepackage{titling}

\pagestyle{fancy}

\pretitle{
\begin{flushright}
\includegraphics[width=3cm,valign=c]{
  sudoku.png}\\
\end{flushright}
\begin{flushleft} \LARGE }
\posttitle{
  \par\end{flushleft}\vskip 0.5em}
\predate{
  \begin{flushleft}\large}
\postdate{
  \par\end{flushleft}}
\preauthor{
  \begin{flushleft}\large}
\postauthor{
  \par\end{flushleft}}
\fancyfoot[L]{\currfilename}
\fancyfoot[R]{
  \includegraphics[width=.8cm]{sudoku.png}}
\fancyhead[L]{\today}

In the Docker image, this file is already in place at /home/joe/shr/preamble.tex. Joe never needs to know it exists.

A pair of hands assembling components into a container, representing the process of packaging an analysis for reproducibility.

Packaging everything together so no piece is left behind.

Case B: Dockerizing a Shiny Application

The Shiny case introduces concerns absent from the R Markdown case: the container must run a persistent web server, bind to a network port accessible from the host, and handle multiple concurrent connections. The Dockerfile structure is similar, but the base image, the CMD, and the docker run flags differ.

The Application

Assume you have a single-file Shiny app (app.R) that displays an interactive scatter plot of the Palmer Penguins dataset.

library(shiny)
library(ggplot2)
library(DT)
library(palmerpenguins)

ui <- fluidPage(
  titlePanel("Palmer Penguins Explorer"),
  sidebarLayout(
    sidebarPanel(
      selectInput(
        "species", "Species:",
        choices = c(
          "All",
          unique(
            as.character(penguins$species)
          )
        )
      ),
      selectInput(
        "x_var", "X Variable:",
        choices = c(
          "bill_length_mm",
          "bill_depth_mm",
          "flipper_length_mm",
          "body_mass_g"
        )
      ),
      selectInput(
        "y_var", "Y Variable:",
        choices = c(
          "flipper_length_mm",
          "bill_length_mm",
          "bill_depth_mm",
          "body_mass_g"
        )
      )
    ),
    mainPanel(
      plotOutput("scatter"),
      DTOutput("table")
    )
  )
)

server <- function(input, output, session) {
  filtered_data <- reactive({
    if (input$species == "All") {
      penguins
    } else {
      penguins[
        penguins$species == input$species,
      ]
    }
  })

  output$scatter <- renderPlot({
    ggplot(
      filtered_data(),
      aes(
        x = .data[[input$x_var]],
        y = .data[[input$y_var]],
        colour = species
      )
    ) +
      geom_point(alpha = 0.7, size = 3) +
      theme_minimal() +
      labs(
        x = input$x_var,
        y = input$y_var,
        colour = "Species"
      )
  })

  output$table <- renderDT({
    filtered_data()
  })
}

shinyApp(ui, server)

The app runs flawlessly on the development machine.

The Naive Approach and Its Errors

You zip the directory and email it to Joe. The following sequence of errors occurred.

Error 1: Shiny Not Installed. there is no package called 'shiny'. Installing it triggers compilation of httpuv and its system dependencies.

Error 2: System Libraries Missing. The httpuv package fails to compile because libssl-dev and libuv1-dev are absent.

Error 3: Package DT Not Found. After installing Shiny, the app fails because DT is missing.

Error 4: palmerpenguins Not Found. The data package is missing.

Error 5: Port Already in Use. Another service occupies port 3838. Joe must identify and stop the conflicting process.

Error 6: Version Incompatibility. Joe’s R version is 4.1; the app uses .data[[]] syntax from rlang 1.0+, which requires a newer ggplot2 than what installed against R 4.1.

After six errors and several hours, Joe has a running application. Every error was environmental.

The Dockerfile

The following Dockerfile starts from rocker/shiny:4, which includes R and Shiny Server pre-installed.

FROM rocker/shiny:4

RUN apt-get update && apt-get install -y \
  libssl-dev \
  libcurl4-openssl-dev \
  libuv1-dev \
  && rm -rf /var/lib/apt/lists/*

RUN R -e "install.packages(c( \
  'ggplot2', \
  'DT', \
  'palmerpenguins', \
  'rlang' \
  ), repos = 'https://cran.r-project.org')"

RUN rm -rf /srv/shiny-server/*

COPY app.R /srv/shiny-server/app.R

EXPOSE 3838

CMD ["/usr/bin/shiny-server"]

Key decisions:

FROM rocker/shiny:4 provides R 4.x with Shiny Server pre-installed, eliminating manual web server configuration.
System libraries are installed before R packages to ensure compilation of binary dependencies (httpuv, openssl).
rm -rf /srv/shiny-server/* removes default sample apps, leaving only the target application.
EXPOSE 3838 documents the port that Shiny Server listens on (it does not publish the port; that requires -p at runtime).
CMD starts Shiny Server when the container launches, so Joe does not need to type any commands inside the container.

Alternative: Single-File Approach

For simpler applications, it is possible to bypass Shiny Server and run the app directly with Rscript. This uses the smaller rocker/r-ver base image.

FROM rocker/r-ver:4

RUN R -e "install.packages(c( \
  'shiny', \
  'ggplot2', \
  'DT', \
  'palmerpenguins' \
  ), repos = 'https://cran.r-project.org')"

COPY app.R /app/app.R

EXPOSE 3838

CMD ["Rscript", "-e", \
  "shiny::runApp('/app/app.R', \
  host='0.0.0.0', port=3838)"]

The trade-off is the loss of Shiny Server features (logging, process management, multi-app serving), but for single-app containers it is often sufficient and produces a smaller image.

Building and Sharing the Image

Build with a tag:

docker build -t rgt47/penguins-shiny \
  --platform=linux/amd64 .

Option 1: Docker Hub

docker push rgt47/penguins-shiny

Joe pulls the image:

docker pull rgt47/penguins-shiny

Option 2: File transfer

docker save rgt47/penguins-shiny \
  | gzip > penguins-shiny.tgz

Joe loads the image:

docker load -i penguins-shiny.tgz

Running the Container

Joe runs the container with port mapping:

docker run -d --rm \
  -p 3838:3838 \
  --name penguins-app \
  rgt47/penguins-shiny

He opens a browser to http://localhost:3838 and the application is running. No errors. No missing packages. No port conflicts.

To stop the application:

docker stop penguins-app

Shiny-Specific: Process Supervision and

Multi-User Deployment

The rocker/shiny image runs the open-source edition of Shiny Server, which does not efficiently handle many concurrent users. For production deployments with authentication and per-user container isolation, consider ShinyProxy. ShinyProxy acts as a reverse proxy and launches a fresh container for each authenticated user, providing isolation without requiring Shiny Server Pro.

For applications requiring persistent user uploads or database writes, mount a host directory to the container:

docker run -d --rm \
  -p 3838:3838 \
  -v "$PWD/uploads":/srv/shiny-server/uploads \
  --name penguins-app \
  rgt47/penguins-shiny

A quiet desk with a closed laptop and a cup of coffee, representing the conclusion of a completed workflow.

When the analysis runs on the first try, on any machine, the work is done.

Things to Watch Out For

Volume mount paths must be absolute. The -v flag requires an absolute path on the host side. Use "$PWD/output" or provide the full path explicitly.
Platform flag matters for Apple Silicon. When building on an M1/M2/M3 Mac, the default architecture is arm64. Use --platform=linux/amd64 to ensure the image runs on Intel-based Linux machines.
The host parameter must be '0.0.0.0' for Shiny. By default, shiny::runApp() binds to 127.0.0.1, which is only accessible inside the container. This is the most common reason a Dockerized Shiny app appears to start but shows a blank page.
EXPOSE does not publish the port. The EXPOSE instruction in a Dockerfile is documentation only. The -p flag at runtime is what makes the port reachable from the host.
Image size can be large. The rocker/verse base image is approximately 2 GB; rocker/shiny is approximately 1.5 GB. Adding packages increases this further. Use rocker/r-ver for a smaller base when the full tidyverse or LaTeX is not required.
R package versions are not pinned. The Dockerfiles above install the latest version of each package at build time. For strict reproducibility, use renv to pin exact package versions and restore from renv.lock inside the Dockerfile.
Non-root user permissions. If the container writes output files as a non-root user, the host directory may need appropriate permissions. The chmod -R 0777 in the Case A Dockerfile is permissive; in production, use more restrictive permissions.

Daily Workflow

Task	Command
Build image (R Markdown)	`docker build -t rgt47/penguin_review --platform=linux/amd64 .`
Build image (Shiny)	`docker build -t rgt47/penguins-shiny --platform=linux/amd64 .`
Push to Docker Hub	`docker push rgt47/`
Save image to file	`docker save rgt47/ \\| gzip > image.tgz`
Load image from file	`docker load -i image.tgz`
Run R Markdown container	`docker run -it --rm -v "$PWD/output":/home/joe/output rgt47/penguin_review`
Run Shiny container	`docker run -d --rm -p 3838:3838 --name penguins-app rgt47/penguins-shiny`
View Shiny logs	`docker logs penguins-app`
Stop Shiny container	`docker stop penguins-app`
List running containers	`docker ps`
Remove stopped containers	`docker container prune`

Uninstall / Rollback

To remove a specific image:

docker rmi rgt47/penguin_review
docker rmi rgt47/penguins-shiny

To remove all stopped containers and dangling images (reclaim disk space):

docker system prune

To remove all unused images (not just dangling):

docker system prune -a

Docker Desktop can be uninstalled via the application’s settings menu on macOS and Windows. On Linux, remove the docker-ce package via the system package manager. Uninstalling Docker does not affect files outside the Docker storage directory (typically /var/lib/docker).

Lessons Learnt

Conceptual Understanding

The naive approach to sharing R code fails because it assumes the recipient’s environment matches the author’s environment. In practice, it never does.
Docker eliminates environment divergence by packaging the complete computing context alongside the analysis code, whether that code produces a static file or a live web application.
The rocker project provides pre-built Docker images for R that include common dependencies, significantly reducing Dockerfile complexity.
Volume mounts are the mechanism for extracting output from containers; port mapping is the mechanism for connecting a container’s network service to the host machine’s browser.

Technical Skills

Building a Dockerfile for R requires installing both R packages (install.packages()) and system libraries (apt install) for packages with compiled dependencies.
The rocker/verse base image covers most R Markdown rendering requirements; rocker/shiny covers Shiny Server deployments; rocker/r-ver is the smallest base for scripts that do not need the full tidyverse or LaTeX.
docker save and docker load provide an alternative to Docker Hub for sharing images on restricted networks.
docker run -d starts a container in detached mode, which is appropriate for long-running server applications like Shiny.

Gotchas and Pitfalls

Forgetting to include auxiliary files (preamble, images, data) in the Dockerfile COPY commands is the most common source of R Markdown build failures.
Forgetting to set host='0.0.0.0' is the most common reason a Dockerized Shiny app appears to start but shows a blank page.
The --platform flag is essential for cross-architecture compatibility between Apple Silicon and Intel machines.
Running docker build without --no-cache may use stale cached layers if upstream packages have been updated; add --no-cache when a fully fresh build is required.

Limitations

Docker Desktop is required on the recipient’s machine, which may not be permitted in all organisational environments.
Base images are large (1.5 to 2 GB before adding packages), making image sharing impractical on low-bandwidth connections.
R package versions are not pinned in these Dockerfiles; future builds may install different versions, breaking reproducibility over time. Integrating renv addresses this but adds complexity.
The Case B Dockerfile does not address authentication or access control; the Shiny app is accessible to anyone who can reach the mapped port.
Shiny Server (open-source edition) does not efficiently handle many concurrent users. For production multi-user deployments, ShinyProxy or Shiny Server Pro is required.
Neither Dockerfile includes a health check; orchestration tools (Kubernetes, Docker Compose) cannot verify the container is actually responding without one.
Docker ensures environment consistency, not analytical validity. Code correctness is a separate concern.

Opportunities for Improvement

Integrate renv into both Dockerfiles to pin exact package versions and ensure long-term reproducibility.
Add a Makefile to the R Markdown image that automates the rendering step, allowing Joe to run make render instead of typing the full R command.
Use multi-stage Docker builds to separate the build environment (with compilation tools) from the runtime environment, reducing the final image size.
Add a docker-compose.yml for analyses that require additional services (e.g., a PostgreSQL database alongside R or Shiny).
Implement ShinyProxy for multi-user Shiny deployments with authentication and per-user container isolation.
Add a HEALTHCHECK instruction to the Shiny Dockerfile (HEALTHCHECK CMD curl -f http://localhost:3838 || exit 1) for container orchestration.
Explore GitHub Actions to automatically build and push Docker images whenever the analysis or application code is updated.

Wrapping Up

The contrast between the naive approach and the Docker approach illustrates a fundamental principle of reproducible research: the computing environment is part of the analysis. Sending an Rmd file or a Shiny application without its environment is like publishing a paper without its methodology section.

The naive approach produced eight separate errors for the R Markdown case and six for the Shiny case, each requiring Joe to diagnose and resolve a dependency issue. The Docker approach produced zero errors in either case. Joe pulled the image, ran a single command, and either opened a rendered PDF in his output/ directory or a working application in his browser.

The Shiny case makes the stakes clearer: a missing dependency in a static analysis produces a clear error message. A missing dependency in a Shiny app produces a blank browser tab. Docker addresses both failure modes by fixing the entire stack at build time.

In conclusion, five points merit emphasis. First, the naive approach produced eight distinct environment errors for R Markdown and six for Shiny before the output could be obtained. Second, Docker eliminates all such errors by packaging the complete environment into a single image. Third, Case A uses rocker/verse (R + tidyverse + LaTeX) with volume mounts for PDF output. Fourth, Case B uses rocker/shiny (R + Shiny Server) with port mapping for browser access. Fifth, the host='0.0.0.0' parameter and the -p flag are the two settings most commonly missed when Dockerizing Shiny applications.

Reproducibility

This workflow was developed on macOS with Docker Desktop 4.x, R 4.4, and Shiny 1.8.

Case A files:

File	Purpose
`peng.Rmd`	R Markdown analysis file
`Dockerfile`	Environment definition
`preamble.tex`	LaTeX header customisation
`sudoku.png`	Logo image for PDF header

Case B files:

File	Purpose
`app.R`	Shiny application code
`Dockerfile`	Environment definition

To reproduce Case A:

docker build -t rgt47/penguin_review \
  --platform=linux/amd64 .

mkdir -p output
docker run -it --rm \
  --platform linux/x86_64 \
  -v "$PWD/output":/home/joe/output \
  rgt47/penguin_review

# Inside the container:
# cd output
# R -e "library(rmarkdown); render('../shr/peng.Rmd')"

To reproduce Case B:

docker build -t rgt47/penguins-shiny \
  --platform=linux/amd64 .

docker run -d --rm \
  -p 3838:3838 \
  --name penguins-app \
  rgt47/penguins-shiny

# Open browser to http://localhost:3838

docker stop penguins-app

Let’s Connect

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: rgtlab.org/contact

I would enjoy hearing from you if:

You spot an error or a better approach to any of the code in this post.
You have suggestions for topics you would like to see covered.
You want to discuss R programming, data science, or reproducible research.
You have questions about anything in this tutorial.
You just want to say hello and connect.

Rendered on 2026-05-17 at 17:08 PDT.
Source: ~/prj/qblog/posts/32-sharermdcodeviadocker/sharermdcodeviadocker/analysis/report/index.qmd

Migrating Off Dropbox: Beyond Dotfiles

Ronald G. Thomas — Thu, 07 May 2026 07:00:00 GMT

A wooden tray on a plain workbench, with one large compartment on the left holding a single round glass weight and three narrower compartments on the right holding three different small objects, suggesting a single category being separated into three purpose-fit ones.

A portable dotfiles repository is only the first half of leaving Dropbox. The other half is the rest of the workflow.

Note

Multi-Machine Setup Series (recommended reading order): Part 1: Migrating off Dropbox | Part 2: Multi-Laptop Bootstrap | Part 3: Security Audit

Introduction

I did not really appreciate how much of my daily workflow depended on Dropbox until I tried to leave it. The dotfiles migration described in post 24 is the obvious half: pull the configuration files out of a cloud-mounted directory, put them under git, write an installer that creates symlinks. Once that work is committed, every dotfile in $HOME becomes version-controlled and reproducible on a new laptop with two commands.

The non-obvious half is everything else. The dotfiles deploy cleanly on a fresh machine, then the new shell tries to cd ~/prj/some-project and fails because the project tree still lives at ~/Library/CloudStorage/Dropbox/prj/. Backup scripts hardcode the same Dropbox path. AI-tool history files live under a Dropbox-synced symlink and accumulate conflict copies on every overlapping write between machines. The configuration is portable; the workflow is not.

This post is the second half of post 24. It frames the problem as three independent layers (dotfiles, project content, append-only history), surveys the sync mechanisms each layer can use, and walks through a migration sequence that does not break a running pipeline mid-move. The companion deliverable is a decision worksheet that takes a reader from current state to a concrete migration plan. The choice of whether to leave Dropbox at all is left to the reader; this post is about doing it deliberately rather than partially.

Status of this migration

This post documents an ongoing migration on the author’s own setup. At time of writing, Layer 1 (dotfiles) is complete: the repository exists at ~/dotfiles/, the installer is written and dry-run-tested, and the architecture below has been exercised end-to-end on this laptop. Layer 2 (project content) and Layer 3 (append-only history) are in progress. The framework is presented as it stands today; the ‘After completing this migration’ results section near the end is labelled as expected outcomes rather than claimed ones, and will be updated with measured numbers once the migration reaches a fresh laptop. The companion plan tracking this work is available in the dotfiles repository as MULTI_LAPTOP_PLAN.md.

Motivations

A new laptop bootstrap should be a one-command operation. Today, after running the dotfiles installer, half the workflow still requires installing Dropbox, signing in, and waiting for several gigabytes to materialise before anything functional happens.
Dropbox sync of append-only files (shell history, AI-tool history, editor undo files) routinely produces conflict copies on multi-machine setups. The 12 conflict copies of history.jsonl discovered in this account on 2026-05-02 are one symptom of an architectural mismatch.
Project content directories under Dropbox commingle with cloud-only artefacts that should not be synced (model output, large derived data, machine-specific caches). Excluding them via .dropboxignore is fragile and error-prone.
The set of services that compete for $HOME (Dropbox, iCloud Drive, Google Drive, OneDrive) keeps growing, and choosing one default for everything is increasingly suboptimal as each service specialises.
The work to reach ‘workflow independence’ is largely the same work as ‘workflow modernisation’: separating concerns by layer, choosing sync mechanisms that match each layer’s semantics, and documenting each move so it can be audited later.

Objectives

Inventory the workflow against three layers (dotfiles, project content, append-only history) and classify every sync-relevant artefact into exactly one layer.
Produce a decision worksheet that, for each layer, names the candidate sync mechanisms and the trade-offs that distinguish them.
Sequence the migration so that the backup pipeline never points at a stale source path and the launchd jobs stay valid throughout the move.
Establish a fresh-machine bootstrap that does not require Dropbox and verifies the entire workflow end to end.

This post documents the author’s own migration. If errors are spotted or better approaches are known, the comment thread below is the right place to note them.

A wooden desk seen from above with three small index cards arranged in a horizontal row, each card showing a hand-drawn symbol, suggesting classification before action.

What is the Problem?

The problem is that ‘sync everything via one cloud provider’ was a reasonable default in 2014, when Dropbox was the only practical option for arbitrary file sync, but it now bundles three semantically distinct concerns into a single mechanism. The three concerns:

Configuration distribution wants version history, atomic deploys, and the ability to roll back. Git solves this; cloud sync only approximates it.
Project content sync wants block-level diffs for large files, conflict semantics that match the file format, and the option to exclude derived artefacts. Different sync mechanisms handle these very differently.
Append-only history wants either single-machine writes or a sync mechanism that understands append semantics. Cloud sync writes the whole file on every change and produces a conflict copy on every overlap.

A useful analogy: it is the same reason a research compendium separates data/, R/, and analysis/. Each directory has different rules about what changes, how it changes, and who is allowed to change it. Lumping them all into one sync mechanism enforces no useful constraint and produces friction at the boundary.

A concrete example from this migration: the file ~/.claude/history.jsonl is an append-only log of prompts entered into the Claude Code CLI. Five machines (athena, Joe, julia.local, MacBook-Air.local, and the author’s primary laptop) write to it through a Dropbox-synced symlink. Most days, only one machine is active and the file syncs cleanly. The day two machines write within the same minute, Dropbox cannot merge the two append streams and creates a conflict copy. Across April 2026 alone, this produced four new conflict copies on this account, each holding several hundred KB of unique prompts; the running total since January is eleven. The file format is not the problem; the choice to sync an append log via a whole-file cloud provider is.

Prerequisites

This post assumes:

Operating system: macOS 13+ or a recent Linux distribution. The sync-mechanism table is applicable to both, but specific path conventions reference macOS.
Already in place: a working dotfiles repository per post 24. The architecture below builds on that foundation; without it, half the moves cannot be made cleanly.
Background knowledge: comfort with git remotes, basic launchd plist structure, rclone or rsync at the level of a single sync command. The post does not require expertise in any of these.
Hardware: a single primary laptop is sufficient. A second machine (or VM) is useful for the migration’s verification step but is not required.
Time required: about a day’s careful work to inventory and decide; about a half-day to execute the moves once the decisions are made.

On a single-machine setup where conflict copies have never appeared, the migration’s payoff is smaller and may not warrant the undertaking. The framework still applies, but Dropbox is unlikely to be causing friction in that scenario.

The Three Layers

Treat the workflow as three layers. Each layer has its own sync semantics, its own candidate mechanisms, and its own failure modes.

Layer 1: Dotfiles distribution

Solved by post 24. The summary, for reference: configuration files live in a git repository at ~/dotfiles/ (outside any cloud-mounted path), an install.sh creates symlinks into $HOME per machine, secrets stay out of the repository entirely. New-machine bootstrap is git clone … ~/dotfiles && cd ~/dotfiles && ./install.sh.

This layer is the easiest to migrate because the files are small, change infrequently, and are nearly always edited from a single machine. Git handles the multi-machine merge cleanly when overlap does occur.

What post 24 explicitly does not address: project content (the next layer), backup-pipeline source paths (also next layer), and append-only history (the third layer). Those failures show up only after the dotfiles layer is migrated, which is why this post exists.

Layer 2: Project content sync

Project content is the directory tree that holds research repositories, data, drafts, and archives. On a Dropbox-bound workflow this is ~/Dropbox/prj/ and similar siblings (docs/, sbx/, work/, shr/).

Project content has different sync characteristics from dotfiles:

It is much larger (tens of GB is typical).
It contains both small text (R, Quarto) and large binary (data, figures, PDFs).
It changes frequently, often on a single machine within a session.
It includes derived artefacts that should not be synced at all (renv libraries, model output, caches).

The candidate sync mechanisms, with the dimensions that matter:

Mechanism	Strength	Weakness
Per-project git remotes (GitHub, GitLab, Codeberg)	Diff-aware history, atomic deploys, free for small repos	Bad with large binary files (Git LFS quotas), each project must be a repo
Syncthing (peer-to-peer, no cloud)	No middleman, no quotas, sync-to-LAN is fast, files are local-first	Two machines must be online together to sync; conflict semantics are simpler than Dropbox’s but still file-level
rsync over SSH to a NAS or VPS	Bandwidth-efficient, deterministic, scriptable	One-way push (or hand-coded bidirectional); requires the destination to exist; no version history without extra work
iCloud Drive (native macOS)	Tight macOS integration, included with Apple ID	macOS-only; opaque sync timing; cannot exclude subdirectories cleanly
Google Drive (rclone or native)	Effectively unlimited storage on enterprise plans	Same whole-file model as Dropbox; conflicts behave the same way

Most readers benefit from a hybrid: per-project git remotes for code (the bulk of the value) and Syncthing or rsync for the data that does not belong in a public or even private repo. The hybrid wins because it lets each artefact go to the mechanism that matches its semantics.

The single most consequential observation in this migration: ‘project content’ is not a homogeneous category. Splitting it into ‘code’ and ‘data’ before choosing a mechanism makes every subsequent decision easier.

Reconstructable local caches: the Maildir case

A subset of what looks like Layer 2 data is in fact a local cache of a canonical source that already lives elsewhere. For these, the right move is not to choose a sync mechanism at all; it is to let each machine rebuild its copy from the authoritative source on demand.

The worked example: a Maildir-based mail setup with mutt, mbsync, and notmuch. The IMAP server holds the canonical mailbox. mbsync mirrors a subset of folders into a local Maildir at ~/.local/share/mail/ (or any other non-synced path; the XDG ~/.local/share// location is a sensible default). notmuch new indexes the Maildir into a local Xapian database. mutt reads the Maildir directly. On a second laptop, the same configs deploy via Layer 1, the same mbsync runs from a launchd job, the local Maildir gets repopulated from IMAP, and notmuch new rebuilds the index. There is nothing to sync between the two laptops; the IMAP server is doing the synchronisation by being the canonical source.

Putting the Maildir in Dropbox is wrong on four counts. It duplicates the work IMAP already does. It triples the disk footprint (IMAP + Dropbox + local). Maildir’s one-message-per-file format produces hundreds of thousands of small files that strain cloud-sync engines. And the notmuch Xapian database is a B-tree-on-disk format that is not safe to share between processes on different machines via file sync.

The same pattern applies to a broader class of local artefacts: Homebrew’s package cache (~/Library/Caches/Homebrew/), language-server indexes, browser caches, ccache, AI-model caches that hash to the same content given the same prompts. None of these need to be synced between machines; each is a derivative of an authoritative source and is rebuilt cheaply on demand. Treat them as per-machine state, exclude them from any sync mechanism, and add the rebuild step to the new-machine bootstrap if it does not happen automatically.

The new-laptop bootstrap for the Maildir case, after install.sh has deployed the configs:

# Restore IMAP/SMTP credentials (manual; from pass / keychain / 1Password).
# Then:
mbsync -a                        # populate ~/.local/share/mail/
notmuch new                      # build the Xapian index
launchctl bootstrap "gui/$UID" \
  ~/Library/LaunchAgents/local.mbsync.plist
launchctl bootstrap "gui/$UID" \
  ~/Library/LaunchAgents/local.notmuch.new.plist

Time depends on mailbox size; a 3 GB mailbox over residential broadband is roughly 30-60 minutes. The two scheduled jobs keep the Maildir current thereafter.

The one cost of this pattern is that mail is not available offline until the first mbsync -a completes. For mutt users this is rarely a meaningful constraint; for users who need immediate offline access on a fresh laptop, an rsync of a recent Maildir snapshot from the previous laptop into ~/.local/share/mail/ skips the IMAP fetch step.

Worked example: relocating an existing Maildir to ~/.local/share/mail/

A common starting point for long-time mutt users is a Maildir at ~/Mail/ (the convention from before the XDG Base Directory specification existed). The relocation to ~/.local/share/mail/ is mechanical, and the cache-and-index can be moved in place rather than re-downloaded. The steps below assume mutt + mbsync + notmuch and a single Gmail account at ~/Mail/gmail/; generalisation to other layouts is direct.

# 1. Stop any running mbsync. If a launchd job runs mbsync on a
#    schedule, bootout it temporarily.
pkill -f mbsync || true
launchctl bootout "gui/$UID/local.mbsync" 2>/dev/null || true

# 2. Cheap insurance: dump the notmuch tag database. Tags are user
#    state; the message index can be rebuilt, the tags cannot.
notmuch dump --output="$HOME/notmuch-dump-$(date +%Y%m%d).txt"

# 3. Move the Maildir AND .notmuch together. They must move
#    atomically to keep the Xapian database's relative paths valid.
mkdir -p ~/.local/share/mail
mv ~/Mail/gmail   ~/.local/share/mail/gmail
mv ~/Mail/.notmuch ~/.local/share/mail/.notmuch

# 4. Update the dotfiles repo (Layer 1, single source of truth)
#    BEFORE editing any live config under $HOME. Edit:
#       ~/dotfiles/editors/notmuch-config         (path = $HOME/.local/share/mail)
#       ~/dotfiles/config/mutt/muttrc             (folder, spoolfile, postponed, record)
#       ~/dotfiles/mbsyncrc                        (Path, Inbox under each MaildirStore)
#    Commit and push. (The dotfiles repo is OUTSIDE cloud-mounted
#    paths per Layer 1, so in-place editors are safe to use here.)

# 5. Reflect the dotfiles changes into $HOME (idempotent).
cd ~/dotfiles && ./install.sh

# 6. Verify.
mbsync -a                  # Resumes incrementally; should NOT re-download.
notmuch search '*' | head  # Confirms the index resolves message paths.
mutt -e 'exit'             # Opens and exits cleanly with the new paths.

# 7. Re-enable the scheduled mbsync once verification passes.
launchctl bootstrap "gui/$UID" \
    ~/Library/LaunchAgents/local.mbsync.plist

Two points worth flagging:

The notmuch Xapian database stores message paths relative to database.path. Moving the Maildir and .notmuch together preserves those relative paths, so notmuch new is not required. If the two directories are moved separately for any reason, the index becomes invalid and notmuch new is needed to rebuild it.
Step 4 (update the dotfiles repo first) is the step the obvious version of this recipe omits. Editing the live ~/.config/mutt/muttrc directly before updating the repo copy makes the two diverge. A second laptop bootstrapped from the repo today would deploy a muttrc that points at a Maildir that does not exist. Always edit the repo file, then install.sh.

The pre-XDG location (~/Mail/) can be deleted after a few days of confirmed working state; keep it around briefly as a rollback target.

Layer 3: Append-only history files

This layer is small in bytes but disproportionate in friction. The relevant files:

~/.zsh_history (shell command history)
~/.viminfo (vim’s last-used files, registers, marks)
~/.claude/history.jsonl (Claude Code prompt history)
Editor undo files (~/.local/state/vim/undo/)
AI-tool histories from any other tools (~/.gemini/, etc.)

Each is append-only at the application level and whole-file at the sync layer. That mismatch is what produces conflict copies. There are three architecturally honest choices:

Per-machine state. Do not sync these files at all. Accept that arrow-key history on one laptop does not include commands typed on another. Simplest; works for most users; what most non-Dropbox dotfile setups assume.
Single-machine writes. Designate one machine as the canonical owner; the others write to local files that are never synced. Practical only if the user genuinely uses one machine 95% of the time.
Append-aware sync. A few sync mechanisms (atuin for shell history, custom JSONL mergers for AI logs) understand append semantics and can merge two divergent streams. More moving parts, but eliminates the conflict-copy class entirely.

The right choice depends on the user. Most readers will be happiest with option 1 plus, optionally, atuin for shell history because it is genuinely useful across machines and well-engineered for the multi-machine case.

The Decision Worksheet

The companion deliverable for this post is a Quarto document at docs/migration-decision-worksheet.qmd. It walks the reader through:

Inventory. List every sync-relevant artefact in $HOME and ~/Dropbox/. Classify each into one of the three layers.
Mechanism selection. For each layer, pick a sync mechanism (or ‘no sync, per-machine’) from the table above.
Migration order. Sequence the moves so the backup pipeline never points at a stale path. The general rule is: dotfiles first (already done if post 24 was followed), then a single project as a pilot, then the rest of project content, then history files.
Verification. Bootstrap a clean user account or VM and confirm the full workflow comes up without Dropbox.

The worksheet is generic; it does not assume any particular project structure or sync mechanism. Completing it for a given setup converts the migration into a series of small, audited moves rather than a single large rewrite.

Migration Sequence

The order of operations matters because the backup pipeline must remain valid at every intermediate step. The sequence below comes from running this migration on a real workflow.

Confirm dotfiles foundation. Verify post 24’s repository works. Run install.sh --dry-run on a clean account or VM. Resolve any failures before proceeding.
Introduce a path-abstraction env var. Add PRJ_ROOT to ~/.zshenv with a fallback that resolves to ~/Dropbox/prj for now. Replace literal ~/Dropbox/prj references in scripts and configs with $PRJ_ROOT. Nothing visible changes, but the next move is now a single-line edit.
Pilot one project. Pick a single small repo. Move it from ~/Dropbox/prj// to ~/prj// (real path, outside cloud). Push to a private git remote if it is not already. Confirm the workflow on it.
Migrate the rest of project content in batches. A batch a day prevents the backup pipeline from going stale. For each batch: move, repoint any in-script paths, push to remotes, verify.
Repoint backup sources. Once $PRJ_ROOT resolves to the new path, edit backup-gdrive and backup-icloud (or their equivalents) to source from $PRJ_ROOT instead of the literal Dropbox path. The launchd plists themselves do not change; only the scripts they call.
Resolve append-only history. Pick option 1, 2, or 3 from the previous section. Apply per-file. Document the choice in the dotfiles repo’s README for future reference.
Verify on a clean account. Boot a fresh user, clone dotfiles, run installer, set up project remotes, confirm the workflow. The first surprise reveals the assumption that was hiding.
Optional: uninstall Dropbox. Most users do not need to actually remove Dropbox. Once the workflow is independent, Dropbox can stay installed as a backup or for sharing files with colleagues, without being load-bearing for the workflow itself.

The sequence is robust to interruption. Stopping after step 3 leaves a working pilot with the backup pipeline intact. Stopping after step 5 leaves the workflow independent; the history-file decision can be deferred.

Two open-top boxes side by side on a workbench, the left box full of mixed objects and the right box divided into three sorted compartments, with a single object mid-transfer between them.

Things to Watch Out For

Seven gotchas to anticipate during the migration. Each lists the symptom and the fix.

The backup pipeline goes stale silently. Symptom: backup-gdrive runs every night, exits 0, but the syncthing folder it backs up has not changed in a week because the source path no longer exists. Fix: rclone sync exits 0 even when the source is empty, so add an explicit [[ -d $PRJ_ROOT ]] guard at the top of every backup script and fail loud on missing source.
Cloud-mounted git working trees corrupt their indexes. Symptom: git status reports phantom changes; git fsck finds dangling refs; the working tree is inconsistent. Fix: never put ~/dotfiles/ inside ~/Dropbox/, ~/Library/CloudStorage/..., or ~/Library/Mobile Documents/. Git’s frequent writes to .git/index race with the cloud provider’s sync engine. The repo MUST be on a non-synced filesystem path.
Symlinks to Dropbox break asymmetrically across machines. Symptom: ~/.config/git -> ~/Library/CloudStorage/Dropbox/dotfiles/config/git on machine A, but on machine B the same symlink is dangling because Dropbox is at a different path or not installed. Fix: symlinks created by install.sh should target $DOTFILES (an env var resolved at install time), not literal Dropbox paths.
Append-only history conflict copies do not announce themselves. Symptom: a file named history (machine's conflicted copy YYYY-MM-DD).jsonl appears next to the canonical file, with no notification, no error, no merge prompt. Fix: a weekly grep for ‘conflicted copy’ across the Dropbox tree, or an explicit migration of the affected files out of cloud sync.
launchd plist substitutions overwrite themselves. Symptom: an edit to ~/dotfiles/launchd/local.backup.research.plist does not change the running job, because install.sh sed-substitutes __USER__ to $USER when writing to ~/Library/LaunchAgents/, and the active plist is the substituted copy. Fix: do not edit the active plist directly; edit the source in ~/dotfiles/launchd/, then re-run install.sh plus launchctl bootout and launchctl bootstrap.
Project content includes runtime artefacts that should not move. Symptom: a 200 MB renv/library/ directory comes along with a project move and now lives at ~/prj//renv/library/, where renv expects to manage it. Fix: confirm .gitignore excludes the right paths before pushing; add **/renv/library/ to the dotfiles repo’s .gitignore if one is accidentally tracked.
In-place editors race with cloud sync and truncate files to zero bytes. Symptom: sed -i.bak file, awk -i inplace, or perl -i against a path under ~/Library/CloudStorage/... returns 0 but leaves the file empty. The .bak saved by -i.bak is also placed in cloud storage and is subject to the same race; on the incident that produced this gotcha, both the file and its backup were lost locally and required Dropbox version-history restore. Fix: never run -i-style editors on cloud-mounted paths. Either use the Edit tool / a real editor, or copy to /tmp, edit, and mv back. (If the dotfiles repo is on a non-synced path per gotcha 2, this hazard is confined to scripts that touch ~/Library/CloudStorage/... from elsewhere.)

Uninstall / Rollback

The most common ‘undo’ is not ‘reinstall Dropbox’ but ‘put project content back under Dropbox while keeping dotfiles independent’. The architecture supports this directly because the layers are independent.

To roll project content back into Dropbox:

Move ~/prj// back to ~/Dropbox/prj//.
Update ~/.zshenv so PRJ_ROOT resolves to the Dropbox path.
The dotfiles repo, backup scripts, and launchd plists do not need any change because they reference $PRJ_ROOT.

This is also the recommended path for a reader who completed the dotfiles work but is not yet ready to migrate project content. Set up the env var, leave the path pointing at Dropbox, defer the project move until later.

To roll back the entire migration: the dotfiles installer’s --dry-run should never have shipped destructively, so the rollback is just rm -rf ~/dotfiles plus restoring $HOME from a Time Machine backup. Most readers will not need this.

A closed wooden chest with three small brass clasps on a stone surface, an open notebook and a capped fountain pen beside it, suggesting a deliberate and completed reorganisation.

What Did We Learn?

Lessons Learnt

Conceptual:

The ‘sync everything via one cloud provider’ default conflates three independent concerns. Separating them makes each easier to reason about.
Migration sequencing matters more than mechanism choice. The wrong mechanism is correctable; a half-migrated workflow is hard to leave for a year.
Per-machine state is a feature, not a bug, for many file types. The right answer for shell history is often ‘do not sync’, not ‘sync better’.

Technical:

A single environment variable ($PRJ_ROOT) provides enough indirection to migrate path references in two passes: first introduce the variable while it still resolves to Dropbox, then change what it resolves to.
launchd plists with __USER__ placeholders deploy cleanly across machines via install.sh sed-substitution; bootstrapping the substituted copy avoids machine-specific files in the dotfiles repo.
Atomic git operations on a non-synced filesystem are reliable; the same operations on a cloud-synced filesystem race with the sync engine and produce subtle corruption.

Gotchas:

‘rclone sync exits 0’ does not mean the sync succeeded; it can mean the source had nothing to send because the path is wrong.
A symlink created on one machine resolves correctly on that machine but may dangle on another. Always test the new-machine case.
Conflict copies of append-only files accumulate silently. Periodic auditing is the only way to notice.

Limitations

The migration described here covers a single-user workflow. Team setups (where the same Dropbox folder is shared with collaborators) need additional considerations not addressed in this post.
Project content moved from Dropbox to per-project git remotes is now subject to GitHub or GitLab quotas, including LFS limits for binary files. Plan accordingly.
The decision worksheet is a generic framework; it does not generate scripts for any particular sync mechanism. Each reader still has to write the rclone or rsync commands for their setup.
Append-aware history sync (option 3) requires either an existing tool (atuin works for shell history) or hand-rolled mergers for proprietary log formats. The post does not provide those mergers.
Migrating off Dropbox does not, by itself, improve the security posture of credentials. Files like ~/.aws/credentials should be addressed separately; cloud sync was not their only exposure.
The post assumes macOS or Linux. Windows-specific path conventions and sync semantics are out of scope.

Opportunities for Improvement

Extend the decision worksheet into a small CLI tool that audits a current setup and generates a draft migration plan.
Build an atuin (or atuin-equivalent) integration template for the dotfiles repo, so option 3 from layer 3 becomes a one-line opt-in.
Add a verify.sh to the dotfiles repo that runs the end-to-end check (dotfiles installed, $PRJ_ROOT valid, backup scripts pointing at it, launchd jobs loaded) on demand and reports pass/fail.
Document a canonical ‘two-laptop’ setup with Syncthing replacing the Dropbox role for shared project content.
Generalise the worksheet’s mechanism table into a maintained list (a small repo or a wiki page) that adds new tools as the ecosystem evolves (Filen, S3-backed, etc.).

Wrapping Up

A portable dotfiles repository is the first half of a Dropbox-free workflow; this post is the framework for the second half. The key idea: stop treating sync as a single dimension and start treating it as three layers (configuration, project content, append-only history) with different semantics and different right answers.

Most of the work is in the inventory and decision step, not in the moves themselves. Once each artefact is classified and each layer has a chosen mechanism, the actual migration is a sequence of small file moves and a few text edits. The pieces that take time are testing the new-machine bootstrap, deciding what to do about history files, and resisting the temptation to migrate everything in one weekend.

Expected outcomes

The architecture aims to deliver three measurable end states. The first is verifiable now (Layer 1 is complete on the author’s primary laptop); the other two will be confirmed and updated with measured numbers when Layer 2 and Layer 3 land.

New-laptop setup is git clone … ~/dotfiles && ./install.sh && (per-project git clones). The plan budget is roughly 30 minutes from clean machine to working environment; the actual figure will be added after a fresh-VM bootstrap is run.
Conflict copies of history.jsonl and similar files should stop recurring once the affected files are moved out of Dropbox sync. The post will be updated with the post-migration count once Layer 3 is applied.
Backup pipelines will source from $PRJ_ROOT, which can point anywhere; once Layer 2 lands, Dropbox is no longer load-bearing for the workflow.

In conclusion, four points merit emphasis. First, the dotfiles layer is the wedge, not the goal; once it is in place, the rest of the migration is mechanical. Second, ‘project content’ is two layers in disguise (code, which wants git, and data, which wants something else); the two should be separated before a mechanism is chosen. Third, some of what looks like ‘data’ is actually a local cache of an authoritative source elsewhere (Maildir from IMAP, Homebrew cache, language-server indexes); for these the right answer is not to sync at all, but to rebuild on each machine from the canonical source. Fourth, append-only files are the only category that benefits from sophisticated sync; for everything else, simpler is better.

Reproducibility

Tested on:

Component	Version
OS	macOS 15.4 (Sequoia)
Shell	zsh 5.9
git	2.45
rclone	1.68
Syncthing	1.27
flock (homebrew)	2.40
Date verified	2026-05-07

The decision worksheet is a generic Quarto document and does not depend on any particular tool versions; the table above captures the environment in which the framework was developed and validated.

Let’s Connect

I would enjoy hearing from you if:

You have made the same migration with different sync-mechanism choices and want to compare notes.
You see a gotcha I missed, especially in the append-only-history layer.
You use Linux or Windows and want to extend the worksheet to your environment.
You just want to say hello and connect.

Rendered on 2026-05-08 at 06:10 PDT.
Source: ~/Dropbox/prj/qblog/posts/64-migrating-off-dropbox/migrating-off-dropbox/analysis/report/index.qmd

A tiered CI strategy for zzcollab research compendia

Ronald G. Thomas — Wed, 06 May 2026 07:00:00 GMT

A layered set of frosted glass panels stacked at increasing depth, each carrying a fainter ink mark, evoking the verification tiers a CI workflow stacks between a commit and a deploy.

Three independent verification tiers, each with its own failure mode and its own definition of ‘passing’.

Correction – May 2026

Two recommendations in this post were superseded by findings from multi-repo deployment in May 2026. Both are annotated inline below at the relevant sections.

r-lib/actions/setup-renv@v2 fails on zzcollab research compendia. All zzcollab compendia include an .Rprofile gate that suppresses renv activation outside a Docker container (ZZCOLLAB_CONTAINER=true must be set). When setup-renv@v2 calls renv::restore() on a host runner, the gate fires, renv declines to activate, and no packages are installed. The CI job either silently passes (nothing installed, nothing verified) or fails with misleading errors. The corrected approach for r-package.yml is r-lib/actions/setup-r-dependencies@v2, which reads DESCRIPTION directly and never invokes renv.

renv::status()$synchronized is an unstable internal field. After a renv version downgrade during renv::restore() (for example, from 1.2.3 to 1.2.2), the field returns FALSE regardless of actual synchronisation state, producing spurious CI failures. The quit(status = 1) guard built on this field should be removed. Relying on renv::restore()’s exit code is sufficient.

The current working architecture (r-package.yml v2.7.0) is documented in Post 69: zzcollab GitHub Actions workflows.

Introduction

I did not realise how much my CI was lying to me until I added a single explicit failure check and watched five ‘passing’ projects turn honestly red. The framework’s default workflow for the zzcollab ecosystem (a Docker-plus-renv pattern) had been showing green checkmarks across the projects I cared about, and I had been trusting those checkmarks the way one trusts the brake light on a car. The revelation was that some of the brake lights were on green because the bulb was burnt out, not because anything had been verified.

The discovery happened almost by accident. I was migrating one research compendium to a tighter CI pattern (r-lib/actions/setup-renv@v2 plus an explicit quit(status = 1) when renv::status() reports inconsistency) and the first run failed. Reading the log I saw a table titled The following package(s) are in an inconsistent state: listing dozens of packages installed and used by source code but never recorded in the lockfile. The previous CI had not been catching this; the new check did. When I extended the migration to four more repositories the same shape of finding appeared in three of them, plus two distinct categories of project-side regression that had been similarly invisible.

This post walks through the migration: a tiered CI model with one template per zzcollab workspace type, an explicit synchronisation gate for lockfiles, and a strategy for getting the framework out of the business of reporting fictional health. The companion white paper (mirrored in the zzcollab repository) records the analytic detail; this post is the procedural distillation.

Motivations

Green checkmarks were structurally misleading. A workflow that prints findings without erroring on them looks identical, in GitHub’s UI, to a workflow that found nothing wrong.
The framework’s single CI default conflated four workspace types. Tool packages, LaTeX manuscript compendia, Quarto compendia, and Quarto blog posts have substantively different CI needs. Applying the same template to all four meant some projects were over-checked (blog posts forced through R CMD check) and some were under-checked (compendia whose render was never tested).
Lockfile drift was the dominant silent regression. Three of five projects in the multi-repo verification had renv.lock pinning many fewer packages than their source actually used. The previous CI did not detect this; researchers were running in environments their lockfiles could not reproduce.
Posit binary identifiers age out of the registry. Lockfiles that pin specific Posit Package Manager binary builds (the -N suffix) eventually fail when those builds are retired from the snapshot. This pattern recurred across two unrelated repos and two different packages (Rcpp 1.1.1-1, S7 0.2.1-1).
The default R version is not stable. setup-r@v2’s r-version: 'release' resolves to whatever is current; lockfiles pin a specific older version; the mismatch surfaces as opaque compilation errors against the wrong header set.

Objectives

By the end of this migration, the zzcollab project will have:

A workspace-type-appropriate CI template chosen from the four-row typology. Tool packages get one shape; the three compendium variants get tiered models with the right Tier 3 renderer.
A Tier 1 lockfile-validation job that fails informatively when renv.lock is out of sync with declared dependencies or actually-used source-code references.
R-version pinning sourced from the lockfile (r-version: 'renv') so the CI environment matches what the project’s developers run locally.
Posit Package Manager URLs that resolve to current binaries, either via date-pinned URLs in renv.lock or via the RENV_CONFIG_REPOS_OVERRIDE env var that setup-renv@v2 sets automatically.

A coffee setting in a quiet workspace, the morning’s first mug poured before the day’s verification work begins. Placeholder imagery; topic-specific replacement pending.

What is the tiered CI model?

A workflow design in which CI is split into independent jobs by verification purpose, each gated on a different cadence and each catching a different class of regression. The architectural analogy is a building’s life-safety inspections: the structural inspection fires once at construction, the fire-alarm test fires periodically, and the daily lock-up check fires at every closing. Each is a separate inspection with a separate sign-off.

In zzcollab CI specifically, three tiers cover most needs. Tier 1 (validate) verifies that the lockfile, the declared dependencies in DESCRIPTION, and the source code’s actual imports agree. Tier 2 (check) is the conventional R CMD check. Tier 3 (render) renders the project’s primary deliverable document (the manuscript or blog post). Workspace type determines which tiers apply: tool packages skip Tier 3, blog posts skip Tier 2, compendia run all three.

Prerequisites

A zzcollab project (compendium or tool package) with a populated renv.lock, DESCRIPTION, and either an analysis/report/ directory or a top-level index.qmd.
A GitHub repository with Actions enabled.
The gh CLI installed locally for triggering and reading runs.
R 4.4 or later locally for renv::snapshot() and renv::status() invocations.
About one hour of attention per repository for the first migration; subsequent migrations take 10-15 minutes once the patterns are in hand.

Migrating an existing zzcollab project

The migration is the same five steps for every workspace type; only the workflow contents differ. Pick one repository to start with and walk it end to end before applying the pattern to others.

Step 1: Identify the workspace type

Inspect the project’s deliverable to choose one of four templates:

analysis/report/report.Rmd exists with output: pdf_document in the YAML header: LaTeX manuscript compendium.
analysis/report/index.qmd (or top-level index.qmd) has document-type: "blog" in the YAML header, or the project lives inside a Quarto blog tree: Quarto blog post.
analysis/report/index.qmd (or report.qmd) without the blog marker, targeting HTML or PDF via Quarto: Quarto analysis compendium.
None of the above; only R/, tests/, man/ populated: tool package.

File presence alone is unreliable for this decision. zzcollab scaffolds the same R-package skeleton (DESCRIPTION, NAMESPACE, R/, tests/) under every paradigm, so the discriminating signal is the YAML header of the primary document, not the directory structure.

Step 2: Replace the workflow

The four templates are recorded in the white paper and reproduced in analysis/configs/ in this companion compendium. The shape common to all four is:

name: 

on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]

jobs:
  validate:
    runs-on: ubuntu-latest
    env:
      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - uses: r-lib/actions/setup-r@v2
        with:
          use-public-rspm: true
          r-version: 'renv'
      - uses: r-lib/actions/setup-renv@v2
      - name: Report renv status (fail on inconsistency)
        shell: Rscript {0}
        run: |
          status <- renv::status()
          if (!isTRUE(status$synchronized)) {
            cat("renv reports the project is not synchronized.\n",
                "Run renv::snapshot() locally and commit ",
                "the updated renv.lock.\n",
                sep = "")
            quit(status = 1)
          }
          cat("renv: lockfile, library, and source are synchronized.\n")

Correction (May 2026)

r-lib/actions/setup-renv@v2 does not work on zzcollab research compendia. The zzcollab .Rprofile gate suppresses renv outside a Docker container (ZZCOLLAB_CONTAINER=true), so setup-renv@v2 cannot install the lockfile on a host runner. The corrected r-package.yml template uses r-lib/actions/setup-r-dependencies@v2 (reads DESCRIPTION, no renv involvement) together with r-lib/actions/check-r-package@v2. See Post 69 for the complete v2.7.0 template.

Three points warrant emphasis. First, r-version: 'renv' reads the R version from the lockfile rather than defaulting to whatever release happens to be. Second, the explicit quit(status = 1) on !isTRUE(status$synchronized) is what turns the silent print into a visible failure. Third, the workflow does not run a Docker container; setup-renv@v2 handles renv installation, lockfile restore, and GitHub Actions caching itself. The Docker container remains useful for local development reproducibility but is not needed in CI.

Step 3: Push and observe failures

Commit and push the new workflow. The first run almost certainly fails. This is expected and informative; the failure mode indicates which class of project-side regression is present.

git add .github/workflows/
git commit -m "Migrate to new-generation tiered CI"
git push origin main
gh run list --limit 5

Examine the failed run with gh run view --log-failed. The failures fall into a small number of categories.

Step 4: Address project-side findings

For each failure category, the fix is project-side, not CI-side. Section ‘Things to Watch Out For’ below catalogues the patterns observed across the multi-repo verification and the action each implies. Once the project-side issues are repaired, the same workflow will pass cleanly.

Step 5: Verify and propagate

Once one repository is green, apply the same workflow template (adjusted for workspace type) to the other repositories in the same project family. The mechanical work shrinks dramatically: copy the YAML, push, observe, fix project-side issues that surface.

A coffee scene mid-pour, the kind of small precision the migration above is asking of CI. Placeholder imagery; topic-specific replacement pending.

Things to Watch Out For

Six recurring patterns to expect during this migration, each with a specific symptom and a specific fix. The first three were observed directly in the five-repository verification; the remaining three are documented in the companion white paper from earlier diagnostic work. None of these are CI-infrastructure noise; all are project-side regressions that the previous CI pattern was masking.

Symptom 1: renv::status() lists packages in ‘inconsistent state’ but the workflow shows green. This was the founding discovery. renv::status() prints findings and returns a list; it does not raise an error. Without an explicit quit(status = 1) on !isTRUE(status$synchronized), the workflow exits 0 and GitHub marks the run successful regardless of how much drift the log records. Fix: the explicit guard shown in Step 2 above.

Correction (May 2026)

The renv::status()$synchronized field is not a stable API. After a renv version downgrade during renv::restore() (observed when the lockfile pins renv 1.2.2 but the host runner installs 1.2.3 first, then downgrades), the field returns FALSE regardless of actual synchronisation state. The quit(status = 1) guard therefore produces spurious CI failures on an otherwise clean project. Relying on renv::restore()’s own exit code, and omitting the renv::status() check entirely, is the more robust pattern.

Symptom 2: ‘package(s) are in an inconsistent state, installed: y, recorded: n, used: y’. The lockfile is incomplete. Source code references packages that the lockfile does not record; those packages are present in the project library because something (usually setup-renv@v2’s aggressive transitive resolution, or a prior renv::install()) put them there. Fix: locally, renv::snapshot() to capture the actual library, then commit the updated renv.lock. The CI should then go green honestly.

Symptom 3: Error: failed to install '', where the version has a -N suffix (Rcpp 1.1.1-1, S7 0.2.1-1). Posit Package Manager has retired the specific binary build the lockfile pins. Fix: locally, refresh the lockfile against the current Posit URL via renv::snapshot(), or pin R$Repositories[0].URL in renv.lock to a date-stamped Posit URL with the OS segment (__linux__/noble/) so binaries are reproducible from a fixed snapshot.

Symptom 4: Compilation errors against R headers, e.g. R_NamespaceRegistry was not declared or R_ext/PrtUtil.h: No such file or directory. The CI is running a different R version than the lockfile pins. The default r-version: 'release' resolves to the current release at run time, which drifts forward as new R versions ship. Fix: add r-version: 'renv' to setup-r@v2’s with: block. This reads the lockfile and matches the pinned version exactly.

Symptom 5: R CMD check found WARNINGs for things like ‘Undocumented code objects’ or ‘Consider adding importFrom’. The previous CI used error_on = 'error' in its hand-rolled rcmdcheck call, which let WARNINGs pass. r-lib/actions/check-r-package@v2 errors on WARNINGs by default. Fix: address the warnings (documentation gaps, NAMESPACE imports, examples). These are real package-quality regressions; the fact that CI surfaced them is the point.

Symptom 6: A workflow with the right structure but fires on every push, including pushes that change only the manuscript. The render tier is expensive (5-15 minutes) and should not run on every push. Fix: path-filter the render workflow to trigger only on changes under analysis/, R/, DESCRIPTION, or renv.lock. The split between r-package.yml (always-on validation) and render-report.yml (path-filtered render) is the standard zzcollab pattern.

Symptom 7 (Docker-anchored compendia only): renv.lock after renv::snapshot() records macOS binary fingerprints instead of Linux binaries, and CI or a new collaborator’s renv::restore() installs different package builds than the developer tested. The lockfile was snapshotted on the developer’s macOS host. renv::snapshot() captures the installed library of the running session; if that session is macOS R, the lockfile records macOS binary hashes that a Linux runner cannot reproduce. The lockfile then describes an environment that only the original developer on that specific machine can fully reproduce, which violates the core zzcollab reproducibility guarantee.

For Docker-anchored compendia the answer is unambiguous: snapshot inside the Docker container. The reasoning rests on a five-pillar coherence check:

Dockerfile – defines the computational environment (R version, OS, system libraries).
renv.lock – must record packages that install cleanly inside that environment, meaning Linux binaries against the container’s system libraries.
.Rprofile – bootstraps renv on every session start.
R/ and analysis/ – the source code.
Raw data – the fixed input.

All five must cohere around the single environment the Dockerfile defines. The Dockerfile is the anchor; the lockfile follows it. A lockfile snapshotted outside the container has slipped its anchor.

Fix: enter the container with make r, ensure all required packages are installed (renv::restore() if the library is not yet populated), then run renv::snapshot(). The lockfile will then record Linux binary fingerprints that match both the container environment and the Ubuntu CI runner.

Consequence for CI design: for a Docker-anchored compendium, running CI inside the container (as the old-generation rocker/tidyverse-based workflow does) is the more principled choice, not a legacy pattern to be discarded. The new-generation setup-renv@v2 pattern trades the container’s reproducibility guarantee for CI simplicity – a trade worth making for tool packages and pure-renv projects, but one to evaluate carefully for a compendium whose five pillars are anchored to a Dockerfile. The recommended next step when migrating a Docker-anchored compendium is therefore not to discard the container-based CI but to tighten it: add the explicit renv::status() synchronisation guard (Symptom 1) inside the container job rather than switching to a containerless workflow.

Symptom 8 (Docker-anchored compendia with mounted TinyTeX): sh: 1: xelatex: Permission denied even after chmod -R a+rx $HOME/.TinyTeX on the host runner. This one has two compounding causes that must be understood separately.

Cause A: chmod -R on Linux does not follow file symlinks. In a TinyTeX installation, ~/.TinyTeX/bin/x86_64-linux/xelatex is a symlink pointing into ~/.TinyTeX/texmf-dist/bin/x86_64-linux/. Running chmod -R a+rx $HOME/.TinyTeX/bin changes the mode of the symlinks themselves, not the targets. The actual ELF binaries in texmf-dist/ may remain non-executable. Expanding the scope to chmod -R a+rx $HOME/.TinyTeX does recurse into texmf-dist/ (because chmod -R does follow directory symlinks), so it addresses Cause A – but Cause B still blocks execution.

Cause B: /root itself is 700. The strategy of mounting TinyTeX into the container as -v $HOME/.TinyTeX:/root/.TinyTeX places the tree under /root inside the container. A zzcollab Dockerfile ends with USER analyst (or equivalent non-root username), so the container process runs without root privileges. The Linux kernel checks directory execute-permission at each component of a path; /root being 700 means that even if every file inside /root/.TinyTeX/ has world-execute permission, the non-root user cannot traverse into it. The error sh: 1: xelatex: Permission denied reflects the kernel blocking path traversal, not a missing execute bit on the binary.

Fix: mount TinyTeX to a world-accessible path rather than under /root. Replace the -v and -e PATH arguments in the docker run call:

# Before (fails for non-root container user)
-v $HOME/.TinyTeX:/root/.TinyTeX \
-e PATH="/root/.TinyTeX/bin/x86_64-linux:$PATH" \

# After (world-accessible mount point)
-v $HOME/.TinyTeX:/opt/tinytex \
-e PATH="/opt/tinytex/bin/x86_64-linux:$PATH" \

/opt is owned by root with permissions 755, so all users can traverse it. The chmod -R a+rx $HOME/.TinyTeX step on the host runner is still needed to ensure the files themselves are world-readable and executable. The two lines together – chmod on the host, mount to /opt/tinytex in the container – resolve both causes.

Symptom 9 (cross-repo portability): A render-report.yml that works for one compendium fails for another because one uses rocker/verse (TinyTeX built-in, runs as root) and the other uses rocker/tidyverse (no LaTeX, non-root user). Projects in the same portfolio may have different Dockerfile base images. Applying a workflow that was fixed for one base to another without adjustment reproduces the original failure.

The /opt/tinytex host-mount pattern from Symptom 8 generalises to both cases and can serve as the single standard template across all compendium repos:

rocker/verse-based repos (root user): The image already ships TinyTeX at /root/.TinyTeX with its bin directory on PATH. The host mount at /opt/tinytex is redundant but harmless: xelatex is found in the container’s own PATH entry before the explicit -e PATH prefix is reached. Both LaTeX sources exist simultaneously; the container’s built-in takes precedence.
rocker/tidyverse-based repos (non-root user): The image has no LaTeX. The host mount at /opt/tinytex is the only source of xelatex. Because /opt is world-traversable (755), the non-root user can access the binaries.

The three-step block that achieves this generalisation:

- name: Install TinyTeX on host runner
  uses: r-lib/actions/setup-tinytex@v2

- name: Make TinyTeX binaries accessible to container
  run: chmod -R a+rx $HOME/.TinyTeX

- name: Restore packages and render all manuscripts
  run: |
    docker run --rm -e CI=true \
      -v ${{ github.workspace }}:/project \
      -v $HOME/.TinyTeX:/opt/tinytex \
      -e PATH="/opt/tinytex/bin/x86_64-linux:$PATH" \
      -w /project compendium-env \
      Rscript -e '...'

Fix: adopt this three-step block as the standard render-report.yml template across all Docker-anchored compendium repos, regardless of Dockerfile base image. The rocker/verse case incurs a small overhead (downloading TinyTeX on the host runner, ~30 seconds) that is acceptable in exchange for a single maintainable template.

Symptom 10 (ggplot2/patchwork version skew): object 'is_ggplot' is not exported by 'namespace:ggplot2' during patchwork installation, even when patchwork appears to be a current CRAN release. This is a compound failure that surfaces when the Docker base image ships ggplot2 3.5.x and the renv lockfile (or the PPM snapshot) pulls patchwork 1.3.2 or later.

patchwork 1.3.2 added @importFrom ggplot2 is_ggplot to its NAMESPACE – it imports is_ggplot as an exported symbol from ggplot2. ggplot2 3.5.1 (which ships with rocker/verse:4.4.2) does not export is_ggplot; the function was introduced as an export in ggplot2 4.0.0. Installing patchwork 1.3.2 into a container that has ggplot2 3.5.1 therefore fails at the ‘prepare package for lazy loading’ step with the export error.

The failure looks like it belongs to zzlongplot or the project’s own code because the error appears during zzlongplot’s installation (which imports patchwork). But the root cause is one level deeper: patchwork itself cannot load against ggplot2 3.5.1.

Fix: update ggplot2 to 4.0.x before (or alongside) installing patchwork. In the lockfile-rebuild script:

pkgs <- c("ggplot2", "patchwork", ...)  # include ggplot2 explicitly
install.packages(pkgs,
  repos = "https://packagemanager.posit.co/cran/__linux__/noble/latest")

ggplot2 4.0.x pulls in S7 as a new dependency – add S7 to the lockfile as part of the same rebuild. After this update, patchwork 1.3.2 loads cleanly and any package that depends on patchwork (such as zzlongplot) can be installed.

Portfolio implication: any zzcollab compendium that uses patchwork and is based on rocker/verse:4.4.2 (which ships ggplot2 3.5.1) will hit this failure when the lockfile is rebuilt against the current PPM snapshot. The fix – updating ggplot2 to 4.x – must be applied consistently across all affected repos during lockfile refresh.

Uninstall / Rollback

The migration is fully reversible. The previous workflow lives in git history; restoring it requires checking out the prior version of .github/workflows/r-package.yml (and any companion files) from the commit before the migration:

git log --oneline -- .github/workflows/r-package.yml
git checkout <prior-sha> -- .github/workflows/r-package.yml
git commit -m "Revert to prior CI workflow"
git push

If only the strict synchronisation check is the source of friction (for example, during a transitional period while several lockfiles are being repaired in parallel), the strict guard can be loosened to informational reporting by removing the quit(status = 1) line. The rest of the new generation pattern remains in place.

An empty cup beside a closed notebook, the morning’s verification work done and the cup ready for the next pour. Placeholder imagery; topic-specific replacement pending.

What did we learn?

Conceptual

CI checks for regressions, and different regressions need different checks. A workflow that runs R CMD check is necessary for a tool package and useless for a Quarto blog post whose deliverable is rendered HTML. The framework’s purpose determines the appropriate verification, not the framework’s default.
A green dashboard with red findings in logs is operationally indistinguishable from an absent CI check. The visible signal is the job exit status. A check that prints evidence of regression but exits 0 is a false sense of safety, not a check.
File presence is a noisy signal for workspace-type detection. zzcollab scaffolds the same R-package skeleton across all paradigms; the discriminating signal lives in the document YAML header (document-type: "blog", output: pdf_document).
The two-workflow split is an existing zzcollab pattern that should be propagated uniformly. The framework already ships r-package.yml and render-report.yml templates; three of eighteen surveyed compendium repos have both deployed. The contribution of this migration is uniform deployment, not a new design.

Technical

r-lib/actions/setup-renv@v2 handles renv installation, lockfile restore, GitHub Actions caching, and the RENV_CONFIG_REPOS_OVERRIDE env var in roughly twenty lines of YAML. It is the canonical R-package CI bootstrap; the Docker-in-CI pattern adds complexity that this action already resolves.
r-version: 'renv' on setup-r@v2 reads the R version from the lockfile. Without it, the workflow’s R version drifts forward with each new R release while the project’s pinned version stays put.
renv::status() returns a list with synchronized = TRUE/FALSE. Capture it explicitly and quit(status = 1) on inconsistency. The job’s exit status is what GitHub reads; nothing else matters.
r-lib/actions/check-r-package@v2 errors on WARNINGs by default. This is stricter than a hand-rolled rcmdcheck::rcmdcheck(error_on = 'error'). The strictness is a feature: it surfaces real package-quality regressions.

Gotchas

The default r-version drifts. setup-r@v2 resolves release at run time, so a workflow that worked yesterday can fail tomorrow when R 4.x.y is released. Pin to the lockfile.
Posit binary identifiers are not stable. A lockfile pinning Rcpp 1.1.1-1 will eventually fail when Posit retires that build. Prefer date-pinned URLs.
The .Rprofile auto-restore can race the workflow. The zzcollab template fires renv::restore() on R session start; the workflow also calls restore. Set ZZCOLLAB_AUTO_RESTORE=false in the workflow env: to give control to the workflow.
Stale render-report.yml variants exist in deployed repos. The templates/.github/ tree (since deleted from canonical zzcollab) shipped a hardcoded report.Rmd matcher; projects that received that variant fail silently for any project using manuscript.Rmd. Re-template via zzc doctor.

Limitations

Project-side fixes are still required. The new CI surfaces drift, missing documentation, and aged-out binaries; it does not repair them. Each surfaced issue requires a developer to address it.
Posit snapshot dates expire. Date-pinned URLs eventually age out (Posit retains snapshots for roughly three to five years). Long-lived projects need periodic snapshot refreshes.
Detection at scaffolding time is not yet automated. The decision among the four workspace templates currently requires a developer to inspect the project and pick. A future enhancement to zzc analysis or zzc doctor could automate this.
The renv::status() synchronisation check assumes renv.lock is the source of truth. Projects that intentionally maintain a smaller lockfile (for example, a package that lists only its declared Imports and trusts the user’s environment for everything else) will fail the synchronisation check by design. The check is appropriate for research compendia; it may not be for some other patterns.
The Docker container is no longer exercised by CI. Projects that rely on the Dockerfile for production deployment should consider adding a separate Docker-build verification job.

Opportunities for Improvement

Auto-detect workspace type at zzc analysis time. Inspect the YAML headers and emit the appropriate workflow template in .github/workflows/ automatically. This removes the manual decision in Step 1 of the migration.
Date-pin the lockfile URL at init time. When zzc initialises a project, write R$Repositories[0].URL as https://packagemanager.posit.co/cran/__linux__/noble/ rather than the unqualified source URL the framework currently emits.
zzc doctor propagation. Extend the doctor command’s full-content workflow replacement (introduced earlier in this work) to detect workspace type and replace stale render-report.yml variants with the canonical .qmd-aware version.
Output verification. Hash the rendered manuscript and fail Tier 3 if the hash differs from a recorded baseline. Catches regressions in figures and tables that pass R CMD check but produce different rendered output.
Schedule-driven freshness checks. Add a weekly cron job that re-runs Tier 1 against the current Posit URL, surfacing binary aging-out before it blocks an active push.
Documentation pass. The four workflow templates and the migration procedure deserve to be embedded in the canonical zzcollab user guide alongside the existing CI_WORKFLOW_ARCHITECTURE.md and ci-workflows-whitepaper.md documents.

Wrapping Up

The migration’s core contribution is making CI honest. The previous green checkmarks across these projects were structurally misleading; they reflected the absence of an error rather than the presence of verification. The new pattern’s first response on most projects is to turn red, but the red is informative and actionable, and the resulting fixes are real improvements to the projects’ reproducibility.

The pattern is not a new design. The two-workflow split (r-package.yml plus render-report.yml), the use of r-lib/actions/setup-renv@v2, and the path-filtering on the render workflow all already exist in zzcollab’s template set, applied inconsistently across deployed projects. The migration described here is the explicit deployment of those existing patterns uniformly, with the addition of a single explicit synchronisation guard that the framework’s templates had been missing.

The companion white paper records the analytic detail behind the choices made here, including the four-workspace typology, the verification of the synchronisation check at scale, and the classification of failure modes observed during the multi-repo deployment. Readers who want to understand why the templates have the shapes they do should consult that document; readers who want to apply the templates should follow this post.

In conclusion, four points merit emphasis. First, a CI check that prints findings without erroring is operationally invisible; the job’s exit status is the only signal GitHub shows. Second, four zzcollab workspace types map to four distinct CI templates (tool package, LaTeX compendium, Quarto compendium, Quarto blog post), and the document YAML header is the discriminating signal, not file presence. Third, r-lib/actions/setup-renv@v2 plus r-version: 'renv' plus a one-line quit(status = 1) guard on renv::status()$synchronized covers the bulk of what an always-on validation tier needs. Fourth, failures from the new pattern point to real project-side regressions (lockfile drift, aged-out Posit binaries, R-version mismatch, missing package documentation); none are CI-infrastructure noise.

Correction to Point 3 (May 2026)

Point 3 above reflects the intent of this post at time of writing, but both mechanisms named have since proved unreliable for zzcollab compendia. setup-renv@v2 cannot install packages on a host runner because the zzcollab .Rprofile gate suppresses renv outside the Docker container. renv::status()$synchronized misfires after renv version downgrades, producing spurious failures. The corrected r-package.yml (v2.7.0) uses setup-r-dependencies@v2 to install from DESCRIPTION and check-r-package@v2 for validation; no renv involvement on the host runner at all. Full template and rationale in Post 69. The four-workspace typology (Point 2) and the exit-status principle (Point 1) remain sound.

Reproducibility

Tested on the configurations below.

Component	Version	Notes
OS (CI runner)	Ubuntu 24.04 (noble)	GitHub Actions ubuntu-latest as of May 2026
OS (local)	macOS 15 (Darwin 25)	for `gh`, `git`, local renv operations
R	4.4.2	matches lockfile pins across surveyed projects
`r-lib/actions/setup-r`	v2	with `r-version: 'renv'`
`r-lib/actions/setup-renv`	v2	handles caching automatically
`r-lib/actions/setup-pandoc`	v2	needed for Tier 2 and Tier 3
`r-lib/actions/setup-tinytex`	v2	needed for LaTeX render tier
`r-lib/actions/check-r-package`	v2	errors on WARNINGs by default
`quarto-dev/quarto-actions/setup`	v2	needed for Quarto render tier
Date of last verification	2026-05-06	five-repo deployment, all surfacing project-side findings

The companion compendium ships analysis/configs/r-package.yml, analysis/configs/render-report.yml, and analysis/configs/blog-render.yml as the four templates referenced in Step 2. Copy the appropriate one to .github/workflows/ in the project.

Let’s Connect

Comments and questions on this post are welcome via the giscus panel below. Corrections, alternative patterns, and reports of edge cases that the four-workspace typology does not cover are particularly useful: the typology was extracted from a sample of roughly twenty projects in two trees, and the natural next step is to test it against external projects whose patterns differ.

The companion white paper at zzcollab/docs/ci-strategy-tiered-model.md is the live document; it will be updated with new findings as they accumulate.

Rendered on 2026-05-22 at 09:22 PDT.
Source: ~/Dropbox/prj/qblog/posts/63-zzcollab-ci-strategy/zzcollab-ci-strategy/analysis/report/index.qmd

Setting up OBS for Live R Coding Screencasts

Ronald G. Thomas — Sat, 02 May 2026 07:00:00 GMT

A live OBS Studio session ready to capture an R analysis.

A short screencast captures the small decisions a written report leaves out.

Introduction

I did not appreciate how much a good screencast teaches until a colleague sent me a five-minute video of an analysis I had read about in a paper a week earlier. The paper had described the model, the data, and the result. The screencast showed the analyst pause over a missing value, change a plot scale mid-thought, and read aloud the output of summary(fit). That brief exposure to the analyst’s working memory taught me more about the analysis than the paper had.

What I discovered, after a few false starts with proprietary tools, was that the open source community already had a complete capture pipeline waiting: OBS Studio for recording, YouTube for distribution, and a small set of recording conventions to glue them together. The pipeline costs nothing, runs on macOS, Linux, and Windows, and produces files that any future viewer can play without an account.

We walk through that pipeline end to end. The post covers OBS installation, scene and audio configuration suitable for an R coding session, a publication checklist for YouTube, and a worked five-minute example using the Palmer Penguins dataset that mirrors the analytical structure of post 14.

Motivations

Written reports describe the finished analysis but not the small decisions the analyst makes between keystrokes. A short screencast exposes those decisions.
Proprietary screencasting tools (Camtasia, ScreenFlow) cost between 100 and 300 dollars and lock recordings into vendor-specific project formats.
Live streaming an analysis to a small group of collaborators (a thesis advisor, a paper co-author, a study team) was awkward without a reproducible setup.
The R community has gravitated toward YouTube as the default home for asynchronous tutorials, but documentation of the recording workflow itself is scattered.

Objectives

Install and configure OBS Studio on macOS or Linux with sensible defaults for R coding screencasts (1080p, 30 fps, hardware encoder).
Establish a three-scene template (code only, code with webcam, title card) and a two-source audio chain (microphone with noise suppression, optional system audio).
Produce a 30-second test recording, edit it with ffmpeg, and upload it to YouTube as an unlisted draft with appropriate metadata.
Walk through a complete five-minute live R coding session based on the Palmer Penguins dataset, ready to record on a second take.

Workspace prepared for the first recording session.

What Is OBS Studio?

OBS Studio (Open Broadcaster Software) is a free, cross-platform video recording and live streaming application released under the GNU General Public License version 2. It plays the same role for video that ffmpeg plays for command-line transcoding: a deeply scriptable, deeply configurable workhorse that the rest of the open source ecosystem builds on. A concrete example: every academic conference I have attended in the past two years that recorded talks for asynchronous viewing used OBS, either directly by the AV team or via a downstream tool that wraps it.

Prerequisites

Operating system. macOS 12 or later, Ubuntu 22.04 or later, or equivalent. Windows 10/11 also works but is not the focus of this post.
Hardware. A laptop or desktop from the past five years. Hardware H.264 encoding (Apple VT on macOS, NVENC on NVIDIA GPUs) reduces CPU load substantially during recording.
Microphone. Any USB or 3.5mm microphone. Built-in laptop microphones produce listenable but distinctly amateur audio.
Prior knowledge. Basic shell command experience and familiarity with the R session one intends to record.
Disk space. Roughly 50 megabytes per minute of recorded video at the recommended quality settings.

Step by Step

Step 1: Install OBS Studio

On macOS:

brew install --cask obs

On Debian or Ubuntu:

sudo apt install obs-studio

Launch OBS once and accept the auto-configuration wizard’s defaults. The wizard probes the system to choose sensible recording resolution and encoder settings.

Verification:

obs --version

A successful install prints the version string (30.x as of 2026-05).

Step 2: Configure Scenes and Sources

A coding screencast typically needs three scenes:

Code only. A single Display Capture or Window Capture source for the editor or terminal. Useful for the bulk of the recording.
Code with webcam. The same capture source plus a small Video Capture Device overlay for the presenter, anchored to a corner.
Title card. A static image source for the opening and closing seconds of the video.

Window Capture is preferred over Display Capture when only a single application needs to be visible: it ignores notification banners, desktop clutter, and accidental tab switches.

Step 3: Audio Routing

Two audio sources matter for a coding screencast: microphone and (optionally) system sound. Add both as separate sources so they can be mixed independently.

For a USB microphone, set the input gain so the audio meter peaks around -12 dB during normal speech. Apply the Noise Suppression filter (RNNoise) and a Compressor filter to even out volume across sentences.

Step 4: Recording Format

In Settings > Output > Recording:

Recording Path. A dedicated ~/screencasts/ directory.
Recording Format. mkv (resilient to crashes; remux to mp4 afterward).
Encoder. Hardware encoder if available (Apple VT H.264 on macOS, NVENC on Linux with NVIDIA GPU).
Rate Control. CQP at quality 18 to 22.

Set frame rate to 30 fps and base resolution to 1920x1080. Higher frame rates are unnecessary for coding content and inflate file size.

Step 5: Practice Run

Record a 30-second test that exercises every scene transition and confirms the audio levels. Play the result back at full screen on a different device. Common issues caught at this stage include muted microphone input, font sizes too small to read at video compression, and a busy desktop background visible behind a window-captured editor.

Mid-recording: the editor on the left, a small webcam tile in the corner.

Step 6: The Five-Minute Analysis

The recorded analysis itself follows a strict template: load, glimpse, plot, model, summary. The full code for the worked example is below; during the screencast each block is typed and explained in real time.

library(palmerpenguins)
library(ggplot2)
library(dplyr)

penguins_clean <- penguins |> tidyr::drop_na()

glimpse(penguins_clean)

Rows: 333
Columns: 8
$ species            Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, Adel…
$ island             Torgersen, Torgersen, Torgersen, Torgersen, Torgerse…
$ bill_length_mm     39.1, 39.5, 40.3, 36.7, 39.3, 38.9, 39.2, 41.1, 38.6…
$ bill_depth_mm      18.7, 17.4, 18.0, 19.3, 20.6, 17.8, 19.6, 17.6, 21.2…
$ flipper_length_mm  181, 186, 195, 193, 190, 181, 195, 182, 191, 198, 18…
$ body_mass_g        3750, 3800, 3250, 3450, 3650, 3625, 4675, 3200, 3800…
$ sex                male, female, female, female, male, female, male, fe…
$ year               2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007…

A brief comment on the dataset acknowledges its provenance (Palmer Station, Antarctica; collected by Dr. Kristen Gorman) and previews the question: how well does flipper length predict body mass?

ggplot(
  penguins_clean,
  aes(x = flipper_length_mm, y = body_mass_g)
) +
  geom_point(alpha = 0.6) +
  geom_smooth(method = 'lm', se = TRUE) +
  labs(
    x = 'Flipper length (mm)',
    y = 'Body mass (g)',
    title = 'Body mass scales linearly with flipper length'
  ) +
  theme_minimal(base_size = 13)

The visual establishes the linear relationship before the model is fit, which keeps the viewer oriented.

fit <- lm(body_mass_g ~ flipper_length_mm,
          data = penguins_clean)
summary(fit)


Call:
lm(formula = body_mass_g ~ flipper_length_mm, data = penguins_clean)

Residuals:
     Min       1Q   Median       3Q      Max 
-1057.33  -259.79   -12.24   242.97  1293.89 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)       -5872.09     310.29  -18.93   <2e-16 ***
flipper_length_mm    50.15       1.54   32.56   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 393.3 on 331 degrees of freedom
Multiple R-squared:  0.7621,    Adjusted R-squared:  0.7614 
F-statistic:  1060 on 1 and 331 DF,  p-value: < 2.2e-16

A flipper length increase of one millimetre is associated with roughly a 50 gram increase in body mass. R-squared is approximately 0.76, which is striking for a single predictor and provides a natural cliffhanger for a follow-up post that introduces species as a covariate (see Palmer Penguins Part 1).

The screencast closes with a one-sentence summary and a verbal pointer to the written post that contains the full analysis.

Step 7: Edit and Upload

Trim the head and tail using ffmpeg:

ffmpeg -i recording.mkv -ss 00:00:03 \
  -to 00:05:12 -c copy clipped.mp4

Upload via the YouTube Studio web interface. Recommended metadata fields:

Title. Mirror the blog post title.
Description. Link to the blog post and to the GitHub repository.
Tags. r, data analysis, screencast, plus dataset tags.
Chapters. Add timestamp markers in the description so viewers can jump to the modelling section.

Step 8: Embed in the Blog Post

Quarto embeds YouTube videos with a single shortcode:

The video should appear early in the post, ideally just after the introduction, so visitors can choose to watch or read.

Default Recording Hotkeys

Action	macOS shortcut	Linux shortcut
Start / stop record	not bound by default	not bound by default
Pause record	not bound by default	not bound by default
Switch scene	configurable per scene	configurable per scene
Mute microphone	configurable	configurable

Bind these in Settings > Hotkeys before the first real recording. Unbound defaults catch most beginners off guard.

Things to Watch Out For

The following gotchas have all caught me at least once. Each lists the symptom and the fix.

Display Capture shows a black rectangle on macOS. Symptom: the captured area is uniformly black even though the screen has content. Fix: grant OBS Screen Recording permission in System Settings > Privacy & Security > Screen Recording, then quit and relaunch OBS. The permission request only appears the first time; if it was dismissed, it must be granted manually.
Audio is silent in the recording but the meter shows activity. Symptom: the OBS audio meter moves during speech, but playback is silent. Fix: check that the microphone source is not muted in the Audio Mixer panel (the speaker icon is hidden until hovered) and confirm the recording’s output channel mix in Settings > Audio > Advanced.
The recording stutters when the editor is in focus. Symptom: intermittent dropped frames during typing. Fix: switch from software (x264) to a hardware encoder, or reduce base resolution to 1280x720 if hardware encoding is unavailable. Coding content at 720p is still readable at typical viewing distances.
YouTube rejects the upload citing ‘invalid format’. Symptom: YouTube Studio displays a generic error after the upload finishes. Fix: most often the recording is in MKV. Remux to MP4 with ffmpeg -i recording.mkv -c copy recording.mp4. MKV is preferred during recording for crash resilience but YouTube prefers MP4.
Font in the recording is unreadable when played at small window sizes. Symptom: code is legible at full screen but pixelated in a 480-wide embed. Fix: increase the editor’s font size to 18 or 20 points before recording. What looks oversized in the editor compresses to comfortable reading size in the final video.
Webcam overlay is mirrored. Symptom: text on a notebook visible to the webcam appears reversed in the recording. Fix: right-click the Video Capture Device source, choose Transform > Flip Horizontal. OBS mirrors the preview by default to feel natural to the speaker but does not mirror the recorded output.
Recording fills the disk during a long session. Symptom: OBS silently stops recording after 20 to 30 minutes. Fix: at the recommended quality, every 10 minutes consumes roughly 500 megabytes. Confirm at least 5 gigabytes of free space before a long session, or switch to a more aggressive crf value (lower visual quality, smaller file).

Uninstall / Rollback

To remove OBS Studio and its configuration entirely:

On macOS:

brew uninstall --cask obs
rm -rf ~/Library/Application\ Support/obs-studio

On Debian or Ubuntu:

sudo apt remove obs-studio
rm -rf ~/.config/obs-studio

Both removals are reversible: a fresh install re-creates the configuration directory with default settings on first launch.

A finished recording, ready to clip and upload.

What Did We Learn?

Conceptual

A short screencast is a different teaching artefact from a written post; both have a place, and pairing them produces a richer resource than either alone.
The capture pipeline (recorder, encoder, hosting) can be assembled entirely from open source and free-tier components, with no long-term lock-in.
Recording forces a kind of editorial discipline that improves the written post too: vague paragraphs become obvious when read aloud.

Technical

Hardware encoders are essential on laptops; software encoding (x264) competes with R for CPU cycles and produces visible stuttering.
MKV during recording, MP4 for upload. The two-stage convention protects against crashes without sacrificing platform compatibility.
Audio quality matters more than video quality. Viewers tolerate 720p video far better than echo-prone or compressed audio.

Gotchas

macOS screen-recording permission is granted per-application and silently fails closed. New OBS installs need a deliberate first-run trip through System Settings.
YouTube’s MP4 preference is documented but not enforced clearly in upload error messages.
Default OBS hotkeys are intentionally unset to avoid clashes; this feels broken until one realises it.

Limitations

The pipeline does not include a post-production editor. For multi-cut tutorials or any work requiring callouts, a separate editor (Kdenlive, DaVinci Resolve, iMovie) is needed.
The Palmer Penguins worked example is a demonstration, not a template for clinical trial visualisation. Audio narration would need different framing for sensitive data.
Live streaming via YouTube introduces 5 to 30 seconds of latency, which makes interactive Q&A clunky compared with a dedicated video conferencing tool.
The post does not address closed captioning or multilingual subtitles, both of which deserve their own treatment.

Opportunities for Improvement

Capture the OBS scene collection as a JSON file under analysis/configs/scene-collection.json so the configuration is reproducible across machines.
Script the audio chain (RNNoise + Compressor) using obs-websocket so the configuration can be applied programmatically.
Compare hardware encoder quality across Apple VT, NVENC, and QuickSync at matched bitrates.
Produce a follow-up post on Kdenlive for non-trivial editing.
Document a captioning workflow with whisper.cpp for automatically generated subtitles.
Build a CI job that validates the scene collection JSON against the OBS schema on each commit.

Wrapping Up

OBS Studio, paired with YouTube and a small set of recording conventions, gives R users a complete open source pipeline for live coding content. The investment in initial setup is modest and pays off across many subsequent recordings: the same scenes, audio chain, and upload conventions apply to every post.

The next step in this series will be a recording of the Palmer Penguins analysis above, published both as a written post and as an embedded five-minute video, so the comparison between the two media can be made directly.

In conclusion, three points merit emphasis. First, the open source pipeline (OBS + ffmpeg + YouTube) is sufficient for publication-quality screencasts of R analyses, with no proprietary tooling required. Second, three scenes, two audio sources, and a 30-second test constitute the minimum viable configuration; everything beyond that is refinement. Third, audio quality and font size are the two settings most often underestimated by first-time recorders, and correcting them before recording saves significant post-production effort.

Reproducibility

Component	Version tested	Notes
OBS Studio	30.x	macOS via Homebrew; Ubuntu via apt
ffmpeg	7.x	for trim and remux
Quarto	1.9.37	for HTML rendering of this post
macOS	26.4 (Tahoe)	hardware encoding via Apple VT
Ubuntu	24.04 LTS	hardware encoding via NVENC
Date verified	2026-05-02

Let’s Connect

Have questions, suggestions, or spot an error? Let me know.

GitHub: rgt47
Email: Contact form

I would enjoy hearing from you if:

You spot an error or a better approach to any of the configuration in this post.
You use a different recorder (SimpleScreenRecorder, ScreenFlow) and want to compare notes.
You have suggestions for follow-up topics in the screencasting series.

Rendered on 2026-05-02 at 10:30 PDT.
Source: ~/prj/qblog/posts/27-setupobs/setupobs/analysis/report/index.qmd

From testthat to tinytest: Converting an R Package Test Suite

R. Glenn Thomas — Sat, 02 May 2026 07:00:00 GMT

Two test-tube racks side by side on a clean lab bench, the right rack noticeably smaller and holding two amber-tinted tubes, symbolising the contraction in scaffolding when a suite moves from testthat to tinytest.

Same coverage, fewer lines: the appeal of tinytest is less about novelty and more about the absence of ceremony.

Introduction

I did not really appreciate how much ceremony testthat carries until I cracked open inst/tinytest/ in a small package and saw a fourteen-line file doing the work of an eighty-line test_that() file in the package I had been maintaining. The expectations were the same. The fixtures were the same. What was missing was the nesting, the helper boilerplate, and roughly half of the imports.

What I discovered, once I started porting the suite, was that tinytest is not ‘testthat lite’. It is a different shape. The unit of testing is the file, not the test_that() block. Tests are flat assertions evaluated in a script, not nested calls inside a closure. There is no setup() and no implicit fixture caching. Several testthat idioms (expect_snapshot, local_edition, expect_message with regex matching) have no direct equivalent and require either a different pattern or, in a few cases, a deliberate decision to drop the test.

We walk through the conversion as a paired-example review. I cover layout differences, the mechanical mapping of expect_* calls, the handful of patterns that do not port cleanly, the CI implications, and a short list of cases where I concluded that staying on testthat was the right call. The companion deliverable, a side-by-side conversion recipe, lives at docs/testthat-to-tinytest-recipe.qmd in this post’s compendium.

Pros and cons at a glance

Before getting into the mechanical details, a level-set on what each framework does well and where it asks something of the package author. This is an extended summary so that readers can calibrate whether the conversion is worth their afternoon.

What `testthat` does well

A mature ecosystem. The framework is the de-facto standard on CRAN, which means the largest body of Stack Overflow answers, blog posts, and worked examples points at testthat idioms. Onboarding contributors who already write R is faster with testthat than with anything else.
Snapshot testing. expect_snapshot(), expect_snapshot_value(), and expect_snapshot_file() cover text, structured-data, and binary outputs respectively, with automatic regeneration on review. This is genuinely useful for packages whose output is a formatted table, a printed object, or a plot.
Rich diff output. Failed assertions are reported through waldo, which produces a side-by-side diff for vectors, lists, and data frames. The diff often points directly at the responsible value, where a bare ‘not equal’ message would not.
IDE integration. RStudio’s test pane reports per-test pass/fail status with one-click navigation. Ctrl-Shift-T runs the suite from the editor.
Parallel test execution. testthat 3.0+ runs tests in parallel with parallel = TRUE in testthat::test_local() or Config/testthat/parallel: true in DESCRIPTION. For suites with long per-file fixtures, the speed-up is noticeable.
Editions for managing breaking changes. The Config/testthat/edition: 3 declaration locks the package to a specific behavioural contract, which makes upgrades predictable. testthat 4 will likely use the same mechanism.
A wider assertion surface. expect_s3_class, expect_s4_class, expect_type, expect_named, expect_no_error, expect_no_warning, expect_no_message, expect_setequal, expect_mapequal, and expect_invisible all have semantics that read clearly at the call site.
test_check() exposes the package internal namespace. Test files reference non-exported helpers and constants by their bare names. For internal-heavy packages, this removes a layer of getFromNamespace() ceremony.
Helper files auto-sourced. Files matching tests/testthat/helper-*.R are sourced before each test file, providing implicit fixture setup.
Skip helpers for environment conditions. skip_on_cran(), skip_on_os(), skip_if_offline(), skip_if_not_installed() are concise and read like the intent they encode.

Where `testthat` asks something of the author

A heavy dependency surface. Installing testthat pulls in waldo, brio, desc, pkgload, processx, cli, digest, evaluate, R6, rlang, and roughly two dozen transitive dependencies. For toy packages or small internal tools, this is most of the package’s install footprint.
Slower CI. A R CMD check of an empty testthat suite takes longer than the same check with tinytest because the framework itself takes time to load and dispatch.
The framework itself depends on third-party packages. This is a philosophical concern more than a practical one, but it does mean a bug or breaking change in any of the transitive dependencies can affect the test suite.
Edition complexity. Editions are useful but add a layer of indirection. Tests that pass under edition 2 may fail under edition 3 (and vice versa). Diagnosing these edition-driven regressions is rare but bewildering when it happens.
Per-test ceremony. Every assertion lives inside a test_that('description', { ... }) block. The description string is sometimes informative and sometimes ceremonial.

What `tinytest` does well

Zero Imports. tinytest is a single source file with no third-party dependencies. The framework is itself testable from a fresh R installation.
Faster R CMD check. The framework loads in milliseconds. For a small package with a few hundred assertions, the wall-clock saving over testthat is on the order of seconds, not minutes, but it compounds across a CI fleet.
File-as-test-unit. Each inst/tinytest/test_*.R file is the unit. The natural one-to-one map between an R/ source file and a test file makes the suite easy to navigate. Tests inside a file are top-level, not nested.
Tests installed with the package. Because tests live in inst/tinytest/, they ship with the installed package and remain runnable via tinytest::test_package('pkg') after install. Users can verify their installation; this is useful for compiled-code packages and packages with system dependencies.
A smaller cognitive surface. No editions, no fixture caching, no helper auto-sourcing, no description strings. When a test fails, the failure mode is straightforward. When it passes, one can be confident it passed for the reason intended.
Predictable behaviour across R versions. With no third-party dependencies, tinytest is unaffected by changes in the broader package ecosystem. A test suite written against tinytest 1.4 in 2024 will still run unchanged in 2030.
Self-evident diagnostics. A tinytest failure reports the file, line, expected value, and actual value with no framework noise. The diagnostic surface is small enough to read at a glance.

Where `tinytest` asks something of the author

No snapshot testing. This is the largest functional gap. Manual reference-file comparison via capture.output()
- expect_equal() works but is ergonomically distant from expect_snapshot().
No parallel execution by default. A cl argument to test_all() accepts a parallel cluster, but the default is serial.
No RStudio test-pane integration. Tests run under R CMD check and the console; the IDE’s ‘Run Tests’ button does not populate tinytest results.
A smaller user community. Searches for specific assertion patterns yield fewer results, and answers are sometimes outdated. The CRAN vignette is the most reliable reference.
A narrower assertion surface. No expect_s3_class, expect_type, expect_named, or expect_no_error. Tests using these need manual rewriting; the recipe section 12 documents the substitutions.
test_package() only sees exports. Tests that reference non-exported package internals must qualify the reference (getFromNamespace() or pkg:::name) or the package must export the object. Recipe section 20 has the detail.
No helper-*.R auto-sourcing. Shared fixtures require explicit source('helper_X.R', local = TRUE) at the top of each consumer file.
No skip_on_os, skip_on_ci, skip_if_offline helpers. The substitutes (if (... condition ...) exit_file('reason') at file scope) are equally clear but more verbose at the use site.
No patrick-style parameterised tests. Plain for loops work, but the failing-iteration identifier is the line number rather than a parameter label.
Less informative diff output. Without waldo, large vector or list comparisons report ‘not equal’ without the side-by-side diff. For complex objects, this lengthens debugging time.

Decision matrix

If your package…	Likely better fit
Has more than five `expect_snapshot_*()` calls	testthat
Is developed primarily in RStudio with the test pane	testthat
Uses `parallel = TRUE` for acceptable CI time	testthat
Has a large team unfamiliar with `tinytest`	testthat
Is a small utility with no snapshots	tinytest
Is tested mainly in CI rather than interactively	tinytest
Wants minimal install footprint	tinytest
Ships compiled code that needs post-install verification	tinytest
Has tests that primarily reference exported functions	either
Is greenfield	either

The matrix is not exhaustive, and most packages have at least one row pulling each direction. The recommendation is not to choose by majority vote but to identify the row that costs the most under the wrong choice. For a package whose CI is too slow, that row is the install-footprint one. For a package whose IDE workflow is critical, the test-pane row dominates.

Motivations

Reading the testthat source after a debugging session and realising how much of the dependency surface (waldo, brio, desc, pkgload, processx, cli) my small packages were pulling in only to call expect_equal half a dozen times.
A growing collection of toy packages where R CMD check time was dominated by test-framework startup rather than the tests themselves, particularly inside containers.
Curiosity about Mark van der Loo’s design argument that a test framework should itself be testable, and that a single-file, zero-dependency framework is more honest about what testing actually requires.
The ergonomic appeal of file-as-test-unit when most of my packages have one source file per concern; the one-to-one map between R/foo.R and inst/tinytest/test_foo.R makes the suite easy to navigate.
A practical interest in seeing what breaks. Several testthat idioms are deeply embedded in the broader R-package ecosystem (snapshot testing, parallel test runners, RStudio’s test pane). Porting forces a clear-eyed audit of what one actually uses.

Objectives

By the end of the conversion, the package should have:

An inst/tinytest/ directory containing one test file per source-code concern, each runnable in isolation via tinytest::run_test_file().
A tests/tinytest.R bootstrap that triggers the suite during R CMD check, replacing the tests/testthat.R bootstrap.
A DESCRIPTION file with tinytest in Suggests, no Config/testthat/edition line, and no testthat dependency.
A CI run (locally or in GitHub Actions) that exercises the suite end-to-end and reports the same number of distinct assertions the testthat suite covered, less any tests that were intentionally retired.

A cappuccino in a dark cup resting on an open paper planner, lit by a shaft of light from a window above. Placeholder ambiance image; will be replaced with subject-specific imagery before publication.

What is tinytest?

tinytest is a single-file R testing framework, written by Mark van der Loo, that ships with no Imports and one test file per package concern. Where testthat organises tests as nested calls inside a test_that() closure, tinytest evaluates each file as a script and treats every top-level expect_* call as an independent assertion.

A useful analogy: testthat is to tinytest as a full-featured build system is to a Makefile. Both produce the same artifact; one is opinionated about structure, the other gets out of the way.

A concrete example. The same assertion in both frameworks:

# testthat
test_that('addition is associative', {
  expect_equal((1 + 2) + 3, 1 + (2 + 3))
})

# tinytest
expect_equal((1 + 2) + 3, 1 + (2 + 3))

The tinytest form is not nested in anything. The file in which it lives is the unit. The label, when reporting, is the file name plus the line number. There is no descriptive string parameter because the assertion is its own description.

Prerequisites

R 4.1 or later (the native pipe is used in the post’s worked examples, but tinytest itself works back to R 3.5).
An existing R package with a tests/testthat/ directory. Greenfield packages can skip the conversion entirely and start with tinytest.
Familiarity with R CMD check and the standard tests/.R bootstrap convention.
Optional but useful: a CI configuration file you control. The examples below use GitHub Actions, but the pattern is the same for GitLab, Jenkins, or a local Makefile.

Installation

tinytest is on CRAN, with no compiled code and no dependencies. Install it from R:

install.packages('tinytest')

Verify the install:

packageVersion('tinytest')
# [1] '1.4.1'

For a package that uses renv, add the dependency to the lockfile:

renv::install('tinytest')
renv::snapshot()

Layout: testthat vs tinytest

The two frameworks place test files in different directories, and the bootstrap files differ as well. The package layout shifts as follows:

Before                            After
------                            -----
tests/                            tests/
  testthat.R                        tinytest.R
  testthat/                       inst/
    test-foo.R                      tinytest/
    test-bar.R                        test_foo.R
    helper-fixtures.R                 test_bar.R
                                      helper_fixtures.R

Three differences worth naming:

Test files move from tests/testthat/ to inst/tinytest/. The inst/ location means the tests are installed alongside the package and remain runnable after install via tinytest::test_package('mypkg'). This is one of the larger conceptual shifts: in tinytest, tests are part of the shipped package, not a development-only artefact.
File-naming convention changes from test-*.R to test_*.R. The underscore is tinytest’s default; the dash also works, but the underscore is what tinytest::test_package() expects without configuration.
The bootstrap file changes name and content. See below.

The bootstrap file

tests/testthat.R looks like this:

library(testthat)
library(mypkg)
test_check('mypkg')

The tinytest equivalent is tests/tinytest.R:

if (requireNamespace('tinytest', quietly = TRUE)) {
  tinytest::test_package('mypkg')
}

The requireNamespace guard is idiomatic for tinytest. Because tinytest lives in Suggests, not Imports, the package should build and check on a system where tinytest is not installed. The guard makes that case a no-op rather than a hard error.

The mechanical conversion

The bulk of the work is rewriting individual test files. The mapping is mostly one-to-one. The full table lives in the companion deliverable; here are the patterns that come up most often.

Pattern 1: a single `test_that()` block

Before:

# tests/testthat/test-arithmetic.R
test_that('addition works', {
  expect_equal(1 + 1, 2)
  expect_equal(2 + 2, 4)
  expect_true(is.numeric(1 + 1))
})

After:

# inst/tinytest/test_arithmetic.R
expect_equal(1 + 1, 2)
expect_equal(2 + 2, 4)
expect_true(is.numeric(1 + 1))

The test_that() wrapper, the description string, and the braces all disappear. The three assertions are now top-level calls in the file.

Pattern 2: multiple `test_that()` blocks per file

Before:

test_that('addition works', {
  expect_equal(1 + 1, 2)
})

test_that('subtraction works', {
  expect_equal(2 - 1, 1)
})

After (option A, single file):

# inst/tinytest/test_arithmetic.R
expect_equal(1 + 1, 2)   # addition
expect_equal(2 - 1, 1)   # subtraction

After (option B, split into two files):

inst/tinytest/test_addition.R
inst/tinytest/test_subtraction.R

Option B is the more idiomatic tinytest choice when the two groups exercise unrelated source files. The framework offers no mechanism for grouping inside a file, so cohesion is enforced by file boundaries instead.

Pattern 3: shared fixtures

testthat resolves shared fixtures via files named helper-*.R, which it sources before each test file. tinytest does not have an equivalent automatic mechanism, but two patterns work:

# inst/tinytest/helper_fixtures.R
make_test_data <- function() {
  data.frame(x = 1:5, y = letters[1:5])
}

# inst/tinytest/test_summary.R
source('helper_fixtures.R', local = TRUE)
df <- make_test_data()
expect_equal(nrow(df), 5)

The local = TRUE argument keeps the fixture’s symbols out of the global environment. The repetition (source(...) at the top of each consumer file) is deliberate: tinytest privileges explicit dependency over magic.

Pattern 4: skip-on-CRAN

Before:

test_that('integration test', {
  skip_on_cran()
  skip_if_not_installed('database_driver')
  ...
})

After:

if (!identical(Sys.getenv('NOT_CRAN'), 'true')) exit_file('Skipping on CRAN')
if (!requireNamespace('database_driver', quietly = TRUE)) {
  exit_file('database_driver not installed')
}
...

exit_file() halts the current file with the supplied message recorded as a skip. There is no tinytest analogue of skip_on_os() or skip_if_offline(), but those checks are small enough that an if (...) exit_file(...) line at the top of the file is sufficient.

Pattern 5: snapshots

This is the pattern that does not port. testthat snapshots (expect_snapshot(), expect_snapshot_value(), expect_snapshot_file()) have no direct equivalent in tinytest. The available substitutes are:

For text output, capture with capture.output() and compare to a reference string with expect_equal().
For binary or large outputs, write a reference file once and compare with tools::md5sum() on each run.
For images, use tinytest::expect_equal_to_reference() or drop the test.

In practice, snapshot-heavy suites are the strongest argument against converting. The migration cost is high, the failure modes are subtler than expect_equal(), and the resulting tests are noticeably less ergonomic than their testthat counterparts.

DESCRIPTION changes

The package metadata changes in three places:

Before                            After
------                            -----
Suggests:                         Suggests:
    testthat (>= 3.0.0)               tinytest
Config/testthat/edition: 3        (line removed)

If testthat is the only test framework being dropped, the removal is straightforward. If the package previously declared both, leave whichever frameworks are still in use.

CI: GitHub Actions

The standard r-lib/actions/check-r-package action runs R CMD check, which exercises whichever bootstrap file is in tests/. No CI changes are strictly required: a package with tests/tinytest.R will run its tinytest suite during R CMD check exactly as a testthat package does.

The one wrinkle is that tinytest errors are currently surfaced through R CMD check as ordinary stop conditions, not as a separate test report. If the CI surface relies on a structured test summary (for example, GitHub’s check-run output), an extra step that calls tinytest::test_package() directly and emits a TAP report may be useful:

- name: Run tinytest with TAP output
  run: |
    Rscript -e 'tinytest::test_package("mypkg",
      side_effects = TRUE) |> as.data.frame() |> print()'

Verification

After the conversion, three commands should produce output consistent with the previous testthat run:

# 1. Bootstrap runs the suite under R CMD check
R CMD check --as-cran .

# 2. Direct invocation reports per-file pass/fail
Rscript -e 'tinytest::test_package(".")'

# 3. Single-file run, useful during development
Rscript -e 'tinytest::run_test_file(
  "inst/tinytest/test_arithmetic.R")'

The total number of assertions reported in step 2 should match the testthat baseline, less any tests retired during the migration. If the count differs and the difference is not explained by an explicit retirement, a test_that() block likely contained more expect_* calls than the rewrite captured.

Daily Workflow

Action	Command
Run the full suite	`Rscript -e 'tinytest::test_package(".")'`
Run a single file	`tinytest::run_test_file('inst/tinytest/test_X.R')`
Run all files in a directory	`tinytest::run_test_dir('inst/tinytest')`
Run during `R CMD check`	`R CMD check .` (uses `tests/tinytest.R`)
Build a coverage report	`covr::package_coverage()` (works with both)
Continuous run on file change	`tinytest::test_package('.', side_effects = TRUE)`

An espresso cup viewed from above on a wide hardwood floor, low-key lighting, single subject in deep shadow. Placeholder ambiance image; will be replaced with subject-specific imagery before publication.

Things to Watch Out For

These are the gotchas that cost me time during the conversion. Several are not in the tinytest documentation because they are artefacts of the migration rather than the framework itself.

Helper files are not auto-sourced. A testthat package with tests/testthat/helper-data.R gets that file sourced before every test file automatically. tinytest does not. Symptom: object 'make_test_data' not found. Fix: add source('helper_fixtures.R', local = TRUE) at the top of each consumer file.
expect_message() and expect_warning() regex matching. testthat accepts a regex as the second argument. tinytest does not. Symptom: tests pass that should not, because any message satisfies the assertion. Fix: assert the message content separately:
```
m <- capture.output(my_function(), type = 'message')
expect_true(grepl('expected pattern', m))
```
expect_error() returns the error invisibly. Capturing it for further inspection requires e <- expect_error(my_function()) which is the same idiom as testthat, but the error class is reported differently. testthat’s class = argument has no tinytest analogue; inspect class(e) manually.
Config/testthat/edition: 3 lingers. Forgetting to remove this line from DESCRIPTION is harmless but confusing for future maintainers. Symptom: a DESCRIPTION that references a framework the package no longer uses. Fix: remove the line manually; no tooling performs this step automatically.
Snapshot tests have no clean port. Listed above as pattern 5; worth re-stating. If a substantial fraction of the suite is expect_snapshot_*(), reconsider the migration.
The RStudio test pane does not light up. tinytest integrates with RStudio’s build pane through R CMD check but does not populate the test runner pane the way testthat does. Symptom: ‘Run Tests’ button in the IDE no longer reports per-test results. Fix: run from the console or terminal; the IDE integration is unlikely to change.
Parallelism requires explicit setup. testthat 3.0+ parallelises tests with parallel = TRUE. tinytest does not parallelise out of the box, though tinytest::test_all() accepts a cl argument for a parallel cluster. Suites with long-running per-file setup may run noticeably slower in serial.
Four testthat 3 assertion functions silently resolve via pkgload. A real conversion of the zzpower package surfaced this: expect_s3_class, expect_type, expect_named, and expect_no_error are exported by testthat but not by tinytest. When a developer runs the converted suite via pkgload::load_all() followed by tinytest::run_test_dir(), pkgload auto-attaches every package in Suggests, including testthat. The four testthat assertions then resolve against the testthat namespace, the run reports ‘all ok’, and the migration appears complete. Symptom: the tinytest assertion count silently undercounts the testthat baseline. Fix: detach testthat before running the suite, or grep the converted files for these four function names and replace them with tinytest equivalents (expect_inherits, expect_true(is.function(...)), expect_equal(sort(names(x)), sort(...)), and unwrapping the expect_no_error block). Recipe section 12 has the full mapping.
Non-exported objects are invisible to tinytest::test_package(). testthat::test_check() runs tests with the package’s internal namespace exposed, so test files can reference non-exported helpers, constants, and S3 methods by their bare names. tinytest::test_package() calls library(pkg), which only attaches exports. A test file that worked under testthat with expect_true(exists('MY_CONST')) will fail under tinytest with ‘object ’MY_CONST’ not found’, even though the original suite passed. Symptom: under pkgload::load_all() + tinytest::run_test_dir(), tests pass; under R CMD check (which uses tinytest::test_package() against the installed package), the same tests fail. Fix: either export the object via roxygen2 (#' @export) and re-run devtools::document(), or qualify the reference in the test using getFromNamespace('MY_CONST', 'pkg') or pkg:::MY_CONST. The qualified-reference variant is the safer choice for objects that should remain internal.
R CMD check on a source directory does not auto-derive Author and Maintainer from Authors@R. Under R 4.5.3 on macOS, R CMD check fails with ‘Required fields missing or empty: Author Maintainer’ when the DESCRIPTION declares only Authors@R. The same DESCRIPTION passes when checked via R CMD build followed by R CMD check . The cause is in tools:::.read_description(), which reads the DCF file without injecting derived fields. Symptom: a check that was working last month suddenly errors at the very first step. Fix: prefer the canonical workflow (R CMD build . then R CMD check pkg_X.Y.Z.tar.gz); if direct source-dir checking is required, add explicit Author: and Maintainer: lines to DESCRIPTION alongside Authors@R.
skip_if_not() inside test_that() was scoped to one test; a flat conversion may halt the entire file. A naive textual rewrite of test_that('memory test', { skip_if_not(...); ... }) produces a top-level if (!cond) exit_file(...) followed by the body. exit_file() halts the entire test file, but in the original the skip applied only to that one test_that block; subsequent blocks ran normally. Symptom: the converted suite passes but reports significantly fewer assertions than the testthat baseline, with one entire file showing zero results. Fix: replace the file-level exit_file() with a block-level if (cond) { ... } wrapper around just the assertions that depend on the condition. Recipe section 6 now documents both forms; pick the one that matches the original scope.
Argument-name differences in expect_error and expect_warning. testthat accepts both regex = and regexp = as the named-argument form for the matching pattern. tinytest accepts only pattern =. Symptom: unused argument (regex = "...") or unused argument (regexp = "...") errors during the tinytest run. Fix: rename both to pattern =. The recipe now lists this in section 12.

Uninstall / Rollback

If the conversion turns out to be a poor fit (snapshot-heavy suite, IDE-dependent workflow, parallelisation requirements), the rollback is straightforward because the original tests/testthat/ directory has not been deleted. The git history preserves it.

# 1. Restore tests/testthat/ from git
git checkout HEAD~N -- tests/testthat tests/testthat.R

# 2. Remove the tinytest scaffolding
rm -rf inst/tinytest tests/tinytest.R

# 3. Restore DESCRIPTION
git checkout HEAD~N -- DESCRIPTION

The HEAD~N placeholder is whatever commit preceded the migration. The cleaner approach is to perform the conversion on a branch and revert the branch if needed, rather than committing the migration to main.

A close-up of a single espresso drip falling from a machine into a grey ceramic mug. Placeholder ambiance image; will be replaced with subject-specific imagery before publication.

What Did We Learn?

Lessons Learnt

Conceptual.

The unit of testing is a design choice, not a given. testthat treats the assertion as the unit and groups assertions into test_that() closures. tinytest treats the file as the unit and treats assertions as top-level statements. Neither is strictly better; the choice has consequences for fixture scope, error reporting, and how new tests are added.
A test framework’s dependency surface matters more than the number of features it offers. A package whose tests pull in forty transitive dependencies pays an installation cost on every CI run, whether or not the features are used.
‘Snapshot testing’ is a separable concern from unit testing. Treating the two as a single problem (which testthat does via expect_snapshot_*()) makes the framework richer but also more entangled. tinytest declines to merge them, which forces a deliberate choice about whether snapshots belong in the test suite at all.
Assertions read more naturally when they are not nested inside a description string. The test-name parameter that test_that() requires is information that the file name and line number carry implicitly; making it explicit is helpful in some cases and ceremonial in others.

Technical.

tinytest::expect_equal() defaults to tolerance = 1e-6, matching base::all.equal() rather than testthat::expect_equal()’s newer default of strict equality with a configurable tolerance. Tests that previously passed under testthat 3 may need an explicit tolerance = 0 on the tinytest side, or vice versa.
The order in which test files run matters more than under testthat. tinytest evaluates files in alphabetical order by default. If a file has side effects on the global state (writes to disk, changes options, alters the working directory), later files will see them.
tinytest::test_package() returns a tinytests object whose as.data.frame() method gives a per-assertion table. This is a more programmatic interface than testthat::test_local()’s console-output focus, and pairs well with custom CI summaries.
Coverage tools (covr) work without modification. The package introspection that covr performs is framework- agnostic; it instruments source files rather than test files.

Gotchas.

Forgetting to rename helper-*.R to helper_*.R (dash vs underscore). tinytest does not source files starting with helper; the convention is purely human-facing, so a typo here is silent.
Treating inst/tinytest/ as private. Because tests live in inst/, they are installed with the package and visible to users. This is a feature (users can run the suite to verify their installation) but it means the test files should not contain credentials, paths to private servers, or other artefacts that do not belong in a published package.
Mixing testthat and tinytest during a partial migration. Both bootstraps will run during R CMD check. If the same assertion is duplicated, the count doubles; if it is split, the report fragments. Pick one and finish before merging.
Underestimating the snapshot-test inventory. Run grep -r 'expect_snapshot' tests/testthat/ before starting. If the count is non-trivial, the migration will be longer than expected.

Limitations

tinytest lacks built-in snapshot testing, parallel test execution, and IDE integration with RStudio’s test pane. For packages that depend on any of these, the migration is a net loss.
The framework’s spartan design means that some testthat affordances (parameterised tests via patrick, fixtures via withr::local_*) require manual reimplementation.
Error messages in tinytest are less informative by default than testthat 3’s waldo-powered diffs. For numeric or list comparison, the diff output is the single feature tinytest users miss most.
The framework has a small user community relative to testthat. Searching for solutions to specific assertion patterns yields fewer results, and the available answers are sometimes outdated.

Opportunities for Improvement

Build a tinytest linter that flags accidental testthat idioms (test_that() calls, skip_on_*(), regex arguments to expect_message) during the migration.
Wrap the common conversion patterns in a small package (tinyport?) that automates the file-rename, header rewrite, and DESCRIPTION update steps.
Add a tinytest-aware action to GitHub’s R-lib actions collection so that the structured test-summary step described above is one click away.
Document the snapshot-equivalent patterns more thoroughly. The CRAN vignette covers the basics; a longer reference that addresses image, JSON, and structured-output snapshots would close a real gap.
Build a per-package decision script that reports the number of expect_snapshot_*() calls, the number of helper-*.R files, and an estimated hour-cost for the migration.
Contribute a parallel = TRUE argument to tinytest::test_package() that mirrors testthat’s built-in parallelisation.

Wrapping Up

The conversion is a small project, on the order of an afternoon for a package with a few hundred assertions and no snapshots. The mechanical work is straightforward and the patterns repeat. What takes the time is the audit: deciding which testthat features the package actually uses, which can be replaced, and which signal that the migration is the wrong call.

The decision matrix is short. If the package uses snapshot tests heavily, depends on RStudio’s IDE integration, or relies on parallel test execution, stay on testthat. If the package is small, has no snapshots, and is tested in CI rather than interactively, the migration is mostly upside: fewer dependencies, faster R CMD check, a smaller cognitive surface.

This post stops short of recommending the conversion as a default. Both frameworks are well-engineered. The choice between them is an architectural decision about how much ceremony a test framework should impose, and that decision is package-specific.

Wrapping Up

In conclusion, six points merit emphasis. First, the mechanical mapping is mostly one-to-one; snapshots are the exception. Second, the inst/tinytest/ placement makes tests installable, which is the largest conceptual shift. Third, helper files are not auto-sourced, so explicit source() calls are required. Fourth, Config/testthat/edition lingers in DESCRIPTION if the line is not removed manually. Fifth, the migration is reversible; performing it on a branch is the recommended practice. Sixth, snapshot-heavy suites are a strong reason to remain on testthat.

Reproducibility

Component	Version	Notes
OS	macOS 15.4	also tested on Ubuntu 24.04
R	4.4.1	matches `renv.lock`
`tinytest`	1.4.1	from CRAN, 2024-02 release
`testthat`	3.2.1	reference baseline
`covr`	3.6.4	optional, used for coverage
Date verified	2026-05-02	claims in this post tested on this date

sessionInfo()

Appendix A: Conversion Atlas

A condensed mapping of the most common testthat idioms to their tinytest equivalents. The full table, with paired examples for each row, is in docs/testthat-to-tinytest-recipe.qmd in this post’s compendium.

testthat	tinytest
`expect_equal(x, y)`	`expect_equal(x, y)`
`expect_identical(x, y)`	`expect_identical(x, y)`
`expect_true(x)`	`expect_true(x)`
`expect_false(x)`	`expect_false(x)`
`expect_error(f(), 'msg')`	`expect_error(f(), pattern = 'msg')`
`expect_warning(f(), 'msg')`	`expect_warning(f(), pattern = 'msg')`
`expect_message(f(), 'msg')`	`expect_message(f(), pattern = 'msg')`
`expect_null(x)`	`expect_null(x)`
`expect_silent(f())`	`expect_silent(f())`
`expect_snapshot(x)`	(no direct equivalent; see pattern 5)
`expect_s3_class(x, 'cls')`	`expect_inherits(x, 'cls')`
`expect_is(x, 'cls')`	`expect_inherits(x, 'cls')`
`expect_type(x, 'closure')`	`expect_true(is.function(x))`
`expect_type(x, 'double')`	`expect_equal(typeof(x), 'double')`
`expect_named(x, c(...))`	`expect_equal(sort(names(x)), sort(c(...)))`
`expect_no_error({block})`	(unwrap; inner assertions stand on their own)
`expect_lt(x, y)`	`expect_true(x < y)`
`expect_gt(x, y)`	`expect_true(x > y)`
`expect_lte(x, y)`	`expect_true(x <= y)`
`expect_gte(x, y)`	`expect_true(x >= y)`
`expect_error(f(), regex = 'msg')`	`expect_error(f(), pattern = 'msg')`
`expect_warning(f(), regexp = 'msg')`	`expect_warning(f(), pattern = 'msg')`
`skip_on_cran()`	`if (!nzchar(Sys.getenv('NOT_CRAN'))) exit_file('CRAN')`
`skip_if_not_installed('pkg')`	`if (!requireNamespace('pkg', quietly = TRUE)) exit_file('pkg missing')`
`skip_on_ci()`	`if (nzchar(Sys.getenv('CI'))) exit_file('CI')`
`skip_if_not(cond, msg)` (per-test)	`if (cond) { ... }` (block-level wrapper)
`context('label')`	(remove; tinytest has no equivalent)
`library(testthat)` in test files	(remove; tinytest is implicit at runtime)
`helper-foo.R` (auto-sourced)	`source('helper_foo.R', local = TRUE)`

Appendix B: A Real Migration on `zzpower`

The recipe was validated on zzpower v0.3.0, an interactive Shiny power-analysis package, on branch testthat-to-tinytest. Pre-migration: 10 testthat files, 1,734 LOC, 602 expectations passing under testthat::test_dir(). Post-migration: 10 tinytest files, 1,625 LOC, 600 assertions passing under tinytest::run_test_dir() with testthat detached.

The 6% line-count reduction (109 lines) came almost entirely from removing test_that() wrappers and de-indenting bodies by two spaces. The 2-assertion delta is exactly the two expect_no_error() outer-wrapper counts that no longer count as assertions in their own right; the inner expectations they guarded are still counted.

The migration was not entirely mechanical. Four testthat 3 assertion functions had no direct port and required manual substitution: expect_s3_class (9 calls; replaced with expect_inherits), expect_type (3 calls; replaced with expect_true(is.function(...)) for the closure case and expect_equal(typeof(...), ...) otherwise), expect_named (1 call; replaced with expect_equal(sort(names(x)), ...)), and expect_no_error (2 calls; the wrapper was removed). These four patterns were not in the recipe’s first draft, because the conversion script (which wrapped a textual brace-aware transform around test_that() calls) preserved them verbatim. The ‘Things to Watch Out For’ section now includes these patterns as gotcha 8, and the recipe has a new section 12 with paired examples.

A subtler trap surfaced during dry-running. pkgload::load_all() auto-attaches every package in Suggests, which means the testthat namespace becomes available even after the migration removes testthat from the bootstrap. The four testthat-only assertion functions then resolve silently against testthat’s exports, the run reports ‘all ok’, and the migration appears complete. The honest dry-run requires either detaching testthat explicitly or removing it from Suggests before verification. Only after tinytest::run_test_dir() reports a result count consistent with the testthat baseline (less the known expect_no_error deltas) is the migration validated.

R CMD check was attempted in two configurations and surfaced two more issues, neither in the recipe’s first draft.

The first configuration, R CMD check , failed at the very first step with ‘Required fields missing or empty: Author Maintainer’. Investigation showed that this is not a parsing issue but a behavioural one: tools:::.read_description(), which R CMD check calls to load the DESCRIPTION, does not auto-derive Author and Maintainer from Authors@R. The same DESCRIPTION passes when checked via the canonical R CMD build . followed by R CMD check pkg_X.Y.Z.tar.gz. The lesson generalises: prefer the build-then-check workflow; if direct source-directory checks are required, declare Author and Maintainer explicitly. Recipe section 17 and gotcha 10 now document this.

The second configuration, R CMD check , passed the DESCRIPTION step but failed inside tests/tinytest.R with ‘could not find function “build_sample_size_inputs”’. The cause is that tinytest::test_package('zzpower') calls library('zzpower'), which only attaches exported objects. zzpower’s test_framework.R referenced six non-exported helpers (build_advanced_settings, build_effect_size_inputs, build_sample_size_inputs, get_effect_size_range, logrank_power, trend_power) by their bare names, which worked under testthat::test_check() but not under tinytest. The fix added six getFromNamespace() assignments at the top of test_framework.R, bringing the helpers into the test file’s scope without modifying the package’s NAMESPACE. After the fix, R CMD check reported Status: OK on the resulting tarball. Recipe section 20 and gotcha 9 document the diagnostic and the fix.

Note that ZZPOWER_CONSTANTS, which my first investigation identified as the offender, is in fact exported (export(ZZPOWER_CONSTANTS) in the generated NAMESPACE) and accessible through the search path after library('zzpower'). The earlier failure was a misdiagnosis from a stale install. The lesson here is operational: when an earlier check-run leaves a stale binary in the test library, its NAMESPACE may not match the source. Always rebuild and reinstall when chasing ‘object not found’ errors that contradict a current getNamespaceExports() inspection.

Neither issue invalidates the recipe; both refine it. The final state on the validation branch: R CMD check zzpower_0.4.0.tar.gz returns Status: OK, with all non-skipped checks passing. The total cost of using zzpower as the validation case study was higher than expected (roughly four hours rather than the ‘half a day to a day’ my pre-migration estimate had) because each unanticipated failure required tracing R-internal code to diagnose. The pay-off is that the recipe has been hardened against three real failure modes (the four testthat-only assertion functions, the pkgload-namespace contamination, and the library vs loadNamespace test- visibility difference) before any reader attempts the migration on their own package.

The migration commit and follow-up commits are on branch testthat-to-tinytest of the zzpower repository.

Time spent: roughly two hours, including writing the brace-aware conversion script (about 30 lines of R), running two dry-runs to surface the testthat-namespace contamination issue, and patching the recipe with the four newly-discovered patterns. The migration commit and follow-up commits are on branch testthat-to-tinytest of the zzpower repository.

Second case study: `zztable1`

zzpower was small enough that the recipe gaps it surfaced might have been idiosyncratic. A second migration on zztable1 (publication-quality summary tables, version 0.5.0; 13 testthat files, 2,393 LOC, 564 passing assertions plus 1 skipped) exercised the recipe at roughly twice the scale and exposed five additional patterns the first case missed.

Pre-migration, zztable1 had: 36 expect_s3_class calls, 24 expect_is calls (testthat’s older alias for the same operation), 19 expect_type calls, 18 expect_lt calls and one expect_gte, and several skip_* invocations including an in-block skip_if_not(capabilities('long.double'), ...) inside a test_that block. It also had two condition matchers using the regex = and regexp = argument names, a library(testthat) line in 10 of the 13 files, and a single context(...) call.

The migration applied the pattern table in section 12 and the conversion-atlas additions, with the following new findings folded back into the recipe:

Ordering assertions (expect_lt, expect_gt, expect_lte, expect_gte) have no tinytest equivalent. Replace with expect_true(x < y) and similar forms. This added 38 substitutions in zztable1; the same pattern added zero in zzpower because that package’s tests happened not to use ordering assertions.
expect_is is testthat’s deprecated alias for expect_s3_class. It behaves identically; the substitute is expect_inherits. zztable1’s test suite used both forms inconsistently across files.
expect_error and expect_warning accept both regex = and regexp = as the matching-argument name. tinytest accepts only pattern =. Both names need to be renamed.
A skip_if_not() call placed inside a test_that() block was scoped to that block alone in testthat. A naive flat conversion places exit_file() at the file level, which halts the entire file. This dropped 12 assertions from test_performance.R until the conversion was rewritten as a block-level if (cond) { ... } wrapper.
library(testthat) calls inside test files and bare context('...') declarations are testthat-specific and must be removed from the converted files. Neither surfaces as an error if the testthat package is still installed (because library(testthat) succeeds and context is exported), but both are noise that will error once testthat is finally removed from Suggests.

After applying these substitutions, 564 of 564 expected assertions passed under tinytest::test_package('zztable1'). R CMD check zztable1_0.5.0.tar.gz reports checking tests ... OK. Three other warnings remain (build directory, dependencies in R code, package vignettes), all stemming from a missing zzobj2fig package in Suggests/Remotes and unrelated to the migration.

Time spent on zztable1: approximately three hours. The incremental cost over zzpower came almost entirely from the five new patterns; the brace-aware conversion script itself ran without modification. The recipe has been updated to cover all nine testthat-only assertion functions and the two scoping issues, so a third migration should not turn up additional surprises at this scale.

Appendix C: The `zzedc` Migration (Origins of the Translator)

The recipe in this post was hardened against the zzedc electronic data capture package before zzpower or zztable1 were attempted. zzedc is the largest package in the portfolio: 51 test files, 989 test blocks, 3,064 assertions. This appendix records why the migration was attempted on that package first, what the Python translator was designed to handle, and the seven-round arc that produced the post-round-7 artifact now in the companion repository.

C.1 When migration is worth the cost

Most R packages should leave testthat alone. The framework is well documented, well integrated with usethis and devtools, and has a much larger user base than any alternative. If a package is already working well on testthat, that is the correct choice.

For a smaller class of packages the dependency calculus is different. testthat 3 brings roughly thirty transitive packages into a package’s Suggests field. tinytest, by Mark van der Loo, brings zero. The migration becomes worth considering in five concrete contexts:

CRAN-bound packages where reviewers see and weigh every dependency.
CI matrices where each row reinstalls testthat and its full transitive tail.
Long-term reproducibility stacks (Docker images, renv lockfiles) where a thirty-package surface ages less gracefully than a one-package surface.
Pharmaverse and FDA-submission contexts, where every dependency appears in a software bill of materials and must be justified.
Embedded R deployments (AWS Lambda, scratch containers, minimal Posit Connect images) where install size matters.

For zzedc, the relevant constraint was the third and fourth: the package targets clinical research environments where the dependency graph is itself a documentation artifact, and the existing research-compendium workflow already exercised tinytest elsewhere in the portfolio. If none of these apply, the migration is not worth the disruption.

C.2 The naive approach and why it fails

The temptation is to reach for sed. Most expect_* names are shared between the two frameworks, and the obvious script handles a useful fraction of cases:

sed -E \
  -e 's/library\(testthat\)/library(tinytest)/' \
  -e '/^context\(/d' \
  tests/testthat/test-*.R

The result parses. It also fails in at least four substantive ways.

First, test_that() has no tinytest analogue. The wrapper must be removed. A naive deletion of the wrapping line and its closing brace breaks block scoping: testthat runs each block in a fresh function frame, so on.exit() fires at the end of the block and local variables do not leak. Top-level tinytest code has neither property.

Second, skip_if(requireNamespace('haven', quietly = TRUE), 'msg') contains a comma inside the inner call. A regex that splits at the first comma will mangle the condition. The translator needs a paren-balanced parser, not a regex.

Third, multi-line string literals containing YAML, SQL, or JSON fixtures often use indentation that is part of the string content. An early version of the translator de-indented test bodies to compensate for the dropped wrapper, which silently corrupted those literals.

Fourth, expect_warning(x, NA) is testthat idiom for ‘expect no warning’. A direct translation passes NA to tinytest::expect_warning and reports a confusing failure. The correct rewrite is expect_silent(x).

These are not edge cases. All four appeared in the first hundred lines of the zzedc test suite.

C.3 What the Python translator handles

The Python translator (testthat_to_tinytest.py) handles the patterns below. Each is a fix for a class of failure surfaced during the zzedc migration.

Block dropping with scope preservation. test_that('desc', { ... }) becomes a # Test: desc comment plus local({ ... }) around the body:

test_that('creates database', {
  con <- dbConnect(SQLite(), ':memory:')
  on.exit(dbDisconnect(con))
  expect_true(dbIsValid(con))
})

becomes

# Test: creates database
local({
  con <- dbConnect(SQLite(), ':memory:')
  on.exit(dbDisconnect(con))
  expect_true(dbIsValid(con))
})

The local() wrapper is the cheapest way to recover testthat’s function-frame semantics without rewriting every test.

Nested BDD blocks. describe(...) { it(...) { ... } } is handled by iterating the block translator to a fixed point. Each pass strips one layer; ten passes covers everything seen in practice.

Comparison rewrites. tinytest does not provide expect_lt/expect_gt/expect_lte/expect_gte as first-class expectations. The translator rewrites them to expect_true(x < y) and so on. Similarly expect_length(x, n) becomes expect_equal(length(x), n), expect_null(x) becomes expect_true(is.null(x)), expect_s3_class(x, 'c') becomes expect_true(inherits(x, 'c')), and expect_no_error(expr) becomes expect_silent(expr).

No-warning idiom. expect_warning(x, NA) and expect_warning(x, regexp = NA) translate to expect_silent(x). The same applies to expect_error(x, NA).

Argument renames. regexp = '...' becomes pattern = '...' for tinytest’s expect_match and related functions.

Paren-balanced skip translation. skip_if(requireNamespace('haven', quietly = TRUE), 'msg') is parsed as a single call with two top-level arguments. The result is if ((requireNamespace('haven', quietly = TRUE))) exit_file('msg').

Block-scoped skip semantics. tinytest’s exit_file() aborts the entire file, whereas testthat’s skip_* aborts only the current block. When a skip_* call is the first directive of a test_that body, the translator wraps the rest of the block in if (!cond) local({ ... }) rather than emitting a top-level exit_file(). The skip stays scoped to its original block.

C.4 The helper-file consolidation problem

testthat auto-loads any file in tests/testthat/ whose name begins with helper-. tinytest has no such convention. It does auto-load files in inst/tinytest/ that begin with an underscore, but only when invoked through tinytest::test_package(); running individual files via tinytest::run_test_file() does not trigger the auto-load.

The convention used for zzedc is to consolidate the four helper-*.R files (helper-test-setup.R, helper-skip.R, helper-db-backends.R, helper-test-utilities.R) into a single inst/tinytest/_setup.R, and to source it explicitly from the top of each translated test file:

if (file.exists('_setup.R')) source('_setup.R')

This is honest: one large setup file is less ergonomic than four small ones, but tinytest’s model is simpler for users to reason about, and the explicit source() keeps individual test files runnable without invoking the package-level harness.

The translator emits the source() line; the consolidated _setup.R is hand-written. The zzedc _setup.R runs to about 680 lines, combining database fixtures, multi-backend test harnesses, custom expectations, and Sys.setenv/Sys.unsetenv lifecycle helpers. The testthat::skip() call is reimplemented as a thin wrapper around exit_file() so existing helper code continues to compile.

C.5 The seven-round migration arc

The zzedc migration proceeded in seven rounds. The arc is a useful empirical record of what a hardened translator does and does not catch on first contact with a real package.

Round 1, naive translator: 0 of 989 tests ran. The first file had a parse error from a YAML literal whose indentation had been stripped by an over-eager de-indent pass; the tinytest runner aborts on a single parse failure. Removing the de-indent step and relying on local({}) for scoping fixed this.

Round 2, after local({}) wrapping: 981 of 989 tests ran. Eight failures remained.

Rounds 3 through 6 each surfaced one new pattern: paren-balanced skip conditions, block-scoped skip wrapping, the regexp = NA idiom, S3 class inheritance, and the custom expect_table_exists helper.

Round 7: 3,064 of 3,064 assertions passing.

The eight final failures resolved into two classes. Seven were indentation-stripping inside YAML literals that survived the first de-indent fix because they were nested two levels deep; removing de-indentation entirely (rather than partially) eliminated the class. The eighth was the expect_warning(x, NA) semantics, which had been masquerading as a passing test under testthat because the regex was NA and matched nothing rather than checking warning suppression.

The version of the translator in the companion repository is the post-round-7 artifact.

C.6 Using the tools

For a single file during development, invoke the Python translator directly:

python3 testthat_to_tinytest.py \
  tests/testthat/test-mything.R \
  inst/tinytest/test_mything.R

The output is a self-contained tinytest file that sources _setup.R from its own directory if present.

For a whole repository, the bash orchestrator (migrate.sh) handles the surrounding DESCRIPTION and scaffold edits. Phase 1 is fully automated for repositories whose tests are three-line testthat stubs (the pattern produced by usethis::use_testthat() followed by no further work). Phase 2 generates per-file conversion previews into .migration-previews/ for repositories with substantive test code, which the user reviews and applies manually:

./migrate.sh phase1 [--dry]
./migrate.sh phase2

Beyond translation, the orchestrator replaces testthat with tinytest in DESCRIPTION, removes Config/testthat/edition: 3 if present, deletes the tests/testthat/ tree, writes a new tests/tinytest.R, regenerates renv.lock from the trimmed DESCRIPTION, and commits the result.

After translation, the manual checklist for a real-test repository is:

Hand-write inst/tinytest/_setup.R consolidating any helper-*.R files. Replace library(testthat) with nothing and testthat::skip(msg) with exit_file(msg) (or with a wrapped form where block scoping is needed).
Inspect any expect_snapshot and local_edition calls; they will not work without manual rewriting.
Run tinytest::run_test_dir('inst/tinytest') and triage the first round of failures.

The translator handles the rote mechanical work: paren-balanced parsing, the seventeen expect_* rewrites, the scope-preserving local({...}) wrapping, the block-scoped skip semantics. It does not replace the judgment required to convert snapshot tests, to consolidate helper files, or to decide which Suggests packages should now move to Imports because the test suite was the only place using them. That judgment is the actual work of the migration; the translator makes the mechanical part fast enough that the judgment becomes the bottleneck rather than the typing.

Rendered on 2026-05-18 at 09:00 PDT.
Source: ~/prj/qblog/posts/62-testthat-to-tinytest/testthat-to-tinytest/analysis/report/index.qmd

Let’s Connect

If you spotted an error, found a pattern that did not port cleanly, or have a different conclusion about the conversion, the comment thread below is the right place. Issues and pull requests against the post’s source are welcome at https://github.com/rgthomas47/qblog.

A 55-Item Initiation Checklist for zzcollab Data Analyses

Ronald G. Thomas — Wed, 29 Apr 2026 07:00:00 GMT

A clean workspace with a printed checklist, an open notebook, and a single CSV file open on screen, signalling deliberate setup before the first line of analysis code is written.

A documented, numbered checklist turns the first week of a data analysis project from improvisation into routine. The gain is not speed; it is the disappearance of the small omissions that surface, weeks later, as untestable models or unreproducible figures.

Introduction

A clinical research collaborator emails a single attachment. The body of the email reads, in part:

Attached is the analytic dataset for Study A, frozen as of last Friday. Twelve columns, two hundred and eighty-seven rows, one row per participant per study visit. The protocol is IRB #2024-XYZ. Please treat as confidential; no PHI but a DUA is in place. An updated extract may follow if a few late withdrawals come in, but the structure will be the same. We expect to write this up for submission in Q3.

Opening the file reveals a header row that reads:

ParticipantID,Status,Sex,Group,VisitAge,VisitType,
Outcome_A_Imp,Outcome_B_Imp,Sub_a,Sub_b,Sub_c,Sub_total_Imp

Some columns are immediately legible (ParticipantID, Sex, Group). Others are not. What does the _Imp suffix mean? Is VisitAge the participant’s age at the visit, or the age of the visit measured from baseline? Is Sub_total_Imp a sum of Sub_a, Sub_b, Sub_c, an average, or something else? Twelve columns, two hundred and eighty-seven rows, no codebook, no README. The clock starts on reply ‘received, will review’. By Q3 a reproducible compendium is needed that another analyst could clone and re-run.

The fifty-five items below are a six-phase preparation checklist that moves this attachment from inbox to publishable compendium. The structure is the same whether the dataset is a clinical trial, an observational cohort, a simulation output, or any other tabular extract that will be analysed in a zzcollab research compendium.

The Hypothetical Client Dataset

To anchor the items, assume the following throughout.

File: studyA_data_2026Q1.csv (a single UTF-8 CSV with a BOM in the header).

Shape: twelve columns, two hundred and eighty-seven rows. One observation per participant per study visit.

Columns (in header order):

Column	Apparent type	Notes from the email
`ParticipantID`	integer	participant identifier
`Status`	categorical	enrolment status
`Sex`	categorical	M or F
`Group`	categorical	randomised arm
`VisitAge`	numeric	age (years)
`VisitType`	categorical	baseline + follow-up
`Outcome_A_Imp`	numeric	primary outcome A
`Outcome_B_Imp`	numeric	primary outcome B
`Sub_a`	integer	subscale item
`Sub_b`	integer	subscale item
`Sub_c`	integer	subscale item
`Sub_total_Imp`	numeric	subscale composite

What is missing:

A codebook or data dictionary.
A README documenting provenance, IRB protocol, embargo, and the meaning of _Imp.
Any indication of which columns can be missing (structural NAs versus unexpected ones).
The pre-specified statistical analysis plan (SAP), other than ‘we will write up Outcomes A and B as primary’.

These gaps are the substrate the checklist operates on.

Why a Checklist?

A checklist is not a workflow. It is a set of named, verifiable end states ordered such that earlier items unblock later ones. The annotation paragraphs that accompany each item are scaffolding for first-time use; the durable feature is the numbering.

A numbered checklist solves three problems that recur on every project. First, it surfaces the steps that are easy to skip under deadline pressure (data dictionary, baseline integrity test, fresh-container reproducibility run). The items most often skipped are also the items reviewers most often ask for. Second, it gives collaborators a shared vocabulary: ‘Item 17 is failing in the new tinytest file’ is a precise bug report. Third, it forces reproducibility verification before results are circulated, not after.

The zzcollab Workspace

The target state is a zzcollab research compendium: an R package layout extended with analysis/, an inst/tinytest/ directory for data-integrity tests, and the Five Pillars (Dockerfile, renv.lock, .Rprofile, source code, data) that together specify the computational environment. The checklist below is written against that layout. A fresh workspace is created with zzc analysis and contains, in broad outline:

projectname/
|-- analysis/
|   |-- data/
|   |   |-- raw_data/         (the client's CSV lives here)
|   |   `-- derived_data/     (cleaned files, model RDS)
|   |-- scripts/              (numbered analysis scripts)
|   |-- report/               (Rmd / qmd reports)
|   `-- figures/              (publication-ready outputs)
|-- R/                        (reusable functions)
|-- inst/tinytest/            (data integrity assertions)
|-- tests/testthat/           (function unit tests)
|-- docs/                     (data dictionary, checklist)
|-- DESCRIPTION
|-- Dockerfile
|-- renv.lock
|-- .Rprofile
|-- Makefile
`-- zzcollab.yaml

Every checklist item references a specific path inside this tree. Reading the items alongside the layout shows how the work product accumulates.

The 55-Item Preparation Checklist

The items are organised into six phases. Each phase corresponds to a deliverable that lives at a documented location.

Phase	Items	Deliverable
1	1-8	A buildable workspace with documented raw data
2	9-22	A cleaned, validated derived dataset
3	23-31	An EDA report with Table 1 and trajectory plots
4	32-38	Tested R functions for summary, modelling, plotting
5	39-45	A primary-analysis report with sensitivity analyses
6	46-55	A reproducible compendium ready for archival

Items not applicable to a particular project should be marked ‘N/A’ with a one-line reason rather than removed. The numbering is meant to be stable across projects so that two analysts can compare progress unambiguously.

Phase 1: Project Setup (Items 1-8)

Phase 1 produces a buildable workspace with raw data in place and provenance documented.

Verify Dockerfile exists and builds successfully. The Dockerfile defines the computational environment that every subsequent step will run inside; an unbuildable Dockerfile blocks every phase that follows.
```
zzc docker --build
docker images studyA            # confirm recent build
```
Verify renv.lock contains required packages. The lockfile should list only direct dependencies at this stage; transitive dependencies are resolved by renv::restore() inside the container. The lockfile is the second of the Five Pillars and pins the package versions that make the analysis reproducible.
```
jq '.Packages | keys' renv.lock | head -20
```
Verify .Rprofile has appropriate R options. Confirm .Rprofile activates renv on session start:
```
# .Rprofile
source('renv/activate.R')
options(renv.config.auto.snapshot = TRUE)
```
The zzcollab template also configures auto-restore on session start so that a fresh container picks up the lockfile automatically.

Verify source code directories exist. The layout is the contract the rest of the checklist relies on; if R/ does not exist, Phase 4 has nowhere to put its functions.

ls -d R/ analysis/scripts/ analysis/data/ tests/ \
      inst/tinytest/ docs/

# Create any missing directories
mkdir -p R analysis/scripts analysis/data/raw_data \
         analysis/data/derived_data \
         analysis/figures analysis/report \
         tests/testthat inst/tinytest docs

Verify raw data is in analysis/data/raw_data/ (read-only). With write permissions on the raw file, an analyst can open it in Excel, edit a cell, save, and never know they have changed the upstream data. With chmod 444 set, every save attempt fails loudly.
```
mv ~/Downloads/studyA_data_2026Q1.csv \
   analysis/data/raw_data/
chmod 444 analysis/data/raw_data/studyA_data_2026Q1.csv
ls -la analysis/data/raw_data/
```

Update analysis/data/README.md with data source, date received, and use restrictions. Document who provided the file (the collaborator’s name and institution), the date and time of receipt, the IRB protocol number (#2024-XYZ), the DUA status, and the embargo. Quote the relevant sentences from the email verbatim. A minimal template:

# Study A Raw Data

- **Source:** Dr. , 
- **Received:** 2026-04-29 via email attachment
- **IRB protocol:** #2024-XYZ
- **DUA:** in place; treat as confidential
- **Embargo:** none stated
- **Update window:** 'an updated extract may follow if
  a few late withdrawals come in' (per email).

## Files

- `studyA_data_2026Q1.csv` (12 cols, 287 rows)

Record data dictionary: column definitions, units, coding schemes. Create docs/data-dictionary.md with one entry per column. Resolve undocumented suffixes (_Imp) with the data provider before writing the entry; do not guess. A minimal template for one column:
```
## ParticipantID

- **Type:** integer
- **Range:** 1001-9999
- **Missing:** never
- **Interpretation:** unique participant identifier
- **Notes:** 4-digit; first digit is study site
```
Repeat for all twelve columns. Treat the dictionary as a contract between the data provider and the analyst; the integrity tests in Phase 2 should validate the contract, not the data.
Install analysis packages: tidyverse, here, janitor, skimr. Enter the container, install the packages, snapshot the lockfile, and add each package to the DESCRIPTION Imports field so the declared dependency set matches the installed environment.
```
make r
```
```
install.packages(c(
  'tidyverse', 'here', 'janitor', 'skimr',
  'tinytest', 'gtsummary', 'broom', 'broom.mixed',
  'lme4', 'patchwork', 'assertthat'
))
renv::snapshot()
```

Phase 2: Data Ingestion and Validation (Items 9-22)

Phase 2 produces a cleaned, validated derived dataset and a test file that asserts every contract from the data dictionary. The deliverables are R/load_data.R, inst/tinytest/test_data_integrity.R, and analysis/scripts/01-clean-data.R.

Create R/load_data.R with a function to read raw data. The function returns the raw data frame, never a printed object. Use here::here() for the path so the function works from any working directory. The CSV carries a UTF-8 BOM in the header; read_csv() handles this automatically.
```
# R/load_data.R
load_raw_data <- function() {
  readr::read_csv(
    here::here(
      'analysis/data/raw_data/studyA_data_2026Q1.csv'
    ),
    show_col_types = FALSE
  )
}
```
Create inst/tinytest/test_data_integrity.R. The test file sources load_raw_data() once and runs every assertion (Items 11-17) against the resulting data frame. Run with tinytest::run_test_file(). The test file is the living version of the data dictionary contract.
```
# inst/tinytest/test_data_integrity.R
library(tinytest)
source(here::here('R/load_data.R'))

dat <- load_raw_data()
```
Test: expected number of columns (12). A mismatch signals that the upstream data export has changed or that the file was corrupted during transfer.
```
N_EXPECTED_COLS <- 12L
expect_equal(ncol(dat), N_EXPECTED_COLS)
```

Test: expected column names present. A failure indicates the data provider has renamed a field or that an export script has changed; either way, the analysis cannot proceed without resolving it.

expected_cols <- c(
  'ParticipantID', 'Status', 'Sex', 'Group',
  'VisitAge', 'VisitType',
  'Outcome_A_Imp', 'Outcome_B_Imp',
  'Sub_a', 'Sub_b', 'Sub_c', 'Sub_total_Imp'
)
expect_true(all(expected_cols %in% names(dat)))

Test: ID columns have no NAs and follow expected format. ParticipantID is integer-valued with no missing values; the documented range is 1001-9999. A missing identifier breaks every downstream join.

expect_true(all(!is.na(dat$ParticipantID)))
expect_true(is.numeric(dat$ParticipantID))
expect_true(all(dat$ParticipantID >= 1001 &
                  dat$ParticipantID <= 9999))

Test: pre-specified categorical variables match allowed levels. Catches typos and miscoded values before they propagate into the model.

allowed_groups <- c('Active', 'Control')
expect_true(all(dat$Group %in% allowed_groups))

allowed_status <- c('Enrolled', 'Withdrawn',
                    'Completed')
expect_true(all(dat$Status %in% allowed_status))

Test: demographic categorical variables match coding scheme. Investigate any unexpected values; do not silently recode them. Differences from the documented set are sometimes legitimate and sometimes errors; only the data provider can tell.
```
allowed_sex <- c('M', 'F')
expect_true(all(dat$Sex %in% allowed_sex))
expect_true(all(!is.na(dat$Sex)))
```
Test: visit variables fall in the pre-specified set. P1 is the baseline assessment; V2-V4 are follow-ups. Any other value indicates a data export error or a silently extended protocol.
```
allowed_visits <- c('P1', 'V2', 'V3', 'V4')
expect_true(all(dat$VisitType %in% allowed_visits))
```

Test: numeric columns are within plausible ranges. Use na.rm = TRUE so that missing follow-up data does not trip the check.

# Age (years)
expect_true(all(dat$VisitAge >= 18 &
                  dat$VisitAge <= 100,
                na.rm = TRUE))

# Likert subscale items (0-3)
for (col in c('Sub_a', 'Sub_b', 'Sub_c')) {
  x <- dat[[col]]
  expect_true(all(x >= 0 & x <= 3, na.rm = TRUE))
}

# Subscale composite (0-9)
expect_true(all(dat$Sub_total_Imp >= 0 &
                  dat$Sub_total_Imp <= 9,
                na.rm = TRUE))

# Outcomes (placeholder bounds; refer to dictionary)
expect_true(all(dat$Outcome_A_Imp >= 36 &
                  dat$Outcome_A_Imp <= 78,
                na.rm = TRUE))
expect_true(all(dat$Outcome_B_Imp >= 0 &
                  dat$Outcome_B_Imp <= 28,
                na.rm = TRUE))

Create analysis/scripts/01-clean-data.R. The script sources load_raw_data(), applies the cleaning steps in Items 19-21, and writes the result to derived data (Item 22). Structure as a linear pipe so the transformations are auditable in sequence.

# analysis/scripts/01-clean-data.R
library(tidyverse)
library(janitor)
library(here)
source(here('R/load_data.R'))

dat_clean <- load_raw_data() |>
  clean_names() |>
  mutate(
    group = factor(group,
                   levels = c('Control', 'Active')),
    sex = factor(sex, levels = c('F', 'M')),
    visit_type = factor(
      visit_type,
      levels = c('P1', 'V2', 'V3', 'V4'),
      ordered = TRUE
    ),
    status = factor(status,
                    levels = c('Enrolled',
                               'Withdrawn',
                               'Completed'))
  )

saveRDS(
  dat_clean,
  here('analysis/data/derived_data/cleaned_data.rds')
)

Clean column names using janitor::clean_names(). clean_names() converts headers to snake_case and normalises the BOM on the first column. Update the data dictionary to record both raw and cleaned names.

# Before clean_names():
#   ParticipantID, Outcome_A_Imp, Sub_total_Imp, ...

dat_clean <- dat |> janitor::clean_names()

# After clean_names():
#   participant_id, outcome_a_imp, sub_total_imp, ...

Convert categorical variables to factors with explicit levels. Place the reference level first (Control before Active, F before M) so model contrasts produce intuitive coefficients. Preserve the time ordering for visit_type (P1 < V2 < V3 < V4).

dat_clean <- dat_clean |>
  mutate(
    # Control as reference => coefficient is the
    # 'Active vs Control' effect
    group = factor(group,
                   levels = c('Control', 'Active')),

    # Female as reference (alphabetical)
    sex = factor(sex, levels = c('F', 'M')),

    # Ordered factor preserves visit sequence
    visit_type = factor(
      visit_type,
      levels = c('P1', 'V2', 'V3', 'V4'),
      ordered = TRUE
    )
  )

Handle missing values explicitly and document approach. Tabulate NAs per column and per visit before deciding on a strategy. The _Imp suffix suggests imputation was already done upstream; confirm with the data provider before writing any imputation yourself.

dat_clean |>
  summarise(across(everything(),
                   ~ sum(is.na(.x)))) |>
  pivot_longer(everything(),
               names_to = 'column',
               values_to = 'n_missing') |>
  arrange(desc(n_missing))

# Per-visit missingness for the primary outcome
dat_clean |>
  group_by(visit_type) |>
  summarise(
    n_total = n(),
    n_missing_a = sum(is.na(outcome_a_imp))
  )

Save cleaned data to analysis/data/derived_data/. All downstream scripts read from derived data, never from raw. This makes the cleaning step the single point of transformation.

saveRDS(
  dat_clean,
  here::here(
    'analysis/data/derived_data/cleaned_data.rds'
  )
)

# Optional: also write a CSV for non-R consumers
readr::write_csv(
  dat_clean,
  here::here(
    'analysis/data/derived_data/cleaned_data.csv'
  )
)

Workspace ambiance: an analyst at the second day of a project, with the printed checklist on the desk and the loaded data frame on screen.

Phase 3: Exploratory Analysis (Items 23-31)

Phase 3 produces the first knittable report. The deliverables are analysis/scripts/02-eda.R and analysis/report/01-eda.Rmd (or .qmd).

Create analysis/scripts/02-eda.R. The second step in the linear pipeline. It reads cleaned data from derived data, never from raw.

# analysis/scripts/02-eda.R
library(tidyverse)
library(skimr)
library(here)

dat <- readRDS(
  here('analysis/data/derived_data/cleaned_data.rds')
)

Generate summary statistics by treatment group. For each numeric outcome and each pre-specified grouping, produce mean, SD, median, IQR, min, max, and count of non-missing observations. Save the summary as a CSV in derived data.

summary_by_group_visit <- dat |>
  group_by(group, visit_type) |>
  summarise(
    across(
      c(outcome_a_imp, outcome_b_imp, sub_total_imp),
      list(
        n    = ~ sum(!is.na(.x)),
        mean = ~ mean(.x, na.rm = TRUE),
        sd   = ~ sd(.x, na.rm = TRUE),
        med  = ~ median(.x, na.rm = TRUE)
      ),
      .names = '{.col}_{.fn}'
    ),
    .groups = 'drop'
  )

readr::write_csv(
  summary_by_group_visit,
  here('analysis/data/derived_data/summary_by_group_visit.csv')
)

Check baseline balance. For randomised trials, imbalance at baseline is informative for downstream covariate adjustment, not a basis for re-randomisation.

baseline <- dat |> filter(visit_type == 'P1')

baseline |>
  group_by(group) |>
  summarise(
    n          = n(),
    mean_age   = mean(visit_age, na.rm = TRUE),
    sd_age     = sd(visit_age, na.rm = TRUE),
    pct_female = mean(sex == 'F') * 100,
    mean_a    = mean(outcome_a_imp, na.rm = TRUE),
    mean_b    = mean(outcome_b_imp, na.rm = TRUE)
  )

Visualise outcome distributions. Inspect for floor and ceiling effects, bimodality, heavy tails, and outliers.

p_dist <- ggplot(dat,
                 aes(x = outcome_a_imp,
                     fill = group)) +
  geom_density(alpha = 0.5) +
  facet_wrap(~ visit_type) +
  labs(title = 'Outcome A by group and visit',
       x = 'Outcome A (imputed)',
       y = 'Density')

ggsave(
  here('analysis/figures/dist-outcome-a.png'),
  p_dist, width = 8, height = 5, dpi = 300
)

Identify potential outliers or data quality issues. Flag observations that fall outside the plausibility ranges from Item 17, or that deviate substantially from the within-group distribution. Document each flagged row with the decision in a tracking file.

flags <- dat |>
  mutate(
    flag_age   = visit_age < 18 | visit_age > 100,
    flag_sub_a = sub_a < 0 | sub_a > 3,
    flag_sub_b = sub_b < 0 | sub_b > 3,
    flag_sub_c = sub_c < 0 | sub_c > 3
  ) |>
  filter(flag_age | flag_sub_a | flag_sub_b |
           flag_sub_c)

if (nrow(flags) > 0) {
  readr::write_csv(
    flags,
    here('analysis/data/derived_data/data-flags.csv')
  )
}

Create analysis/report/01-eda.Rmd (or .qmd). The first report is a stand-alone Rmd or qmd that knits to HTML or PDF. It loads cleaned data, reads the EDA outputs from derived data, and presents the results in narrative form.
```
quarto render analysis/report/01-eda.qmd
```

Document sample size and missingness patterns. Report the number of participants enrolled, randomised, and analysed at each visit. Tabulate missingness by variable and by visit so the reader can judge the analytic implications.

# Per-visit sample sizes by group
dat |>
  count(visit_type, group) |>
  pivot_wider(names_from = group, values_from = n)

# Per-visit missingness for the primary outcome
dat |>
  group_by(visit_type) |>
  summarise(
    n_total    = n(),
    n_observed = sum(!is.na(outcome_a_imp)),
    pct_obs    = 100 * n_observed / n_total
  )

Include baseline characteristics table (Table 1). Use gtsummary::tbl_summary() stratified by group. For randomised trials, avoid statistical tests of baseline balance; they are uninformative when randomisation succeeded.

library(gtsummary)

tab1 <- dat |>
  filter(visit_type == 'P1') |>
  select(group, sex, visit_age,
         outcome_a_imp, outcome_b_imp,
         sub_total_imp) |>
  tbl_summary(
    by = group,
    statistic = list(
      all_continuous() ~ '{mean} ({sd})',
      all_categorical() ~ '{n} ({p}%)'
    )
  ) |>
  add_overall()

tab1

Include outcome trajectory plots by visit and group. Plot the mean trajectory over visit_type with error bars; spaghetti plots of individual trajectories are a useful supplement when sample sizes are modest.

p_traj <- dat |>
  group_by(group, visit_type) |>
  summarise(
    m  = mean(outcome_a_imp, na.rm = TRUE),
    se = sd(outcome_a_imp, na.rm = TRUE) /
           sqrt(sum(!is.na(outcome_a_imp))),
    .groups = 'drop'
  ) |>
  ggplot(aes(x = visit_type, y = m,
             colour = group, group = group)) +
  geom_line() +
  geom_errorbar(aes(ymin = m - se, ymax = m + se),
                width = 0.1) +
  labs(x = 'Visit', y = 'Outcome A (mean +/- SE)')

ggsave(
  here('analysis/figures/trajectory-outcome-a.png'),
  p_traj, width = 6, height = 4, dpi = 300
)

Phase 4: Analysis Functions (Items 32-38)

Phase 4 produces tested R functions for summary, modelling, and plotting. The deliverables are R/summarize_outcomes.R, R/model_outcomes.R, R/plot_outcomes.R, and corresponding test files under inst/tinytest/.

Create R/summarize_outcomes.R with descriptive statistics functions. Each function takes a data frame and returns a data frame, never a printed object. Document arguments and return values with roxygen2.

# R/summarize_outcomes.R

#' Mean and SD of an outcome by group and visit
#'
#' @param dat cleaned data frame
#' @param outcome column name as a string
#' @return tibble with group, visit_type, n, mean, sd
#' @export
summarize_outcome_by_group_visit <- function(dat,
                                              outcome) {
  stopifnot(outcome %in% names(dat))
  dat |>
    dplyr::group_by(group, visit_type) |>
    dplyr::summarise(
      n    = sum(!is.na(.data[[outcome]])),
      mean = mean(.data[[outcome]], na.rm = TRUE),
      sd   = sd(.data[[outcome]], na.rm = TRUE),
      .groups = 'drop'
    )
}

Create R/model_outcomes.R with statistical modelling functions. Wrap each call to lm(), lmer(), or glm() in a function so the model specification is captured in code rather than reproduced inline.

# R/model_outcomes.R

#' Fit a linear mixed model for a primary outcome
#'
#' Random intercept for participant; fixed effects of
#' group, visit, their interaction, and baseline
#' covariates.
#'
#' @param dat cleaned data frame
#' @param outcome column name as a string
#' @return fitted lmerMod object
#' @export
fit_primary_lmm <- function(dat, outcome) {
  stopifnot(outcome %in% names(dat))
  f <- stats::as.formula(
    paste0(
      outcome,
      ' ~ group * visit_type + visit_age + sex + ',
      '(1 | participant_id)'
    )
  )
  lme4::lmer(f, data = dat)
}

Create R/plot_outcomes.R with visualisation functions. Plotting functions return a ggplot; they do not call ggsave(). Returning the object lets reports compose panels with patchwork.

# R/plot_outcomes.R

#' Mean +/- SE trajectory plot for one outcome
#'
#' @param dat cleaned data frame
#' @param outcome column name as a string
#' @return ggplot object
#' @export
plot_trajectory <- function(dat, outcome) {
  assertthat::assert_that(outcome %in% names(dat))
  assertthat::assert_that('group' %in% names(dat))
  assertthat::assert_that('visit_type' %in% names(dat))

  dat |>
    ggplot2::ggplot(
      ggplot2::aes(x = visit_type,
                   y = .data[[outcome]],
                   colour = group, group = group)) +
    ggplot2::stat_summary(fun = mean,
                          geom = 'line') +
    ggplot2::stat_summary(fun.data = mean_se,
                          geom = 'errorbar',
                          width = 0.1)
}

Create inst/tinytest/test_summarize.R. Each test uses a small hand-constructed data frame with known summary statistics.

# inst/tinytest/test_summarize.R
library(tinytest)
source(here::here('R/summarize_outcomes.R'))

tiny <- tibble::tribble(
  ~participant_id, ~group, ~visit_type, ~outcome_a_imp,
  1001L, 'Active',  'P1', 5.0,
  1002L, 'Active',  'P1', 7.0,
  1003L, 'Control', 'P1', 4.0,
  1004L, 'Control', 'P1', 6.0
)

out <- summarize_outcome_by_group_visit(
  tiny, 'outcome_a_imp')
expect_equal(out$mean[out$group == 'Active'],  6.0)
expect_equal(out$mean[out$group == 'Control'], 5.0)
expect_equal(nrow(out), 2L)

Create inst/tinytest/test_model.R. Test modelling functions against simulated data with known parameters. Use set.seed() for reproducibility.

# inst/tinytest/test_model.R
library(tinytest)
library(lme4)
source(here::here('R/model_outcomes.R'))

set.seed(42)
n <- 200
sim <- tibble::tibble(
  participant_id = rep(1:(n / 2), each = 2),
  group = factor(
    rep(c('Active', 'Control'), each = n / 2)),
  visit_type = factor(
    rep(c('P1', 'V2'), times = n / 2)),
  visit_age = rnorm(n, 50, 10),
  sex = factor(sample(c('M', 'F'), n, replace = TRUE))
)
# True effect of Active vs Control = 2.0
sim$outcome_a_imp <- 5 +
  2 * (sim$group == 'Active') + rnorm(n, 0, 1)

fit <- fit_primary_lmm(sim, 'outcome_a_imp')
est <- lme4::fixef(fit)[['groupActive']]
expect_true(abs(est - 2) < 0.5)

Test edge cases: missing data handling. The _Imp columns should not have NAs, but their unimputed counterparts might. Lock down the documented behaviour with a test.

tiny_with_na <- tiny
tiny_with_na$outcome_a_imp[1] <- NA_real_

out <- summarize_outcome_by_group_visit(
  tiny_with_na, 'outcome_a_imp')
# Active row: n drops from 2 to 1, mean from 6 to 7
expect_equal(out$n[out$group == 'Active'],    1L)
expect_equal(out$mean[out$group == 'Active'], 7.0)

Test edge cases: single-group scenarios. A pre-specified subgroup may turn out empty in the sample. Assert either a graceful empty result or a clear error.

tiny_single <- tiny |>
  dplyr::filter(group == 'Active')

out <- summarize_outcome_by_group_visit(
  tiny_single, 'outcome_a_imp')
expect_equal(nrow(out), 1L)
expect_equal(out$group[1], 'Active')

Phase 5: Primary Analysis (Items 39-45)

Phase 5 produces the primary-analysis report, including secondary outcomes and sensitivity analyses. The deliverables are analysis/scripts/03-primary-analysis.R, saved fitted-model objects in analysis/data/derived_data/, and analysis/report/02-results.Rmd (or .qmd).

Create analysis/scripts/03-primary-analysis.R. The script orchestrates; R/ holds the reusable code.

# analysis/scripts/03-primary-analysis.R
library(here)
library(broom.mixed)
source(here('R/model_outcomes.R'))

dat <- readRDS(
  here('analysis/data/derived_data/cleaned_data.rds')
)

fit_a <- fit_primary_lmm(dat, 'outcome_a_imp')
fit_b <- fit_primary_lmm(dat, 'outcome_b_imp')
fit_s <- fit_primary_lmm(dat, 'sub_total_imp')

Implement the pre-specified analysis plan. Follow the SAP line by line. Mark any deviation from the SAP with a code comment that begins # SAP DEVIATION: and document the reason.

# SAP Section 4.1: primary contrast 'Active vs Control'
# at the final follow-up visit (V4)
primary_contrast_a <- emmeans::emmeans(
  fit_a,
  pairwise ~ group | visit_type
)$contrasts |>
  as.data.frame() |>
  dplyr::filter(visit_type == 'V4')

# SAP DEVIATION: V3 contrast was added post-hoc at
# the request of the DSMB; report as exploratory.

Save model objects to analysis/data/derived_data/. Persist each fitted model with saveRDS(); save the tidy and glance outputs as CSV for direct inclusion in tables. The derived data folder becomes the single source of truth for model results.

saveRDS(
  fit_a,
  here('analysis/data/derived_data/model_outcome_a_lmm.rds')
)
saveRDS(
  fit_b,
  here('analysis/data/derived_data/model_outcome_b_lmm.rds')
)

readr::write_csv(
  broom.mixed::tidy(fit_a),
  here('analysis/data/derived_data/tidy_outcome_a.csv')
)
readr::write_csv(
  broom.mixed::glance(fit_a),
  here('analysis/data/derived_data/glance_outcome_a.csv')
)

Create analysis/report/02-results.Rmd (or .qmd). The results report loads the saved model objects and renders the result tables and figures. It contains no model-fitting code; if it does, the script and report can drift apart.

# In the report's R chunk, NOT in the script
fit_a <- readRDS(here::here(
  'analysis/data/derived_data/model_outcome_a_lmm.rds'))
tidy_a <- readr::read_csv(here::here(
  'analysis/data/derived_data/tidy_outcome_a.csv'))

knitr::kable(tidy_a, digits = 3)

Document primary outcome analysis in the report. Present the primary outcome model first, with a results paragraph that includes the point estimate, confidence interval, and inferential statistic for the pre-specified primary contrast.

A typical results sentence template:

The mean Outcome A at V4 was lower in the Active
arm than in the Control arm (estimated difference
-3.2 points, 95% CI -5.1 to -1.3, p = 0.001),
consistent with the pre-specified primary
hypothesis. The fitted model was a linear mixed
model with a random intercept per participant and
fixed effects for group, visit, their interaction,
and baseline covariates (age, sex):

`outcome_a_imp ~ group * visit_type + visit_age +
sex + (1 | participant_id)`

Document secondary outcomes in the report. Present each secondary outcome with the same structure as the primary. Acknowledge the multiple-comparison context; either pre-specify a correction or label the analyses as exploratory.

# Holm-adjusted p-values across the three primary
# contrasts (Outcome A, Outcome B, Sub_total)
p_unadj <- c(
  outcome_a    = 0.001,
  outcome_b    = 0.04,
  sub_total    = 0.18
)
p_holm <- p.adjust(p_unadj, method = 'holm')
p_holm

Document sensitivity analyses in the report. For each modelling assumption that could plausibly affect the conclusion, run a sensitivity analysis and document its result alongside the primary.

# Sensitivity 1: complete-case using the unimputed
# underlying column (if available from the data
# provider) instead of the _Imp column.
fit_a_cc <- fit_primary_lmm(dat, 'outcome_a_raw')

# Sensitivity 2: alternative covariate set (drop
# baseline age)
f_alt <- as.formula(
  'outcome_a_imp ~ group * visit_type + sex + ',
  '(1 | participant_id)')
fit_a_alt <- lme4::lmer(f_alt, data = dat)

# Compare estimates from primary vs sensitivity
dplyr::bind_rows(
  primary    = broom.mixed::tidy(fit_a),
  complete   = broom.mixed::tidy(fit_a_cc),
  no_age     = broom.mixed::tidy(fit_a_alt),
  .id = 'analysis'
) |>
  dplyr::filter(term == 'groupActive')

Editor view of the checklist artefact open alongside the working zzcollab project, showing how items are referenced by number from commit messages and TODO notes.

Phase 6: Documentation and Reproducibility (Items 46-55)

Phase 6 closes the loop: DESCRIPTION metadata, a project README, a rebuilt Docker image, and a fresh-container end-to-end reproducibility run.

Update DESCRIPTION: add a meaningful Title and Description. Title is one sentence, fewer than sixty-five characters. Description is one to three sentences naming the study population, primary outcomes, and analytic approach.

Package: studya
Title: Study A: Active vs Control on Outcomes A and B
Version: 0.0.0.9000
Authors@R:
    person('First', 'Last', email = 'you@example.com',
           role = c('aut', 'cre'))
Description: Reproducible compendium for Study A, an
    IRB-approved randomised comparison of Active and
    Control on Outcomes A and B across four study
    visits (P1 baseline, V2-V4 follow-ups). Linear
    mixed models with random participant intercepts.

Update DESCRIPTION: add or verify Authors. Use the Authors@R field with person() entries; do not silently leave the placeholder ‘Your Name’.

Authors@R: c(
    person('Anna', 'Analyst',
           email = 'analyst@example.org',
           role = c('aut', 'cre'),
           comment = c(ORCID = '0000-0000-0000-0000')),
    person('Bob', 'Bioscientist',
           email = 'bob@example.org',
           role = 'ctb'))

Verify all package dependencies listed in DESCRIPTION. Any package called but not declared will fail in a clean container. DESCRIPTION lists direct dependencies only; renv.lock pins the transitive closure.
```
# Inside the container
deps <- renv::dependencies()
unique(deps$Package) |> sort()
```
```
make check-renv          # validates Imports vs code
```
Create or update the project README.md. The README is the entry point for anyone reading the repository for the first time. Include a directory tree (tree -L 2), the research question, and the reproduction commands.
```
tree -L 2 -I 'renv|_freeze|_site' .
```

Document reproduction instructions in the README. Provide a copy-pasteable sequence of shell commands that reproduces the analysis from a clean checkout. Test on a machine that does not already have the project set up.

# Reproduce Study A end-to-end
git clone https://github.com/<owner>/studya.git
cd studya
make docker-build

# Inside the container, run the linear pipeline
make r

source('analysis/scripts/01-clean-data.R')
source('analysis/scripts/02-eda.R')
source('analysis/scripts/03-primary-analysis.R')
quarto::quarto_render('analysis/report/01-eda.qmd')
quarto::quarto_render('analysis/report/02-results.qmd')

Rebuild Docker image: make docker-build. Tag with the current git SHA so future readers can identify the exact build that produced the published results.
```
make docker-build
GIT_SHA=$(git rev-parse --short HEAD)
docker tag studya:latest studya:${GIT_SHA}
docker images studya
```

Verify reproducibility: run all scripts in a fresh container. The end-to-end run is the canonical reproducibility test.

GIT_SHA=$(git rev-parse --short HEAD)

docker run --rm -v "$(pwd):/project" -w /project \
  studya:${GIT_SHA} \
  Rscript analysis/scripts/01-clean-data.R

docker run --rm -v "$(pwd):/project" -w /project \
  studya:${GIT_SHA} \
  Rscript analysis/scripts/02-eda.R

docker run --rm -v "$(pwd):/project" -w /project \
  studya:${GIT_SHA} \
  Rscript analysis/scripts/03-primary-analysis.R

docker run --rm -v "$(pwd):/project" -w /project \
  studya:${GIT_SHA} \
  quarto render analysis/report/02-results.qmd

Confirm all outputs match expected results. Bit-for-bit reproducibility is a strong signal; numerically identical floating-point results without timestamp or render-metadata noise is the realistic target.

# Hash the derived data and figures from the fresh
# run, compare against the previously committed
# versions
git stash
shasum -a 256 analysis/data/derived_data/*.rds \
              analysis/figures/*.png > /tmp/fresh.sha256
git stash pop
shasum -a 256 analysis/data/derived_data/*.rds \
              analysis/figures/*.png > /tmp/committed.sha256
diff /tmp/fresh.sha256 /tmp/committed.sha256

Run make test and verify all tests pass. A failing test at the publication stage is a stop-the-line event.

make test
# Equivalent inside R:
# tinytest::run_test_dir('inst/tinytest')
# testthat::test_local()

Final review: all Five Pillars complete and consistent. Walk through each pillar one last time and confirm that the artefacts on disk tell a single, consistent story.

# Pillar 1: Dockerfile R version
grep '^FROM' Dockerfile

# Pillar 2: renv.lock R version
jq '.R.Version' renv.lock

# Pillar 3: .Rprofile activates renv
grep -E 'renv/activate' .Rprofile

# Pillar 4: source code is sourceable
Rscript -e 'devtools::load_all()'

# Pillar 5: raw data is read-only and documented
ls -la analysis/data/raw_data/
test -f analysis/data/README.md && \
  echo 'data README present'
test -f docs/data-dictionary.md && \
  echo 'data dictionary present'

The grep, jq, and ls outputs together must agree: same R version in the Dockerfile and renv.lock, renv activation in .Rprofile, sourceable code, and read-only documented raw data.

Daily Workflow

The checklist is not consulted line by line every day. It is consulted at three moments:

Project kick-off. Read the entire checklist before doing anything else. Skim, do not implement; the goal is to know what is coming so the early decisions (directory layout, factor coding, file naming) do not need to be unwound later.
Phase boundaries. At the end of each phase, work through the checklist items for that phase explicitly. Tick each item off in a private copy (docs/analysis-checklist-.md is a reasonable convention) so the project’s progress is auditable.
Reproducibility gate. Before circulating any externally facing report, run Phase 6 in full. This is the gate that catches the inconsistency between the author’s laptop and a fresh container.

The numbered structure is also useful in commit messages (‘Item 17: add plausibility-bound assertions for outcomes A and B’) and in TODO notes (‘Item 41: still need to save fitted models as RDS’).

Command	Action
`quarto render docs/analysis-checklist.qmd`	Render the standalone checklist
`make r`	Enter the Docker container
`make test`	Run the tinytest and testthat suites
`make check-renv`	Validate package dependencies
`make docker-build`	Rebuild the image after package changes

Things to Watch Out For

The checklist is a tool, not a contract. Items are meant to be skipped consciously, not silently. If the project has no longitudinal structure, Item 16 (visit variables) is not applicable; mark it ‘N/A’ with a one-line reason rather than removing it from the file.
Phase 1 takes longer than expected. First-time users of zzcollab routinely underestimate Item 6 (data provenance README) and Item 7 (data dictionary). Both feel like overhead and pay off only weeks later, when they are the difference between a clean Phase 2 and a month of detective work. The _Imp ambiguity in the example dataset is a concrete instance: resolving it in Item 7 takes ten minutes; discovering it in Phase 5 takes days.
The integrity tests are not unit tests. Items 11-17 assert contracts about the raw data, not behaviours of functions. They live in inst/tinytest/, not tests/testthat/, by convention. Mixing the two leads to tests that pass against frozen fixtures but fail against live data.
Item 41 (save model objects) is easy to forget. Without saved RDS objects, the report in Phase 5 re-fits models on every render, which couples the report to a particular package version and can produce different results across renders. Save once, load thereafter.
Item 52 (fresh-container rerun) catches errors that nothing else does. A pipeline that runs on the author’s laptop with a stale R session can fail in a fresh container because of a missing library() call, a hardcoded path, or a package not declared in DESCRIPTION. Run Item 52 weekly during active analysis, not just at the end.
Item 55 is not a formality. The Five Pillars review has caught real inconsistencies (mismatched R versions, undeclared dependencies, raw data committed under the wrong path) on every project that used it. Treat it as a separate review pass, with at least one fresh pair of eyes.

When the Updated Extract Arrives

The email mentioned that ‘an updated extract may follow if a few late withdrawals come in’. This is the moment the Phase 2 tests earn their keep. The procedure is four commands:

# 1. Drop the new file in and lock it down.
mv ~/Downloads/studyA_data_2026Q2.csv \
   analysis/data/raw_data/
chmod 444 analysis/data/raw_data/studyA_data_2026Q2.csv

# 2. Update the data README with the new receipt date
#    and the provider's notes on what changed.
$EDITOR analysis/data/README.md

# 3. Update load_raw_data() to point at the new file
#    (or accept a filename argument), then run the
#    integrity tests in the container.
make r

# 4. Inside R: run integrity tests, then re-source
#    the pipeline if they pass.
tinytest::run_test_file(
  'inst/tinytest/test_data_integrity.R')
# All passed -> safe to re-run the pipeline.

source('analysis/scripts/01-clean-data.R')
source('analysis/scripts/02-eda.R')
source('analysis/scripts/03-primary-analysis.R')

quarto::quarto_render('analysis/report/01-eda.qmd')
quarto::quarto_render('analysis/report/02-results.qmd')

If Items 11-17 pass silently, the contract held and the data are just smaller or larger. If any of them fails loudly, the contract changed; investigate before proceeding. A pipeline without integrity tests requires the analyst to re-derive every assumption about the new file by hand; a pipeline with the Phase 2 tests in place reduces the update to a build-and-render cycle.

A finished compendium archived alongside the rendered report, with the standalone checklist filed in docs/.

What Did We Learn?

Lessons Learnt

Conceptual Understanding:

A checklist is a tool for surfacing the steps that are easy to skip, not for enforcing a particular workflow. The same checklist will produce slightly different artefacts in two projects; that is the intended outcome.
The phase boundaries are useful precisely because they are arbitrary. Anything that lets a project pause and audit itself before moving on is worth the friction.
The numbered structure is more valuable than the prose. The annotation paragraphs are scaffolding for first-time use; experienced users only consult the numbers.
Reproducibility is a property of the workspace as a whole, not of any single artefact. The Five Pillars review at Item 55 is the only point that audits the workspace rather than its parts.

Technical Skills:

Quarto is a better authoring surface for the checklist than LaTeX. The same source renders to HTML for inline reading and PDF for printing, and the document carries a YAML header that the analyst can edit per project.
A standalone docs/analysis-checklist.qmd lives outside the analysis pipeline, so changes to the checklist do not retrigger expensive analysis re-renders.
tinytest in inst/tinytest/ is well suited to the data-integrity tests in Items 11-17. Its lightweight API is appropriate for assertions about a single loaded data frame.
Saving fitted model objects as RDS in analysis/data/derived_data/ decouples Phase 5 from the report rendering and makes report regeneration cheap.

Gotchas and Pitfalls:

Forgetting to set raw data files read-only (Item 5) produces silent corruption that is impossible to detect without a checksum.
Conflating the data dictionary (a contract) with the cleaned data documentation (a description) blurs the audit trail. Keep them separate.
Running tinytest from the host instead of from inside the container exposes the tests to whatever R version the host happens to have, defeating the purpose of the zzcollab environment.
A passing make test does not certify reproducibility; only Item 52 (fresh-container rerun) does. Several projects in the wild conflate the two.

Limitations

Six phases is one ordering, not the ordering. A simulation study or a methods paper will want to reorder Phases 3 and 4. The checklist accommodates this by numbering items, not by enforcing serial execution.
The checklist assumes a tabular outcome. Imaging pipelines, NLP corpora, and high-throughput genomics will need different Phase 2 contents.
No version-control discipline. The checklist does not prescribe a git branching model, commit message format, or PR review workflow. Those belong in a separate document.
No statistical analysis plan template. Item 40 says ‘follow the SAP’; it does not provide one. Drafting the SAP is the analyst’s responsibility.
No coverage of stakeholder-facing artefacts. The checklist ends at Item 55 (reproducible compendium). It says nothing about manuscript drafting, conference posters, or oral presentations.

Opportunities for Improvement

Several avenues would be worth exploring further:

Add a Phase 0 with project-charter items (research question, target population, success criteria) that precede the workspace creation in Item 1.
Add a ‘completion column’ to the rendered checklist with a date and analyst initials, so the artefact doubles as a sign-off log.
Extend Phase 6 with archival items (DOI assignment, Zenodo deposit, OSF preregistration) for projects that require external archival of the compendium.
Provide a tinytest harness that takes the checklist as input and asserts each item’s existence (tests/integration/test-checklist-coverage.R).
Translate the checklist into a Quarto project template that initialises an empty docs/analysis-checklist.qmd alongside the boilerplate Dockerfile, Makefile, DESCRIPTION produced by zzc analysis.

Wrapping Up

A 55-item checklist is enough structure to take a single CSV in the inbox to a publishable, reproducible compendium, and short enough to read end to end before doing anything. The six phases correspond to deliverables that map directly onto the zzcollab workspace: a buildable container, a validated dataset, an EDA report, tested functions, a primary-analysis report, and an archival compendium.

The most valuable lesson from running the checklist against the example dataset (a single CSV, no codebook, an updated extract on the way) was that Phase 1 and Phase 2 absorb most of the apparent overhead and produce most of the durable value. By the time Phase 3 begins, the analysis runs against a typed, tested, documented dataset, and the rest of the project is a series of auditable transformations on it.

In conclusion, four points merit emphasis. First, a six-phase, 55-item checklist drives a zzcollab analysis from a CSV in the inbox to a published, reproducible report. Second, phase boundaries are the natural audit points; Phase 6 is the only phase that audits the workspace as a whole. Third, the Phase 2 integrity tests pay off when the updated extract arrives. Fourth, the numbered structure is the durable feature; the prose is scaffolding for first-time users.

Reproducibility

Tested configuration:

Component	Version
Operating system	macOS 15.4
zzcollab	2.4.0
R	4.5.1
Quarto	1.5.x
renv	1.0.x
Docker Desktop	4.x
Last verified	2026-04-29

Configuration files:

docs/analysis-checklist.qmd: the standalone generic checklist, archived alongside this post for offline reference.
Dockerfile, renv.lock, .Rprofile: the Five Pillars computational environment shared with every zzcollab workspace.
Makefile: standard zzcollab targets (r, test, check-renv, docker-build).

To reproduce end-to-end:

git clone <this-post-repo>
cd zzcollab-analysis-checklist
make docker-build
quarto render index.qmd --to pdf

Appendix A: Printable Reference Card

The annotated walkthrough above is the durable teaching content. The compact table below is the operational artefact, intended to be printed as a single sheet and ticked off in pen as a project progresses. The PDF render ships a fully-gridded version with cell borders; the HTML render uses the standard browser table.

#	Item	Done	Date completed	Notes
	Phase 1: Project Setup
1	Dockerfile exists and builds
2	renv.lock lists required packages
3	.Rprofile activates renv
4	Source code directories exist
5	Raw data is read-only in raw_data/
6	analysis/data/README.md documents provenance
7	Data dictionary written
8	Analysis packages installed and snapshotted
	Phase 2: Data Ingestion and Validation
9	R/load_data.R defines load_raw_data()
10	inst/tinytest/test_data_integrity.R created
11	Test: expected number of columns
12	Test: expected column names
13	Test: ID columns no NAs, expected format
14	Test: pre-specified categorical levels match
15	Test: demographic categorical levels match
16	Test: visit variables in pre-specified set
17	Test: numeric ranges plausible
18	analysis/scripts/01-clean-data.R written
19	janitor::clean_names() applied
20	Categorical variables converted to factors
21	Missing values handled and documented
22	Cleaned data saved to derived_data/
	Phase 3: Exploratory Analysis
23	analysis/scripts/02-eda.R written
24	Summary statistics by group computed
25	Baseline balance checked
26	Outcome distributions visualised
27	Outliers and data-quality flags documented
28	analysis/report/01-eda.qmd written
29	Sample size and missingness documented
30	Table 1 included
31	Trajectory plots included
	Phase 4: Analysis Functions
32	R/summarize_outcomes.R written
33	R/model_outcomes.R written
34	R/plot_outcomes.R written
35	inst/tinytest/test_summarize.R passes
36	inst/tinytest/test_model.R passes
37	Missing-data edge cases tested
38	Single-group edge cases tested
	Phase 5: Primary Analysis
39	analysis/scripts/03-primary-analysis.R written
40	Pre-specified analysis plan implemented
41	Fitted model objects saved to derived_data/
42	analysis/report/02-results.qmd written
43	Primary outcome documented
44	Secondary outcomes documented
45	Sensitivity analyses documented
	Phase 6: Documentation and Reproducibility
46	DESCRIPTION Title and Description updated
47	DESCRIPTION Authors set
48	DESCRIPTION Imports verified
49	Project README.md created or updated
50	Reproduction instructions documented
51	Docker image rebuilt and tagged with git SHA
52	Fresh-container end-to-end run passes
53	Outputs match committed versions
54	make test passes in fresh container
55	Five Pillars consistency review complete

A two-up print on letter or A4 stationery yields a single double-sided card. Items marked ‘N/A’ for the project at hand should carry a brief reason in the Notes column rather than a tick.

Let’s Connect

Have questions, suggestions, or spot an error? Let me know.

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: Contact form

I would enjoy hearing from you if:

You spot an error or a stronger ordering for any phase in the checklist.
You have suggestions for items that should be added (or removed).
You have used a similar checklist in a different research domain and want to compare notes.
You just want to say hello and connect.

Rendered on 2026-04-30 at 08:06 PDT.
Source: ~/Dropbox/prj/qblog/posts/61-zzcollab-analysis-checklist/zzcollab-analysis-checklist/analysis/report/index.qmd

Refactoring a Personal Toolbox: Scripts versus Shell Functions

Ronald G. Thomas — Sat, 25 Apr 2026 07:00:00 GMT

A workshop pegboard, organised so that each tool sits in its own labelled outline.

A personal toolbox earns its keep when each helper sits in the place that fits its job, with no doubt about which drawer to open.

Introduction

Most quantitative researchers maintain a personal toolbox of small shell helpers. It generally takes two forms: a ~/bin directory populated with executable scripts, and a .zshrc (or .bashrc) file containing a layer of custom aliases and functions sourced into every interactive session. The contents accumulate gradually over years of incremental work, each entry typically introduced in response to a specific irritation: a one-liner written to bypass an awkward pipeline, a research-notes capture script drafted during a manuscript revision, a git workflow function codified after a co-author inadvertently committed a secret.

After several years of accretion, the toolbox remains functional but the boundary between ‘script in ~/bin’ and ‘function in .zshrc’ has eroded. Some scripts in ~/bin are aliases in disguise (a single cd followed by the launch of an editor). Some functions in .zshrc span hundreds of lines of program logic that have no need for the calling shell’s state. The toolbox remains serviceable, but it has become difficult to reason about, difficult to audit for security, and noticeably slow to load on shell startup.

We document a principled refactor of such a toolbox. The worked example is drawn from a biostatistician’s workflow (reproducible R analyses, Docker-based research compendia, frequent git commits, occasional HPC submission), but the underlying rule applies to any Unix-flavoured personal workspace.

A companion plan that operationalises the rule, with a concrete phase sequence and effort estimates, is referenced under See Also.

Motivations

To establish a single rule that decides, for any helper, whether it belongs in ~/bin or in shell config.
To remove from .zshrc the logic that has no business mutating the current shell, lowering startup time and improving auditability.
To bring versioned scripts under a linter (shellcheck) so that legacy quoting bugs surface before they bite.
To eliminate microscripts that exist purely because the author did not realise the shell already has a primitive for them (the alias).
To make the toolbox legible to future-self and to collaborators who inherit it during onboarding or Sabbatical handoff.

Objectives

State a guiding principle that resolves the function-versus-script question in every encountered case.
Provide a categorisation matrix so that an existing toolbox can be triaged in a single sitting.
Sequence the refactor into seven small phases, each independently shippable and reversible.
Document the gotchas that only surface when scripts are extracted out of a long-lived shell environment.

Errors and better approaches are welcome; see the Feedback section at the end.

A second monitor displaying terminal panes during an active refactor session.

Background: function or script?

The function-versus-script rule developed here rests on a small set of recurring terms (process, state, namespace, scope, testing, reproducibility). Each is defined briefly below before the substantive discussion begins, so that the later prose can be read against a fixed vocabulary.

Terminology

Process. An operating-system execution unit with its own memory, environment, file descriptors, and a unique process identifier (PID). Every running program is a process. A shell, an editor, a script, and a database server are each a separate process; the operating system isolates them so that one cannot directly read or write another’s memory. Communication between processes is restricted to explicit channels: arguments and environment passed at launch, file descriptors, signals, and exit codes.

State. The information a process holds in memory at a given moment. For a shell process, the state includes the current working directory, the values of environment and local variables, the defined aliases and functions, the shell options, the history list, the directory stack, and the job table. State is what distinguishes one interactive shell session from another, and it is what makes the function-versus-script question consequential at all: a function can read and write the calling shell’s state, whereas a script cannot.

Namespace. The set of names (variables, functions, aliases) visible to a particular execution context. The interactive shell maintains one namespace; a child process launched from it receives a copy of the exported portion and nothing else. Two processes can hold variables of the same name without collision because each has its own namespace. The R analogy below uses the term in the same sense: an R session has a namespace into which library() calls bind names.

Scope. The region of code in which a given name is defined and accessible. Shell functions and scripts have distinct scopes by construction: a variable set inside a script is visible only within that script’s process unless explicitly exported to children, while a variable set inside an interactive function (without local) is visible to the entire shell session. Scope is the language-level property that follows from process boundaries and namespace rules.

Testing. Exercising code in a controlled environment with known inputs and asserting that the observed outputs match expectations. In shell work, testing typically means invoking a script with prepared arguments, capturing its stdout, stderr, and exit code, and comparing them against reference values; static analysers such as shellcheck address a complementary class of issues without execution. A helper that cannot be invoked with a controlled input set, in isolation from a live interactive session, cannot be tested in this sense.

Reproducibility. The property that re-running the same code on the same inputs in the same environment yields the same outputs. Reproducibility requires both that the code be captured (version control, an executable file at a stable location) and that the execution environment be characterised (declared dependencies, pinned versions, an explicit shebang). A helper defined inline in a personal startup file fails the first requirement; a script in ~/bin under git satisfies it.

How the shell executes code

To classify a helper sensibly, it helps to begin with how the shell actually executes code.

When a user opens a terminal, the shell launches as a long-lived process. It reads a startup file (.zshrc, .bashrc, or similar), constructs its environment (exported variables, defined aliases and functions, the current working directory, the history file, the directory stack, key bindings, completion definitions), and then waits for input at a prompt. Every command the user enters is interpreted in the context of that single process and the in-memory state it carries.

Within this model, a helper can run in one of two fundamentally different ways: inside the same process as the interactive shell, or inside a new process spawned from it. The distinction is mechanical and unambiguous, and it determines what the helper is permitted to change.

Functions: code in the current process

A shell function is a named block of code, defined as name() { ... } (or function name { ... }), that executes in the current shell process. Because it runs there, it has direct access to that process’s state. It can:

Change the working directory via cd, with the change persisting for subsequent commands at the same prompt.
export, set, or unset environment variables that future commands in the session will inherit.
Define or redefine aliases, options (setopt), key bindings, and completion functions.
Manipulate the directory stack (pushd, popd), the command history, and the job table.

Functions are most commonly defined in a startup file so that they are available in every interactive session. The cost is borne by every shell invocation: each function is parsed in full before the prompt appears, regardless of whether it is ever called.

Scripts: code in a child process

A shell script is a file with execute permission that the shell runs by spawning a child process. The mechanism is the standard Unix fork and execve pair: the parent shell calls fork, which produces a near-identical copy of the shell process (the child), and the child then calls execve to replace its own program image with the interpreter named in the script’s shebang line (#!/usr/bin/env bash, #!/usr/bin/env zsh, and so on). The parent typically waits for the child to finish; the child runs the script and eventually exits.

The parent-child relationship

From the operating system’s perspective the parent and child are two distinct processes. Each has its own process identifier (PID), its own address space (a private region of memory neither process can read from the other), its own working directory, its own copy of the environment, and its own file descriptor table. At the moment of fork, the child receives a snapshot of the parent’s exported environment, the parent’s open file descriptors (stdin, stdout, stderr), the command-line arguments passed at invocation, and a copy of the parent’s working directory. That snapshot is the entirety of what crosses the boundary. After the fork, the two processes are independent: state changes the child makes are invisible to the parent, and state changes the parent makes are invisible to the child. When the child exits, the parent collects three pieces of information and nothing else: the bytes the child wrote to stdout, the bytes the child wrote to stderr, and the integer exit code returned by the child’s final instruction.

What this means for `cd` (and everything like it)

The practical consequence is that a script cannot mutate the state of the shell that invoked it. The point is precise enough to deserve a careful statement: a script can call cd, and the call succeeds. The child process moves to the new directory and any subsequent commands in the script run from there. What the script cannot do is cd the parent shell, because the parent and child each hold their own working directory, and the child’s was a copy from the start. A common source of early confusion is the user who places cd /some/project inside a script, executes it, and is puzzled to find the prompt unchanged. The script ran, the child process did cd, the child exited, and the parent shell, never having shared a working directory with the child, remained where it was.

The same argument applies to environment variables, aliases, shell options, the directory stack, and every other element of shell state: a script can change them inside its own process, but the changes are discarded the moment that process terminates. This is the same process isolation that allows a crashing program to leave the rest of the system intact.

The one explicit exception: `source`

The shell built-in source (equivalently, .) does not spawn a child process. It reads the named file and executes its commands directly in the current shell. A cd inside a sourced file therefore does change the calling shell’s working directory, and an export inside a sourced file does set a variable in the calling shell’s namespace. Sourcing is the mechanism by which .zshrc contributes its definitions to the interactive session in the first place, and it is the standard workaround when a helper genuinely needs to leave the user in a new directory but is too long to live as a function.

The cost is the loss of isolation: a sourced file shares the calling shell’s state and can read, modify, or clobber any name defined there. An error in a sourced file (an unset variable under set -u, a stray exit) terminates the interactive session rather than a child process. For these reasons, sourcing is reserved for files whose authors explicitly intend to operate on the parent’s state; ordinary helpers should be invoked, not sourced.

Why the distinction matters

These two execution models are not interchangeable. Some helpers genuinely require the function form because their entire purpose is to mutate the parent shell’s state: a wrapper that prepares a project directory and leaves the user inside it, a helper that loads a credential into the current session via export, a function that toggles a shell option for the remainder of the session. Other helpers fit equally well in either form, but the choice has measurable consequences for testability, auditability, startup performance, security review, and the reader’s mental model of where logic lives.

A toolbox that has drifted out of alignment shows two symmetric inversions: scripts that should have been aliases (because the ‘script’ is a single command line that does not even merit its own file), and functions that should have been scripts (because the ‘function’ performs no operation that requires access to the parent shell’s state). The next section defines the rule that resolves both.

Guiding principle

The rule is short:

A helper should be a shell function only when it must mutate the calling shell. Otherwise it should be a versioned, executable script in ~/bin.

The remainder of this section unpacks the rule in terms readily familiar to an audience trained in scope, reproducibility, and the boundaries between processes.

A familiar analogy: R functions and Rscript files

R users have lived with the same boundary for years without naming it explicitly. A pure helper that takes inputs, returns a value, and produces no side effects is written as an R function, often grouped with related functions in an R package. A one-shot batch job that loads a dataset, fits a model, writes an .rds file, and emits a PDF report is written as a script and executed with Rscript or rendered through Quarto. No experienced R user would propose to embed an entire batch pipeline in .Rprofile, even though .Rprofile is technically capable of running it. The pipeline does not require access to the interactive session’s namespace, and locating it there would slow every R session, obscure it from version control review, and prevent any tooling from analysing it as a self-contained unit.

The shell case is the same boundary at a different scale. A shell function corresponds to a function loaded into the user’s current R session: it can read and modify that session’s globals, and it lives or dies with the session. A shell script corresponds to Rscript invoked from the command line: it receives a fresh interpreter, communicates back through stdout and an exit code, and exits cleanly. The process boundary that makes Rscript reliable for batch work is the same boundary that makes a shell script reliable for non-interactive use. Conflating the two in shell-land carries the same costs as it would in R-land; the shell community has historically been less attentive to the conflation.

Scope, in four lines

Concretely, a helper must be a function when it does any of the following in a way that should outlast the helper’s run:

Change the working directory.
Export, set, or unset shell variables.
Modify aliases, options, key bindings, completion definitions.
Manipulate the directory stack, history, or job table.

Each item on this list shares a single property: the change must persist in the calling shell’s process, which is precisely what a child process is structurally incapable of accomplishing. Anything outside the list (running a program, reading or writing files, formatting output, fetching from a URL, calling git, calling make, calling a database, parsing JSON) communicates with the surrounding world through file descriptors and exit codes alone, which is exactly what the script form is designed for. The rule scales cleanly: a function that both runs a long pipeline and needs to cd at the end is two helpers, one calling the other.

Why the inversion is expensive

A long shell function in .zshrc carries five practical costs that map directly onto practices already familiar from renv, package versioning, and unit testing.

Startup latency. A function in .zshrc is parsed every time the shell starts. A script is parsed only when invoked. A two-hundred- line zsh function that the user calls a few times a day is re-parsed dozens of times for each one execution. The latency cost is the same shape as library() calls in .Rprofile: convenient on the surface, expensive in aggregate.

Lint and static analysis. shellcheck reads scripts; it does not introspect functions buried in dotfiles. Quoting bugs, unset variables, and incorrect test predicates that would be caught immediately in a script silently survive in a .zshrc function. The R parallel is lintr against package code versus lintr against ad-hoc chunks in .Rprofile.

Version control granularity. A script is a file. Its diffs are local, its history is local, blame is meaningful, and its commits do not interact with unrelated config changes. A function in a shared .zshrc competes with PATH exports, plugin lines, keybindings, and PROMPT changes for git history attention. Any non-trivial helper deserves its own git history, the same way a non-trivial R helper deserves its own R/foo.R.

Testability. A script can be invoked under a controlled environment with arguments, redirected I/O, and a captured exit code. The same logic embedded in a .zshrc function can only be tested by sourcing the dotfile and calling the function in a live shell, which is roughly as defensible as testing R code by copy-pasting it into the console.

Reuse across hosts and contexts. Scripts in ~/bin are picked up by anything that respects PATH: cron, launchd, Make rules, Docker containers that mount the home directory, HPC submission scripts, scheduled GitHub Actions runners that source the toolbox. Functions are visible only to interactive zsh sessions. A research-backup helper hidden in .zshrc cannot be wired to a launchd job; the same logic in ~/bin can be scheduled in seconds. This boundary maps almost exactly to the line between ‘helpers exported by a package’ and ‘helpers defined inline in a notebook’ in R work.

Why the inversion happens

A second pattern to notice: the inversions are not random. Helpers that grow up as one-liners (cd somewhere && open something) are written as scripts because the author did not yet know about aliases. Helpers that grow up gradually (a git workflow that started as a snippet, then accreted secret-scanning, then commit- message templating, then a confirmation prompt) end up as functions because they were edited each time a new shell was open and .zshrc was already in the editor. The forces driving each inversion are completely different, but the symptom (a toolbox where category does not predict location) is identical.

Recognising both forces matters because the corrective for each is different. The microscript that should have been an alias becomes an alias and disappears. The accumulated function that should have been a script becomes a script and gets the full hardening treatment: shebang, set -euo pipefail, quoted variables, a -h/--help flag, a shellcheck pass, and a place in version history.

When the rule cuts the other way

For completeness, the rule does sometimes route a current script into the function form. A helper that the author scripted but that genuinely needs to leave the user in a different directory is mis-categorised as a script. The give-away symptom: ‘why does my shell not stay in the project directory after I run this?’. That helper’s logic should move to a function, or, more cleanly, the script should be retained for non-interactive callers (cron, make) and a thin shell function should call it and apply the final cd.

What the rule is not

The rule is not ‘all logic must leave .zshrc’. The dirstack helpers, the ff fzf-then-cd shortcut, the navigation jumps to project subdirectories: all are legitimately functions, because they exist precisely to mutate the calling shell. The point is to keep the function tier narrow and obviously appropriate, so that when a future reader sees a function in .zshrc, the existence of that function is itself evidence that it had to be one.

Symptoms of an unaligned toolbox

Three signs reliably indicate that a toolbox needs a refactor along these lines.

The shell startup file has crossed a thousand lines. Most of the bulk is unlikely to be configuration. It will be logic that should have been extracted long ago.

Microscripts in ~/bin outnumber non-trivial scripts. A directory with thirty entries, half of which are under five lines, is mostly aliases waiting to be promoted.

At least one shell function imports a non-trivial external program. A .zshrc function that calls gitleaks, aws, pandoc, or jq is almost certainly mis-located. External programs imply a real workflow, and real workflows belong in versioned files.

A categorisation matrix

The matrix below sorts every helper a typical toolbox will contain into one of five fates. Triaging an existing toolbox is a single-sitting exercise: read each helper, ask the four questions in the principle, and place it.

Fate	Trigger
Keep as function	Helper must `cd`, `export`, modify aliases or history
Move from function to script	Function does no shell-state work; it just runs a program
Convert script to alias	Script is a single command line with no real argument parsing
Keep as script (harden)	Script is correctly placed but lacks shebang, quoting, lint
Decommission	Helper duplicates another, is referenced nowhere, or is dead

The first row is the smallest in any toolbox the author has inspected, typically four to twenty helpers. The remaining rows absorb everything else.

Editor showing a shell function being lifted into its own file under ‘~/bin’.

A seven-phase refactor

Each phase below is independently shippable and reversible. The ordering is by leverage: the early phases produce most of the visible improvement.

Phase 1: extract the largest function

In every toolbox the author has examined, a single function dominates by line count. Often it is a custom git commit workflow or a research-notes capture pipeline that grew over the years. That function alone typically contains 30 to 70 percent of the non-config bulk in .zshrc.

The extraction is mechanical:

Create a new file in ~/bin named after the helper. Mark it executable.
Add a shebang that matches the syntax in use. Functions written with zsh-only constructs (${(f)...}, ${array:#}, setopt semantics) require #!/usr/bin/env zsh; do not assume bash will parse them.
Apply standard hardening: set -u always, set -e if the function does not deliberately tolerate failure, set -o pipefail if any command piping is involved.
Decide on helpers. Sub-functions called only by the extracted helper can move with it (inline, or in a lib/ directory the helper sources). External helpers used by other functions stay in shell config.
Delete the original function from the shell startup file.
Verify by exercising the helper end-to-end on a scratch workspace, including any failure paths.

This phase alone is usually responsible for a measurable shell startup speedup and for moving the largest single block of code under linting.

Phase 2: extract the small functions

After the large extraction, several smaller functions remain that also fail the principle: a Mathematica wrapper, a fuzzy file finder that calls vim, a make invocation that runs from any subdirectory. Each is a one-file extraction at most. Time per helper is small; the cumulative effect is roughly another fifty to eighty lines out of .zshrc.

Phase 3: convert microscripts to aliases

Inspect every script under fifteen lines in ~/bin. Any whose body reduces to a single command pipeline (with no flag parsing, no branching, no output processing) is more honestly expressed as an alias. Examples that look exactly like aliases when written inline:

nohup /Applications/Ghostty.app/Contents/MacOS/ghostty \
  >/dev/null 2>&1 &

Promote each such script to an alias in shell config and delete the file. The benefit is twofold: the helper now appears in alias output (so it is discoverable by listing aliases), and the toolbox shrinks by one file per promotion.

Phase 4: standardise the rest

The remaining scripts in ~/bin are correctly scripts. They typically need light hardening:

Add a shebang, #!/usr/bin/env bash for bash-portable scripts.
Add set -euo pipefail unless the script explicitly chooses a more permissive policy and documents why.
Replace backticks with $(...).
Quote variable expansions: "$1", "$PWD", and so on.
Resolve shellcheck warnings, or annotate with a disable comment that names a reason.
Add a -h/--help flag for any script over thirty lines.
Decide an extension policy and apply it consistently; standard Unix practice is to drop .sh from executables and reserve the extension for files that are explicitly sourced. Confirm that any scheduler entries (launchd, systemd, cron) referring to the script by name are updated together.

This is the long-tail phase. It can be done one helper at a time, amortised over normal work.

Phase 5 (optional): autoload remaining functions

For shell startup files that still feel heavy after phases 1 to 4, move the remaining functions out of the startup file into a function directory and autoload them:

fpath=(~/.zsh/functions $fpath)
autoload -Uz d ff za zw zy zf zt zs zp zr z0 zm ze zo zc zg

Each function lives in its own file under the function path. They load on first call, not on shell start. Skip this phase if the remaining functions are few or if having their bodies visible in the startup file is more valuable than the small startup cost.

Phase 6: install a guardrail

Add a shellcheck pre-commit hook scoped to ~/bin/*, excluding binaries, R scripts, Python scripts, and any archive/ directory. The point is not to catch a flood of issues at install time (phase 4 already handled those) but to prevent regressions as the toolbox continues to evolve. A single Make target, make lint, that runs shellcheck over every executable is sufficient.

Phase 7: decommission

Triage each remaining helper for redundancy. Common findings: two clipboard-to-notes helpers that do roughly the same thing under different names, a PDF viewer launcher that exists in two versions because the author once wanted to try a different viewer, an installer left behind from a tool that was removed years ago. Pick one of each, retire the rest. Git history preserves the removed code; the working tree should not.

A clean desk after the refactor; the same tools are present, but each sits where it belongs.

Things to watch out for

These pitfalls are easy to underestimate before starting and easy to recognise once seen.

Zsh-only constructs. Many shell functions accumulate zsh conveniences that bash will not parse: ${(f)var} for newline splitting, (s::) for string-splitting flags, parameter modifiers like ${var:#}. When extracting such a function to a script, the shebang must match the syntax. A bash shebang on a zsh function will fail in unhelpful ways.
Schedulers refer to file names. If ~/bin/foo.sh is referenced by a launchd plist, a systemd unit, a cron entry, or a Makefile, renaming to ~/bin/foo will silently break the scheduler. Search for every occurrence before renaming, and land both changes in the same commit.
PATH ordering. A script extracted from a function inherits PATH from the calling shell, which is usually fine. The exception is when the function previously ran inside a block that prepended a directory to PATH. The new script needs to do its own PATH adjustment or use absolute paths.
Working directory at invocation. A function inherits the caller’s $PWD. A script does too, but if the script was previously a function that called itself recursively or that cd’d for its own purposes, that behaviour is now subprocess-local and may surprise the parent.
History and dirstack contamination. A function that previously ran pushd/popd to navigate around silently lost that capability when extracted to a script. If the helper genuinely needed the navigation, the helper was mis-classified and the rule routes it back to the function tier.
shellcheck false positives. External programs called with computed arguments ("${args[@]}") sometimes trigger SC2068, SC2086, or SC2154. Disable per line with a comment that names the reason; do not blanket-disable the warning across the project.
Secrets in extracted code. A function previously sourced from .zshrc may have benefited from environment variables set elsewhere in the same file. The extracted script does not have that benefit. Either source the relevant env file explicitly inside the script, or read secrets via a tool such as pass or a credential helper.

Daily workflow after the refactor

The refactor pays back every day in three ways.

Discoverability. A new helper added to ~/bin shows up under tab completion against $PATH. A new function added to the shell config shows up only in the current session and only after a reload.

Auditability. Every helper in ~/bin is one git log away from full history; one shellcheck away from lint; one cat away from a complete reading. A helper buried in a thousand-line startup file requires the reader to first locate it.

Composability. Scripts in ~/bin can be called from anything that respects PATH: Make rules, scheduled jobs, Docker containers mounted with the home directory, HPC submission scripts, peer collaborators who cloned the dotfiles repository. Functions can only be called from interactive shells that have sourced them. The composability gain is the most important long-term return on the refactor.

What did we learn?

Lessons learnt

Conceptual:

The function-versus-script question reduces to a single test: does the helper need to mutate the calling shell? Almost every other consideration follows from that one decision.
The forces that produce inversions in each direction are different. Microscripts grow up because the author did not know about aliases; oversized functions grow up because the startup file is the file that happens to be open.
A mature toolbox is mostly scripts, a small set of legitimate functions, and a layer of aliases. Any other distribution is evidence of accumulated misalignment.

Technical:

A long zsh function may not be portable to bash, even when the prose-level intent looks generic. Confirm the shebang against the syntactic features actually in use.
shellcheck is the highest-leverage tool to apply at the end of an extraction. It catches more than half of the latent quoting and unset-variable bugs in legacy scripts.
Naming and extension conventions only need to be applied consistently within a toolbox. The choice of .sh versus no extension matters less than the absence of a third option appearing for no reason.

Gotchas:

Renaming a script that a scheduler refers to silently breaks the schedule. Always grep for the file name across launchd/, systemd/, Makefile, and any related repositories before renaming.
Functions that depend on shell-specific syntax cannot be blindly retitled with a bash shebang. Prefer the zsh shebang unless the syntax has been explicitly portabilised.

Limitations

The refactor described here addresses helpers in shell-land specifically. It does not:

Address Python, R, or other-language helpers whose execution model differs from shell scripts.
Replace a real configuration management system. Tools such as Ansible, Nix, or chezmoi solve a larger problem and may be appropriate when the personal toolbox grows past a few dozen files or needs to be deployed to multiple machines.
Provide automated migration. Each helper still needs a human to read it, classify it, and verify the extraction.
Compose with shell frameworks that load logic from many files (Oh My Zsh, prezto). Those frameworks have their own conventions; the rule still applies, but the implementation details change.

Opportunities for improvement

Encode the rule as a small linter that scans .zshrc for functions and flags any whose body never references shell state. This would surface candidates for extraction over time rather than at one big-bang refactor.
Add a CI workflow that runs shellcheck over ~/bin on every push to the dotfiles repository.
Generate ~/bin/nav (a navigation cheat sheet) from the autoloaded function metadata, so adding a new shortcut automatically updates the help output.
Build a thin packaging layer around shared helpers so they can be installed on collaborator machines without copying files by hand. A simple Make target plus a manifest is usually sufficient.
Track helper invocation frequency via the shell’s precmd hook for one week per year, and use the result to prune helpers nobody actually calls.

Wrapping up

A personal toolbox is software, even when it is small, even when it is private, even when it grew organically. Treating it as software (with a rule for how it is structured, lint applied to the parts that should be linted, version control applied to the parts that change, and a maintenance pass when the bulk crosses some threshold) pays back every day for the next decade.

The rule that drives the refactor is short enough to remember: function only when shell state must change, script otherwise. The work to apply it is finite, ordered, and reversible. A single afternoon, divided across the seven phases above, is enough to move a toolbox from accumulated drift back into alignment.

In conclusion, four points merit emphasis. First, the rule reduces every helper-placement decision to a single question about shell state. Second, the largest single win is extracting the dominant function out of the shell startup file and into ~/bin. Third, microscripts that read like one-liners belong as aliases, not as files. Fourth, a shellcheck guardrail prevents the same drift from re-accumulating.

Reproducibility

Tested configuration:

Component	Version
Operating system	macOS 15.4
Shell	zsh 5.9
shellcheck	0.10.0
Last verified	2026-04-25

Configuration files:

analysis/configs/refactor_plan.md: the seven-phase plan in the format used to drive the worked refactor.

Feedback

Corrections, suggestions, and questions are welcome. Please open an issue or pull request on the GitHub repository or send an email to user@example.com.

Building a statistical computing textbook in the Age of AI

Ronald G. Thomas — Fri, 24 Apr 2026 07:00:00 GMT

A shelf of two books rendered as a Quarto site, symbolising the dual-volume structure that emerged during this drafting session.

A textbook’s subtitle is a commitment. ‘In the Age of AI’ commits the author to explaining where the human-LLM division of labour actually falls for each topic.

Introduction

I did not really appreciate how much structural decision-making goes into a textbook until I tried to draft two of them at once. A methods volume (‘Statistical Computing in the Age of AI’) and a workflow companion (‘Biostatistics Practicum’) were both overdue, and I wanted them to share enough scaffolding that a student could move between them without reorienting.

The turning point came when I stopped treating ‘in the Age of AI’ as a subtitle and started treating it as a structural obligation. Three short prose sections of ‘Prompts to try’ at the end of each chapter was decoration. What the subtitle required was an explicit front-loaded treatment of what the human statistician contributes, paired with an end-of-chapter verification workflow that exercises those contributions.

We walk through the decisions that produced the two books’ current state: chapter template, visual design, bibliography growth, and the specific AI-collaboration pattern. The post is a retrospective on one working session, not a finished tutorial. Readers drafting a domain-specific textbook can probably save themselves some of the back-tracking documented here.

More formally, we document here the convergence of the Knowledge Management and LLM-Augmented Editing concerns of the Workflow Construct described in post 52. Authoring a textbook is the largest-scale knowledge-management task most academics will undertake; doing so ‘in the age of AI’ surfaces specifically how the LLM-augmented editing layer interacts with the file-system, document, and bibliographic layers below it. The retrospective documented here is one working session worth of evidence on how those layers compose under sustained use.

Motivations

Existing statistical computing textbooks are excellent but largely predate widespread LLM use. A curriculum written as if 2019 were still the current year trains students for a workplace that no longer exists.
Graduate biostatistics students arrive with LLMs already in hand. Pretending otherwise produces a textbook that does not match their reading context.
I wanted a dual-volume approach: a methods book (numerical algorithms, models, bootstrap, Bayesian) and a practicum (reproducibility, Git, Docker, SAS, CDISC). The two should share scaffolding but remain independently readable.
A peer-program survey (22 US biostat MS programmes) had surfaced specific content gaps (missing data, SAS, survival, Bayesian computation) that I wanted to close without restructuring the books.
I wanted to document the decision-making as it happened so that the next instructor building out a similar text has a worked example to follow.

Objectives

Scaffold two Quarto books (a methods volume and a practicum companion) with compatible chapter structure, shared citation conventions, and visually distinct rendering.
Establish a ‘statistician’s contribution’ / ‘collaborating with an LLM’ pattern that earns the ‘in the Age of AI’ subtitle.
Ground chapter outlines in existing lecture materials, a comprehensive peer-MS-programme survey, and Jenny Bryan’s STAT 545 philosophical framing.
Apply front-matter and code-block improvements (code annotations, first-class callouts, typographic differentiation) as a named ‘polish’ pass.

The process is documented as it unfolded. Corrections and better approaches are welcome; see the contact section at the end.

Workspace ambiance image: a desk with two browser tabs open side-by-side, each rendering a different Quarto book under the same file-system hierarchy.

What is a Quarto book?

A Quarto book is a directory of .qmd files (Markdown with executable R code chunks) bound together by a _quarto.yml configuration, rendered to a multi-format output (HTML, PDF, EPUB, Word) by the quarto command-line tool. Think of it as a technical-writing Makefile: chapters, parts, cross-references, and output formats are declared in YAML, and quarto render produces a website plus a printable PDF from the same source.

The specific convention I follow is the Posit book-family pattern (R for Data Science, Advanced R, R Packages). Each content chapter carries a three-question diagnostic quiz at the top, collapsible ‘check your understanding’ callouts distributed through the content, exercises and a ‘further reading’ block at the foot, and the quiz answers at the very end.

Prerequisites

This process assumed the following:

Operating system: macOS 15.4, though the setup is portable to Linux and Windows.
R 4.4+, RStudio optional.
Quarto 1.5+ (the book format changed non-trivially between 1.4 and 1.5; earlier versions will misbehave).
Git and a GitHub account.
An LLM used as a drafting and verification assistant. The session used Claude Code via terminal; ChatGPT or Gemini would work similarly.
Existing lecture materials (speaker notes .md, slides .qmd, R code .R) for the methods volume; these become the source for chapter content.
Time required: roughly 10 hours for the initial scaffold and framework decisions, excluding content porting.

Installation

No new installation was required beyond the existing Quarto and R setup. The zzcollab framework (an opinionated research-compendium scaffold) created the working directory structure automatically. The only unusual tool was a Python helper script for batch edits across 45 chapter files, written ad-hoc during the session.

quarto --version
# 1.9.37
R --version
# R version 4.5.3 (2026-02-28)

Configuration: the chapter template

The single most consequential decision was the chapter template. After several iterations, the structure below became the pattern for every content chapter in both books.

# Chapter structure (applied to every content chapter)
sections:
  - Prerequisites             # Advanced-R-style 3-question quiz
  - Learning objectives
  - Orientation
  - The statistician's contribution   # Age-of-AI pillar 1
  - Content sections                  # with Check-your-understanding callouts
  - Collaborating with an LLM on X    # Age-of-AI pillar 2
  - Exercises
  - Further reading
  - Practice test                     # only where a course test bank exists
  - Prerequisites answers             # placed at the very end

The two sections that earn the subtitle are ‘The statistician’s contribution’ (front-loaded consciousness raising: the two to five decisions a statistician cannot delegate to an LLM) and ‘Collaborating with an LLM on X’ (end-of-chapter verification prompts, each with a Prompt / What to watch for / Verification triple).

Example of the statistician’s-contribution section for the bootstrap chapter:

## The statistician's contribution {#sec-bootstrap-human}

Before handing a dataset to a large language model with the
instruction to 'bootstrap the standard error', four decisions
require the statistician's judgment:

1. Is the bootstrap appropriate for this statistic at all?
2. What is the dependence structure of the data?
3. Which confidence-interval method matches the statistic's
   behaviour?
4. Is the bootstrap distribution itself plausible?

Each decision gets a paragraph of explanation, with the failure modes named explicitly (extrema, time-series data, skewed bootstrap distributions) and the chapter that follows providing the vocabulary to make each decision.

Terminal showing the book’s sidebar in the browser, with the two books’ shared chapter structure visible as identical tree layouts under different part-labels.

The four structural decisions

The session’s work sorted into four structural decisions, each with downstream consequences across 45 chapters.

Decision 1: book boundary and overlap

The methods volume covers statistical-theory content an MS programme teaches in a typical one-quarter course: programming in R, numerical linear algebra, optimisation, simulation, bootstrap, linear models, GLM, mixed models, survival, Bayesian. The practicum covers the workflow that surrounds the methods: reproducibility infrastructure, Git, Docker, rrtools, renv, Quarto, tidyverse wrangling, CDISC data standards, SAS, and two full case studies.

The boundary is enforced by discipline in the table of contents. Where a chapter in one volume touches a topic covered in the other, the prose uses a textual pointer (‘see the Missing Data chapter of the companion Biostatistics Practicum volume’) rather than a Quarto cross-reference, because @sec-* anchors do not resolve across distinct books.

Decision 2: Age-of-AI framing

The original chapter template ended with a single ‘With AI assistance’ callout containing three prompts. Reviewing the material as a whole showed that three decorative prompts do not earn the ‘in the Age of AI’ subtitle; they behave as a sidebar rather than as part of the intellectual framework.

The replacement is two structured sections per chapter: a front-loaded ‘The statistician’s contribution’ (what the LLM cannot do on the reader’s behalf) and an end-of-chapter ‘Collaborating with an LLM on X’ (what to verify when the LLM produces output). The preface and introduction were rewritten to name the four-category framework (reliable, unreliable, cannot-due-to-missing-context, cannot-due-to-accountability) that organises the book’s position.

Decision 3: sourcing chapter content

Each chapter’s source material is distributed across three file types: a speaker_notes_expanded.md that carries the prose narrative, a lecture*_slides_*.qmd that carries both inline code blocks and ‘dynamic’ .fragment .question / .answer pairs, and a lecture*_R_code.R with supplementary code examples. Early chapter ports missed the inline slide code and the fragment pairs entirely because the porting workflow only read the speaker notes and the R file.

The revised workflow reads all three sources, integrates inline code from the slides rather than from the stand-alone R file (the two sometimes disagree), and converts each slide’s dynamic Q&A fragment into a collapsible ‘Check your understanding’ callout placed after the relevant content section.

Decision 4: visual differentiation

Opening the two books in adjacent browser tabs, they were indistinguishable at a glance. Both used the same primary colour, the same body face, the same monospace face, the same SCSS rules. The fix was a deliberate typographic split: the methods volume renders body text in Source Serif 4 (fallbacks Charter, Georgia), the practicum renders body text in Source Sans 3 (fallbacks Inter, Helvetica Neue). Monospace stays JetBrains Mono in both. The SCSS change was ten lines per book.

Complementary configuration changes in _quarto.yml:

format:
  html:
    code-annotations: hover
    code-copy: hover
    code-overflow: wrap
    code-tools:
      source: repo
      toggle: false
    code-block-bg: true
    code-block-border-left: '#1f4e79'
    highlight-style: arrow

Each of these settings is a named Quarto feature. code-annotations: hover enables numbered-disc annotations inline in code. code-tools puts a ‘view source’ chevron on each page that links to the source .qmd in the book’s GitHub repository. code-block-border-left replaces a hand-rolled SCSS rule with Quarto-native styling synchronised with dark-mode theming. highlight-style: arrow is the accessibility-tuned palette and renders legibly in both HTML and PDF.

Verification

After every structural change, both books were re-rendered and the warnings were checked:

cd path/to/methods/textbook
quarto render --to html 2>&1 | grep -iE 'error|warn'

cd path/to/practicum
quarto render --to html 2>&1 | grep -iE 'error|warn'

Clean output (empty grep result) is the signal to continue. Citeproc warnings (‘citation X not found’) always flagged a bibliography gap and were fixed immediately; bib entries were added incrementally as content ported in.

Daily Workflow

The tools that came up most often:

Command	Action
`quarto render --to html`	Full book render (HTML only, fast)
`quarto render path/to/chapter.qmd`	Single chapter render for rapid iteration
`quarto preview`	Live-reload preview in browser
`quarto check knitr`	Confirm the R engine is configured
`grep -cE '^@' references.bib`	Count bibliography entries (growth log)
`ls [0-9][0-9]-*.qmd` + `wc -l`	Confirm expected chapter count

The bulk batch-edit pattern (rename a heading across 45 chapters, insert a section block after a given anchor) was handled by short ad-hoc Python scripts using pathlib and a content-map dictionary. Shell sed worked for single-line substitutions but became unmanageable for multi-line insertions.

A workflow scene showing a simple diagram of the two books’ structure: seven parts in the methods volume, seven parts in the practicum, each containing the same chapter template.

Things to Watch Out For

Quarto’s auto-numbering silently breaks if the introduction chapter is not unnumbered. 00-intro.qmd without .unnumbered gets counted as Chapter 1, shifting every subsequent chapter’s rendered number up by one. Filename prefix says 09-bootstrap.qmd, browser says ‘Chapter 10’. Symptom: every cross-reference in the text reads one number off. Fix: # Introduction {#sec-intro .unnumbered}.
Subsection numeric prefixes collide with Quarto’s auto-numbering. Writing ### 1. Correlation and the Fisher-z transformation renders as ‘9.12.1 1. Correlation…’. Fix: drop the prefix, let Quarto number.
Missing bibliography entries warn but do not fail. [WARNING] Citeproc: citation mcelreath2020rethinking not found is easy to miss in a full-render log. Pipe the render through grep -iE 'error|warn' every time.
Cross-book references do not resolve. @sec-cdisc in the methods volume cannot find the anchor if that anchor lives in the practicum volume. Fix: use prose pointers (‘see the CDISC chapter of the companion volume’).
The Quiz answers heading label mismatches the Prerequisites quiz label. Advanced R uses ‘Quiz’ / ‘Quiz answers’; my chapters used ‘Prerequisites’ / ‘Quiz answers’, which readers found confusing. Fix: rename the bottom section ‘Prerequisites answers’ and move it to the very end of the chapter (after the Practice test, where one exists).
Speaker notes, slide deck, and R code file each carry different content for the same lecture. Slide .qmd files are the primary source for inline code; the stand-alone .R file often omits blocks that only appear inline in the deck. Read all three.
Dynamic .fragment .question / .answer blocks in slides are easy to miss. These are the interactive ‘check your understanding’ moments in a live lecture. In book form they become collapsible callouts. Grep for fragment .question across the lecture directories to find them.

Uninstall / Rollback

Nothing installed that needed rollback. All work is in Git; git reset --hard undoes any session.

What Did We Learn?

Lessons Learnt

Conceptual Understanding:

A subtitle is a structural commitment, not a tagline. ‘In the Age of AI’ obliged the book to explain where the human-LLM division of labour falls for each topic. Three prose prompts at the end of a chapter did not discharge the commitment.
Two books are easier to draft than one, because the scope limits become explicit. Each volume has a boundary, and content that straddles the boundary gets a textual pointer rather than a cross-book link.
Peer programs converge on a fairly consistent MS curriculum. A 22-programme survey found 17/22 require SAS, 14/22 require missing data, 13/22 require Bayesian computation. Content gaps close faster when the reference distribution is visible.
The Posit book family (r4ds, r-pkgs, Advanced R, ggplot2) is an excellent structural template, but its conventions do not by themselves earn an ‘Age of AI’ subtitle. That work is additive.

Technical Skills:

Quarto’s code-annotation feature is under-used. It is the single most distinctive affordance for a computational textbook and was off by default in both books.
Visual differentiation costs almost nothing: a ten-line SCSS change gives each of two sibling books a distinct body face.
Collapsible callouts (.callout-tip collapse='true') are the ideal home for slide-style ‘check your understanding’ fragments in a book context. The reader sees the question, thinks, then clicks to reveal the answer.
Bibliography growth should track content porting. Retrofitting a references.bib at the end is painful; adding entries as chapters fill in is trivial.

Gotchas and Pitfalls:

The .unnumbered class on introduction chapters is load-bearing. Without it, every subsequent chapter renders with a shifted number.
Subsection numeric prefixes duplicate Quarto’s auto-numbering (‘9.12.1 1.’). Let Quarto do the numbering.
Tidyverse-style @sec-* cross-references are invisible in a rendered TOC. They resolve correctly but only in the body text.
Cross-book @sec-* anchors do not resolve. Use prose pointers.

Limitations

Single author, not peer-reviewed. Content decisions were informed by an LLM assistant but not by a disciplinary editorial board.
Most chapters are still scaffolds. The skeleton (Prerequisites, Learning Objectives, Exercises, Further Reading, Quiz answers) is complete, but the interior content sections are TODO placeholders in all but a handful of chapters.
The bootstrap chapter was ported end-to-end as a reference implementation. The remaining 19 methods chapters and most practicum chapters await a similar pass.
Practice tests are only present in chapters where the course’s test bank contains matching material. New chapters (Bayesian, survival) have no practice test by design.
No automated progress tracking. A simple markdown table mapping chapter to ‘scaffold / ported / polished’ status would make collaboration easier.

Opportunities for Improvement

Port the remaining 19 content chapters’ speaker notes, slide code, and dynamic Q&A fragments into chapter bodies following the bootstrap chapter’s workflow.
Implement the four named callouts (Note / Verification / Pitfall / LLM prompt) consistently, migrating existing prose callouts as each chapter is touched.
Migrate the ‘Check your understanding’ Q&A fragments from untitled callout-tip collapse='true' to the named callout-tip title='Check' form so they pattern-match across chapters.
Add a colophon page to each book documenting the build environment, theme, and font choices.
Source hero and portrait images (Efron, Bates, Bryan, Wickham, Marwick) and drop them at images/cover.png and images/portraits/*.jpg.
Push the two book repositories to GitHub (rgt47/scai, rgt47/practicum) so the code-tools ‘view source’ chevrons resolve.
Configure Netlify deploys at rgtlab.org/scai and rgtlab.org/practicum via a path-based proxy from the main site.

Wrapping Up

A textbook’s subtitle is a commitment. ‘Statistical Computing in the Age of AI’ obliged the book to explain where the human-LLM division of labour falls for each topic, and the explanation had to be structural rather than decorative. The current template delivers that through two sections per chapter (statistician’s contribution plus collaborating with an LLM) backed by a four-category framework in the introduction.

The secondary lesson was that a dual-volume approach clarifies scope. One book tries to be everything to everyone; two books can afford to be boundary-conscious, each pointing to the other where their topics connect. The methods volume covers what modelling looks like; the practicum covers everything that surrounds modelling. A student who reads both comes out with a complete picture and has an easier time than the single-volume reader because each book’s scope is visibly narrower.

In conclusion, four points merit emphasis. First, a subtitle is a structural obligation: it must be earned with sections, not sidebars. Second, two books with shared scaffolding and distinct visual identity are easier to maintain than one oversized volume. Third, Quarto’s code-annotation, code-tools, and first-class callout features should be configured on day one, not bolted on during a polish pass. Fourth, when porting lecture content, the slides, the speaker notes, and the R scripts should be read separately, as they do not fully overlap.

Reproducibility

Tested configuration:

Component	Version
Operating system	macOS 15.4
Quarto	1.9.37
R	4.5.3
knitr	1.51
rmarkdown	2.31
Shell	zsh 5.9
Last verified	2026-04-24

Book repositories:

Methods volume: ~/prj/tch/methods-textbook/textbook/
Practicum volume: ~/prj/tch/biostatistics-practicum/

To reproduce the two-book scaffold:

# Methods volume
mkdir path/to/methods-book && cd path/to/methods-book
quarto create project book
# Edit _quarto.yml with the chapter-template sections.

# Practicum volume (same)
mkdir path/to/practicum-book && cd path/to/practicum-book
quarto create project book
# Edit _quarto.yml with the chapter-template sections.

The chapter-template structure (Prerequisites, Learning Objectives, Orientation, The statistician’s contribution, content, Check-your-understanding callouts, Collaborating with an LLM, Exercises, Further reading, Practice test, Prerequisites answers) is a narrative convention, not a Quarto feature; applying it is manual.

Let’s Connect

Have questions, suggestions, or spot an error? Let me know.

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: Contact form

Feedback is welcome in the following cases:

Drafting a similar domain-specific textbook and wishing to compare structural decisions.
A better approach to the human-LLM division of labour framework.
Teaching a biostatistics MS programme with different curriculum gaps than the ones this process surfaced.
A general connection is of interest.

A pocket terminal for your Linux laptop with ttyd and Tailscale

Ronald G. Thomas — Wed, 15 Apr 2026 07:00:00 GMT

A phone displaying a terminal window connected to a laptop over a private network, illustrating remote shell access from a mobile device.

A browser-based interface to a persistent shell, reachable over an authenticated private network, provides a workable mobile surface for observing and intervening in long-running tasks.

Introduction

The ergonomics of operating an SSH client on a mobile device become apparent only when the task is non-trivial. Identity-file selection, modifier-key entry on a soft keyboard, reliable reconnection after the device sleeps, and the rendering of a monospace font at phone or tablet scale each impose non-negligible overhead. In practice these frictions accumulate to the point where many users defer routine oversight work until they regain physical access to the host machine. The resulting delay is not merely inconvenient; it creates an asymmetry between the pace at which background work progresses and the pace at which an operator can respond to it.

Two components, used together, reduce this overhead substantially. The first, ttyd, is a small C daemon that serves a terminal emulator over HTTP using the xterm.js JavaScript library; any standards-compliant browser becomes a client without additional software. The second, Tailscale, builds a peer-to-peer WireGuard mesh between enrolled devices and provides an authenticated overlay network, so that the ttyd listener is addressable only by devices already admitted to the private tailnet rather than by arbitrary hosts on the public internet. The combination yields a single URL that is reachable from an enrolled phone, is invisible to every other network the laptop is attached to, and resumes a persistent tmux session on each visit.

We document the configuration end to end. The coverage includes installation of ttyd, Tailscale, and tmux on an Ubuntu 24.04 laptop; binding of the ttyd listener to the Tailscale network interface; wrapping of the service in a per-user systemd unit that survives logout and reboot; client-side configuration on iOS; and optional hardening with a TLS certificate issued by tailscale cert. Appendices cover a sample work session and the full teardown procedure. Corrections and alternative approaches are welcome.

More formally, we document here an entry point to the Remote Access concern of the Workflow Construct described in post 52. Tailscale plus ttyd is one specific solution to the cross-machine, cross-network reachability problem; the broader Remote Access family also includes mosh for high-latency-link resilience and standard SSH for the unauthenticated-network case. The present post is the browser-shell-on-mobile leaf of that family.

Motivations

The configuration described here was motivated by the following requirements, each tied to a concrete limitation of the alternatives:

Monitoring and occasional intervention in long-running interactive sessions, including agentic coding workflows that may run for hours, without requiring physical access to the host machine.
Avoidance of identity-file management and modifier-key entry on a small touchscreen, which are the two principal ergonomic costs of native SSH clients on iOS.
Elimination of any public-internet exposure on the laptop. Opening inbound port 22 through a home router, combined with dynamic DNS, would grant global addressability to a device that regularly attaches to untrusted networks; this is an unacceptable risk profile for a personal workstation.
Network-location independence. A single stable URL should be usable from the home network, the institutional network, and a cellular carrier’s network without client-side reconfiguration.
Generalisation to future deployments. The component choices (a WireGuard-based overlay, an HTTP-native terminal, a per-user systemd unit) transfer cleanly to a cloud-hosted VPS workstation, which was identified as a plausible next step.

Objectives

The scope of this post is the following set of verifiable deliverables:

Install ttyd, Tailscale, and tmux on a Linux laptop and confirm each by version query.
Run ttyd bound exclusively to the Tailscale network interface, protected by HTTP basic authentication, and attached to a persistent tmux session.
Wrap the service in a per-user systemd unit that remains active across logout and reboot.
Connect to the service from an iPhone browser over the tailnet, verify that interactive input and output behave correctly, and document a complete teardown procedure.

The configuration is presented as a reproducible reference rather than a tutorial. Each command and configuration directive is annotated with the rationale for its inclusion.

Workspace ambiance: a Linux laptop on a desk with a phone propped beside it, mirroring the same tmux session.

What is ttyd?

ttyd is a small C daemon, approximately six thousand lines of source, that serves a terminal emulator over HTTP. On the server side it allocates a pseudo-terminal (pty), spawns a process attached to it (commonly a shell or a multiplexer such as tmux), and exposes the pty’s input and output through a WebSocket on an HTTP listener. On the client side the browser loads the xterm.js library, opens the WebSocket, and renders a full VT100-family terminal in a element; keystrokes are forwarded as they are generated, and output bytes are written back to the terminal buffer.

In functional terms the role of ttyd is analogous to that of an SSH daemon, but the transport is HTTP(S) and the client is a browser rather than a native terminal emulator. This has three practical consequences. First, no client-side software installation is required; any device with a recent browser is a viable client. Second, the authentication and transport-security story is different: ttyd itself provides only HTTP basic authentication, and is not intended to be exposed to untrusted networks without an enclosing security layer (a reverse proxy with client-certificate authentication, or in this configuration, a Tailscale overlay). Third, session persistence across client disconnections is delegated: closing the browser tab tears down the WebSocket, but the pty and its attached process continue to run as long as ttyd is running, which is why this configuration always attaches ttyd to a tmux session rather than to a bare shell.

As a minimal example, ttyd -W -p 7681 zsh starts a writable shell on port 7681; any host that can establish a TCP connection to that endpoint obtains an interactive terminal. The Tailscale layer constrains the meaning of ‘any host that can reach that endpoint’ to devices that have been authenticated into the tailnet, and routes the traffic over an encrypted WireGuard tunnel rather than over a TCP connection that transits intermediate networks in clear text.

Architecture Overview

The runtime system is best understood as three layered components, each with a distinct responsibility:

The terminal process. tmux runs as a long-lived user process on the laptop, maintaining one or more pty pairs and the associated shell state. Its lifecycle is independent of any network connection and outlasts both the ttyd daemon and the browser.
The HTTP gateway. ttyd runs as a systemd user service, attaches to the tmux session specified on its command line, and serves the terminal contents over HTTP. It listens only on the Tailscale virtual interface (tailscale0), so its socket is unreachable from the physical wired, wireless, or cellular interfaces of the host.
The transport overlay. Tailscale runs as a system-level daemon (tailscaled) and maintains the tailscale0 WireGuard interface. All packets arriving on that interface have already been authenticated against the tailnet’s public-key infrastructure and decrypted from their WireGuard envelopes. On the client (the phone), the Tailscale iOS app maintains the corresponding peer relationship and presents the tailnet as a routeable network.

The request path for a single keystroke therefore traverses: browser on phone, iOS Tailscale VPN, WireGuard tunnel (direct peer-to-peer where possible, or via a DERP relay when both endpoints are behind restrictive NATs), tailscale0 on the laptop, ttyd listener on port 7681, the pty master owned by ttyd, and finally the tmux process that is the pty slave. The response path returns bytes by the reverse route. All segments between the phone and the laptop are encrypted by WireGuard; the segment between ttyd and tailscale0 is local to the loopback-like virtual interface and does not leave the host.

Security Model

The configuration is designed to withstand the following threats, in approximate order of likelihood: (a) an attacker scanning the public IP addresses of the laptop’s current network for open ports, (b) an attacker on the same Layer-2 network as the laptop (for example, a shared coffee-shop SSID) attempting to observe or connect to local services, (c) an attacker who compromises a single device on the tailnet and attempts to move laterally, and (d) an attacker with access to the basic-authentication password but not to the tailnet.

Three layered controls address these threats:

Network-layer authentication (Tailscale). Only devices that have completed the OAuth-style authentication flow with the tailnet’s coordination server and received a signed node key can establish WireGuard sessions with the laptop. This eliminates threats (a) and (b): the ttyd port is not even visible to hosts outside the tailnet. An attacker on a shared SSID sees only the encrypted WireGuard packets, which are indistinguishable from other UDP traffic.
Interface binding (ttyd -i tailscale0). The listener is bound to the Tailscale virtual interface rather than to 0.0.0.0. If the tailnet were somehow compromised at the coordination-server level, or if Tailscale itself were misconfigured to route traffic from unexpected sources into tailscale0, the service would still be unreachable from the physical interfaces. This is a defence-in-depth control.
Application-layer authentication (HTTP basic auth). A username and password are required to open a WebSocket, providing a final barrier against threat (c): an attacker who has compromised one tailnet device cannot open a shell on the laptop without also possessing the ttyd credentials. Basic authentication is not a strong control in isolation (the credentials are sent on every request and are visible in ps output on the host), but it is appropriate as the innermost layer of a multi-layer model.

Threat (d) is out of scope: an attacker with the basic-auth password but no tailnet credentials has no routable path to the service. This is the intended shape of the model: Tailscale provides confidentiality and network-layer authentication; basic authentication provides an independent defence against compromised tailnet peers; the interface binding ensures that neither of the other two controls is single-point-of-failure.

Appendix A describes an optional additional layer (TLS via tailscale cert) that closes the remaining residual risk of plaintext credentials on the wire between ttyd and the tailscale0 interface, at a modest operational cost.

Prerequisites

The configuration assumes the following environment:

Operating system: Ubuntu 24.04 on the laptop acting as the server. Other Linux distributions can be accommodated with minor substitutions in the package-manager invocations.
Client device: an iPhone running iOS 17 or later with Safari as the browser. Android and desktop browsers function identically with respect to the xterm.js client.
Hardware: any laptop with sufficient resources to run Ubuntu; the runtime footprint of the services described here is negligible.
Prior installation state: a Linux user account with sudo privileges, and a Tailscale account (the free tier is adequate for the use cases described here).
Prior knowledge: familiarity with systemd unit-file syntax and general Linux administration at the shell. Prior experience with tmux is helpful but not required.
Time investment: approximately thirty minutes for installation, configuration, and verification of the first successful browser connection.

Where any assumption fails to hold, the commands below may require adaptation.

Installation

On Ubuntu, the three required packages are available from the distribution repositories and the Tailscale install script:

sudo apt update
sudo apt install -y ttyd tmux
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

The tailscale up command prints an authentication URL. Open it in any browser, sign in, and confirm the device appears in the tailnet.

Note

If the distribution ships a ttyd older than 1.7, install a prebuilt binary from the ttyd releases page instead. Earlier versions lack the -i interface-binding flag used below.

Confirm the installs succeeded:

ttyd --version        # Expected: ttyd version 1.7.x
tailscale version     # Expected: 1.64.x or later
tmux -V               # Expected: tmux 3.4 or later
tailscale ip -4       # Prints the laptop's 100.x.y.z address

Configuration

Two configuration artifacts drive the setup: a single ttyd command line and a systemd user unit that wraps it. Both are stored in full under analysis/configs/ in the post repository.

The ttyd invocation

ttyd -W \
  -p 7681 \
  -i tailscale0 \
  -c user:'REPLACE_WITH_STRONG_PASSWORD' \
  tmux new -A -s main

Rationale for each flag:

-W enables write input on the pty. Without this flag the browser session is read-only, which is not useful for the interactive work that motivates the configuration.
-p 7681 specifies the TCP listening port. Any unused port is acceptable; 7681 is the ttyd project default and is used here for consistency with upstream documentation.
-i tailscale0 binds the listener exclusively to the Tailscale virtual interface. The service is therefore unreachable from physical wired, wireless, or cellular interfaces of the laptop, regardless of which network the laptop is attached to at any given moment. The interface name should be confirmed with ip -br link, as it is nominally fixed but may differ on non-default configurations.
-c user:password enables HTTP basic authentication. The credentials are an application-layer defence-in-depth measure; the primary access control is the Tailscale interface binding, and the basic-auth credentials are an independent safeguard against the scenario in which a tailnet peer is itself compromised.
tmux new -A -s main invokes tmux in a mode that creates the session if it does not exist and attaches to it otherwise. This form is idempotent, which is essential inside a systemd ExecStart directive. It also ensures that every browser connection attaches to the same session, so that shell state persists across client disconnections.

The systemd user unit

Manual launches do not survive logout. The unit below, saved as ~/.config/systemd/user/ttyd.service, runs the service as the logged-in user and restarts on failure.

# ~/.config/systemd/user/ttyd.service
# Tested on ttyd 1.7.7, Ubuntu 24.04
[Unit]
Description=ttyd terminal over Tailscale
After=network-online.target tailscaled.service
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/ttyd -W -p 7681 -i tailscale0 \
  -c user:REPLACE_WITH_STRONG_PASSWORD \
  /usr/bin/tmux new -A -s main
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

Enable and start:

systemctl --user daemon-reload
systemctl --user enable --now ttyd.service
systemctl --user status ttyd.service

To allow the user service to start before the user logs in at the console (so the laptop is reachable after an unattended reboot):

sudo loginctl enable-linger "$USER"

Restrict the unit file permissions so the basic-auth password is not world-readable:

chmod 600 ~/.config/systemd/user/ttyd.service

Editor view of the systemd unit file alongside a browser-based terminal, illustrating the configuration-to-runtime relationship.

Verification

Three checks confirm the stack is operating correctly.

# 1. Service is running
systemctl --user is-active ttyd.service
# Expected: active

# 2. Port is bound to the Tailscale interface only
ss -ltnp | grep 7681
# Expected: LISTEN on 100.x.y.z:7681 (tailnet address),
# NOT on 0.0.0.0:7681

# 3. End-to-end reachability from a second tailnet device
curl -u user:PASSWORD -I \
  http://$(tailscale ip -4):7681
# Expected: HTTP/1.1 200 OK

If step 2 shows 0.0.0.0:7681, the -i tailscale0 flag is not taking effect; the service is exposed to every network the laptop joins and must not be started until fixed.

Connecting from the iPhone

The client-side procedure consists of four steps:

Install the Tailscale application from the App Store.
Authenticate with the same account used on the laptop and confirm that the laptop appears under Devices in the Tailscale client.
Enable the Tailscale VPN toggle. iOS will report an active VPN connection in the status bar.
In Safari, navigate to the service URL:
```
http://laptop.tailnet-name.ts.net:7681
```
The MagicDNS name is reported by tailscale status on the laptop. The bare-IP form (http://100.x.y.z:7681) is functionally equivalent and is useful for diagnostic purposes when MagicDNS resolution is suspected to be failing.
Supply the basic-authentication credentials. The browser will render an interactive terminal attached to the main tmux session.

Tip

Adding the URL to the iPhone home screen via Safari’s Share > Add to Home Screen creates a standalone web-application icon. When launched from that icon, the page runs in a full-screen container without browser chrome, which improves usable display area on a small screen.

Daily Workflow

The table below summarises the commands and gestures that are most frequently required during routine use:

Command or gesture	Action
Open home-screen icon	Attach to `main` tmux session
`Ctrl-b d` (within tmux)	Detach without terminating the session
`Ctrl-b c`	Create a new window within the session
`Ctrl-b n` / `Ctrl-b p`	Cycle to next / previous tmux window
`systemctl --user restart ttyd.service`	Restart the terminal service
`journalctl --user -u ttyd.service -f`	Follow the service log in real time
`tailscale status`	List reachable devices on the tailnet
Pinch-zoom in Safari	Adjust terminal font size

A Bluetooth keyboard paired with the iPhone substantially improves the input ergonomics, to the point that the configuration becomes suitable for sustained interactive work rather than brief observation. Safari forwards the Ctrl, Esc, Tab, and arrow keys to xterm.js without rebinding. A keyboard on which Caps Lock has been remapped to Ctrl (an iOS system setting) reduces hand-position strain during extended tmux use.

Operational Considerations

The following issues arise sufficiently often during initial deployment and routine use to warrant explicit documentation. Each is presented with its symptom and its remediation.

Incorrect interface binding. If ss -ltnp shows 0.0.0.0:7681 rather than the tailnet address, the -i tailscale0 directive has not taken effect. The interface name should be confirmed with ip -br link, as it may differ on systems with non-default Tailscale installations or with systemd-networkd renaming. The service should not be left running until the binding has been verified to be correct, because a listener on 0.0.0.0 is reachable from every network the laptop attaches to.
Lid-close suspension. A laptop whose power-management policy suspends the system on lid closure is not reachable over the network while suspended. The remediation is one of: physically maintain an open lid; modify the desktop environment’s lid-close behaviour to ‘do nothing’; or, for transient tasks, wrap the command with systemd-inhibit --what=sleep to prevent suspension for the duration of the inhibition.
Loss of session persistence. If closing the browser terminates the shell, the ExecStart directive is launching a bare shell directly rather than attaching to a tmux session. Only the tmux new -A -s main form, with tmux as the terminal’s attached process, delivers persistence across client disconnections.
Service termination at logout. Without loginctl enable-linger , systemd user services are terminated when the user’s last interactive session ends. Lingering must be enabled once per user and, being a system-level setting, requires sudo.
Plaintext credentials in the unit file. The basic-authentication password is stored in plaintext in the ExecStart line of the unit file. The unit permissions must be restricted with chmod 600, and the unit file must not be committed to a shared repository. For higher sensitivity requirements, use a systemd credentials file (LoadCredential=) or migrate to Tailscale Serve as described under Opportunities for Improvement.
MagicDNS resolution failures. Tailnet hostnames must include the full tailnet suffix (for example, laptop.tailnet-name.ts.net). A bare hostname resolves only if MagicDNS is enabled in the Tailscale admin console and if the client has accepted the DNS configuration pushed by the Tailscale daemon; clients that predate the MagicDNS enablement may require a Tailscale toggle cycle to refresh.
iOS input autocorrection. Safari on iOS auto-capitalises the first character entered into a text field by default, which can corrupt the first character of a password or a command. The appropriate remediations are to disable auto-capitalisation under Settings > General > Keyboard, or to use a Bluetooth keyboard, which is not subject to the on-screen keyboard’s input transformations.

Uninstall / Rollback

To remove the setup, disable the service and uninstall the packages.

# 1. Stop and disable the service
systemctl --user disable --now ttyd.service
rm ~/.config/systemd/user/ttyd.service
systemctl --user daemon-reload

# 2. Disable lingering if no other user service needs it
sudo loginctl disable-linger "$USER"

# 3. Uninstall packages
sudo apt remove --purge -y ttyd
# Leave tmux and tailscale if used elsewhere; otherwise:
sudo apt remove --purge -y tmux
sudo tailscale down
sudo apt remove --purge -y tailscale

Remove the laptop from the tailnet via the Tailscale admin console if the device is being retired.

Workflow scene: the laptop closed on a desk while a phone shows the same tmux session mid-task, suggesting continuity without co-location.

Discussion

Observations

Conceptual observations:

A browser is an adequate terminal client when the network transport is handled by a component specifically engineered for authenticated overlay networking. The SSH daemon is one of several defensible approaches to remote shell access; in this configuration the transport responsibility is relocated from SSH to WireGuard, and the authentication responsibility is distributed across WireGuard, ttyd basic auth, and the interface binding.
MagicDNS eliminates the operational burden of dynamic DNS, port forwarding, and firewall-rule maintenance for personal-scale remote access. The trade-off is dependence on the Tailscale coordination server for name resolution.
The interface-binding flag on ttyd is the single most consequential hardening decision in the configuration. A service bound to 0.0.0.0 with HTTP basic authentication represents a qualitatively different risk profile from a service bound to a WireGuard interface; the former is subject to public-internet scanning, the latter is not.
Per-user systemd units, combined with loginctl enable-linger, provide a clean abstraction for background services that require user-level file system access without requiring system-level privileges beyond the initial linger enablement.

Technical observations:

The tmux new -A -s form is both a creator and an attacher, and is therefore idempotent. This makes it safe to invoke from an ExecStart directive, where a non-idempotent invocation would cause a restart loop.
The ss -ltnp command is more direct and more precise than netstat -ltnp for determining the interface on which a service is listening, and it is the standard utility on distributions that have replaced net-tools with iproute2.
The tailscale cert command issues short-lived TLS certificates for MagicDNS hostnames, enabling end-to-end HTTPS without involvement of a public certificate authority. Certificates expire and must be reissued on a schedule, as discussed in Appendix A.
The Safari Add to Home Screen action produces a standalone web-application surface that removes browser chrome and recovers screen area, which materially improves usability on any URL that is accessed with regularity.

Recurring Pitfalls:

Omission of the -W flag yields a read-only terminal, which presents as a non-interactive first connection and is easily misdiagnosed as a network or authentication problem.
Binding to tailscale0 without first confirming the interface name (via ip -br link) produces a service that fails to start and logs an interface-not-found error; the service is inert but not obviously so on casual inspection.
The basic-authentication password is visible both in ps output on the host and in the unit file. It should be classified as a moderate-sensitivity secret: rotated on suspicion of exposure, but not requiring the operational controls appropriate to a high-sensitivity credential such as an SSH private key.
iOS aggressively reclaims memory from backgrounded Safari tabs. The WebSocket reconnects transparently on tab reactivation, but any command-line input that had been typed but not submitted at the time of backgrounding is lost.

Limitations

The configuration has several explicit constraints that should be understood at the outset:

Single-user scope. The unit file and basic-authentication credentials are per-user. Multi-user access requires either parallel service instances on distinct ports or a front-end reverse proxy that demultiplexes on authenticated identity.
Absence of audit logging. ttyd does not log the commands executed within the pty. Where accountability or forensic capability is required, a separate shell-level or system-level auditing facility (for example, auditd with appropriate rules) must be configured.
Input ergonomics on a phone-scale device. Even with a Bluetooth keyboard and the small-screen optimisations, extended authorship of new code on a handheld device remains slower than on a conventional workstation. The configuration is well suited to monitoring, intervention, and short modifications; it is not a substitute for a primary development environment.
Dependence on Tailscale infrastructure. New connections require reachability to the Tailscale coordination server for key exchange and peer discovery. Existing WireGuard sessions are unaffected by a coordination-server outage, but reconnection after a client restart requires coordination-server availability.
Absence of session recording. The configuration does not record session input or output. Tools such as asciinema, tmux-logging, or script(1) can be added if session provenance is required.

Opportunities for Improvement

Several extensions would strengthen the configuration without departing from its overall design:

Adopt end-to-end TLS using tailscale cert, so that an accidental mis-binding of ttyd to a non-Tailscale interface would not expose an unencrypted session. Appendix A documents the procedure.
Replace HTTP basic authentication with Tailscale Serve or Tailscale Funnel, both of which delegate authentication to the tailnet identity of the client. This removes the need for a shared password and eliminates the associated secret-management burden.
Deploy the same configuration on an always-on VPS reached through Tailscale. The benefit is session durability across laptop sleep or shutdown; the cost is the VPS itself and the slightly more involved administrative story.
Instantiate a second, read-only ttyd on a distinct port that runs a scoped command such as htop or a log-tailing command. This provides an observation-only surface suitable for short status checks without opening the full shell.
Curate a phone-optimised tmux configuration (tmux.conf.phone) with shortened status-bar labels and fewer segments, and load it conditionally when ttyd detects a small-screen client.
Encapsulate the installation in an idempotent shell script under analysis/configs/, suitable for provisioning identical configurations on additional machines.

Conclusion

A terminal served over HTTP and reached through a private WireGuard mesh constitutes a workable mobile surface for a laptop that may be closed, stationed in another room, or physically distant from the operator. The marginal cost is modest: three package installations, a single systemd user unit, and an initial configuration effort of approximately thirty minutes. The benefit is that long-running work on the laptop, including interactive agentic coding sessions that may run for several hours, remains observable and steerable without any exposure of the laptop to the public internet, without dynamic DNS infrastructure, and without reliance on a native SSH client on the mobile device.

The principal design insight is that a browser is an adequate terminal client when the transport-layer authentication and confidentiality problems are delegated to a component engineered specifically for that purpose. Once Tailscale and ttyd are in place, the same configuration generalises to a cloud-hosted VPS: only the server endpoint changes, and the tailnet membership, the ttyd invocation, and the client-side procedure are invariant.

In conclusion, four points merit emphasis. First, the single most consequential line in the configuration is -i tailscale0 on the ttyd invocation: that binding decision removes the service from every non-tailnet interface and makes the rest of the security model meaningful. Second, a per-user systemd unit combined with loginctl enable-linger provides a logout-safe and reboot-safe service without any system-level privileges beyond the initial linger enablement. Third, the tmux new -A -s main argument inside the ExecStart directive is the mechanism that transforms an ephemeral browser tab into a durable working session. Fourth, the Safari ‘Add to Home Screen’ action yields a standalone web-application presentation that recovers enough screen area to make the terminal practically usable on phone-scale displays.

Extended Glossary

The following terms appear throughout the post. Definitions are compact and operational, prioritising the usage relevant to this configuration over full generality.

ACL (access control list). In Tailscale, a declarative policy document that restricts which tailnet members may reach which services on which ports. Default policy is permissive; explicit ACLs tighten the model.

Basic authentication (HTTP). The authentication scheme defined by RFC 7617 in which the client sends a header of the form Authorization: Basic base64(user:password) on each request. Sufficient when transport-layer confidentiality is assured by another mechanism; inadequate alone over plaintext HTTP.

Coordination server (Tailscale). The central component of the Tailscale control plane, responsible for device authentication, public-key distribution, and ACL enforcement. It does not carry data traffic; user traffic flows peer-to-peer over WireGuard or via DERP relays.

DERP (Designated Encrypted Relay for Packets). Tailscale-operated relay servers used when two peers cannot establish a direct WireGuard connection due to symmetric or restrictive NAT. Traffic through a DERP relay remains end-to-end encrypted; the relay cannot read it.

Dynamic DNS. A service that updates a DNS A or AAAA record in response to changes in an endpoint’s public IP address. Historically used for home-hosted services but requires an open public port on the host. This configuration deliberately avoids it in favour of MagicDNS over Tailscale.

ExecStart. A systemd unit directive that specifies the command line to run when the service is started. Must be an absolute path with no shell interpretation; metacharacters are passed literally to the process.

HTTP. The Hypertext Transfer Protocol, as used here in its unencrypted form over a Tailscale-encrypted transport. The ttyd listener speaks HTTP, and the security model delegates confidentiality to the WireGuard layer below it.

HTTPS. HTTP over TLS. In this configuration, HTTPS is optional (Appendix A) because the transport is already authenticated and encrypted by Tailscale; adding HTTPS provides defence in depth against an accidental mis-binding of the listener.

Interface binding. The act of restricting a listening socket to accept connections only on a named network interface, typically by specifying an interface name or an IP address belonging to that interface. Here, ttyd -i tailscale0 binds the listener to the Tailscale virtual interface.

Linger (systemd). A setting, enabled per user by loginctl enable-linger , that causes systemd to keep a user’s services running even when no session is active on that user’s behalf. Required for a systemd user unit to persist across logout and reboot.

MagicDNS. A Tailscale feature that automatically resolves hostnames of the form ..ts.net to tailnet IP addresses, with the resolution delivered via the tailnet VPN rather than the system resolver. Eliminates the need for dynamic DNS or manual hosts-file maintenance.

NAT (network address translation). The technique by which a router rewrites source or destination addresses in IP packets, commonly used to share a single public IPv4 address among many private hosts. WireGuard and DERP together are responsible for establishing communication across NATs without requiring port forwarding.

NAT traversal. The set of techniques used to establish peer-to-peer connectivity between hosts behind NAT devices, including STUN-style discovery and UDP hole punching. Tailscale performs NAT traversal automatically; DERP is the fallback when it fails.

Overlay network. A network built logically on top of an underlying physical network, with its own address space and its own forwarding rules. The Tailscale 100.x.y.z/10 address space is an overlay over the underlying internet.

Posit Package Manager. A binary R package mirror (formerly RStudio Package Manager) used by the zzcollab Docker image to accelerate package installation. Not directly relevant to the ttyd configuration, but present in the repository’s reproducibility infrastructure.

Pseudo-terminal (pty). A pair of character-device endpoints in the kernel, the master and the slave, that emulates a serial line between a terminal program and a process. ttyd owns the master end; tmux (or another attached process) reads from and writes to the slave end.

Quarto. The open-source scientific and technical publishing system used to render this post. Produces HTML, PDF, and other formats from a unified source document. Relevant to the blog infrastructure rather than to the ttyd setup.

renv. The R package-version pinning and isolation tool used throughout the zzcollab research framework. Not directly used by the ttyd configuration.

Rocker. A family of Docker images providing R and common toolchains; rocker/tidyverse is the base image for this blog’s rendering environment. Again, infrastructure rather than subject matter.

Session (tmux). A named, persistent grouping of tmux windows and panes that survives detachment from any client. The main session used here is created by tmux new -A -s main and is attached by all subsequent ttyd connections.

ss. A socket-statistics utility from the iproute2 suite, preferred over the legacy netstat tool for listing listening sockets. The invocation ss -ltnp lists TCP listeners with their owning process and the bound interface.

SSH (Secure Shell). A cryptographic network protocol for remote shell access, defined principally by RFC 4251 through RFC 4254. This configuration deliberately does not use SSH for inbound access, although SSH may still be used for other purposes on the laptop.

systemd. The init system and service manager used by most contemporary Linux distributions. Responsible for launching, supervising, and logging long-running services.

systemd user unit. A service definition stored under ~/.config/systemd/user/ and managed with systemctl --user. Runs as the user rather than as root; requires loginctl enable-linger to persist across logout.

Tailnet. A private network of devices enrolled in a single Tailscale account. Members are mutually addressable; non-members are not routable to tailnet addresses.

Tailscale. A commercial product that builds a peer-to-peer WireGuard mesh between enrolled devices, coupled with a coordination server for authentication and key distribution. The free tier is adequate for personal use.

Tailscale Funnel. An optional Tailscale feature that exposes a selected tailnet service to the public internet through a Tailscale-operated reverse proxy, with TLS termination and public-DNS resolution. Not used in this configuration.

Tailscale Serve. An optional Tailscale feature that provides TLS termination and routing for tailnet-internal HTTPS services, using certificates issued automatically via tailscale cert. A plausible future replacement for the HTTP + basic-auth configuration described here.

TLS (Transport Layer Security). The successor to SSL, and the protocol used to encrypt HTTP into HTTPS. Optional in this configuration because Tailscale already provides confidentiality at the WireGuard layer.

tmux. A terminal multiplexer that maintains persistent sessions containing windows and panes. Essential to this configuration because it decouples the shell lifetime from the lifetime of any individual client connection.

ttyd. The daemon that is the subject of this post. Serves a terminal over HTTP via WebSocket, using xterm.js on the client side.

WebSocket. A protocol (RFC 6455) that upgrades an HTTP connection to a full-duplex, persistent framed-TCP channel, used by ttyd to stream keystrokes and output between the browser and the pty.

WireGuard. A modern encrypted tunnel protocol, noted for its small codebase and its use of Curve25519 and ChaCha20-Poly1305. Tailscale builds its overlay on WireGuard but adds a coordination plane, NAT traversal, and identity management on top.

xterm.js. A JavaScript terminal emulator library that implements VT100/VT220/xterm escape-sequence handling in the browser. Rendered by ttyd on every client connection.

Reproducibility

Tested configuration:

Component	Version
Operating system	Ubuntu 24.04
ttyd	1.7.7
Tailscale	1.64.2
tmux	3.4
Shell	zsh 5.9
Client	iOS 17.4, Safari
Last verified	2026-04-15

Configuration files:

analysis/configs/ttyd.service: the full systemd user unit
analysis/configs/install.sh: an idempotent install script covering ttyd, tmux, and Tailscale
analysis/configs/tmux.conf.phone: an optional tmux configuration tuned for small screens

To reproduce end-to-end:

bash analysis/configs/install.sh
sudo tailscale up
install -m 600 analysis/configs/ttyd.service \
  ~/.config/systemd/user/ttyd.service
# edit the unit and substitute a real password
systemctl --user daemon-reload
systemctl --user enable --now ttyd.service
sudo loginctl enable-linger "$USER"

Appendix A: TLS with tailscale cert

For transport encryption end-to-end (rather than relying on Tailscale’s WireGuard tunnel alone), issue a certificate for the MagicDNS name and point ttyd at it.

sudo mkdir -p /etc/ttyd
cd /etc/ttyd
sudo tailscale cert laptop.tailnet-name.ts.net

Adjust the unit’s ExecStart to include:

--ssl \
--ssl-cert /etc/ttyd/laptop.tailnet-name.ts.net.crt \
--ssl-key  /etc/ttyd/laptop.tailnet-name.ts.net.key \

Access the service at https://laptop.tailnet-name.ts.net:7681. Certificates issued by tailscale cert expire; schedule a weekly systemd timer to reissue and reload:

systemctl --user edit --force --full ttyd-cert-renew.timer

Appendix B: Sample Work Session

The following sequence illustrates a representative end-to-end session, under the assumption that the configuration above is complete and the laptop is running.

Step 1. At the workstation, open a local terminal and attach to the shared tmux session:

tmux attach -t main

Step 2. Initiate a long-running interactive task (for example, an agentic coding session or a computation that will run for an extended period). Detach from the tmux session with Ctrl-b d. The task continues to execute.

Step 3. Depart from the workstation. The laptop remains powered and attached to a network. From any enrolled tailnet device, launch the home-screen icon for the ttyd URL and authenticate with the configured credentials. The browser attaches to the same tmux session, and the task’s current output appears.

Step 4. Observe the output, issue intervention commands as required, and close the browser tab when finished. Closing the tab terminates only the WebSocket connection to ttyd; the tmux session and its subprocesses continue to run.

Step 5. On return to the workstation, reattach locally with tmux attach -t main. The session state is continuous across the entire sequence; no work is lost at any transition.

Appendix C: Teardown (Undo)

Full removal in order:

systemctl --user disable --now ttyd.service
rm ~/.config/systemd/user/ttyd.service
sudo loginctl disable-linger "$USER" (if no other user service needs it)
sudo apt remove --purge -y ttyd tmux
sudo tailscale down
sudo apt remove --purge -y tailscale
Remove the device from the Tailscale admin console.
Delete any issued certificates under /etc/ttyd/.

Warning

If the laptop is being decommissioned rather than just retired from this workflow, revoke its Tailscale auth key in the admin console. A device removed only locally can reappear on the tailnet if the machine is recovered.

Contact

Corrections, alternative approaches, and substantive discussion are welcome.

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: Contact form

Feedback is particularly useful on the following topics:

Corrections or improvements to any of the configuration or the associated reasoning.
Client-side experience on platforms other than iOS (Android, Chrome OS, iPadOS tablet use).
Extensions of the pattern, for example Tailscale Serve, Tailscale Funnel, mutual TLS, or certificate-based client authentication, and the operational trade-offs encountered.
Reports of deployments of this configuration on a cloud VPS rather than on a mobile laptop.

Dynamic Column Names in R: Seven Approaches Compared

Zenn — Mon, 16 Feb 2026 08:00:00 GMT

Introduction

While reviewing some production R code recently, the following pattern appeared:

data |>
  mutate(
    !!paste0(measure, "_bl") := baseline_value,
    !!paste0(measure, "_cng") := current - baseline_value
  )

The !! and := operators were unfamiliar. After investigation, they proved to be part of R’s tidy evaluation system: a powerful but often misunderstood feature for programmatic data manipulation.

This exploration led to cataloguing seven different approaches to creating columns with dynamic names in R:

Base R using [[ assignment
Base R using do.call() and setNames()
The “classic” tidyverse approach with !! and :=
The modern tidyverse glue-style syntax
The rlang expression splicing pattern
The data.table approach
The collapse package approach

The Problem

Consider a function that calculates change scores for any cognitive measure. Given a dataframe with columns rid (subject ID), vis (visit), and a measure like mmse, we wish to create two new columns:

mmse_bl: the baseline value for each subject
mmse_cng: the change from baseline at each visit

The column names must be constructed dynamically because the function should work for any measure (mmse, adas13, cdr, etc.).

Here is our sample data:

library(dplyr)


Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

df <- tibble(
  rid = rep(c("001", "002"), each = 3),
  vis = rep(c("bl", "m06", "m12"), 2),
  mmse = c(28, 27, 25, 30, 29, 28)
)

df

# A tibble: 6 × 3
  rid   vis    mmse
    
1 001   bl       28
2 001   m06      27
3 001   m12      25
4 002   bl       30
5 002   m06      29
6 002   m12      28

Approach 1: Base R with `[[` Assignment

The most straightforward approach uses base R’s [[ assignment operator:

calculate_change_base <- function(data, measure) {
  bl_col <- paste0(measure, "_bl")
  cng_col <- paste0(measure, "_cng")

  result <- data

  for (id in unique(data$rid)) {
    idx <- data$rid == id
    baseline <- data[[measure]][idx & data$vis == "bl"][1]
    result[[bl_col]][idx] <- baseline
    result[[cng_col]][idx] <- data[[measure]][idx] - baseline
  }

  result
}

calculate_change_base(df, "mmse")

# A tibble: 6 × 5
  rid   vis    mmse mmse_bl mmse_cng
           
1 001   bl       28      28        0
2 001   m06      27      28       -1
3 001   m12      25      28       -3
4 002   bl       30      30        0
5 002   m06      29      30       -1
6 002   m12      28      30       -2

Discussion

How it works: The [[ operator accepts character strings for column names, unlike $ which requires literal names. Writing df[["mmse"]] is equivalent to df$mmse, but the former allows df[[variable]] where variable contains the column name.

Advantages:

No dependencies beyond base R
Syntax is familiar to programmers from other languages
Explicit and easy to debug

Disadvantages:

Verbose, especially with grouped operations
Requires explicit loops for by-group calculations
Mutable state pattern (result is modified in place)
Does not integrate with dplyr pipelines
Performance degrades with many groups

When to use: Small scripts with no package dependencies, or when interfacing with code from other languages where this pattern is standard.

Approach 2: Base R with `do.call()` and `setNames()`

A more functional base R approach avoids explicit loops:

calculate_change_docall <- function(data, measure) {
  bl_col <- paste0(measure, "_bl")
  cng_col <- paste0(measure, "_cng")

  baselines <- tapply(
    data[[measure]][data$vis == "bl"],
    data$rid[data$vis == "bl"],
    FUN = `[`, 1
  )

  bl_values <- baselines[data$rid]
  cng_values <- data[[measure]] - bl_values

  new_cols <- setNames(
    list(as.numeric(bl_values), as.numeric(cng_values)),
    c(bl_col, cng_col)
  )

  do.call(cbind, c(list(data), new_cols))
}

calculate_change_docall(df, "mmse")

  rid vis mmse mmse_bl mmse_cng
1 001  bl   28      28        0
2 001 m06   27      28       -1
3 001 m12   25      28       -3
4 002  bl   30      30        0
5 002 m06   29      30       -1
6 002 m12   28      30       -2

Discussion

How it works: setNames() creates a named list where the names come from a character vector. do.call(cbind, ...) then binds these columns to the original dataframe. The tapply() function computes grouped summaries without explicit loops.

Advantages:

No external dependencies
Functional style (no mutable state)
Vectorized operations for better performance

Disadvantages:

Dense, less readable syntax
tapply() returns an array requiring careful indexing
Type coercion issues (note the as.numeric() calls)
Returns a data.frame, not a tibble

When to use: Package development where minimising dependencies is critical, or performance-sensitive code where the overhead of dplyr is unacceptable.

Tidy evaluation approaches.

Approach 3: Classic Tidyverse with `!!` and `:=`

The tidyverse introduced tidy evaluation to handle dynamic column names. Two operators are central:

:= (the “walrus” operator): Allows the left-hand side of an assignment to be evaluated
!! (bang-bang): Unquotes an expression, forcing immediate evaluation

calculate_change_classic <- function(data, measure) {
  bl_col <- paste0(measure, "_bl")
  cng_col <- paste0(measure, "_cng")

  data |>
    group_by(rid) |>
    mutate(
      !!bl_col := .data[[measure]][vis == "bl"][1],
      !!cng_col := .data[[measure]] - .data[[bl_col]]
    ) |>
    ungroup()
}

calculate_change_classic(df, "mmse")

# A tibble: 6 × 5
  rid   vis    mmse mmse_bl mmse_cng
           
1 001   bl       28      28        0
2 001   m06      27      28       -1
3 001   m12      25      28       -3
4 002   bl       30      30        0
5 002   m06      29      30       -1
6 002   m12      28      30       -2

Discussion

How it works: Standard mutate() syntax like mutate(new_col = value) treats new_col as a literal name. The = operator does not evaluate its left-hand side. The := operator changes this behaviour. When mutate(!!bl_col := value) is written, the !! forces bl_col to be evaluated (yielding "mmse_bl"), and := uses that string as the column name.

The .data pronoun explicitly references columns in the dataframe, avoiding ambiguity between dataframe columns and local variables.

Advantages:

Integrates seamlessly with dplyr pipelines
Grouped operations are trivial
Declarative, readable intent (once the syntax is learnt)

Disadvantages:

Unfamiliar syntax for newcomers (!! and := are not standard R)
Requires understanding tidy evaluation concepts
The rlang package must be available (loaded with dplyr)

When to use: Legacy tidyverse code (pre-dplyr 1.0), or when the full power of quasiquotation is needed for complex metaprogramming.

Approach 4: Modern Tidyverse with Glue Syntax

Starting with dplyr 1.0 (June 2020), a cleaner syntax emerged using glue-style interpolation:

calculate_change_modern <- function(data, measure) {
  data |>
    group_by(rid) |>
    mutate(
      "{measure}_bl" := .data[[measure]][vis == "bl"][1],
      "{measure}_cng" := .data[[measure]] - .data[[paste0(measure, "_bl")]]
    ) |>
    ungroup()
}

calculate_change_modern(df, "mmse")

# A tibble: 6 × 5
  rid   vis    mmse mmse_bl mmse_cng
           
1 001   bl       28      28        0
2 001   m06      27      28       -1
3 001   m12      25      28       -3
4 002   bl       30      30        0
5 002   m06      29      30       -1
6 002   m12      28      30       -2

Discussion

How it works: The string "{measure}_bl" is interpolated like glue::glue(), substituting the value of measure directly. This occurs at the level of the column name, with := still required to enable left-hand side evaluation.

Advantages:

Most readable of all tidyverse approaches
Reduces cognitive load (no need to remember !! semantics)
Consistent with glue syntax used elsewhere in the tidyverse

Disadvantages:

Requires dplyr >= 1.0
Still requires := (cannot use =)
Less flexible than !! for complex quasiquotation

When to use: New tidyverse code. This is the current recommended approach for dynamic column names in dplyr.

Approach 5: rlang Expression Splicing

For more explicit metaprogramming, rlang provides expr() for building expressions and !!! for splicing them into function calls:

library(rlang)

calculate_change_rlang <- function(data, measure) {
  bl_col <- paste0(measure, "_bl")
  cng_col <- paste0(measure, "_cng")

  exprs <- list(
    expr(.data[[!!measure]][vis == "bl"][1]),
    expr(.data[[!!measure]] - .data[[!!bl_col]])
  )
  names(exprs) <- c(bl_col, cng_col)

  data |>
    group_by(rid) |>
    mutate(!!!exprs) |>
    ungroup()
}

calculate_change_rlang(df, "mmse")

# A tibble: 6 × 5
  rid   vis    mmse mmse_bl mmse_cng
           
1 001   bl       28      28        0
2 001   m06      27      28       -1
3 001   m12      25      28       -3
4 002   bl       30      30        0
5 002   m06      29      30       -1
6 002   m12      28      30       -2

Discussion

How it works: expr() captures an expression without evaluating it, while !! inside expr() forces evaluation of specific parts. The result is a list of named expressions. The !!! (splice) operator unpacks this list into mutate(), equivalent to writing each expression as a separate argument.

Advantages:

Build expressions programmatically before evaluation
Useful when the number of columns is dynamic
Expressions can be inspected, modified, or logged before use
Clear separation between expression construction and evaluation

Disadvantages:

More verbose for simple cases
Requires understanding quasiquotation deeply
Overkill for straightforward dynamic naming

When to use: Complex metaprogramming where expressions must be built dynamically, such as generating an unknown number of columns based on input data, or when inspecting or logging expressions before evaluation is desirable.

High-performance alternatives.

Approach 6: data.table

The data.table package has its own := operator that predates tidyverse adoption:

library(data.table)


Attaching package: 'data.table'

The following object is masked from 'package:rlang':

    :=

The following objects are masked from 'package:dplyr':

    between, first, last

calculate_change_dt <- function(data, measure) {
  bl_col <- paste0(measure, "_bl")
  cng_col <- paste0(measure, "_cng")

  dt <- as.data.table(data)

  dt[, (bl_col) := .SD[[measure]][vis == "bl"][1], by = rid]
  dt[, (cng_col) := .SD[[measure]] - .SD[[bl_col]], by = rid]

  dt[]
}

calculate_change_dt(df, "mmse")

      rid    vis  mmse mmse_bl mmse_cng
            
1:    001     bl    28      28        0
2:    001    m06    27      28       -1
3:    001    m12    25      28       -3
4:    002     bl    30      30        0
5:    002    m06    29      30       -1
6:    002    m12    28      30       -2

Discussion

How it works: In data.table, wrapping a variable in parentheses on the left-hand side of := forces evaluation: (bl_col) evaluates to "mmse_bl". Without parentheses, bl_col := value would create a column literally named “bl_col”. The .SD pronoun (Subset of Data) references the current group’s data.

Advantages:

Extremely fast, especially for large datasets
Memory efficient (modifies in place by reference)
Mature, battle-tested codebase
The by argument handles grouping concisely

Disadvantages:

Different mental model from base R and tidyverse
Modify-by-reference semantics can cause subtle bugs
Less readable for those unfamiliar with data.table idioms
Parentheses syntax (col) is easy to forget

When to use: Performance-critical applications, very large datasets (millions of rows), or codebases already using data.table.

Minimalist high-speed data tools.

Approach 7: collapse

The collapse package offers high-performance data manipulation with its own idioms:

library(collapse)

collapse 2.1.6, see ?`collapse-package` or ?`collapse-documentation`


Attaching package: 'collapse'

The following object is masked from 'package:data.table':

    fdroplevels

The following object is masked from 'package:stats':

    D

calculate_change_collapse <- function(data, measure) {
  bl_col <- paste0(measure, "_bl")
  cng_col <- paste0(measure, "_cng")

  bl_data <- fsubset(data, vis == "bl")
  bl_values <- get_vars(bl_data, c("rid", measure))
  names(bl_values)[2] <- bl_col

  result <- join(data, bl_values, on = "rid", how = "left")
  result[[cng_col]] <- result[[measure]] - result[[bl_col]]

  result
}

calculate_change_collapse(df, "mmse")

left join: data[rid] 6/6 (100%) <3:1st> bl_values[rid] 2/2 (100%)

# A tibble: 6 × 5
  rid   vis    mmse mmse_bl mmse_cng
           
1 001   bl       28      28        0
2 001   m06      27      28       -1
3 001   m12      25      28       -3
4 002   bl       30      30        0
5 002   m06      29      30       -1
6 002   m12      28      30       -2

Discussion

How it works: collapse provides fast versions of common operations (fsubset, get_vars, join, etc.). Unlike dplyr, collapse does not use tidy evaluation, so dynamic column access requires base R syntax like get_vars() with character vectors and direct [[ assignment. The approach above uses a join strategy rather than grouped mutation.

Advantages:

Extremely fast (often faster than data.table for certain operations)
Low memory footprint
Works with both data.frames and data.tables
Good for time series and panel data

Disadvantages:

Smaller user community than tidyverse or data.table
Inconsistent tidy evaluation support across functions
Often requires mixing collapse functions with base R syntax
Less comprehensive documentation

When to use: Performance-critical code, especially time series or panel data analysis. Also useful when speed is needed but data.table’s reference semantics are to be avoided.

Comparison Summary

Approach	Dependencies	Readability	Performance	Best For
Base R `[[`	None	Moderate	Low	No-dependency scripts
Base R `do.call`	None	Low	Moderate	Package development
Classic `!!` `:=`	dplyr	Low	Moderate	Legacy tidyverse code
Modern glue	dplyr >= 1.0	High	Moderate	New tidyverse code
rlang `!!!` splice	rlang	Moderate	Moderate	Complex metaprogramming
data.table	data.table	Moderate	High	Large datasets
collapse	collapse	Moderate	Very High	Performance-critical work

Practical Example: Multiple Measures

The real power emerges when processing multiple measures:

df_multi <- tibble(
  rid = rep(c("001", "002"), each = 3),
  vis = rep(c("bl", "m06", "m12"), 2),
  mmse = c(28, 27, 25, 30, 29, 28),
  adas = c(12, 14, 18, 10, 12, 15),
  cdr = c(0.5, 0.5, 1.0, 0.5, 0.5, 0.5)
)

calculate_all_changes <- function(data, measures) {
  result <- data |> group_by(rid)

  for (measure in measures) {
    result <- result |>
      mutate(
        "{measure}_bl" := .data[[measure]][vis == "bl"][1],
        "{measure}_cng" := .data[[measure]] - .data[[paste0(measure, "_bl")]]
      )
  }

  result |> ungroup()
}

calculate_all_changes(df_multi, c("mmse", "adas", "cdr"))

# A tibble: 6 × 11
  rid   vis    mmse  adas   cdr mmse_bl mmse_cng adas_bl adas_cng cdr_bl cdr_cng
                         
1 001   bl       28    12   0.5      28        0      12        0    0.5     0  
2 001   m06      27    14   0.5      28       -1      12        2    0.5     0  
3 001   m12      25    18   1        28       -3      12        6    0.5     0.5
4 002   bl       30    10   0.5      30        0      10        0    0.5     0  
5 002   m06      29    12   0.5      30       -1      10        2    0.5     0  
6 002   m12      28    15   0.5      30       -2      10        5    0.5     0

Common Pitfalls

1. Forgetting := on the LHS (tidyverse)

# Wrong: creates column literally named "{measure}_bl"
mutate("{measure}_bl" = value)

# Correct: evaluates the string
mutate("{measure}_bl" := value)

2. Forgetting parentheses (data.table)

# Wrong: creates column named "bl_col"
dt[, bl_col := value]

# Correct: evaluates bl_col to get "mmse_bl"
dt[, (bl_col) := value]

3. Confusing column references

# Ambiguous: is 'measure' a column or variable?
mutate("{measure}_bl" := measure[vis == "bl"][1])

# Clear: .data[[measure]] references the column
mutate("{measure}_bl" := .data[[measure]][vis == "bl"][1])

4. Reference semantics in data.table

dt <- as.data.table(df)
dt2 <- dt  # This is NOT a copy!
dt2[, new_col := 1]  # Modifies both dt and dt2

# Safe copy:
dt2 <- copy(dt)

What Did We Learn?

Lessons Learnt

:= enables dynamic LHS: In both tidyverse and data.table, this operator allows the left side of an assignment to be evaluated
!! is the unquote operator: Forces immediate evaluation of an expression in tidy eval contexts
Glue syntax is cleaner: "{var}_suffix" is more readable than !!paste0(var, "_suffix")
Parentheses matter in data.table: (col) evaluates, col is literal
.data removes ambiguity: Always use .data[[col]] for column references in functions
Choose based on context: Readability (glue) vs performance (data.table) vs no dependencies (base R)

Reproducibility

sessionInfo()

R version 4.5.3 (2026-03-11)
Platform: aarch64-apple-darwin25.3.0
Running under: macOS Tahoe 26.5

Matrix products: default
BLAS:   /opt/homebrew/Cellar/openblas/0.3.32/lib/libopenblasp-r0.3.32.dylib 
LAPACK: /opt/homebrew/Cellar/r/4.5.3/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.1

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Los_Angeles
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] collapse_2.1.6    data.table_1.17.8 rlang_1.2.0       dplyr_1.2.1      

loaded via a namespace (and not attached):
 [1] digest_0.6.37     utf8_1.2.6        R6_2.6.1          fastmap_1.2.0    
 [5] tidyselect_1.2.1  xfun_0.56         magrittr_2.0.5    glue_1.8.0       
 [9] tibble_3.3.1      knitr_1.50        parallel_4.5.3    pkgconfig_2.0.3  
[13] htmltools_0.5.8.1 generics_0.1.4    rmarkdown_2.29    lifecycle_1.0.5  
[17] cli_3.6.6         vctrs_0.7.3       compiler_4.5.3    tools_4.5.3      
[21] pillar_1.11.1     evaluate_1.0.5    Rcpp_1.1.0        yaml_2.3.10      
[25] jsonlite_2.0.0    htmlwidgets_1.6.4

Let’s Connect

Have questions, suggestions, or spot an error? Let me know.

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: Contact form

I would enjoy hearing from readers who:

Spot an error or a better approach to any of the code in this post.
Have suggestions for topics to cover.
Want to discuss R programming, data science, or reproducible research.
Have questions about anything in this tutorial.
Simply want to say hello and connect.

Updating an R Package: A Complete Development Workflow

Ronald G. Thomas — Wed, 11 Feb 2026 08:00:00 GMT

Terminal showing an R package build in progress

Setting up a reproducible R package development workflow.

Introduction

I did not really understand how much discipline goes into modifying an R package until I tried to push a change to one of my own projects and watched CI fail on three platforms simultaneously. It turns out that editing an R file is the easy part; the surrounding workflow of branching, documenting, testing, building, and integrating with GitHub Actions is what separates a quick fix from a reliable contribution.

The scenario is straightforward: an existing R package is hosted on GitHub and a feature must be added, a bug fixed, or error handling improved. The change itself might be small, but the process of getting it safely merged involves a surprising number of steps.

This post documents the complete workflow I learned while modifying my own package, zzdataframe2graphic. Every step from creating a feature branch to cleaning up after a merge is covered, with the goal of making the process repeatable for future changes.

More formally, this post documents the R-packages layer (Layer 13) of the Workflow Construct described in post 52, specifically the maintenance loop that keeps an existing package healthy across a working biostatistician’s many simultaneous projects. The construct’s R-packages layer holds a small set of author-maintained zz* packages; this post is the how-to-update-one companion to post 35, which is the how-to-author-one entry point.

Motivations

I kept making ad hoc changes to my R packages without a systematic process, and occasionally broke things on other platforms.
I wanted a clear mental model for the branch-edit-test-merge cycle so I could contribute to open source R projects with confidence.
I had GitHub Actions YAML files in my repositories but did not fully understand what they did or how to set them up from scratch.
I needed a reference I could return to each time I make a package change, rather than re-learning the steps every time.
I wanted to understand how devtools, testthat, and roxygen2 fit together in a single coherent workflow.

Objectives

Create a feature branch, make code changes, and update roxygen2 documentation in an R package.
Write and run unit tests locally with testthat and build the package with devtools.
Set up three GitHub Actions workflows (R-CMD-check, test coverage, pkgdown) and understand what each one does.
Execute the full pull request lifecycle: push, review, address CI failures, merge, and clean up.

I am documenting my learning process here. Errors and better approaches are welcome; see the contact section.

Settling in for a focused development session.

Settling in for a focused development session.

Prerequisites and Setup

This workflow assumes familiarity with R, basic Git commands, and a GitHub account. The following tools should be installed:

install.packages(
  c("devtools", "testthat", "roxygen2",
    "rcmdcheck", "covr", "pkgdown")
)

Git should also be configured on the system with SSH or HTTPS access to GitHub repositories. The examples below use a package called zzdataframe2graphic, but the process applies to any R package hosted on GitHub.

What is an R Package Development Workflow?

An R package development workflow is a structured sequence of steps that takes a code change from an idea to a tested, documented, and merged contribution. Think of it as a checklist for responsible software modification: rather than editing a file and hoping for the best, one creates an isolated branch, updates documentation automatically, runs tests on multiple platforms, and merges only when everything passes.

The key tools in this workflow are devtools (which orchestrates building, testing, and checking), roxygen2 (which generates documentation from inline comments), testthat (which provides a structured testing framework), and GitHub Actions (which runs automated checks on every push). Together, they form a safety net that catches problems before they reach production.

Getting Started: Creating a Feature Branch

Every change begins with a new branch. This keeps the main branch stable during development.

git checkout main
git pull origin main
git checkout -b feature-name

The first two commands ensure work starts from the latest version of the codebase. The third creates a new branch and switches to it. All subsequent changes happen on this branch, isolated from main until the work is ready to merge.

Editing the R Code

With the branch created, navigate to the relevant source file and make your changes. For an R package, the source files live in the R/ directory:

Open the target file (e.g., R/zzdataframe2graphic.R).
Make the code changes.
Update or add roxygen2 comments above any modified or new functions.
Save the file.

After editing, regenerate the documentation:

devtools::document()

This command reads the roxygen2 comments and updates the NAMESPACE file and the man/ directory. It is important to run this every time function signatures change, parameters are added, or exported functions are modified.

Code review on a laptop screen

Taking a step back to review the changes before testing.

Writing and Running Tests

Every code change should be accompanied by tests. The testthat framework provides a clean structure for this:

test_that("new feature works as expected", {
  result <- your_function(test_input)
  expect_equal(result, expected_output)
})

Add test cases to the appropriate file in tests/testthat/. Then run the full test suite locally:

devtools::test()

Fix any failures before proceeding. If the change is substantial, consider adding multiple test cases covering edge cases and expected error conditions.

Building and Checking the Package

Once tests pass, build and check the package:

devtools::build()
devtools::check()

The check() function runs R CMD check, which is the standard quality gate for R packages. It verifies that documentation is complete, examples run without error, tests pass, and the package can be installed cleanly.

For more rigorous checking, run against additional platforms:

devtools::check_win_devel()
devtools::check_mac_release()
rcmdcheck::rcmdcheck(args = c("--as-cran"))

The --as-cran flag applies the stricter checks that CRAN uses when evaluating package submissions.

Updating the DESCRIPTION File

Before committing, update the package metadata:

Increment the version number (e.g., from 0.2.0 to 0.2.1).
Update the Date field.
Add any new dependencies to Imports or Suggests.

The Git and GitHub Workflow

With all local checks passing, it is time to commit, push, and open a pull request.

Staging and Committing

git status
git add R/zzdataframe2graphic.R
git add tests/testthat/test-zzdataframe2graphic.R
git add DESCRIPTION
git add man/*
git add NAMESPACE

Review the staged changes with git status, then commit with a descriptive message:

git commit -m "feat: add error handling to
  zzdataframe2graphic

- Added input validation for data frame columns
- Updated roxygen2 documentation
- Added unit tests for edge cases"

Pushing and Creating a Pull Request

git push origin feature-name

Then create a pull request on GitHub:

Navigate to the repository on GitHub.
Click the “Pull requests” tab.
Click “New pull request.”
Set base to main and compare to feature-name.
Fill in the PR description: what changed, why, how to test, and any related issues.

GitHub Actions: Automated CI/CD

GitHub Actions automate the testing process on every push or pull request. A typical R package uses three workflows:

R-CMD-check runs R CMD check across multiple operating systems:

Windows (latest)
macOS (latest)
Ubuntu (latest, with R-release, R-devel, and R-oldrel)

This workflow triggers on pushes and pull requests to the main branch.

Test coverage runs all package tests, generates coverage reports, and identifies which parts of the code need additional testing.

pkgdown builds the package documentation website and deploys it to GitHub Pages whenever changes are pushed to the main branch.

Setting Up GitHub Actions

To add these workflows to a package:

mkdir -p .github/workflows

Then add three YAML configuration files:

.github/workflows/R-CMD-check.yaml
.github/workflows/test-coverage.yaml
.github/workflows/pkgdown.yaml

Configure repository permissions:

Go to Settings, then Actions, then General.
Set “Workflow permissions” to “Read and write.”
Enable GitHub Pages deployment.

The r-lib/actions repository on GitHub provides standard workflow templates for all three.

Addressing CI Failures

If any workflow fails after pushing:

Check the “Actions” tab on GitHub.
Review the failure logs for each workflow.
Fix the issues locally.
Push the fixes:

git add .
git commit -m "fix: address CI failures"
git push origin feature-name

The pull request will automatically re-run the workflows.

Merging and Cleaning Up

Once all checks pass and the PR is approved:

Choose a merge strategy (usually “Squash and merge”).
Update the PR title if needed.
Click “Squash and merge.”

Then clean up locally:

git checkout main
git pull origin main
git branch -d feature-name

Creating a Release (Optional)

For significant changes, create a GitHub release:

Navigate to the “Releases” section.
Click “Create new release.”
Tag the version (e.g., v0.2.1).
Add release notes summarizing the changes.
Publish the release.

Verification

After completing the branch-edit-test cycle, confirm that each stage of the pipeline passes.

devtools::document()
devtools::test()

devtools::check(args = c("--as-cran"))

git log --oneline -5

If document() runs without warnings, test() reports no failures, and check() returns 0 errors, 0 warnings, 0 notes, the package is ready to push.

Daily Workflow

Task	Command
Create feature branch	`git checkout -b feature-name`
Regenerate docs	`devtools::document()`
Run tests	`devtools::test()`
Build package	`devtools::build()`
Full check	`devtools::check(args = c("--as-cran"))`
Push branch	`git push origin feature-name`
Clean up after merge	`git checkout main && git pull && git branch -d feature-name`

Things to Watch Out For

Run devtools::document() before committing. I have forgotten this step more than once, leading to CI failures because the NAMESPACE file was out of date.
Stage files explicitly rather than using git add . The R build process creates temporary files that should not be committed. Staging specific files avoids accidental inclusion of build artifacts.
Check on multiple platforms. A package that passes on macOS may fail on Windows due to path separators or system dependency differences. The multi-platform R-CMD-check workflow catches these issues.
Keep commits focused. A PR that changes one function, updates its documentation, and adds its tests is easy to review. A PR that touches fifteen files across unrelated features is difficult to evaluate.
Address CI failures immediately. Letting failures accumulate makes debugging harder. Fix each failure as it appears and push the fix before moving on.

Uninstall / Rollback

To revert a package change that has not yet been merged, delete the feature branch locally and on the remote.

git checkout main
git branch -D feature-name
git push origin --delete feature-name

If the change has already been merged, revert the merge commit on main:

git revert <merge-commit-hash>
git push origin main

To remove the development toolchain entirely:

remove.packages(c(
  "devtools", "testthat", "roxygen2",
  "rcmdcheck", "covr", "pkgdown"
))

Books and notes on a research desk

Reflecting on the lessons from a full development cycle.

What Did We Learn?

Lessons Learnt

Conceptual Understanding:

The R package structure is not just an organizational convenience; it is a contract that R CMD check enforces across documentation, dependencies, and test coverage.
Feature branches isolate risk. Working directly on main means every mistake is immediately visible to collaborators and CI systems.
Automated CI/CD on three operating systems catches platform-specific issues that local testing on a single machine cannot detect.
The pull request is not just a merge mechanism; it is a documentation artifact that records what changed, why, and what tests confirmed the change works.

Technical Skills:

devtools::document() regenerates NAMESPACE and man/ from roxygen2 comments, eliminating manual documentation maintenance.
devtools::check() with --as-cran applies the strictest quality standards and is worth running before every push.
The r-lib/actions repository provides ready-to-use GitHub Actions YAML templates for R-CMD-check, test coverage, and pkgdown.
rcmdcheck::rcmdcheck() provides more detailed output than devtools::check() and is useful for diagnosing obscure failures.

Gotchas and Pitfalls:

Forgetting to run devtools::document() after changing roxygen2 comments causes NAMESPACE mismatches that fail CI silently.
Using git add . can accidentally stage build artifacts, .Rhistory, or .DS_Store files that do not belong in the repository.
GitHub Actions workflows require “Read and write” permissions under repository settings; the default “Read” permission causes pkgdown deployment to fail without a clear error message.
Test coverage workflows may require LaTeX dependencies for vignette building; missing system dependencies produce cryptic errors that do not mention LaTeX directly.

Limitations

This workflow assumes a single-developer context. Multi-contributor projects require additional conventions around code review, branch protection rules, and merge conflict resolution.
The GitHub Actions templates from r-lib/actions target CRAN-style packages. Packages with non-standard system dependencies may need custom workflow modifications.
Test coverage reports measure which lines of code are executed during tests, not whether the tests are meaningful. High coverage does not guarantee high quality.
The --as-cran check is conservative and may flag issues that are acceptable for internal or non-CRAN packages.
This post does not cover continuous deployment to package repositories (e.g., r-universe or drat) or automated version bumping.

Opportunities for Improvement

Add pre-commit hooks that run devtools::document() and devtools::test() automatically before each commit.
Integrate lintr into the CI pipeline to enforce consistent code style across all contributions.
Set up branch protection rules on main to require passing CI checks before merging.
Add a NEWS.md file to track user-facing changes with each version increment.
Explore usethis::use_github_action() to generate workflow files directly from R rather than copying YAML templates manually.
Consider adding a code coverage badge to the README to make test coverage visible at a glance.

Wrapping Up

The R package development workflow is more involved than simply editing code and pushing to GitHub, but each step exists for a reason. The branching strategy protects the main branch, the documentation tools keep help files synchronized with code, the testing framework catches regressions, and GitHub Actions verify everything works across platforms.

What I learned most from going through this process is that the overhead of a proper workflow pays for itself quickly. The first time CI catches a platform-specific bug that I would never have found on my own machine, the entire setup justified its existence.

For anyone starting out with R package development, my advice is to set up the full workflow once, even for a small package, and then follow it consistently. The steps become automatic after a few iterations.

In conclusion, four points merit emphasis. First, always branch before making changes and keep each branch focused on a single feature or fix. Second, run devtools::document(), devtools::test(), and devtools::check() locally before pushing. Third, set up R-CMD-check, test coverage, and pkgdown GitHub Actions workflows using the templates from r-lib/actions. Fourth, treat the pull request as both a merge mechanism and a documentation record of what changed and why.

Reproducibility

This post describes a workflow rather than an analysis pipeline. To follow along, one will need:

R 4.4 or later
Git 2.30 or later
A GitHub account with repository access
The packages listed in the Prerequisites section

The key commands in sequence:

git checkout -b feature-name
# Edit R source files and roxygen2 comments

devtools::document()
devtools::test()
devtools::build()
devtools::check()

git add <specific-files>
git commit -m "feat: description of change"
git push origin feature-name
# Open PR on GitHub, wait for Actions, merge
git checkout main && git pull origin main
git branch -d feature-name

sessionInfo()

R version 4.5.3 (2026-03-11)
Platform: aarch64-apple-darwin25.3.0
Running under: macOS Tahoe 26.5

Matrix products: default
BLAS:   /opt/homebrew/Cellar/openblas/0.3.32/lib/libopenblasp-r0.3.32.dylib 
LAPACK: /opt/homebrew/Cellar/r/4.5.3/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.1

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Los_Angeles
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] htmlwidgets_1.6.4 compiler_4.5.3    fastmap_1.2.0     cli_3.6.6        
 [5] tools_4.5.3       htmltools_0.5.8.1 parallel_4.5.3    yaml_2.3.10      
 [9] rmarkdown_2.29    knitr_1.50        jsonlite_2.0.0    xfun_0.56        
[13] digest_0.6.37     rlang_1.2.0       evaluate_1.0.5

Let’s Connect

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: rgtlab.org/contact

I would enjoy hearing from you if:

An error or a better approach to any of the code in this post comes to mind.
There are suggestions for topics to see covered.
The interest is in discussing R programming, data science, or reproducible research.
There are questions about anything in this tutorial.
The goal is simply to say hello and connect.

Setting Up Neovim as a Data Science IDE

Ronald G. Thomas — Wed, 11 Feb 2026 08:00:00 GMT

Neovim logo and editor theme in a terminal

Neovim: a modern fork of Vim designed for extensibility through Lua.

Introduction

I did not really know how much a keyboard-centric text editor could reshape the way I write code until I committed to learning Neovim. For years I relied on conventional mouse-driven editors, accepting their speed as a given. The transition was not immediate, but the cumulative effect on daily productivity was substantial.

Neovim is a fork of Vim that retains the modal editing philosophy while adding first-class Lua scripting, an asynchronous plugin architecture, and a built-in terminal emulator. These features make it particularly well suited for data science workflows where one frequently switches between writing code, inspecting output, and editing manuscripts.

We describe a practical Neovim configuration for data science development on macOS. The setup covers plugin management with Lazy, core editor settings, R integration through Nvim-R, and a set of key mappings that streamline the edit-run-inspect cycle. An appendix addresses Ubuntu adjustments and working with multiple Neovim configurations.

More formally, we document the Editor layer (Layer 5) of the Workflow Construct described in post 52. Neovim is the choice of editor at this layer; the layer is substitutable (with Vim, VS Code, RStudio, or a graphical IDE), but the contract the layer holds, modal scriptable text manipulation that behaves identically locally and over SSH, is what the higher layers (Vim plugins, snippets, REPL integration) all assume. The configuration documented here is the editor-layer counterpart to the dotfiles repository (post 24).

Motivations

I was spending too much time reaching for the mouse during analysis sessions and wanted a purely keyboard-driven workflow.
My existing IDE felt sluggish when editing large R scripts and Quarto documents; Neovim starts instantaneously and never lags.
I needed a single editor that could handle R, Python, Julia, and LaTeX without switching tools.
The Nvim-R plugin offered a tighter REPL integration than what I had experienced in other editors.
I wanted my entire editor configuration stored in version-controlled text files that I could replicate on any machine.
Learning Lua as a configuration language felt like a worthwhile investment given its growing adoption in the Neovim ecosystem.

Objectives

Install Neovim and set up convenience shell aliases for terminal and GUI modes.
Build a modular Lua configuration with a clear directory hierarchy under ~/.config/nvim.
Install and configure essential plugins for data science work using the Lazy plugin manager.
Configure Nvim-R for interactive R development with custom key mappings for sending code, inspecting objects, and rendering documents.

I am documenting my learning process here. If you spot errors or have better approaches, please let me know.

Settling in for editor configuration.

The Neovim project logo. Neovim extends Vim with Lua scripting and an asynchronous architecture.

Prerequisites and Setup

This guide assumes a macOS environment with Homebrew installed. The simplest approach is to install both the terminal and GUI versions of Neovim:

brew install neovim neovim-qt

Convenience aliases in .zshrc are recommended:

alias ng="nvim-qt"
alias nt="nvim"

The mnemonic is straightforward: nt for terminal, ng for GUI. Ubuntu Linux users should see the appendix at the end of this post for platform-specific adjustments.

Assumed knowledge. Basic familiarity with the terminal, a general understanding of what a text editor does, and comfort navigating the file system from the command line. No prior Vim experience is required, though it helps considerably.

What is Neovim?

Neovim is a modern reimplementation of Vim, the ubiquitous Unix text editor. Where Vim relies on its own scripting language (Vimscript), Neovim adds first-class support for Lua, a lightweight and fast scripting language. This means configuration files are written in a language that is easier to read, debug, and extend than traditional Vimscript.

The core idea behind modal editing is that the keyboard serves different purposes depending on the current mode. In Normal mode, keys move the cursor and manipulate text. In Insert mode, keys type characters. In Visual mode, keys select regions. This separation eliminates the need for modifier-heavy shortcuts and allows rapid text manipulation once the muscle memory develops.

For data scientists, Neovim offers a compelling combination: a built-in terminal for running REPLs, a rich plugin ecosystem for language-specific tooling, and a configuration system that lives entirely in plain text files under version control.

Directory Structure and Configuration Hierarchy

The standard location for Neovim configuration files on Unix-like systems is ~/.config/nvim. The main entry point is either init.vim (Vimscript) or init.lua (Lua). In this post we use Lua exclusively.

Below is the file hierarchy we construct. All the code in the lua subdirectory could be bundled into init.lua, but separating concerns into individual files is clearer and easier to maintain.

.
|-- ginit.vim
|-- init.lua
|-- lazy-lock.json
|-- lua
|   |-- basics.lua
|   |-- leap-config.lua
|   |-- nvim-R-config.lua
|   |-- nvim-cmp-config.lua
|   |-- nvim-telescope-config.lua
|   |-- nvim-tree-config.lua
|   `-- treesitter-config.lua
|-- my_snippets
|   |-- all.snippets
|   |-- tex
|   |-- R
|   |-- python
|   |-- julia
|   |-- giles.tex.snipppets
|   |-- mail.snippets
|   |-- r.snippets
|   |-- rmd.snippets
|   |-- snippets.snippets
|   |-- tex.snippets
|   |-- text.snippets
|   `-- txt.snippets
|-- spell
|   |-- en.utf-8.add
|   `-- en.utf-8.add.spl

Each .lua file under the lua/ directory is loaded by name from init.lua using Lua’s require() function. The my_snippets/ directory holds UltiSnips snippet files organized by language. The spell/ directory contains custom dictionary additions for Neovim’s built-in spell checker.

Plugin Management with Lazy

Neovim on its own is useful but limited. Plugins extend its capabilities dramatically. To install plugins we need a plugin manager. Several exist; we use Lazy, which is fast, declarative, and written entirely in Lua.

To install Lazy, clone its repository into Neovim’s data directory:

git clone https://github.com/folke/lazy.nvim.git \
  ~/.local/share/nvim/lazy/lazy.nvim

The init.lua Entry Point

The init.lua file is the first file Neovim reads on startup. It sets leader keys, prepends Lazy to the runtime path, and then loads each configuration module in sequence.

vim.g.mapleader = ","
vim.g.maplocalleader = " "
vim.opt.rtp:prepend(
  "~/.local/share/nvim/lazy/lazy.nvim"
)
require('plugins')
require('basics')
require('nvim-tree-config')
require('nvim-R-config')
require('nvim-telescope-config')
require('leap').add_default_mappings()
require('leap-config')
require('lualine').setup()

The comma leader key and space local leader are personal preferences. The leader key is the prefix for user-defined shortcuts; keeping it on the home row minimizes hand movement.

Plugin List and Descriptions

The following block is the plugin specification passed to Lazy. Plugins are grouped by purpose: a minimal data science core, optional utilities, and Neovim-specific enhancements.

require('lazy').setup({
  -- minimal data science setup
  'jalvesaq/Nvim-R',
  'lervag/vimtex',
  'SirVer/ultisnips',
  'jalvesaq/vimcmdline',
  -- optional utilities
  "nvim-lualine/lualine.nvim",
  "bluz71/vim-moonfly-colors",
  'junegunn/vim-peekaboo',
  'tpope/vim-commentary',
  'francoiscabrol/ranger.vim',
  'machakann/vim-highlightedyank',
  'tpope/vim-surround',
  'ggandor/leap.nvim',
  -- neovim specific
  'nvim-lua/plenary.nvim',
  'nvim-tree/nvim-web-devicons',
  'nvim-tree/nvim-tree.lua',
  'nvim-telescope/telescope.nvim',
  'nvim-treesitter/nvim-treesitter',
  'neovim/nvim-lspconfig',
})

Minimal data science setup. Nvim-R provides a full R development environment with REPL integration. Vimtex handles LaTeX editing and compilation. UltiSnips manages code snippets for rapid boilerplate insertion. Vimcmdline extends REPL support to Python and Julia.

Optional utilities. Lualine provides a status bar. Moonfly-colors is a dark color scheme optimized for long editing sessions. Peekaboo previews register contents. Commentary toggles comments. Ranger integrates a file manager. Highlighted-yank provides visual feedback on yanked text. Surround manipulates paired delimiters. Leap enables rapid cursor movement by two-character search.

Neovim specific. Plenary provides shared Lua utilities. Web-devicons and nvim-tree deliver a file explorer with icons. Telescope is a fuzzy finder for files, buffers, and grep results. Treesitter enables syntax-aware highlighting and code navigation. Lspconfig connects to language servers for autocompletion and diagnostics.

Setup Basics

The basics.lua file contains core editor settings and key mappings. These settings apply globally regardless of file type.

local map = vim.keymap.set
local opts = {noremap = true}
vim.cmd([[
"    paste registers into terminal
tnoremap   \
  '"'.nr2char(getchar()).'pi'
set background=dark
colorscheme moonfly
let $FZF_DEFAULT_COMMAND = 'rg --files --hidden'
set completeopt=menu,menuone,noinsert,noselect
set number relativenumber
set textwidth=80
set cursorline
set clipboard=unnamed
set iskeyword-=_
set hlsearch
set splitright
set hidden
set incsearch
set noswapfile
set showmatch
set ignorecase
set smartcase
set gdefault
filetype plugin on
set encoding=utf-8
set nobackup
set nowritebackup
set updatetime=300
set signcolumn=yes
set colorcolumn=80
set timeoutlen=1000 ttimeoutlen=10
let g:UltiSnipsSnippetDirectories = \
  ['~/.config/nvim/my_snippets']
let g:UltiSnipsExpandTrigger=""
let g:UltiSnipsJumpForwardTrigger=""
let g:UltiSnipsJumpBackwardTrigger=""
nnoremap U \
  call UltiSnips#RefreshSnippets()
autocmd BufWinEnter,WinEnter term://* startinsert
]])

A few settings worth highlighting: relativenumber shows line distances from the cursor, which makes vertical motions (e.g., 12j to jump 12 lines down) intuitive. colorcolumn=80 draws a vertical guide at 80 characters to encourage readable line lengths. clipboard=unnamed connects Neovim’s yank register to the system clipboard. splitright opens vertical splits to the right, matching the natural left-to-right reading direction.

Key Mappings

The following mappings are defined in basics.lua after the vim.cmd block. They use the Lua vim.keymap.set API.

map('n', ':', ';', opts)
map('n', ';', ':', opts)
map('n', 'u',
  ':UltiSnipsEdit', opts)
map('n', 'U',
  'call UltiSnips#RefreshSnippets()',
  opts)
map('n', '',
  '', opts)
map('n', '-', '$', opts)
map('n', 'w', 'vipgq', opts)
map('n', 'v',
  ':edit ~/.config/nvim/init.lua', opts)
map('n', 'n',
  ':edit ~/.config/nvim/lua/basics.lua',
  opts)
map('n', 'a', 'ggVG', opts)
map('n', 't', ':tab split', opts)
map('n', 'y', ':vert sb3', opts)
map('n', '0',
  ':ls!:b', opts)
map('n', '', 'w', opts)
map('n', '1', ':b1', opts)
map('n', '2', ':b2', opts)
map('n', '3', ':b3', opts)
map('t', 'ZZ', "q('yes')", opts)
map('t', 'ZQ', "q('no')", opts)
map('v', '-', '$', opts)
map('t', '0',
  '\\>:ls!:b', opts)
map('t', '', '\\>', opts)
map('t', ',,',
  '\\>w', opts)
map('i', '', '`^', opts)

The semicolon and colon swap (':' to ';' and vice versa) deserves special mention. In Vim, the colon enters command mode (one of the most frequent actions). Swapping it to the semicolon key eliminates the need to hold Shift, saving thousands of keystrokes over time.

Soft mood lighting over a coding workspace

A well-configured development environment reduces cognitive overhead and allows sustained focus on the analytical task.

R Development Setup with Nvim-R

The nvim-R-config.lua file configures the Nvim-R plugin, which transforms Neovim into a capable R IDE with REPL integration, object inspection, and document rendering.

vim.cmd([[
iabb  x %>%
iabb  z %in%
let R_auto_start = 2
let R_enable_comment = 1
let R_hl_term = 0
let R_clear_line = 1
let R_pdfviewer = "zathura"
let R_assign = 2
let R_latexcmd = ['xelatex']
augroup rmarkdown
autocmd!
autocmd FileType rmd,r nnoremap  \
   :call SendLineToR("down")
autocmd FileType rmd,r nnoremap  \
  ' :call RMakeRmd("default")
autocmd FileType rmd,r noremap \
  i :call RAction("dim")
autocmd FileType rmd,r noremap \
  h :call RAction("head")
autocmd FileType rmd,r noremap \
  p :call RAction("print")
autocmd FileType rmd,r noremap \
  q :call RAction("length")
autocmd FileType rmd,r noremap \
  n :call RAction("nvim.names")
autocmd FileType rmd,r vmap  \
   sd
autocmd FileType rmd,r nmap  \
  j gn
autocmd FileType rmd,r nmap  \
  k gN
autocmd FileType rmd,r nmap  \
  l cd
augroup END
]])

Key configuration choices. R_auto_start = 2 opens an R console automatically when an R file is loaded. R_assign = 2 converts two underscores into the <- assignment operator. The iabb abbreviations expand x to %>% and z to %in% in insert mode, reducing keystrokes for common pipe and membership operators.

Custom autocmds. The augroup rmarkdown block defines file-type-specific mappings for R and Rmd files. Pressing Enter sends the current line to the R console and moves down. Space-prefixed mappings provide quick object inspection: i for dimensions, h for head, p for print. This tight coupling between editor and REPL is what makes Nvim-R particularly effective for interactive data analysis.

Ubuntu Tweaks and Multiple Configurations

On Ubuntu, Neovim is available through the system package manager, though the version may lag behind the latest release. For the most current version, use the Neovim PPA or download the AppImage directly from the Neovim releases page.

sudo add-apt-repository ppa:neovim-ppa/unstable
sudo apt update
sudo apt install neovim

Working with Multiple Configurations

Neovim supports the NVIM_APPNAME environment variable, which allows running entirely separate configurations side by side. This is useful for testing new plugin setups without disturbing a working configuration.

For a thorough walkthrough, see Switching Configs in Neovim by Michael Uloth.

To start Neovim with an alternative configuration stored in ~/.config/test_nvim:

NVIM_APPNAME=test_nvim nvim

This tells Neovim to read its configuration from ~/.config/test_nvim/init.lua instead of the default ~/.config/nvim/init.lua. Each configuration maintains its own plugin state, cache, and data directories.

Verification

After installing Neovim and copying the configuration:

# 1. Version check
nvim --version | head -1

# 2. Plugin status (inside Neovim)
# :Lazy          — verify all plugins installed
# :checkhealth   — diagnose provider issues

# 3. R integration smoke test
# Open an .R file:
nvim test.R
# Press \rf to start R console
# Press \d to send the current line to R

If :checkhealth shows red for clipboard, install xclip (brew install xclip or sudo apt install xclip).

Daily Workflow

Keybinding / Command	Action
`\rf`	Start R console
`\d`	Send current line to R
`\ss`	Send selection to R
`\ro`	View R object in pager
	Fuzzy-find files (Telescope)
	Toggle file tree
`:Lazy update`	Update all plugins
`:Lazy restore`	Restore pinned versions
`NVIM_APPNAME=test nvim`	Launch alternate config

Things to Watch Out For

Plugin version conflicts. Lazy locks plugin versions in lazy-lock.json. If a plugin update breaks something, restore the lock file from version control and run :Lazy restore.
Clipboard integration. On headless Linux servers, the clipboard=unnamed setting requires xclip or xsel to be installed. Without these, yanking to the system clipboard silently fails.
Terminal mode escape. The default escape sequence in Neovim’s built-in terminal is , which is awkward. The mapping map('t', '', '', opts) in our configuration fixes this, but it means a literal Escape cannot be typed in terminal mode without an alternative binding.
R console startup. With R_auto_start = 2, Nvim-R opens the R console automatically. On slower machines or when R has a heavy .Rprofile, this can cause a noticeable delay. Set R_auto_start = 0 to start R manually with \rf.
Snippet directory paths. UltiSnips expects snippet directories to be on Neovim’s runtime path. If snippets are not expanding, verify the path in g:UltiSnipsSnippetDirectories matches the actual directory location.

Uninstall / Rollback

To remove Neovim and its configuration:

# 1. Remove configuration
rm -ri ~/.config/nvim
rm -ri ~/.local/share/nvim   # plugin data
rm -ri ~/.cache/nvim          # cache

# 2. Uninstall Neovim
brew uninstall neovim          # macOS
sudo apt remove neovim         # Ubuntu / Debian

To keep Neovim but reset to defaults, rename the config:

mv ~/.config/nvim ~/.config/nvim.backup

Neovim will start with no plugins and default settings.

UCSD Geisel Library at dusk

The pursuit of an efficient development environment parallels the broader scholarly commitment to refining one’s tools and methods.

What Did We Learn?

Lessons Learnt

Conceptual Understanding:

Modal editing is not merely a different interface; it represents a fundamentally different model of text manipulation where composable commands replace mouse-driven selection.
The separation of configuration into modular Lua files mirrors the separation-of-concerns principle in software engineering and makes each component independently testable.
Neovim’s asynchronous architecture means plugins run without blocking the editor, which is why operations like fuzzy search and tree-sitter parsing feel instantaneous even on large files.
The REPL-integrated workflow (edit code, send to console, inspect output) collapses the feedback loop in interactive data analysis to a single keystroke.

Technical Skills:

Writing Lua configuration files from scratch, including understanding the vim.cmd, vim.keymap.set, and vim.opt APIs.
Installing and managing plugins through the Lazy plugin manager, including reading lazy-lock.json for reproducible plugin states.
Configuring Nvim-R for interactive R development with custom autocmds, abbreviations, and file-type-specific key mappings.
Setting up UltiSnips with language-specific snippet directories for rapid code template insertion.

Gotchas and Pitfalls:

Forgetting to set noremap = true on key mappings can cause recursive mapping chains that freeze the editor or produce unexpected behavior.
The iskeyword-=_ setting (which treats underscores as word boundaries) improves snake_case navigation but breaks word completion for identifiers containing underscores.
Neovim’s built-in terminal uses terminal mode, not normal mode. Mappings defined for normal mode do not apply until one escapes back to normal mode with the configured escape binding.
The gdefault setting applies substitutions globally by default (no need for the /g flag), which is convenient but reverses the meaning of /g in substitute commands — adding /g now means “first match only.”

Limitations

This configuration targets macOS as the primary platform. While most settings transfer directly to Linux, clipboard integration, font rendering, and GUI behavior may require platform-specific adjustments.
The plugin selection reflects one practitioner’s workflow. Users working primarily in Python or Julia may need different language server configurations and REPL integrations.
Nvim-R connects to a single R session. Users who need multiple concurrent R sessions or remote R connections may find this limiting compared to RStudio’s session management.
The configuration does not include a debugger setup. Interactive debugging in R and Python requires additional plugins (nvim-dap) and configuration not covered here.
Snippet management through UltiSnips requires learning a domain-specific snippet syntax. Users unfamiliar with snippet engines face an additional learning curve.
This post does not address Neovim’s built-in LSP (Language Server Protocol) configuration in depth. Full autocompletion and diagnostics require language-server setup that varies by language.

Opportunities for Improvement

Add LSP configuration for R (languageserver), Python (pyright), and Julia (LanguageServer.jl) to enable autocompletion, go-to-definition, and inline diagnostics.
Replace UltiSnips with LuaSnip, a pure-Lua snippet engine that integrates more naturally with Neovim’s Lua ecosystem and does not require Python.
Configure nvim-cmp for intelligent autocompletion that combines LSP suggestions, buffer words, and snippet expansions in a unified menu.
Add a DAP (Debug Adapter Protocol) configuration for step-through debugging of R and Python scripts directly within Neovim.
Create a custom Quarto filetype plugin with mappings for rendering, previewing, and navigating between code chunks in .qmd files.
Explore the Which-Key plugin to provide a discoverable popup menu of available key mappings, reducing the memorization burden for new users.

Wrapping Up

Configuring Neovim for data science is an investment that pays dividends over time. The initial learning curve is real — modal editing requires deliberate practice before it becomes natural. However, once the muscle memory develops, the speed and precision of keyboard-driven editing fundamentally changes the coding experience.

The configuration presented here provides a functional starting point: Lazy manages plugins, basics.lua establishes sensible defaults, and Nvim-R connects the editor to an interactive R console. The modular structure means each component can be modified independently as the workflow evolves.

In conclusion, four points merit emphasis. First, Neovim’s Lua-based configuration is readable, version-controllable, and modular, which makes it practical to maintain across machines. Second, the Lazy plugin manager provides reproducible plugin states through its lock file, enabling consistent environments after cloning. Third, Nvim-R transforms Neovim into a capable R IDE with single-keystroke code execution and object inspection. Fourth, investing time in key mappings and snippets yields compounding productivity gains across every editing session.

Anyone considering the switch should start with a minimal configuration and add plugins gradually. The Neovim ecosystem is large, and configuring everything at once is overwhelming. Actual workflow demands should dictate what to add next.

Reproducibility

This post describes a configuration-only workflow with no data analysis pipeline. The configuration files reside in ~/.config/nvim/ and are managed through version control.

Software versions used:

nvim --version | head -1
brew list --versions neovim

Configuration files described in this post:

~/.config/nvim/init.lua: Entry point, leader keys, module loading
~/.config/nvim/lua/basics.lua: Core settings and key mappings
~/.config/nvim/lua/nvim-R-config.lua: R development configuration
~/.config/nvim/lua/plugins.lua: Lazy plugin specification
~/.config/nvim/my_snippets/: UltiSnips snippet files by language

To replicate this setup, clone the dotfiles repository (or copy the files above) into ~/.config/nvim/, then open Neovim. Lazy will automatically install all specified plugins on first launch.

Let’s Connect

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: rgtlab.org/contact

I would enjoy hearing from you if:

You spot an error or a better approach to any of the code in this post.
You have suggestions for topics you would like to see covered.
You want to discuss R programming, data science, or reproducible research.
You have questions about anything in this tutorial.
You just want to say hello and connect.

Reproducible Blog Posts with ZZCOLLAB: A Quarto Workflow

Ronald G. Thomas — Tue, 10 Feb 2026 08:00:00 GMT

A Quarto-themed workspace representing reproducible document authoring

Quarto provides the rendering engine; ZZCOLLAB provides the reproducible scaffolding around it.

Introduction

I did not really know how much infrastructure a blog post needed until I returned to one of my own R-based posts six months after writing it and watched it fail to render. A package had updated, a system library had shifted, and the code that once produced clean diagnostic plots now threw cryptic errors. The rendered HTML was frozen in time, but the source code was dead.

Technical blog posts that embed R code face a well-documented fragility problem. Package updates introduce breaking changes, data sources move or disappear, and system-level dependencies shift beneath the analysis. A post that rendered correctly in 2024 may fail silently in 2025, producing different results or refusing to compile entirely.

The conventional response is to freeze rendered output and accept that the source code is a snapshot rather than a living document. We describe an alternative: treating each blog post as a standalone ZZCOLLAB reproducible research project, complete with its own Docker environment, pinned package versions, and continuous integration pipeline.

Motivations

I was frustrated that returning to a six-month-old post required hours of debugging broken package dependencies before I could even render it again.
I wanted each blog post to be independently clonable so that a collaborator could reproduce the results without asking me what versions of R and ggplot2 I originally used.
I needed a workflow that scaled to 40+ posts without requiring a monorepo that coupled unrelated analyses together.
I was already using ZZCOLLAB for research projects and wanted a consistent mental model across all my computational work, whether a journal article or a tutorial post.
I wanted CI/CD to catch breakage immediately on push rather than discovering it months later when a reader reports a dead link or a wrong figure.

Objectives

Document the lead-in directory pattern that separates git repositories from archival material.
Explain the dual-symlink architecture that reconciles Quarto discovery with rrtools/ZZCOLLAB conventions.
Walk through the steps for creating, initializing, and publishing a new blog post project.
Describe the five-pillar reproducibility framework as applied to the blog post context.

I am documenting my learning process here. If you spot errors or have better approaches, please let me know.

A workspace ready for reproducible publishing.

What is a Reproducible Blog Post?

A reproducible blog post is a technical document whose computational results can be regenerated from source by anyone, on any machine, at any point in the future. It goes beyond simply sharing code alongside prose. Concretely, a reader can clone a single repository, build a Docker container, and produce an identical HTML document without installing anything beyond Docker itself.

The distinction matters because most technical blog posts are reproducible only in theory. They share code snippets but omit the system libraries, the exact package versions, and the data preparation steps that produced the published figures. A truly reproducible blog post packages all of these elements together, much as a research compendium packages the materials needed to reproduce a journal article.

The Problem

Consider a blog post that fits a linear model to the Palmer Penguins dataset and generates diagnostic plots. The post depends on:

A specific R version (4.5.1)
The palmerpenguins package (for data)
ggplot2, broom, patchwork (for analysis)
System libraries for PNG rendering (libpng, cairo)
Quarto itself (for HTML generation)

Any of these may change independently. Without explicit version management, reproducing the post requires reconstructing the original environment through trial and error.

The Solution

Each blog post becomes a self-contained project with five components:

Dockerfile: defines the computational environment
renv.lock: pins exact R package versions
.Rprofile: configures R session behavior
Source code: analysis scripts and narrative
Data: raw and derived datasets

A collaborator (or the author, six months later) can clone the repository, run make docker-build && make r, and reproduce the post in an identical environment.

The Lead-In Pattern

Blog post projects follow the same directory convention used across all research software projects in this lab:

~/Dropbox/prj/qblog/posts/
+-- 29-setupquarto/          # lead-in directory
|   +-- archive/             # old drafts, artifacts
|   +-- setupquarto/         # git repo (the project)
|       +-- .git/
|       +-- Dockerfile
|       +-- index.qmd -> analysis/report/index.qmd
|       +-- ...

The lead-in directory (29-setupquarto/) is not managed by git. It serves as a container for the git repository and an archive/ directory for superseded drafts, exploratory notebooks, and other material that does not belong in version control.

The project directory (setupquarto/) is the git repository. Its name matches the GitHub repository name and omits the numeric prefix.

This pattern mirrors the layout used for research software under ~/prj/sfw/:

~/prj/sfw/07-zzcollab/
|   +-- archive/
|   +-- zzcollab/            # git repo

The numeric prefix provides chronological ordering in file browsers without polluting repository names.

Creating a New Post

Step 1: Create the Lead-In

mkdir -p \
  ~/Dropbox/prj/qblog/posts/43-newpost/archive
cd ~/Dropbox/prj/qblog/posts/43-newpost

Step 2: Initialize the ZZCOLLAB Project

Copy the scaffold from an existing post or use the ZZCOLLAB CLI:

# Option A: Copy from template
cp -r \
  ~/Dropbox/prj/qblog/posts/39-templatepost/\
templatepost \
  ./newpost

# Option B: Use zzcollab CLI
zzc analysis newpost

Both approaches produce the same directory structure.

Step 3: Write the Post

The blog post content lives at analysis/report/index.qmd. A root-level symlink provides Quarto compatibility:

# Already created by scaffold:
# newpost/index.qmd -> analysis/report/index.qmd

Open analysis/report/index.qmd and write. Analysis code that produces figures or derived data belongs in analysis/scripts/. Reusable utility functions belong in R/.

Step 4: Initialize Git and Push

cd newpost
git init -b main
git add .
git commit -m \
  "Initial scaffold for blog post: newpost"
gh repo create rgt47/newpost --public \
    --source=. --remote=origin
git remote set-url origin \
    git@github.com:rgt47/newpost.git
git push -u origin main

Each blog post is its own GitHub repository. Collaborators clone a single post and receive the complete reproducible environment.

Connecting to Quarto

The Quarto site lives at ~/Dropbox/prj/qblog/. It discovers blog posts through a glob pattern in blog/index.qmd:

# blog/index.qmd
listing:
  contents: "../posts/*/*/index.qmd"
  type: default
  sort: "date desc"
  categories: true

The double-wildcard posts/*/*/index.qmd matches the lead-in/project nesting. Quarto resolves the index.qmd symlink transparently and reads the YAML front matter to populate the blog listing.

How Quarto Discovers Posts

The resolution chain for a blog post:

blog/index.qmd
  +-- glob: ../posts/*/*/index.qmd
      +-- matches:
          posts/29-setupquarto/setupquarto/index.qmd
          +-- symlink: analysis/report/index.qmd
              +-- actual file with YAML front matter

Quarto reads the title, date, categories, description, image, and draft fields from the YAML header. Posts with draft: false are excluded from the listing.

Site Configuration

The site-level _quarto.yml defines navigation, theme, and rendering defaults. It does not enumerate posts; the listing glob handles discovery automatically. Adding a new post requires only creating the lead-in directory and project, with no modification to _quarto.yml or any other site-level file.

GitHub Integration

One Repository per Post

Each blog post is a standalone GitHub repository under the rgt47 organization. This design provides:

Independent version history: each post tracks its own commits without polluting a monorepo log
Isolated CI/CD: a failing build in one post does not block others
Focused collaboration: a contributor clones only the post they are reviewing
Independent dependency management: each post pins its own package versions

CI/CD Pipeline

Each repository includes .github/workflows/blog-render.yml, which executes on push:

Build the Docker image from Dockerfile
Restore R packages from renv.lock
Run unit tests (tests/testthat/)
Execute analysis scripts (analysis/scripts/)
Render the Quarto report to HTML
Upload the rendered artifact

The pipeline confirms that the post renders successfully in a clean environment. If a package update breaks the build, the failure is detected immediately rather than months later.

Collaborator Workflow

A collaborator reproducing or reviewing a post:

git clone \
  git@github.com:rgt47/setupquarto.git
cd setupquarto
make docker-build    # build the Docker image
make r               # enter the container
# Inside the container:
Rscript analysis/scripts/01_prepare_data.R
quarto render analysis/report/index.qmd

The Makefile automates common operations. make r mounts the project directory into the container and drops the user into an R-ready shell.

The Five Pillars in Blog Context

ZZCOLLAB’s reproducibility framework rests on five components. Each serves a specific role in the blog post context:

1. Dockerfile: Defines the base R version (rocker/tidyverse:4.5.1), system dependencies, and the analyst user. Ensures the computational environment is identical across machines and CI/CD runners.
2. renv.lock: Pins exact package versions. A minimal lockfile (172 bytes) specifies only the R version and CRAN repository; renv::install() populates it from the DESCRIPTION file on first use. A full lockfile captures the complete dependency tree.
3. .Rprofile: Configures R session behavior. Activates renv, sets repository mirrors, enables auto-snapshot, and detects whether the session is running inside a ZZCOLLAB container.
4. Source code: Analysis scripts in analysis/scripts/ produce figures and derived data. The narrative in analysis/report/index.qmd loads pre-computed results rather than running analysis inline. This separation ensures that rendering the narrative is fast and that analysis code is independently testable.
5. Data: Raw data in analysis/data/raw_data/ is immutable. Derived data in analysis/data/derived_data/ is produced by analysis scripts and may be regenerated. Each dataset includes a README.md documenting its provenance.

Directory Structure Reference

Complete annotated structure of a blog post project:

setupquarto/                     # git repo root
+-- .git/
+-- .github/
|   +-- workflows/
|       +-- blog-render.yml      # CI/CD pipeline
+-- .gitignore
+-- .Rbuildignore
+-- .Rprofile                    # renv activation
+-- .zzcollab/                   # zzcollab metadata
+-- analysis/
|   +-- data/
|   |   +-- raw_data/            # immutable inputs
|   |   +-- derived_data/        # script outputs
|   +-- figures/                 # generated plots
|   +-- media/
|   |   +-- images/              # photos, diagrams
|   +-- report/
|   |   +-- index.qmd            # the blog post
|   |   +-- data -> ../data      # convenience
|   |   +-- figures -> ../figures
|   |   +-- media -> ../media
|   +-- scripts/
|       +-- 01_prepare_data.R
|       +-- 02_fit_models.R
|       +-- 03_generate_figures.R
+-- R/                           # reusable functions
+-- tests/
|   +-- testthat/                # unit tests
|   +-- integration/             # pipeline tests
+-- Dockerfile
+-- Makefile
+-- DESCRIPTION                  # R package metadata
+-- NAMESPACE
+-- renv.lock
+-- renv/
|   +-- activate.R
+-- index.qmd -> analysis/report/index.qmd
+-- data -> analysis/data
+-- figures -> analysis/figures
+-- media -> analysis/media

A quiet study environment for focused work

Reproducible workflows require careful architecture, much like building a well-organized library.

The Symlink Architecture

The project uses two layers of symlinks to reconcile three competing requirements: (1) Quarto expects index.qmd at the directory root it discovers via glob, (2) the rrtools/ZZCOLLAB convention places the narrative at analysis/report/index.qmd, and (3) relative paths in the .qmd file must resolve to analysis/data/, analysis/figures/, and analysis/media/ regardless of which directory Quarto considers the “working directory” during rendering.

Layer 1: Root-Level Symlinks

setupquarto/                  # git repo root
+-- index.qmd -> analysis/report/index.qmd
+-- data -> analysis/data
+-- figures -> analysis/figures
+-- media -> analysis/media

Purpose: Quarto’s blog listing glob (../posts/*/*/index.qmd) matches files at the project root. The index.qmd symlink makes the post discoverable without duplicating the file.

The data/, figures/, and media/ symlinks allow Quarto to resolve relative image and data paths when it renders from the project root. If the .qmd references figures/eda-overview.png, Quarto resolves it through the root-level figures/ symlink to analysis/figures/eda-overview.png.

Git behavior: Git stores symlinks as text files containing the relative target path. On clone, git recreates the symlinks. The git add . command captures symlinks automatically:

$ git ls-files -s index.qmd
120000  0   index.qmd

The 120000 mode indicates a symbolic link.

Layer 2: Report-Level Symlinks

analysis/report/
+-- index.qmd            # the actual blog post
+-- data -> ../data      # resolves to analysis/data
+-- figures -> ../figures # resolves to
|                         # analysis/figures
+-- media -> ../media     # resolves to
                          # analysis/media

Purpose: When an author edits index.qmd inside analysis/report/ and uses a relative path like data/derived_data/results.csv, the data symlink resolves to ../data (i.e., analysis/data). This ensures paths work correctly whether Quarto renders from the project root (following the root-level symlink) or from analysis/report/ directly.

Without these symlinks, an ![](figures/plot.png) reference in index.qmd would resolve differently depending on the rendering context:

From root: setupquarto/figures/plot.png (works via root symlink)
From analysis/report/: analysis/report/figures/plot.png (fails: no such directory)

The report-level symlinks make both contexts resolve to analysis/figures/plot.png.

Layer 3: The Quarto Site Glob

At the site level, Quarto discovers posts through the listing in blog/index.qmd:

listing:
  contents: "../posts/*/*/index.qmd"

The resolution chain:

blog/index.qmd (listing page)
  |
  +-- glob: ../posts/*/*/index.qmd
      |
      +-- posts/29-setupquarto/    <- lead-in
      |   +-- setupquarto/         <- project
      |       +-- index.qmd        <- symlink
      |           +-- target:
      |               analysis/report/index.qmd
      |               +-- YAML: title, date, ...
      |
      +-- posts/08-palmerpenguins.../
          +-- palmerpenguinspart1/
              +-- index.qmd ->
                  analysis/report/index.qmd

A critical constraint discovered during development: Quarto’s listing glob does not follow symlinked directories. If posts/29-setupquarto/ were itself a symlink to an external path, Quarto would skip it entirely. The lead-in directory must be a real directory; only the index.qmd file inside may be a symlink. This is why the project lives inside qblog/posts/ rather than being symlinked from an external location.

Why Not a Single Symlink?

An earlier design placed project repositories outside the Quarto site (at ~/prj/blog/NN-name/name/) and used directory-level symlinks:

# DOES NOT WORK:
# Quarto skips symlinked directories
posts/29-setupquarto ->
  ~/prj/blog/29-setupquarto/setupquarto/

Quarto’s internal file discovery skips symlinked directories during glob matching. Individual posts could still be rendered by explicit path (quarto render posts/29-setupquarto/index.qmd), but the blog listing page would not discover them. This forced the current architecture where project directories reside physically inside the qblog tree.

Image Path Resolution

Images referenced in the .qmd file demonstrate the symlink chain in action. For project-local images stored in analysis/media/images/:

![](media/images/diagram.png)

This resolves through the report-level symlink: media -> ../media -> analysis/media/images/diagram.png.

Creating the Symlinks

The migration script creates both layers:

# Root-level symlinks
ln -s analysis/report/index.qmd index.qmd
ln -s analysis/data data
ln -s analysis/figures figures
ln -s analysis/media media

# Report-level symlinks
cd analysis/report
ln -s ../data data
ln -s ../figures figures
ln -s ../media media

These are relative symlinks, which is critical for portability. An absolute symlink (/Users/zenn/Dropbox/...) would break on any other machine. Relative symlinks survive cloning, moving the project, and CI/CD environments.

Verifying Symlink Integrity

To confirm symlinks are correctly configured:

# From the project root:
ls -la index.qmd data figures media
# Should show -> targets

# From analysis/report/:
ls -la data figures media
# Should show -> ../data, ../figures, ../media

# Verify resolution:
file index.qmd
# Should show: "symbolic link to ..."

# Verify the target exists:
readlink -f index.qmd
# Should show the absolute path to the actual file

Things to Watch Out For

Symlinks on Windows. Windows requires developer mode or administrative privileges to create symbolic links. If collaborators use Windows, document this requirement prominently or provide a setup script that checks permissions.
Dropbox follows symlinks. Dropbox does not sync symlinks as symlinks; it follows them and syncs the target content. The index.qmd symlink at the project root becomes a regular file on other Dropbox-connected machines. Treat git as the canonical source, not Dropbox.
Quarto skips symlinked directories. This was the most time-consuming discovery in the entire project. Directory-level symlinks in posts/ are invisible to the blog listing glob. Only file-level symlinks work.
git config core.symlinks. Some git configurations on Windows disable symlink support. Collaborators may need to run git config core.symlinks true to restore correct behavior after cloning.
Freeze cache and Docker. Quarto’s freeze feature caches rendered output outside the Docker container. If a post executes R code during rendering (rather than loading pre-computed results), the freeze cache may produce inconsistent results between local and CI environments.

Daily Workflow

Command	Action
`make r`	Enter the Docker container
`make docker-build`	Rebuild the Docker image
`make docker-post-render`	Render the post inside Docker
`make check-renv`	Validate renv.lock against code
`quarto render index.qmd --to html`	Render locally (outside Docker)
`quarto preview`	Live preview with auto-reload

Uninstall / Rollback

To remove the zzcollab scaffolding from a post:

# 1. Remove Docker artifacts
docker rmi $(docker images -q <post-image>) 2>/dev/null
rm Dockerfile Makefile

# 2. Remove renv
rm -ri renv/ renv.lock .Rprofile

# 3. Remove symlinks (keeps the target files intact)
rm index.qmd data figures media

# 4. Move index.qmd out of analysis/report/ to the root
cp analysis/report/index.qmd ./index.qmd

The post’s .qmd content is unchanged; only the reproducibility scaffolding is removed.

UCSD Geisel Library, a space for focused research

Academic research and technical writing share a common requirement: disciplined organization of complex materials.

What Did We Learn?

Lessons Learnt

Conceptual Understanding:

Reproducibility for blog posts is not merely about sharing code. It requires packaging the entire computational environment: R version, system libraries, package versions, and data provenance.
The lead-in pattern (numbered directory containing a git repository and an archive folder) provides a clean separation between version-controlled and exploratory material.
Quarto’s glob-based post discovery is powerful but has a critical limitation: it does not follow symlinked directories. Understanding this constraint drove the entire architecture.
Each blog post as its own repository provides independence at the cost of administrative overhead. For a lab producing dozens of posts, this trade-off is worth explicit consideration.

Technical Skills:

Creating and verifying dual-layer relative symlinks that work across rendering contexts (project root vs. analysis/report/).
Configuring Quarto listing globs with the double wildcard pattern (posts/*/*/index.qmd) to match the lead-in/project nesting.
Setting up CI/CD pipelines that build Docker images, restore renv environments, and render Quarto reports in a single workflow.
Using git ls-files -s to verify that git stores symlinks with the 120000 mode and the correct relative target path.

Gotchas and Pitfalls:

Absolute symlinks break portability. Always use relative paths when creating symlinks, even if the absolute path seems more readable during initial setup.
A full site render across 42+ posts is prohibitively slow. In practice, render only the modified post locally and rely on CI/CD for validation.
The renv.lock must be committed to each post repository individually. A shared lockfile across posts defeats the purpose of independent dependency management.
Dropbox silently converts symlinks to regular files on sync. Never rely on Dropbox as the primary synchronization mechanism for projects that use symlinks.

Limitations

Symlink fragility across operating systems. macOS and Linux handle symlinks transparently; Windows requires special configuration. This limits cross-platform collaboration without additional setup documentation.
Repository proliferation. With 42 posts, this workflow creates 42 GitHub repositories. Actions minutes, Dependabot alerts, and repository settings must be managed individually.
Quarto freeze interaction. The freeze cache lives outside the Docker container and may produce inconsistent results if posts execute R code during rendering rather than loading pre-computed outputs.
Dropbox and symlinks are incompatible. Dropbox follows symlinks and syncs target content, making git the only reliable synchronization mechanism.
Rendering cost. A full site render requires entering each post’s Docker container, restoring packages, and executing Quarto. For 42 posts, this is prohibitively slow without selective rendering.
Initial setup overhead. The scaffold, symlinks, CI/CD workflow, and renv initialization require non-trivial effort for each new post, though this is amortized over the post’s lifetime.

Opportunities for Improvement

Automate post scaffolding. A single CLI command (zzc blog 43-newpost) could create the lead-in directory, initialize the ZZCOLLAB project, create both symlink layers, and push an initial commit to GitHub.
Shared Dockerfile caching. Posts that use the same base profile (e.g., ubuntu_x11_analysis) could share a pre-built Docker image from a container registry, eliminating redundant builds.
Selective site rendering. A script that detects which posts have changed since the last render and re-renders only those would make full-site builds feasible.
Symlink validation hook. A pre-commit git hook could verify that all expected symlinks exist and point to valid targets, catching broken links before they reach CI/CD.
Centralized dependency dashboard. A monitoring tool that scans all 42 renv.lock files and flags posts using outdated or vulnerable package versions would simplify maintenance.
Template compliance checker. An automated script that validates YAML front matter fields, symlink integrity, and directory structure against the ZZCOLLAB blog post specification.

Wrapping Up

This post documented a workflow for treating each blog post as a standalone reproducible research project using ZZCOLLAB, Docker, renv, and Quarto. The architecture emerged from a concrete frustration: returning to a post that no longer rendered and having no reliable way to reconstruct the environment that originally produced it.

The workflow is not without costs. Each post generates its own GitHub repository, requires its own Docker image, and demands its own CI/CD pipeline. The initial setup overhead is real. But the payoff is substantial: any post can be cloned and reproduced by anyone, at any time, without guessing at package versions or system dependencies.

In conclusion, five points merit emphasis. First, the lead-in pattern (NN-name/name/) cleanly separates git repositories from archival material. Second, dual-layer relative symlinks reconcile Quarto’s discovery mechanism with the rrtools/ZZCOLLAB directory convention. Third, Quarto’s listing glob does not follow symlinked directories: this constraint drove the entire architecture. Fourth, the five pillars (Dockerfile, renv.lock, .Rprofile, source code, data) provide a complete reproducibility contract for each post. Fifth, CI/CD catches breakage on push rather than months later when a reader reports a problem.

Appendix: rrtools as the Predecessor Pattern

ZZCOLLAB did not appear in a vacuum. The compendium pattern documented above is a refinement of the rrtools package introduced by Marwick, Boettiger, and Mullen (2018), which in turn operationalised the ‘research compendium’ construct introduced by Gentleman and Temple Lang (2007). This appendix preserves the rrtools framing for readers who encounter the older pattern in the literature or in existing project repositories, and clarifies what zzcollab adds.

The problem rrtools (and zzcollab) tried to solve

When sharing R code with a collaborator, several predictable failure modes arise: different R versions on each machine; mismatched package versions; missing system dependencies (pandoc, LaTeX, image libraries); missing supplementary files referenced by the analysis (bibliography files, LaTeX preambles, datasets, images); and collaborator-specific R startup configurations (.Rprofile, .Renviron).

A real-world scenario unfolds like this:

The author emails an R Markdown file to a colleague, Joe.
Joe attempts to run it with R -e "source('peng1.Rmd')".
R is not installed on Joe’s system.
After installing R, Joe gets an error: ‘could not find function render’.
Joe installs the rmarkdown package.
Now pandoc is missing.
After installing pandoc, a required package is missing.
After installing the package, supplementary files are missing (bibliography, images).
The cycle continues until both parties give up or one party invests an afternoon in environment debugging.

The rrtools framework addressed this by defining a fixed research-compendium directory layout (R/, data/, vignettes/, analysis/) and pairing it with a Dockerfile that pinned the R version, the system libraries, and the package versions. The compendium directory was itself an R package, which gave it a DESCRIPTION file as a canonical manifest of dependencies.

What zzcollab adds

The zzcollab framework retains rrtools’s directory layout and Dockerfile-plus-DESCRIPTION pattern, and extends it on three axes:

Profile-based base images. rrtools assumed each project would author its own Dockerfile from rocker/r-ver or similar. zzcollab provides named profiles (minimal, analysis, modeling, publishing, shiny) with pre-built base images, reducing the per-project Docker work to selecting a profile.
Renv integration as a first-class layer. The original rrtools approach used DESCRIPTION for dependency declaration; zzcollab additionally pins exact package versions via renv.lock. This is a stricter reproducibility contract.
Make-driven workflow. zzcollab projects ship a Makefile that exposes make r (enter the container), make check-renv (validate package state), make test, and make docker-render-qmd as the canonical entry points, rather than requiring the analyst to remember per-project Docker commands.

For readers maintaining an existing rrtools project, the migration to zzcollab is mechanical (copy the project’s R/, analysis/, data/ into a new zzcollab scaffold) and reversible. The two patterns coexist: a project on rrtools still satisfies the compendium-tier requirements of the Workflow Construct described in post 52; zzcollab is the construct’s recommended implementation of that tier in 2026, but rrtools remains a valid alternative.

Reproducibility

Environment: R 4.5.1, Quarto 1.6+, Docker (rocker/tidyverse:4.5.1)

Project template: posts/39-templatepost/

Site configuration: _quarto.yml at repository root

Migration script: migrate_posts.sh at repository root (converts flat post directories to lead-in/project structure)

sessionInfo()

R version 4.5.3 (2026-03-11)
Platform: aarch64-apple-darwin25.3.0
Running under: macOS Tahoe 26.5

Matrix products: default
BLAS:   /opt/homebrew/Cellar/openblas/0.3.32/lib/libopenblasp-r0.3.32.dylib 
LAPACK: /opt/homebrew/Cellar/r/4.5.3/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.1

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Los_Angeles
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] htmlwidgets_1.6.4 compiler_4.5.3    fastmap_1.2.0     cli_3.6.6        
 [5] tools_4.5.3       htmltools_0.5.8.1 parallel_4.5.3    yaml_2.3.10      
 [9] rmarkdown_2.29    knitr_1.50        jsonlite_2.0.0    xfun_0.56        
[13] digest_0.6.37     rlang_1.2.0       evaluate_1.0.5

Let’s Connect

GitHub: rgt47
Twitter/X: @rgt47
LinkedIn: Ronald Glenn Thomas
Email: rgtlab.org/contact

I would enjoy hearing from you if:

You spot an error or a better approach to any of the code in this post.
You have suggestions for topics you would like to see covered.
You want to discuss R programming, data science, or reproducible research.
You have questions about anything in this tutorial.
You just want to say hello and connect.

rgtlab — Ronald G. Thomas

Functional Plot Generation with purrr

Introduction

Motivations

Objectives

Prerequisites and Setup

What is Functional Plot Generation?

Getting Started: Preparing the Data

Building the Plotting Function

Mapping Across Species and Variable Pairs

Assembling the Grid with Patchwork

Extracting Individual Plots

Defining the Layout

Composing the Final Figure

Things to Watch Out For

What Did We Learn?

Lessons Learnt

Limitations

Opportunities for Improvement

Wrapping Up

See Also

Reproducibility

Let’s Connect

Related posts in this cluster

Setting Up Multi-Language Quarto Documents on macOS

Introduction

Motivations

Objectives

Prerequisites and Setup

What is Multi-Language Quarto?

Getting Started: Basic Setup

Stage 1: R and RStudio

Stage 2: Python

Stage 3: Julia

Stage 4: Observable JS

Verifying Each Language

R Environment Check

Python Environment Check

Julia Environment Check

Testing a Minimal Example

Rapid Conversion of Draft R Scripts to Formal Rmd Reports

Introduction

Motivations

Objectives

Prerequisites and Setup

What is Script-to-Report Conversion?

Getting Started: The Problem with Raw Scripts

Approach 1: knitr::spin() (The Fastest Path)

How spin() Annotation Works

Annotating an Existing Script

Running spin()

Approach 2: Manual Conversion

The Conversion Steps

A Minimal Template for Manual Conversion

Install Linux Mint on a MacBook Air

Introduction

Motivations

Objectives

Prerequisites and Setup

What is Linux Mint?

Getting Started: Installation

Download and Verify the ISO

Burn the ISO to USB

Boot from USB and Install

Installation Dialog

Post-Install Connectivity

Configuring the Desktop

Keyboard and Trackpad

Display Settings

Power Management

Installing the Software Stack

Copy Configuration Files from Host

Install Core Utilities

Set Up the Shell

Set Up Symbolic Links

Install Additional Applications

Install Dropbox

Install Docker

Checking Our Work

Troubleshooting the Rendering Pipeline

Approach 1: `knitr::spin()` (The Fastest Path)