Designing research tooling for complexity and flexibility

Hybrid Cohort Selection

Role: Sole designer on product team (3 engineers, 1 PM); partnered closely with Flatiron R team
Timeline: Apr-July 2022 (discovery and first release); Sep-Dec 2023 (multi-arm)
Users: ~20 Research Scientists, now 25 Data Analysts
Contributions: Discovery research, internal smoke / landing page testing, complex diagram setup and visual design

Outcome

Project teams need to find specific patients (a cohort) from a database of millions. We redesigned fragmented cohort selection to make it a single-user workflow in a clear user interface. This Hybrid Cohort Selection eliminated 2 weeks of alignment per project with 100% adoption, supporting complex use cases that represent 40% of deliveries today up from 10% of projects in 2023.

Two employees demoing HCS at company All-Hands. — Directors of Research Sciences and Data Science demoing HCS at all-hands.

Context

Projects begin in a scoping phase, where a research scientist (RS) builds out the filtering steps to get to the final cohort of interest. They work off their own snapshot of the patient db + their own templated and custom R code. The actual delivery pipelines have to be configured in Python by an engineer on a slightly different database who recreates the scoping work to the best of their ability. Cohort selection had to happen twice and then RS and engineers have to align and understand why patient counts & individuals may not be the same. Projects would block off two weeks for cohort alignment before kicking off in earnest.

Discovery — As part of a larger discovery on patient identification, we prioritized alignment as the most frequent, wasteful process

Marketing / Comprehension

The team collaborated with our platforms team, who were working to create workflow packages that allowed R users to directly view and manipulate prod data (via Workbench + Posit Connect to S3). While we saw a workflow benefit to RS directly interacting with production dbs to eliminate alignment, we also debated if it was worth the increased complexity and infrastructure changes. I said we should bring it the users and let them react to a future role. We watched a couple live reactions to our “you could own cohort selection” landing page explainer and sent it to the wider Research organization.

Feedback was positive though users had several follow-ups, especially around ambiguous language and concerns certain project types wouldn’t be supported. This technique helped us recruit a testing group, prioritize risks to investigate, and pick ubiquitous language early on.

Updating the UI

We originally wanted cohort selection to be more tightly coupled with RS R work with more in-app editing. But projects at Flatiron were evolving faster than the tech could support it; as we were building Cohort Selection, the eventual end user kept changing at a BU level. While we anchored to RS first, my designs reflect the uncertainty of full ownership. It was possible setup of standard and custom variables would still be split up, and it was important to visually create separate running spaces for both. The new Hybrid Cohort Selection was designed for change, with an emphasis on quickly spinning up projects, and pulling functional R code from S3 regardless of its original source.

A view of cohort selection with editable R code within the app. — While we were excited to integrate the entirety of cohort selection into our app, our users didn't perceive their R code so segmented, and this exposed us to organizational risk.

Increased complexity

More and more projects relied on multi-arm cohorts, where nodes could split, have different criteria applied, and merge back. In old cohort select, an engineer was responsible for creating all nodes, then going back and deciding how they tied back to each other. This was not readable to anyone else.

old ui — The old version of cohort selection denotes splits with green lines and a "combine sources" criteria below them. Most users in testing did not know this was a multi-arm cohort.

Our original tests focused on a better single column view of the nodes, and while users could piece together the workflow, they unanimously asked for a visual, node based editing view. When we tried a hacky version of this in Miro, we realized RS were hoping for far more creativity than Blocks pipelines would allow—like splits not remerging until the end—and decided we needed to improve the main workflow. We simultaneously built out a view-only tree view for extra confidence.

A first test with much lighter colors and less emphasis on splitting and merging — Our first test with light colors and no visual differentiation of the split and merge hit 30% comprehension amongst data analysts.

We pushed pushed several iterations with clearer color and naming iterations until we hit full user comprehension and users could explain back the most complex project in multi-arm to date.

By getting users to create the structure of the split first and already distinctly displaying how branches would merge, we were able to visually explain how multi-arm worked and keep expectations in check.

HCS Now

As of 2026, 40% of projects use multi-arm. Even with cohort selection now owned by a completely different user group, there have been no reported setup issues in the last 18 months.

A recent project using two arms and subarms with the associated visuals.