Documentation

How to use Myelin

Discovery in the browser, YAML workflows on BU SCC—from GEO search and contrast design to samples.tsv, QC Keep/Exclude, and qsub. Practical flow only; no local dev or ops setup here.

Overview

What Myelin does

Myelin coordinates dataset discovery and workflow configuration: GEO search, editable metadata, sample sets with saved contrasts, and machine-readable pipeline exports. Computation runs on BU SCC; the web app is the control plane for curation.

Supported YAML workflows in myelin-pipeline-script include the recommended QC-gated path (atac_preprocessing_qc.yaml → Myelin QC review → differential_footprinting.yaml), the legacy end-to-end run (atac_footprinting.yaml), QC backfill (qc_only.yaml), and standalone TOBIAS matrix postprocess.

Exports map to one batch directory on SCC:samples.tsv plus growing stage outputs—not ad hoc scattered paths.

Interface

Typical flow in the web app

  1. Open the lab URL or forwarded port on a login node.

  2. On Search, use New search for GEO keywords or Past searches to reload a prior query (sign in to sync history across sessions). Pick a GSE, inspect the metadata table, select samples, and add them to a new or existing sample set.

  3. Open your set under Sample sets. Edit metadata inline (condition, exposure, cell type, tissue, SRR, etc.) and configure Contrasts per GSE — see Metadata & exports.

  4. Export a Human review (H) zip to check metadata and saved pairs, then download samples.tsv (M) once contrasts are saved. Select table rows first for a subset export.

  5. Copy the batch folder (at minimum samples.tsv) to project disk on SCC, e.g. under projectnb/. Prefer the QC-gated ATAC path: submit preprocessing → review Keep/Exclude in Myelin → export filtered samples.tsv → run differential footprinting.

  6. Submit from SCC with myelin-pipeline-script and the workflow YAML that matches your goal.

  7. Track progress with qstat, BATCH_DIR/logs/run_manifest.json, and stage logs under BATCH_DIR/logs/. After preprocessing QC, open QC review (or the panel on the sample set) to mark Keep / Exclude.

Sample set

Metadata, contrasts & exports

The metadata table groups samples by GSE. condition and exposure display as separate columns; the pipeline machine label merges them (condition_exposure when both are set). Fill or correct labels before configuring contrasts.

In Contrasts (study pairs), pick control vs experiment labels per GSE using the per-study wizard or manual rows. The AI pairing reference panel suggests differential-footprinting contrasts from your metadata — apply suggestions, then click Save pairs to persist. Machine export is blocked until at least one pair is saved.

Human (H)

Zip with the full metadata table and saved study-pair summary — for review before you run the pipeline. No saved pairs required. After QC review, the same export is enriched with QC metrics and SCC BAM/peak paths.

Machine (M)

samples.tsv only — merged labels, GSE, replicate, SRR, and contrast_code. The submitter derives study_pairs.tsv and srr.list at run time. Requires saved contrasts. After QC review, this becomes samples.tsv (QC kept) — Keep rows only, with SCC paths for differential footprinting.

Checkpoint

QC-gated ATAC (recommended)

Instead of running download → footprinting in one shot, stop after biological QC, decide Keep / Exclude in Myelin, then continue differential footprinting on the filtered set. SCC posts the QC report and artifact paths back to the sample set via the ingest callback.

  1. Submit workflows/atac_preprocessing_qc.yaml with --myelin-api-url, --myelin-sample-set-id, and MYELIN_SCC_INGEST_TOKEN matching the web app secret.
  2. When the sample set shows awaiting review, open QC review or the panel on the sample-set page. Mark each run Keep or Exclude, then save.
  3. Download samples.tsv (QC kept) into a new SCC batch directory (paths from the QC run are already embedded).
  4. Submit workflows/differential_footprinting.yaml. The submitter reuses steps 5–7 when a condition’s selected SRR set is unchanged; otherwise it rebuilds only the affected merges / peaks / reference peaksets before ATACorrect → footprints → optional BINDetect.

The legacy end-to-end atac_footprinting.yaml remains available when you intentionally skip the Myelin QC checkpoint.

Preprocessing → Myelin callback
export MYELIN_SCC_INGEST_TOKEN='same secret as MYELIN_SCC_INGEST_TOKEN in the web app'

PYTHON=python3 ./bin/submit.sh /projectnb/.../qc_batch \
  -w workflows/atac_preprocessing_qc.yaml \
  --myelin-api-url https://your-myelin.example \
  --myelin-sample-set-id <sample-set-uuid>
After QC review — differential
# Place reviewed samples.tsv (QC kept) in a new batch dir, then:
PYTHON=python3 ./bin/submit.sh /projectnb/.../differential_batch \
  -w workflows/differential_footprinting.yaml

File layout

Batch directory & input tables

Before SCC jobs submit, place at least samples.tsv in your batch folder. The submitter writes derived files on first run:

samples.tsv (M export)

Recommended columns: run_accession, condition (merged machine label), replicate, gsm, study_id, contrast_code. Download from the sample set after saving metadata and contrasts.

contrast_code (P-A-R)

Tokens like 1-1-1 encode contrast, arm, and replicate (2-1-1,3-1-1 when a sample belongs to multiple contrasts). At submit, submit_workflow.py materializes study_pairs.tsv and srr.list from this file. Legacy batches may still ship a hand-written study_pairs.tsv instead.

Human review zip (H) is for checking only — do not copy it to SCC. Keep one batch folder per run under lab project paths, e.g. .../batch/batch_YYYYMMDD_HHMMSS/.

After git pull on the pipeline repo, re-export samples.tsv if contrast encoding or metadata changed — derived files in the batch dir are rewritten on the next submit.

HPC

BU Shared Computing Cluster (SCC)

The ATAC DAG runs via qsub on BU’s Shared Computing Cluster (SCC). Jobs use Sun Grid Engine #$ directives; see BU’s Submitting your batch job. Your onboarding lists SSH host and project (-P) names.

Login vs compute: use the login shell for git, edits, and submit wrappers; real work executes on compute nodes via the queue. Aim large data at projectnb/-style space from your PI.

Load tools with BU module. Batch snippets often start with #!/bin/bash -l so modules load correctly under qsub.

Repository

Get & update myelin-pipeline-script

Pipeline scripts live in myelin-pipeline-script (separate from this web repo). Clone from the URL your lab shares.

First clone on SCC
cd /path/where/you/keep/code
git clone <repository-url> myelin-pipeline-script
cd myelin-pipeline-script
Update before submitting
cd /path/to/myelin-pipeline-script
git status
git fetch origin
git pull origin main
Swap main for your branch when needed; stash/commit local edits before pulling. Pull only updates tracked files—not your batches on project disk.

Execution

Submit the pipeline from SCC

Run from the repository root. Load Python ≥ 3.7 on SCC, install deps once, then point bin/submit.sh at your batch folder and a workflow YAML.

One-time setup
module purge
module load python3/3.10.12   # or any >=3.7 module on SCC
python3 --version
cd /path/to/myelin-pipeline-script
python3 -m pip install --user -r requirements.txt
YAML workflows
  • workflows/atac_preprocessing_qc.yaml — download through peaks + biological QC; posts report and SCC paths to Myelin (QC-gated).
  • workflows/differential_footprinting.yaml — after QC review: rebuild changed merges/peaks only when needed, then ATACorrect → footprints → optional BINDetect.
  • workflows/atac_footprinting.yaml — legacy end-to-end: download through TOBIAS footprinting; optional BINDetect; matrix postprocess as final stage.
  • workflows/qc_only.yaml — per-sample biological QC, replicate correlation (step 7b), and aggregated QC report when BAMs and peak BEDs already exist.
  • workflows/tobias_postprocess.yaml — rerun matrix postprocess only on a completed batch.
Shell
cd /path/to/myelin-pipeline-script

# Dry-run (writes logs/run_manifest.json, no qsub):
PYTHON=python3 ./bin/submit.sh batch/my_condition_study \
  -w workflows/atac_preprocessing_qc.yaml \
  --dry-run

# Recommended: QC-gated preprocessing (see #qc-gated for callback flags):
PYTHON=python3 ./bin/submit.sh batch/my_condition_study \
  -w workflows/atac_preprocessing_qc.yaml \
  --myelin-api-url https://your-myelin.example \
  --myelin-sample-set-id <sample-set-uuid>

# After QC review — differential footprinting:
PYTHON=python3 ./bin/submit.sh batch/differential_batch \
  -w workflows/differential_footprinting.yaml

# Legacy full ATAC footprinting (no Myelin QC checkpoint):
PYTHON=python3 ./bin/submit.sh batch/my_condition_study \
  -w workflows/atac_footprinting.yaml

# QC backfill on existing batch (4b + 7b + report):
PYTHON=python3 ./bin/submit.sh batch/my_condition_study \
  -w workflows/qc_only.yaml

# Same via myelin CLI alias:
PYTHON=python3 ./bin/myelin submit -b batch/my_condition_study \
  -w workflows/atac_preprocessing_qc.yaml

submit_workflow.py topologically sorts stages, wires SGE hold_jid, and rewrites logs/run_manifest.json with real job IDs (dry-run keeps them null).

The old shell pipeline/submit_pipeline.sh orchestrator is archived under archive/legacy_pipeline/ — use YAML submit for new batches.

On disk

Outputs & logs

Inside BATCH_DIR, expect:

  • samples/, bam/, peaks/, tobias/ — stage outputs
  • logs/run_manifest.json — submitted stage DAG and scheduler job IDs
  • logs/<stage_log_subdir>/ — per-stage wrappers and stdout/stderr
  • logs/stage3_tobias_postprocess/ — matrix postprocess logs
  • tobias/.../ — FootprintScores, optional BINDetect, TF×500 bp bin log2FC matrices

Scheduler

Monitor & cancel jobs (SCC)

SGE queries
qstat -u <your_username>
qdel <job_id_1> <job_id_2> ...
Scripts print IDs at submit — pair with tail -f on BATCH_DIR/logs/ for narratives.

Optional rerun

TOBIAS postprocess (matrix job)

Legacy full ATAC footprinting already runs matrix postprocess as the final workflow stage (tobias_postprocess_matrices), waiting on FootprintScores and optional BINDetect. Differential footprinting stops after BINDetect — run matrix postprocess separately if you need TF×500 bp bin matrices. To rerun matrices alone on a finished batch:

Shell
cd /path/to/myelin-pipeline-script

PYTHON=python3 ./bin/submit.sh /projectnb/.../batch/my_condition_study \
  -w workflows/tobias_postprocess.yaml \
  --dry-run

PYTHON=python3 ./bin/submit.sh /projectnb/.../batch/my_condition_study \
  -w workflows/tobias_postprocess.yaml
Use absolute batch paths on SCC. Primary output is continuous TF×500 bp bin log2FC matrices (TFBSlog2FCsum_*_500bpBins*.txt); see scripts/tobias_postprocess/README.md in the pipeline repo for env vars and optional binary matrix export.