Data sources ============ Every dataset seqtree ships, downloads, or reads in a benchmark. Provenance is stated as **experimental** (measured from a biological sample), **derived** (computed from experimental data), or **synthetic** (generated by a model). Never conflate the three. Bundled with the wheel ---------------------- ``python/seqtree/data/control_human_trb_aa.txt`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :widths: 20 80 * - what - 250,000 human TRB junction amino-acid sequences, the null distribution ``P₀`` * - provenance - **experimental**, then **derived** by uniform subsampling * - origin - HuggingFace ``antigenomics/vdjdb-cdr3-frequencies``, human TRB pooled clonotype table * - format - one junction per line, productive only (20 standard amino acids) * - regenerate - ``python bench/gen_control.py --species human --chain trb --size 250000 --seed 0`` The upstream table is sorted by clonotype count, descending. The sample is a **seeded reservoir sample over unique clonotypes**, then shuffled, so any prefix ``bundled[:k]`` is itself a uniform sub-sample. Taking the first *k* rows of the upstream table instead yields the abundance head, which is 25.8× more self-similar and carries 3.1× the ball mass — it violates ``ass:indep`` in ``appendix/evalue.tex`` and corrupts every E-value. See ``CHANGELOG.md`` [0.3.0] Breaking. Out-of-frame clonotypes (VDJtools ``_``) and in-frame stops (``*``) are dropped. ``_`` marks a *collapsed run* of untranslatable positions, so the residue count is already lost and no amino-acid repair is possible; and out-of-frame junctions escape thymic selection, making them a sample of ``P_gen``, which ``lem:hierarchy`` proves is not ``P₀``. Substitution matrices — ``src/blosum62.inc``, ``pam250.inc``, ``pam100.inc``, ``structural.inc`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :widths: 20 80 * - provenance - **derived** (published log-odds / interaction-strength matrices) * - origin - BLOSUM62 (NCBI); EMBOSS EPAM250, EPAM100; Miyazawa–Jernigan interaction strengths, transcribed into ``MJ_CONTACT`` in ``bench/gen_matrices.py`` from the ``MJ`` rows of ``tcren-ms/src/tcren/data/MJ_Keskin_potentials.csv`` (Miyazawa & Jernigan, J Mol Biol 1996, as tabulated by Keskin et al., Protein Sci 1998) * - regenerate - ``python bench/gen_matrices.py`` Stored as non-negative penalties via the Gram transform ``pen[a][b] = s[a][a] + s[b][b] − 2·s[a][b]``, clamped at zero, so ``penalty(a, a) == 0`` and the score defines a ball. **Known corruption in the MJ source.** ``MJ_Keskin_potentials.csv`` has no ``A,N`` row: the lower-triangle slot where ``N-A`` belongs is mangled to ``V,1`` (mirrored ``1,V``) for *both* the ``MJ`` and ``Keskin`` potentials. ``MJ_CONTACT`` carries the correct ``A-N = 0.15`` from that row, so the constant will not diff cleanly against the CSV at that pair. The ``Keskin`` rows are unused here. Fix not yet upstreamed to ``tcren-ms``. Downloaded on demand -------------------- ``load_control(name)`` → ``~/.cache/seqtree/control_*.sqtree`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :widths: 20 80 * - what - larger control repertoires, e.g. ``mouse_trb_aa`` (694,241 productive clonotypes) * - provenance - **experimental**, then **derived** by uniform subsampling * - origin - HuggingFace ``antigenomics/vdjdb-cdr3-frequencies`` * - fetch - ``seqtree.load_control("mouse_trb_aa")`` (uses ``huggingface_hub``, an optional dependency) The cached reference index is content-addressed by the bundled asset or download source, requested size, alphabet and sampling seed. Superseded index files are pruned after rebuilding. Benchmark inputs — not shipped, passed by path ---------------------------------------------- These live in sibling repositories. seqtree does not depend on either; the benchmarks that read them take ``--prototypes`` / ``--vdjdb`` arguments and parse the files with the standard library. VDJdb slim dump ~~~~~~~~~~~~~~~ .. list-table:: :widths: 20 80 * - what - TCR↔epitope pairs with ``v.end`` / ``j.start`` templated-flank annotations * - provenance - **experimental** (curated from published studies) * - origin - ``../mirpy/airr_benchmark/vdjdb/vdjdb-2025-12-29/vdjdb.slim.txt.gz`` * - upstream - https://github.com/antigenomics/vdjdb-db — release 2025-12-29 * - covers - human TRA/TRB, mouse TRA/TRB, macaque TRB. Dotted column names (``v.segm``, ``antigen.epitope``) * - used by - ``bench/gap_rules_by_locus.py`` (criterion A) Per-locus prototype sets ~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :widths: 20 80 * - what - 10,000 clonotypes per (species, locus): ``v_call``, ``j_call``, ``junction_aa`` * - provenance - **experimental** where a real control repertoire exists, otherwise **synthetic** (OLGA) * - origin - ``../mirpy/mir/resources/prototypes/{species}_{locus}.tsv`` * - covers - human TRA TRB TRG TRD IGH IGK IGL; mouse TRA TRB * - regenerate - ``python ../mirpy/mir/resources/prototypes/generate_prototypes.py`` (committed; never rebuilt at test time) * - used by - ``bench/gap_rules_by_locus.py`` (criteria B and C) Junctions, not IMGT CDR3: they include both anchors (``C…F`` or ``C…W``). Median lengths run from 12 (IGK) to 20 (IGH), with IGH reaching 50 — which is why a fixed gap-position set calibrated on length-14 TRB does not transfer. IMGT gene library ~~~~~~~~~~~~~~~~~ .. list-table:: :widths: 20 80 * - what - germline V/D/J nucleotide sequences, 2,710 (species, allele) entries * - provenance - **experimental** (IMGT reference directory) * - origin - ``../mirpy/mir/resources/gene_library/imgt_gene_library.txt`` * - caveat - **V genes translate in frame 0; J genes do not** (TRBJ2-7 is frame 1, TRBJ1-1 frame 2). The companion ``region_annotations.txt`` carries ``jcdr3_aa``, which excludes the conserved F/W — it is CDR3 convention, not junction, and it disagrees with the translated library on some alleles (TRBV1\*01, IGHV1-2\*01). Not currently read by any committed benchmark. Recorded because a germline-templated-flank prior was built against it and **falsified**: it recovers the structurally correct block position 0.4% of the time on TRA against 42.4% for a central prior. UniProt reference proteomes (``isalgo/pmhc_data``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :widths: 20 80 * - what - whole-proteome FASTA: ``mouse`` (UP000000589, 54,857 records / 23,131,234 residues), ``human`` (UP000005640, 147,506 records / 69,578,135 residues), plus ``ecoli_K12_UP000000625`` and ``fluA_PR8_UP000009255`` as size steps * - provenance - **experimental** (UniProt reference proteomes, mirrored) * - origin - HuggingFace dataset ``isalgo/pmhc_data``, path ``proteome/.fasta.gz`` * - regenerate - ``hf_hub_download("isalgo/pmhc_data", "proteome/human.fasta.gz", repo_type="dataset")`` * - used by - ``bench/bench_text_index.py`` (large tier only, ``RUN_BENCHMARK=1``) Carries residues the amino-acid codec does not name — **36 ``U``** (selenocysteine) in human and **33** in mouse — plus 8,417 human and 4,263 mouse ``X``. ``TextIndex`` keeps the unnamed ones as holes and reports the count, rather than refusing the build or silently treating them as wildcards; ``X``/``B``/``Z``/``*`` are real symbols that match only themselves.