B0.1 — Architecture crosses

Status: the original 172-configuration experiment is complete. This page records that experiment. On 2026-09-16 the failed stochastic-trend decoder and joint_trend recipes were removed; the active machine-readable grid now contains 135 configurations. Historical results retain their original counts. See B1 for the independent-target masked-vintage successor.

Names: B0.0 is the completed post-scaling crosses experiment, stored as data/experiments/b0-us-cross-4/. Its results retain their original objective. B0.1 is this new experiment; its manager name and output folder are B0.1 and data/experiments/B0.1/. The historical folder is not moved; B0.0 is its documented experiment label.

Revised design

Use six prespecified architecture references, vary one factor at a time around each, and globally deduplicate identical recipes. Keep two focused interaction panels rather than crossing every axis with every other axis. Direct quantile prediction is removed. All models generate samples, from which the usual 23 exported quantiles are computed for WIS.

The main questions remain spatial/cross-target information exchange, pathogen-versus-target parameter sharing, temporal representations, and explicit level/growth uncertainty. Matched architectural checks, independent-model controls, calibration, and sample-based mixtures remain in scope.

This replaces the proposed full factorial. Reference recipes are contrasting hypotheses, not winners selected from outer-season scores. One-factor effects apply around their reference; the two interaction panels identify only the specified interactions, not all interactions in the architecture space.

Fixed data, objective, and training contract

• Same finalized six-channel dataset and frozen hub support as B0.0. No new observations, augmentation, wastewater, or vintage reconstruction.
• No raw-count or rate-only admission transform, including controls. Retain square root, fourth root, and log1p of admission rates per 100,000. CRPS and WIS still score native admission counts and ED proportions.
• Target weights [1,1,1,.5,.5,.5], without extra flu weighting.
• Geography: 80% equal-weight states/DC + 20% native US.
• Average available targets with those weights within each season, then average seasons equally. Missing challenge support is not imputed.
• Training: native-unit fair CRPS / fitting-partition channel/location Q95; ranking: location-level model/ensemble total-WIS ratios. Retain the adopted scale rules.
• Every configuration gets three seeds: 42, 43, 44, and all three outer season folds. No first-seed pruning.
• Adam, learning rate .001, weight decay zero, batch 8, gradient clipping 5.
• Default cap 300 epochs; compare cap 100; patience 30 for both. This interprets the epoch request as maximum budgets while preserving the agreed use of early stopping. No patience-zero or 1,000-epoch runs are planned.
• All six references receive the 100-versus-300 comparison as a one-factor change. Other cross variants and interaction panels use their reference's default cap 300. Both caps are used in the conv and independent-sharing controls. This is not epochs multiplied by every other axis.
• If both budgets select the same epoch before 100, they may produce the same refit. Report selected epoch and cap hits; never label a stopped fit as having trained for its entire budget. A cap effect is not a clean fixed-epoch effect.
• 128 training members, 256 fixed validation members, 256 evaluation members. B0.1 implementation must update/expose validation member count (the current implementation uses 32).
• Default width 64, global latent 16, local latent 4 when enabled. Latent 32 is used only in the two historical conv topology controls.
• Keep all six observed input channels, masks, population/US features, dynamics, and calendar features except the explicitly identified feature ablations.
• Native US is directly predicted. Normalizers, fitting, and stopping exclude the outer held-out season. This remains retrospective exploratory season CV.

A. Six reference architectures and their crosses

Common defaults: fourth-root admissions, logit ED, 12-week context, width 64, latent 16, shared state/US head, no location embedding or extra common factor, dynamics/calendar on, cap 300 and patience 30.

Reference	Encoder	Decoder	Output sharing	Information exchange	Noise
local_mlp	MLP	Existing residual	Shared	None	Global
spatial_conv	Existing convolution	Existing residual	Shared	Shared spatial	Global
pathogen_multiscale	Multiscale convolution	Existing residual	Three pathogen heads	Pathogen spatial	Global + local
target_multiscale	Multiscale convolution	Existing residual	Six target heads	Target spatial	Global + local
joint_multiscale	Multiscale convolution	Existing residual	Six target heads	Joint location–target	Global + local
joint_trend	Multiscale convolution	Stochastic trend	Six target heads	Joint location–target	Global + local

Logit ED is a common design choice, not a claim that it wins for every reference; each reference also tests linear and fourth-root ED. The old linear-ED MLP candidate is represented by the corresponding cross variant.

Vary one factor at a time around every reference

Axis	Tested levels
Temporal encoder	MLP; existing convolution; multiscale convolution
Decoder	Existing modulated residual; stochastic damped trend
Output sharing	Shared; three pathogen heads; six target heads
Information exchange	None; shared spatial; pathogen spatial; target spatial; joint location–target
Noise	Global; global plus per-location
Admission transform	Square root; fourth root; log1p
ED transform	Linear; logit; fourth root
Lookback	8, 12, 26 weeks
Width	64, 128
Recent dynamics	On, off
Calendar	On, off
Learned location-ID embedding	Absent, dimension 8
State/US output heads	Shared, separate
Extra common noise factor	Off, on
Epoch cap	100, 300; patience fixed at 30

Each reference has 23 non-default one-factor variants plus itself: 144 candidate entries, which deduplicate to 131 unique configurations.

Matched architectural checks are included in these crosses, not another large separate factorial. Calendar-off removes both annual sine/cosine and Christmas timing. Dynamics-off removes the extra recent-dynamics feature block; basic masks and calendar remain. Location embeddings use existing IDs, not new data. Geography, utility weights, optimizer, batch size, and sample counts stay fixed.

Information exchange: the central capability comparison

All variants retain the same local six-channel context branch. Remote exchange tokens contain only the histories named by their scope: all six for a shared location token, two outcomes for a pathogen token, or one outcome for a target token. Do not silently insert pooled six-channel remote context into every target token; that would blur the cross-target information comparison.

None. Each location can use its own six-channel history; there is no cross-location feature exchange. Shared weights still transfer learning between locations during fitting.

Shared spatial attention. One token per location pools that location's six-channel history: 52 location tokens. One common attention representation feeds the forecasts. This is the existing spatial-attention concept.

Pathogen-specific spatial attention. Pool admission/ED histories into one location/pathogen token: 156 tokens total, with attention restricted to the 52 locations within a pathogen. Flu can learn a different spatial pattern from COVID while its two outcomes initially share the spatial representation. Keep pathogen-conditioned attention parameters shared across the three groups.

Target-specific spatial attention. Retain one history token per location and target. For a given target, attention sees that target at all 52 locations; it can learn different location weights for flu admissions and COVID ED. Use shared attention parameters conditioned on target identity, with a same-target attention mask, so specialization does not require six fully independent attention networks. Local six-channel context remains available.

Joint location–target attention. Retain all 52 × 6 = 312 tokens and permit attention across both locations and targets. A location's flu-admission representation can attend to flu ED, another location's admissions, or another pathogen's history. This tests explicit cross-target/cross-location information selection. It does not introduce new data or establish a causal relationship.

For attribution, these attention variants mix context representations before the horizon decoder. Do not quietly make only one variant horizon-conditioned: that would change two architectural properties at once. Horizon-conditioned queries or attention to separate past-week tokens are sensible later capability extensions if the 312-token model shows value; they are not hidden extra axes in the present counts.

Attention has no geographic-neighbor restriction. The main inputs contain population and US indicators, not physical adjacency. A learned location-ID embedding is tested separately. Attention weights are not validated geographic transmission estimates.

Pathogen heads versus target heads

• Shared: common encoder and decoder parameters, with target embeddings.
• Three pathogen heads: shared feature extractor, then separate flu, COVID, and RSV forecasting heads. Each pathogen head covers admissions and ED, with target identity retained inside the head.
• Six target heads: shared feature extractor, then a distinct forecasting head for each pathogen/outcome pair. Flu admissions and flu ED can have different observation mappings and uncertainty responses.

Apply this choice to both decoder families: it changes the learnable output readout/modulation parameters, not the available inputs or supervision weights. Use the same per-head design/width and report the resulting parameter counts. More heads add parameters, so gains are evidence for the whole sharing design; matched width checks help distinguish specialization from generic capacity.

Heads alone cannot determine whether sharing the encoder is harmful. Block E adds independently fitted models for that reason.

Stochastic trend decoder: precise hypothesis

The existing decoder learns all residual dynamics and noise responses from features plus the horizon embedding. The trend decoder supplies an explicit starting level and growth component in the positive/bounded decoder's working space. An illustrative residual is

delta_m(h) = beta * D_phi(h) + residual(h) + a_m + D_phi(h) * b_m,

where D_phi(h) = sum(phi**k for k=0..h-1) and 0 < phi < 1.

• beta is recent valid growth plus a learned correction.
• Damping phi and noise amplitudes depend on context.
• a_m represents uncertainty in current level; b_m uncertainty in growth.
• A small deterministic horizon residual permits departures from a straight trend. The noise terms are generated from the same global/local member draws across the four horizons, preserving sampled path identity.
• The existing positive/bounded inverse mapping produces native counts and ED proportions. All loss remains in native units.
• Missing recent slopes use a flagged fallback; missing observations are not treated as observed zeros.

This is a soft tendency toward smooth local progression, not a rule requiring monotone paths or a single peak. It is not expected to infer a rebound lacking predictive evidence. Marginal CRPS alone still does not identify the true joint path distribution, so trajectory diagnostics remain necessary.

Multiscale encoder

Use three small temporal convolution branches with kernel width 3 and dilations 1, 2, and 4; two layers per branch yield approximate receptive fields of 5, 9, and 17 weeks. Each receives values and validity masks. Pool recent and broader features and project the concatenated branches back to the common width. Split branch widths to keep total size reasonably comparable to the existing convolution; record measured parameter counts. No future observation is included in a context convolution. This tests short-versus-medium-scale dynamics while holding the rest of the model fixed.

B. Focused attention × output-sharing panels

At local_mlp and joint_trend, fully cross five exchange modes × three output-sharing modes, retaining all other reference settings. This gives 30 candidate entries, of which 15 are new beyond block A.

This is the central interaction check: does the benefit of target/pathogen/joint attention depend on using a shared or specialist output head? The two contrasting references test whether that interaction changes with the temporal/decoder recipe. Do not infer all encoder/noise/transform interactions from these panels.

C. Focused admission × ED transform panels

At the same two references, fully cross three admission transforms × three ED transforms, keeping other reference settings fixed. This gives 18 candidate entries, of which 8 are new after the preceding blocks.

A shared six-channel model can respond differently to a transform pair than to either isolated change. This panel covers that possibility without multiplying the transform grid across all architectural variants.

D. Previous conv topology controls

Retain the old full conv topology with both legacy and residual2 decoders: 12 weeks, width 64, latent 32, fourth-root admissions, logit ED, shared spatial attention, global plus local noise, separate US heads. Test both 100- and 300-epoch caps with patience 30: 4 additional configurations.

They use the adopted loss/ranking, so these are architectural carryovers, not reproductions of historical B0.0 scores. Residual2 appears only here; raw/rate controls are excluded.

E. Independent pathogen/target controls, with matched shared models

Compare three fitting partitions:

1. One shared model predicting all six targets.
2. Three independent pathogen models, each predicting admissions and ED at their 1/.5 relative weights.
3. Six independent target models, each predicting one outcome.

Every component retains all six observed input channels. Export only trained outputs and assemble independent components into complete six-target bundles for normal ranking. Separate heads in block A share an encoder; this block checks whether sharing the encoder itself causes negative transfer.

Cross these three fitting partitions with three temporal encoders × two caps (100/300). Hold the remaining settings at legacy decoder, no spatial exchange, global noise, 12 weeks, width 64, fourth-root admissions, logit ED.

This is 18 candidate entries, including shared models for direct comparison; 14 are new after the preceding blocks. Twelve of the 18 are independent bundles: six pathogen bundles and six target bundles. This replaces the earlier 54-bundle transform sweep. The transform interactions are handled in block C.

Independent bundles have more total parameters/compute than one shared model; report this alongside the width ablation rather than claiming parameter-matched transfer effects. No learned cross-target joint uncertainty is supplied by six independent models. Three pathogen models share uncertainty only within each pathogen. This block does not estimate independence-by-spatial-attention effects.

Size and accounting

Deduplicate across blocks in this order; identical recipes run only once and retain all reference/panel memberships for analysis.

Block	Candidate entries	Additional unique configurations
A: six references and one-factor crosses	144	131
B: attention × output sharing	30	15
C: admission × ED transforms	18	8
D: previous conv controls, both caps	4	4
E: shared/independent controls, both caps	18	14
Total	214	172

172 configurations × 3 seeds = 516 CV evaluations, containing 1,548 outer evaluation folds. There are 160 single-model recipes, six three-model pathogen bundles, and six six-model target bundles: 1,926 outer component-model refits. One inner fit for stopping and one final refit per component gives 3,852 training calls before any reuse of identical stopped trajectories. These are counts, not runtime estimates. No configuration is removed because its first seed performs poorly.

The former 5,366-configuration full factorial and direct quantile block are replaced by this design. The JSON contains only the active crosses plan.

Explicit exclusions

• Direct quantile prediction. Keep sample-based models and quantile export for WIS; these are different from training a direct quantile head.
• Raw-count and rate-only input transforms, including all controls.
• Alternative target/geography/season utility weights.
• Training without early stopping, and the blanket 1,000-epoch budget. Retain the requested 100/300 caps, with the interpretation stated above.
• A full Cartesian product of all architectural, transform, and training axes.
• Large learning-rate, optimizer, batch, latent-size, and dropout grids.
• New history, augmentation, wastewater, vintages, and age/subtype inputs.
• Diffusion/flow models, full-season rollout, peak prediction, and extra horizons.
• A broad residual2 decoder grid or joint-loss coefficient sweep. Residual2 remains a carryover control; assess sampled-path failures before changing the dependence objective.

Calibration and mixtures after fitting

Evaluate both uncalibrated and modestly calibrated finalists. Compare no correction, pooled target/horizon width correction, and width plus bias correction where training-side errors support it. Correction parameters must come from each outer training partition, not the season used to score the correction. The current saved inner forecasts come from a model subsequently refitted; calibration transfer is an explicit assumption tested on outer folds.

Compare single-seed models, equal-weight three-seed mixtures, and a few prespecified architecture mixtures. Retain whole members, regenerate enough samples from checkpoints, and score the mixture's actual quantiles. Do not use mean constituent WIS as mixture WIS. For any optimized blend, learn weights within the outer fitting partition.

Evaluation: capabilities as well as overall rank

Always report all six targets, every target/season, states/DC versus US, and each horizon beside the season-first composite. Additionally:

• Growth, decline, and turning-point/rebound errors and coverage.
• Low/high burden, location size, and calendar phase, including COVID outside winter where frozen evaluation support exists. A year-round diagnostic without ensemble support must be labeled separately from ensemble skill.
• Calibration curves and WIS components: dispersion, underprediction, overprediction. Do not treat narrower intervals as automatically better.
• Temporal increments, four-week sums, sampled peak-within-window distributions, and dependence diagnostics for sampled models; no joint claim from marginal WIS.
• Paired architecture differences at identical transforms/history/noise/head choices. Report interaction patterns, not just global feature importance.
• Parameter count, training time, evaluation time, and memory per recipe.

The strongest capability result would be repeatable gains from target-specific or joint attention, or from sharing at the right level, across matched settings and relevant disease/season slices. Attention visualizations alone are not forecast-skill evidence. Three seeds address training variability; a large configuration count does not create new independent seasons.

Implementation and execution requirements

The encoders, residual/trend decoders, sharing modes, scoped exchange, independent bundles, feature ablations, and 256-member fixed validation draws are implemented. b01_suite.py translates the conceptual JSON options into executable scenario flags. jobs.csv stores one canonical string per unique recipe; design.json retains every reference/panel membership, including recipes shared by multiple contrasts.

Before a full launch:

1. Generate/deduplicate the complete job manifest; verify the above counts, complete target support, and zero raw/rate recipes.
2. Check each architecture's native-unit output, masks, fixed validation noise, and checkpoint reconstruction, plus a short fit of the largest actual tensor shapes.
3. Preflight positive ensemble WIS denominators on frozen location-level support.
4. Use a versioned code snapshot and save data/objective/configuration hashes.
5. Replace the patron launcher’s old hard-coded task range 6–59 with the full manifest. Allocate whole GPUs and run measured numbers of configurations per GPU. Fit each configuration's seeds sequentially, as the current manager does.
6. Measure memory/time for width-128/joint-attention and 26-week/joint-attention cases before choosing lanes. B0.0's six-lane L40 setting and runtime are not evidence that six larger B0.1 models will fit. Reserve CPU threads to match actual workers.
7. Save all evaluated quantiles, fitting/stopping logs, per-location totals, and enough metadata to regenerate members. Keeping all 256 members for every fit is not required; regenerate full mixture samples for selected comparisons.

The initial resource preference is the patron GPUs, with batching sized from those measurements. No live GPU availability or B0.1 runtime has been measured for this specification. No jobs are submitted in this turn.

Run B0.1

The prepared experiment is data/experiments/B0.1/: 172 configurations, seeds 42–44, 516 three-season CV evaluations. Its jobs.csv contains all identifying strings; design.json retains every reference/panel membership.

From the repository root:

# Refresh only before launch, never while workers are running.
.venv/bin/python -m tapestry.models.manager plan -e B0.1 --suite B0.1 --device cuda
mkdir -p output/slurm
sbatch --array=0-3 --export=ALL,LANES=6 scripts/jlessler.sbatch B0.1
sbatch --array=0-1 --nodelist=g1803jles02 --export=ALL,LANES=6 scripts/jlessler.sbatch B0.1

LANES=6 means six concurrent fitting subprocesses on each GPU, hence 36 across the four L40 and two H100 GPUs. Each allocation requests 12 CPU cores, 110 GB host memory and 48 hours. Six lanes are the user's chosen operating point; there is no automatic memory-based reduction or profiling gate. The brief profiles submitted before that instruction completed; no broad profiling or test suite is required before launch.

All six GPUs use one NFS-locked queue (dispatch.json). A claim covers one configuration/seed's three outer folds; only one seed per configuration can run at a time. After each seed the configuration returns to the queue, so the next seed can move between GPUs. An idle peer gets a ten-second chance to take a just-finished configuration before the same GPU reclaims it.

Scheduling cost is an explicit heuristic: component count × epoch cap × squared relative width × square-root relative lookback, multiplied by encoder factors (MLP 1, conv 1.5, multiscale 2), decoder factors (legacy 1, residual2 2, trend .8), and a factor 1.4 for joint attention. These are priorities, not measured runtime predictions; early stopping can make actual durations quite different. Underloaded GPUs take long remaining seed chains; GPUs already carrying above-average estimated work take short jobs. This spreads large and small configurations, and faster GPUs keep taking work without a fixed node-specific task list. A final tail is still possible because an active seed cannot be migrated.

Queue recovery uses squeue -a -r because jlessler is hidden. A seed is only reclaimed when its owning allocation disappears, never solely because it is old. Failed seeds are recorded and skipped for the rest of that dispatch; review and reset failed entries before retrying. Completed seed artifacts are reused. The source snapshot under code/ is fixed for the run. preflight.json records source/data/support hashes and positive frozen ensemble WIS denominators.

.venv/bin/python -m tapestry.models.manager status -e B0.1
.venv/bin/python -m tapestry.models.manager rank -e B0.1
.venv/bin/python -m tapestry.models.manager list --suite B0.1

The reference aliases are B0.1/local_mlp, B0.1/spatial_conv, B0.1/pathogen_multiscale, B0.1/target_multiscale, B0.1/joint_multiscale, and B0.1/joint_trend. The full joint-trend string is:

b0:h12:tr_4rt:ed_logit:geo1:dyn1:lw_obj:enc_msc:sp_joint:hd_sh:dec_trend:nz_loc:us_none:z16:w64:ep300:pat30:bs8:m128:lr0.001:hs_targ:cal1:id0:fit_all:vm256:wd0

hs controls output sharing, hd state/US heads, fit independent fitting, and vm validation members. Seeds and evaluation-member count are also saved in the experiment settings and each attempt record.

Implementation choices and validation

• All models retain a local six-channel branch. Shared spatial attention retains the existing full-context spatial block. Other exchange modes use separate scope-restricted value/mask histories, group embeddings and the enabled population/US/location-ID features. Pathogen and target groups are batched separately through shared attention weights, exactly enforcing their scope; joint exchange attends over all 312 tokens. Exchange precedes the horizon decoder.
• Multiscale branches split the requested width across three branches, assigning the remainder to the dilation-four branch. Each pools its last timestep and mean. Per-component measured parameter counts are saved with every fit.
• Trend growth is the difference of the last two adjacent context weeks in decoder working space: inverse softplus of normalized admissions/root ED, or native logit ED. Missing recent pairs use zero growth plus a validity flag, not zero-valued observations. Context predicts a growth correction, sigmoid damping, and positive level/growth amplitudes. The same projected global/local draws are reused across all four horizons. The learned horizon residual has a small output initialization.
• Calendar-on includes Christmas timing independently of the dynamics switch. Calendar-off removes all three calendar features; dynamics-off removes only the 30 slope/acceleration/age/validity features. Location IDs follow the saved dataset order. The population and native-US flags stay enabled in this suite.
• Independent component i uses seed + 10000*i, selects its own stopping epoch, and refits on its trained target group with all six observed inputs. Input normalizers use the entire permitted fitting partition, identically across components. Each component samples fresh noise; assembled member indices represent independence across components. Only its supervised outputs enter the exported bundle. Model counts/parameters/time sum across components.
• Checkpoints include complete constructor settings; bundles contain each component checkpoint and its channel group. validation_model.pt retains the stopped models for later training-side calibration. Evaluation seed and sample batch size are recorded for exact member regeneration. New head modules replace the old checkpoint layout; earlier weights must be regenerated for this code.
• Cap-hit, actual stopping epoch, refit epochs, training/evaluation time, parameter count and peak CUDA allocation are recorded. The cap remains 100/300 with patience 30. Full calibration, mixtures and scientific diagnostics are later analyses of fitted models, not part of this preparation run.

Validation used real-data forward/backward checks for all 172 recipes, including canonical-string and CLI round trips. Small two-epoch runs exercised all three outer folds for shared, three-pathogen and six-target fitting with stopping, refitting, quantile export and exact saved-member regeneration from checkpoints. These reduced-size runs verify execution, not forecast skill. The full test suite was not run. One-off profiling and concurrency drivers were removed after recording the operational findings in the Longleaf documentation.

Log

• 2026-09-15: user named the previous experiment B0.0 and the new experiment B0.1, excluded raw/rate representations, and requested specification before implementation or launch.
• 2026-09-15: emphasized spatial exchange and pathogen/target specialization; included matched architecture checks and independent-model controls.

• 2026-09-15: user rejected direct quantile prediction, retained matched checks and earlier architectures/controls, requested 100/300 epochs, and preferred crosses around a few defaults. Replaced the full factorial with six reference crosses and two focused interaction panels: 172 unique configurations. Interpreted epoch counts as caps with patience 30, retaining early stopping.

• 2026-09-15: implemented all B0.1 model axes, independent bundles, canonical scenario strings, deduplicated suite, source snapshot and manifest-driven patron launcher. Prepared the 172-job/five-seed experiment without submitting GPU jobs.

• 2026-09-15: user requested both patron nodes and six lanes per GPU. Replaced static slices with a shared, size-prioritized seed queue. First allocations 1237570/1237571 were stopped after detecting a hidden-partition recovery check missing -a; fixed before restarting.

Termination notification

An independent CPU job with afterany dependencies on both GPU arrays sends one aggregate notification to https://ntfy.sh/tapestry after all six allocations end. This includes normal exit, failure, cancellation and timeout. Priority is always 3, as requested. The message reports completed/total CV evaluations, unfinished work, failed/interrupted/incomplete/pending/running/unknown counts, fully completed configurations and Slurm allocation outcomes. One CV evaluation is one recipe/seed across its three outer folds. Historical interrupted attempts do not count against a seed that subsequently completed. A stale running attempt whose allocation ended is reported as interrupted. Termination does not imply all work succeeded.

For the current arrays (1237640 and 1237641), notification job 1245582 is attached. The notifier runs independently of the training snapshot and needs no training restart. A setup notification verified delivery. No per-epoch, per-seed or periodic alerts are sent. Receipts and exact message counts are saved under data/experiments/B0.1/notifications/. HTTP delivery is attempted up to three times; a persistent network/service failure appears in the notifier's Slurm log.

Notifications are now on by default for every experiment: plan stages the notifier into the experiment's notifications/ directory and scripts/jlessler.sbatch queues the follow-up job itself, with --dependency=afterany, so a timeout or cancellation still sends a summary. Set NTFY=0 to suppress it, or NTFY_URL to retarget the topic. To send one by hand for allocations launched another way:

sbatch --dependency=afterany:ARRAY_A:ARRAY_B scripts/notify.sbatch B0.1 ARRAY_A ARRAY_B

The topic and publishing API follow ntfy's publishing documentation.

Three-seed amendment

On user request, the live run was reduced uniformly to seeds 42, 43, 44. Seeds 45 and 46 were still pending for every configuration: 344 CV evaluations were removed under the shared queue lock. The new total is 516 CV evaluations, 1,548 outer folds and 1,926 component refits. No fit was restarted or discarded. This is a 40% reduction in planned evaluations, not a promise of a 40% reduction in remaining wall time; early stopping and configuration sizes vary. Seed variability will now be assessed using three replicates. The GPU allocations and priority-3 termination notification remain active; the notifier reads the updated job manifest. The original code snapshot remains fixed; seed_amendment.json records the live manifest change. Future suite planning reads the three seeds from the amended JSON.

Increased concurrency

The user accepted losing unfinished fits to try more concurrency. After sampling approximately 19 GB used on one L40 and one H100, the run was relaunched with 8 lanes per L40 and 12 per H100 (56 total), one CPU thread per fit. Both sampled GPUs reported 100% utilization: spare memory establishes capacity, not a guaranteed throughput gain. Eight/twelve lanes are a conservative experimental operating point; actual memory depends on the changing recipe mix. Completed CV evaluations are retained; interrupted fits restart from their beginning. Seeds remain 42–44. TAPESTRY_TORCH_THREADS=1 controls Torch, OMP and MKL thread counts and the actual Torch count is recorded in each CV manifest. The priority-3 notification is moved to the replacement arrays so the intentional cancellation does not send a premature termination alert.

sbatch --array=0-3 --cpus-per-task=8 --export=ALL,LANES=8,TAPESTRY_TORCH_THREADS=1 scripts/jlessler.sbatch B0.1
sbatch --array=0-1 --nodelist=g1803jles02 --cpus-per-task=12 --export=ALL,LANES=12,TAPESTRY_TORCH_THREADS=1 scripts/jlessler.sbatch B0.1

Paused for concurrency measurement

The 8/12-lane run and its notifier were cancelled at user request. Completed results are retained. A controlled L40 benchmark compared 1, 4 and 8 concurrent fits of exactly the same eight-model workload. No automatic full-suite restart is scheduled.

Restart after the L40 benchmark

At user request, restarted with 8 lanes on each of all six GPUs (48 fits), 12 allocated CPU cores per GPU, one numerical-library thread per fit, and seeds 42–44. The extra four cores per allocation provide headroom for export and scoring. Four L40 allocations reserve 48 of that node's 56 cores; the two H100 allocations reserve 24 cores. Thirteen completed CV evaluations are reused; 503 remain. Eight lanes is supported by the L40 comparison and is the user's chosen setting for H100s as well; H100 concurrency was not benchmarked. The existing training snapshot is reused without changing model code. Each GPU job has a 48-hour limit. A new afterany CPU job sends the aggregate termination summary to ntfy with priority 3. Current allocation IDs are in launch.json.

sbatch --array=0-3 --cpus-per-task=12 --export=ALL,LANES=8,TAPESTRY_TORCH_THREADS=1 scripts/jlessler.sbatch B0.1
sbatch --array=0-1 --nodelist=g1803jles02 --cpus-per-task=12 --export=ALL,LANES=8,TAPESTRY_TORCH_THREADS=1 scripts/jlessler.sbatch B0.1