Python API¶
The public acquisition types are re-exported by tapestry.data. Fetcher and
explorer APIs are documented separately because most consumers should interact
with them through the command line.
Data package¶
data
¶
Raw, vintage-aware covariate acquisition and storage.
DatasetSpec
dataclass
¶
A catalog entry for one independently retrievable raw dataset.
FileRecord
dataclass
¶
Integrity and size information for one acquired file.
RevisionMode
¶
Bases: StrEnum
How historical versions can be recovered from a source.
SnapshotManifest
dataclass
¶
Machine-readable provenance for an immutable acquisition snapshot.
Repository¶
RawDataRepository
¶
Filesystem repository consumed by acquisition now and model datasets later.
mirrors_dir
property
¶
raw_dir
property
¶
root = Path(root).expanduser().resolve()
instance-attribute
¶
__init__(root)
¶
_atomic_write_json(path, value)
staticmethod
¶
begin_snapshot(spec)
¶
initialize(catalog)
¶
latest(dataset_key)
¶
list_snapshots(dataset_key)
¶
snapshot_path(manifest)
¶
verify_snapshot(manifest)
¶
Raise when a payload file is missing or differs from its manifest.
Data models¶
models
¶
Stable metadata objects shared by acquisition code and downstream datasets.
RevisionMode
¶
Bases: StrEnum
How historical versions can be recovered from a source.
DatasetSpec
dataclass
¶
A catalog entry for one independently retrievable raw dataset.
FileRecord
dataclass
¶
Integrity and size information for one acquired file.
SnapshotManifest
dataclass
¶
Machine-readable provenance for an immutable acquisition snapshot.
Geography¶
geography
¶
Normalize state identifiers and select native state/national observations.
normalize_state(value)
¶
Return a postal abbreviation for a state-like publisher value.
observation_geography(row, resolutions, path_geo_type=None)
¶
Return a native state code or US; reject finer and regional observations.
A state column on a county/site row is context, not its observation support. Only a national-only source may omit geographic columns altogether.
Shared post-intake selection¶
SelectedData
¶
Bases: RawTables
Post-intake table/measure selection shared by the explorer and downstream.
audit = []
instance-attribute
¶
__init__(data_root)
¶
_catalog()
¶
_table_sources(artifact)
¶
_tar_sources(artifact)
¶
_validated_hub_rows(table)
¶
Choose unique rows only after checking the entire canonical table.
Spill keys to SQLite so cross-batch conflicts cannot be averaged or resolved by file order. Retain compact row offsets for repeat reads.
iter_records(*, group=None, dataset_key=None, available_by=None)
¶
Stream native selected records, optionally excluding releases after a cutoff.
All eligible revisions are emitted. Consumers must resolve full snapshots and revisions themselves, rather than treating them as independent events. Unknown historical availability is excluded when a cutoff is supplied.
selected_tables(*, group=None, dataset_key=None)
¶
Consume each yielded table before advancing; tar files are temporary.
summary()
¶
table_decision(key, name, names)
¶
Explorer index¶
ExplorerIndex
¶
Bases: SelectedData
Build and query a disposable long-form index of plottable observations.
_write_lock = threading.Lock()
instance-attribute
¶
batch_rows = batch_rows
instance-attribute
¶
cache_mb = cache_mb
instance-attribute
¶
index_path = Path(index_path).expanduser().resolve() if index_path is not None else self.data_root / '.explorer' / 'series.sqlite3'
instance-attribute
¶
revision_ledger_path = Path(ledger_path).expanduser().resolve() if ledger_path is not None else self.index_path.parent / 'revisions.parquet'
instance-attribute
¶
strict = strict
instance-attribute
¶
__init__(data_root, index_path=None, ledger_path=None, *, batch_rows=100000, cache_mb=256, strict=False)
¶
_build_lock()
¶
Reject competing CLI builders, including ones sharing a ledger.
_check_build(connection, ledger_path, fingerprint)
¶
_create_indexes(connection)
staticmethod
¶
Create query indexes once, after all rows have been inserted.
_create_schema(connection)
staticmethod
¶
_cutoff(as_of)
staticmethod
¶
_date_column(columns, catalog_column)
staticmethod
¶
_dimension_columns(profiles, *, excluded)
staticmethod
¶
_flush_points(connection, points, ledger_writer)
staticmethod
¶
_index_artifact(connection, artifact, progress, ledger_writer)
¶
_index_table(connection, source, progress, ledger_writer)
¶
_ledger_errors(meta, path)
staticmethod
¶
Check the footer and pair identity without scanning 77M+ rows.
_location_code(location)
staticmethod
¶
_record_source(connection, artifact, *, status, message=None, rows_seen=0, state_column=None, date_column=None, numeric_columns=())
¶
_revision_rows(series_id, state)
¶
Read one series/state slice from the columnar revision ledger.
_series_id(connection, source, value_column, state_column, date_column, dimensions)
staticmethod
¶
_state_column(profiles)
staticmethod
¶
_storage_errors(connection, ledger_path)
classmethod
¶
build(*, progress=_print_progress)
¶
Build privately, validate, then replace the two derived files.
Stop the server before rebuilding: two file renames are not atomic as a pair. Matching build IDs detect interrupted publication on next use.
connect()
¶
data(state, series_ids, *, scale=False, as_of=None)
¶
ensure(*, rebuild=False, progress=_print_progress)
¶
is_current()
¶
list_series(state, *, query='', cadence='', vintage='', support='', freshness='', limit=250, offset=0)
¶
overview()
¶
source_fingerprint()
¶
status(*, validate=False)
¶
Inspect saved build health without rebuilding or opening a browser.
Validation checks SQLite and the Parquet footer, not every raw value. Source errors are reported separately from storage integrity failures.