Data Source Status

Rulemaking Procedures for Pipeline and Hazardous Materials Safety Programs

Pipeline Safety Enforcement and Regulatory Procedures

Transportation of Natural and Other Gas by Pipeline: Annual Reports, Incident Reports, and Safety-Related Condition Reports

Transportation of Natural and Other Gas by Pipeline: Minimum Federal Safety Standards

Liquefied Natural Gas Facilities: Federal Safety Standards

Response Plans for Onshore Oil Pipelines

Transportation of Hazardous Liquids by Pipeline

Protection of Underground Pipelines from Excavation Activity

Regulations for Grants to Aid State Pipeline Safety Programs

Drug and Alcohol Testing

Procedures for Transportation Workplace Drug and Alcohol Testing Programs

Defined terms extracted from CFR section text — master "definitions" sections plus inline "means/includes" definitions, scoped per part during the eCFR sync

Implicit relationship extraction via LLM-powered analysis

Pipeline safety authorization acts and extracted mandates via Congress.gov API — enriched with fulfillment linking and authority citations

Runs scripts/enrichment/embedding-backfill.mjs for all models registered in MODEL_REGISTRY (null-only). Includes OversightReport (added 1bce63e2) and MfFailureProfile (dac69619).

Case documents (NOPVs, consent orders) and violation allegations — scraped from PRIMIS case pages after enforcement sync

PHMSA enforcement manual, Chief Counsel memos, procedural guides

Rebuilds mff_failure_profiles from mechanical_fitting_failures (~90K rows → ~1K profiles).

Drains pending phmsa_files via pdf-parse + Ghostscript/Tesseract OCR fallback. Bounded to 200/run to stay under 30-min cron timeout.

Federal Register documents — Rules, Proposed Rules (NPRMs), and Notices from PHMSA via federalregister.gov API

Spec 068 — projects DataQualityAnnotation insertions into FeedItem rows on a 60s cadence. Idempotency keyed on (sourceEventId, type).

Daily $-cap (default $1.5) on the Anthropic Sonnet 4 summarization step used by FR Rule embedding. Trips RED when a pre-flight estimate would exceed the cap; resets at midnight local time. Spec 156.

PHMSA-related court cases and public laws via GovInfo API

Hierarchical community detection (Leiden) + summarization + embeddings

Spec 175 rss crawler for Associated Press. Headline + URL + ≤200-char excerpt only.

Spec 175 rss crawler for E&E News. Headline + URL + ≤200-char excerpt only.

Spec 175 html-scrape crawler for Houston Chronicle. Headline + URL + ≤200-char excerpt only.

Spec 175 rss crawler for Oil & Gas Journal. Headline + URL + ≤200-char excerpt only.

Spec 175 html-scrape crawler for PHMSA. Headline + URL + ≤200-char excerpt only.

Spec 175 rss crawler for ProPublica. Headline + URL + ≤200-char excerpt only.

Spec 175 rss crawler for Reuters. Headline + URL + ≤200-char excerpt only.

Spec 175 rss crawler for Texas Tribune. Headline + URL + ≤200-char excerpt only.

Deterministic Leiden community detection over the jargon (Term) subgraph

Graph projection of PostgreSQL data to AuraDB Neo4j

NTSB pipeline accident/incident investigations and safety recommendations via CAROL API

Spec M6 operator-match-backfill (NTSB/IAN fuzzy match → matchedOperatorId) + spec 098 terms-backfill (CFR definition sections → Term table). Both DB-only, idempotent.

Regex-extracts NTSB rec numbers (P-NN-NNN) and GAO/OIG report numbers from federal_register_documents.textPlain; populates linkage tables. Also parses OversightRecommendation rows from new report bodies.

GAO + DOT OIG report corpus expansion. Pulls from congressional_mandates.fulfillmentDocs, govinfo.gov GAOREPORTS, and oig.dot.gov audits.

Drug & Alcohol Management Information System operator + business-unit lists — auto-fetched XLSX

Enforcement guidance organized by CFR section with violation examples and evidence requirements

FAQ document: Batch 1 FAQs (PHMSA-2019-0225)

FAQ document: Batch 2 RIN-1 FAQs

FAQ document: DIMP FAQs

FAQ document: Gas Transmission Integrity Management FAQs

FAQ document: Gathering Pipelines FAQs

FAQ document: Gathering Type C & R FAQs

FAQ document: Liquid IM FAQs

FAQ document: LNG Operations FAQs

FAQ document: Temporary-Permanent Repair FAQs

PDF/XLSX/CSV/DOCX files downloaded from phmsa.dot.gov — text extracted, CFR references linked

Enriched incident data with significance flags, narratives, MAP-7 causes, costs — 10 XLSX files by system type and era

Inspection Activity question sets for pipeline safety audits

Safety Condition Reports (SRCR) + Integrity-Assurance / IM notification leading indicators — auto-fetched XLSX

Browser-based page crawler for phmsa.dot.gov metadata extraction

Operator annual report datasets (gas distribution / transmission / gathering / hazardous liquid / LNG / UNGS) — auto-fetched XLSX/ZIP

Pipeline enforcement case data from PRIMIS

Official regulatory interpretation letters from the PHMSA Office of Pipeline Safety (49 CFR Parts 192 & 195)

Federal Register notices and advisory bulletins from PHMSA covering pipeline safety (49 CFR Parts 186–199). Source: Federal Register API.

17K+ pipeline operators with system type classifications — source of truth for operator identity

Master list of PHMSA operator identification numbers — 17K+ operators

Special permits and state waivers from PHMSA: issued, denied, withdrawn/expired/terminated (6 source pages)

XML sitemap change detection for phmsa.dot.gov (~8,296 URLs)

Event-driven sync orchestration — triggers scrapers based on detected page changes

Pages in our database not present in the PHMSA sitemap XML — discovered via link crawling, monitored for re-crawl

PHMSA PRIMIS enforcement cases and orders — structured enforcement records with operatorId linking

PHMSA regulatory dockets, documents, and public comments via regulations.gov API

Structured deadlines extracted from CFR requirement text — frequencies, intervals, calendar dates, event triggers

Parent-child operator groupings across 8 safety programs — maps submitting OPIDs to primary OPIDs

Spec 204 GDELT DOC 2.0 search-discovery backend availability + response-shape monitor (keyless, free, unguaranteed). RED on API outage / non-JSON drift across all queries.

Spec 204 GDELT DOC 2.0 query "("pipeline leak" OR "pipeline failure" OR "pipeline rupture" OR "pipeline explosion" OR "pipeline spill" OR "pipeline blast" OR "pipeline fire" OR "gas line explosion" OR "gas pipeline explosion" OR "crude oil spill" OR "natural gas pipeline" OR "hazardous liquid release") sourcecountry:US". Headline + canonical URL + ≤200-char excerpt only.

LLM-generated qualitative summaries for every CFR section. Generated once by Opus 4.7 backfill, refreshed on upstream change.

Ongoing Smart Interpretation extraction (spec 184 F14 Path B) — rule-based promoters + budget-capped LLM extraction on newly-ingested data.

NTSB section/rec edges (F02/F03), IAN/SCR/IncidentDetail section linking (F08/F09/F10), SPR Operator edges (F12), FR amendments (F13), CongressionalMandate edges (F14), GovInfo section linking (F16), NTSB-Enforcement crossmatch (F21), SP CFR backfills (F04 v2/F05/F05 v2). All DB-only, idempotent.

Fetches special-permit PDFs via Regulations.gov + (optional) PHMSA relay and extracts textPlain so the SP CFR backfills downstream have source text. Bounded to 100/run. Needs REGULATIONS_GOV_API_KEY at runtime.

Structured conditions parsed from special permit grant letters and state waiver documents

Active PHMSA rulemakings from RegInfo.gov Unified Agenda — tracks the regulatory pipeline

Spec 189 — match engine: for every spec-068 FeedItem, evaluates declared AlertRules (F01) + ACTIVE canvas topic profiles (F02) into per-user Alert rows surfaced in-canvas, on /feed, and via email. Event-driven off FeedItem inserts.