# AUDIT — resolution-equity Honest self-critique of the resolution-equity case study. Originally written for the 2026-04-20 rebuild; revised 2026-07 for a full cold-run rebuild on corrected, hour-grain data (see §10 for the dated addendum). Voice is pragmatic, not paper-cadence. Readers interested in the publication-style write-up should go to `MANUSCRIPT.md`. ## 1. What this showcase does and does not claim **It claims**: for the first-of-each-month sub-sample of NYC 311 noise-related complaints between 2020-01-01 and 2026-06-30, the measured mean resolution time at the community-board × month level is **not** patterned in the direction a naive equity story would predict. Bronx resolves faster than Manhattan; within-borough heterogeneity dwarfs between-borough heterogeneity (Theil T_between / T_overall = 8.68%). Seasonal variation (~139.7 hours peak-to-trough amplitude, 11.4% of variance share) exceeds the between-borough gap in absolute magnitude, though residual month-to-month variation (83.5% of variance share) dominates the series. **It does not claim**: - Causal identification. All regressions are descriptive conditional associations. Treatment here is a composite construct (borough demographic exposure) that co-varies with dozens of unobservables; the coefficients are correlations with fixed-effects structure, not policy counterfactuals. - Tract-level equity. Our analysis is community-board × month. A tract-level analysis (~2,300 × 78 ≈ 179,000 cells) per the modernization plan would require a tract↔community-district crosswalk that `nyc311` does not ship and that we did not add under this branch. - Completeness of the 311 system. Only resolved complaints enter the analysis (99.1% of the 164,515-record pull had `closed_date` populated; a further 1,156 rows (0.7%) were dropped as `closed_date <= created_date` or duration > 365 days, leaving 161,862 usable). Complaints that never resolve are excluded. ## 2. Hour-grain resolution latency — RESOLVED UPSTREAM (nyc311 1.0.4) *Status (2026-07)*: **Closed** — [`nyc311` 1.0.4](https://github.com/random-walks/nyc311/releases/tag/v1.0.4) fixes a `bulk_fetch` CSV-cache defect that dropped `closed_date` and truncated `created_date` / `closed_date` to whole calendar dates. This study now pins `nyc311 >= 1.0.4` and computes resolution latency at hour grain (`closed_date` minus `created_date`) straight from `nyc311.pipeline.bulk_fetch`. `notebooks/01_load_and_explore.py` fetches the full-corpus stream and filters to day-1-of-month rows client-side; there is no hand-rolled Socrata pagination. Original gap (preserved for context): before 1.0.4 the on-disk CSV cache quantized every timestamp to midnight, so every latency this study computed collapsed to an integer number of days. The published April 2026 write-up therefore reported resolution time in **days**, not hours — a silent data defect, not a modeling choice. The 1.0.4 patch preserves full second-resolution timestamps in the cache, and this cold-run rebuild is the first pass to measure resolution time at hour grain (median 0.86 hrs, mean 39.06 hrs across 161,862 usable records). ## 3. Day-1-of-month sampling Notebook 01 calls `nyc311.pipeline.bulk_fetch` over the 2020-01 → 2026-06 window and keeps the 164,515 records whose `created_date` fell on the 1st of a calendar month, filtered client-side. This keeps the first-run round-trip short, at the cost of a within-month sampling bias we have not formally corrected. Two things make the bias small in practice for this analysis: - Resolution-time distributions are agency-level — DEP and NYPD SLA deadlines are the same on every day of the month — so first-of-month draws are not atypical on the outcome dimension. - Monthly means are still computed per community-board × month cell, and the minimum-5-complaints-per-cell filter excludes cells where sampling noise would swamp the signal. After the filter we retain 4,559 of 5,382 possible cell-months (84.7%). We did *not* bootstrap a sampling-bias correction or run the full panel to confirm the bias is small. A follow-up run with a richer sample (days 1-14 of each month) is the natural next iteration. ## 4. Why statsmodels OLS instead of `factor_factory.engines.panel_reg` `factor_factory.engines.panel_reg.estimate` in 1.0.2 builds its regressor matrix from the `Panel.outcome_col` plus `regressors=` tuple. The engine's record-view flow attaches covariates as per-record attributes; the outcome is a *count* (number of records per cell). Our outcome is a *custom float* (mean resolution hours within a cell), not a raw count. Threading a custom float outcome through the engine requires either (a) constructing a synthetic record set where cell weights encode the mean — lossy — or (b) calling `pyfixest` directly outside the engine wrapper. We chose the simplest faithful path: `statsmodels.formula.api.ols` with HC1-robust SE for the pooled specs and cluster-on-board SE for the TWFE spec. The reporting discipline (SE, 95% CI, *t*, exact *p*, R², *n*, df) matches factor-factory's `PanelRegResult` field-for-field (see `artifacts/twfe_results.json`), so downstream automation is not blocked. **Upstream fix**: a future `factor-factory` release that accepts a pre-computed `outcome_col` on a `Panel` built from a pandas DataFrame (rather than requiring record aggregation) would remove this engine-fit friction. ## 5. Spatial geometry — real centroids Notebook 04 places each community board at its `nyc-geo-toolkit` representative-point centroid — `centroids_from_boundaries( layer='community_district', representative=True)`, which returns `shapely.representative_point()` rather than the bare geometric centroid, guaranteed to fall inside the polygon (this matters for non-convex CBs along the Brooklyn / Queens shorelines, e.g. Coney Island CB 14, Far Rockaway CB 14). An earlier build used a hash-placed-centroid shim; that shim is retired and no longer appears anywhere in the pipeline. **Current Moran's *I* on pooled resolution hours** (real centroids, extended 2020-01 → 2026-06 window, 999 permutation reps): | band (km) | mean neighbors | Moran's *I* | z | p_perm | |---|---:|---:|---:|---:| | 1 km | 0.00 | undefined | — | — | | 2 km | 0.44 | +0.102 | 0.44 | .380 | | 5 km | 5.90 | +0.107 | 1.77 | .055 | **Neither band reaches conventional significance.** The 1 km band is undefined (no CB pair falls within 1 km at community-board scale); the 2 km band is clearly non-significant (p = .380); the 5 km band is marginal (p = .055), just above α = .05. Both point estimates are small and positive but the permutation null cannot be rejected. LISA at 2 km labels all 59 matched boards "ns" — no HH/LL/HL/LH clusters. The honest current read is **no statistically significant spatial clustering**. Note: an interim note drafted against the pre-rebuild artifacts (broken, shorter window) expected the 5 km band to clear significance (p ≈ .03). That number was an artifact of the day-grain, 2020-2024 run; on the corrected hour-grain, extended-window data the 5 km band is only marginal and the conclusion is a null, not a boroughwide-clustering finding. **Unmatched CBs**: 10 of the 69 panel community boards do not match the 59 geometries shipped by nyc-geo-toolkit (pseudo-CBs like "UNSPECIFIED BRONX" and airport catchment zones). Those rows drop out of the spatial analysis but remain in the panel regressions in notebook 03. ## 6. Engine / adapter inventory | Engine | Family | Status | Notes | |---|---|---|---| | `factor_factory.engines.stl` | STL | used | Ran directly on city-monthly series; not through Panel adapter | | `factor_factory.engines.changepoint` | PELT | used | ruptures under the hood | | `factor_factory.engines.inequality` | Theil | *partially* | TheilEngine's canonical path wants a count-outcome Panel; we reproduced the decomposition math inline (code path verified against engine formula) | | `factor_factory.engines.panel_reg` | TWFE | *not used* | See §4 above | | `factor_factory.engines.spatial` | Moran's I / LISA | *not used* | Needs a Panel with `(latitude, longitude)` record-view; our CB-aggregated outcome is not a per-record panel. Hand-rolled distance-band version substituted. | | `nyc311.pipeline.bulk_fetch` | — | used | 1.0.4+ preserves full-timestamp `closed_date` in the cache; hour-grain latency; §2 resolved | | `nyc_geo_toolkit.centroids_from_boundaries` | geometry | used | representative-point mode; real centroids; see §5 | | `statsmodels.formula.api.ols` | OLS | used | HC1 + cluster-on-board SE for the four equity specs | Net: factor-factory earns 2 of 5 expected uses. The gap is primarily an API contract issue (engines assume count-outcome Panels built from raw records; we have pre-aggregated float outcomes). A 1.1.x engine-Panel adapter that accepts pre-aggregated outcomes would close this gap for *every* aggregate-outcome case-study this ecosystem will ever host. ## 7. Reproducibility checks Checked: - `uv run jellycell run` over notebooks 01-05 re-runs them cleanly on a cold cache (verified manually this session). - `uv run jellycell render` produces HTML under `site/` for all five notebooks + an index. - `uv run jellycell lint` exits 0 with no violations. - The tearsheet export writes five files under `manuscripts/tearsheets/` without error. - All `artifacts/*.json` are stdlib-only JSON (no NumPy-specific serializers) and re-readable with plain `json.loads`. Not checked (deferred): - Byte-identical regeneration of `FINDINGS.md` across two clean runs. The findings tearsheet uses fixed override values so it regenerates deterministically, but we have not run the diff-empty check twice. ## 8. Reproducibility notes A few implementation choices are shaped by the reproducible-notebook toolchain rather than the analysis itself, and are recorded here so a reader regenerating the study understands them: - Every notebook cell imports what it uses inline (no shared setup cell), so cells cache and re-run independently. - Figures are inlined as static images for portability across notebook viewers. - All committed `artifacts/*.json` are plain stdlib-serializable JSON (no NumPy-specific encoders), re-readable with `json.loads`. ## 9. What would the next iteration look like 1. **Descriptor-level stratification** — the highest-value follow-up. The Bronx-faster gap is most plausibly a complaint-descriptor-mix artifact: Bronx volume skews to "Loud Music / Party" (routes to NYPD, closes fast at scene); Manhattan skews to commercial / construction descriptors (route to DEP with multi-day SLA targets). The cleaned record stream already carries the `descriptor` column (`artifacts/clean_records.parquet`); a within-descriptor panel that compares resolution times across boroughs would test whether the gap survives descriptor control or dissolves into a routing / composition artifact — the load-bearing question for any equity claim here. 2. **Tract-level panel** via `nyc-geo-toolkit` crosswalk (~30 more observations per borough; dilutes within-borough Theil slightly). 3. **Tract-level ACS covariates** (not just borough): real Oaxaca-Blinder decomposition into race, poverty, age endowments becomes well-defined. 4. **Sampling fidelity**: pull days 1-14 of each month to reduce within-month sampling noise. 5. **Additional outcome**: `pct_same_day` as a robustness DV. Already computed per cell (column `pct_same_day` on `artifacts/panel.parquet`); regress it with the same four M1-M4 specs and report alongside resolution-hours. (The two items that appeared here in the April 2026 revision — `closed_date` upstream in `nyc311` and real CB centroids — are both resolved as of this rebuild; see §2, §5, and the §10 addendum.) ## 10. Addendum — 2026-07 cold-run rebuild This revision documents a full cold-run rebuild of the study. The prior (April 2026) run shipped with a silently broken data cache; every number in it should be treated as superseded by the current artifacts. Four things changed: 1. **Hour-grain resolution latency (nyc311 1.0.4).** The published run computed resolution time in whole **days** because the `bulk_fetch` CSV cache dropped `closed_date` and truncated timestamps to calendar dates. `nyc311` 1.0.4 fixes the cache; the study now pins `nyc311 >= 1.0.4` and measures latency at hour grain (median 0.86 hrs, mean 39.06 hrs). See §2. 2. **Extended window.** The panel now spans 2020-01 → 2026-06 (78 months) rather than 2020-2024 (60 months): 164,515 day-1 records fetched, 161,862 usable, 4,559 cell-months across 69 community boards. Headline gap widened from the day-grain estimate to a 64.6-hour Bronx-faster gap (Bronx 8.63 hrs vs Manhattan 73.24 hrs). 3. **Real-centroid spatial re-run.** Moran's *I* was recomputed on `nyc-geo-toolkit` representative-point centroids over the extended window. The result is **no statistically significant spatial clustering** (2 km p = .380; 5 km p = .055, marginal; 1 km undefined; LISA all-ns). This overturns an interim note that expected a significant 5 km band — that expectation was computed on the broken, shorter-window artifacts. See §5. 4. **Centroid-merge label-format fix.** The community-board labels in the panel are normalized to the format the `nyc-geo-toolkit` centroid layer keys on, so cells merge to geometry correctly; 59 of 69 boards now match (10 pseudo-CB / catch-all rows remain unmatched, as before). Net effect on the narrative: the direction of the headline (Bronx resolves faster than Manhattan) holds, but the magnitude, the panel dimensions, the inequality shares, the seasonality share, and the spatial finding all changed. The honest current story is a large Bronx-faster gap that is almost entirely within-borough (91%), most plausibly an agency-routing / descriptor-mix artifact, with no significant spatial clustering and no COVID structural break — not a demographic-bias finding.