Fair Lending Lab
Hypothesis testing on CFPB HMDA mortgage application data. Five hypotheses were preregistered before the data was touched, then estimated with controls: the raw denial-rate gap is +10.7 percentage points, and 44 percent of it survives adjustment for debt-to-income, loan-to-value, loan amount, property value, and income.
Evidence
Screenshots from the live dashboard against Massachusetts 2023 HMDA LAR data.
Five preregistered hypotheses, all rejecting
Observations from the scored Massachusetts 2023 prototype. These are screening signals in observed covariates, not findings of discrimination. See the causal framing section below.
- H1 race disparity, raw. Black non-Hispanic applicants face a risk difference of +10.7 percentage points (95% CI 8.9 to 12.4) versus White non-Hispanic applicants for first-lien conventional owner-occupied home-purchase loans (two-proportion z = 18.194, p = 2.9e-74). Stratified by income band with BH-FDR adjustment, the disparity persists in every income band.
- H1 race disparity, adjusted. Controlling for debt-to-income, combined loan-to-value, loan amount, property value, and income, the gap falls to +4.73 percentage points (95% CI 3.26 to 6.20), an odds ratio of 2.41 (95% CI 2.00 to 2.90). 44.4 percent of the raw gap survives adjustment (bootstrap 95% CI 37.4 to 52.1 percent). Adding lender and metro fixed effects takes it to +3.55 pp, or 33.3 percent surviving. See the adjusted-model section below.
- H2 ethnicity disparity. Hispanic applicants face a +5.5 percentage point risk difference (95% CI 4.4 to 6.7) versus White non-Hispanic applicants (two-proportion z, p < 0.001). This one has not yet been re-estimated with controls; only H1 has.
- H3 rate spread on priced loans. Among priced originated loans, Black borrowers carry a higher mean rate spread than White borrowers, Hedges g = 0.42 (95% CI 0.35 to 0.49), Welch t p = 6.3e-22, confirmed by Mann-Whitney and a 5,000 permutation test plus a conjugate Bayesian sensitivity.
- H4 lender effect. Denial rates differ materially across the top 10 lenders by application volume in this market, F(9, 15,519) = 56.78, p = 1.4e-102, eta squared = 0.032. Pairwise two-proportion z tests are reported with BH-FDR and Bonferroni adjustment.
- H5 low income residual. Within the lowest income band (under $50k applicant income), the Black vs White disparity is +18.5 percentage points (95% CI 5.4 to 31.6, p = 0.001), a screening signal that the gap is not fully explained by a coarse income control.
- Family-wise correction. All five primary tests reject at BH-FDR q = 0.05 and at Bonferroni alpha / m.
Problem
The HMDA Loan Application Register is one of the largest public datasets in financial services. Reading it well requires more than a t test. Sample sizes are big enough that any difference becomes statistically significant, so power analysis, effect sizes, multiple-comparison correction, and explicit causal framing are what separate a credible BI analysis from a misleading one.
The project needed to demonstrate a disciplined inferential workflow on a sensitive policy dataset: preregistered hypotheses with directional H0/H1, multiple statistical methods that have to agree, family-wise correction, effect sizes with confidence intervals, and a clear separation between association and causation.
Users and decisions
Fair-lending and compliance analysts are the intended users. The dashboard helps them screen observed disparities, compare effect size and uncertainty across hypotheses, and decide where matched supervisory data, loan-file review, or further modeling is warranted.
The dashboard is portfolio methodology. It is not a regulatory finding, a discrimination claim against any lender or applicant pool, or a substitute for the matched-data analyses used in actual fair-lending review.
Preregistered tests over 41,287 real HMDA applications
A Python backend ingests the CFPB FFIEC HMDA LAR for a chosen state and year, curates first-lien conventional owner-occupied home-purchase applications into a Postgres fact table, and runs five preregistered hypotheses through a layered method stack (parametric, non-parametric, permutation, bootstrap, Bayesian sensitivity). Results are cached as JSONB and exposed by a read-only FastAPI service.
A Next.js frontend renders the same data in a clean analyst console: KPI tiles, a lead-finding callout, a disparity-ruler forest plot, per-hypothesis cards with primary and secondary tests, stratified sensitivity tables, a family-wise correction view, and a methods tab. Backend deploys to a Linux VPS behind nginx via systemd, frontend deploys to Cloudflare Pages.
Architecture
Data flow
The pipeline pulls a state-year LAR from the CFPB Data Browser CSV endpoint, projects the 99 raw columns down to the 34 needed for analysis, and COPYs the raw rows into a Postgres staging table. A curated SQL step filters to comparable applications and engineers analysis columns (race rollup, ethnicity rollup, income band, loan amount band, priced-loan flag, denial flag).
Each preregistered hypothesis pulls its own slice, runs the full method stack, writes a primary plus secondary result block plus assumption checks plus power block to the analysis-runs log, and upserts a denormalized JSON payload to the results cache. The API serves the cache, the dashboard hydrates from the API on load.
Tools used
Key features
- Five preregistered hypotheses with explicit H0, H1, direction, and pre-test effect-of-interest target.
- Method stack: Welch t, Mann-Whitney with rank-biserial r, one-way ANOVA with eta squared and omega squared, Kruskal-Wallis with epsilon squared, two-proportion z, Wald risk-difference CI, Haldane-Anscombe-corrected odds ratio CI, 5,000 sample permutation test, 2,000 sample percentile bootstrap, and a conjugate Normal Inv-Chi-Sq Bayesian sensitivity with BIC-approximation Bayes factor.
- Family-wise correction: Benjamini-Hochberg FDR at q = 0.05 and Bonferroni FWER at alpha / m across the five primary tests.
- Stratified sensitivity for the headline H1 across income bands, with BH-FDR adjustment over strata.
- Pairwise post-hoc tests for the lender-effect ANOVA, with BH-FDR and Bonferroni adjustment over the 45 pairwise comparisons.
- Power and minimum detectable effect reported for every test so a non-rejection is interpretable.
- Causal framing caveat on every hypothesis: HMDA omits credit score and full underwriting, so reported disparities are screening signals, not findings of discrimination.
- Deterministic seed across NumPy, SciPy, permutation, bootstrap, and pandas sampling; the notebook reproduces bit identically.
- Pytest plus Hypothesis property tests on the stats helpers, plus a registry sanity test that scans hypothesis text for forbidden characters.
- CI workflow that lints with ruff, runs pytest against a real Postgres service, builds the Next.js frontend, and scans the repo for forbidden em or en dashes.
What this analysis can and cannot claim
Appropriate use: portfolio demonstration of hypothesis testing on a real public dataset, including power analysis, effect-size reporting, multiple-method validation, multiplicity correction, and explicit limitation framing.
Inappropriate use: regulatory determinations, discrimination findings against any specific lender, legal claims against any borrower group, or any operational decision that would normally require matched supervisory data plus loan-file audit pairs.
What the analysis can claim. The reported disparities are statistical associations in HMDA-observed covariates. They are consistent with a real residual disparity after partial adjustment, with omitted-variable bias from credit-quality features HMDA does not contain, and with selection effects in which lenders different groups apply to. The job here is to surface that pattern with calibrated uncertainty, not to assign cause.
What it would take to claim discrimination. Matched supervisory HMDA with credit-bureau records, loan-file review, audit-pair testing, or a counterfactual design. None of those are in scope for a public-data portfolio project, which is why every hypothesis card in the dashboard carries the same caveat in plain language.
Controls cut the disparity from +10.7 pp to +4.73 pp
Headline numbers from the live Massachusetts 2023 dataset. Re-runs against any other state or year are a one-line config change.
The adjusted model
The original analysis reported a raw marginal disparity. This is the same comparison with the underwriting variables HMDA actually carries, fitted as a logistic regression on the H1 cohort of 24,819 applications (1,806 Black non-Hispanic, 23,013 White non-Hispanic). Each row adds controls to the row above it.
- Raw, no controls. Risk difference +10.66 pp. Odds ratio 3.374 (95% CI 2.775 to 4.101, cluster-robust by lender). This reproduces the published +10.7 pp.
- Primary specification. Debt-to-income, combined loan-to-value, log loan amount, log property value, log income. Risk difference +4.73 pp (95% CI 3.26 to 6.20). Odds ratio 2.407 (95% CI 1.996 to 2.901). 44.4 percent of the raw gap survives.
- Plus automated underwriting result. +4.88 pp (3.39 to 6.37). 45.8 percent surviving.
- Plus metro fixed effects. +4.38 pp (2.92 to 5.83). 41.1 percent surviving.
- Plus sex and applicant age. +3.72 pp (2.28 to 5.16). 34.9 percent surviving.
- Fullest specification, adding lender fixed effects. +3.55 pp (2.33 to 4.76). 33.3 percent surviving. Lenders with fewer than 50 applications are pooled into a single level, so 13.5 percent of the cohort is not being compared within an individual lender.
Which control does the work. Entered alone, combined loan-to-value leaves 81.0 percent of the gap, income leaves 76.9 percent, and debt-to-income leaves 63.3 percent. Debt-to-income and combined loan-to-value together account for most of the reduction, taking the gap to +4.64 pp on their own. A complete-case sensitivity that drops the 380 rows with any missing control gives +4.67 pp, so the result is not an artifact of imputation.
Reproducing it. From the platform/ directory: PYTHONPATH=. python -m flab.analysis.adjusted_model --csv data/raw/hmda_2023_MA.csv. The bootstrap is seeded, so the interval reproduces exactly. It defaults to 500 replicates and takes about six minutes; pass --n-boot 50 for a fast check. Output is written to data/processed/adjusted_model.json.
Limitations
HMDA omits credit score, full underwriting detail, property appraisal, and post-application history. Massachusetts 2023 is one state-year of one product line. The disparities here are statistical associations, not causal findings.
Restricting H3 to priced originated loans is a conditioning-on-collider risk: the same underwriting that produces a higher denial rate may also push observed borrowers toward the priced segment. The result is informative about the priced-loan population, not about the underlying borrower population.
The original headline was unadjusted, and the method stack did not fix that. The +10.7 pp figure is a raw marginal comparison of denial rates. Running Welch t, Mann-Whitney, a 5,000-sample permutation test, a 2,000-sample bootstrap, and a Bayesian sensitivity over that same unadjusted quantity buys robustness to distributional assumptions, which at n = 41,287 was never the binding constraint. Omitted-variable bias was the binding constraint, and every layer in the stack inherited it. Layered agreement meant the methods agreed about a marginal difference, not that the difference survived controls. The adjusted model below was added for that reason.
Adjustment does not make this causal. HMDA carries no credit score, no reserves, no appraisal detail, and no compensating factors, so the surviving 4.73 pp is an upper bound on any lender effect plus whatever the unobservables carry. Automated-underwriting result, loan amount, and combined loan-to-value are themselves partly downstream of lender behavior, so specifications that include them risk mediator or collider bias in either direction. That is why the financials-only specification is reported as primary and the underwriting layer is shown separately rather than folded in.
Decisions and rejected alternatives
Method breadth over covariate adjustment, which was the wrong trade. I built five inferential layers across one unadjusted estimand instead of one adjusted model. Breadth demonstrates range; a single logistic regression carrying the HMDA controls demonstrates judgment, and judgment was what the question needed. That model has since been fitted and is reported above, and it changed the headline: 44.4 percent of the raw gap survives controls, not all of it. The robustness stack is still worth having, but it answers a question that was never in doubt.
Multiplicity correction that cannot change a decision. BH-FDR and Bonferroni are applied across five primary tests whose p-values sit far below any threshold a correction could move them across. No correction changes a single conclusion. The real forking-paths exposure lives in the income-band stratification and the H5 subgroup slice, whose confidence interval spans 5.4 to 31.6 percentage points, and nothing in the current design records how many cuts were examined before that one was reported.
A reported p-value was a floating-point artifact. H1 was published as p < 1e-300. That is not a p-value, it is double-precision underflow: the true two-sided value computed from the z statistic is 2.9e-74. The conclusion is unaffected, since both are far below any threshold in use, but reporting a number produced by the limits of the float type rather than by the data is the kind of detail a careful reader checks. The other hypotheses are reported as p < 0.001 pending the same recomputation.
An ANOVA F-test on a binary outcome for H4. Denial is 0/1, so F(9, 15,519) is the wrong instrument even though the pairwise two-proportion z tests underneath it are correct. The lender effect is also unadjusted, which confounds lender behavior with the composition of each lender's applicant pool.
Preregistering the hypotheses, which I would do again. Fixing H0, H1, direction, and the effect of interest before touching the data is the single decision on this project that most constrains me, and it is the reason the caveats above can be stated honestly rather than discovered by a reviewer.
What I would build next
Add a propensity-score matched-pair sensitivity analysis on the observed covariates, a logistic regression with all available HMDA controls, and an MSA-level fixed-effect model. Add multi-year ingest so trend tests (did the disparity move after a specific rule change) become single-config queries. Extend to additional state-year files by parameter; the pipeline is dataset agnostic.