# Data Sources and Versioning Strategy

This document outlines the data sources used for the Net Worth and Income Comparator, how data is parsed/adjusted, and the architectural plan for managing public vs. secret/private data streams.

---

## 1. Data Sources

### U.S. Household Income (CPS ASEC 2023 / 2024)
- **Source**: U.S. Census Bureau, Current Population Survey (CPS) Annual Social and Economic Supplement (ASEC).
- **Table Reference**: HINC-02 (Age of Householder by Total Money Income).
- **Data Definition**: Pre-tax total household money income. Includes wages, business earnings, interest, dividends, social security, pension payments, and other regular cash streams of all household members.
- **Adjustments**:
  - Income ranges are interpolated between key percentiles (10th, 25th, 50th, 75th, 90th, 95th, 99th) to calculate exact percentile scores.
  - Adjusted for Household Size (scale factors derived from national poverty/income statistics).

### U.S. Household Net Worth (SCF 2022)
- **Source**: Federal Reserve Survey of Consumer Finances (SCF) 2022.
- **Data Definition**: Total assets minus total liabilities. Assets include financial assets (bank accounts, stocks, retirement accounts) and non-financial assets (primary residence, other real estate, vehicles, business equity). Liabilities include mortgages, student loans, credit card balances, and other debt.
- **Adjustments**:
  - Grouped by householder age brackets (Under 35, 35-44, 45-54, 55-64, 65-74, 75+).
  - Percentiles are interpolated log-linearly due to the extreme right-skew of wealth distributions.

### Education-Level Slicing (Earmarked / Paywalled)
- **Source**: U.S. Census Bureau ASEC tables for Educational Attainment (e.g., Table PINC-03).
- **Education Brackets**: GED/No Diploma, High School Diploma, College (Associate/Bachelor's), Grad School (Master's/Professional), PhD.
- **Integration**: To be integrated as an optional dashboard input. A paywall logic will prompt users when attempting to unlock comparative charts filtered by education level.

---

## 2. Public vs. Private Repository Mirroring Strategy

To keep the most current/production version of our dataset **SECRET** (e.g. for premium subscribers) while sharing a public codebase with slightly older data (1-2 versions/months behind), we will implement a dual-repository workflow:

```mermaid
graph TD
    A[Private GitHub Repo: networth-income-comparator] -->|Contains latest code & data_current.json| B(Production Deployment)
    A -->|GitHub Actions Sync Trigger| C{Version Lag & Filter Step}
    C -->|Replaces data_current.json with data_delayed.json| D[Public GitHub Repo: networth-income-comparator-public]
    D -->|Deploy Public Demo| E(Public Demo Deployment)
```

### Git & Sync Workflow Details:
1. **Developer Work**: Code and datasets are committed to this **private** repository (`networth-income-comparator`).
2. **Data Structure**:
   - `data_current.json`: Contains the latest, up-to-date data.
   - `data_delayed.json`: Contains the dataset from 1-2 versions prior (e.g. 1 month old or historical baseline).
3. **Automated Mirror Script**:
   - A GitHub Action workflow `.github/workflows/mirror.yml` runs on pushes to the `main` branch.
   - It checks out the code, copies `data_delayed.json` to overwrite `data_current.json` (so the public app only sees the delayed data), deletes any private configuration/files, and pushes the code to a separate **public** GitHub repository.
   - This ensures the public repository has the latest UI enhancements, bug fixes, and features, but is always running on the delayed/free tier dataset.