diff --git a/AGENTS.md b/AGENTS.md index 298bee1..c29546d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,6 +9,41 @@ * **Never** call `refresh_data.py` from within a Task Command. This environment has **no internet access**, so attempts to fetch external data will fail. Data in `data/fred.db` must already be present. +## Policy for Handling UNRATE Null Values + +When generating the UNRATE vs. Oil chart, the UNRATE series may include a `NaN` for the most recent month if that month’s data has not yet been published. To prevent the script from raising a `ValueError: UNRATE contains null values`, follow this policy: + +1. **Clamp to Last Valid Index** + + * After loading `unrate` and `oil` with `fetch_series(...)`, immediately determine the last non-NaN date in the `unrate` DataFrame using: + + ```python + last_date = unrate.last_valid_index() + ``` + * Truncate both `unrate` and `oil` so that any trailing `NaN` row in `unrate` (and corresponding index in `oil`) is removed: + + ```python + unrate = unrate.loc[:last_date] + oil = oil.loc[:last_date] + ``` + * Only after this truncation should you call `validate_series(unrate, oil)`. This ensures that neither DataFrame contains nulls and that they share a valid overlapping date range. + +2. **Rationale** + + * Doing a forward-fill or dropping all NaNs could inadvertently hide missing data or distort recent observations. Clamping to the last officially published month preserves data purity. + * By slicing both series at `last_valid_index()`, you ensure that the chart is plotted only with officially released data and no “estimate” for the current month. + +3. **Implementation in `lagged_oil_unrate_chart_styled.py`** + + * Right after `unrate, oil = fetch_series(start_dt, end_dt, args.db)`, insert: + + ```python + last_date = unrate.last_valid_index() + unrate = unrate.loc[:last_date] + oil = oil.loc[:last_date] + ``` + * Then proceed to `validate_series(unrate, oil)` and the usual plotting logic. + ## Project Overview This repository, **python\_charts**, provides a framework for fetching economic time series from FRED, storing them locally, and producing custom matplotlib charts with configurable parameters. The core components are: @@ -17,6 +52,8 @@ This repository, **python\_charts**, provides a framework for fetching economic * **`Makefile`**: Convenient high‐level targets (`install`, `data`, `plot`, `plot‐custom`, `test`, `clean`) that funnel through `startup.sh` for reproducible builds. * **`scripts/refresh_data.py`**: Downloads UNRATE and DCOILWTICO series into `data/fred.db`—**only run in an internet-connected environment**, not in Codex/CI. * **`scripts/lagged_oil_unrate_chart_styled.py`**: Generates the lagged oil vs. unemployment chart with fixed axes styling and log scale. +* **`scripts/bitcoin_m2_chart.py`**: Produces a Bitcoin vs. global M2 overlay chart with adjustable day offset. +* **`scripts/custom_chart.py`**: Plots arbitrary FRED series stored in `data/fred.db`. Future extensions will add additional custom chart scripts under `scripts/`, all driven by the same bootstrapping and command conventions. @@ -30,7 +67,7 @@ Future extensions will add additional custom chart scripts under `scripts/`, all ./startup.sh ``` - Installs/upgrades dependencies globally and runs the default lagged‐oil chart with an 18-month offset. + Installs/upgrades dependencies globally and runs the default lagged‐oil chart with an 18‐month offset. * **Custom plot**: @@ -41,7 +78,7 @@ Future extensions will add additional custom chart scripts under `scripts/`, all * **Data refresh (offline environments)**: * **Do not** run `refresh_data.py` in Codex/CI — data must already be in `data/fred.db`. - * To update data in an internet-connected shell, run: + * To update data in an internet‐connected shell, run: ```bash python scripts/refresh_data.py @@ -63,7 +100,7 @@ Future extensions will add additional custom chart scripts under `scripts/`, all * **Python**: Target version ≥ 3.8. Use type hints for all function signatures. Follow PEP8 with Black formatting. * **Bash**: `startup.sh` uses `set -euo pipefail`. -* **Makefile**: Use tab-indented commands. All targets should funnel through `startup.sh`. +* **Makefile**: Use tab‐indented commands. All targets should funnel through `startup.sh`. ## Testing & Validation @@ -93,6 +130,4 @@ Future extensions will add additional custom chart scripts under `scripts/`, all ```bash ./startup.sh --offset 18 - ``` - -*End of AGENTS.md — tailored for python\_charts repo* \ No newline at end of file + ``` \ No newline at end of file diff --git a/README.md b/README.md index 915e08c..36f6c7f 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # Python Charts -This initial chart is a self-contained toolkit that: +This initial chart is a self‐contained toolkit that: -* Fetches U.S. unemployment rate (UNRATE) and WTI crude oil price (DCOILWTICO) from FRED -* Stores the series in a local SQLite database (`data/fred.db`) -* Produces a dual-axis plot of unemployment vs. lagged oil price, with customizable lag, date range, and styling +* Fetches U.S. unemployment rate (UNRATE) and WTI crude oil price (DCOILWTICO) from FRED :contentReference[oaicite:1]{index=1} +* Stores the series in a local SQLite database (`data/fred.db`) :contentReference[oaicite:2]{index=2} +* Produces a dual‐axis plot of unemployment vs. lagged oil price, with customizable lag, date range, and styling :contentReference[oaicite:3]{index=3} --- @@ -15,7 +15,7 @@ This initial chart is a self-contained toolkit that: git clone https://github.com/your-org/python_charts.git cd python_charts -# One-command: install deps, download data, and plot +# One‐command: install deps, download data, and plot ./startup.sh --offset 12 --end 2025-05-31 --extend-years 5 ```` @@ -27,8 +27,8 @@ This will: ```bash pip install --upgrade -r requirements.txt ``` -3. Download UNRATE & DCOILWTICO into `data/fred.db` if missing -4. Generate the chart with a 12-month oil lead, extending the x-axis by 5 years +3. Download UNRATE & DCOILWTICO into `data/fred.db` if missing ([Stack Overflow][3], [martinheinz.dev][4]) +4. Generate the chart with a 12‐month oil lead, extending the x‐axis by 5 years ([martinheinz.dev][4], [Real Python][1]) --- @@ -39,115 +39,207 @@ python_charts/ ├── data/ │ └── fred.db # Local FRED snapshot (SQLite) ├── scripts/ -│ ├── refresh_data.py # Download and store FRED series -│ ├── lagged_oil_unrate_chart_styled.py # Plotting script -│ └── pyproject.toml # Packaging metadata -├── Makefile # High-level targets (install, plot, etc.) -├── startup.sh # Installs deps globally, fetches data, runs commands +│ ├── __init__.py # Marks this folder as a Python package +│ ├── common.py # Data‐loading, validation, and save_figure utils +│ ├── refresh_data.py # Downloads and stores FRED series +│ ├── lagged_oil_unrate_chart_styled.py # Offline plot of UNRATE vs. WTI oil with lag +│ ├── bitcoin_m2_chart.py # Plot Bitcoin vs. Global M2 overlay +│ └── custom_chart.py # Generic plotting for arbitrary FRED series +├── outputs/ # Generated chart files (PNGs) +├── Makefile # High‐level targets (install, data, plot, etc.) +├── startup.sh # Installs dependencies, ensures data, runs commands ├── requirements.txt # Python dependencies └── README.md # This file ``` +> **Note:** An empty `scripts/__init__.py` file has been added so that Python treats `scripts/` as a package. Because of this, each script can import `common.py` via: +> +> ```python +> from common import fetch_series_db, save_figure, validate_dataframe, validate_overlap +> ``` +> +> rather than `from scripts.common import …`. ([Real Python][1], [Software Engineering Stack Exchange][2]) + --- ## ⚙️ Usage -### Refresh FRED data +### 1. Refresh FRED Data + +To refresh (or initially populate) the SQLite database with FRED series, run: ```bash ./startup.sh python scripts/refresh_data.py --series UNRATE CPIAUCSL --start 1960-01-01 --output outputs/chart.png ``` -Downloads UNRATE and WTI series (from 1948 to today) into `data/fred.db`. +This command: + +* Connects to FRED via `pandas_datareader` +* Downloads the specified series (e.g., `UNRATE` and `CPIAUCSL`) from the start date through today +* Writes each series to its own table in `data/fred.db` +* Saves a PNG of any default plot (if `--output` is provided) to `outputs/` ([Stack Overflow][3], [GitHub][5]) #### `refresh_data.py` options -| Option | Description | -| ---------- | ----------------------------------- | -| `--series` | One or more FRED series names | -| `--start` | Start date in `YYYY-MM-DD` format | -| `--end` | End date in `YYYY-MM-DD` format | +| Option | Description | +| ---------- | -------------------------------------------------------- | +| `--series` | One or more FRED series names (e.g., UNRATE, DCOILWTICO) | +| `--start` | Start date in `YYYY-MM-DD` format | +| `--end` | End date in `YYYY-MM-DD` format (defaults to today) | -To add new data and plot it later: - -1. Download series: - - ```bash - ./startup.sh python scripts/refresh_data.py --series - ``` - -2. Generate a chart with those series: - - ```bash - ./startup.sh python scripts/custom_chart.py --series - ``` - -### Generate the chart +##### Example: ```bash -# Default 18-month lag -./startup.sh - -# Custom lag, date range, and extension -./startup.sh --offset 6 --start 1980-01-01 --end 2025-05-31 --extend-years 2 +./startup.sh python scripts/refresh_data.py --series UNRATE DCOILWTICO CBBTCUSD GLOBAL_M2 --start 1973-01-01 ``` -The plotting script validates that both series contain data for the -requested period before rendering the chart. +This populates `fred.db` with four series (UNRATE, WTI, Bitcoin, and Global M2). ([Stack Overflow][3], [martinheinz.dev][4]) -If no `--output` path is provided, each script automatically saves the figure to -`outputs/