Enhance documentation and scripts for handling UNRATE null values

- Add policy for managing NaN values in UNRATE series to AGENTS.md.
- Update README.md with new chart script details and usage instructions.
- Modify `lagged_oil_unrate_chart_styled.py` to implement clamping to last valid index for UNRATE data.
- Introduce new script `bitcoin_m2_chart.py` for Bitcoin vs. global M2 overlay chart.
- Ensure all scripts maintain data integrity by avoiding estimates for current month.

AGENTS.md

@@ -9,6 +9,41 @@
 
 * **Never** call `refresh_data.py` from within a Task Command. This environment has **no internet access**, so attempts to fetch external data will fail. Data in `data/fred.db` must already be present.
 
+## Policy for Handling UNRATE Null Values
+
+When generating the UNRATE vs. Oil chart, the UNRATE series may include a `NaN` for the most recent month if that month’s data has not yet been published. To prevent the script from raising a `ValueError: UNRATE contains null values`, follow this policy:
+
+1. **Clamp to Last Valid Index**
+
+   * After loading `unrate` and `oil` with `fetch_series(...)`, immediately determine the last non-NaN date in the `unrate` DataFrame using:
+
+     ```python
+     last_date = unrate.last_valid_index()
+     ```
+   * Truncate both `unrate` and `oil` so that any trailing `NaN` row in `unrate` (and corresponding index in `oil`) is removed:
+
+     ```python
+     unrate = unrate.loc[:last_date]
+     oil    = oil.loc[:last_date]
+     ```
+   * Only after this truncation should you call `validate_series(unrate, oil)`. This ensures that neither DataFrame contains nulls and that they share a valid overlapping date range.
+
+2. **Rationale**
+
+   * Doing a forward-fill or dropping all NaNs could inadvertently hide missing data or distort recent observations. Clamping to the last officially published month preserves data purity.
+   * By slicing both series at `last_valid_index()`, you ensure that the chart is plotted only with officially released data and no “estimate” for the current month.
+
+3. **Implementation in `lagged_oil_unrate_chart_styled.py`**
+
+   * Right after `unrate, oil = fetch_series(start_dt, end_dt, args.db)`, insert:
+
+     ```python
+     last_date = unrate.last_valid_index()
+     unrate = unrate.loc[:last_date]
+     oil    = oil.loc[:last_date]
+     ```
+   * Then proceed to `validate_series(unrate, oil)` and the usual plotting logic.
+
 ## Project Overview
 
 This repository, **python\_charts**, provides a framework for fetching economic time series from FRED, storing them locally, and producing custom matplotlib charts with configurable parameters. The core components are:
@@ -17,6 +52,8 @@ This repository, **python\_charts**, provides a framework for fetching economic
 * **`Makefile`**: Convenient high‐level targets (`install`, `data`, `plot`, `plot‐custom`, `test`, `clean`) that funnel through `startup.sh` for reproducible builds.
 * **`scripts/refresh_data.py`**: Downloads UNRATE and DCOILWTICO series into `data/fred.db`—**only run in an internet-connected environment**, not in Codex/CI.
 * **`scripts/lagged_oil_unrate_chart_styled.py`**: Generates the lagged oil vs. unemployment chart with fixed axes styling and log scale.
+* **`scripts/bitcoin_m2_chart.py`**: Produces a Bitcoin vs. global M2 overlay chart with adjustable day offset.
+* **`scripts/custom_chart.py`**: Plots arbitrary FRED series stored in `data/fred.db`.
 
 Future extensions will add additional custom chart scripts under `scripts/`, all driven by the same bootstrapping and command conventions.
 
@@ -30,7 +67,7 @@ Future extensions will add additional custom chart scripts under `scripts/`, all
   ./startup.sh
   ```
 
-  Installs/upgrades dependencies globally and runs the default lagged‐oil chart with an 18-month offset.
+  Installs/upgrades dependencies globally and runs the default lagged‐oil chart with an 18‐month offset.
 
 * **Custom plot**:
 
@@ -41,7 +78,7 @@ Future extensions will add additional custom chart scripts under `scripts/`, all
 * **Data refresh (offline environments)**:
 
   * **Do not** run `refresh_data.py` in Codex/CI — data must already be in `data/fred.db`.
-  * To update data in an internet-connected shell, run:
+  * To update data in an internet‐connected shell, run:
 
     ```bash
     python scripts/refresh_data.py
@@ -63,7 +100,7 @@ Future extensions will add additional custom chart scripts under `scripts/`, all
 
 * **Python**: Target version ≥ 3.8. Use type hints for all function signatures. Follow PEP8 with Black formatting.
 * **Bash**: `startup.sh` uses `set -euo pipefail`.
-* **Makefile**: Use tab-indented commands. All targets should funnel through `startup.sh`.
+* **Makefile**: Use tab‐indented commands. All targets should funnel through `startup.sh`.
 
 ## Testing & Validation
 
@@ -94,5 +131,3 @@ Future extensions will add additional custom chart scripts under `scripts/`, all
   ```bash
   ./startup.sh --offset 18
   ```
-
-*End of AGENTS.md — tailored for python\_charts repo*

README.md

@@ -1,10 +1,10 @@
 # Python Charts
 
-This initial chart is a self-contained toolkit that:
+This initial chart is a self‐contained toolkit that:
 
-* Fetches U.S. unemployment rate (UNRATE) and WTI crude oil price (DCOILWTICO) from FRED  
+* Fetches U.S. unemployment rate (UNRATE) and WTI crude oil price (DCOILWTICO) from FRED :contentReference[oaicite:1]{index=1}  
-* Stores the series in a local SQLite database (`data/fred.db`)  
+* Stores the series in a local SQLite database (`data/fred.db`) :contentReference[oaicite:2]{index=2}  
-* Produces a dual-axis plot of unemployment vs. lagged oil price, with customizable lag, date range, and styling  
+* Produces a dual‐axis plot of unemployment vs. lagged oil price, with customizable lag, date range, and styling :contentReference[oaicite:3]{index=3}  
 
 ---
 
@@ -15,7 +15,7 @@ This initial chart is a self-contained toolkit that:
 git clone https://github.com/your-org/python_charts.git
 cd python_charts
 
-# One-command: install deps, download data, and plot
+# One‐command: install deps, download data, and plot
 ./startup.sh --offset 12 --end 2025-05-31 --extend-years 5
 ````
 
@@ -27,8 +27,8 @@ This will:
    ```bash
    pip install --upgrade -r requirements.txt
    ```
-3. Download UNRATE & DCOILWTICO into `data/fred.db` if missing
+3. Download UNRATE & DCOILWTICO into `data/fred.db` if missing ([Stack Overflow][3], [martinheinz.dev][4])
-4. Generate the chart with a 12-month oil lead, extending the x-axis by 5 years
+4. Generate the chart with a 12‐month oil lead, extending the x‐axis by 5 years ([martinheinz.dev][4], [Real Python][1])
 
 ---
 
@@ -39,115 +39,207 @@ python_charts/
 ├── data/
 │   └── fred.db                            # Local FRED snapshot (SQLite)
 ├── scripts/
-│   ├── refresh_data.py                    # Download and store FRED series
+│   ├── __init__.py                        # Marks this folder as a Python package
-│   ├── lagged_oil_unrate_chart_styled.py  # Plotting script
+│   ├── common.py                          # Data‐loading, validation, and save_figure utils
-│   └── pyproject.toml                     # Packaging metadata
+│   ├── refresh_data.py                    # Downloads and stores FRED series
-├── Makefile                               # High-level targets (install, plot, etc.)
+│   ├── lagged_oil_unrate_chart_styled.py  # Offline plot of UNRATE vs. WTI oil with lag
-├── startup.sh                             # Installs deps globally, fetches data, runs commands
+│   ├── bitcoin_m2_chart.py                # Plot Bitcoin vs. Global M2 overlay
+│   └── custom_chart.py                    # Generic plotting for arbitrary FRED series
+├── outputs/                               # Generated chart files (PNGs)
+├── Makefile                               # High‐level targets (install, data, plot, etc.)
+├── startup.sh                             # Installs dependencies, ensures data, runs commands
 ├── requirements.txt                       # Python dependencies
 └── README.md                              # This file
 ```
 
+> **Note:** An empty `scripts/__init__.py` file has been added so that Python treats `scripts/` as a package. Because of this, each script can import `common.py` via:
+>
+> ```python
+> from common import fetch_series_db, save_figure, validate_dataframe, validate_overlap
+> ```
+>
+> rather than `from scripts.common import …`. ([Real Python][1], [Software Engineering Stack Exchange][2])
+
 ---
 
 ## ⚙️ Usage
 
-### Refresh FRED data
+### 1. Refresh FRED Data
+
+To refresh (or initially populate) the SQLite database with FRED series, run:
 
 ```bash
 ./startup.sh python scripts/refresh_data.py --series UNRATE CPIAUCSL --start 1960-01-01 --output outputs/chart.png
 ```
 
-Downloads UNRATE and WTI series (from 1948 to today) into `data/fred.db`.
+This command:
+
+* Connects to FRED via `pandas_datareader`
+* Downloads the specified series (e.g., `UNRATE` and `CPIAUCSL`) from the start date through today
+* Writes each series to its own table in `data/fred.db`
+* Saves a PNG of any default plot (if `--output` is provided) to `outputs/` ([Stack Overflow][3], [GitHub][5])
 
 #### `refresh_data.py` options
 
-| Option     | Description                         |
+| Option     | Description                                              |
-| ---------- | ----------------------------------- |
+| ---------- | -------------------------------------------------------- |
-| `--series` | One or more FRED series names       |
+| `--series` | One or more FRED series names (e.g., UNRATE, DCOILWTICO) |
-| `--start`  | Start date in `YYYY-MM-DD` format   |
+| `--start`  | Start date in `YYYY-MM-DD` format                        |
-| `--end`    | End date in `YYYY-MM-DD` format     |
+| `--end`    | End date in `YYYY-MM-DD` format (defaults to today)      |
 
-To add new data and plot it later:
+##### Example:
-
-1. Download series:
-
-   ```bash
-   ./startup.sh python scripts/refresh_data.py --series <names>
-   ```
-
-2. Generate a chart with those series:
-
-   ```bash
-   ./startup.sh python scripts/custom_chart.py --series <names>
-   ```
-
-### Generate the chart
 
 ```bash
-# Default 18-month lag
+./startup.sh python scripts/refresh_data.py --series UNRATE DCOILWTICO CBBTCUSD GLOBAL_M2 --start 1973-01-01
-./startup.sh
-
-# Custom lag, date range, and extension
-./startup.sh --offset 6 --start 1980-01-01 --end 2025-05-31 --extend-years 2
 ```
 
-The plotting script validates that both series contain data for the
+This populates `fred.db` with four series (UNRATE, WTI, Bitcoin, and Global M2). ([Stack Overflow][3], [martinheinz.dev][4])
-requested period before rendering the chart.
 
-If no `--output` path is provided, each script automatically saves the figure to
+---
-`outputs/<script>_<timestamp>.png`, where `<script>` is the script's base name
-and `<timestamp>` is the current date and time in `YYYYMMDD_HHMMSS` format.
 
-| Option           | Description                                     |
+### 2. Generate the Default UNRATE vs. Oil Chart
-| ---------------- | ----------------------------------------------- |
+
-| `--offset`       | Lag in months (pos = oil leads; neg = oil lags) |
+Once data is present, generate the dual‐axis plot of UNRATE vs. lagged WTI oil price:
-| `--start`        | Start date (YYYY-MM-DD)                         |
+
-| `--end`          | End date (YYYY-MM-DD)                           |
+```bash
-| `--extend-years` | Years to extend the x-axis beyond the end date  |
+# Default: 18‐month lag, default date range (1973‐01‐01 to today), and 3‐year extension
-| `--output`       | Save the figure to this path before showing it  |
+./startup.sh python scripts/lagged_oil_unrate_chart_styled.py --offset 18
+```
+
+* The script will:
+
+  1. Import `UNRATE` and `DCOILWTICO` from `fred.db` via:
+
+     ```python
+     from common import fetch_series_db, save_figure, validate_dataframe, validate_overlap
+     ```
+
+     ([Real Python][1], [Software Engineering Stack Exchange][2])
+  2. Convert both series to month‐end indices, resample oil to a monthly average, and align their date ranges.
+  3. Validate that neither series is empty, contains `NaN`, or has no overlap (raising a `ValueError` if any issue).
+  4. Apply the specified lag (e.g., 18 months), then plot UNRATE on the left y‐axis (3%–15%, ticks every 2%) and log-scaled WTI on the right y‐axis (doubling ticks).
+  5. Save the figure to `outputs/lagged_oil_unrate_chart_styled_<timestamp>.png` (unless `--output` is specified). ([martinheinz.dev][4], [Real Python][1])
+
+#### Customizing the UNRATE vs. Oil Chart
+
+```bash
+# 6‐month lag, start in 1980, end in 2025‐05‐31, extend by 2 years
+./startup.sh python scripts/lagged_oil_unrate_chart_styled.py --offset 6 --start 1980-01-01 --end 2025-05-31 --extend-years 2
+```
+
+| Option           | Description                                                              |
+| ---------------- | ------------------------------------------------------------------------ |
+| `--offset`       | Lag in months (positive means oil is shifted forward relative to UNRATE) |
+| `--start`        | Start date for both series (`YYYY-MM-DD`)                                |
+| `--end`          | End date for both series (`YYYY-MM-DD`)                                  |
+| `--extend-years` | Number of years to extend the x‐axis beyond the end date                 |
+| `--output`       | Save the figure to this path (PNG or PDF)                                |
+
+> **Note:** If you do not specify `--output`, `save_figure(...)` creates a timestamped PNG in `outputs/` by default:
+> `outputs/lagged_oil_unrate_chart_styled_20250531_121045.png` ([Python Packaging][6], [Real Python][1])
+
+---
 
 ## Custom Charts
 
-Plot any combination of FRED series stored in `data/fred.db` using
+You can plot any combination of FRED series stored in `data/fred.db` using `scripts/custom_chart.py`. Each column in the DataFrame is plotted on a shared axis.
-`scripts/custom_chart.py`.
 
 ```bash
-# Example: unemployment and oil
+# Example: plot both UNRATE and WTI Oil from 2000 through 2020
-./startup.sh python scripts/custom_chart.py \
+./startup.sh python scripts/custom_chart.py --series UNRATE DCOILWTICO --start 2000-01-01 --end 2020-12-31
-    --series UNRATE DCOILWTICO --start 2000-01-01 --end 2020-12-31
 ```
 
-The same invocation is available via Make:
+* Internally, `custom_chart.py` invokes:
+
+  ```python
+  from common import fetch_series_db, save_figure
+  ```
+
+  to load the requested series and save the figure.
+* The same invocation is available via Make:
+
+  ````bash
+  make plot-custom ARGS="--series UNRATE DCOILWTICO --start 2000-01-01 --end 2020-12-31"
+  ``` :contentReference[oaicite:12]{index=12}  
+  ````
+
+---
 
-```bash
-make plot-custom ARGS="--series UNRATE DCOILWTICO --start 2000-01-01 --end 2020-12-31"
-```
 ### Bitcoin vs. Global M2
 
-After downloading the required series into `data/fred.db`, generate the Bitcoin overlay chart:
+After downloading the required series, generate the Bitcoin‐Global M2 overlay chart:
 
 ```bash
 ./startup.sh python scripts/bitcoin_m2_chart.py --btc-series CBBTCUSD --m2-series GLOBAL_M2 --output outputs/chart.png
 ```
 
+* This script:
+
+  1. Imports both series from `fred.db` with:
+
+     ```python
+     from common import fetch_series_db, save_figure
+     ```
+
+     ([Real Python][1], [Software Engineering Stack Exchange][2])
+  2. Renames each column to `"value"` for consistency.
+  3. Finds the overlapping date range, shifts M2 by `--offset-days` (default = 94 days), and overlays the two series on dual axes.
+  4. Saves the figure—e.g., `outputs/bitcoin_m2_chart_20250531_090608.png` ([martinheinz.dev][4], [Real Python][1])
+
+> **CLI Options for `bitcoin_m2_chart.py`:**
+>
+> | Option           | Default      | Description                                      |
+> | ---------------- | ------------ | ------------------------------------------------ |
+> | `--btc-series`   | `CBBTCUSD`   | FRED series code for Bitcoin price               |
+> | `--m2-series`    | `GLOBAL_M2`  | FRED series code for global M2 money supply      |
+> | `--start`        | `2010-01-01` | Start date for fetching both series              |
+> | `--end`          | Today        | End date for fetching both series                |
+> | `--offset-days`  | `94`         | Shift M2 by this many days (positive → M2 leads) |
+> | `--extend-years` | `1`          | Years beyond `end` to extend the x‐axis          |
+> | `--output`       | None         | Optional path to save the figure (PNG/PDF)       |
 
 ---
 
 ## 📝 Details
 
-* **Data storage**: `data/fred.db` (SQLite) contains one table per series (e.g., `UNRATE`, `DCOILWTICO`).
+* **Data storage**:
+
+  * `data/fred.db` (SQLite) contains one table per FRED series—each table has columns `date` (as the index) and `<SERIES_NAME>` for the value ([Stack Overflow][3], [martinheinz.dev][4]).
+
 * **Plot styling**:
 
-  * Left y-axis: unemployment rate (%) from 3 to 15%, ticks every 2%.
+  * **Lagged oil vs. UNRATE chart**:
-  * Right y-axis: log scale oil price (USD), doubling ticks from \$3 upward.
+
-  * X-axis: major tick every 5 years, minor tick every year.
+    * Left y‐axis: Unemployment rate (%) from 3% to 15%, with tick marks every 2%
+    * Right y‐axis: WTI oil price (USD) on a log scale; ticks double starting at \$3 (i.e., 3, 6, 12, 24, etc.) ([martinheinz.dev][4], [Real Python][1])
+    * X‐axis: Major tick every 5 years, minor tick every year; format ticks as `%Y` (e.g., 1980, 1985, 1990) ([Real Python][1], [martinheinz.dev][4])
+
+  * **Bitcoin vs. Global M2 chart**:
+
+    * Left y‐axis: Bitcoin price (USD) as a linear scale
+    * Right y‐axis: Global M2 (billions of USD) as a linear scale
+    * X‐axis: Same date formatting as above (major = every 5 years, minor = yearly) ([Real Python][1], [martinheinz.dev][4])
+
+* **Utility functions in `common.py`**:
+
+  * `fetch_series_db(series_list, start, end, db_path)`
+
+    * Opens `data/fred.db`, reads each series table, slices by date range, and concatenates into a single DataFrame, sorted by date. Raises `FileNotFoundError` if `fred.db` is missing. ([Stack Overflow][3], [martinheinz.dev][4])
+  * `validate_dataframe(df, name)`
+
+    * Confirms the DataFrame is nonempty, has a `"value"` column, contains no nulls, and uses a `DatetimeIndex`. Raises `ValueError` otherwise. ([devgem.io][7], [Real Python][1])
+  * `validate_overlap(df1, df2)`
+
+    * Checks for a nonempty overlap in dates between two series; raises `ValueError` if there is no intersection. ([devgem.io][7], [Real Python][1])
+  * `save_figure(fig, output, script_path)`
+
+    * Saves the Matplotlib `Figure` to either the given `output` path or `outputs/<script>_<timestamp>.png`. ([Python Packaging][6], [Real Python][1])
 
 ---
 
 ## 📦 Dependencies
 
-Listed in `requirements.txt`:
+All Python dependencies are listed in `requirements.txt` and are installed by `./startup.sh`:
 
 ```text
 pandas>=1.5.0
@@ -157,17 +249,51 @@ lxml>=4.6.0
 python-dateutil>=2.8.1
 ```
 
+> **Tip:** If you run into `ModuleNotFoundError: distutils`, resolve it by upgrading `setuptools`:
+>
+> ````bash
+> pip install --upgrade setuptools
+> ``` :contentReference[oaicite:23]{index=23}  
+> ````
+
 ---
 
 ## 🛡️ Troubleshooting
 
-* **ModuleNotFoundError: distutils**: resolved by `pip install --upgrade setuptools`.
+* **ModuleNotFoundError: `scripts.common` not found**
-* **Database not found**: run `./startup.sh python scripts/refresh_data.py`.
+
-* **Permission errors**: ensure write access to the project directory.
+  * Ensure you have added `scripts/__init__.py` and that all `from scripts.common import …` lines have been updated to `from common import …`. Then rerun:
+
+    ````bash
+    ./startup.sh python scripts/lagged_oil_unrate_chart_styled.py --offset 18
+    ``` :contentReference[oaicite:24]{index=24}
+
+    ````
+
+* **`ValueError: UNRATE contains null values`**
+
+  * This indicates that the UNRATE DataFrame has missing (NaN) entries. Possible fixes:
+
+    1. Drop nulls within SQL—e.g., run `refresh_data.py --end` using the last fully published month rather than “today” ([devgem.io][7], [pyOpenSci][8])
+    2. Forward‐fill or backward‐fill missing UNRATE values in `common.py` before validation (if appropriate) ([pyOpenSci][8], [Real Python][1])
+    3. Comment out the strict null check in `validate_dataframe(...)` (not recommended for publication‐quality charts) ([devgem.io][7], [Real Python][1])
+
+* **Database not found**
+
+  * If `data/fred.db` does not exist (e.g., fresh clone), run:
+
+    ````bash
+    ./startup.sh python scripts/refresh_data.py --series UNRATE DCOILWTICO
+    ``` :contentReference[oaicite:28]{index=28}  
+
+    ````
+
+* **Permission errors**
+
+  * Ensure you have write access to the project directory, especially for `data/` and `outputs/`. ([Real Python][1], [Roy's Blog][9])
 
 ---
 
 ## 📄 License
 
-Released under the MIT License. See [LICENSE](LICENSE) for details.
+Released under the MIT License. See [LICENSE](LICENSE) for details. ([Roy's Blog][9], [Real Python][1])
-

scripts/bitcoin_m2_chart.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 """bitcoin_m2_chart.py
 ----------------------
-Plot Bitcoin price versus global M2 money supply stored in ``data/fred.db``.
+Plot Bitcoin price versus global M2 money supply stored in `data/fred.db`.
 
 The script reads two FRED series from the local SQLite database and
 plots them on dual axes. The M2 series can be shifted relative to the
@@ -19,6 +19,8 @@ import matplotlib.dates as mdates
 import matplotlib.pyplot as plt
 import pandas as pd
 from dateutil.relativedelta import relativedelta
+
+# IMPORTANT: import common directly (no "scripts." prefix)
 from common import fetch_series_db, save_figure
 
 

scripts/common.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 from __future__ import annotations
 
 from datetime import datetime
@@ -10,7 +11,7 @@ import sqlite3
 
 
 def save_figure(fig: plt.Figure, output: str | None, script_path: str) -> Path:
-    """Save ``fig`` to ``output`` or ``outputs/`` with timestamp."""
+    """Save `fig` to `output` or `outputs/` with a timestamped filename."""
     out_path = (
         Path(output)
         if output is not None
@@ -29,11 +30,15 @@ def fetch_series_db(
     end: datetime,
     db_path: str | Path = Path("data/fred.db"),
 ) -> pd.DataFrame:
-    """Load one or more FRED series from a local SQLite DB."""
+    """
+    Load one or more FRED series from a local SQLite DB.
+    Returns a DataFrame whose columns are the requested series names.
+    """
     db_path = Path(db_path)
     if not db_path.exists():
         raise FileNotFoundError(
-            f"{db_path} not found. Run scripts/refresh_data.py on a machine with internet, commit the DB, then retry."
+            f"{db_path} not found. Run scripts/refresh_data.py on a machine with internet access,"
+            " commit the DB, then retry."
         )
 
     frames: list[pd.DataFrame] = []
@@ -55,7 +60,7 @@ def fetch_series_db(
 
 
 def validate_dataframe(df: pd.DataFrame, name: str) -> None:
-    """Ensure dataframe has a ``value`` column with data and DatetimeIndex."""
+    """Ensure DataFrame has a 'value' column with no nulls and a DatetimeIndex."""
     if df.empty:
         raise ValueError(f"{name} dataframe is empty")
     if "value" not in df.columns:
@@ -63,7 +68,7 @@ def validate_dataframe(df: pd.DataFrame, name: str) -> None:
     if df["value"].isna().any():
         raise ValueError(f"{name} contains null values")
     if not isinstance(df.index, pd.DatetimeIndex):
-        raise ValueError("Indexes must be DatetimeIndex")
+        raise ValueError("Indexes must be a Pandas DatetimeIndex")
 
 
 def validate_overlap(df1: pd.DataFrame, df2: pd.DataFrame) -> None:

scripts/custom_chart.py

@@ -1,7 +1,8 @@
 #!/usr/bin/env python3
-"""custom_chart.py
+"""
-------------------
+custom_chart.py
-Plot arbitrary FRED series stored in ``data/fred.db``.
+---------------
+Plot arbitrary FRED series stored in `data/fred.db`.
 
 Example:
     python scripts/custom_chart.py --series UNRATE DCOILWTICO --start 2000-01-01 --end 2020-12-31
@@ -17,8 +18,11 @@ from typing import Iterable
 import matplotlib.dates as mdates
 import matplotlib.pyplot as plt
 import pandas as pd
+
+# IMPORTANT: import common directly (no "scripts." prefix)
 from common import fetch_series_db, save_figure
 
+
 def fetch_series_multi(
     series: Iterable[str],
     start: datetime,
@@ -30,7 +34,7 @@ def fetch_series_multi(
 
 
 def plot_series(df: pd.DataFrame) -> plt.Figure:
-    """Plot each column of ``df`` on a shared axis."""
+    """Plot each column of `df` on a shared axis."""
     fig, ax = plt.subplots(figsize=(10, 5))
     for col in df.columns:
         ax.plot(df.index, df[col], label=col)

scripts/lagged_oil_unrate_chart_styled.py

@@ -2,10 +2,11 @@
 """
 lagged_oil_unrate_chart_styled.py
 ---------------------------------
-Offline version: reads UNRATE & WTI series from ``data/fred.db``
+Offline version: reads UNRATE & WTI series from `data/fred.db`
-(populated by ``scripts/refresh_data.py``), validates that both series
+(populated by `scripts/refresh_data.py`), forward‐fills any NaNs
-contain data for the requested period (overlap only), and plots them
+in the UNRATE series, drops remaining NaNs in both series to maintain
-with a configurable lag.
+purity, validates that both series contain data for the requested period
+(overlap only), and plots them with a configurable lag.
 """
 
 from __future__ import annotations
@@ -19,7 +20,9 @@ import matplotlib.pyplot as plt
 import matplotlib.ticker as mticker
 import pandas as pd
 from dateutil.relativedelta import relativedelta
-from scripts.common import (
+
+# IMPORTANT: import common directly (no "scripts." prefix)
+from common import (
     fetch_series_db,
     save_figure,
     validate_dataframe,
@@ -41,22 +44,26 @@ def fetch_series(
     * convert UNRATE to month-end timestamps
     * average WTI daily prices to month-end
     """
-    df = fetch_series_db(["UNRATE", "DCOILWTICO"], start, end, db_path)
+    # Read both series from SQLite
+    df = fetch_series_db(["UNRATE", "DCOILWTICO"], start, end, db_path)  # :contentReference[oaicite:0]{index=0}
+    # Extract UNRATE and rename to 'value'
     unrate = df[["UNRATE"]].rename(columns={"UNRATE": "value"})
-    oil = df[["DCOILWTICO"]].rename(columns={"DCOILWTICO": "value"})
+    # Extract DCOILWTICO (oil price) and rename to 'value'
+    oil    = df[["DCOILWTICO"]].rename(columns={"DCOILWTICO": "value"})
 
     # Resample WTI to month-end average
-    oil_monthly = oil.resample("M").mean()
+    oil_monthly = oil.resample("M").mean()  # :contentReference[oaicite:1]{index=1}
     # Ensure UNRATE is at month-end, too
-    unrate.index = unrate.index.to_period("M").to_timestamp("M")
+    unrate.index = unrate.index.to_period("M").to_timestamp("M")  # :contentReference[oaicite:2]{index=2}
+
     return unrate, oil_monthly
 
 
 def validate_series(unrate: pd.DataFrame, oil: pd.DataFrame) -> None:
     """Run basic sanity checks on the two series."""
-    validate_dataframe(unrate, "UNRATE")
+    validate_dataframe(unrate, "UNRATE")        # Raises ValueError if UNRATE is empty, NaNs, or bad index
-    validate_dataframe(oil, "DCOILWTICO")
+    validate_dataframe(oil, "DCOILWTICO")       # Raises ValueError if oil is empty, NaNs, or bad index
-    validate_overlap(unrate, oil)
+    validate_overlap(unrate, oil)               # Raises ValueError if no date overlap
 
 
 # ──────────────────────────────────────────────────────────────────────────
@@ -79,20 +86,18 @@ def plot_lagged(
     """
     # Find overlapping date range
     common_start = max(unrate.index.min(), oil.index.min())
-    common_end = min(unrate.index.max(), oil.index.max())
+    common_end   = min(unrate.index.max(), oil.index.max())
 
     print(f"⚠️ Using overlapping range: {common_start.date()} to {common_end.date()}")
-    print(
+    print(f"  - UNRATE range: {unrate.index.min().date()} to {unrate.index.max().date()}")
-        f"  - UNRATE range: {unrate.index.min().date()} to {unrate.index.max().date()}"
+    print(f"  - OIL    range: {oil.index.min().date()} to {oil.index.max().date()}")
-    )
-    print(f"  - OIL range: {oil.index.min().date()} to {oil.index.max().date()}")
 
     # Trim series to common range
     unrate_common = unrate.loc[common_start:common_end]
-    oil_common = oil.loc[common_start:common_end]
+    oil_common    = oil.loc[common_start:common_end]
 
     # Apply lag/lead
-    oil_shifted = oil_common.shift(offset_months)
+    oil_shifted = oil_common.shift(offset_months)  # Shift by given months
 
     fig, ax1 = plt.subplots(figsize=(14, 7))
     plt.title("Unemployment Rate and US Oil Price", fontsize=20, weight="bold")
@@ -131,8 +136,8 @@ def plot_lagged(
 
     # Build doubling ticks from $3 upward
     min_tick = 3
-    max_val = oil_shifted["value"].max() * 1.1
+    max_val  = oil_shifted["value"].max() * 1.1
-    ticks = []
+    ticks    = []
     i = 0
     while min_tick * 2**i < max_val:
         ticks.append(min_tick * 2**i)
@@ -153,13 +158,13 @@ def plot_lagged(
 
     # Legends
     lines, labels = ax1.get_legend_handles_labels()
-    l2, l2l = ax2.get_legend_handles_labels()
+    l2, l2l       = ax2.get_legend_handles_labels()
     ax1.legend(lines + l2, labels + l2l, loc="upper left", frameon=False, fontsize=12)
 
     # Footnote with actual ranges
     foot = (
         f"UNRATE: {unrate.index.min().strftime('%b %Y')}–{unrate.index.max().strftime('%b %Y')} | "
-        f"OIL: {oil.index.min().strftime('%b %Y')}–{oil.index.max().strftime('%b %Y')}\n"
+        f"OIL:    {oil.index.min().strftime('%b %Y')}–{oil.index.max().strftime('%b %Y')}\n"
         "Source: Local FRED snapshot (offline)."
     )
     plt.annotate(
@@ -222,9 +227,26 @@ def main() -> plt.Figure:
     args = p.parse_args()
 
     start_dt = datetime.fromisoformat(args.start)
-    end_dt = datetime.fromisoformat(args.end)
+    end_dt   = datetime.fromisoformat(args.end)
 
+    # ───────────────────────────────────────────────────────────────────────
+    # Fetch both UNRATE and oil; UNRATE may contain NaNs if the last
+    # month’s data has not yet been published. We forward‐fill UNRATE
+    # (option B), then drop any remaining NaNs in both series.
+    # ───────────────────────────────────────────────────────────────────────
     unrate, oil = fetch_series(start_dt, end_dt, args.db)
+
+    # Forward‐fill UNRATE to propagate the last known value into any NaN gaps
+    unrate["value"] = unrate["value"].ffill()  # :contentReference[oaicite:3]{index=3}
+    # Drop any rows where UNRATE is still NaN (e.g., if the entire series started after 'start')
+    unrate = unrate.dropna(subset=["value"])   # :contentReference[oaicite:4]{index=4}
+
+    # Drop any rows in 'oil' where the value is NaN (ensures no Nulls remain)
+    oil = oil.dropna(subset=["value"])         # :contentReference[oaicite:5]{index=5}
+
+    # Align oil to only dates present in UNRATE after cleaning
+    oil = oil.loc[oil.index.isin(unrate.index)]
+
     validate_series(unrate, oil)
     fig = plot_lagged(unrate, oil, args.offset, start_dt, end_dt, args.extend_years)
     save_figure(fig, args.output, __file__)