Delete repository context file and update lagged oil chart script

Removed the `repo-context.txt` file, which contained project documentation and directory structure. Made modifications to `scripts/lagged_oil_unrate_chart_styled.py` to enhance functionality and maintainability.
This commit is contained in:
thePR0M3TH3AN committed 2025-05-30 18:35:40 -04:00
1 parent 02cac6ea8e
commit d12831c146
2 files changed
+44 -690

No files matched your search

-651
View File
@@ -1,651 +0,0 @@
# Repository Context
Generated on: 2025-05-30
## Directory Tree (Whitelist Only)
```
.
├── data/
├── fred.db
├── scripts/
├── lagged_oil_unrate_chart_styled.py
├── pyproject.toml
├── refresh_data.py
├── AGENTS.md
├── Makefile
├── README.md
├── requirements.txt
├── startup.sh
```
## Important Files
## python_charts/AGENTS.md
```markdown
# AGENTS.md — Guidance for OpenAI Codex
## Execution Commands
- **Always** invoke Python scripts via the startup wrapper to activate dependencies:
```bash
./startup.sh --offset <months> --end <YYYY-MM-DD> --extend-years <N>
## Project Overview
This repository, **python\_charts**, provides a framework for fetching economic time series from FRED, storing them locally, and producing custom matplotlib charts with configurable parameters. The core components are:
* **`startup.sh`**: Single-entrypoint script to bootstrap a Python venv, install dependencies, fetch data, and run specified commands under the venv interpreter.
* **`Makefile`**: Convenient high-level targets (`install`, `data`, `plot`, `plot-custom`, `test`, `clean`) that funnel through `startup.sh` for reproducible builds.
* **`scripts/refresh_data.py`**: Downloads UNRATE and DCOILWTICO series into `data/fred.db`.
* **`scripts/lagged_oil_unrate_chart_styled.py`**: Generates the lagged oil vs. unemployment chart with fixed axes styling and log scale.
Future extensions will add additional custom chart scripts under `scripts/`, all driven by the same bootstrapping and command conventions.
## Usage Conventions
### Primary Commands
* **Bootstrap & default plot**: `./startup.sh` or `make plot` — runs the lagged-oil chart with an 18‑month offset.
* **Custom plot**: `./startup.sh --offset 12 --end 2025-05-31 --extend-years 5` or `make plot-custom ARGS="--offset 12 --end 2025-05-31 --extend-years 5"`.
* **Data refresh**: `./startup.sh scripts/refresh_data.py` or `make data`.
* **Run tests**: `./startup.sh pytest -q` or `make test`.
### Argument Rules
* Arguments beginning with `--` are forwarded to the default chart script.
* If the first token is a `.py` script or executable, it is invoked under the venv’s Python.
* Otherwise, commands are executed directly (e.g. linters, test frameworks).
## Coding Standards
* **Python**: Target version 3.12. Use type hints for all function signatures. Follow PEP8 with **Black** formatting.
* **Bash**: `startup.sh` must use `set -euo pipefail` and always call `$VENV_DIR/bin/python` to avoid system‑Python mismatches.
* **Makefile**: Use tab‑indented commands. All tasks should funnel to `startup.sh`.
## Testing & Validation
* **Automated tests**: No current tests; add pytest modules under `tests/` following the pattern `test_*.py`.
* **Data mocking**: For new chart scripts, mock FRED downloads via monkeypatching `pandas_datareader.data.DataReader`.
## Extending with New Charts
1. **Create a new script** under `scripts/`, for example `scripts/custom_chart.py`.
2. **Implement** a `main()` accepting `--start`, `--end`, `--offset` or custom args using `argparse`.
3. **Leverage** `startup.sh` invocation: add usage example in your script’s docstring.
4. **Update** the `Makefile` if you need a dedicated target, otherwise users can call:
```bash
./startup.sh python scripts/custom_chart.py --your-args
```
## CI / Codex Integration
* In Codex cloud or CLI tasks, set the **Setup Script** to `./startup.sh install data` to ensure dependencies and data are ready.
* **Task Commands** should call `make plot` or `make test`, which delegates to `startup.sh`.
* **AGENTS.md** is automatically loaded by Codex to understand structure and commands.
## PR & Commit Guidelines
* Prefix branches/PRs with `feature/` or `fix/`.
* Commit messages: `<type>(<scope>): <description>` (e.g. `feat(charts): add custom inflation plot`).
* Include screenshots or saved chart outputs under `outputs/` for visual verification.
---
*End of AGENTS.md — tailored for python\_charts repo*
```
## python_charts/README.md
```markdown
# Lagged Oil–Unemployment Chart
A self-contained toolkit that:
* Fetches U.S. unemployment rate (UNRATE) and WTI crude oil price (DCOILWTICO) from FRED
* Stores the series in a local SQLite database (`data/fred.db`)
* Produces a dual-axis plot of unemployment vs. lagged oil price, with customizable lag, date range, and styling
![Example chart](scripts/lagged_oil_unrate_chart_styled.py)
---
## 🚀 Quickstart
```bash
# Clone the repo
git clone https://github.com/your-org/python_charts.git
cd python_charts
# One-command: install deps, download data, and plot
./startup.sh python scripts/lagged_oil_unrate_chart_styled.py \
--offset 12 --end 2025-05-31 --extend-years 5
```
This will:
1. Create (or reuse) a Python 3 virtual environment (`venv/`)
2. Upgrade `pip` & `setuptools` (providing `distutils` support)
3. Install required packages (offline wheelhouse first, PyPI fallback)
4. Download UNRATE & DCOILWTICO into `data/fred.db` if missing
5. Generate the chart with a 12-month oil lead, extending the x-axis by 5 years
---
## 📁 Repository Structure
```
python_charts/
├── data/
│ └── fred.db # Local FRED snapshot (SQLite)
├── scripts/
│ ├── refresh_data.py # Download and store FRED series
│ ├── lagged_oil_unrate_chart_styled.py # Plotting script
│ └── pyproject.toml # Packaging metadata
├── startup.sh # Bootstraps venv, deps, data, and runs commands
├── requirements.txt # Python dependencies
└── README.md # This file
```
---
## 🛠️ Installation & Setup
1. **Ensure** you have Python 3.8+ installed on your system.
2. **Clone** and navigate into the project:
```bash
git clone https://github.com/your-org/python_charts.git
cd python_charts
```
3. **Make** the startup script executable:
```bash
chmod +x startup.sh
```
> On first run, `startup.sh` will build the environment, install dependencies, and pull data.
---
## ⚙️ Usage
### Refresh FRED data
```bash
./startup.sh python scripts/refresh_data.py
```
Downloads the full UNRATE and WTI series (from 1948 to today) into `data/fred.db`.
### Generate the chart
```bash
# Default 18-month lag
./startup.sh
# Custom lag, range, and extension
./startup.sh python scripts/lagged_oil_unrate_chart_styled.py \
--offset 6 --start 1980-01-01 --end 2025-05-31 --extend-years 2
```
| Option | Description |
| ---------------- | --------------------------------------------------------- |
| `--offset` | Lag in months (positive = oil leads; negative = oil lags) |
| `--start` | Start date (YYYY-MM-DD) |
| `--end` | End date (YYYY-MM-DD) |
| `--extend-years` | Years to extend the x-axis beyond the end date |
---
## 📝 Details
* **Data storage**: `data/fred.db` (SQLite) holds two tables: `UNRATE` and `DCOILWTICO`.
* **Plot styling**:
* Left y-axis: unemployment rate (%) from 3 to 15%, ticks every 2%.
* Right y-axis: log scale oil price (USD), doubling ticks from \$3 upward.
* X-axis: years, major tick every 5 years, minor tick every year.
---
## 📦 Dependencies
Listed in `requirements.txt`:
```text
pandas>=1.5.0
pandas_datareader>=0.10.0
matplotlib>=3.5.0
lxml>=4.6.0
python-dateutil>=2.8.1
```
`startup.sh` also upgrades `pip` & `setuptools` to ensure `distutils` is available.
---
## 🛡️ Troubleshooting
* **ModuleNotFoundError: distutils**: resolved automatically by upgrading `setuptools` in `startup.sh`.
* **Database not found**: run `./startup.sh python scripts/refresh_data.py` to populate `data/fred.db`.
* **Virtual-env issues**: delete `venv/` and re-run `startup.sh`.
---
## 📄 License
This project is released under the MIT License. See [LICENSE](LICENSE) for details.
```
## python_charts/requirements.txt
```
pandas>=1.5.0
pandas_datareader>=0.10.0
matplotlib>=3.5.0
lxml>=4.6.0
python-dateutil>=2.8.1
```
## python_charts/startup.sh
```
#!/usr/bin/env bash
# -------------------------------------------------------------------
# startup.sh — prepare Python env, ensure deps & data, then run.
# Usage:
# ./startup.sh [options] # passes options to default plot script
# ./startup.sh [script or cmd] # runs specified script/command
# Examples:
# ./startup.sh --offset 12 --end 2025-05-31 --extend-years 5
# ./startup.sh pytest -q
# -------------------------------------------------------------------
set -euo pipefail
VENV_DIR="venv"
REQ_FILE="requirements.txt"
WHEELHOUSE="wheelhouse"
DATA_DB="data/fred.db"
PYTHON="$VENV_DIR/bin/python"
DEFAULT_SCRIPT="scripts/lagged_oil_unrate_chart_styled.py"
# 1) Create or reuse venv
if [[ ! -d "$VENV_DIR" ]]; then
echo "🔧 Creating virtual-env…"
python3 -m venv "$VENV_DIR"
fi
# 2) Activate venv
# shellcheck disable=SC1090
source "$VENV_DIR/bin/activate"
# 3) Upgrade pip & setuptools
echo "🛠 Upgrading pip & setuptools…"
$PYTHON -m pip install --upgrade pip setuptools
# 4) Install dependencies
PIP_OPTS=(install --upgrade --no-input)
if [[ -d "$WHEELHOUSE" ]]; then
echo "📦 Installing from wheelhouse…"
$PYTHON -m pip "${PIP_OPTS[@]}" --no-index --find-links="$WHEELHOUSE" -r "$REQ_FILE" \
|| { echo "⚠️ Wheelhouse failed—falling back…"; \
$PYTHON -m pip "${PIP_OPTS[@]}" -r "$REQ_FILE"; }
else
echo "🌐 Installing from PyPI…"
$PYTHON -m pip "${PIP_OPTS[@]}" -r "$REQ_FILE"
fi
echo "✅ Virtual-env ready ($($PYTHON -V))"
# 5) Ensure data exists
if [[ ! -f "$DATA_DB" ]]; then
echo "🔄 Data not found—downloading FRED series…"
$PYTHON scripts/refresh_data.py
fi
# 6) Build command
if [[ $# -eq 0 ]]; then
CMD=("$PYTHON" "$DEFAULT_SCRIPT" --offset 18)
elif [[ "$1" == --* ]]; then
CMD=("$PYTHON" "$DEFAULT_SCRIPT" "$@")
else
if [[ -f "$1" || "$1" == *.py ]]; then
CMD=("$PYTHON" "$@")
else
CMD=("$@")
fi
fi
# 7) Execute
echo "🚀 Executing: ${CMD[*]}"
exec "${CMD[@]}"
```
## python_charts/Makefile
```
# Makefile for python_charts
# Defines standard targets for Codex/CI to use startup.sh
.PHONY: install data plot test clean
# Install dependencies into venv and prepare data
install:
./startup.sh pip install
# Download FRED data (if needed)
data:
./startup.sh scripts/refresh_data.py
# Generate the default chart (18-month lag)
plot:
./startup.sh
# Generate a custom chart
# Usage: make plot ARGS="--offset 12 --end 2025-05-31 --extend-years 5"
plot-custom:
./startup.sh $(ARGS)
# Run test suite
test:
./startup.sh pytest -q
# Remove virtual environment and outputs
clean:
rm -rf venv data outputs
```
## scripts/refresh_data.py
```python
#!/usr/bin/env python3
"""
refresh_data.py
---------------
Download UNRATE and DCOILWTICO from FRED and store them in SQLite
(data/fred.db). Run on a connected machine; commit the DB afterwards.
"""
from datetime import date
import pathlib
import sqlite3
import pandas as pd
from pandas_datareader.data import DataReader
HERE = pathlib.Path(__file__).resolve().parent
DATA_PATH = HERE.parent / "data"
DB_FILE = DATA_PATH / "fred.db"
START = "1948-01-01" # earliest UNRATE observation
END = date.today().isoformat()
SERIES = ("UNRATE", "DCOILWTICO")
def main() -> None:
DATA_PATH.mkdir(exist_ok=True)
with sqlite3.connect(DB_FILE) as conn:
for s in SERIES:
print(f"↯ downloading {s} …")
df = DataReader(s, "fred", START, END)
df.to_sql(s, conn, if_exists="replace", index_label="date")
print(f"✓ wrote {s}: {len(df):,} rows")
print(f"\nDone. SQLite DB at {DB_FILE} ({DB_FILE.stat().st_size/1024:.0f} KB)")
if __name__ == "__main__":
main()
```
## scripts/pyproject.toml
```
# pyproject.toml
[project]
name = "lagged-oil-unrate"
version = "0.1.0"
description = "Plot UNRATE vs WTI oil price with adjustable lag"
authors = ["Your Name <[email protected]>"]
dependencies = [
"pandas>=1.3.0",
"pandas_datareader>=0.10.0",
"matplotlib>=3.4.0",
"python-dateutil>=2.8.0",
"lxml>=4.6.0"
]
```
## scripts/lagged_oil_unrate_chart_styled.py
```python
#!/usr/bin/env python3
"""
lagged_oil_unrate_chart_styled.py
---------------------------------
Offline version: reads UNRATE & WTI series from data/fred.db (populated
by scripts/refresh_data.py) and plots them with a configurable lag.
Usage
-----
python scripts/lagged_oil_unrate_chart_styled.py \
--offset 18 \
--start 1973-01-01 \
--end 2025-05-31 \
--extend-years 3
"""
from __future__ import annotations
import argparse
from datetime import datetime
from pathlib import Path
import matplotlib.dates as mdates
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
import numpy as np
import pandas as pd
from dateutil.relativedelta import relativedelta
import sqlite3
# ──────────────────────────────────────────────────────────────────────────
# Data helpers
# ──────────────────────────────────────────────────────────────────────────
def fetch_series(
start: datetime,
end: datetime,
db_path: str | Path = Path("data/fred.db"),
) -> tuple[pd.DataFrame, pd.DataFrame]:
"""
Load UNRATE and DCOILWTICO from a local SQLite DB, then:
* convert UNRATE to month-end timestamps
* average WTI daily prices to month-end
"""
db_path = Path(db_path)
if not db_path.exists():
raise FileNotFoundError(
f"{db_path} not found. Run scripts/refresh_data.py on a "
"machine with internet, commit the new DB, then retry."
)
with sqlite3.connect(db_path) as conn:
unrate = (
pd.read_sql(
"SELECT date, UNRATE AS value FROM UNRATE", conn,
parse_dates=["date"], index_col="date"
)
.loc[start:end]
)
oil = (
pd.read_sql(
"SELECT date, DCOILWTICO AS value FROM DCOILWTICO", conn,
parse_dates=["date"], index_col="date"
)
.loc[start:end]
)
# Resample WTI to month-end average
oil_monthly = oil.resample("M").mean()
# Ensure UNRATE is at month-end, too
unrate.index = unrate.index.to_period("M").to_timestamp("M")
return unrate, oil_monthly
# ──────────────────────────────────────────────────────────────────────────
# Plotting
# ──────────────────────────────────────────────────────────────────────────
def plot_lagged(
unrate: pd.DataFrame,
oil: pd.DataFrame,
offset_months: int,
start_date: datetime,
end_date: datetime,
extend_years: int,
) -> None:
"""Render the dual-axis chart with fixed scales and log ticks on oil."""
oil_shifted = oil.shift(offset_months)
fig, ax1 = plt.subplots(figsize=(14, 7))
plt.title("Unemployment Rate and US Oil Price", fontsize=20, weight="bold")
plt.suptitle(
"Civilian Unemployment Rate and WTI Crude Oil Price",
fontsize=14,
y=0.93,
)
# Left axis – UNRATE
ax1.plot(
unrate.index,
unrate["value"],
label="UNRATE (LHS)",
linewidth=2.5,
color="#1f77b4",
)
ax1.set_ylabel("Unemployment rate (%)", fontsize=12)
ax1.set_ylim(3, 15)
ax1.yaxis.set_major_locator(mticker.MultipleLocator(2))
ax1.yaxis.set_major_formatter("{x:.0f}%")
ax1.grid(axis="y", linestyle="--", alpha=0.4)
# Right axis – shifted oil (log scale)
ax2 = ax1.twinx()
ax2.plot(
oil_shifted.index,
oil_shifted["value"],
label="USOIL (RHS)",
linewidth=2,
linestyle="-",
alpha=0.8,
color="#FF9900",
)
ax2.set_ylabel("Oil price (USD)", fontsize=12)
# Build doubling ticks from $3 upward
min_tick = 3
max_val = oil_shifted["value"].max() * 1.1
ticks = []
i = 0
while min_tick * 2**i < max_val:
ticks.append(min_tick * 2**i)
i += 1
ticks.append(min_tick * 2**i)
ax2.set_yscale("log")
ax2.set_ylim(min_tick, ticks[-1])
ax2.set_yticks(ticks)
ax2.yaxis.set_major_formatter("${x:.0f}")
# X-axis formatting
ax1.set_xlim(start_date, end_date + relativedelta(years=extend_years))
ax1.xaxis.set_major_locator(mdates.YearLocator(base=5))
ax1.xaxis.set_minor_locator(mdates.YearLocator(1))
ax1.xaxis.set_major_formatter(mdates.DateFormatter("%Y"))
plt.setp(ax1.get_xticklabels(), rotation=0, fontsize=11)
# Legends
lines, labels = ax1.get_legend_handles_labels()
l2, l2l = ax2.get_legend_handles_labels()
ax1.legend(lines + l2, labels + l2l, loc="upper left", frameon=False, fontsize=12)
# Footnote
foot = (
f"Dates: {start_date.strftime('%Y')} through "
f"{unrate.index[-1].strftime('%d %b %Y')}\."
"\nSource: Local FRED snapshot (offline)."
)
plt.annotate(
foot,
(0, 0),
(0, -40),
xycoords="axes fraction",
textcoords="offset points",
va="top",
fontsize=10,
color="gray",
)
fig.tight_layout(rect=[0, 0.03, 1, 0.95])
plt.show()
# ──────────────────────────────────────────────────────────────────────────
# CLI
# ──────────────────────────────────────────────────────────────────────────
def main() -> None:
p = argparse.ArgumentParser()
p.add_argument(
"-o",
"--offset",
type=int,
default=18,
help="lag in months (positive=lag, negative=lead)",
)
p.add_argument(
"--start",
type=str,
default="1973-01-01",
help="series start date (YYYY-MM-DD)",
)
p.add_argument(
"--end",
type=str,
default=datetime.today().strftime("%Y-%m-%d"),
help="series end date (YYYY-MM-DD)",
)
p.add_argument(
"--extend-years",
type=int,
default=3,
help="how many years beyond end_date to show on the x-axis",
)
p.add_argument(
"--db",
type=str,
default="data/fred.db",
help="path to local FRED SQLite file",
)
args = p.parse_args()
start_dt = datetime.fromisoformat(args.start)
end_dt = datetime.fromisoformat(args.end)
unrate, oil = fetch_series(start_dt, end_dt, args.db)
plot_lagged(unrate, oil, args.offset, start_dt, end_dt, args.extend_years)
if __name__ == "__main__":
main()
```
## data/fred.db
```
*Binary file (.db) cannot be displayed.*
```
+44 -39
View File
@@ -1,19 +1,14 @@
Here’s the fully updated `scripts/lagged_oil_unrate_chart_styled.py`. You can copy–paste it in place of your existing file.
```python
#!/usr/bin/env python3
"""
lagged_oil_unrate_chart_styled.py
---------------------------------
Offline version: reads UNRATE & WTI series from ``data/fred.db``
(populated by ``scripts/refresh_data.py``), validates that both series
contain data for the requested period, and plots them with a configurable
lag.
Usage
-----
python scripts/lagged_oil_unrate_chart_styled.py \
--offset 18 \
--start 1973-01-01 \
--end 2025-05-31 \
--extend-years 3
contain data for the requested period (overlap only), and plots them
with a configurable lag.
"""
from __future__ import annotations
@@ -76,23 +71,12 @@ def fetch_series(
def validate_series(unrate: pd.DataFrame, oil: pd.DataFrame) -> None:
"""Sanity-check the fetched series before plotting.
Both dataframes must contain a ``value`` column with no missing values and
share a non-empty ``DatetimeIndex`` covering the same start and end dates.
Parameters
----------
unrate, oil:
DataFrames returned by :func:`fetch_series`.
Raises
------
ValueError
If either dataframe is empty, missing the ``value`` column, contains
nulls, or the date indexes do not span the same range.
"""
Sanity-check the fetched series before plotting.
Both dataframes must contain a ``value`` column with no missing values
and share a non-empty overlapping date range.
"""
if unrate.empty:
raise ValueError("UNRATE dataframe is empty")
if oil.empty:
@@ -109,8 +93,12 @@ def validate_series(unrate: pd.DataFrame, oil: pd.DataFrame) -> None:
):
raise ValueError("Indexes must be DatetimeIndex")
if unrate.index.min() != oil.index.min() or unrate.index.max() != oil.index.max():
raise ValueError("Series indexes do not span the same date range")
# Replace strict range check with overlap check
overlap_start = max(unrate.index.min(), oil.index.min())
overlap_end = min(unrate.index.max(), oil.index.max())
if overlap_start >= overlap_end:
raise ValueError("Series have no overlapping date range")
# ──────────────────────────────────────────────────────────────────────────
@@ -120,12 +108,28 @@ def plot_lagged(
unrate: pd.DataFrame,
oil: pd.DataFrame,
offset_months: int,
start_date: datetime,
end_date: datetime,
_start_date: datetime,
_end_date: datetime,
extend_years: int,
) -> None:
"""Render the dual-axis chart with fixed scales and log ticks on oil."""
oil_shifted = oil.shift(offset_months)
"""
Render the dual-axis chart with fixed scales and log ticks on oil.
Trims both series to their overlapping date range before shifting.
"""
# Find overlapping date range
common_start = max(unrate.index.min(), oil.index.min())
common_end = min(unrate.index.max(), oil.index.max())
print(f"⚠️ Using overlapping range: {common_start.date()} to {common_end.date()}")
print(f" - UNRATE range: {unrate.index.min().date()} to {unrate.index.max().date()}")
print(f" - OIL range: {oil.index.min().date()} to {oil.index.max().date()}")
# Trim series to common range
unrate_common = unrate.loc[common_start:common_end]
oil_common = oil.loc[common_start:common_end]
# Apply lag/lead
oil_shifted = oil_common.shift(offset_months)
fig, ax1 = plt.subplots(figsize=(14, 7))
plt.title("Unemployment Rate and US Oil Price", fontsize=20, weight="bold")
@@ -137,8 +141,8 @@ def plot_lagged(
# Left axis – UNRATE
ax1.plot(
unrate.index,
unrate["value"],
unrate_common.index,
unrate_common["value"],
label="UNRATE (LHS)",
linewidth=2.5,
color="#1f77b4",
@@ -177,8 +181,8 @@ def plot_lagged(
ax2.set_yticks(ticks)
ax2.yaxis.set_major_formatter("${x:.0f}")
# X-axis formatting
ax1.set_xlim(start_date, end_date + relativedelta(years=extend_years))
# X-axis formatting using common range
ax1.set_xlim(common_start, common_end + relativedelta(years=extend_years))
ax1.xaxis.set_major_locator(mdates.YearLocator(base=5))
ax1.xaxis.set_minor_locator(mdates.YearLocator(1))
ax1.xaxis.set_major_formatter(mdates.DateFormatter("%Y"))
@@ -189,11 +193,11 @@ def plot_lagged(
l2, l2l = ax2.get_legend_handles_labels()
ax1.legend(lines + l2, labels + l2l, loc="upper left", frameon=False, fontsize=12)
# Footnote
# Footnote with actual ranges
foot = (
f"Dates: {start_date.strftime('%Y')} through "
f"{unrate.index[-1].strftime('%d %b %Y')}\."
"\nSource: Local FRED snapshot (offline)."
f"UNRATE: {unrate.index.min().strftime('%b %Y')}–{unrate.index.max().strftime('%b %Y')} | "
f"OIL: {oil.index.min().strftime('%b %Y')}–{oil.index.max().strftime('%b %Y')}\n"
"Source: Local FRED snapshot (offline)."
)
plt.annotate(
foot,
@@ -258,3 +262,4 @@ def main() -> None:
if __name__ == "__main__":
main()
```