mirror of
https://github.com/PR0M3TH3AN/python_charts.git
synced 2026-10-27 08:50:39 +00:00
thePR0M3TH3AN
1d6f38fedd
- Introduce a Makefile to define standard targets for installation, data preparation, plotting, testing, and cleaning. - Update `startup.sh` to enhance clarity and functionality, including better command handling and virtual environment management. - Document changes in `repo-context.txt` to reflect the addition of the Makefile.
575 lines
16 KiB
Plaintext
575 lines
16 KiB
Plaintext
# Repository Context
|
||
|
||
Generated on: 2025-05-30
|
||
|
||
## Directory Tree (Whitelist Only)
|
||
|
||
```
|
||
.
|
||
├── data/
|
||
├── fred.db
|
||
├── scripts/
|
||
├── lagged_oil_unrate_chart_styled.py
|
||
├── pyproject.toml
|
||
├── refresh_data.py
|
||
├── Makefile
|
||
├── README.md
|
||
├── requirements.txt
|
||
├── startup.sh
|
||
```
|
||
|
||
## Important Files
|
||
|
||
## python_charts/README.md
|
||
```markdown
|
||
# Lagged Oil–Unemployment Chart
|
||
|
||
A self-contained toolkit that:
|
||
|
||
* Fetches U.S. unemployment rate (UNRATE) and WTI crude oil price (DCOILWTICO) from FRED
|
||
* Stores the series in a local SQLite database (`data/fred.db`)
|
||
* Produces a dual-axis plot of unemployment vs. lagged oil price, with customizable lag, date range, and styling
|
||
|
||
|
||
|
||
---
|
||
|
||
## 🚀 Quickstart
|
||
|
||
```bash
|
||
# Clone the repo
|
||
git clone https://github.com/your-org/python_charts.git
|
||
cd python_charts
|
||
|
||
# One-command: install deps, download data, and plot
|
||
./startup.sh python scripts/lagged_oil_unrate_chart_styled.py \
|
||
--offset 12 --end 2025-05-31 --extend-years 5
|
||
```
|
||
|
||
This will:
|
||
|
||
1. Create (or reuse) a Python 3 virtual environment (`venv/`)
|
||
2. Upgrade `pip` & `setuptools` (providing `distutils` support)
|
||
3. Install required packages (offline wheelhouse first, PyPI fallback)
|
||
4. Download UNRATE & DCOILWTICO into `data/fred.db` if missing
|
||
5. Generate the chart with a 12-month oil lead, extending the x-axis by 5 years
|
||
|
||
---
|
||
|
||
## 📁 Repository Structure
|
||
|
||
```
|
||
python_charts/
|
||
├── data/
|
||
│ └── fred.db # Local FRED snapshot (SQLite)
|
||
├── scripts/
|
||
│ ├── refresh_data.py # Download and store FRED series
|
||
│ ├── lagged_oil_unrate_chart_styled.py # Plotting script
|
||
│ └── pyproject.toml # Packaging metadata
|
||
├── startup.sh # Bootstraps venv, deps, data, and runs commands
|
||
├── requirements.txt # Python dependencies
|
||
└── README.md # This file
|
||
```
|
||
|
||
---
|
||
|
||
## 🛠️ Installation & Setup
|
||
|
||
1. **Ensure** you have Python 3.8+ installed on your system.
|
||
2. **Clone** and navigate into the project:
|
||
|
||
```bash
|
||
git clone https://github.com/your-org/python_charts.git
|
||
cd python_charts
|
||
```
|
||
3. **Make** the startup script executable:
|
||
|
||
```bash
|
||
chmod +x startup.sh
|
||
```
|
||
|
||
> On first run, `startup.sh` will build the environment, install dependencies, and pull data.
|
||
|
||
---
|
||
|
||
## ⚙️ Usage
|
||
|
||
### Refresh FRED data
|
||
|
||
```bash
|
||
./startup.sh python scripts/refresh_data.py
|
||
```
|
||
|
||
Downloads the full UNRATE and WTI series (from 1948 to today) into `data/fred.db`.
|
||
|
||
### Generate the chart
|
||
|
||
```bash
|
||
# Default 18-month lag
|
||
./startup.sh
|
||
|
||
# Custom lag, range, and extension
|
||
./startup.sh python scripts/lagged_oil_unrate_chart_styled.py \
|
||
--offset 6 --start 1980-01-01 --end 2025-05-31 --extend-years 2
|
||
```
|
||
|
||
| Option | Description |
|
||
| ---------------- | --------------------------------------------------------- |
|
||
| `--offset` | Lag in months (positive = oil leads; negative = oil lags) |
|
||
| `--start` | Start date (YYYY-MM-DD) |
|
||
| `--end` | End date (YYYY-MM-DD) |
|
||
| `--extend-years` | Years to extend the x-axis beyond the end date |
|
||
|
||
---
|
||
|
||
## 📝 Details
|
||
|
||
* **Data storage**: `data/fred.db` (SQLite) holds two tables: `UNRATE` and `DCOILWTICO`.
|
||
* **Plot styling**:
|
||
|
||
* Left y-axis: unemployment rate (%) from 3 to 15%, ticks every 2%.
|
||
* Right y-axis: log scale oil price (USD), doubling ticks from \$3 upward.
|
||
* X-axis: years, major tick every 5 years, minor tick every year.
|
||
|
||
---
|
||
|
||
## 📦 Dependencies
|
||
|
||
Listed in `requirements.txt`:
|
||
|
||
```text
|
||
pandas>=1.5.0
|
||
pandas_datareader>=0.10.0
|
||
matplotlib>=3.5.0
|
||
lxml>=4.6.0
|
||
python-dateutil>=2.8.1
|
||
```
|
||
|
||
`startup.sh` also upgrades `pip` & `setuptools` to ensure `distutils` is available.
|
||
|
||
---
|
||
|
||
## 🛡️ Troubleshooting
|
||
|
||
* **ModuleNotFoundError: distutils**: resolved automatically by upgrading `setuptools` in `startup.sh`.
|
||
* **Database not found**: run `./startup.sh python scripts/refresh_data.py` to populate `data/fred.db`.
|
||
* **Virtual-env issues**: delete `venv/` and re-run `startup.sh`.
|
||
|
||
---
|
||
|
||
## 📄 License
|
||
|
||
This project is released under the MIT License. See [LICENSE](LICENSE) for details.
|
||
|
||
```
|
||
|
||
## python_charts/requirements.txt
|
||
```
|
||
pandas>=1.5.0
|
||
pandas_datareader>=0.10.0
|
||
matplotlib>=3.5.0
|
||
lxml>=4.6.0
|
||
python-dateutil>=2.8.1
|
||
|
||
```
|
||
|
||
## python_charts/startup.sh
|
||
```
|
||
#!/usr/bin/env bash
|
||
# -------------------------------------------------------------------
|
||
# startup.sh — prepare Python env, ensure deps & data, then run.
|
||
# Usage:
|
||
# ./startup.sh [options] # passes options to default plot script
|
||
# ./startup.sh [script or cmd] # runs specified script/command
|
||
# Examples:
|
||
# ./startup.sh --offset 12 --end 2025-05-31 --extend-years 5
|
||
# ./startup.sh pytest -q
|
||
# -------------------------------------------------------------------
|
||
set -euo pipefail
|
||
|
||
VENV_DIR="venv"
|
||
REQ_FILE="requirements.txt"
|
||
WHEELHOUSE="wheelhouse"
|
||
DATA_DB="data/fred.db"
|
||
PYTHON="$VENV_DIR/bin/python"
|
||
DEFAULT_SCRIPT="scripts/lagged_oil_unrate_chart_styled.py"
|
||
|
||
# 1) Create or reuse venv
|
||
if [[ ! -d "$VENV_DIR" ]]; then
|
||
echo "🔧 Creating virtual-env…"
|
||
python3 -m venv "$VENV_DIR"
|
||
fi
|
||
|
||
# 2) Activate venv
|
||
# shellcheck disable=SC1090
|
||
source "$VENV_DIR/bin/activate"
|
||
|
||
# 3) Upgrade pip & setuptools
|
||
echo "🛠 Upgrading pip & setuptools…"
|
||
$PYTHON -m pip install --upgrade pip setuptools
|
||
|
||
# 4) Install dependencies
|
||
PIP_OPTS=(install --upgrade --no-input)
|
||
if [[ -d "$WHEELHOUSE" ]]; then
|
||
echo "📦 Installing from wheelhouse…"
|
||
$PYTHON -m pip "${PIP_OPTS[@]}" --no-index --find-links="$WHEELHOUSE" -r "$REQ_FILE" \
|
||
|| { echo "⚠️ Wheelhouse failed—falling back…"; \
|
||
$PYTHON -m pip "${PIP_OPTS[@]}" -r "$REQ_FILE"; }
|
||
else
|
||
echo "🌐 Installing from PyPI…"
|
||
$PYTHON -m pip "${PIP_OPTS[@]}" -r "$REQ_FILE"
|
||
fi
|
||
|
||
echo "✅ Virtual-env ready ($($PYTHON -V))"
|
||
|
||
# 5) Ensure data exists
|
||
if [[ ! -f "$DATA_DB" ]]; then
|
||
echo "🔄 Data not found—downloading FRED series…"
|
||
$PYTHON scripts/refresh_data.py
|
||
fi
|
||
|
||
# 6) Build command
|
||
if [[ $# -eq 0 ]]; then
|
||
CMD=("$PYTHON" "$DEFAULT_SCRIPT" --offset 18)
|
||
elif [[ "$1" == --* ]]; then
|
||
CMD=("$PYTHON" "$DEFAULT_SCRIPT" "$@")
|
||
else
|
||
if [[ -f "$1" || "$1" == *.py ]]; then
|
||
CMD=("$PYTHON" "$@")
|
||
else
|
||
CMD=("$@")
|
||
fi
|
||
fi
|
||
|
||
# 7) Execute
|
||
echo "🚀 Executing: ${CMD[*]}"
|
||
exec "${CMD[@]}"
|
||
|
||
```
|
||
|
||
## python_charts/Makefile
|
||
```
|
||
# Makefile for python_charts
|
||
# Defines standard targets for Codex/CI to use startup.sh
|
||
|
||
.PHONY: install data plot test clean
|
||
|
||
# Install dependencies into venv and prepare data
|
||
install:
|
||
./startup.sh pip install
|
||
|
||
# Download FRED data (if needed)
|
||
data:
|
||
./startup.sh scripts/refresh_data.py
|
||
|
||
# Generate the default chart (18-month lag)
|
||
plot:
|
||
./startup.sh
|
||
|
||
# Generate a custom chart
|
||
# Usage: make plot ARGS="--offset 12 --end 2025-05-31 --extend-years 5"
|
||
plot-custom:
|
||
./startup.sh $(ARGS)
|
||
|
||
# Run test suite
|
||
test:
|
||
./startup.sh pytest -q
|
||
|
||
# Remove virtual environment and outputs
|
||
clean:
|
||
rm -rf venv data outputs
|
||
|
||
```
|
||
|
||
## scripts/refresh_data.py
|
||
```python
|
||
#!/usr/bin/env python3
|
||
"""
|
||
refresh_data.py
|
||
---------------
|
||
Download UNRATE and DCOILWTICO from FRED and store them in SQLite
|
||
(data/fred.db). Run on a connected machine; commit the DB afterwards.
|
||
"""
|
||
|
||
from datetime import date
|
||
import pathlib
|
||
import sqlite3
|
||
|
||
import pandas as pd
|
||
from pandas_datareader.data import DataReader
|
||
|
||
HERE = pathlib.Path(__file__).resolve().parent
|
||
DATA_PATH = HERE.parent / "data"
|
||
DB_FILE = DATA_PATH / "fred.db"
|
||
|
||
START = "1948-01-01" # earliest UNRATE observation
|
||
END = date.today().isoformat()
|
||
SERIES = ("UNRATE", "DCOILWTICO")
|
||
|
||
|
||
def main() -> None:
|
||
DATA_PATH.mkdir(exist_ok=True)
|
||
with sqlite3.connect(DB_FILE) as conn:
|
||
for s in SERIES:
|
||
print(f"↯ downloading {s} …")
|
||
df = DataReader(s, "fred", START, END)
|
||
df.to_sql(s, conn, if_exists="replace", index_label="date")
|
||
print(f"✓ wrote {s}: {len(df):,} rows")
|
||
print(f"\nDone. SQLite DB at {DB_FILE} ({DB_FILE.stat().st_size/1024:.0f} KB)")
|
||
|
||
|
||
if __name__ == "__main__":
|
||
main()
|
||
|
||
```
|
||
|
||
## scripts/pyproject.toml
|
||
```
|
||
# pyproject.toml
|
||
|
||
[project]
|
||
name = "lagged-oil-unrate"
|
||
version = "0.1.0"
|
||
description = "Plot UNRATE vs WTI oil price with adjustable lag"
|
||
authors = ["Your Name <[email protected]>"]
|
||
dependencies = [
|
||
"pandas>=1.3.0",
|
||
"pandas_datareader>=0.10.0",
|
||
"matplotlib>=3.4.0",
|
||
"python-dateutil>=2.8.0",
|
||
"lxml>=4.6.0"
|
||
]
|
||
|
||
```
|
||
|
||
## scripts/lagged_oil_unrate_chart_styled.py
|
||
```python
|
||
#!/usr/bin/env python3
|
||
"""
|
||
lagged_oil_unrate_chart_styled.py
|
||
---------------------------------
|
||
Offline version: reads UNRATE & WTI series from data/fred.db (populated
|
||
by scripts/refresh_data.py) and plots them with a configurable lag.
|
||
|
||
Usage
|
||
-----
|
||
python scripts/lagged_oil_unrate_chart_styled.py \
|
||
--offset 18 \
|
||
--start 1973-01-01 \
|
||
--end 2025-05-31 \
|
||
--extend-years 3
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import argparse
|
||
from datetime import datetime
|
||
from pathlib import Path
|
||
|
||
import matplotlib.dates as mdates
|
||
import matplotlib.pyplot as plt
|
||
import matplotlib.ticker as mticker
|
||
import numpy as np
|
||
import pandas as pd
|
||
from dateutil.relativedelta import relativedelta
|
||
import sqlite3
|
||
|
||
|
||
# ──────────────────────────────────────────────────────────────────────────
|
||
# Data helpers
|
||
# ──────────────────────────────────────────────────────────────────────────
|
||
def fetch_series(
|
||
start: datetime,
|
||
end: datetime,
|
||
db_path: str | Path = Path("data/fred.db"),
|
||
) -> tuple[pd.DataFrame, pd.DataFrame]:
|
||
"""
|
||
Load UNRATE and DCOILWTICO from a local SQLite DB, then:
|
||
|
||
* convert UNRATE to month-end timestamps
|
||
* average WTI daily prices to month-end
|
||
"""
|
||
db_path = Path(db_path)
|
||
if not db_path.exists():
|
||
raise FileNotFoundError(
|
||
f"{db_path} not found. Run scripts/refresh_data.py on a "
|
||
"machine with internet, commit the new DB, then retry."
|
||
)
|
||
|
||
with sqlite3.connect(db_path) as conn:
|
||
unrate = (
|
||
pd.read_sql(
|
||
"SELECT date, UNRATE AS value FROM UNRATE", conn,
|
||
parse_dates=["date"], index_col="date"
|
||
)
|
||
.loc[start:end]
|
||
)
|
||
oil = (
|
||
pd.read_sql(
|
||
"SELECT date, DCOILWTICO AS value FROM DCOILWTICO", conn,
|
||
parse_dates=["date"], index_col="date"
|
||
)
|
||
.loc[start:end]
|
||
)
|
||
|
||
# Resample WTI to month-end average
|
||
oil_monthly = oil.resample("M").mean()
|
||
# Ensure UNRATE is at month-end, too
|
||
unrate.index = unrate.index.to_period("M").to_timestamp("M")
|
||
return unrate, oil_monthly
|
||
|
||
|
||
# ──────────────────────────────────────────────────────────────────────────
|
||
# Plotting
|
||
# ──────────────────────────────────────────────────────────────────────────
|
||
def plot_lagged(
|
||
unrate: pd.DataFrame,
|
||
oil: pd.DataFrame,
|
||
offset_months: int,
|
||
start_date: datetime,
|
||
end_date: datetime,
|
||
extend_years: int,
|
||
) -> None:
|
||
"""Render the dual-axis chart with fixed scales and log ticks on oil."""
|
||
oil_shifted = oil.shift(offset_months)
|
||
|
||
fig, ax1 = plt.subplots(figsize=(14, 7))
|
||
plt.title("Unemployment Rate and US Oil Price", fontsize=20, weight="bold")
|
||
plt.suptitle(
|
||
"Civilian Unemployment Rate and WTI Crude Oil Price",
|
||
fontsize=14,
|
||
y=0.93,
|
||
)
|
||
|
||
# Left axis – UNRATE
|
||
ax1.plot(
|
||
unrate.index,
|
||
unrate["value"],
|
||
label="UNRATE (LHS)",
|
||
linewidth=2.5,
|
||
color="#1f77b4",
|
||
)
|
||
ax1.set_ylabel("Unemployment rate (%)", fontsize=12)
|
||
ax1.set_ylim(3, 15)
|
||
ax1.yaxis.set_major_locator(mticker.MultipleLocator(2))
|
||
ax1.yaxis.set_major_formatter("{x:.0f}%")
|
||
ax1.grid(axis="y", linestyle="--", alpha=0.4)
|
||
|
||
# Right axis – shifted oil (log scale)
|
||
ax2 = ax1.twinx()
|
||
ax2.plot(
|
||
oil_shifted.index,
|
||
oil_shifted["value"],
|
||
label="USOIL (RHS)",
|
||
linewidth=2,
|
||
linestyle="-",
|
||
alpha=0.8,
|
||
color="#FF9900",
|
||
)
|
||
ax2.set_ylabel("Oil price (USD)", fontsize=12)
|
||
|
||
# Build doubling ticks from $3 upward
|
||
min_tick = 3
|
||
max_val = oil_shifted["value"].max() * 1.1
|
||
ticks = []
|
||
i = 0
|
||
while min_tick * 2**i < max_val:
|
||
ticks.append(min_tick * 2**i)
|
||
i += 1
|
||
ticks.append(min_tick * 2**i)
|
||
|
||
ax2.set_yscale("log")
|
||
ax2.set_ylim(min_tick, ticks[-1])
|
||
ax2.set_yticks(ticks)
|
||
ax2.yaxis.set_major_formatter("${x:.0f}")
|
||
|
||
# X-axis formatting
|
||
ax1.set_xlim(start_date, end_date + relativedelta(years=extend_years))
|
||
ax1.xaxis.set_major_locator(mdates.YearLocator(base=5))
|
||
ax1.xaxis.set_minor_locator(mdates.YearLocator(1))
|
||
ax1.xaxis.set_major_formatter(mdates.DateFormatter("%Y"))
|
||
plt.setp(ax1.get_xticklabels(), rotation=0, fontsize=11)
|
||
|
||
# Legends
|
||
lines, labels = ax1.get_legend_handles_labels()
|
||
l2, l2l = ax2.get_legend_handles_labels()
|
||
ax1.legend(lines + l2, labels + l2l, loc="upper left", frameon=False, fontsize=12)
|
||
|
||
# Footnote
|
||
foot = (
|
||
f"Dates: {start_date.strftime('%Y')} through "
|
||
f"{unrate.index[-1].strftime('%d %b %Y')}\."
|
||
"\nSource: Local FRED snapshot (offline)."
|
||
)
|
||
plt.annotate(
|
||
foot,
|
||
(0, 0),
|
||
(0, -40),
|
||
xycoords="axes fraction",
|
||
textcoords="offset points",
|
||
va="top",
|
||
fontsize=10,
|
||
color="gray",
|
||
)
|
||
|
||
fig.tight_layout(rect=[0, 0.03, 1, 0.95])
|
||
plt.show()
|
||
|
||
|
||
# ──────────────────────────────────────────────────────────────────────────
|
||
# CLI
|
||
# ──────────────────────────────────────────────────────────────────────────
|
||
def main() -> None:
|
||
p = argparse.ArgumentParser()
|
||
p.add_argument(
|
||
"-o",
|
||
"--offset",
|
||
type=int,
|
||
default=18,
|
||
help="lag in months (positive=lag, negative=lead)",
|
||
)
|
||
p.add_argument(
|
||
"--start",
|
||
type=str,
|
||
default="1973-01-01",
|
||
help="series start date (YYYY-MM-DD)",
|
||
)
|
||
p.add_argument(
|
||
"--end",
|
||
type=str,
|
||
default=datetime.today().strftime("%Y-%m-%d"),
|
||
help="series end date (YYYY-MM-DD)",
|
||
)
|
||
p.add_argument(
|
||
"--extend-years",
|
||
type=int,
|
||
default=3,
|
||
help="how many years beyond end_date to show on the x-axis",
|
||
)
|
||
p.add_argument(
|
||
"--db",
|
||
type=str,
|
||
default="data/fred.db",
|
||
help="path to local FRED SQLite file",
|
||
)
|
||
args = p.parse_args()
|
||
|
||
start_dt = datetime.fromisoformat(args.start)
|
||
end_dt = datetime.fromisoformat(args.end)
|
||
|
||
unrate, oil = fetch_series(start_dt, end_dt, args.db)
|
||
plot_lagged(unrate, oil, args.offset, start_dt, end_dt, args.extend_years)
|
||
|
||
|
||
if __name__ == "__main__":
|
||
main()
|
||
|
||
```
|
||
|
||
## data/fred.db
|
||
```
|
||
*Binary file (.db) cannot be displayed.*
|
||
|
||
```
|
||
|