diff --git a/README.md b/README.md index bebdeda..77b2f03 100644 --- a/README.md +++ b/README.md @@ -1,190 +1,138 @@ -# RePrompt – Repository-Context Generator +# Lagged Oil–Unemployment Chart -**RePrompt** creates a single `repo-context.txt` file that gives AI coding assistants -(e.g. ChatGPT) a concise, opinionated snapshot of your repository. -It combines an overview, directory tree, highlighted file contents, and optional static -sections such as to-do lists—all configurable through a Streamlit UI. +A self-contained toolkit that: -> **Works great with** -> [mckaywrigley’s XML parser](https://github.com/mckaywrigley/o1-xml-parser/tree/main). +* Fetches U.S. unemployment rate (UNRATE) and WTI crude oil price (DCOILWTICO) from FRED +* Stores the series in a local SQLite database (`data/fred.db`) +* Produces a dual-axis plot of unemployment vs. lagged oil price, with customizable lag, date range, and styling + +![Example chart](scripts/lagged_oil_unrate_chart_styled.py) --- -## ✨ Features - -| Category | What it does | -|----------|--------------| -| **Streamlit UI** | Choose a target repo, set include/exclude rules, and generate the context file in one screen. | -| **Context builder** | Produces `repo-context.txt` tailored for AI assistants. | -| **Configurable tree** | Exclude or include directories/files interactively (defaults skip noisy folders like `node_modules`, `.git`, etc.). | -| **File highlights** | Embeds syntax-highlighted contents of “important” files you pick. | -| **Static sections** | Auto-appends `overview.txt`, `to-do_list.txt`, or any custom text snippets. | -| **Save & load config** | Persist your include/exclude selections for future sessions. | - ---- - -## 🛠 Prerequisites - -* **Python 3.7 +** – [download](https://www.python.org/downloads/) -* **Git** – [download](https://git-scm.com/downloads) - ---- - -## 🗂 Directory layout - -``` - -RePrompt/ -├── src/ -│ ├── app.py # Streamlit entry-point -│ ├── generate\_repo\_context.py # Core context-builder -│ ├── config.yaml # Default config (auto-generated if absent) -│ ├── index.html # Single-page static explanation -│ └── requirements.txt # Python dependencies -└── README.md - -```` - ---- - -## 🚀 Installation - -### 1 Clone the repo +## 🚀 Quickstart ```bash -git clone https://github.com/PR0M3TH3AN/RePrompt.git -cd RePrompt -```` +# Clone the repo +git clone https://github.com/your-org/python_charts.git +cd python_charts -### 2 Create & activate a virtual-environment - -
-Windows (PowerShell) - -```powershell -python -m venv venv -venv\Scripts\Activate.ps1 +# One-command: install deps, download data, and plot +./startup.sh python scripts/lagged_oil_unrate_chart_styled.py \ + --offset 12 --end 2025-05-31 --extend-years 5 ``` -
+This will: -
-macOS / Linux +1. Create (or reuse) a Python 3 virtual environment (`venv/`) +2. Upgrade `pip` & `setuptools` (providing `distutils` support) +3. Install required packages (offline wheelhouse first, PyPI fallback) +4. Download UNRATE & DCOILWTICO into `data/fred.db` if missing +5. Generate the chart with a 12-month oil lead, extending the x-axis by 5 years + +--- + +## 📁 Repository Structure -```bash -python3 -m venv venv -source venv/bin/activate ``` - -
- -### 3 Install dependencies - -```bash -python -m pip install --upgrade pip -pip install -r src/requirements.txt -``` - -`requirements.txt` contains only three packages: - -```text -streamlit -PyYAML -pyperclip +python_charts/ +├── data/ +│ └── fred.db # Local FRED snapshot (SQLite) +├── scripts/ +│ ├── refresh_data.py # Download and store FRED series +│ ├── lagged_oil_unrate_chart_styled.py # Plotting script +│ └── pyproject.toml # Packaging metadata +├── startup.sh # Bootstraps venv, deps, data, and runs commands +├── requirements.txt # Python dependencies +└── README.md # This file ``` --- -## ▶️ Running the app +## 🛠️ Installation & Setup -```bash -streamlit run src/app.py -``` +1. **Ensure** you have Python 3.8+ installed on your system. +2. **Clone** and navigate into the project: -The UI opens at **[http://localhost:8501](http://localhost:8501)**. + ```bash + git clone https://github.com/your-org/python_charts.git + cd python_charts + ``` +3. **Make** the startup script executable: ---- - -## 📝 Usage walk-through - -1. **Config file** - A starter `config.yaml` is created in `src/`: - - ```yaml - exclude_dirs: - - node_modules - - venv - - __pycache__ - - .git - - dist - - build - - logs - - .idea - - .vscode - - important_files: [] - custom_sections: [] + ```bash + chmod +x startup.sh ``` -2. **Select repo** - Click **“Choose Folder”** in the sidebar, navigate to the repository you - want to summarise, and select it. - -3. **Fine-tune files** - - * Tick directories to *include* in the rendered tree. - * Exclude noisy or irrelevant files. - * Review the final list. - -4. **Generate** - Hit **“Generate Context File”**. - Download or copy the result; optionally save the configuration for later. +> On first run, `startup.sh` will build the environment, install dependencies, and pull data. --- -## 🔧 Customisation tips +## ⚙️ Usage -### Change default exclusions - -Edit these constants in `src/app.py`: - -```python -DEFAULT_EXCLUDED_DIRS = [ - "node_modules", "venv", "__pycache__", ".git", - "dist", "build", "logs", ".idea", ".vscode" -] -DEFAULT_EXCLUDED_FILES = ["repo-context.txt"] -``` - -### Different port - -If port 8501 is busy: +### Refresh FRED data ```bash -streamlit run src/app.py --server.port 8502 +./startup.sh python scripts/refresh_data.py ``` ---- +Downloads the full UNRATE and WTI series (from 1948 to today) into `data/fred.db`. -## ❓ Troubleshooting +### Generate the chart -| Problem | Fix | -| -------------------------------------- | ------------------------------------------------------------- | -| `Tcl_AsyncDelete` warning | Harmless—ignore it. | -| “Permission denied” when creating dirs | Run your terminal as admin or check folder write permissions. | -| Packages won’t install | `python -m pip install --upgrade pip` then reinstall. | -| Port already in use | Kill other Streamlit processes or use `--server.port`. | +```bash +# Default 18-month lag +./startup.sh + +# Custom lag, range, and extension +./startup.sh python scripts/lagged_oil_unrate_chart_styled.py \ + --offset 6 --start 1980-01-01 --end 2025-05-31 --extend-years 2 +``` + +| Option | Description | +| ---------------- | --------------------------------------------------------- | +| `--offset` | Lag in months (positive = oil leads; negative = oil lags) | +| `--start` | Start date (YYYY-MM-DD) | +| `--end` | End date (YYYY-MM-DD) | +| `--extend-years` | Years to extend the x-axis beyond the end date | --- -## 🤝 Contributing +## 📝 Details -1. Fork → create a feature branch -2. Commit & push your changes -3. Open a Pull Request +* **Data storage**: `data/fred.db` (SQLite) holds two tables: `UNRATE` and `DCOILWTICO`. +* **Plot styling**: + + * Left y-axis: unemployment rate (%) from 3 to 15%, ticks every 2%. + * Right y-axis: log scale oil price (USD), doubling ticks from \$3 upward. + * X-axis: years, major tick every 5 years, minor tick every year. + +--- + +## 📦 Dependencies + +Listed in `requirements.txt`: + +```text +pandas>=1.5.0 +pandas_datareader>=0.10.0 +matplotlib>=3.5.0 +lxml>=4.6.0 +python-dateutil>=2.8.1 +``` + +`startup.sh` also upgrades `pip` & `setuptools` to ensure `distutils` is available. + +--- + +## 🛡️ Troubleshooting + +* **ModuleNotFoundError: distutils**: resolved automatically by upgrading `setuptools` in `startup.sh`. +* **Database not found**: run `./startup.sh python scripts/refresh_data.py` to populate `data/fred.db`. +* **Virtual-env issues**: delete `venv/` and re-run `startup.sh`. --- ## 📄 License -Distributed under the **MIT License** – see `LICENSE` for details. +This project is released under the MIT License. See [LICENSE](LICENSE) for details. diff --git a/data/fred.db b/data/fred.db new file mode 100644 index 0000000..daadfbd Binary files /dev/null and b/data/fred.db differ diff --git a/repo-context.txt b/repo-context.txt index b22a9f0..22118d1 100644 --- a/repo-context.txt +++ b/repo-context.txt @@ -6,6 +6,8 @@ Generated on: 2025-05-30 ``` . + ├── data/ + ├── fred.db ├── scripts/ ├── lagged_oil_unrate_chart_styled.py ├── pyproject.toml @@ -19,126 +21,8 @@ Generated on: 2025-05-30 ## python_charts/README.md ```markdown -# Lagged Oil vs. Unemployment Charting - -This repository provides two Python scripts to fetch the latest US Unemployment Rate (UNRATE) and WTI crude oil price (DCOILWTICO) from FRED, apply a configurable lag or lead to the oil series, and generate polished dual‑axis charts. - -## Files - -* **`lagged_oil_unrate_chart.py`** - A lightweight script that: - - 1. Downloads monthly UNRATE and averages daily WTI to monthly. - 2. Shifts the oil series by `N` months (default = 18). - 3. Plots both series on a dual‑axis Matplotlib chart. - -* **`lagged_oil_unrate_chart_styled.py`** - A more polished version that adds: - - * Title, subtitle, footnote with date range and source. - * Clean dual‑axis styling with percentage and dollar‑format ticks. - * Customizable date range and extended x‑axis (e.g., 1973–2028). - * CLI flags for offset, start/end dates, and axis extension. - -* **`requirements.txt`** - Lists all Python dependencies for both scripts. - -## Prerequisites - -* Python 3.7 or later -* Internet connection (to fetch data from FRED) - -## Setup - -Clone or download this repository: - -```bash -git clone https://your-repo-url.git -cd your-repo-folder -``` - -### Create a Virtual Environment & Install Dependencies - -#### On Linux / macOS - -```bash -# 1. Create a venv -python3 -m venv venv - -# 2. Activate it -source venv/bin/activate - -# 3. Upgrade pip and install requirements -pip install --upgrade pip -pip install -r requirements.txt -``` - -#### On Windows (PowerShell) - -```powershell -# 1. Create a venv -python -m venv venv - -# 2. Activate it -venv\Scripts\Activate.ps1 - -# 3. Upgrade pip and install requirements -python -m pip install --upgrade pip -pip install -r requirements.txt -``` - -> **Note:** If you get execution policy errors on Windows, you may need to run: -> -> ```powershell -> Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser -> ``` - -## Usage - -Once the virtual environment is active and dependencies are installed, run either script from the project root. - -### 1. Basic Lag Script - -```bash -# Default: oil lags unemployment by 18 months (~1.5 years) -python lagged_oil_unrate_chart.py - -# Other offsets (in months) -python lagged_oil_unrate_chart.py --offset 6 # 6‑month lag -python lagged_oil_unrate_chart.py --offset -12 # 12‑month lead -``` - -### 2. Styled & Configurable Chart - -```bash -python lagged_oil_unrate_chart_styled.py \ - --offset 18 \ - --start 1973-01-01 \ - --end 2025-05-31 \ - --extend-years 3 -``` - -| Flag | Description | -| ------------------ | -------------------------------------------------------------- | -| `--offset N` | Lag (positive) or lead (negative) in months (default = 18) | -| `--start DATE` | Series start date (`YYYY-MM-DD`, default = `1973-01-01`) | -| `--end DATE` | Series end date (`YYYY-MM-DD`, default = today) | -| `--extend-years M` | How many years beyond `end` to extend the x‑axis (default = 3) | - -## Customization - -* **Normalization**: Insert a normalization step (e.g., percent change from a base date). -* **Export**: Modify the script to save figures with `plt.savefig('chart.png', dpi=300)`. -* **Statistics**: Add `ta.correl()` or use Pandas to compute rolling correlations. - -## Troubleshooting - -* **"ModuleNotFoundError"**: Ensure your venv is activated, then reinstall dependencies. -* **SSL/DataReader errors**: Upgrade `pandas_datareader` or switch to an alternate data source. - -## License - -This project is released under the MIT License. +./startup.sh python scripts/lagged_oil_unrate_chart_styled.py \ + --offset 12 --end 2025-05-31 --extend-years 5 ``` @@ -155,15 +39,66 @@ python-dateutil>=2.8.1 ## python_charts/startup.sh ``` #!/usr/bin/env bash +# ------------------------------------------------------------------- +# startup.sh — prepare Python env, ensure dependencies, FRED data, +# and run a command inside the venv +# Usage: +# ./startup.sh [command…] +# Automatically: +# - Upgrades pip & setuptools (ensuring distutils on Python 3.12+) +# - Creates venv if missing +# - Installs requirements +# - Downloads data/fred.db if missing +# ------------------------------------------------------------------- set -euo pipefail -python3 -m venv venv -source venv/bin/activate +VENV_DIR="venv" +REQ_FILE="requirements.txt" +WHEELHOUSE="wheelhouse" +DATA_DB="data/fred.db" -pip install --no-index --find-links=wheelhouse -r requirements.txt +# 1) Create or reuse venv +if [[ ! -d "$VENV_DIR" ]]; then + echo "🔧 Creating virtual-env…" + python3 -m venv "$VENV_DIR" +fi +# Activate +# shellcheck disable=SC1090 +source "$VENV_DIR/bin/activate" -echo "✅ Virtual-env ready (offline install)." +# 2) Upgrade pip & setuptools (to provide distutils on Py3.12+) +echo "🛠 Upgrading pip & setuptools…" +pip install --upgrade pip setuptools +# 3) Install project dependencies +PIP_OPTS=(install --upgrade --no-input) +if [[ -d "$WHEELHOUSE" ]]; then + echo "📦 Installing from wheelhouse…" + pip "${PIP_OPTS[@]}" --no-index --find-links="$WHEELHOUSE" -r "$REQ_FILE" \ + || { echo "⚠️ Wheelhouse failed—falling back to PyPI…"; \ + pip "${PIP_OPTS[@]}" -r "$REQ_FILE"; } +else + echo "🌐 Installing from PyPI…" + pip "${PIP_OPTS[@]}" -r "$REQ_FILE" +fi + +echo "✅ Virtual-env ready ($(python -V))" + +# 4) Ensure data/fred.db exists +if [[ ! -f "$DATA_DB" ]]; then + echo "🔄 $DATA_DB not found—downloading FRED series…" + python scripts/refresh_data.py +fi + +# 5) Decide what to run +if [[ $# -gt 0 ]]; then + CMD=("$@") +else + CMD=(python scripts/lagged_oil_unrate_chart_styled.py --offset 18) +fi + +echo "🚀 Executing: ${CMD[*]}" +exec "${CMD[@]}" ``` ## scripts/refresh_data.py @@ -253,6 +188,8 @@ from pathlib import Path import matplotlib.dates as mdates import matplotlib.pyplot as plt +import matplotlib.ticker as mticker +import numpy as np import pandas as pd from dateutil.relativedelta import relativedelta import sqlite3 @@ -313,7 +250,7 @@ def plot_lagged( end_date: datetime, extend_years: int, ) -> None: - """Render the dual-axis chart.""" + """Render the dual-axis chart with fixed scales and log ticks on oil.""" oil_shifted = oil.shift(offset_months) fig, ax1 = plt.subplots(figsize=(14, 7)) @@ -333,11 +270,12 @@ def plot_lagged( color="#1f77b4", ) ax1.set_ylabel("Unemployment rate (%)", fontsize=12) - ax1.set_ylim(0, unrate["value"].max() * 1.1) + ax1.set_ylim(3, 15) + ax1.yaxis.set_major_locator(mticker.MultipleLocator(2)) ax1.yaxis.set_major_formatter("{x:.0f}%") ax1.grid(axis="y", linestyle="--", alpha=0.4) - # Right axis – shifted oil + # Right axis – shifted oil (log scale) ax2 = ax1.twinx() ax2.plot( oil_shifted.index, @@ -349,10 +287,23 @@ def plot_lagged( color="#FF9900", ) ax2.set_ylabel("Oil price (USD)", fontsize=12) - ax2.set_ylim(0, oil_shifted["value"].max() * 1.1) - ax2.yaxis.set_major_formatter("${x:,.0f}") - # X-axis + # Build doubling ticks from $3 upward + min_tick = 3 + max_val = oil_shifted["value"].max() * 1.1 + ticks = [] + i = 0 + while min_tick * 2**i < max_val: + ticks.append(min_tick * 2**i) + i += 1 + ticks.append(min_tick * 2**i) + + ax2.set_yscale("log") + ax2.set_ylim(min_tick, ticks[-1]) + ax2.set_yticks(ticks) + ax2.yaxis.set_major_formatter("${x:.0f}") + + # X-axis formatting ax1.set_xlim(start_date, end_date + relativedelta(years=extend_years)) ax1.xaxis.set_major_locator(mdates.YearLocator(base=5)) ax1.xaxis.set_minor_locator(mdates.YearLocator(1)) @@ -367,8 +318,8 @@ def plot_lagged( # Footnote foot = ( f"Dates: {start_date.strftime('%Y')} through " - f"{unrate.index[-1].strftime('%d %b %Y')}.\n" - "Source: Local FRED snapshot (offline)." + f"{unrate.index[-1].strftime('%d %b %Y')}\." + "\nSource: Local FRED snapshot (offline)." ) plt.annotate( foot, @@ -435,3 +386,9 @@ if __name__ == "__main__": ``` +## data/fred.db +``` +*Binary file (.db) cannot be displayed.* + +``` + diff --git a/scripts/lagged_oil_unrate_chart_styled.py b/scripts/lagged_oil_unrate_chart_styled.py index 49dea8b..1050046 100644 --- a/scripts/lagged_oil_unrate_chart_styled.py +++ b/scripts/lagged_oil_unrate_chart_styled.py @@ -22,6 +22,8 @@ from pathlib import Path import matplotlib.dates as mdates import matplotlib.pyplot as plt +import matplotlib.ticker as mticker +import numpy as np import pandas as pd from dateutil.relativedelta import relativedelta import sqlite3 @@ -82,7 +84,7 @@ def plot_lagged( end_date: datetime, extend_years: int, ) -> None: - """Render the dual-axis chart.""" + """Render the dual-axis chart with fixed scales and log ticks on oil.""" oil_shifted = oil.shift(offset_months) fig, ax1 = plt.subplots(figsize=(14, 7)) @@ -102,11 +104,12 @@ def plot_lagged( color="#1f77b4", ) ax1.set_ylabel("Unemployment rate (%)", fontsize=12) - ax1.set_ylim(0, unrate["value"].max() * 1.1) + ax1.set_ylim(3, 15) + ax1.yaxis.set_major_locator(mticker.MultipleLocator(2)) ax1.yaxis.set_major_formatter("{x:.0f}%") ax1.grid(axis="y", linestyle="--", alpha=0.4) - # Right axis – shifted oil + # Right axis – shifted oil (log scale) ax2 = ax1.twinx() ax2.plot( oil_shifted.index, @@ -118,10 +121,23 @@ def plot_lagged( color="#FF9900", ) ax2.set_ylabel("Oil price (USD)", fontsize=12) - ax2.set_ylim(0, oil_shifted["value"].max() * 1.1) - ax2.yaxis.set_major_formatter("${x:,.0f}") - # X-axis + # Build doubling ticks from $3 upward + min_tick = 3 + max_val = oil_shifted["value"].max() * 1.1 + ticks = [] + i = 0 + while min_tick * 2**i < max_val: + ticks.append(min_tick * 2**i) + i += 1 + ticks.append(min_tick * 2**i) + + ax2.set_yscale("log") + ax2.set_ylim(min_tick, ticks[-1]) + ax2.set_yticks(ticks) + ax2.yaxis.set_major_formatter("${x:.0f}") + + # X-axis formatting ax1.set_xlim(start_date, end_date + relativedelta(years=extend_years)) ax1.xaxis.set_major_locator(mdates.YearLocator(base=5)) ax1.xaxis.set_minor_locator(mdates.YearLocator(1)) @@ -136,8 +152,8 @@ def plot_lagged( # Footnote foot = ( f"Dates: {start_date.strftime('%Y')} through " - f"{unrate.index[-1].strftime('%d %b %Y')}.\n" - "Source: Local FRED snapshot (offline)." + f"{unrate.index[-1].strftime('%d %b %Y')}\." + "\nSource: Local FRED snapshot (offline)." ) plt.annotate( foot, diff --git a/startup.sh b/startup.sh index 18be739..05b1229 100755 --- a/startup.sh +++ b/startup.sh @@ -1,9 +1,61 @@ #!/usr/bin/env bash +# ------------------------------------------------------------------- +# startup.sh — prepare Python env, ensure dependencies, FRED data, +# and run a command inside the venv +# Usage: +# ./startup.sh [command…] +# Automatically: +# - Upgrades pip & setuptools (ensuring distutils on Python 3.12+) +# - Creates venv if missing +# - Installs requirements +# - Downloads data/fred.db if missing +# ------------------------------------------------------------------- set -euo pipefail -python3 -m venv venv -source venv/bin/activate +VENV_DIR="venv" +REQ_FILE="requirements.txt" +WHEELHOUSE="wheelhouse" +DATA_DB="data/fred.db" -pip install --no-index --find-links=wheelhouse -r requirements.txt +# 1) Create or reuse venv +if [[ ! -d "$VENV_DIR" ]]; then + echo "🔧 Creating virtual-env…" + python3 -m venv "$VENV_DIR" +fi +# Activate +# shellcheck disable=SC1090 +source "$VENV_DIR/bin/activate" -echo "✅ Virtual-env ready (offline install)." +# 2) Upgrade pip & setuptools (to provide distutils on Py3.12+) +echo "🛠 Upgrading pip & setuptools…" +pip install --upgrade pip setuptools + +# 3) Install project dependencies +PIP_OPTS=(install --upgrade --no-input) +if [[ -d "$WHEELHOUSE" ]]; then + echo "📦 Installing from wheelhouse…" + pip "${PIP_OPTS[@]}" --no-index --find-links="$WHEELHOUSE" -r "$REQ_FILE" \ + || { echo "⚠️ Wheelhouse failed—falling back to PyPI…"; \ + pip "${PIP_OPTS[@]}" -r "$REQ_FILE"; } +else + echo "🌐 Installing from PyPI…" + pip "${PIP_OPTS[@]}" -r "$REQ_FILE" +fi + +echo "✅ Virtual-env ready ($(python -V))" + +# 4) Ensure data/fred.db exists +if [[ ! -f "$DATA_DB" ]]; then + echo "🔄 $DATA_DB not found—downloading FRED series…" + python scripts/refresh_data.py +fi + +# 5) Decide what to run +if [[ $# -gt 0 ]]; then + CMD=("$@") +else + CMD=(python scripts/lagged_oil_unrate_chart_styled.py --offset 18) +fi + +echo "🚀 Executing: ${CMD[*]}" +exec "${CMD[@]}" \ No newline at end of file