Update README.md with formatting improvements and command clarifications

- Correct markdown formatting for links in the instructions.
- Clarify command usage for running scripts, including direct module invocation.
- Enhance descriptions of the data refresh and chart generation processes.
This commit is contained in:
thePR0M3TH3AN committed 2025-05-31 18:22:08 -04:00
1 parent 59de89e66f
commit 7a7231a69c
2 files changed
+71 -44

No files matched your search

+71 -44
View File
@@ -38,8 +38,8 @@ This will:
```bash ```bash
pip install --upgrade -r requirements.txt pip install --upgrade -r requirements.txt
``` ```
3. Download UNRATE & DCOILWTICO into `data/fred.db` if missing ([Stack Overflow][3], [martinheinz.dev][4]) 3. Download UNRATE & DCOILWTICO into `data/fred.db` if missing (\[Stack Overflow]\[3], \[martinheinz.dev]\[4])
4. Generate the chart with a 12‐month oil lead, extending the x‐axis by 5 years ([martinheinz.dev][4], [Real Python][1]) 4. Generate the chart with a 12‐month oil lead, extending the x‐axis by 5 years (\[martinheinz.dev]\[4], \[Real Python]\[1])
--- ---
@@ -69,7 +69,7 @@ python_charts/
> from common import fetch_series_db, save_figure, validate_dataframe, validate_overlap > from common import fetch_series_db, save_figure, validate_dataframe, validate_overlap
> ``` > ```
> >
> rather than `from scripts.common import …`. ([Real Python][1], [Software Engineering Stack Exchange][2]) > rather than `from scripts.common import …`. (\[Real Python]\[1], \[Software Engineering Stack Exchange]\[2])
--- ---
@@ -80,7 +80,11 @@ python_charts/
To refresh (or initially populate) the SQLite database with FRED series, run: To refresh (or initially populate) the SQLite database with FRED series, run:
```bash ```bash
./startup.sh python scripts/refresh_data.py --series UNRATE CPIAUCSL --start 1960-01-01 --output outputs/chart.png # Either let startup.sh detect the script path:
./startup.sh scripts/refresh_data.py --series UNRATE CPIAUCSL --start 1960-01-01 --output outputs/chart.png
# Or invoke it directly as a module (dependencies must already be installed):
python -m scripts.refresh_data --series UNRATE CPIAUCSL --start 1960-01-01 --output outputs/chart.png
``` ```
This command: This command:
@@ -88,7 +92,7 @@ This command:
* Connects to FRED via `pandas_datareader` * Connects to FRED via `pandas_datareader`
* Downloads the specified series (e.g., `UNRATE` and `CPIAUCSL`) from the start date through today * Downloads the specified series (e.g., `UNRATE` and `CPIAUCSL`) from the start date through today
* Writes each series to its own table in `data/fred.db` * Writes each series to its own table in `data/fred.db`
* Saves a PNG of any default plot (if `--output` is provided) to `outputs/` ([Stack Overflow][3], [GitHub][5]) * Saves a PNG of any default plot (if `--output` is provided) to `outputs/` (\[Stack Overflow]\[3], \[GitHub]\[5])
#### `refresh_data.py` options #### `refresh_data.py` options
@@ -101,11 +105,10 @@ This command:
##### Example: ##### Example:
```bash ```bash
./startup.sh python scripts/refresh_data.py --series UNRATE DCOILWTICO CBBTCUSD GLOBAL_M2 --start 1973-01-01 # Populate fred.db with four series (UNRATE, WTI, Bitcoin, and Global M2)
./startup.sh scripts/refresh_data.py --series UNRATE DCOILWTICO CBBTCUSD GLOBAL_M2 --start 1973-01-01
``` ```
This populates `fred.db` with four series (UNRATE, WTI, Bitcoin, and Global M2). ([Stack Overflow][3], [martinheinz.dev][4])
--- ---
### 2. Generate the Default UNRATE vs. Oil Chart ### 2. Generate the Default UNRATE vs. Oil Chart
@@ -114,28 +117,33 @@ Once data is present, generate the dual‐axis plot of UNRATE vs. lagged WTI oil
```bash ```bash
# Default: 18‐month lag, default date range (1973‐01‐01 to today), and 3‐year extension # Default: 18‐month lag, default date range (1973‐01‐01 to today), and 3‐year extension
./startup.sh python scripts/lagged_oil_unrate_chart_styled.py --offset 18 ./startup.sh --offset 18
``` ```
* The script will: What happens under the hood:
1. Import `UNRATE` and `DCOILWTICO` from `fred.db` via: 1. `startup.sh` sees `--offset 18` and forwards it to the default module `scripts.lagged_oil_unrate_chart_styled` as:
```bash
python -m scripts.lagged_oil_unrate_chart_styled --offset 18
```
2. The script imports `UNRATE` and `DCOILWTICO` from `fred.db` via:
```python ```python
from common import fetch_series_db, save_figure, validate_dataframe, validate_overlap from common import fetch_series_db, save_figure, validate_dataframe, validate_overlap
``` ```
([Real Python][1], [Software Engineering Stack Exchange][2]) (\[Real Python]\[1], \[Software Engineering Stack Exchange]\[2])
2. Convert both series to month‐end indices, resample oil to a monthly average, and align their date ranges. 3. Converts both series to month‐end indices, resamples oil to a monthly average, and aligns their date ranges.
3. Validate that neither series is empty, contains `NaN`, or has no overlap (raising a `ValueError` if any issue). 4. Validates that neither series is empty, contains `NaN`, or has no overlap (raising a `ValueError` if any issue).
4. Apply the specified lag (e.g., 18 months), then plot UNRATE on the left y‐axis (3%–15%, ticks every 2%) and log-scaled WTI on the right y‐axis (doubling ticks). 5. Applies the specified lag (e.g., 18 months), then plots UNRATE on the left y‐axis and log‐scaled WTI on the right y‐axis.
5. Save the figure to `outputs/lagged_oil_unrate_chart_styled_<timestamp>.png` (unless `--output` is specified). ([martinheinz.dev][4], [Real Python][1]) 6. Saves the figure to `outputs/lagged_oil_unrate_chart_styled_<timestamp>.png` (unless `--output` is specified). (\[martinheinz.dev]\[4], \[Real Python]\[1])
#### Customizing the UNRATE vs. Oil Chart #### Customizing the UNRATE vs. Oil Chart
```bash ```bash
# 6‐month lag, start in 1980, end in 2025‐05‐31, extend by 2 years # 6‐month lag, start in 1980, end in 2025‐05‐31, extend by 2 years
./startup.sh python scripts/lagged_oil_unrate_chart_styled.py --offset 6 --start 1980-01-01 --end 2025-05-31 --extend-years 2 ./startup.sh --offset 6 --start 1980-01-01 --end 2025-05-31 --extend-years 2
``` ```
| Option | Description | | Option | Description |
@@ -147,7 +155,7 @@ Once data is present, generate the dual‐axis plot of UNRATE vs. lagged WTI oil
| `--output` | Save the figure to this path (PNG or PDF) | | `--output` | Save the figure to this path (PNG or PDF) |
> **Note:** If you do not specify `--output`, `save_figure(...)` creates a timestamped PNG in `outputs/` by default: > **Note:** If you do not specify `--output`, `save_figure(...)` creates a timestamped PNG in `outputs/` by default:
> `outputs/lagged_oil_unrate_chart_styled_20250531_121045.png` ([Python Packaging][6], [Real Python][1]) > `outputs/lagged_oil_unrate_chart_styled_20250531_121045.png` (\[Python Packaging]\[6], \[Real Python]\[1])
--- ---
@@ -156,8 +164,11 @@ Once data is present, generate the dual‐axis plot of UNRATE vs. lagged WTI oil
You can plot any combination of FRED series stored in `data/fred.db` using `scripts/custom_chart.py`. Each column in the DataFrame is plotted on a shared axis. You can plot any combination of FRED series stored in `data/fred.db` using `scripts/custom_chart.py`. Each column in the DataFrame is plotted on a shared axis.
```bash ```bash
# Example: plot both UNRATE and WTI Oil from 2000 through 2020 # Either let startup.sh detect the script path:
./startup.sh python scripts/custom_chart.py --series UNRATE DCOILWTICO --start 2000-01-01 --end 2020-12-31 ./startup.sh scripts/custom_chart.py --series UNRATE DCOILWTICO --start 2000-01-01 --end 2020-12-31
# Or invoke it directly as a module (dependencies must already be installed):
python -m scripts.custom_chart --series UNRATE DCOILWTICO --start 2000-01-01 --end 2020-12-31
``` ```
* Internally, `custom_chart.py` invokes: * Internally, `custom_chart.py` invokes:
@@ -179,17 +190,24 @@ You can plot any combination of FRED series stored in `data/fred.db` using `scri
### Bitcoin vs. Global M2 ### Bitcoin vs. Global M2
Before running `bitcoin_m2_chart.py`, make sure the Bitcoin price Before running `bitcoin_m2_chart.py`, make sure the Bitcoin price
(`CBBTCUSD`) and global M2 (`GLOBAL_M2`) series exist in `data/fred.db`. (`CBBTCUSD`) and global M2 (`GLOBAL_M2`) series exist in `data/fred.db`. If the database is missing these tables, fetch them with `refresh_data.py`:
If the database is missing these tables, fetch them with `refresh_data.py`:
```bash ```bash
./startup.sh python scripts/refresh_data.py --series CBBTCUSD GLOBAL_M2 --start 2010-01-01 # Either let startup.sh detect the script path:
./startup.sh scripts/refresh_data.py --series CBBTCUSD GLOBAL_M2 --start 2010-01-01
# Or invoke it directly as a module (dependencies must already be installed):
python -m scripts.refresh_data --series CBBTCUSD GLOBAL_M2 --start 2010-01-01
``` ```
Once the data is present, generate the Bitcoin‐Global M2 overlay chart: Once the data is present, generate the Bitcoin‐Global M2 overlay chart:
```bash ```bash
./startup.sh python scripts/bitcoin_m2_chart.py --btc-series CBBTCUSD --m2-series GLOBAL_M2 --output outputs/chart.png # Either let startup.sh detect the script path:
./startup.sh scripts/bitcoin_m2_chart.py --btc-series CBBTCUSD --m2-series GLOBAL_M2 --output outputs/chart.png
# Or invoke it directly as a module (dependencies must already be installed):
python -m scripts.bitcoin_m2_chart --btc-series CBBTCUSD --m2-series GLOBAL_M2 --output outputs/chart.png
``` ```
* This script: * This script:
@@ -200,10 +218,10 @@ Once the data is present, generate the Bitcoin‐Global M2 overlay chart:
from common import fetch_series_db, save_figure from common import fetch_series_db, save_figure
``` ```
([Real Python][1], [Software Engineering Stack Exchange][2]) (\[Real Python]\[1], \[Software Engineering Stack Exchange]\[2])
2. Renames each column to `"value"` for consistency. 2. Renames each column to `"value"` for consistency.
3. Finds the overlapping date range, shifts M2 by `--offset` (default = 94 days), and overlays the two series on dual axes. 3. Finds the overlapping date range, shifts M2 by `--offset` (default = 94 days), and overlays the two series on dual axes.
4. Saves the figure—e.g., `outputs/bitcoin_m2_chart_20250531_090608.png` ([martinheinz.dev][4], [Real Python][1]) 4. Saves the figure—e.g., `outputs/bitcoin_m2_chart_20250531_090608.png` (\[martinheinz.dev]\[4], \[Real Python]\[1])
> **CLI Options for `bitcoin_m2_chart.py`:** > **CLI Options for `bitcoin_m2_chart.py`:**
> >
@@ -223,36 +241,36 @@ Once the data is present, generate the Bitcoin‐Global M2 overlay chart:
* **Data storage**: * **Data storage**:
* `data/fred.db` (SQLite) contains one table per FRED series—each table has columns `date` (as the index) and `<SERIES_NAME>` for the value ([Stack Overflow][3], [martinheinz.dev][4]). * `data/fred.db` (SQLite) contains one table per FRED series—each table has columns `date` (as the index) and `<SERIES_NAME>` for the value (\[Stack Overflow]\[3], \[martinheinz.dev]\[4]).
* **Plot styling**: * **Plot styling**:
* **Lagged oil vs. UNRATE chart**: * **Lagged oil vs. UNRATE chart**:
* Left y‐axis: Unemployment rate (%) from 3% to 15%, with tick marks every 2% * Left y‐axis: Unemployment rate (%) from 3% to 15%, with tick marks every 2%
* Right y‐axis: WTI oil price (USD) on a log scale; ticks double starting at \$3 (i.e., 3, 6, 12, 24, etc.) ([martinheinz.dev][4], [Real Python][1]) * Right y‐axis: WTI oil price (USD) on a log scale; ticks double starting at \$3 (i.e., 3, 6, 12, 24, etc.) (\[martinheinz.dev]\[4], \[Real Python]\[1])
* X‐axis: Major tick every 5 years, minor tick every year; format ticks as `%Y` (e.g., 1980, 1985, 1990) ([Real Python][1], [martinheinz.dev][4]) * X‐axis: Major tick every 5 years, minor tick every year; format ticks as `%Y` (e.g., 1980, 1985, 1990) (\[Real Python]\[1], \[martinheinz.dev]\[4])
* **Bitcoin vs. Global M2 chart**: * **Bitcoin vs. Global M2 chart**:
* Left y‐axis: Bitcoin price (USD) as a linear scale * Left y‐axis: Bitcoin price (USD) as a linear scale
* Right y‐axis: Global M2 (billions of USD) as a linear scale * Right y‐axis: Global M2 (billions of USD) as a linear scale
* X‐axis: Same date formatting as above (major = every 5 years, minor = yearly) ([Real Python][1], [martinheinz.dev][4]) * X‐axis: Same date formatting as above (major = every 5 years, minor = yearly) (\[Real Python]\[1], \[martinheinz.dev]\[4])
* **Utility functions in `common.py`**: * **Utility functions in `common.py`**:
* `fetch_series_db(series_list, start, end, db_path)` * `fetch_series_db(series_list, start, end, db_path)`
* Opens `data/fred.db`, reads each series table, slices by date range, and concatenates into a single DataFrame, sorted by date. Raises `FileNotFoundError` if `fred.db` is missing. ([Stack Overflow][3], [martinheinz.dev][4]) * Opens `data/fred.db`, reads each series table, slices by date range, and concatenates into a single DataFrame, sorted by date. Raises `FileNotFoundError` if `fred.db` is missing. (\[Stack Overflow]\[3], \[martinheinz.dev]\[4])
* `validate_dataframe(df, name)` * `validate_dataframe(df, name)`
* Confirms the DataFrame is nonempty, has a `"value"` column, contains no nulls, and uses a `DatetimeIndex`. Raises `ValueError` otherwise. ([devgem.io][7], [Real Python][1]) * Confirms the DataFrame is nonempty, has a `"value"` column, contains no nulls, and uses a `DatetimeIndex`. Raises `ValueError` otherwise. (\[devgem.io]\[7], \[Real Python]\[1])
* `validate_overlap(df1, df2)` * `validate_overlap(df1, df2)`
* Checks for a nonempty overlap in dates between two series; raises `ValueError` if there is no intersection. ([devgem.io][7], [Real Python][1]) * Checks for a nonempty overlap in dates between two series; raises `ValueError` if there is no intersection. (\[devgem.io]\[7], \[Real Python]\[1])
* `save_figure(fig, output, script_path)` * `save_figure(fig, output, script_path)`
* Saves the Matplotlib `Figure` to either the given `output` path or `outputs/<script>_<timestamp>.png`. ([Python Packaging][6], [Real Python][1]) * Saves the Matplotlib `Figure` to either the given `output` path or `outputs/<script>_<timestamp>.png`. (\[Python Packaging]\[6], \[Real Python]\[1])
--- ---
@@ -283,36 +301,45 @@ python-dateutil>=2.8.1
* Ensure you have added `scripts/__init__.py` and that all `from scripts.common import …` lines have been updated to `from common import …`. Then rerun: * Ensure you have added `scripts/__init__.py` and that all `from scripts.common import …` lines have been updated to `from common import …`. Then rerun:
````bash ```bash
./startup.sh python scripts/lagged_oil_unrate_chart_styled.py --offset 18 # Run the default lagged-oil chart via startup.sh
``` :contentReference[oaicite:24]{index=24} ./startup.sh --offset 18
```
```` or, if you prefer the module form:
```bash
python -m scripts.lagged_oil_unrate_chart_styled --offset 18
```
* **`ValueError: UNRATE contains null values`** * **`ValueError: UNRATE contains null values`**
* This indicates that the UNRATE DataFrame has missing (NaN) entries. Possible fixes: * This indicates that the UNRATE DataFrame has missing (NaN) entries. Possible fixes:
1. Drop nulls within SQL—e.g., run `refresh_data.py --end` using the last fully published month rather than “today” ([devgem.io][7], [pyOpenSci][8]) 1. Drop nulls within SQL—e.g., run `refresh_data.py --end` using the last fully published month rather than “today” (\[devgem.io]\[7], \[pyOpenSci]\[8])
2. Forward‐fill or backward‐fill missing UNRATE values in `common.py` before validation (if appropriate) ([pyOpenSci][8], [Real Python][1]) 2. Forward‐fill or backward‐fill missing UNRATE values in `common.py` before validation (if appropriate) (\[pyOpenSci]\[8], \[Real Python]\[1])
3. Comment out the strict null check in `validate_dataframe(...)` (not recommended for publication‐quality charts) ([devgem.io][7], [Real Python][1]) 3. Comment out the strict null check in `validate_dataframe(...)` (not recommended for publication‐quality charts) (\[devgem.io]\[7], \[Real Python]\[1])
* **Database not found** * **Database not found**
* If `data/fred.db` does not exist (e.g., fresh clone), run: * If `data/fred.db` does not exist (e.g., fresh clone), run:
````bash ````bash
./startup.sh python scripts/refresh_data.py --series UNRATE DCOILWTICO # Using startup.sh to detect the script path
./startup.sh scripts/refresh_data.py --series UNRATE DCOILWTICO
# Or run it as a module directly
python -m scripts.refresh_data --series UNRATE DCOILWTICO
``` :contentReference[oaicite:28]{index=28} ``` :contentReference[oaicite:28]{index=28}
```` ````
* **Permission errors** * **Permission errors**
* Ensure you have write access to the project directory, especially for `data/` and `outputs/`. ([Real Python][1], [Roy's Blog][9]) * Ensure you have write access to the project directory, especially for `data/` and `outputs/`. (\[Real Python]\[1], \[Roy's Blog]\[9])
--- ---
## 📄 License ## 📄 License
Released under the MIT License. See [LICENSE](LICENSE) for details. ([Roy's Blog][9], [Real Python][1]) Released under the MIT License. See [LICENSE](LICENSE) for details. (\[Roy's Blog]\[9], \[Real Python]\[1])
Binary file not shown.

After

Width:  |  Height:  |  Size: 129 KiB