Python for GenAI: turn policy files into clean records

Set up a project you can reproduce

Install a supported Python 3 release and create a folder named northstar-assistant. Use a virtual environment so this project's dependencies do not silently alter another project. In PowerShell, create the environment with the commands below. Calling its interpreter directly avoids shell activation issues.

python -m venv .venv
.venv\Scripts\python --version

On macOS or Linux, the interpreter is .venv/bin/python. The ingestion example uses only the standard library. When you later add packages, install through the selected interpreter's -m pip command and record the versions you actually tested. Do not copy a large dependency list before you have a use for it.

Create the small document collection

Make a policies folder. Save returns.txt with “Unopened items may be returned within 30 days of delivery. Clearance items are excluded.” Save delivery.txt with “Standard delivery takes 3–5 business days after dispatch.” Save damage.txt with “Report damaged items within 7 days of delivery and include a photograph.” These are invented training policies, not statements about i-360 or any real retailer.

Lists hold your collection of records. A dictionary gives each record named fields. Strings contain text; integers can track counts. Functions separate reading from later retrieval. Those basics are enough for the first stage. Tuples suit fixed pairs, while sets help detect duplicate IDs. Type hints document intent, but runtime validation still belongs at file and API boundaries.

Read files without hiding failures

Save this as ingest.py next to the policies directory. The explicit UTF-8 encoding makes the file format predictable. Sorting provides stable output, and the document ID comes from the filename rather than a model-generated label.

from pathlib import Path
import json

def load_policies(folder: Path) -> list[dict[str, str]]:
    records = []
    for path in sorted(folder.glob("*.txt")):
        text = path.read_text(encoding="utf-8").strip()
        if not text:
            raise ValueError(f"Empty policy: {path.name}")
        records.append({"id": path.stem, "text": text})
    if not records:
        raise ValueError("No policy files found")
    return records

if __name__ == "__main__":
    records = load_policies(Path(__file__).parent / "policies")
    print(json.dumps(records, ensure_ascii=False, indent=2))

Run .venv\Scripts\python ingest.py on Windows. Expect three objects containing id and text. A missing collection or blank document should fail visibly. Catch an exception only where you can add useful context or recover; returning an empty list for every error makes broken ingestion look like a valid empty knowledge base.

Add external services only at a clear boundary

An HTTP call can time out, return an error status, or deliver a body that does not match your expectations. Keep that behavior inside a small client function and validate the result before adding it to your records. Store API credentials outside source files and never place them in browser JavaScript.

Async programming helps overlap network waits; it does not make every CPU-heavy task faster. Introduce it after you have a working synchronous path and a reason to handle several requests concurrently. PDF extraction is another boundary: selectable text and scanned images need different handling. Inspect extracted paragraphs before assuming a successful library call means usable content.

References and further reading

Use these primary references for deeper study and current API details. Examples in this lesson use fictional Northstar data.

• Python documentation —