Real world automation

Learning goals

Combine paths, CSV, logs, backups, and API calls into predictable automation with visible results.

Explanation

Automation follows a small pipeline: read inputs, validate them, process them, and report results. Real scripts also need collision handling, useful logging, and a clear failure policy. The image includes file organization, CSV filtering, log analysis, REST calls, backups, and DevOps workflows. Lesson 15 provides the REST example; the examples here demonstrate the local operations.

Use a new disposable folder for this lesson. Create an inbox folder containing copies of a few .txt and .csv files. Do not point practice code at your only copy of important work.

Example 1 Preview a file organizer

from pathlib import Path
import shutil

root = Path("inbox")
dry_run = True
if not root.is_dir():
    raise FileNotFoundError("Create the practice inbox first")
for source in sorted(root.iterdir()):
    if not source.is_file():
        continue
    group = source.suffix.lower().lstrip(".") or "no_extension"
    target = root / group / source.name
    if target.exists():
        print("SKIP existing:", target)
        continue
    print("PLAN" if dry_run else "MOVE", source, "->", target)
    if not dry_run:
        target.parent.mkdir(exist_ok=True)
        shutil.move(str(source), str(target))

Expected result: planned moves such as inbox/notes.txt to inbox/txt/notes.txt. Files do not move while dry_run is True. After inspecting the plan, setting it to False performs the moves. Existing destinations are skipped. This is a small single-user example; a concurrent production tool needs stronger collision controls.

Example 2 Filter CSV and count log errors

Use students.csv created in Lesson 11, copied into this lesson's folder. Create app.log containing these three lines: INFO Started, ERROR Disk full, ERROR Upload failed.

import csv

with open("students.csv", newline="", encoding="utf-8") as file:
    rows = list(csv.DictReader(file))
selected = [row for row in rows if int(row["score"]) >= 90]
with open("top_students.csv", "w", newline="",
          encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=["name", "score"])
    writer.writeheader()
    writer.writerows(selected)

with open("app.log", encoding="utf-8") as file:
    errors = [line.rstrip() for line in file
              if line.startswith("ERROR ")]
print("Selected students:", len(selected))
print("Error lines:", len(errors))

Expected output with the stated fixtures: Selected students: 1 and Error lines: 2. top_students.csv contains Sara's record. This example assumes valid scores; a task below adds a malformed-row policy.

Example 3 Create a timestamped backup

from datetime import datetime, timezone
from pathlib import Path
import shutil
import zipfile

source = Path("inbox")
if not source.is_dir():
    raise FileNotFoundError("Create the practice inbox first")
Path("backups").mkdir(exist_ok=True)
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
archive = shutil.make_archive(str(Path("backups") / stamp),
                              "zip", root_dir=source)
with zipfile.ZipFile(archive) as backup:
    print("Archive readable:", backup.testzip() is None)
    print("Entries:", len(backup.namelist()))

The archive is placed outside inbox, preventing it from including itself. The name and entry count vary. A readable ZIP is only an initial check; periodically restore a backup into a separate folder and compare the files.

In DevOps, similar scripts can collect metrics, upload artifacts, invoke container tools, or trigger a build. Use service-specific SDKs and permissions for those operations. Avoid hardcoding credentials; pass configuration through environment variables or an appropriate secret manager.

Student tasks

1. Add a summary of planned, moved, and skipped files to the organizer. Verify a second run does not lose files.

2. Extend CSV processing to report invalid scores with row numbers and continue processing valid rows. Test a blank score and "abc".

3. Count INFO, WARNING, and ERROR records separately in a practice log.

4. Restore the generated ZIP into a separate restore_test folder and compare the restored file contents with the originals.

Completion check: submit the inputs, an execution log, and evidence that the backup can be restored.