LLM Ecosystem, Tools and Future Trends

The system stack

An application combines user experience, orchestration, a model interface, infrastructure, and data. An orchestration framework can help coordinate retrieval and tools, but it does not replace evaluation, authorization, or sound application design.

Tool categories

The source names orchestration libraries such as LangChain and LlamaIndex, agent frameworks, model repositories, hosted APIs, and vector stores. Treat these as examples of categories, not a current recommendation or exhaustive list. Features and licenses change.

Choose by requirements

Compare deployment constraints, quality on your test set, latency, memory, cost, licensing, maintainability, and data handling. For vector storage, consider filters, hybrid search, access enforcement, scale, backup, and operations. A small prototype may not need a separate vector service.

Trends as design questions

Multimodal inputs, smaller models, agents, local inference, domain adaptation, streaming, and robotics are directions highlighted by the sheet. Ask which one solves a measured user problem. More autonomy also requires explicit action boundaries and recovery paths.

Worked example

Two fictional deployment options are scored against weighted criteria. These are invented values for learning, not benchmarks or vendor ratings.

Download lesson 14 Python example

weights = {"quality": 0.5, "latency": 0.3, "operations": 0.2}
options = {
    "Option A": {"quality": 4, "latency": 3, "operations": 5},
    "Option B": {"quality": 5, "latency": 4, "operations": 2}
}
for name, scores in options.items():
    total = sum(weights[k]*scores[k] for k in weights)
    print(name, round(total, 2))

Expected output

Option A 3.9
Option B 4.1

Option A scores 3.9 and Option B scores 4.1. These scores only make sense if higher always means better and all candidates first satisfy mandatory requirements.

Practice and self-check

Common mistake

A weighted average must not hide a failed hard requirement. An option that cannot meet required data boundaries should be filtered out before scoring.

Student tasks

1. Change the weights to favor simpler operations and recalculate the ranking.
2. Write three hard constraints and three weighted preferences for a private document assistant.
3. Draw which components hold documents, embeddings, prompts, and audit logs.