I have watched a lot of capable people get stuck moving from "I built something in a notebook" to "it runs in production." Almost none of them got stuck on the AI part.
They got stuck because the code only existed on one laptop. Because the API key was committed to the repo. Because it worked locally and died in the container with a dependency error. Because there was no way to roll back when the prompt change made things worse.
These are software engineering problems. They are boring. They are also exactly why most AI pilots do not reach production.
Python — the parts that matter for AI work
You do not need to be a Python expert. You need a specific, fairly narrow subset:
- Data structures and comprehensions — you will spend most of your time reshaping JSON responses. Dicts, lists, and comprehensions are the daily tools.
- Async and concurrency — this one is not optional. LLM calls take seconds. Making twenty of them sequentially takes a minute; making them concurrently with
asyncio.gathertakes seconds. The difference between a usable and unusable system is often just this. - Type hints and Pydantic — models return unstructured text and you need structured data. Pydantic models give you validation, useful errors, and a schema you can hand to the model for structured output. This single library removes an enormous category of bug.
- Error handling and retries — APIs rate-limit, time out, and return malformed JSON. Exponential backoff with jitter is table stakes, not an optimisation.
- Virtual environments —
uvorvenv. "Works on my machine" is nearly always a dependency-isolation failure.
REST APIs — you are an API consumer first
Nearly all AI engineering is calling somebody's HTTP API and handling what comes back. What you need to actually understand:
- Status codes as control flow — 429 means back off and retry; 400 means your request is wrong and retrying will not help; 500 means retry with backoff. Treating all errors identically produces systems that hammer a rate-limited endpoint until they are banned.
- Authentication and key handling — keys go in environment variables or a secrets manager. Never in code, never in a notebook you will share, never in the repo. Rotate them.
- Streaming responses — server-sent events. The difference between a UI that feels instant and one that appears frozen for eight seconds.
- Timeouts — always set them explicitly. A hung request with no timeout will eventually take down a worker pool.
- Idempotency — if a request times out, did it actually run? For anything that costs money or changes state, this question needs a real answer.
Git — the discipline that makes prompts reviewable
Git matters more in AI work than in conventional software, for a reason that is not obvious at first: prompts are code.
A prompt change alters system behaviour as surely as a logic change. If prompts live in a Google Doc, or hardcoded inline with no history, then you cannot answer "what changed on Tuesday when quality dropped?" — and you cannot roll back.
Keep prompts in version control as files. Review prompt changes in pull requests. Tag versions and record which version produced which output in your traces. This is the connective tissue between Git and observability, and it is what lets you correlate a behaviour change to a specific commit.
Beyond that: branching, meaningful commit messages, pull requests, and the ability to revert. Nothing exotic.
Cloud basics — enough to deploy and not overspend
- Containers — Docker. Package the app with its dependencies so the environment is identical everywhere. This eliminates the largest single category of deployment failure.
- Secrets management — cloud-native secret stores, not environment files checked into the repo.
- Networking fundamentals — enough to reason about why a service cannot reach a database, and about egress when your compliance team asks where prompt data travels.
- Cost model — AI workloads have a genuinely unusual cost profile: usage-based, spiky, and easy to accidentally 10x with a retry loop. Set budget alerts before your first production deploy.
- Data residency — for Malaysian teams under PDPA, and especially in financial services under BNM RMiT expectations, knowing which region processes your data is a compliance requirement rather than an architectural preference.
What you can safely skip
Worth saying explicitly, because people burn months here: you do not need to be able to derive backpropagation, implement a transformer from scratch, or train a model. Those are ML research skills. AI engineering is systems engineering applied to probabilistic components.
Understand tokenisation, context windows, temperature, and why models hallucinate — conceptually. Skip the mathematics unless you are doing research.
A realistic learning path
- Weeks 1–2 — Python: async, Pydantic, error handling. Build a script calling an LLM API with retries and structured output.
- Weeks 3–4 — Wrap it in FastAPI. Add streaming. Containerise it with Docker.
- Weeks 5–6 — Git discipline: prompts as versioned files, PR review, tagged releases. Deploy the container somewhere real.
- Weeks 7–8 — Add evaluation and tracing. Now you have a system rather than a script.
Eight weeks of unglamorous work puts you ahead of a substantial share of people currently calling themselves AI engineers — because the fundamentals are what most people skipped on the way to the interesting part.
Our AI Engineering programme covers this foundation and the production stack built on top of it — HRD Corp SBL-KHAS claimable for eligible Malaysian employers.
