GenAI Engineering: LLM Foundations

Large Language Models power almost every AI product shipped in 2026 — yet most engineers using them have never opened the box. GenAI Engineering: LLM Foundations teaches how an LLM actually predicts the next token, why a Transformer attends the way it does, what really happens during RLHF, and why models hallucinate — through 1,700+ practice MCQs with instant explanations on every wrong answer.

This is the entry module of the Generative AI Engineering track on Abekus — the four-course path to becoming an AI Engineer. It assumes no prior AI or ML background, but goes deep enough that ML practitioners pick up the parts the day-job never forced them to learn — RoPE scaling, PagedAttention, DPO vs RLHF, the lost-in-the-middle problem, judge-model bias.

• Format — 1,700+ MCQs with instant explanations
• Duration — about 19 hours of focused practice, typically 3–5 weeks at 30–40 questions a day
• Level — beginner to advanced; no prior AI/ML experience required
• Cost — free, with a public-URL completion certificate
• Audience — engineers pivoting to AI Engineer roles, AI/ML interview candidates, self-taught builders
• Next courses in the track — Prompt Engineering, RAG Systems, AI Agents

Three audiences land on this page. Working software engineers and data scientists who are about to start using LLM APIs at work and want grounding before they ship — not another "build a chatbot in 20 minutes" tutorial. Final-year engineering and MCA students targeting AI Engineer / ML Engineer interviews, where bias-variance is a tier-1 question but so is "explain RoPE" and "what is KV cache". Self-taught builders who have shipped something with OpenAI's SDK and now realise they cannot reason about why their output is bad without understanding decoding, alignment, or hallucination mechanics.

Foundations

• Language Models — next-token prediction, autoregressive generation, the curse of dimensionality in n-grams, why scale unlocks emergent abilities like in-context learning, chain of thought and instruction following, and how base models differ from instruct and chat variants
• Tokenization — BPE, WordPiece, SentencePiece schemes, why non-English text costs more tokens, leading-space conventions, byte fallback, special tokens, and the tradeoffs of vocabulary size against subword splitting on rare words
• Embeddings — dense vector representations, contextual vs static embeddings, cosine vs dot product similarity, CLS vs mean vs max pooling, the anisotropy problem, embedding arithmetic (king − man + woman), and out-of-distribution behaviour
• Transformer Intuition — query-key-value attention, scaled dot product attention, multi-head splits and induction heads, positional encodings (sinusoidal, RoPE, ALiBi), causal vs padding masks, and the KV cache during generation

Depth

• Pretraining and Training Objectives — causal LM, masked LM, span corruption, denoising objectives, the Chinchilla compute-optimal token-to-parameter ratio, data deduplication impact, contamination, and large-scale optimisation (AdamW, warmup, mixed precision)
• Decoding and Sampling — greedy vs beam search, beam search degeneration, temperature scaling, top-k, top-p, min-p, repetition penalty, structured-output enforcement, JSON-schema guided decoding, and speculative decoding
• Context Windows and Long Context — KV-cache memory growth, quadratic attention cost, position interpolation, RoPE scaling, sliding window attention, Flash Attention, the lost-in-the-middle problem, and needle-in-a-haystack evaluation
• Instruction Tuning and Alignment — SFT vs base behaviour, chat template formatting, RLHF reward modelling and PPO, DPO, Constitutional AI, and alignment failure modes like sycophancy, reward hacking, and refusal overgeneralisation

Interview readiness

• Hallucination and Factuality — why next-token objectives diverge from truth, knowledge cutoff, extrapolation outside training, log-probability as confidence proxy, self-consistency sampling, citation enforcement, and tool-augmented factuality
• LLM Evaluation — MMLU, GSM8K, HumanEval, MBPP, BigBench and MT-Bench benchmarks; LLM-as-judge and pairwise preference grading; benchmark contamination, prompt sensitivity, judge-model bias, and metric gaming
• LLM Families and Architectures — GPT, Claude, Llama, Mistral and Gemini characteristics; dense vs MoE; grouped-query and multi-query attention; int4 quantization, KV-cache compression, PagedAttention, and continuous batching
• LLM Foundations Mastery — head-to-head comparisons (encoder vs decoder, SFT vs RLHF vs DPO, RoPE vs ALiBi, top-k vs top-p) and when-to-use selection across temperature, beam search, context window, base vs instruct, and model size

LLM Foundations teaches how the model works under the hood; Prompt Engineering teaches how to make a model do what you want from the outside. If you have never thought about tokenization, attention, or decoding parameters, take LLM Foundations first — every advanced prompting pattern (chain of thought, self-consistency, structured output, prompt injection defence) makes much more sense once you understand what is happening inside the model when it reads your prompt. If you already know transformers and just want to get good at the API surface, you can start with Prompt Engineering and circle back. Most learners take both, in this order.

Most engineers using LLMs in production carry a few persistent misconceptions. This course is designed to surface and correct them through targeted MCQs. Among the most common:

• KV-cache memory grows linearly with sequence length per layer — not constantly — and quickly dominates GPU memory at long context lengths.
• Temperature 0 does not guarantee deterministic outputs in production — server-side batching, sampling library defaults, and GPU non-determinism all leak in.
• A log-probability is not a confidence score. A model can produce high log-probs on confidently wrong outputs, especially after instruction tuning.
• RoPE and ALiBi are not interchangeable — RoPE extrapolates poorly beyond the training context window without explicit scaling, while ALiBi degrades more gracefully.
• MoE models have a parameter count and an active parameter count — only the latter drives per-token compute.
• Lost-in-the-middle is real and reproducible — retrieved chunks placed at the middle of a long context are weighted less than chunks at either end, regardless of relevance.
• LLM-as-judge introduces its own biases — judge models tend to prefer longer answers, answers from the same family, and answers in their own writing style.
• Sycophancy is a measurable alignment failure — RLHF models will agree with confidently wrong user assertions more often than base models will.

Each of these has a dedicated cluster of practice questions in the curriculum.

Active recall beats re-reading. A learner who answers 60 LLM questions, gets 18 wrong, and reads the explanation each time will remember the material a month later — while a learner who watches the same content as a 4-hour video will not. The Abekus format puts you on the hook for an answer every 40 seconds, surfaces the gap immediately, and the AI guide re-surfaces missed concepts in later sessions so weak spots compound less.

One question at a time. Pick an answer. If you are wrong, the explanation appears immediately — usually a paragraph that walks through the reasoning, names the concept by its standard term, and points at the related labels in the curriculum. The AI guide watches your accuracy by subtopic and prioritises the weak ones in the next practice session. There is no video, no scrolling lecture, no playback speed to fiddle with — just focused retrieval practice.

The honest math: 1,700+ MCQs at about 40 seconds each (including reading explanations on the wrong ones) is roughly 19 hours of pure focus. Spread that over 30–40 questions a day and you finish in about 4 weeks. At 60 questions a day, about 2 weeks. The course is designed for ad-hoc sessions of 20–30 minutes — you do not need to block off a Saturday. Most learners finish the four-module Generative AI Engineering track in 3–4 months at this pace.

The mastery topic at the end is built around the questions that actually get asked in AI Engineer technical screens in 2026 — encoder vs decoder, KV cache, RoPE vs ALiBi, RLHF vs DPO, when to lower temperature, when to use beam search, what hallucination really is. Every "In Practice" label is a named real-world system (LLM in semantic search, LLM in structured data extraction, LLM in code completion) so the scenario-based round has practiced ground. The course will not teach you to build a RAG system or an agent end-to-end — those are the third and fourth modules in the track — but it will make you able to reason about why a given system is failing.

The natural next step is Prompt Engineering — the second module in the Generative AI Engineering track. RAG Systems and AI Agents come third and fourth, in either order. Finishing all four unlocks the series certificate. Independently of the GenAI track, learners targeting Data Scientist or ML Engineer roles often pair this with the Machine Learning Foundations and Statistics for Data Science courses on Abekus.