Skip to content

The Connected System

How all seven tools hand off to each other, end to end

Core question: You've learned the tools one at a time. But production is not seven islands โ€” it's one machine with seven moving parts. Where exactly does each tool stop, and the next one begin?

โฑ๏ธ Time: ~40 min padho ยท ๐ŸŽš๏ธ Level: Intermediate (synthesis) ยท ๐Ÿ“‹ Pehle chahiye: M0โ€“M7, M7.5

Is module ke baad tum kar paoge: - 2 loops, 8 bridges, 5 threads whiteboard pe draw karo โ€” bina notes ke - Kisi bhi production failure ka diagnosis shuru karo: which bridge broke? - Ek git push se user ke browser tak ka poora path ek sentence mein sunaao โ€” 30-minute interview answer ki tarah

โ†ฉ๏ธ Recall gate โ€” shuru karne se pehle

Pichhle modules se 3 sawaal. Pehle memory se jawab do, phir kholo. (Yeh retrieve karna hi lifetime yaad rakhta hai โ€” dobara padhna nahi.)

  1. (M7) Argo CD "pull model" hai โ€” push model (CI direct deploy) ke mukable mein iska ek concrete security fayda kya hai?
  2. (M6) CI pipeline ne image ECR pe push kar di. Cluster mein naya version pohunchane ke liye CI seedha kubectl apply karta hai โ€” sahi ya galat? Agar galat, toh kya karta hai?
  3. (M4) K8s reconciliation loop kya compare karta hai, aur agar koi pod manually delete kar do, toh kya hota hai?

Jawab

  1. Pull mein Argo cluster ke andar se Git read karta hai โ€” cluster credentials kabhi bahar nahi jaate; CI ko kubeconfig ki zaroorat nahi.   2. Galat โ€” CI sirf k8s manifest mein image tag update karta hai aur Git commit karta hai (Bridge 5); Argo baad mein Git se pull karta hai.   3. Desired state (spec replicas:3) vs current state (live pods:2) โ€” controller loop missing pod wapas banata hai desired poora karne ke liye.

This is the chapter that turns a pile of tools into a mental model. Read it after M0โ€“M7. Everything here is synthesis โ€” no new tools, just the joints between the ones you know. Master this and a whiteboard interview becomes a story you can tell for 30 minutes without pausing.


The 60-second version

The whole stack is two reconciliation loops sharing one Git repo. The outer loop (Terraform โ†’ Ansible) builds the place things run โ€” rarely, carefully; its servers are Pets. The inner loop (Git โ†’ CI โ†’ Registry โ†’ GitOps โ†’ Kubernetes โ†’ Users) ships the app โ€” constantly, automatically; its pods are Cattle. Between every pair of tools is a handoff โ€” a specific output one tool produces that the next consumes. There are 8 of these bridges. And under all of it run 5 golden threads โ€” ideas (reconciliation, state-outside, preview-before-apply, push-vs-pull, idempotency) that repeat in every single tool.

๐Ÿง  Memory hook: 2 loops, 8 bridges, 5 threads. If you can draw those three things, you understand DevOps.


๐Ÿงฉ ๐ŸŽฌ Interactive: sab kaise jude hain โ€” ek baton, poora toolchain8 handoffs step-by-step, har stage se uske deep explainer tak โ€” is chapter ka living version. โ–ถ Kholo

1. The two loops (the map)

flowchart LR
    Git["GIT<br/>source of truth"]:::shared
    TF["Terraform<br/>VPC ยท EC2 ยท RDS ยท ECR"]:::infra
    Ans["Ansible<br/>kubeadm ยท Calico"]:::infra
    K8s["Kubernetes Cluster<br/>where loops meet"]:::run
    CI["CI / Actions<br/>test + build"]:::ci
    ECR["Registry / ECR<br/>immutable image"]:::ci
    Argo["Argo CD<br/>pull-based GitOps"]:::ci
    Users(["Users"]):::run

    Git -->|"outer loop"| TF
    TF -->|"B1 ยท server IPs"| Ans
    Ans -->|"B2 ยท live cluster"| K8s
    Git -->|"inner loop"| CI
    CI -->|"B3 ยท image push"| ECR
    CI -->|"B5 ยท tag commit"| Git
    ECR -. "B3 ยท kubelet pull" .-> K8s
    Git -. "B6 ยท watches" .-> Argo
    Argo -->|"B6 ยท apply"| K8s
    K8s --> Users

    classDef shared fill:#fff9c4,stroke:#f9a825,color:#4a3800;
    classDef infra fill:#fce4ec,stroke:#d81b60,color:#880e4f;
    classDef ci fill:#e3f2fd,stroke:#1976d2,color:#0d47a1;
    classDef run fill:#e0f2f1,stroke:#00897b,color:#004d40;

Outer loop (Pets, runs rarely) and inner loop (Cattle, every push) both read from Git and converge at the Kubernetes cluster.

                        โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
                        โ”‚            GIT  โ€” single source of truth    โ”‚
                        โ”‚   (infra code + app code + K8s manifests)   โ”‚
                        โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                                โ”‚                          โ”‚
   โ•โ•โ•โ•โ•โ•โ• OUTER LOOP โ•โ•โ•โ•โ•โ•โ•โ•โ•โ–ผโ•โ•โ•โ•โ•โ•โ•—       โ•”โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ–ผโ•โ•โ•โ•โ•โ•โ• INNER LOOP โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
   โ•‘  INFRASTRUCTURE  ("build the      โ•‘       โ•‘  DELIVERY  ("ship the app")             โ•‘
   โ•‘  house") โ€” runs RARELY, Pets      โ•‘       โ•‘  runs EVERY PUSH, Cattle                โ•‘
   โ•‘                                   โ•‘       โ•‘                                         โ•‘
   โ•‘  Terraform โ”€โ–ถ Cloud (VPC, EC2,    โ•‘       โ•‘  Code โ”€โ–ถ CI (test+build) โ”€โ–ถ image โ”€โ–ถ    โ•‘
   โ•‘  RDS, ECR, S3/DynamoDB)           โ•‘       โ•‘  Registry (ECR) โ”€โ–ถ manifest-update in   โ•‘
   โ•‘      โ”‚                            โ•‘       โ•‘  Git โ”€โ–ถ Argo CD (pull) โ”€โ–ถ Kubernetes โ”€โ–ถ โ•‘
   โ•‘      โ–ผ                            โ•‘       โ•‘  Service โ”€โ–ถ Ingress โ”€โ–ถ LB โ”€โ–ถ DNS โ”€โ–ถ     โ•‘
   โ•‘  Ansible (configure hosts,        โ•‘       โ•‘  Users                                  โ•‘
   โ•‘  kubeadm, Calico) โ”€โ–ถ live cluster โ•‘       โ•‘                                         โ•‘
   โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•       โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•

The two loops meet in exactly one place: the Kubernetes cluster. The outer loop produces it; the inner loop deploys into it. Git is the shared brain both loops read from.

๐Ÿ‡ฎ๐Ÿ‡ณ Hinglish intuition: Outer loop = neev aur ghar banao (ek baar, mazbooti se). Inner loop = rozana khana banao aur serve karo (baar-baar, machine se). Dono ki common cheez ek hi: recipe book (Git).

The two loops sharing one Git repo, meeting at the Kubernetes cluster

Figure 1: the two loops โ€” outer (infra/Pets) and inner (delivery/Cattle) โ€” meeting at the Kubernetes cluster, Git as the shared source of truth.

The full production pipeline with every handoff (bridge) labelled, from code to user

Figure 2: the same system unrolled left-to-right โ€” every handoff (the 8 bridges below) shown as an arrow from one tool to the next.

Full-resolution renders (and extra views) live in the repo root: devops_two_loops_mental_model.png, two_loops_infra_vs_app.png, devops_full_pipeline_handoffs.png, full_production_flow_end_to_end.png, complete_production_pipeline_flow.png, kubernetes_controller_chain.png.


๐Ÿš€ ๐ŸŽฌ Interactive: Ek git push โ†’ live podPoora inner loop chalte hue dekho โ€” CI โ†’ registry โ†’ Argo pull โ†’ rolling update โ†’ live. โ–ถ Kholo

2. The 8 Bridges

A bridge is a handoff: the concrete artifact tool A produces that tool B consumes. Learn the artifact and the failure mode when the bridge breaks, and you can debug any seam.

[Terraform]โ”€โ‘ โ”€โ–ถ[Ansible]โ”€โ‘กโ”€โ–ถ[Kubernetes cluster]        โ† outer loop bridges
     โ”‚                                    โ–ฒ
     โ”‚(builds ECR, RDS)                   โ”‚โ‘ข (image pulled from ECR into a Pod)
     โ–ผ                                    โ”‚
[Docker image]โ”€โ‘ขโ”€โ–ถ[Registry/ECR]โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
[git push]โ”€โ‘ฃโ”€โ–ถ[CI/Actions]โ”€โ‘คโ”€โ–ถ[Git manifest]โ”€โ‘ฅโ”€โ–ถ[Argo CD]โ”€โ–ถ[Kubernetes]
[Pod]โ”€โ‘ฆโ”€โ–ถ[Service]โ”€โ–ถ[Ingress]โ”€โ–ถ[User]        [Pod]โ”€โ‘งโ”€โ–ถ[RDS]
# Bridge The artifact that crosses Failure mode when it breaks
1 Terraform โ†’ Ansible terraform output server public IPs โ†’ written into Ansible's inventory.ini Wrong/empty IP โ†’ Ansible UNREACHABLE
2 Ansible โ†’ Kubernetes 3 playbooks (common โ†’ kubeadm init on master โ†’ kubeadm join on workers) produce a live cluster Missing containerd/swap-on โ†’ kubeadm fails; wrong join token โ†’ worker never joins
3 Docker โ†’ Pod image pushed to ECR, then pulled by kubelet onto a node Bad tag / no ECR auth โ†’ ImagePullBackOff
4 Push โ†’ CI a git push to main matches on: push: โ†’ runs the workflow Wrong branch filter โ†’ pipeline never triggers
5 โญ CI โ†’ Git manifest CI updates the image tag inside the K8s manifest (via sed) and commits โ€” it does not apply to the cluster Forgot to update the manifest โ†’ cluster keeps the old image forever
6 Git โ†’ Argo โ†’ Kubernetes Argo pulls the changed manifest and applies it โ†’ rolling update Argo OutOfSync never syncs โ†’ new version never lands
7 Pod โ†’ Service โ†’ User Service (stable IP) fronts ready pods via EndpointSlice; NodePort/Ingress exposes it No ready pods in the slice โ†’ Connection refused / 503
8 Pod โ†’ RDS app connects psycopg2 โ†’ rds-endpoint:5432 using an env-var host + a secret password Wrong SG / password โ†’ connection timeout or auth error

Bridge 5 is the one interviewers probe. Say it precisely: "CI never touches the cluster. It only changes the desired state in Git โ€” updates the image tag and commits. Argo CD, running inside the cluster, notices Git changed and pulls it. That separation is why the pipeline needs no cluster credentials sitting in GitHub."

๐Ÿ‡ฎ๐Ÿ‡ณ Hinglish intuition (Bridge 5): CI menu update karta hai (Git me naya tag likhta), khud khana serve nahi karta. Serve karna Argo ka kaam hai. Isliye CI ke paas cluster ki chaabi rakhne ki zaroorat nahi.


3. The 5 Golden Threads

If the 8 bridges are the joints, the 5 threads are the DNA โ€” the same idea wearing a different costume in each tool. Spot them and 9 tools collapse into 1 idea.

Thread 1 โ€” Reconciliation

A control loop constantly compares desired state to current state and drives current โ†’ desired. It never stops.

Tool Desired state Who reconciles
Terraform .tf files + tfstate you, when you run apply
Ansible playbook state=present you, when you run the playbook
Kubernetes Deployment replicas: 3 the controller loop, continuously
Argo CD the manifest in Git Argo, continuously (~3 min)

The leap: Terraform/Ansible reconcile when you ask; Kubernetes/Argo reconcile forever, on their own. That's why a deleted pod comes back and a hand-edited object gets reverted. Learn reconciliation once in M4, and Argo's selfHeal is free.

๐Ÿ‡ฎ๐Ÿ‡ณ Hinglish intuition: Chowkidar jo 24/7 ginta hai โ€” "3 hone chahiye, 2 hain, ek aur banao." Yehi loop har jagah hai.

Thread 2 โ€” State outside, compute disposable

Keep durable state (database, tfstate) outside the throwaway compute. Then servers and pods become cattle โ€” kill and replace freely.

  • tfstate โ†’ S3 (not your laptop) โ†’ any machine can run Terraform.
  • App is stateless โ†’ pod dies โ†’ identical replacement, no data lost.
  • DB is stateful โ†’ usually lives in RDS outside the cluster, backed up โ€” never a casual pod (i.e. never a bare Deployment with an emptyDir). Running it in-cluster is legitimate, but only deliberately: a StatefulSet with volumeClaimTemplates, tested backups, and someone owning failover. See the explicit trade-off in 28 โ€” Helm in the real world, where billfree runs a self-managed Postgres StatefulSet on purpose. Managed = less control, far less operational burden; self-managed = full control, and the pager is yours.
  • Because compute is disposable, you can run it on Spot instances (M5) and save 70%.

๐Ÿ‡ฎ๐Ÿ‡ณ Hinglish intuition: State bahar nikaalo โ†’ server/pod disposable ban jaata hai. Yehi cattle-not-pets ka asli mechanism hai.

Thread 3 โ€” Preview before apply

Never change reality blind. Every tool has a dry-run.

Tool Preview command It shows
Terraform terraform plan the diff before it touches AWS
Ansible ansible-playbook --check what would change
Kubernetes kubectl apply --dry-run=server validation without applying
CI/CD the test gate "does it even build & pass?" before shipping

๐Ÿ‡ฎ๐Ÿ‡ณ Hinglish intuition: Bill dekho, phir payment karo. plan = bill, apply = payment.

Thread 4 โ€” Push vs Pull

Who initiates the change โ€” the doer reaching in, or the target reaching out?

Model Tools Security consequence
Push Ansible, GitHub Actions The initiator holds the target's credentials (SSH keys, cluster creds)
Pull Argo CD The agent lives inside the target and pulls โ€” no external creds leave the cluster
The middle Git Neither pushes nor pulls; it just holds the truth both sides agree on

This is why GitOps is considered more secure than CI-driven kubectl apply: with pull, GitHub never needs your cluster's admin credentials.

Thread 5 โ€” Idempotent

Run it again, nothing new happens. The operation is SET, not +=.

terraform apply twice โ†’ no duplicate VPC. Ansible playbook twice โ†’ second run is all ok, zero changed. kubectl apply twice โ†’ same object. This is what makes automation safe to retry โ€” the single most important property for a tool a machine runs unattended.

๐Ÿ‡ฎ๐Ÿ‡ณ Hinglish intuition: Switch (idempotent: "ON" dabao 10 baar, bulb ON hi rahega) vs Counter (+=: har dabane pe badhta). DevOps tools switch jaise hain.


4. The full end-to-end walkthrough

Now watch a single feature travel the entire system, crossing every bridge in order.

Day 0 โ€” Setup (outer loop, done once)

1. terraform apply        โ†’ VPC, 3ร— EC2, RDS, ECR, S3+DynamoDB lock created   [state โ†’ S3, Thread 2]
2. terraform output       โ†’ 3 server IPs                                       [Bridge 1]
3. edit inventory.ini      โ†’ paste the IPs                                     [Bridge 1]
4. ansible-playbook 1-common.yml   โ†’ containerd, swapoff, sysctl on all 3     [Thread 5: idempotent]
5. ansible-playbook 2-master.yml   โ†’ kubeadm init + Calico CNI                 [Bridge 2]
6. ansible-playbook 3-workers.yml  โ†’ kubeadm join ร— 2                          [Bridge 2]
   โ–ถ Result: a live Kubernetes cluster. The two loops now share this cluster.
flowchart LR
    Push["git push<br/>to main"]:::ci
    Test["CI ยท pytest<br/>test gate"]:::ci
    Build["CI ยท docker build<br/>push to ECR"]:::ci
    ManifestUp["CI updates<br/>image tag in Git"]:::ci
    ArgoSync["Argo CD<br/>OutOfSync then apply"]:::ci
    RollingUp["K8s rolling update<br/>new ReplicaSet"]:::run
    Svc["Service<br/>EndpointSlice"]:::run
    RDS[("RDS<br/>state outside")]:::run
    User(["User<br/>200 OK"]):::run

    Push -->|"B4 triggers"| Test
    Test -->|"pass"| Build
    Build -->|"B3 image in ECR"| ManifestUp
    ManifestUp -->|"B5 commit"| ArgoSync
    ArgoSync -->|"B6 apply"| RollingUp
    RollingUp -->|"B7 ready pods"| Svc
    Svc --> User
    RollingUp -->|"B8 connect"| RDS

    classDef ci fill:#e3f2fd,stroke:#1976d2,color:#0d47a1;
    classDef run fill:#e0f2f1,stroke:#00897b,color:#004d40;

Every git push crosses Bridges 4 โ†’ 5 โ†’ 6 โ†’ 7 (and Bridge 8 for the database) before reaching the user.

Day 1+ โ€” Delivery (inner loop, every push)

7.  git push origin main            โ†’ matches on: push: [main]                 [Bridge 4]
8.  CI: pytest                       โ†’ test gate; fail = nothing ships          [Thread 3]
9.  CI: docker build + push to ECR   โ†’ image tagged with ${{ github.sha }}      [Bridge 3, immutable tag]
10. CI: sed image tag in k8s/deploy.yaml โ†’ git commit + push                    [Bridge 5 โญ: change Git, NOT cluster]
11. Argo CD notices Git changed      โ†’ 3-way diff โ†’ OutOfSync โ†’ apply           [Bridge 6, Thread 1]
12. Kubernetes rolling update        โ†’ new ReplicaSet, readiness-gated          [Thread 1]
13. Service EndpointSlice updates     โ†’ traffic shifts to new pods              [Bridge 7]
14. Pod โ†’ RDS:5432                    โ†’ serves the request                      [Bridge 8]
15. User hits Ingress โ†’ Service โ†’ Pod โ†’ 200 OK

When something goes wrong

Bug in prod?      โ†’ git revert <bad commit> โ†’ Argo pulls the old manifest โ†’ rollback.  [Thread 1 + Git = time machine]
Someone kubectl-edited a live pod?  โ†’ Argo selfHeal reverts it to match Git.           [Thread 1 + Thread 4]
tfstate deleted?  โ†’ Terraform thinks NOTHING exists โ†’ would try to rebuild everything.  [Thread 2: guard your state]

๐Ÿ‡ฎ๐Ÿ‡ณ Hinglish intuition (rollback): Time machine โ†ฉ๏ธ โ€” galti hui to Git me peeche jao, Argo cluster ko wapas us waqt pe le aayega. Rollback = sirf Git history.


5. The connection web (one glance)

flowchart TD
    TF["Terraform<br/>builds VPC ยท EC2 ยท RDS ยท ECR ยท S3"]:::infra
    Ans["Ansible<br/>kubeadm ยท cluster bootstrap"]:::infra
    Git["GIT<br/>source of truth"]:::shared
    CI["CI / Actions"]:::ci
    ECR["Registry / ECR"]:::ci
    Argo["Argo CD<br/>pull-based"]:::ci
    K8s["K8s Cluster<br/>Deployment ยท Service ยท Ingress"]:::run
    RDS[("RDS<br/>state outside cluster")]:::run
    Users(["Users"]):::run

    TF -->|"B1 ยท IPs"| Ans
    TF -->|"creates"| ECR
    TF -->|"creates"| RDS
    Ans -->|"B2 ยท live cluster"| K8s
    Git -->|"B4 ยท triggers"| CI
    CI -->|"B3 ยท image push"| ECR
    CI -->|"B5 ยท tag update"| Git
    ECR -. "B3 ยท kubelet pull" .-> K8s
    Git -. "B6 ยท watches" .-> Argo
    Argo -->|"B6 ยท apply"| K8s
    K8s -->|"B7"| Users
    K8s -->|"B8 ยท connect"| RDS

    classDef shared fill:#fff9c4,stroke:#f9a825,color:#4a3800;
    classDef infra fill:#fce4ec,stroke:#d81b60,color:#880e4f;
    classDef ci fill:#e3f2fd,stroke:#1976d2,color:#0d47a1;
    classDef run fill:#e0f2f1,stroke:#00897b,color:#004d40;

All eight bridges in one view โ€” Terraform builds the foundation, Git is the hub both loops read from, all paths converge at the Kubernetes cluster.

                         โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ TERRAFORM โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
                         โ”‚  builds: VPC, EC2, RDS, ECR, S3    โ”‚
                         โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                    IPs (Bridge1)โ”‚                 โ”‚creates ECR & RDS
                                 โ–ผ                 โ”‚
                            ANSIBLE                 โ”‚
                    (kubeadm โ†’ cluster, Bridge2)    โ”‚
                                 โ”‚                 โ”‚
              โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
              โ”‚                 KUBERNETES  CLUSTER                   โ”‚
              โ”‚   Deploymentโ†’ReplicaSetโ†’Pod   โ€ข   Serviceโ†’Ingress     โ”‚
              โ”‚        โ–ฒ pulls image (Bridge3)   โ”‚ serves users       โ”‚
              โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ฒโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                       โ”‚              โ”‚Bridge6    โ”‚ Bridge7โ†’ USERS
                  ECR  โ”‚         ARGO CD (pull)   โ”‚
                       โ”‚              โ–ฒ           โ”‚ Bridge8
              โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”       โ”‚Bridge5    โ–ผ
              โ”‚ DOCKER image  โ”‚โ—€โ”€โ”€ CI/ACTIONS โ”€โ”€ GIT โ”€โ”€โ–ถ RDS (state, outside)
              โ”‚  (Bridge4: git push triggers CI)   โ”‚
              โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

  Threads woven through ALL boxes:  โ‘  reconcile  โ‘ก state-outside  โ‘ข preview  โ‘ฃ push/pull  โ‘ค idempotent

6. Who owns what, in a real org

Systems thinking includes people. On a real team these boxes have different owners โ€” knowing this is what separates a junior from someone who's shipped in production.

Layer Typical owner What they're paged for
Terraform / cloud infra Platform / Infra / Cloud team VPC, IAM, account limits, cost
Ansible / host config Platform / SRE drift, patching, cluster bootstrap
Docker image / app code Product / app developers the app builds and runs
CI/CD pipeline Platform provides, devs consume broken builds, flaky tests
Kubernetes cluster Platform / SRE node health, capacity, upgrades
GitOps / Argo Platform / SRE sync failures, prod drift
Observability & on-call SRE + the owning dev team SLO burn, incidents (see M8)
The database (RDS) DBA / data platform backups, failover, slow queries

Senior insight: the boundaries above are also blast-radius boundaries. When you design a change, ask "which of these owners do I need to tell, and what breaks if this box fails?" That question is 80% of an architecture review.


7. Who does what โ€” the tool recall drill

Section 6 was about people. This one is about tools: which tool owns which job. Interviewers probe this constantly ("who deploys to the cluster?", "who runs the container?") โ€” so drill it until the answer is instant. Cover the right column and quiz yourself.

The layer stack (memorize this ONE picture):

 App โ†’ Pods / Containers          โ† KUBERNETES  (schedules, restarts, scales)
 Node OS: containerd, kubeadm      โ† ANSIBLE     (installs & configures the node)
 The server: VM, VPC, subnet, disk โ† TERRAFORM   (creates the infrastructure)

๐Ÿ‡ฎ๐Ÿ‡ณ Hinglish intuition: "Terraform building banata, Ansible kitchen set karta (stove/utensils = containerd/kubeadm), Kubernetes chef hai jo dishes (pods) pakata." Teen tools, teen layers, isi order me. Ansible cook nahi karta; K8s plumbing nahi bichata.

Task โ†’ owner (the master table):

The task Who does it Layer
Create the VM / VPC / disk Terraform infra
Install runtime + kubeadm, disable swap, kernel tuning Ansible node OS
Form the cluster (init/join) kubeadm (run by Ansible) node OS
Build the container image Docker (on the CI runner) build
Store the image Registry (ECR / GHCR / Hub) build
Actually run a container containerd (under kubelet) node
Decide which node a pod runs on kube-scheduler control plane
Keep N replicas alive (self-heal) Deployment โ†’ ReplicaSet cluster
Route traffic to the right pods Service (+ kube-proxy) / Ingress cluster
Persist data beyond a pod PVC โ†’ PV (โ†’ EBS) storage
Hold config / secrets ConfigMap / Secret cluster
Test โ†’ build โ†’ scan the code CI (Jenkins / GitHub Actions) pipeline
Deploy to the cluster ArgoCD (GitOps) โ€” not CI! delivery
Observe (metrics/logs/alerts) Prometheus / Grafana / Loki observability

โš ๏ธ The trap most people fall into: thinking CI deploys. It doesn't โ€” CI writes to Git; ArgoCD reads from Git and deploys. That separation (Bridge 5) is the whole point of GitOps.

Where does each tool RUN? (another recall-killer)

Laptop / CI runner (clients) Master node Worker nodes
terraform (calls cloud API) api-server + etcd kubelet
ansible (SSH push) scheduler, controllers kube-proxy
kubectl (calls API) ArgoCD (pods) containerd โ† your pods run here
docker build, git push Prometheus / Grafana (pods)

Golden rule: clients (Terraform, Ansible, kubectl, docker-build) run from your laptop/CI and command remote things; servers (containerd, kubelet, ArgoCD) run on the cluster. ArgoCD lives inside the cluster โ€” that's what makes GitOps pull-based.

Rapid-fire (cover the answers, drill until instant):

Who does X? โ€” jawab dekho
  • Creates the EC2 instance? โ†’ Terraform
  • Installs containerd + kubeadm? โ†’ Ansible
  • Forms the cluster? โ†’ kubeadm
  • Builds the Docker image? โ†’ CI (Docker on the runner)
  • Stores the image? โ†’ Registry
  • Runs the container on a node? โ†’ containerd (under kubelet)
  • Decides which node? โ†’ scheduler
  • Recreates a dead pod? โ†’ ReplicaSet (via the Deployment)
  • Load-balances to pods? โ†’ Service
  • Routes external HTTP by path? โ†’ Ingress
  • Gives a pod persistent storage? โ†’ PVC โ†’ PV (EBS)
  • Deploys the new image to the cluster? โ†’ ArgoCD (from Git) โ€” not CI
  • Watches Git for changes? โ†’ ArgoCD
  • Collects metrics? โ†’ Prometheus ยท Shows dashboards? โ†’ Grafana

๐Ÿ“Ž The actual commands for each tool live in Command Cheat-Sheets & Labs โ€” this drill is who, that chapter is how.


Summary

  • The stack is 2 loops (infra/outer/Pets + delivery/inner/Cattle) sharing 1 Git repo; they meet at the Kubernetes cluster.
  • 8 bridges are the handoffs; each has a concrete artifact and a signature failure mode. Bridge 5 (CI changes Git, not the cluster) is the one to nail.
  • 5 golden threads โ€” reconciliation, state-outside, preview-before-apply, push-vs-pull, idempotent โ€” recur in every tool. Learn each once, reuse everywhere.
  • A feature travels: push โ†’ test โ†’ build โ†’ ECR โ†’ manifest-update โ†’ Argo โ†’ rolling update โ†’ Service โ†’ RDS โ†’ user. Rollback is just git revert.
  • Real orgs split these layers across owners; the boundaries double as blast-radius boundaries.

Self-check quiz

Pehle memory se jawab do, phir neeche kholo.

  1. Name the 8 bridges and one failure mode for each.
  2. Which bridge is the CIโ†’CD seam, and why does CI not touch the cluster?
  3. Give the same golden thread (reconciliation) as it appears in four different tools.
  4. tfstate is deleted. What does the next terraform plan think, and why is that dangerous?
  5. Why is Argo's pull model considered more secure than CI-driven kubectl apply (push)?
  6. Someone runs kubectl edit on a live Deployment. Walk through what Argo does and which thread explains it.
  7. Where does durable state live in this system, and what does that let you do with compute?
Jawab dekho
  1. B1 TFโ†’Ansible: IPsโ†’inventory; UNREACHABLE. B2 Ansibleโ†’K8s: 3 playbooks live cluster banate hain; kubeadm fail / wrong join token. B3 Dockerโ†’Pod: ECR image kubelet pull karta hai; ImagePullBackOff. B4 pushโ†’CI: git push workflow trigger karta hai; wrong branch filter. B5โญ CIโ†’Git manifest: image tag update+commit, cluster seedha nahi; manifest update bhool gaye โ†’ old image chalta rehta. B6 Gitโ†’Argoโ†’K8s: Argo pull+apply karta hai; OutOfSync sync nahi hua. B7 Podโ†’Serviceโ†’User: EndpointSlice ready pods route karta hai; koi ready pods nahi โ†’ 503. B8 Podโ†’RDS: psycopg2 se RDS:5432 connect; wrong SG ya bad password โ†’ timeout.
  2. Bridge 5. CI cluster ko seedha nahi chhuta โ€” sirf Git mein manifest update karta hai. Argo (cluster ke andar) Git se pull karke apply karta hai. Isliye CI ke paas cluster credentials nahi chahiye.
  3. Terraform: apply .tf files vs tfstate/live AWS compare karta hai. Ansible: playbook state=present vs host reality. Kubernetes: Deployment spec vs live ReplicaSet/Pod count, continuously. Argo CD: Git manifest vs live cluster, har ~3 min.
  4. Terraform koi state nahi dekhta โ†’ sochta hai kuch bhi provisioned nahi โ†’ plan "sab kuch create karo" kehta hai โ†’ apply karo toh duplicate VPC/EC2/RDS banane ki koshish โ€” name conflicts ya costly duplicates. State file = Terraform ki yaadaasht.
  5. Pull (Argo): creds cluster ke andar hain, Argo Git ko outbound read karta hai โ€” CI ke paas kuch nahi. Push (CI kubectl): CI ke paas kubeconfig โ€” CI hack = cluster gaya.
  6. Argo OutOfSync detect karta hai (Cause B โ€” cluster drifted). selfHeal:true hai toh Git manifest wapas apply karta hai, manual edit revert. Thread 1 (reconciliation) โ€” current state ko desired state tak drive karo, hamesha.
  7. Durable state: tfstate โ†’ S3; DB โ†’ RDS (cluster ke bahar). Yeh compute ko disposable banata hai โ€” pods aur servers Cattle hain, kill and replace freely, koi data loss nahi.

Hands-on lab

On paper (no cloud spend), from memory: 1. Draw the two loops and label which tools live in each. 2. Draw all 8 bridges as arrows; annotate each arrow with its artifact and failure mode. 3. Pick one golden thread and write the one line proving it appears in Terraform, Ansible, Kubernetes, and Argo. 4. Trace a single git push all the way to a user's browser, naming every bridge it crosses. 5. Now trace a rollback and a selfHeal event. Compare โ€” which threads does each use?

โœ… Sahi hua to aisa dikhega: If you can do all five from memory, you're ready for a systems-design interview.

Interview questions

  1. "Walk me through what happens from git push to the change being live for users." (This chapter, section 4 โ€” the whole answer.)
  2. "CI or Argo โ€” who applies to the cluster? Why is that separation valuable?"
  3. "What's the difference between drift and lost state, and how does each get fixed?" (Thread 1 + Thread 2; see M1.)
  4. "Your Terraform state file is gone. What's your recovery plan and what's the blast radius?"
  5. "Why do we keep the database in RDS instead of a Kubernetes pod?" (Thread 2 โ€” state outside; see M4.)
  6. "Explain push vs pull deployment models and their security tradeoff." (Thread 4.)

Production challenge

A deploy went out 20 minutes ago. Users report 502s. You have: green CI, Argo showing Synced/Healthy, pods Running. Using the bridges as a checklist, write the ordered sequence of checks you'd run โ€” from Bridge 7 (Serviceโ†’Pod, is anything in the EndpointSlice?) back through Bridge 6 (did Argo actually apply the intended image?) to Bridge 8 (can the pod reach RDS?). Name the single command you'd run at each bridge. (Cross-reference the per-hop debug commands in M9 ยงThe full instrumented request lifecycle.)


Next โ†’ M8 โ€” Observability & SRE: now that you can draw the system, learn to see inside it while it's running.