> For the complete documentation index, see [llms.txt](https://naz.gitbook.io/my-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://naz.gitbook.io/my-docs/get-started/roadmap_for_backend_from_first_principles.md).

# Backend From First Principles — Roadmap

## 1. Why This Series Exists

Backend engineering is a very wide scope. When we say backend engineering, it is much more than building a set of CRUD APIs. Backend engineering is about building **reliable**, **scalable**, **fault-tolerant**, and **maintainable** codebases and efficient systems.

If one were to start today to learn backend development, there are a lot of resources — at least a hundred. But how do you decide what to learn? How do you prioritize? How do you see the big picture when and how all these different concepts come together?

This is the reason it takes people years to get their head around a lot of these concepts and principles. People primarily start with a limited scope of training — whether it is from a college, a boot camp, or a simple course — and they eventually build on top of that with trial and error and with the help of other developers over time.

The second problem is people start with backend development from a particular **language or framework's point of view** — it could be Express, Spring Boot, or Ruby on Rails. The problem with that is you look at the problems you solve with the lens of your particular language and ecosystem, and there are blind spots.

Imagine you have to switch to a different language. Let's say you were working with Ruby on Rails for years and one day your company decided to migrate to Golang for performance reasons. In that situation, how much of your knowledge can you transfer if you don't understand the underlying systems?

This series is built on **foundational concepts** of a backend system — drawn from various books, open-source codebases, and industry practice. The goal is language-agnostic understanding.

***

## 2. High-Level Understanding of Backend Systems

We start with a very high-level understanding of how backend systems work behind the scenes. We look at how a request from a browser flows through different hops — the network, firewalls, over the internet — and how it is routed to a backend server situated in a remote AWS server. Then how the server responds to that request.

We look at what the response looks like. That should give a pretty vivid idea of how systems communicate — how a client communicates with a server, and how the server responds.

```mermaid
sequenceDiagram
    participant B as Browser
    participant FW as Firewall
    participant INT as Internet
    participant SRV as Backend Server

    B->>FW: HTTP Request
    FW->>INT: Forward
    INT->>SRV: Route to Server
    SRV->>INT: HTTP Response
    INT->>FW: Forward
    FW->>B: Deliver Response
```

***

## 3. HTTP Protocol

From there we move to understanding the **HTTP protocol** — what role it plays and how communication is established through HTTP. We look at how HTTP raw messages look like and what HTTP headers are. The role of headers, the different types of headers like **request headers**, **representational headers**, **general headers**, and **security headers**.

We look at different types of HTTP methods — **GET**, **POST**, **PUT**, **DELETE** — when to use them and what the semantics and principles are behind them. We look at what **CORS** is and how it works — how a simple request differs from a pre-flight request and how a pre-flight request looks from the browser to the server and back.

Then we look at HTTP responses — the structure of it and different status codes that a server returns. When to return which type of code and what the most commonly used HTTP status codes are.

We also cover:

* **HTTP caching** — ETags, max-age headers, different caching techniques
* **HTTP/1.1 vs HTTP/2.0 vs HTTP/3.0** — differences between them
* **Content negotiation** between client and server
* **Persistent connections** in HTTP
* **HTTP compression** — gzip, deflate, Brotli
* **Security** — SSL/TLS and HTTPS

***

## 4. Routing

Next we look at **routing** — how routing maps URLs to server-side logic and the connection between routing and HTTP methods.

Different components of routes:

* **Path parameters** and **query parameters**
* **Static routes**, **dynamic routes**, **nested routes**
* **Hierarchical routes**, **catch-all/wildcard routes**
* **Regular expression based routes**

We cover **API versioning** using HTTP — different versioning techniques, best ways to deprecate an API, and industry best practices. We look at the benefits of **route grouping** and how it helps with versioning, permissions, and shared middleware. We see how to secure routes and optimize route matching performance.

***

## 5. Serialization and Deserialization

This means how a server, before sending data over the network, translates data into a particular format — and after receiving data from a client over the internet, how it translates that data to its own native format.

We look at the different formats used:

| Format Type    | Examples  | Tradeoff                   |
| -------------- | --------- | -------------------------- |
| **Text-based** | JSON, XML | Readable, slower           |
| **Binary**     | Protobuf  | Faster, not human-readable |

We explore **JSON** in depth — the structure, different data types (strings, numbers, booleans, arrays, objects), how serialization of nested objects and collections works, and how deserializing into native data structures works (Python dictionaries, Golang structs, JavaScript objects).

Common errors while dealing with JSON:

* Handling missing or extra fields
* Dealing with null values
* Date serialization and time zone issues
* Implementing custom serialization

We look at error handling in serialization/deserialization — invalid data, data conversion errors, unknown fields — and security concerns like injection attacks. We see why validation should happen **before** deserialization, and how to validate JSON schemas using **JSON Schema validation**.

The performance tradeoff: text-based formats are readable but slower; binary formats are faster but you can't easily inspect the payload.

***

## 6. Authentication and Authorization

Why we use it, different types of authentication — **stateful** and **stateless**. We cover:

* **Basic authentication**
* **Bearer token authentication**
* **Sessions**, **JWTs**, **cookies**
* Deep dive into **OAuth** protocol and **OpenID Connect**
* **API keys** and **multi-factor authentication**
* **Salting**, **hashing**, and different cryptographic techniques

For authorization we explore:

* **ABAC** (Attribute-Based Access Control)
* **RBAC** (Role-Based Access Control)
* **ReBAC** (Relationship-Based Access Control)

Security best practices:

* Securing cookies, avoiding **CSRF** and **XSS**
* **MITM** attack prevention
* **Audit logging** — recording authentication/authorization events
* Escalating error messages properly, preventing information leakage
* Handling edge cases — consistency across failure modes, rate limiting, account lockout
* Avoiding **timing attacks** — attackers exploiting time differences in error responses to infer valid credentials

***

## 7. Validation and Transformation

Different types of validation:

* **Syntactic** — checking if input is a valid email, phone number, or date format
* **Semantic** — a date of birth cannot be in the future; age should be between 1 and 120
* **Type validation** — checking if the input matches expected types (string, integer, array, object)

We look at the difference between **client-side validation** (improves UX with instant feedback) and **server-side validation** (the true security implementation — the gateway to business logic). Server-side validation matters even if client-side is already implemented.

The importance of **failing fast** — reducing unnecessary processing by returning early. Keeping consistency between front-end and back-end validation.

**Transformations**:

* Type casting (string to number, number to string)
* Date format conversion
* Normalization (lowercasing emails, trimming whitespace, adding country codes)
* Sanitization for security (preventing SQL injection)

Complex validation logic:

* **Relationship validation** — password and confirm password must match
* **Conditional validation** — partner field only required if married is true
* **Chain validation** — lowercase, remove special characters, then check length

Error handling in validation: meaningful error messages, aggregating all validation errors in one response, escalating messages (saying "invalid credentials" instead of "invalid password").

***

## 8. Middleware

What middleware is, when to use them, and common use cases. The role of middleware in a request cycle — **pre-request** middleware and **post-response** middleware.

The flow of middleware — chaining: a middleware is executed in sequence, passing control to the next middleware until the request reaches its final handler. How to order middleware appropriately:

1. Log the request
2. Check authentication
3. Do validation
4. Route handling
5. Error handling

How the `next()` function works in middleware and how middleware can exit the request pipeline early by handling 404 errors.

Common middlewares:

* **Security** — adding headers (X-Content-Type, HSTS, CSP)
* **CORS** — adding appropriate CORS headers
* **CSRF protection**
* **Rate limiting**
* **Authentication** — reusing route-protection logic
* **Logging and monitoring** — request logging, structured logging
* **Error handling** — catching and formatting application-level errors
* **Compression** — compressing response bodies
* **Data parsing** — handling JSON, URL-encoded forms, file uploads

Performance and scalability: keeping middleware lightweight, ensuring correct order, understanding how order affects performance and security.

***

## 9. Request Context

Request context is the metadata passed through middlewares, controllers, and services. It is a **request-scoped state** — valid only for that request.

Components of request context:

* **Request metadata** — HTTP method, URL, headers, query parameters, body
* **Session and user information** — injected during authentication middleware
* **Tracking information** — unique request IDs, trace IDs
* **Request-specific data** — caching data, permission checks

Use cases: authentication, rate limiting, tracing, logging. The connection between middlewares and request contexts. Different types of timeouts — request timeouts, custom timeouts, cancellation signals.

Best practices: keep it lightweight, ensure context data is cleaned up after the request lifecycle, avoid tightly coupling components through context.

***

## 10. Handlers, Controllers, and Services

The **MVC pattern** — what handlers, controllers, and services are, and the responsibilities of each. Reducing code with middleware. Centralized error handling in handlers. Consistent success and error message formats in controllers.

***

## 11. CRUD Deep Dive

How CRUD operations map to HTTP methods and the common APIs associated with each:

| Method          | Operation                     | Common Status Code           |
| --------------- | ----------------------------- | ---------------------------- |
| **POST**        | Create / Submit               | 201 Created, 400 Bad Request |
| **GET**         | Fetch list or single resource | 200 OK                       |
| **PUT / PATCH** | Update resource               | 200 OK                       |
| **DELETE**      | Delete resource               | 204 No Content               |

We look at how to implement **pagination**, **search APIs**, **sorting**, and **filtering**. Best practices: strict validation, consistent response formatting, limiting payload, redacting sensitive fields, error handling, authentication and authorization.

***

## 12. RESTful Architecture and Best Practices

What RESTful architecture is and best practices for implementing REST APIs — designing APIs around resources and sticking to HTTP semantics. Filtering and pagination best practices. Different types of versioning: **URI versioning**, **header versioning**, **query string**, **media type**.

Designing APIs with **OpenAPI spec** in mind. Content negotiation. Capturing exceptions and providing meaningful messages. Supporting client-side caching with ETags. Optimizing large requests and responses.

***

## 13. Databases

Relational vs non-relational — differences and when to use which. Theoretical concepts like **ACID** and **CAP theorem**. Basic querying and joins. Database design best practices: schema design, indexing.

Optimization methods: query optimization, caching, connection pooling. Data integrity: constraints, validations, transactions, concurrency. How **ORMs** work, whether to use one, and the tradeoffs. Database migrations.

***

## 14. Business Logic Layer (BLL)

The role of the business logic layer and the different layers of a request cycle:

```mermaid
flowchart TB
    PL["Presentation Layer\n(Routing, Middleware, Handlers, Controllers)"]
    BLL["Business Logic Layer\n(Services, Domain Models, Business Rules)"]
    DAL["Data Access Layer\n(Database Queries, Inserts, Deletes)"]

    PL --> BLL
    BLL --> DAL
```

Design principles: separation of concerns, single responsibility, open/closed, dependency inversion. Components of BLL: services, domain models (User, Order), business rules, business validation logic. Service layer design best practices. Error propagation from service layer to presentation layer.

***

## 15. Caching

What caching is and how it differs from database persistence. Different types: **memory caching**, **browser caching**, **database caching**. Client-side vs server-side caching.

Caching strategies:

* **Cache-aside**
* **Write-through**
* **Write-behind (write-back)**
* **Read-through**

Cache eviction strategies: **LRU**, **LFU**, **TTL**, **FIFO**. Cache invalidation: manual, TTL-based, event-based.

Levels of caching:

* **Level 1** — in-memory (fast, small)
* **Level 2** — network distributed (slower, larger)
* **Hierarchical** — combines L1 and L2; frequently used data in fast small cache, less frequently used in slower larger cache

Web app caching — static assets, API responses using headers. Database caching — storing results of heavy joins in Redis. Cache hit vs cache miss ratio and how to optimize them.

***

## 16. Transactional Emails

Common use cases, anatomy of a transactional email: subject, preheader, body, header, main content, CTA, footer. How to personalize with dynamic parameters.

***

## 17. Task Queuing and Scheduling

Common use cases for queuing: sending emails, processing image files, third-party API integration (payment processing, webhooks), offloading heavy computation (batch processing).

Scheduling use cases: database backups, recurring notifications and reminders, data synchronization, maintenance (clearing logs or caches).

Components of a task queue: **producer**, **queue**, **consumer**, **broker**, **backend**. Task dependencies — chain or parent-child relationships. Task groups for concurrent execution. Error handling and retries. Task prioritization and rate limiting.

***

## 18. Elasticsearch

Why we use Elasticsearch and how it works behind the scenes: inverted index, term frequency, inverse document frequency, segments, shards.

Use cases: typeahead experience, log analytics, social media search (full-text search for profiles, posts, comments). Creating and managing indexes. Search types: basic, full-text, relevance scoring.

Optimizing search: text vs keyword fields, analyzers, boosting, pagination. Advanced patterns: filtering, aggregation, fuzzy search. Kibana for a user-friendly interface.

Best practices: defining field mappings explicitly, optimizing shard count, indexing in batches, avoiding wildcards.

***

## 19. Error Handling

Different types of errors: syntax, runtime, logical. Error handling strategies: fail-safe, fail-fast, graceful degradation, prevention.

Practices: catching early, not swallowing errors, custom error types, failing gracefully, logging with stack traces. Global error handlers. User-facing errors: friendly messages, actionable feedback. Monitoring and logging tools: Sentry, ELK stack. Alert systems: email-based, Slack-based.

***

## 20. Config Management

How config management helps with flexibility and decouples environment-specific settings from application logic. Use cases: different environments, safely managing sensitive data (API keys, database passwords, private certificates), enabling/disabling features without changing codebase.

Types of configs:

* **Static** — DB credentials, API endpoints
* **Dynamic** — feature flags, rate limits
* **Sensitive** — credentials, tokens, secrets

Sources: env files, JSON, YAML. Differences between environment variables vs command-line flags vs static files.

***

## 21. Logging, Monitoring, and Observability

Differences between logging, tracing, monitoring, and observability. Types of logs: system, application, access, security. Log levels: debug, info, warn, error, fatal.

Structured vs unstructured logging. Best practices: centralized logging, log rotation and retention, contextual and meaningful logs, avoiding sensitive data.

Monitoring types: infrastructure, application performance, uptime. Tools: Prometheus, Grafana. Alert management: defining thresholds, creating alerts, avoiding alert fatigue.

Observability — three pillars: **logs**, **metrics**, **traces**. Security and compliance of log management.

***

## 22. Graceful Shutdown

Why we need it — server restarts, scaling in cloud environments, microservices, long-running jobs. Signal handling: SIGTERM, SIGINT, SILL signals.

Steps:

1. Capture signal
2. Stop accepting requests
3. Complete in-flight requests
4. Close external resources (database connections, open files)
5. Terminate the app

***

## 23. Security

Different aspects of backend security — avoiding attacks: SQL injection, NoSQL injection, XSS, CSRF, broken authentication, insecure deserialization.

Principles of secure software design: least privilege, defense in depth, fail-secure defaults, separation of duties, security by design. Input validation and sanitization, rate limits, CSP, CORS, same-site cookies, monitoring events.

***

## 24. Scaling and Performance

Performance metrics: response time, resource utilization, identifying bottlenecks. Caching and database optimization — avoiding N+1 query problems, proper use of joins, lazy loading.

Database indexes to speed up read operations on frequent fields (foreign keys, search fields). Processing data in batches. Avoiding memory leaks — closing file handles, database connections, cleaning up. Minimizing network overhead — reducing payload size, using compression.

Performance testing and profiling. Best practices: focus on clear and maintainable code first, write modular code, ensure graceful degradation, offload non-critical tasks to background processes.

***

## 25. Concurrency and Parallelism

The difference between concurrency and parallelism. How **concurrency** helps with IO-bound tasks. How **parallelism** helps with CPU-bound tasks.

***

## 26. Object Storage and Large Files

Common use cases for object storage (AWS S3). Managing large files with chunking and streaming. Multipart file uploads.

***

## 27. Real-Time Backend Systems

WebSockets, Server-Sent Events (SSE), and pub/sub architecture.

***

## 28. Testing and Code Quality

Types of testing: unit, integration, end-to-end, functional, regression, performance, load and stress, user acceptance, security. Test-driven development (TDD). Automating tests in CI/CD environments. Code quality with linting and formatting tools.

Quality metrics: **cyclomatic complexity** (counting possible paths through code), **maintainability index** (quantifying how easy it is to maintain based on complexity, lines of code, and other factors).

***

## 29. 12-Factor App

A set of principles for building modern, scalable, maintainable software-as-a-service apps.

***

## 30. OpenAPI Standards

The need for standards, benefits, use cases (documentation automation). The ecosystem: Swagger, Corden, Postman. History: Swagger to OpenAPI transition. Key concepts: API paths, request/response definitions, parameters, schemas.

Document structure: metadata, paths, components, security definitions, responses. New features in OpenAPI 3.0 and 3.1. Tools: Swagger UI, Corden, Postman. **API-first development** — define your OpenAPI spec first, then build the APIs.

***

## 31. Webhooks

Use cases: notifications, third-party integrations. API vs webhook — polling (client-initiated) vs pushing (server-initiated).

Key components: webhook URL, event triggers, payload, HTTP method, response handling. Best practices: signature verification (HMAC), quick response, retry logic, logging. Testing with ngrok. Real-world use cases: Stripe, GitHub, Slack, Discord, Twilio.

***

## 32. DevOps for Backend Engineers

Core concepts: continuous integration, continuous delivery, continuous deployment. Practices: infrastructure as code, config management, version control.

Tools: Docker (containers), Kubernetes (orchestration), CI/CD pipelines. Scaling: horizontal vs vertical. Deployment strategies: blue-green, rolling deployment.

***

## Resource

* [Roadmap for backend from first principles — Sriniously](https://www.youtube.com/watch?v=0Rwb4Xmlcwc)