> For the complete documentation index, see [llms.txt](https://naz.gitbook.io/my-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://naz.gitbook.io/my-docs/backend-learning/understanding_http_for_backend_engineers.md).
# Lesson 1: Understanding HTTP for Backend Engineers
## Backend Scope and HTTP Intro
Backend is huge and if we start discussing every single component that could be part of it we will be stuck here for years. So what we will do is we will only discuss those topics which are used in majority of the code bases, let's say 90% of them.
With that in mind let's talk about HTTP protocol, the medium through which our browsers talk to our servers either to send data or to receive data from it.
There are a lot of other ways and protocols which clients and servers use to communicate with each other and HTTP being one of the most used ones we will focus on that.
***
## Statelessness
There are two ideas which are at the heart of HTTP protocol. The first one being **statelessness**.
**Statelessness** basically means it has no memory of past interactions. Each HTTP request carries all the necessary information for the server to process it such as headers or URLs and methods.
After the server responds it forgets about the request. If a client makes another request the server treats it as a new and unrelated event.
This also means **self-contained requests**. Since the server does not remember past requests each request must include all the necessary data such as authentication tokens or session information to handle that specific interaction.
* For example in the case of accessing a user profile the client has to provide credentials like cookies or tokens on every request for the server to know which user is requesting the data.
### Benefits of Statelessness
* **Simplicity** -- Stateless design simplifies server architecture because the server does not need to store session information which would otherwise require additional resources and complexity.
* **Scalability** -- Stateless protocol makes it easy to distribute requests across multiple servers because no single server needs to keep track of a session.
* **Fault tolerance** -- If the server crashes it does not affect the state of a client interaction as there is no session or memory of a request that needs to be restored.
Because HTTP is stateless developers often implement state management techniques like cookies or sessions or tokens to maintain continuity in interactions where needed like user logins or shopping carts.
***
## Client Server Model
The second idea is **client server model**. In a typical HTTP request flow there is a client and there is a server always.
* **Client** -- typically a web browser or an application which initiates the communication by sending a request to the server. The client is responsible for providing all information needed by the server such as the URL of the resource or the headers.
* **Server** -- hosts resources like websites or APIs or other content and waits for incoming requests from the clients. When the server receives a request it processes it and sends back the appropriate response such as a web page or data or error message or a JSON file or a text file or any other kind of content.
HTTP protocol states that communication is always initiated by the client to get some kind of response by the server.
```mermaid
sequenceDiagram
participant Client as "Client
(browser or app)"
participant Server as "Server
(website or API)"
Client->>Server: "Request with URL and headers"
Note over Server: "Process request
stateless, no memory of past"
Server-->>Client: "Response with page or data or error"
Note over Client,Server: "Next request treated as new event"
```
***
## HTTP and HTTPS Are Interchangeable Here
Throughout our discussion we can safely assume that HTTP and HTTPS are interchangeable because HTTPS to oversimplify is just a more secure version of HTTP.
The underlying principles are the same with more security features like encryption and security certificates or TLS stuff like that which borders too much into the network engineering domain.
***
## TCP and the Application Layer
To send some kind of request or receive some kind of response first the client and the server need to establish some kind of connection mechanism.
For that HTTP uses **TCP** which is a transmission protocol. HTTP does not require the underlying transport protocol to be connection based, it only requires it to be reliable and not lose messages, at minimum presenting an error in such cases.
Among the two most common transport protocols on the internet which are TCP and UDP, TCP is considered to be more reliable. HTTP therefore relies on the TCP standard which is connection based.
This is called an **OSI model** which is often referred when we are talking about sending and receiving data over a network.
We as backend engineers often deal with the top layer, **layer 7, the application layer**. A lot of discussions for example the TCP handshake and establishing connections and TLS encryptions, they often border into these layers, they are mostly network engineering concepts.
If we start exploring those it will be a rabbit hole. So what we will do is we will focus on the application layer with a brief about what happens in the network layer.
HTTP uses TCP and TCP uses something like a **3-way handshake**.
```mermaid
flowchart TB
APP["Layer 7
Application Layer
HTTP, backend focus"]
TRANS["Layer 4
Transport Layer
TCP reliable, 3-way handshake"]
NET["Layer 3 and below
Network layers
packet delivery"]
APP --> TRANS --> NET
```
***
## Evolution of HTTP
Throughout the years we had different versions of HTTP which kept redefining how clients and servers send and receive data.
* **HTTP 1.0** -- each request opened a new connection. This led to inefficiencies since a connection had to be established and closed for every request and response which slowed performance.
* **HTTP 1.1** -- introduced persistent connections allowing multiple requests and responses over the same TCP connection that was established before sending the request which significantly improved performance. It also added chunk transfer encoding and better caching mechanisms.
* **HTTP 2.0** -- introduced multiplexing allowing multiple requests or responses over a single connection. It uses binary framing instead of text and supports header compression with HPACK and server push allowing servers to send resources before the client requests them.
* **HTTP 3.0** -- built on QUIC protocol, a transport layer protocol over UDP instead of TCP. It is designed over UDP which improved performance with faster connection establishment, reduced latency and better handling of packet loss. It also continues to support multiplexing without head of line blocking which is still an issue in HTTP 2.0.
All we need to remember from all these discussions is that client and servers establish some kind of network connection and messages are sent and received.
| Version | Connection model | Key addition |
| -------- | ----------------------------------- | ----------------------------------------------------- |
| HTTP 1.0 | New connection per request | Headers, status codes, content types |
| HTTP 1.1 | Persistent connections | Chunk transfer encoding, better caching |
| HTTP 2.0 | Multiplexing over single connection | Binary framing, HPACK header compression, server push |
| HTTP 3.0 | QUIC over UDP | Faster establishment, no head of line blocking |
```mermaid
flowchart LR
H10["HTTP 1.0
one connection per request"] --> H11["HTTP 1.1
persistent connection"]
H11 --> H2["HTTP 2.0
multiplexing plus binary framing"]
H2 --> H3["HTTP 3.0
QUIC over UDP"]
```
***
## HTTP Messages
Now that we mentioned messages let's look at what HTTP messages look like, request messages or response messages.
A **request message** is the one that is sent by the client and a **response message** is the one that is received by the client from the server.
Below is a full example of a request and response with many parameters so that we can cover different components one by one.
**Request message:**
```
PUT /api/users/12345 HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36
Content-Type: application/json
Content-Length: 123
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
Accept: application/json
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Referer: https://example.com/dashboard
Cookie: sessionId=abc123xyz456; lang=en-US
{
"firstName": "John",
"lastName": "Doe",
"email": "john.doe@example.com",
"age": 30
}
```
**Response message:**
```
HTTP/1.1 200 OK
Date: Fri, 20 Sep 2024 12:00:00 GMT
Content-Type: application/json
Content-Length: 85
Server: Apache/2.4.41 (Ubuntu)
Cache-Control: no-store
X-Request-ID: abcdef123456
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Set-Cookie: sessionId=abc123xyz456; Path=/; Secure; HttpOnly
Vary: Accept-Encoding
Connection: keep-alive
{
"message": "User updated successfully",
"userId": 12345,
"status": "success"
}
```
### Breaking Down the Request
* `PUT` is the **request method**.
* `/api/users/12345` is the **resource URL** we are requesting from the server.
* `HTTP/1.1` is the **HTTP version**, currently the most used one.
* `Host: example.com` is the **host**, the frontend domain.
* All the lines after the method line until the blank line are called **headers**.
There is a blank line after all the headers to signify that the headers are over and the body starts. The JSON object below the blank line is the **request body**, the information the client wants to send to the server.
### Breaking Down the Response
* `HTTP/1.1` is the **HTTP version**.
* `200` is the **status code** and `OK` is the value of the status code, meaning the request was successful.
* The lines after that are **response headers**.
* After another blank line comes the **response body**.
```mermaid
sequenceDiagram
participant Client as "Client
(example.com dashboard)"
participant Server as "Server
(Apache 2.4.41 Ubuntu)"
Client->>Server: "PUT /api/users/12345
headers plus JSON body"
Note over Server: "Status 200 OK
build response headers plus body"
Server-->>Client: "200 OK
message User updated successfully"
```
***
## Why Do We Need HTTP Headers
Headers are basically key value pairs of different parameters that are sent over a request or received over a response.
The next question is why do we need headers, why not send all these values in the URL or in the request body, why create a different section just for sending more metadata about the request or the response.
### The Parcel Analogy
Take a real life example. We send parcels and we receive them. The address, the phone number of the recipient, the state, the PIN code, do we keep it inside the package or do we write it on top.
We write it on top because the one who is taking the parcel from the sender to the receiver needs to know different information in order to successfully transmit the package through different modes of transmission.
If we kept all that information inside the parcel they have to open it again and again. Keeping the metadata on top of the parcel gives a quick way of checking different metadata about the package.
HTTP headers can be thought of something like that.
```mermaid
flowchart LR
subgraph Parcel["Parcel analogy"]
TOP["Label on top
address plus phone plus PIN"]
INSIDE["Contents inside
actual goods"]
TOP --> INSIDE
end
subgraph HTTP["HTTP message"]
HEAD["Headers
quick metadata check"]
BODY["Body
actual data"]
HEAD --> BODY
end
Parcel -->|"same idea"| HTTP
```
***
## Types of HTTP Headers
* **Request headers** -- sent by the client to the server to provide information about the request itself. User-Agent identifies what kind of client it is, a browser or Postman or a server or a mobile app. Authorization sends credentials like bearer token to identify the user. Accept headers provide information like what kind of content we are expecting whether it is JSON or text or HTML. Request headers help server understand the client's environment, its preferences and its capabilities.
* **General headers** -- used in both requests and responses which have metadata about the message itself. For example the date of the message and caching mechanisms like no-cache or max-age and the connection information whether to keep it alive or to close it.
* **Representation headers** -- deal with the representation of the resource being transmitted whether it is a request body or a response message. Content-Type describes the media type, it could be JSON or HTML. Content-Length describes the size of the resource in bytes. Content-Encoding specifies encoding like gzip or deflate. ETag is a unique identifier mostly used for caching. Representation headers ensure the clients and servers know how to interpret and process the request or the response.
* **Security headers** -- enhance the security of the request and response by controlling behaviors like content loading, cookies and encryption.
* **HSTS** ensures that the client only communicates with the server over HTTPS preventing protocol downgrade attacks.
* **Content-Security-Policy** restricts the sources from which content like JavaScript or CSS and images can be loaded helping prevent cross site scripting attacks.
* **X-Frame-Options** prevents the web page from being embedded in iframe mitigating clickjacking attacks.
* **X-Content-Type-Options** ensures that the browser does not try to guess the MIME type preventing MIME sniffing attacks.
* **Cookies with HttpOnly or Secure flags** secure cookies by making them inaccessible with JavaScript and ensuring they are sent over HTTPS.
Security headers protect the client and server by controlling how the browser behaves with resources and enforcing security policies.
***
## Extensibility and Remote Control
There are two ideas here while talking about HTTP headers.
### Extensibility
HTTP is highly extensible because headers can be easily added or customized without altering the underlying protocol. We only have to add metadata and the whole flow of the interaction changes.
* Headers can be defined for various purposes making HTTP adaptable to new technologies.
* Security enhancements like Strict-Transport-Security force secure connections.
* Developers can create custom headers like X-Custom-Header for their own use cases.
* Content negotiation with Accept and Accept-Language and Accept-Encoding allows servers to serve different versions of the content depending on the client's preference.
### Remote Control
HTTP headers act as a remote control on the server side. They allow the client to send instructions or preferences to the server influencing how the server responds or processes requests.
* **Content negotiation** -- clients can request specific formats using Accept header and the server responds with the appropriate format. If the client says it wants HTML the server sends HTML and if the client says it wants JSON the server sends JSON.
* **Caching and expiration control** -- the server can use headers like Cache-Control or Expires to control how long a resource should be cached by the client.
* **Authentication** -- the client can authenticate itself through Authorization header influencing access control decisions.
***
## HTTP Methods
HTTP methods exist to represent different kinds of actions that a client like a browser or an API consumer can request on a server.
Instead of every request doing the same thing methods define the **intent**. The keyword here is the intent of the interaction. This gives clear semantic meaning to each type of action.
* **GET** -- we use this to fetch data from the server and it should not modify anything on the server.
* **POST** -- we use this to create data in the server and POST requests have a body because that is how you send the user data to the server.
* **PATCH** -- is used to update data, for example a user profile page where they update their name, and this also has a request body.
* **PUT** -- is also used to update data but what sets it apart from PATCH is whatever data comes in the request body should completely replace the previous instance. PATCH can be thought of as an append action or a selective replacement while PUT is a complete replacement. Even though a lot of times developers use PUT when they should be using PATCH and go against the semantics.
* **DELETE** -- as the name suggests we use this to delete some kind of resource from the server.
The thumb rule is always use PATCH unless you have a specific use case for PUT.
| Method | Intent | Body |
| ------- | -------------------------- | ---- |
| GET | Fetch data, modify nothing | No |
| POST | Create data | Yes |
| PATCH | Selective update or append | Yes |
| PUT | Complete replacement | Yes |
| DELETE | Remove resource | No |
| OPTIONS | Fetch server capabilities | No |
***
## Idempotent vs Non-Idempotent
**Idempotent** means these HTTP methods can be called multiple times and we can expect the same kind of result.
We generally consider GET or PUT or DELETE in the category of idempotent methods.
* **GET** -- Fetching data does not matter how many times you fetch it, the data should be same, you should not modify anything in the server, so GET is obviously idempotent.
* **PUT** -- Completely replaces the resource in the server so it does not matter how many times you replace the old with the new data, the result will be the same, so that is also idempotent.
* **DELETE** -- You can only delete a resource once because after that it is already deleted, you cannot perform the action multiple times and expect different results, the result will always be the same.
**Non-idempotent** means the method produces different results for the same kind of request.
* **POST** -- Usually we consider POST to be non-idempotent because it produces different results for the same kind of request. Once you submit a request to create data, let's say a user creates a note with your app, the first time they submit the request a new note is created and when they do it a second time another new note is created. There are two different results for the same kind of request.
```mermaid
flowchart TB
subgraph IDEM["Idempotent
same result on repeat"]
GET["GET fetch"]
PUT["PUT full replace"]
DEL["DELETE remove once"]
end
subgraph NON["Non-idempotent
new result on repeat"]
POST["POST create
note 1 then note 2"]
PATCH["PATCH partial update"]
end
```
***
## OPTIONS Method and CORS Workflow
We have one other method which is called the **OPTIONS** method and this has a very interesting use case which is used in the **CORS** flow.
You probably won't use it directly as a developer but you will see it in your browser network tab in pre-flight requests.
The OPTIONS method is used to fetch the capabilities of the server for a cross origin request, also known as CORS.
### Same Origin Policy
Browsers follow the **same origin policy** which restricts web pages from making requests to a domain different from the one serving the web page.
**CORS** is a security mechanism enforced by browsers to control how web applications interact with resources hosted on different domains.
Without CORS browsers block requests made from a web application running on one origin like example.com to a different origin like api.anotherexample.com for security reasons.
CORS allows servers to specify who can access their resources and how.
In a cross origin request there are primarily two types of flows, one is a **simple request flow** and another one is a **pre-flighted request flow**.
### Simple Request Flow
Imagine our client frontend is at the domain example.com and our server is at api.example.com and we make a GET request.
The client sends a request, the browser automatically adds the Origin header to indicate the origin of the request. The request uses a simple method which are usually GET, POST or HEAD.
```
GET /api/products/123 HTTP/1.1
Host: api.anotherdomain.com
Origin: https://example.com
Accept: application/json
```
The server checks the Origin header against its CORS policy. If the origin is allowed the server includes the Access-Control-Allow-Origin header in the response.
```
HTTP/1.1 200 OK
Content-Type: application/json
Access-Control-Allow-Origin: https://example.com
{
"product": {
"id": 123,
"name": "Example Product",
"price": 29.99
}
}
```
The browser sees that the domain example.com is different from the host domain api.anotherdomain.com and since these two are different the browser checks whether the server's response has the Access-Control-Allow-Origin header.
If the server responds with the domain of the client example.com or with star which means allow all origins, then the browser lets the response go through to the JavaScript client requesting the resource.
If the server does not add the corresponding CORS headers or does not allow this particular domain, it excludes the Access-Control-Allow-Origin header from the response:
```
HTTP/1.1 200 OK
Content-Type: application/json
{
"product": {
"id": 123,
"name": "Example Product",
"price": 29.99
}
}
```
When the browser parses it, it sees the absence of this header and blocks the response from being passed. You will get an error in the console and in the network tab as a CORS error.
```mermaid
sequenceDiagram
participant Page as "Page
example.com"
participant Browser as "Browser"
participant API as "API
api.anotherdomain.com"
Page->>Browser: "GET /api/products/123"
Browser->>API: "GET plus Origin example.com"
API-->>Browser: "200 plus Access-Control-Allow-Origin"
Note over Browser: "Origin allowed, pass to page"
Browser-->>Page: "Product 123 Example Product 29.99"
```
### Pre-Flighted Request Flow
These are the three conditions the browser checks before it decides it has to do a **pre-flight request**, which means a request before the original request to inquire about capabilities.
It is an either-or situation, either of these conditions have to be true, plus it has to be a cross origin request where our domain and the server domain are different.
1. The method is not GET, POST or HEAD, for example it is a PUT or DELETE request.
2. The request includes non-simple headers, basically anything apart from general or simple request headers, for example an Authorization header or custom headers.
3. The request has a content type other than application/x-www-form-urlencoded or multipart/form-data or text/plain, these are called simple content types.
If our client requests the data to be in JSON it will be considered a pre-flighted request. Since mostly we deal with JSON data most of our requests are pre-flighted requests.
A pre-flight request is made with an OPTIONS method. It has the method as OPTIONS, the resource URL, the host of the API, the origin of the frontend, and Access-Control-Request-Method asking whether this method is supported for this URL, saying I am making a cross origin request and do you support this PUT method for this route.
It also asks whether you support a particular header. The browser sends the OPTIONS request to the server, it does not include actual data, no request body, it is just a general inquiry about its capabilities.
```
OPTIONS /api/users/12345 HTTP/1.1
Host: api.anotherdomain.com
Origin: https://example.com
Access-Control-Request-Method: PUT
Access-Control-Request-Headers: Authorization, Content-Type
```
If the server properly handles CORS it responds with the following. If the server does not handle CORS then it won't respond with this and the request will be automatically blocked.
```
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: https://example.com
Access-Control-Allow-Methods: PUT, DELETE
Access-Control-Allow-Headers: Authorization, Content-Type
Access-Control-Max-Age: 86400
```
The status code 204 means no content, just general information.
### The Four Important Headers
* **Access-Control-Allow-Origin** -- says yes I allow the client's domain example.com, or it can respond with star meaning all clients.
* **Access-Control-Allow-Methods** -- says yes I allow PUT and DELETE for this route.
* **Access-Control-Allow-Headers** -- says yes I allow Authorization header.
* **Access-Control-Max-Age** -- says don't make more pre-flight requests, these configs will be the same for at least the next 24 hours, so you don't have to keep making pre-flight requests before every route. This saves bandwidth for both servers and clients.
The browser checks off all these conditions and then sends the final original request that the client wanted to make and the server responds with whatever operations are required.
```mermaid
sequenceDiagram
participant Browser as "Browser
example.com"
participant Server as "Server
api.anotherdomain.com"
Browser->>Server: "OPTIONS plus
Request-Method PUT"
Note over Server: "Check origin and method and headers"
Server-->>Browser: "204 No Content
Allow-Origin plus Allow-Methods
plus Allow-Headers plus Max-Age"
Note over Browser: "All checked off, send original"
Browser->>Server: "PUT plus JSON body"
Server-->>Browser: "200 OK plus result"
```
***
## Response Status Codes
HTTP response codes exist to communicate the result of a request in a standardized way. You can just look at the response code and see whether the request was successful or what is the state of the server without judging from the response body.
We don't have to make those decisions like if successful I expected this structure, if unsuccessful this structure, if the server crashed a null object.
### Why Status Codes Matter
* They quickly inform the client whether the request was successful, resulted in an error or requires further action.
* They also help clients handle errors by providing specific codes, for example unauthorized access returns 401 so the client can log the user out saying you have to log in again. Or a 400 bad request from an invalid form value, the client can ask the user to fix the form and resubmit.
* Standardization: HTTP response codes are standardized across all web services enabling consistency regardless of platform or language. It does not matter whether you make a server in Python or Golang or Rust or JavaScript or Ruby, if the request was successful you return 200, if you created something you return 201.
* Before status codes clients would have to guess the outcome based on the content of the response leading to inconsistencies. Status codes provide a universal language that all clients and servers understand.
On a high level response codes are three-digit numbers starting with 1, 2, 3, 4 or 5. Starting with 1 are informational, 2 are success, 3 are redirection, 4 are client errors, 5 are server errors.
### 1xx Informational
* **100 Continue** -- sent by the server to indicate that it has received the headers and the client can proceed to send the request body, commonly used in large uploads.
* **101 Switching Protocols** -- indicates the server is switching protocols as requested by the client such as upgrading from HTTP to WebSocket.
These are not the mostly used codes in day-to-day life, so let's focus on 2, 3, 4 and 5.
### 2xx Success
* **200 OK** -- the most common code, the request was successful and the server is returning the requested resource, for example a successful GET.
* **201 Created** -- indicates the request has been fulfilled and resulted in creation of a new resource, for example POST for new form submissions.
* **204 No Content** -- indicates the request was successful but there is no content, only headers. We use it for DELETE, the client deletes the resource and the server says it is deleted but there is nothing to return.
### 3xx Redirection
* **301 Moved Permanently** -- means the resource has been permanently moved to a new URL and future requests should use it. For example initially you had a route called /user and you moved it to /person, to maintain backwards compatibility so old applications don't break you add a 301 for this route and redirect to /person.
* **302 Temporary Redirect** -- means the resource is temporarily at a different URL but the client should continue to use the original URL, for example running a campaign for a couple of hours to show a different UI but later you want to revert.
* **304 Not Modified** -- indicates the resource has not been modified since the last time the client requested it, mostly used with conditional GET to allow efficient caching with ETags, telling the client to use the cached response instead of downloading a new one.
### 4xx Client Error
* **400 Bad Request** -- fires when the client sends invalid or illogical data, for example you expect a number and the client sends a string, you expect an email and the client sends a phone number. You are letting the client know there is something wrong with the request format, fix it and make a new request.
* **401 Unauthorized** -- fires when a request requires authentication but the client failed to provide valid credentials or is not authenticated at all, for example you expect a JWT token and the token has expired or the client did not send it.
* **403 Forbidden** -- means the server understood the request but refuses to authorize, even if the client is authenticated, for example user A trying to delete a resource of user B, the server says you don't have the necessary permissions.
* **404 Not Found** -- the most famous code, fires when the client requests a resource that is unavailable either because the URL is incorrect or the resource has been deleted.
* **405 Method Not Allowed** -- fires when an invalid HTTP method is used such as trying to PUT to a resource that only accepts GET or POST, often because of typos like doing PATCH instead of PUT.
* **409 Conflict** -- has a number of use cases, for example in your app users create folders and folder names must be unique, when they submit a POST you check whether the folder already exists and respond with 409 so the client knows to try a new name.
* **429 Too Many Requests** -- is used for rate limiting, if the client makes too many requests in an interval, for example you allow at most 60 requests in 1 second and the client exceeds that.
### 5xx Server Error
* **500 Internal Server Error** -- is for unexpected conditions, some process broke or unhandled exceptions, something unexpected happened at the server, so instead of returning empty or hanging we respond with 500.
* **501 Not Implemented** -- is when the server does not support the requested HTTP method or functionality but plans to add it soon.
* **502 Bad Gateway** -- we usually see in proxies like Nginx, when a server acts as a proxy in a load balancer or reverse proxy and the upstream server returns an invalid response, this is handled mostly by proxies and load balancers.
* **503 Service Unavailable** -- is when the service is temporarily unable to handle the request such as during high traffic or maintenance, letting the client know it should try again later.
* **504 Gateway Timeout** -- is similar to 502 but means the upstream server failed to respond within the timeout period, for example Nginx could not get a response from our original server running behind it.
| Range | Meaning | Common codes |
| ----- | ------------- | ---------------------------------------------- |
| 1xx | Informational | 100 Continue, 101 Switching Protocols |
| 2xx | Success | 200 OK, 201 Created, 204 No Content |
| 3xx | Redirection | 301 Permanent, 302 Temporary, 304 Not Modified |
| 4xx | Client error | 400, 401, 403, 404, 405, 409, 429 |
| 5xx | Server error | 500, 501, 502, 503, 504 |
***
## HTTP Caching
HTTP caching is a technique to store copies of responses for reuse reducing the need to repeatedly request the server.
This improves load time, reduces bandwidth and decreases server load because the client does not need to download a lot of data and the server does not need to send a lot of data if the data is not changed.
The client can just reuse the old data if the data is not changed.
### How Caching Works
When a client first requests a resource it sends a GET request. The server responds with the resource along with caching headers.
* **Cache-Control** -- tells the client how long to maintain the cache for this resource, for example max 10 seconds.
* **ETag** -- basically a hash. ETags are usually hashes computed from a response, the server takes the response, hashes it and sends that hash as ETag.
* **Last-Modified** -- the last time this resource was modified, judging from this we decide whether to use the cached resource or request a new one.
```
HTTP/1.1 200 OK
X-Powered-By: Express
Access-Control-Allow-Origin: *
Cache-Control: max-age=10, public
ETag: 3141
Last-Modified: Fri, 27 Sep 2024 13:48:33 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 112
{
"content": "This is the updated content ...",
"lastModified": "Fri, 27 Sep 2024 ...",
"etag": "3141"
}
```
### Conditional Requests
When the client requests the same resource again it sends the headers If-None-Match and If-Modified-Since. What the client is saying to the server is if the ETag of the requested resource is not the same as this one which we have in the browser, or if the resource has been modified after this, meaning we have the outdated version, then send us the updated resource, otherwise I will just use my cached version.
```
GET /api/resource HTTP/1.1
Host: localhost:3001
Cache-Control: max-age=0
If-None-Match: 3141
If-Modified-Since: Fri, 27 Sep 2024 13:48:33 GMT
Origin: http://[::1]:5173
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
```
The server responds with **304 Not Modified**. It checked the If-None-Match which is the ETag and the Last-Modified and since the resource matched it sends 304 which means the resource has not been modified since the last time you fetched it so you can use your cached version.
### When Resources Change
If the resource is updated, the server responds with 200 OK and sends a new ETag. When the client requests again with the old ETag, the server responds with 200 instead of 304 because the resource has been modified, along with the updated ETag.
When the client fetches again with the new ETag and Last-Modified, the server checks it and since it is the latest version it again responds with 304 Not Modified.
### Production Considerations
In a production setting it gets complicated because the server has to manually implement and manage all these ETags and if by mistake it forgets to update an ETag then the client will continue to use the cached outdated resource which is not a good idea.
Nowadays we have better solutions like React Query which is complete client side caching, the client has the complete power over when to use a cached resource and when to refetch, at what interval, which in my opinion is a much better solution compared to traditional HTTP based caching.
But it is good to know that we have this option if our use case is simple enough.
```mermaid
sequenceDiagram
participant Browser as "Browser
cache empty"
participant Server as "Server
Express"
Browser->>Server: "GET /api/resource"
Server-->>Browser: "200 plus ETag 3141
plus max-age 10 public"
Note over Browser: "Store ETag 3141"
Browser->>Server: "GET plus If-None-Match 3141"
Server-->>Browser: "304 Not Modified
use cached copy"
Browser->>Server: "POST update resource"
Server-->>Browser: "200 plus new ETag 2943"
Browser->>Server: "GET plus If-None-Match 3141 old"
Server-->>Browser: "200 plus ETag 2943
resource changed"
Browser->>Server: "GET plus If-None-Match 2943"
Server-->>Browser: "304 Not Modified"
```
***
## HTTP Content Negotiation
Another important topic in the client server model is **content negotiation**. We already looked at headers like Accept and Content-Type application/json.
This is a mechanism using which client and server agree on the best format to exchange data. The client can indicate its preferred format like JSON or XML or HTML and the server will try to respond with a compatible format or a fallback format.
### Three Types of Negotiation
* **Media type** -- the client specifies the desired format through the Accept header which is application/json or XML.
* **Language negotiation** -- the client requests content in a specific language using Accept-Language, it could be English or Spanish.
* **Encoding negotiation** -- the client specifies which encoding it supports using Accept-Encoding like gzip or deflate and the server responds with that compression format.
For example when the client sends Accept-Language English, Accept application/json and Accept-Encoding gzip, deflate, br, the server responds with JSON in English.
When the client changes Accept-Language to Spanish, the server updates the response to Spanish since the client prefers Spanish.
When the client changes the format from JSON to XML with Accept-Language Spanish, the server responds with XML in Spanish.
The client lets the server know its preferences whether data format or language and the server sends according to that format, making the life of the client easier.
```mermaid
sequenceDiagram
participant Client as "Client"
participant Server as "Server"
Client->>Server: "GET plus Accept json
plus Language en"
Server-->>Client: "JSON in English"
Client->>Server: "GET plus Accept json
plus Language es"
Server-->>Client: "JSON in Spanish"
Client->>Server: "GET plus Accept xml
plus Language es"
Server-->>Client: "XML in Spanish"
```
***
## HTTP Compression
There is one interesting topic under the same umbrella which is **HTTP based compression**, either gzip, deflate or other formats.
When a server sends a large response, for example a JSON file with 11,000 entries, the response size can be around 26 MB uncompressed. With gzip compression enabled on the server side the size drops to around 3.8 MB. The response includes Content-Encoding gzip because the client says it accepts gzip as one of the encoding formats.
That is a huge difference and imagine every client having to download the uncompressed file, it costs a lot of bandwidth.
That is why we need compression, if the response size is very large we compress it and on the client side the browser decompresses it and gets the same response.
```mermaid
flowchart LR
BIG["Large file
11000 entries"] -->|"gzip on"| SMALL["3.8 MB
Content-Encoding gzip"]
BIG -->|"gzip off"| HUGE["26 MB
uncompressed"]
SMALL -->|"browser decompresses"| SAME["Same data"]
HUGE --> SAME
```
***
## Persistent Connections and Keep-Alive
This is another important topic which we won't really work with, it's good to know that it exists behind the scenes.
In the early days of HTTP specifically HTTP 1.0 each request response cycle required a separate connection to the server.
This created inefficiencies since establishing and closing TCP connections is resource intensive and slow.
To address this **persistent connections** were introduced in HTTP 1.1. With persistent connections a single TCP connection can be reused for multiple requests and responses avoiding the overhead of opening and closing a connection for every interaction.
For achieving that they introduced the header called **keep-alive**. Keep-alive is the mechanism that enables persistent connections, it allows the client and server to reuse the same connection for multiple request responses until one of them decides to close it.
### How It Works in Practice
* In HTTP 1.1 connections are persistent by default, we won't have to do anything explicitly, they remain open for further requests unless explicitly closed, and multiple requests and responses can be sent over a single connection.
* This reduces latency and saves resources as fewer connections need to be established.
* The Connection: keep-alive header is still sometimes used to explicitly ask the server to keep the connection open.
* This header can also include options like how long the connection should remain open with a timeout or how many requests can be sent before the connection is closed with the max value, for example Keep-Alive: timeout=5.
* When Connection is set to close the connection is closed after the response is sent. This is the behavior in HTTP 1.0 by default and it can still be explicitly enforced in HTTP 1.1.
You usually won't be working with them, the default values work fine.
```mermaid
sequenceDiagram
participant Client as "Client"
participant Server as "Server"
Note over Client,Server: "HTTP 1.0, one TCP per cycle"
Client->>Server: "Open TCP, GET, close TCP"
Client->>Server: "Open TCP, GET, close TCP"
Note over Client,Server: "HTTP 1.1 persistent by default"
Client->>Server: "Open TCP once"
Client->>Server: "GET 1 over same TCP"
Server-->>Client: "Response 1, keep-alive timeout 5"
Client->>Server: "GET 2 over same TCP"
Server-->>Client: "Response 2"
```
***
## Multipart Data and Chunked Transfer
One last topic is handling large requests and responses, how the server takes in large requests like files, video files, image files, audio files, any kind of file which is very large compared to our typical JSON and how the client can receive large responses.
### Multipart Uploads
Multipart is usually used for sending large files from the client to the server.
The difference from our typical JSON body is in multipart the binary data of the file is transferred to the server in parts, that's why it is called multipart.
A multipart request is a POST with Content-Type multipart/form-data and a boundary parameter. Since the binary data is transferred in parts the boundary parameter specifies what separates the parts, it acts as a delimiter.
The delimiter appears at the start of the binary data and at the end when all the binary data ends in the request body.
The server reads the file and responds with some details of the file to say the upload was successful. Whenever we want to transfer large files to the server we should use multipart requests.
```
POST /api/upload HTTP/1.1
Host: example.com
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Length: 34212
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="picture"; filename="photo.png"
Content-Type: image/png
------WebKitFormBoundary7MA4YWxkTrZu0gW--
```
### Chunked Transfer for Large Responses
For receiving large responses from the server the data is streamed to the client in chunks.
The request is a normal GET. In the response the three important things are:
* **Content-Type** says text/event-stream which streams the data to the client through different events.
* **Connection: keep-alive** says keep the connection alive until all the data is sent.
* **Transfer-Encoding: chunked** indicates the data is being sent in chunks.
The server keeps sending data in chunks and the client keeps appending all the data it receives and constructs the whole file. That is how we transfer a large file from a server to the client using chunk transfer or text event stream.
```
HTTP/1.1 200 OK
Content-Type: text/event-stream
Connection: keep-alive
Transfer-Encoding: chunked
```
```mermaid
sequenceDiagram
participant Client as "Client"
participant Server as "Server"
Note over Client,Server: "Upload with multipart"
Client->>Server: "POST multipart form-data
boundary delimiter plus binary parts"
Server-->>Client: "200 plus file details"
Note over Client,Server: "Download with chunked stream"
Client->>Server: "GET large text file"
Server-->>Client: "chunk 1 text event-stream"
Server-->>Client: "chunk 2 keep-alive"
Server-->>Client: "chunk N file complete"
```
***
## SSL, TLS and HTTPS
The last thing before we end is a brief idea about SSL or TLS or HTTPS even though we don't explicitly work with them.
* **SSL** was the original protocol for securing communications between client like a web browser and server. It encrypts data so that sensitive information like passwords or credit card numbers cannot be intercepted by attackers. Currently SSL is outdated due to security vulnerabilities and has been replaced by TLS.
* **TLS** is the modern and more secure version of SSL. It encrypts data in transit ensuring that any data sent between the client and server is protected from interception and tampering. TLS uses certificates to authenticate the server and establish an encrypted connection preventing eavesdropping and data breaches. TLS is continuously updated with newer versions offering better security.
* **HTTPS** is basically HTTP with more secure features which are provided by SSL or TLS, initially SSL and now TLS. When you visit a website using HTTPS, TLS encrypts the communication between your browser and the server. This protects sensitive data like login credentials from being intercepted by attackers.
That much information is more than enough on this topic to work on the application level.
```mermaid
sequenceDiagram
participant Browser as "Browser"
participant Server as "Server"
Browser->>Server: "HTTPS request"
Server-->>Browser: "Certificate"
Note over Browser: "Verify certificate"
Browser->>Server: "Encrypted session begins"
Server-->>Browser: "Encrypted HTTP data"
```
***
## Review
Backend is huge so we only cover what is used in 90% of codebases. HTTP is the medium through which browsers talk to servers.
Statelessness means no memory of past interactions, every request is self-contained with headers, URLs, methods, tokens. Client server model means the client always initiates.
HTTP uses TCP which is reliable and connection based, and we focus on layer 7 application layer.
HTTP 1.0 opened a new connection per request, 1.1 added persistent connections and caching, 2.0 added multiplexing with binary framing and HPACK, 3.0 uses QUIC over UDP.
Messages have method, URL, version, headers, blank line, body. Headers are key value metadata like the parcel label, with request, general, representation and security types, giving extensibility and remote control.
Methods define intent, PATCH is selective, PUT is complete replacement, thumb rule always use PATCH unless you need PUT. GET, PUT, DELETE are idempotent, POST is non-idempotent.
CORS with OPTIONS pre-flight controls cross origin access, simple versus pre-flight, browser blocks without Access-Control-Allow-Origin.
Status codes are the universal language, 200, 201, 204, 301, 302, 304, 400, 401, 403, 404, 405, 409, 429, 500, 501, 502, 503, 504.
Caching with Cache-Control max-age, ETag, Last-Modified, 304 Not Modified, or React Query for client side.
Content negotiation with Accept, Accept-Language, Accept-Encoding. Compression with gzip shrinks large responses significantly.
Persistent connections with keep-alive are default in 1.1. Multipart with boundary uploads files, chunked text event-stream downloads large files. HTTPS is HTTP plus TLS.
***
## See also
* [HTTP 1 vs HTTP 2 vs HTTP 3](https://github.com/nahomkasa999/Notes/tree/main/Blogs/blog-http-evolution.md) — how the protocol evolved across versions
* [HTTP 1 vs HTTP 2 vs HTTP 3 (CCNA)](/my-docs/basic-networking/http_1_vs_http_2_vs_http_3.md) — the networking perspective on the same evolution
***
## Resource
* [5. Understanding HTTP for backend engineers, where it all starts -- Sriniously](https://youtu.be/a3C1DMswClQ?si=SBCIlPoyLl36Mvgf)