> For the complete documentation index, see [llms.txt](https://naz.gitbook.io/my-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://naz.gitbook.io/my-docs/backend-learning/validations_and_transformations_for_backend_engineers.md). # Lesson 5: Validations and Transformations for Backend Engineers ## API Architecture Overview A typical backend executes in layers. ### The Repository Layer The bottom layer is the **repository layer** -- it deals with database connections and query execution: insertions, deletions, and everything related to data. The database can be a traditional relational database, Redis, or any other kind of persistent storage. ### The Service Layer Above it sits the **service layer**, which executes business logic. A service method typically calls one or more repository methods, and can also send notifications to devices, send emails to users, store data, or make webhook calls. All the functionality an API call needs lives here. ### The Controller Layer Above the service layer is the **controller layer**. It calls the service method associated with the API and returns whatever data the service method produces to the user over the HTTP connection. Controller and service stay separate so that HTTP-related concerns live in their own layer: which error or success code to return, what format the response takes, and what validations to run. The controller handles all data coming from clients and all data going out to clients, and internally calls the service layer. The service layer may or may not call the repository layer, depending on the requirements. So an API call reaches the controller layer, which interacts with the service layer, which interacts with the repository layer, and the data returned from the service layer goes back to whoever made the API call. ```mermaid flowchart TB CLIENT["Client"] --> CTRL["Controller layer
status codes, response format, validations"] CTRL --> SVC["Service layer
business logic"] SVC --> REPO["Repository layer
queries, inserts, deletes"] REPO --> DB[("Database
relational, Redis, other storage")] ``` *** ## The Validation and Transformation Pipeline ### Where the Pipeline Sits Validations and transformations happen at one exact point: after the route is matched and its controller method is identified, but before any significant logic runs in the controller -- before calling service methods or executing business logic. ```mermaid flowchart LR REQ["Client request
JSON, query params,
path params, headers"] --> ROUTE["Route matching"] ROUTE --> PIPE["Validation and
transformation pipeline"] PIPE --> CTRLL["Controller logic"] CTRLL --> SVC2["Service layer"] SVC2 --> REPO2["Repository layer"] ``` The idea behind the pipeline: every piece of data clients send -- JSON payload, query parameters, path parameters, headers -- must be in the expected format before it enters the server. ### How the Pipeline Checks Data Consider an API that expects a JSON payload with a field called **name**: a string between 5 and 100 characters. Before the data reaches the service layer or any database method, the validation pipeline checks the structure. The pipeline is a **middleware** or utility function reused in every API, given a **schema** to validate against. It walks the fields of the JSON in order: 1. **Presence** -- is there a field called `name`? If not, it sends an error right away: the name field is required. 2. **Type** -- is the value a string? If the client sent an array where a string is expected, the error says a string was expected. 3. **Constraint** -- is the length within 5 to 100 characters? A paragraph-long value fails as too long; a two-character string fails the expected length. Each gets its own customized error message. ```mermaid flowchart TB START["JSON payload arrives"] --> P["Field present?"] P -->|No| E1["Error: field is required"] P -->|Yes| T["Type matches?"] T -->|No| E2["Error: expected type X"] T -->|Yes| C["Constraints satisfied?
length, range, format"] C -->|No| E3["Error: constraint message"] C -->|Yes| OK["Pass to controller logic"] ``` This guarantees the payload has at least the expected structure before the API runs any database operation or business logic. ### Why the Entry Point Matters Imagine the validation pipeline does not exist, and a client sends a JSON payload with `name` set to `0` instead of a string. The data reaches the controller layer unchecked. The controller calls the service layer, which executes the business logic. The service calls a repository method that creates a new row -- for example, a new book with the name the user provided. Relational databases enforce data type constraints. When the books table was created, the `name` column was declared as `text` and `NOT NULL` (Postgres). The repository executes the insert with the value `0`. Postgres validates data types before any operation, so the database call fails. Depending on the server's error handling, the client now gets a **500** -- internal server error, meaning something unexpected happened in the server. That is a poor user experience for a basic form API. With validation at the entry point, the same payload is rejected immediately with a **400** -- bad request: the data does not satisfy the constraints, so retry with the appropriate format. Doing validation at the entry point means the system never breaks under unexpected state. ```mermaid sequenceDiagram participant C as Client participant API as Server (no validation) participant DB as Postgres C->>API: POST name = 0 API->>DB: INSERT name = 0 DB-->>API: Type error (expects text) API-->>C: 500 Internal Server Error Note over C,DB: With validation at the entry point,
the request is rejected with 400
before reaching the database ``` *** ## Syntactic Validation Demo A running server exposes `POST /api/validations/syntactic`. Sending an empty JSON payload returns an array of errors: the API expects three fields -- **email**, **phone**, and **date** -- all required. Filling them in with a random string for email, a number for phone, and `2025-11-05` for the date returns two errors: * **email** -- invalid email format * **phone** -- expecting a string but received a number Fixing the email to a valid address clears the email error. Converting the phone value to a string returns a **200** success code. *** ## The Three Types of Validation These are not strict categories -- more types exist depending on requirements -- but these three cover most cases. ### Syntactic Validation The value must follow a particular structure. An email has a structure: a local value, the `@` character, a domain name, and a top-level domain. The validation algorithm checks whether the provided string satisfies that structure. Phone numbers work the same way: a country code followed by a set number of digits, with the digit count depending on the country code. Dates also have a structure -- for example year, then month, then day -- and the check is whether the string follows the format the API expects. ### Semantic Validation The value must make sense. A date-of-birth field cannot hold a future date: if today is 2025-11-01, then 2026-01-13 is rejected because a date of birth cannot be in the future. An **age** field with the value 365 is rejected because it makes no sense -- at least not yet. ### Type Validation The data type must match. Is the field a string, a number, a boolean, an array, or a nested JSON payload? A JSON value can itself hold strings, numbers, booleans, and arrays, each checked in turn. ```mermaid flowchart TB V["Validation pipeline"] --> SYN["Syntactic
follows the structure?
email, phone, date formats"] V --> SEM["Semantic
makes sense?
no future birth dates,
sane age ranges"] V --> TYP["Type
correct data type?
string, number,
boolean, array"] ``` These are guidelines to keep in mind while designing APIs: how strict to be with data, how strict the validation pipeline should be, and how specific the rules get. *** ## Data Transformation Techniques Validation confirms the data matches the expected structure. **Transformation** executes operations on the user-provided data -- before validation, after validation, or both -- to convert it into the format the server wants. ### Casting Query Parameters Consider a paginated API: `GET /bookmarks?page=2&limit=20`. Query parameters need validation too, before any significant logic runs. The requirements might be: * **page** -- a number, greater than 0 and smaller than 500 * **limit** -- a number, greater than 0 and smaller than 10000 But query parameters arrive at the server as **strings** by default. The browser sends `page` as the string `"2"` and `limit` as the string `"20"`. The first validation requirement -- "must be a number" -- fails immediately, and the user gets an error even though nothing was wrong with the request. The client is not at fault: query parameter values are always strings. So the server takes responsibility and **casts** the data type -- forcing one data type to convert into another -- turning the string into a number, then running the validation requirements. This phase is called transformation: data from the client goes through changes before validations can run. It works the other way too -- after validation, operations can reshape data into the structure the service layer expects. ```mermaid flowchart LR Q["Query string
page = '2', limit = '20'
(strings)"] --> TR["Transformation
cast string to number"] TR --> VAL["Validation
number and range checks"] VAL --> PASS["Pass to controller"] ``` ### One Pipeline for Validation and Transformation The process of converting data into a desirable format is called transformation. Validations and transformations are usually paired in a single pipeline, so all input-data logic -- every requirement and every operation before business logic -- stays in one place instead of being scattered across the codebase. *** ## Semantic Validation Demo `POST /api/validations/semantic` expects two fields: **dateOfBirth** (required) and **age** (a number). ### Empty Payload Sending an empty payload returns both errors at once: ``` POST /api/validations/semantic HTTP/1.1 Host: localhost Content-Type: application/json {} ``` ``` HTTP/1.1 400 Bad Request Content-Type: application/json { "errors": [ { "field": "dateOfBirth", "message": "dateOfBirth is required" }, { "field": "age", "message": "Expected number, received none" } ] } ``` Notice the side benefit: even without API documentation, the validation errors tell the client exactly which fields are expected and what values they take. ### Valid Data ``` POST /api/validations/semantic HTTP/1.1 Host: localhost Content-Type: application/json { "dateOfBirth": "1995-06-12", "age": 43 } ``` ``` HTTP/1.1 200 OK Content-Type: application/json { "dateOfBirth": "1995-06-12", "age": 43 } ``` The server echoes back the values it received. ### A Date in the Future Changing the year to 2026: ``` POST /api/validations/semantic HTTP/1.1 Host: localhost Content-Type: application/json { "dateOfBirth": "2026-06-12", "age": 43 } ``` ``` HTTP/1.1 400 Bad Request Content-Type: application/json { "errors": [ { "field": "dateOfBirth", "message": "Date of birth cannot be in the future" } ] } ``` ### An Impossible Age Reverting the date and setting the age to 430: ``` POST /api/validations/semantic HTTP/1.1 Host: localhost Content-Type: application/json { "dateOfBirth": "1995-06-12", "age": 430 } ``` ``` HTTP/1.1 400 Bad Request Content-Type: application/json { "errors": [ { "field": "age", "message": "Number must be less than or equal to 120" } ] } ``` A sensible human age range is 1 to 120. *** ## Complex Validation Scenarios `POST /api/validations/complex` expects three fields: **password**, **passwordConfirmation**, and **married**. ### Fields That Must Match Sending mismatched strings for the two password fields returns "passwords don't match" on the confirmation field, plus "string must contain at least eight characters" on the password field itself. Most platforms pair a password field with a confirmation field for security and data integrity -- the confirmation must always match the password. Fixing both -- eight or more characters, identical values -- returns success. ### Fields That Depend on Each Other Setting **married** to `true` triggers another error: "partner name is required when married is true". The **partner** field is optional when married is false, but becomes required when married is true. Adding a partner name returns success with all provided fields. So validation constraints can relate fields to each other: two fields must hold the same value, or one field becomes required depending on another field's value. ```mermaid flowchart TB IN["Payload: password,
passwordConfirmation, married"] --> M["password equals
passwordConfirmation?"] M -->|No| E1["Error: passwords do not match"] M -->|Yes| L["password at least
8 characters?"] L -->|No| E2["Error: minimum 8 characters"] L -->|Yes| MAR["married is true?"] MAR -->|Yes| P["partner provided?"] P -->|No| E3["Error: partner name required
when married is true"] P -->|Yes| OK1["Success"] MAR -->|No| OK1 ``` *** ## Transformation Demo `POST /api/validations/transformation` expects **email**, **phone**, and **date**. Sending an email with mixed upper and lower case, a phone number without the leading `+`, and a date in one format returns all values reshaped: the email comes back in all lower case, the phone number gains the `+` prefix, and the date is recalculated into a different format. The server transformed the data into the format the service layer requires, before executing anything on it. *** ## Type Validation Demo `POST /api/validations/types` expects four fields: **stringField**, **numberField**, **arrayField**, and **booleanField**. Sending strings for all four returns type errors: expected number received string, expected array received string, expected boolean received string. Fixing the number to `10` clears that error. Sending `[1, 2]` for the array introduces a deeper error: element 0 is expected to be a string. Changing the elements to strings and the boolean to `false` returns success with all values echoed back. Type validation can reach inside arrays and enforce a type per element. *** ## Frontend Validation Is Not Backend Validation Forms often validate input before submission: a **name** field checks minimum and maximum characters and data type, shows errors inline, and only calls the API when the data satisfies the constraints. That frontend validation exists for **user experience** -- immediate feedback that something is wrong with the data. It is not for security or data integrity. Backend validation is mandatory and serves **security and data integrity**. It does not matter what the frontend validates, because a server has many kinds of clients. A web app can validate thoroughly, but an API client like Postman or Insomnia talks to the API directly -- there is no frontend interface acting as a proxy, so no frontend validation exists at all. If the backend depended on frontend validation for security, the server would break the moment the client changed. When designing APIs, the server-side validation logic must be as strict and as specific as possible, without assuming anything about what the client validates. ```mermaid flowchart LR FORM["Form input"] --> FE["Frontend validation
purpose: user experience
immediate feedback"] FE -->|Valid| CALL["API call"] FE -->|Invalid| SHOW["Show inline errors
no API call"] CALL --> BE["Backend validation
purpose: security,
data integrity"] BE --> SVC3["Service layer"] ``` A companion frontend demo shows both working together: submitting a bad email and phone number triggers frontend errors with no API call at all. Fixing the email to a valid address submits the form, the payload goes out, and the server answers 200. Client-side validation gives the user immediate feedback; server-side validation performs the mandatory security checks. *** ## Review * Validations and transformations are a set of rules and guidelines for designing APIs, mostly about data integrity and security. * A backend runs in layers: repository (database), service (business logic), controller (HTTP concerns). Requests flow controller to service to repository. * The validation and transformation pipeline sits at the entry point: after route matching, before any controller logic or service calls. * The pipeline checks presence, then type, then constraints, and rejects bad data with a **400** instead of letting it fail deep inside with a **500**. * The three common validation types: **syntactic** (structure), **semantic** (makes sense), and **type** (correct data type). * **Transformation** converts data into the desirable format -- for example casting query-string values from strings to numbers -- and pairs with validation in a single pipeline. * Validation errors double as documentation: they tell the client exactly which fields and values an API expects. * Frontend validation is for user experience; backend validation is for security and data integrity, and it must never depend on the client. *** ## In Production Today The episode teaches the concepts; in real production codebases the pipeline is usually built with dedicated validation libraries rather than hand-written checks. The pattern is the same everywhere: declare the schema once, validate at the entry point, return **400** with structured errors, and never let unvalidated data reach the service layer. * **Zod** -- the current default in the TypeScript/Node.js world. Schemas are declared once and double as types; `.transform()` and `.coerce` handle the transformation step (for example coercing query-string numbers). Used with tRPC, Hono, Express, and Fastify. * **Pydantic v2** -- the standard for Python backends, built into FastAPI. Request bodies are validated and coerced automatically from type annotations; custom validators cover semantic rules like "date of birth cannot be in the future". * **go-playground/validator** -- the Go ecosystem's choice, with struct tags such as `validate:"required,email,min=5,max=100"`. * **class-validator + class-transformer** -- the NestJS stack, where decorators declare rules and `@Transform()` reshapes values before validation. * **Joi / Yup** -- the older Node.js workhorses, still common in Express codebases. * **JSON Schema + OpenAPI** -- language-agnostic contracts; frameworks can validate requests against the published spec automatically. *** ## Resource * [9. Validations and transformations for backend engineers -- Sriniously](https://www.youtube.com/watch?v=qedj_JjjL-U)