Networking

This chapter is a general networking and communication-protocol learning path. It explains how systems communicate, how the major protocol families relate to one another, and how to choose and diagnose them.

It is intentionally not an HTTP API chapter. For HTTP methods, CRUD vs REST, headers, status codes, authentication, file upload, caching and CORS, use HTTP API semantics.

It is also intentionally not an Embedded & IoT chapter. MQTT, CoAP, Bluetooth/BLE, Zigbee, Thread, Matter, LoRaWAN, CAN, LIN, Modbus, UART, I²C, SPI, USB and similar device-oriented technologies belong in a separate Embedded & IoT communications learning path where they can be explained together and at the correct layers.

1. Protocols are a stack, not a flat list

A protocol is a set of rules that lets peers communicate. Real communication usually uses several protocols at once.

A practical Internet-oriented stack looks like this:

One HTTPS request may therefore involve:

DNS
  -> IP routing
  -> TCP or QUIC
  -> TLS
  -> HTTP
  -> JSON / HTML / another representation

These categories are easy to mix up:

• JSON, XML, Protocol Buffers and MessagePack are data formats, not network protocols.
• REST is an architectural style, not a protocol.
• GraphQL is a query language and execution model, not a transport protocol.
• gRPC is an RPC framework that commonly uses HTTP/2.
• SSE is an HTTP streaming mechanism, not a replacement transport layer.

OSI vs TCP/IP

The seven-layer OSI model is useful vocabulary, while the Internet protocol suite is usually the more practical model for real systems.

Do not force every modern technology into exactly one OSI box. Layering is a model for reasoning, not a law of nature.

2. IP, addresses, ports and sockets

IP

Internet Protocol provides addressing and packet delivery across interconnected networks. IPv4 and IPv6 are the two major versions in use.

IP itself does not promise that packets arrive, arrive once, or arrive in order. Higher layers provide the communication behavior an application needs.

Addresses

An IP address identifies a network interface/addressable endpoint in the IP layer. A hostname such as \api.example.com\ is not an IP address; DNS may resolve it to one or more IPv4 or IPv6 addresses.

Ports

TCP and UDP use port numbers to identify services/process endpoints on a host.

A default port is a convention, not proof of which application protocol is actually running.

Socket

A socket is an operating-system/application abstraction for network communication. A TCP connection is commonly identified by source IP, source port, destination IP and destination port.

This explains how one server can serve many simultaneous connections on the same listening port: every connection has a different endpoint tuple.

3. TCP and UDP

TCP

TCP provides an ordered, reliable byte stream between two endpoints.

Important properties:

• connection-oriented;
• retransmits lost data;
• preserves byte order;
• includes flow control and congestion control;
• exposes a byte stream, not application message boundaries.

Because TCP is a byte stream, an application protocol must define its own framing. One application message can be split across several TCP segments, and several small messages can appear in one read operation.

Common TCP-based protocols include HTTP/1.1, HTTP/2, TLS, SSH, SMTP and IMAP.

UDP

UDP provides independent datagrams. It does not add TCP-style connection establishment, delivery guarantees, ordering or retransmission.

That does not mean every UDP-based application is unreliable. A protocol built on UDP can implement its own reliability, ordering, congestion control and connection behavior. QUIC is the most important modern example.

TCP vs UDP

The choice is not simply “TCP is safe, UDP is fast.” The application protocol and workload determine which trade-offs matter.

4. QUIC: modern transport over UDP

QUIC is a secure, connection-oriented, multiplexed transport built on UDP.

It exists partly to move transport evolution out of operating-system TCP stacks and to improve connection establishment and multiplexing behavior.

Important concepts:

• QUIC integrates TLS 1.3 security into its handshake;
• one connection can contain multiple independent streams;
• loss affecting one stream does not impose TCP-style head-of-line blocking on unrelated streams;
• connection identifiers can help connections survive network-path changes, such as moving from Wi-Fi to mobile data;
• QUIC traffic is carried in UDP datagrams.

HTTP/3 uses QUIC as its transport.

QUIC is not HTTP/3. QUIC is the transport; HTTP/3 is an application protocol carried over it.

5. TLS: security for protocols

TLS protects data in transit between peers.

It provides three central properties:

1. Confidentiality — protected traffic should not be readable by passive observers.
2. Integrity — modification in transit should be detectable.
3. Authentication — normally the client authenticates the server certificate; mutual TLS can authenticate both peers.

A simplified HTTPS flow is:

DNS -> transport connection -> TLS handshake -> HTTP messages

Important TLS concepts:

• certificate;
• certificate authority and trust chain;
• hostname validation;
• certificate validity period;
• TLS version and cipher negotiation;
• SNI for selecting the intended virtual host;
• ALPN for negotiating an application protocol such as HTTP/2;
• session resumption;
• mutual TLS (mTLS).

HTTPS does not mean the application user is authenticated. TLS protects and authenticates the connection. Application authentication and authorization are separate concerns.

TLS is not specific to HTTP. It can protect SMTP, IMAP, AMQP and many other application protocols.

6. DNS: names before connections

DNS maps names to information needed to locate services.

High-value record types:

Recursive and authoritative resolution

A client usually asks a recursive resolver. The resolver may answer from cache or query the DNS hierarchy until it reaches authoritative data.

TTL and caching

DNS records have a TTL that influences how long answers may be cached. After a DNS change, different resolvers can temporarily return different results.

DNS is more than “hostname to IP”

DNS also participates in:

• mail routing through MX records;
• service discovery through SRV records;
• ownership/security configuration through TXT records;
• load distribution and failover patterns through multiple records and DNS providers.

DNS can use both UDP and TCP. Modern encrypted DNS variants also exist, including DNS over HTTPS and DNS over TLS.

7. HTTP/1.1, HTTP/2 and HTTP/3

All modern HTTP versions preserve the core HTTP semantics: methods, resource targets, fields/headers and status codes. The major differences are in framing and transport behavior.

HTTP/1.1

HTTP/1.1 uses textual message syntax and normally runs over TCP. Persistent connections let several requests reuse one connection, but it does not provide HTTP-level multiplexing of many concurrent streams in the way HTTP/2 and HTTP/3 do.

HTTP/2

HTTP/2 keeps HTTP semantics but adds:

• binary framing;
• multiplexed streams;
• header compression;
• stream prioritization mechanisms.

It normally runs over a single TCP connection on the public web. HTTP-level streams are independent, but packet loss in the shared TCP connection can still delay delivery for all streams.

HTTP/3

HTTP/3 maps HTTP semantics onto QUIC instead of TCP.

The separate HTTP API semantics chapter covers methods, REST, headers, status codes, authentication, files, caching and CORS.

8. WebSocket, SSE and polling

These technologies solve related real-time/update problems but use different communication models.

WebSocket

WebSocket provides a long-lived, bidirectional message channel between peers. In its classic web form, the connection starts with an HTTP Upgrade handshake and then switches to WebSocket framing.

Strong fits include:

• chat;
• collaborative editing;
• multiplayer/game state;
• trading/live dashboards;
• interactive control channels.

The dedicated WebSocket: build, test & debug chapter covers the protocol in depth.

Server-Sent Events (SSE)

SSE is an HTTP-based server-to-client event stream. A browser keeps an HTTP response open and receives text events over time.

It is a good fit when the server needs to push updates but the client can continue sending commands using ordinary HTTP requests.

Polling and long polling

Polling is an application pattern over HTTP. The client repeatedly asks whether anything changed.

Long polling keeps one request pending until an update or timeout occurs, then the client creates another request.

9. gRPC and RPC communication

Remote Procedure Call (RPC) systems make remote operations look more like method/function calls than resource-oriented HTTP interactions.

gRPC is a modern RPC framework. Services are defined by a contract, commonly with Protocol Buffers, and gRPC usually runs over HTTP/2.

Four gRPC interaction shapes are important:

1. Unary — one request, one response.
2. Server streaming — one request, many responses.
3. Client streaming — many requests, one response.
4. Bidirectional streaming — both sides stream messages.

Important concepts include:

• service and method definitions;
• message schemas;
• metadata;
• deadlines;
• cancellation;
• gRPC status codes;
• streaming;
• Protocol Buffer compatibility.

Do not describe gRPC as “JSON over HTTP/2.” Its contract, framing and common serialization model are different from a typical JSON/REST-style API.

10. Messaging and AMQP

Request/response is only one communication pattern. Messaging systems decouple producers and consumers in time and topology.

Common messaging concepts include:

• producer and consumer;
• broker;
• queue or address;
• publish/subscribe;
• acknowledgement / settlement;
• redelivery;
• retry;
• dead-letter handling;
• message TTL;
• ordering guarantees;
• backpressure;
• eventual consistency.

AMQP

AMQP is a binary application-layer messaging protocol family/standard used for interoperable messaging systems.

Be careful not to equate a broker product with a single protocol. A broker may support several protocols, and product-level concepts such as exchanges, queues and routing rules can extend beyond what a protocol specification itself defines.

Why MQTT is not covered here in depth

MQTT is a valid application-layer publish/subscribe protocol and can be used outside embedded systems. However, its most important learning context is IoT/device communication: constrained devices, brokers, intermittent connectivity, retained state, sessions and QoS. Covering MQTT deeply here while omitting BLE, CoAP, Thread, LoRaWAN and device buses creates an arbitrary curriculum boundary.

For that reason, MQTT is moved to the separate Embedded & IoT communications track. This chapter focuses on broadly applicable Internet/backend networking concepts.

11. Email protocols: SMTP, IMAP and POP3

Email uses different protocols for transport and mailbox access.

SMTP

SMTP is used to submit and transfer email.

A simplified path is:

mail client --SMTP--> sending server --SMTP--> receiving server

IMAP

IMAP is designed for mailbox access and synchronization while messages remain managed on the server. It supports mailboxes/folders, message flags and multi-client synchronization.

POP3

POP3 is a simpler retrieval protocol historically oriented around downloading messages from a mailbox.

MIME

MIME is not a transport protocol. It defines how email can represent structured bodies, content types, attachments and encodings.

A complete mail flow therefore combines several standards rather than “using one email protocol.”

12. FTP, FTPS, SSH and SFTP

These names are commonly confused.

FTP

FTP is a file-transfer protocol with separate control/data connection behavior. Classic FTP does not provide modern encrypted transport by itself.

FTPS

FTPS is FTP protected with TLS.

SSH

SSH provides a secure channel for remote login, command execution and related capabilities.

SFTP

SFTP means SSH File Transfer Protocol. It runs through SSH and is not “FTP plus SSH.” It is a different protocol and connection model.

13. Choosing a communication mechanism

Protocol choice should begin with the interaction model and constraints, not popularity.

Real systems frequently combine several mechanisms. A product may use HTTPS for its public API, gRPC internally, WebSocket for live updates, AMQP for asynchronous workflows and SMTP for notifications.

14. Diagnose communication layer by layer

When a system cannot communicate, isolate the failing layer.

1. Name resolution — does the name resolve to the expected destination?
2. Network reachability — is there a route to the destination?
3. Port/service reachability — is the expected TCP/UDP service accessible?
4. Transport — can the TCP/QUIC/etc. connection be established and maintained?
5. TLS — does certificate validation and protocol negotiation succeed?
6. Application protocol — are both peers speaking compatible protocol versions?
7. Authentication/authorization — does the application accept the identity and permissions?
8. Payload/contract — is the application message valid?
9. State and timing — are retries, timeouts, ordering, caching or asynchronous state involved?

Useful tools:

A packet capture is powerful, but it is not the first tool for every problem. Start with the highest layer that can quickly confirm or reject the current hypothesis.

15. Practice

Exercise 1 — classify the stack

For an HTTPS request returning JSON, identify the role of DNS, IP, TCP or QUIC, TLS, HTTP and JSON. Explain why JSON is not “the protocol used by the API.”

Exercise 2 — TCP vs UDP

Explain why DNS often uses UDP while HTTP/1.1 normally uses TCP, then explain why HTTP/3 can still be reliable even though QUIC runs over UDP.

Exercise 3 — HTTP versions

Compare HTTP/1.1, HTTP/2 and HTTP/3 without discussing methods or status codes. Focus only on transport, framing and multiplexing.

Exercise 4 — real-time design

Choose between polling, SSE and WebSocket for:

• a build-progress screen;
• a chat application;
• a dashboard refreshed every five minutes.

Defend each choice.

Exercise 5 — diagnose a failure

A browser reports that \https://api.example.com\ is unavailable. Build an investigation sequence from DNS to TLS to HTTP rather than immediately assuming an application bug.

16. QA quick reference

The learning path above is intentionally general. For testing work, use this compact lens rather than treating every section as QA-specific content.

The most useful principle is test the failure model of the protocol you actually use rather than applying one generic network checklist to every technology.

Embedded & IoT scope boundary

Embedded and IoT communication deserves its own layered curriculum rather than being mixed into this Internet/backend protocol survey.

That separate track should distinguish:

These are not all peers at one protocol layer. For example, Thread is an IPv6-based low-power mesh network, Matter is an application-layer ecosystem over IP networks such as Thread/Wi-Fi/Ethernet, and MQTT/CoAP sit at the application layer. Bluetooth LE is a complete wireless stack with its own radio/link and higher-layer concepts. Treating all of them as one flat “protocol list” would repeat the same structural mistake this Networking chapter is designed to avoid.

Sources

• RFC 8200 — Internet Protocol, Version 6
• RFC 9293 — Transmission Control Protocol (TCP)
• RFC 768 — User Datagram Protocol
• RFC 9000 — QUIC: A UDP-Based Multiplexed and Secure Transport
• RFC 8446 — TLS 1.3
• RFC 1034 — Domain Names: Concepts and Facilities
• RFC 9112 — HTTP/1.1
• RFC 9113 — HTTP/2
• RFC 9114 — HTTP/3
• RFC 6455 — The WebSocket Protocol
• gRPC Core Concepts
• OASIS AMQP 1.0
• RFC 5321 — SMTP
• RFC 9051 — IMAP4rev2
• RFC 1939 — POP3
• RFC 959 — FTP
• RFC 4251 — SSH Protocol Architecture
• Bluetooth SIG — Bluetooth technology overview
• Thread Group — What is Thread?
• RFC 7252 — Constrained Application Protocol (CoAP)