Understanding the process of HTTP requests when you type a URL in the browser

tags: HTTP requests, browser, URL, DNS lookup, TCP connection, request, response, HTML parsing

When you type something into the Google search box and press enter, have you ever wondered what happens behind the scenes? This article will explain the step-by-step process of how browsers perform page requests using the HTTP/1.1 protocol.

The HTTP protocol

Note that this article focuses on HTTPS connections differ from regular HTTP connections.

I analyze URL requests only

Modern browsers can determine if the text entered in the address bar is a valid URL or a search term. For the purpose of this explanation, let’s assume you type an actual URL.

Once you enter the URL and press enter, the browser constructs the full URL. If you only entered a domain name, like flaviocopes.com, the browser automatically adds http:// to the beginning, defaulting to the HTTP protocol.

DNS Lookup phase

The browser initiates a DNS lookup to obtain the server’s IP address. Although domain names are human-friendly, computers need the IP address to determine the exact location of the server.

First, the browser checks its local DNS cache to see if the domain has been recently resolved. If not found, the browser uses the gethostbyname POSIX system call to fetch the host information.

gethostbyname

The gethostbyname function first looks in the local hosts file, usually located in /etc/hosts on macOS or Linux, to see if the system provides the necessary information. If no information is found, the system makes a request to the DNS server.

The DNS server’s address is specified in the system preferences, often provided by the internet service provider. Popular DNS servers include 8.8.8.8 (Google public DNS) and 1.1.1.1 (CloudFlare DNS).

The browser uses the UDP protocol to perform the DNS request. Note that UDP is a connectionless protocol designed for lightweight message transmission.

The DNS server may have the domain IP in its cache. If not, it forwards the request to the root DNS server. The root DNS server consists of 13 servers distributed worldwide and serves as the backbone of the internet.

The root DNS server does not have the address for every domain. Instead, it knows where the top-level DNS resolvers are located. These top-level domains include extensions like .com, .it, and .pizza.

For example, if you’re searching for flaviocopes.com, the root DNS server returns the IP of the .com TLD server. The DNS resolver caches the IP of the TLD server to minimize future requests to the root DNS server.

The TLD DNS server contains the IP addresses of the authoritative Name Servers for the requested domain. These authoritative Name Servers are managed by the hosting provider and are usually redundant for backup.

For instance:

  • ns1.dreamhost.com
  • ns2.dreamhost.com
  • ns3.dreamhost.com

The DNS resolver starts with the first Name Server and requests the IP address for the specific domain (including subdomains). This Name Server serves as the ultimate source of truth for the IP address.

With the IP address obtained, we can proceed to the next steps.

TCP request handshaking

Now that the server’s IP is available, the browser can establish a TCP connection. Before data transfer can occur, the TCP connection requires a handshaking process.

Once the connection is established, we can transmit the request.

Sending the request

The request is a plain text document structured according to the communication protocol. It consists of three parts: the request line, the request header, and the request body.

  • The request line includes the HTTP method, the resource location, and the protocol version. For example: GET / HTTP/1.1.
  • The request header contains field: value pairs that set specific values. Mandatory fields include Host and Connection, while other fields are optional. For example: 
    Host: flaviocopes.com
Connection: close
  • The request body is optional for GET requests, but it is often used in POST requests and other verbs. It can contain data in JSON format.

The response

After receiving the request, the server processes it and sends back a response. The response begins with a status code and message. A successful response with a status code of 200 starts with 200 OK. Other status codes include 404 Not Found, 403 Forbidden, 301 Moved Permanently, 500 Internal Server Error, 304 Not Modified, and 401 Unauthorized.

The response also includes a list of HTTP headers and the response body, which, in the case of browser requests, is typically HTML.

Parse the HTML

Once the browser receives the HTML, it starts parsing it. The browser repeats this process for all the resources required by the page, such as CSS files, images, the favicon, JavaScript files, and more.

Note that how browsers render the page is out of the scope of this post. However, it is crucial to understand that the described process applies not only to HTML pages but to any resource requested over HTTP.