Making Web Requests in Python: A Beginner's Guide

Interact with the web using Python. Published by Prince Jan on February 10, 2021

Web requests are a powerful way to get data from websites. This tutorial will show you how to use the Requests module in Python to do this in a few simple steps.

A web request can also be known as an HTTP request which is the protocol that interacts between the client and the server. We make web requests everytime when visiting websites via browsers(clients).

Well, in python, web requests are made through more specific functions, such as get(), request(), post(), etc and the Python Requests module gives us access to this functions

The Python Requests is a great library. It allows you to make GET and POST requests with the options of passing url parameters, adding headers, posting form data and more.

With the library, you can make requests to almost every website/webpage, but it power lies in accessing APIs to retrieve data in the form of JSON which you can then work with in your code, applications and scripts.

Installing the Requests library

To install the python Requests library, run the command:

pip install requests

This installs the Requests module and dependencies if any.

Read More: 8 Excellent Python Courses on Udemy (2021)

 

Making a simple request with the Requests module

Let's try to make a simple request the python website, https://www.python.org ( you can use any website):

import requests

r = requests.get("https://www.python.org/")

First, we begin by importing the Requests module. From the Requests module, we use a get() function with our website's url passed in. With that line of code, a request is made to https://www.python.org and the response is saved into the r variable.

Making a request with the Requests module requires an internet connection.

Now we have a Response object called r. We can retrieve all the information we need from this object.

After a request is made, we get a repsonse from the website's server and we can read the server's response:

r.status_code    # 200

The status response code of 200 indicates a successful request. You can access the source code of the webpage by:

print(r.text)        # prints webpage's source code

The entire source code of the webpage will be printed in your console. This is useful when doing web scraping.

 

Making requests to APIs

By accessing APIs, we can tap into the full functionality of the Requests module. In this tutorial we will use an API that returns some fake data( this is just for learning, we will use a real one later ) from https://jsonplaceholder.typicode.com/posts . The fake data that will be returned represents data of a blog; there are blog posts, users and comments.

import requests
r = requests.get("https://jsonplaceholder.typicode.com/posts")

print(r.status_code)    # 200

Now, we have a Response object, r. We can retrieve all the information we need from it.

Response Content

We can read the content in various ways by using the attributes and functions provided by the Request module

r.status_code    # 200

r.encoding       # utf-8

r.url            # https://jsonplaceholder.typicode.com/posts

r.json()           # returns the json data

r.text           # returns the sourcecode

r.content        # returns the reponse body as bytes

r.status_code returns a code indicating whether a request was successful or not. 200 means successful. Common status codes that you have probably seen are 200, 404, and 500. 404 means client error and 500 means there was a server error.

r.encoding returns the encoding of the response based on the http headers.

r.url returns the requested url

r.json returns any json data from the response.

r.text returns response is text format

r.content returns the response formated in bytes

 

Working with JSON response data

The json data returned from https://jsonplaceholder.typicode.com/posts contains a lot of post data, we are going to need only one post data, therefore we will use this url https://jsonplaceholder.typicode.com/posts/1 which returns the first post object.

import requests
r = requests.get("https://jsonplaceholder.typicode.com/posts/1")

print(r.json())

Output:

{'userId': 1, 'id': 1, 'title': 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit', 'body': 'quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto'}

Note that this is just some dummy data representing a post/article with usedId, Id, title and a body. r.json() parses the response into a python-compatible data type i.e a dictionary or list.

 

import requests
r = requests.get("https://jsonplaceholder.typicode.com/posts/1")

data = r.json()                    # json response is stored in 'data'

post_num = data['id']
user = data['userId']
title = data['title']
body = data['body']

print(f"Post number: {post_num}")
print(f"Written by user: {user}")
print("Title: " + title)
print()                             # just for space
print(body)

Output

Post number: 1
Written by user: 1
Title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit

quia et suscipit
suscipit recusandae consequuntur expedita et cum
reprehenderit molestiae ut ut quas totam
nostrum rerum est autem sunt rem eveniet architecto

 

Passing URL Parameters

It is possible to pass parameters in the url's query string when working with Requests. This allows you to pass some sort of data. URL parameters are a common way of passing data, you often see them after the ? in a url. An example is https://www.google.com/search?q=hello  this is the url of google when you search for "hello", therefore the q=hello is the url parameter.

But we will use our dummy API to retrieve a comments related to a particular post.

To pass parameters to the url, pass the parameters as a dictionary to the params keyword argument. for ?q=hello will be {'q':'hello'} or ?postId=1 will be {"postId":1}

import requests

r = requests.get("https://jsonplaceholder.typicode.com/comments/", params={"postId":1})

print(r.url)           # https://jsonplaceholder.typicode.com/comments/?postId=1

You will notice that r.url returns a url that has appended a parameter to it; ?postId=1.

import requests

my_parameters = {"postId":2, "userId":8}

r = requests.get("https://jsonplaceholder.typicode.com/comments/", params=my_parameters)

print(r.url)          # https://jsonplaceholder.typicode.com/comments/?postId=2&userId=8

 

Read More: 8 Excellent Python Courses on Udemy (2021)


Subscribe to receive updates!

0 Comments

Leave a comment

* Email would not published