Python: Consuming HTTP APIs

Hyper Text Transfer Protocol (HTTP) based APIs are ubiquious in the Internet these days. Whether they are in the form of Simple Object Access Protocol (SOAP), Representational State Transfer (REST) or simple GET/POST requests, HTTP based API's are the de-facto standard for consuming services over the web.

In this article, we take a look how to consume these HTTP API's from Python. There are three main modules that can be used to make HTTP API calls in Python, urllib, httplib and requests. Under the hood, requests uses urllib which in turn uses some functionality of httplib. Hence requests provides the most abstraction and ease of use among the three and we will be using that in this article.

The following code shows how to make simple GET and POST requests using the requests module. The GET will download the main page of the site https://cbelwal.blogspot.com which the POST will make a POST call and will get the 405 error. Note that we are accessing the URL over SSL and thankfully the requests module abstracts the SSL handshake and other protocol specific details. One word of caution is that some older version of the requests module will not check for the validity of the web server's certificate (like a web browser does). So you will have to check the server validity on your own if you desire that functionality, or better still use the latest version of the request library which has the option to check for certificate validity. To disable check for certificate validity, use the flag verify=False as one of the options to the function.

In our code we connect to a normal webpage to print the HTML response from a GET and POST call to the web server. These are basic calls with no JSON payload sent or received from the server. In the next article we will make REST calls to an Azure API and send receive payload. 

# If requests module is missing type this in the python console:
# pip install requests
import requests

class demoHTTPClass:
    def __init__(self):
        # Make a call over SSL to specific URL
        self.url = "https://cbelwal.blogspot.com"

    # Make a get request to the webserver
    # This will return the HTML of the landing page
    def getFromURL(self):
        # verify=True is the default option and which check
        # certificate validity. Using verify=False will disable check
        # for certificate validity
        req = requests.get(self.url, verify=True)
        print('------------------- In getFromURL ')
        print('Response code:', req.status_code)
        print('Response header:', req.headers)
        print('------------------- Downloaded content ')
        print(req.text)

    # Make a post request to the webserver
    # Server will return HTML with 405 error as this method is not supported
    def postToURL(self):
        # verify=True is the default option and which check
        # certificate validity. Using verify=False will disable check
        # for certificate validity. Though not recommended, we have used
        # verify=False just to show the warning that comes up when this option is used.
        print('------------------- In postToURL ')
        req = requests.post(self.url, verify=False)
        print('Response code:', req.status_code)
        print('Response header:', req.headers)
        print('------------------- ', req.headers)
        print('------------------- Downloaded content ')
        print(req.text)


# main() function which contain the high level routines
def main():
    dhc = demoHTTPClass()
    dhc.getFromURL()
    dhc.postToURL()

# Call the main() function
main()

The output of the above code is given below. For the sake of brevity, we have cut parts of the HTML page received from the GET request.

------------------- In getFromURL 
Response code: 200
Response header: {'Content-Type': 'text/html; charset=UTF-8', 'Expires': 'Wed, 12 Dec 2018 14:20:13 GMT', 'Date': 'Wed, 12 Dec 2018 14:20:13 GMT', 'Cache-Control': 'private, max-age=0', 'Last-Modified': 'Wed, 12 Dec 2018 14:19:19 GMT', 'ETag': 'W/"cadbd71c124ab6556f7e19f3df51fcacddd7b71d33ec2e2324f303494c5b0a7b"', 'Content-Encoding': 'gzip', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Server': 'GSE', 'Alt-Svc': 'quic=":443"; ma=2592000; v="44,43,39,35"', 'Transfer-Encoding': 'chunked'}
------------------- Downloaded content 
<!DOCTYPE html>
<html dir='ltr' xmlns='http://www.w3.org/1999/xhtml' xmlns:b='http://www.google.com/2005/gml/b' xmlns:data='http://www.google.com/2005/gml/data' xmlns:expr='http://www.google.com/2005/gml/expr'>
<head>
<meta content='width=device-width, initial-scale=1' name='viewport'/>
<meta content='cc7fd325af51b28443d2a1638a1984b5' name='propeller'/>
<title>Chaitanya Belwal's Blog</title>
...
...
...
------------------- In postToURL 
C:\Users\Chaitanya Belwal\PycharmProjects\SocketsProgramming\venv\lib\site-packages\urllib3\connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
Response code: 405
Response header: {'Content-Type': 'text/html; charset=UTF-8', 'Content-Encoding': 'gzip', 'Date': 'Tue, 18 Dec 2018 14:23:08 GMT', 'Expires': 'Tue, 18 Dec 2018 14:23:08 GMT', 'Cache-Control': 'private, max-age=0', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Server': 'GSE', 'Alt-Svc': 'quic=":443"; ma=2592000; v="44,43,39,35"', 'Transfer-Encoding': 'chunked'}
-------------------  {'Content-Type': 'text/html; charset=UTF-8', 'Content-Encoding': 'gzip', 'Date': 'Tue, 18 Dec 2018 14:23:08 GMT', 'Expires': 'Tue, 18 Dec 2018 14:23:08 GMT', 'Cache-Control': 'private, max-age=0', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Server': 'GSE', 'Alt-Svc': 'quic=":443"; ma=2592000; v="44,43,39,35"', 'Transfer-Encoding': 'chunked'}
------------------- Downloaded content 


Method Not Allowed



Method Not Allowed
Error 405




Process finished with exit code 0

Comments

  1. CyanousJuly 12, 2021 at 9:43 PM

    Nice blog..clearly explained…Thankyou so much for your wonderful information…Looking for the best programming platforms in Hyderabad contact cyanous software solutions now.

    Best programming platform services in Hyderabad
    Best software & web development company in Hyderabad

    ReplyDelete

Post a Comment

Popular posts from this blog

Part III: Backpropagation mechanics for a Convolutional Neural Network

Image
In part-II of this article, we derived the weight update equation for the backpropagation operation of a simple Convolutional Neural Network (CNN). The input for the CNN considered in part-II is a grayscale image, hence, the input is in the form of a single 4x4 matrix. CNNs used in practice however, use color images where each of the Red, Green and Blue (RGB) color spectrums serve as input. Hence, a CNN with the inputs color coded with their respective spectrums look like: In this article we expand upon the original CNN where the input is represented by 3, 4x4 matrices each presenting the pixel intensities for the RGB color spectrum. Hence is we represent each of the pixels with the CNN given is represented by: In this CNN, there are 3, 4x4 input matrices, one 2x2 filter matrix (also known as kernel), a single convolution layer with 1 unit, a single ReLu layer, a single pooling layer (which applies the MaxPool function) and a single fully connected (FC) layer. The elements
Read more

Introducing Convolution Neural Networks with a simple architecture

Image
Convolution Neural Networks (CNNs) are different from standard Neural Networks (NNs) as their input is 2-dimensional vs. 1-dimensional in a standard neural network. While the underlying principles between CNNs and NNs are same, CNNs do introduce some new concepts. In this article, we take a look at how CNNs operate using a very simple CNN architecture, given below. In this simple CNN, there is one 4x4 input matrix, one 2x2 filter matrix (also known as kernel ), a single convolution layer with 1 unit, 1 Rectified Linear Unit (ReLu) layer with one unit, a single pooling layer and a single fully connected (FC) layer. The elements of the filter matrix are equivalent to the unit weights in a standard NN and are updated during the backpropagation phase. Real-life CNNs are significantly more complex than this with several repeating layers of convolutional, ReLu and pooling layers forming a true deep learning network. All the matrix dimensions are also magnitudes higher to represent hig
Read more

Deriving Pythagoras' theorem using Machine Learning

Image
In an earlier article which introduced machine learning, we had seen a comparison of equations derived from scientific principles vs. those derived via machine learning. In this article, we will see how Dythagoras a citizen of the far away planet Darth in the Dilky Day galaxy uses machine learning to derive a correlation between the length of 3 sides of a right-angle triangle. Dythagoras - resemblance to Pythagoras not intentional Unlike humans, the citizens of Darth (Darthians) are not very bright mathematically but are capable of observations and making tools. They had been handed a computer and the Dasmic machine learning library by human astronauts of a SpaceX expedition who had visited them a few months ago. The citizens of Darth need to shorten the travel time between 2 of their biggest cities of Aville and Cville. Currently the only way to goto Cville from Aville is via Bville that takes several hours. Darthians need to know the direct distance between Aville and Cvill
Read more