Introduction
We will discuss here how to build a simple online visitor tracking system using Python Flask. Every website owner would like to know the progress to the count of the online visitors because it gives them overall idea on the website’s analytical measurements. It will also give them how to make improvements on the websites.
We will use here @app.before_request decorator which Flask API provides to track your visitor when a visitor lands on your website.
This example will give you an idea how to implement such system to track visitors who visit your website on daily basis.
Prerequisites
Python 3.8.0, Flask 1.1.1, MySQL 8.0.17, PyMySQL 0.9.3
Related Posts:
Creating MySQL Table
We need to create MySQL table to store visitor information.
In the following table we store how many times a visitor has visited application or website. We store visitor’s IP address, which URLs visitor visits, from which page or URL visitor navigates, if there is any query parameters in the URL, on which browser visitor opens the website.
We have also another flag whether the visitor is unique or not. So it depends on how you are going to implement the functionality. In this basic example I have not worked on this flag.
CREATE TABLE `visits_log` (
`log_id` int unsigned NOT NULL AUTO_INCREMENT,
`no_of_visits` int unsigned NULL,
`ip_address` varchar(20) NULL,
`requested_url` tinytext NULL,
`referer_page` tinytext NULL,
`page_name` tinytext NULL,
`query_string` tinytext NULL,
`user_agent` tinytext NULL,
`is_unique` tinyint NOT NULL DEFAULT '0',
`access_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`log_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;Creating Project Directory
Create a project root directory called python-flask-mysql-online-visitor-tracker as per your chosen location.
We may not mention the project’s root directory name in the subsequent sections but we will assume that we are creating files with respect to the project’s root directory.
Configure Flask
We here configure application through flask framework. Create a file called app.py with the below code.
Here we need to assign secret key otherwise session will not work in Python. The secret key, ideally, should be in encrypted format.
We have also configured the session timeout – 30 minutes because flask expires session once you close the browser unless you have a permanent session.
Basically a session will exist for 30 minutes and in this 30 minutes a user’s visits will be unique for a particular URL. So to set the flag is_unique in table vistis_log you need to work on this.
from flask import Flask
from datetime import timedelta
app = Flask(__name__)
app.secret_key = "secret key"
app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(minutes=30)Database Configuration
We create the below db.py Python script to setup the MySQL database configurations for connecting to database and storing visitor information into visits_log table.
We need to configure database connection with flask module and that’s why we have imported app module and setup the MySQL configuration with flask module.
Make sure to change the database configuration values according to your database setup.
from app import app
from flaskext.mysql import MySQL
mysql = MySQL()
# MySQL configurations
app.config['MYSQL_DATABASE_USER'] = 'root'
app.config['MYSQL_DATABASE_PASSWORD'] = 'root'
app.config['MYSQL_DATABASE_DB'] = 'roytuts'
app.config['MYSQL_DATABASE_HOST'] = 'localhost'
mysql.init_app(app)Configure Utility
I will create one general purpose utility Python script that will provide some utility functions to be used throughout the whole application wherever required.
We create a file called config.php with the following source code.
We check the key DNT in the requested http header and if its value is 1 then we don’t track visitors otherwise we will track the visitors if other conditions also meet.
We ignore IPs, for example, localhost, 127.0.0.1 etc because from localhost you won’t get real visitors.
I have created another function that will help us whether we want to consider a user is unique for a particular time period. For example, as I had said earlier that we want to consider a visitor unique if he/she is active for 30 minutes on the same browser.
from flask import request, session
DNT_TRACK = True #False
IGNORE_IPS = set(['127.0.0.1'])
def is_tracking_allowed():
#print(request.headers)
if 'DNT' in request.headers and request.headers['DNT'] == 1:
return False
if request.remote_addr in IGNORE_IPS:
return False
return True
def track_session():
if 'track_session' in session and session['track_session'] == True:
return True
else:
return FalseLog Visitors
Now we will log visitors into MySQL database table. We create a file called visitor.py.
We first check whether session tracking is allowed, i.e., if you are going to consider a visitor unique for a time period.
If session tracking is allowed then we check if the visitor is visiting the same page within 30 minutes. If visitor visits the same page then we don’t store the same information into database otherwise we store the details into database.
We have used session.modified = True otherwise you won’t be able to update the session key’s value.
If session tracking is not allowed then every time a visitor visits a page we store details into MySQL table. In this case your database table will grow unnecessarily.
import config
import pymysql
from db import mysql
from flask import request, session
def track_visitor():
if not config.is_tracking_allowed():
return
else:
ip_address = request.remote_addr
requested_url = request.url
referer_page = request.referrer
page_name = request.path
query_string = request.query_string
user_agent = request.user_agent.string
if config.track_session():
log_id = session['log_id'] if 'log_id' in session else 0
no_of_visits = session['no_of_visits']
current_page = request.url
previous_page = session['current_page'] if 'current_page' in session else ''
if previous_page != current_page:
log_visitor(ip_address, requested_url, referer_page, page_name, query_string, user_agent, no_of_visits)
else:
conn = None
cursor = None
session.modified = True
try:
conn = mysql.connect()
cursor = conn.cursor()
log_id = log_visitor(ip_address, requested_url, referer_page, page_name, query_string, user_agent)
#print('log_id', log_id)
if log_id > 0:
sql = 'select max(no_of_visits) as next from visits_log limit 1'
conn = mysql.connect()
cursor = conn.cursor(pymysql.cursors.DictCursor)
cursor.execute(sql)
row = cursor.fetchone()
count = 0
if row['next']:
count += 1
else:
count = 1
sql = 'UPDATE visits_log set no_of_visits = %s WHERE log_id = %s'
data = (count, log_id,)
cursor.execute(sql, data)
conn.commit()
session['track_session'] = True
session['no_of_visits'] = count
session['current_page'] = requested_url
else:
session['track_session'] = False
except Exception as e:
print(e)
session['track_session'] = False
finally:
cursor.close()
conn.close()
def log_visitor(ip_address, requested_url, referer_page, page_name, query_string, user_agent, no_of_visits=None):
sql = None
data = None
conn = None
cursor = None
log_id = 0
if no_of_visits == None:
sql = "INSERT INTO visits_log(no_of_visits, ip_address, requested_url, referer_page, page_name, query_string, user_agent) VALUES(%s, %s, %s, %s, %s, %s, %s)"
data = (no_of_visits, ip_address, requested_url, referer_page, page_name, query_string, user_agent,)
else:
sql = "INSERT INTO visits_log(ip_address, requested_url, referer_page, page_name, query_string, user_agent) VALUES(%s, %s, %s, %s, %s, %s)"
data = (ip_address, requested_url, referer_page, page_name, query_string, user_agent,)
try:
conn = mysql.connect()
cursor = conn.cursor()
cursor.execute(sql, data)
conn.commit()
log_id = cursor.lastrowid
return log_id
except Exception as e:
print(e)
finally:
cursor.close()
conn.close()Track Visitors
We have already created the required functionality to store information into database table. Now we will see how to track visitors when he/she visits your web pages.
Create a Python script main.py with the following source code.
import visitor
from app import app
from flask import jsonify
@app.before_request
def do_something_when_a_request_comes_in():
visitor.track_visitor()
@app.route('/')
def home():
return jsonify({'msg' : 'hello'})
if __name__ == "__main__":
app.run()In the above Python script we have used a decorator from Flask API @app.before_request which will trigger your function when a request comes in your web page. So you don’t need to write a hook but Flask API already provides for you to use on application.
We have another function that returns simple hello message when you hit the URL http://localhost:5000.
This is a simple application that will track your visitors. To track referrer page you need to have more navigation on your website.
Testing the Application
Now navigate to the project directory and execute the command python main.py, your server will be started on default port 5000.
If you want to change the port then you can change the line app.run() to app.run(port=5001), where 5001 is the new port.
Now you can hit the URL http://localhost:5000 on your browser and you will see visitors details are logged into MySQL table.
You can see I have accessed this application few times from different browsers such as Chrome, FireFox, Internet Explorer in the below output from visits_log table.
As I have said earlier this application has only one URL and that’s why you won’t see value for referrer page for most of entries. You should have few navigations on your website and users should navigate from one page to another page then only you get proper value for referrer page column.
Now you can create a UI or front-end where you can track your visitors’ statistics on a graph.
Source Code
Thanks for reading.
Very nice!
nice, thanks…