COMP519 Web Programming
Lecture 18: CGI Programming
Handouts
Ullrich Hustadt
Department of Computer Science
School of Electrical Engineering, Electronics, and Computer Science
University of Liverpool
Contents
1 CGI
Overview
CGI I/O
2 Python CGI Programs
Motivation
Python Primer
Example
Processing Environment Variables
Processing Form Data: The cgi Module
3 Revision and Further Reading
COMP519 Web Programming Lecture 18 Slide L18 1
CGI Overview
Common Gateway Interface CGI
The Common Gateway Interface (CGI) is a standard method
for web servers to use an external application, a CGI program,
to dynamically generate web pages
1 A web client generates a client request,
for example, from an HTML form, and sends it to a web server
2 The web server selects a CGI program to handle the request,
converts the client request to a CGI request, executes the program
3 The CGI program then processes the CGI request and
the server passes the program’s response back to the client
COMP519 Web Programming Lecture 18 Slide L18 2
CGI CGI I/O
Client requests
In the following we focus on client requests that are generated
using HTML forms
< html lang =" en - GB " >
<head >< title > My HTML Form </ title > </ head >
<body >
< form action =
" http :// student . csc . liv . ac . uk / cgi - bin / c giwrap / uh / pro cess "
method =" post ">
<label > Enter your user name :
< input type =" text " name =" username ">
</ label >< br >
<label > Enter your full name :
< input type =" text " name =" fullname ">
</ label >< br >
< input type =" submit " value =" Click for respon se ">
</ form >
</ body >
</ html >
COMP519 Web Programming Lecture 18 Slide L18 3
CGI CGI I/O
Client requests
In the following we focus on client requests that are generated
using HTML forms
< html lang =" en - GB" >
<head > < title > My H T ML Form </ title > </ head >
<body >
< form act ion =" h t tp :// stu d ent . csc . liv . ac . uk / cgi - bin / c g iwr ap / uh / pro ces s "
met h od =" po st " >
<label > En t er your user name :< i nput typ e =" text " na m e =" us e rn ame " > </ label >< br >
<label > En t er your full name :< i nput typ e =" text " na m e =" fu l ln ame " > </ label >< br >
< inpu t ty p e =" sub mit " v alue =" C lick for re spo nse ">
</form >
</body >
</html >
COMP519 Web Programming Lecture 18 Slide L18 4
CGI CGI I/O
Encoding of input data
Input data from an HTML form is sent URL-encoded as sequence of
key-value pairs: key1=value1&key2=value2&...
username = dave & fullname = David %20 Davidson
Keys may not be unique (for example, in the case of checkboxes)
Form controls without name do not appear
All characters except A-Z, a-z, 0-9, -, _, .,
(unreserved characters)
are encoded
ASCII characters that are not unreserved characters are represented
using ASCII codes (preceded by %)
A space is represented as %20 or +
+ is represented as %2B
% is represented as %25
username = cath & fullname = Catherin e +O %27 Donnell
COMP519 Web Programming Lecture 18 Slide L18 5
CGI CGI I/O
Request methods: GET versus POST
The two main request methods used with HTML forms
are GET and POST:
GET:
Form data is appended to the URI in the request
(limited to 1KB to 8KB characters depending on both browser and server)
<scheme> "://" <server-name> ":" <server-port>
<script-path> <extra-path> "?" <query-string>
Form data is accessed by the CGI program via environment variables,
name/value pairs that are part of the environment in which a
process/programs is run by the operating system
Requests remain in the browser history and can be bookmarked
Requests should not be used for sensitive data, e.g. passwords
GET / cgi - bin / cgiwr ap / uh / process ? use rnam e = dave & fulln ame = David
,+ Davidson HTTP /1.1
Host : s tudent . csc . liv . ac . uk
COMP519 Web Programming Lecture 18 Slide L18 6
CGI CGI I/O
Request methods: GET versus POST
The two main request methods used with HTML forms
are GET and POST:
POST:
Form data is appended to end of the request (after headers and blank line)
There is no limit on the length/size of the form data
Form data can be accessed by the CGI program via standard input
Form data is not necessarily URL-encoded (but URL-encoding is the default)
Requests do not remain in the browser history and cannot be bookmarked
Requests are suitable for the transfer of sensitive data, e.g. passwords
POST /cgi - bin / cgiwr ap / uh / process HTTP /1.1
Host : s tudent . csc . liv . ac . uk
username = dave & fu llna me = David + Da vids on
COMP519 Web Programming Lecture 18 Slide L18 7
CGI CGI I/O
Environment variables: GET
Env variable Meaning
QUERY_STRING The query information passed to the program
REQUEST_METHOD The request method that was used
PATH_INFO Extra path information passed to a CGI program
PATH_TRANSLATED Translation of PATH_INFO from virtual to physical
path
SCRIPT_NAME The relative virtual path of the CGI program
SCRIPT_FILENAME The physical path of the CGI program
GET http :// s tudent . csc . liv . ac . uk / cgi - bin / c g iwrap / uh / demo / more / dirs ?
us e rname = dave & ful l name = David + Davi d son
QUERY _ S T R I N G use r name = dave & ful l name = David + Davi d son
REQUEST_ME T H O D GET
PAT H _ INFO / more / dir s
PATH_TRANSLATED / users / www / extern a l / docs / more / dirs
SCRI P T _ N AME / cgi - bin / cgi wrap / uh / demo
SCRIPT_FILENAME / users / loco / uh / public _ h t m l / cgi - bin / demo
STDIN # em pty
COMP519 Web Programming Lecture 18 Slide L18 8
CGI CGI I/O
Environment variables: GET
Env variable Meaning
QUERY_STRING The query information passed to the program
REQUEST_METHOD The request method that was used
PATH_INFO Extra path information passed to a CGI program
PATH_TRANSLATED Translation of PATH_INFO from virtual to physical
path
SCRIPT_NAME The relative virtual path of the CGI program
SCRIPT_FILENAME The physical path of the CGI program
GET http :// s tudent . csc . liv . ac . uk / cgi - bin / c g iwrap / uh / proc e ss / more / dirs ?
us e rname =2%60 n+d %2 Bt +e+s %27 t& full n ame = Pet er + New ton
QUERY _ S T R I N G use r name =2 %60 n+d %2 Bt +e+s %27 t& full n ame = Pet er + New ton
REQUEST_ME T H O D GET
PAT H _ INFO / more / dir s
PATH_TRANSLATED / users / www / extern a l / docs / more / dirs
SCRI P T _ N AME / cgi - bin / cgi wrap / uh / p r ocess
SCRIPT_FILENAME / users / loco / uh / public _ h t m l / cgi - bin / p r ocess
STDIN # em pty
COMP519 Web Programming Lecture 18 Slide L18 9
CGI CGI I/O
Environment variables: POST
Env variable Meaning
QUERY_STRING The query information passed to the program
REQUEST_METHOD The request method that was used
SCRIPT_NAME The relative virtual path of the CGI program
SCRIPT_FILENAME The physical path of the CGI program
POST /cgi - bin / cg iwrap / uh / demo
Host : st udent . csc . liv . ac . uk
us e rname =2%60 n+d %2 Bt +e+s %27 t& full n ame = Pet er + New ton
QUERY _ S T R I N G # em pty
REQUEST_ME T H O D POST
SCRI P T _ N AME / cgi - bin / cgi wrap / uh / demo
SCRIPT_FILENAME / users / loco / uh / public _ h t m l / cgi - bin / demo
STDIN us e rname =2%60 n+d %2 Bt +e+s %27 t& full n ame = Pet er + New ton
COMP519 Web Programming Lecture 18 Slide L18 10
CGI CGI I/O
More environment variables
Env variable Meaning
HTTP_ACCEPT A list of the MIME types that the client can accept
HTTP_REFERER The URL of the document that the client points
to before accessing the CGI program
HTTP_USER_AGENT The browser the client is using to issue the request
REMOTE_ADDR The remote IP address of the user making the
request
REMOTE_HOST The remote hostname of the user making the re-
quest
SERVER_NAME The server’s hostname
SERVER_PORT The port number of the host on which the server
is running
SERVER_SOFTWARE The name and version of the server software
COMP519 Web Programming Lecture 18 Slide L18 11
Python CGI Programs Motivation
CGI programs and Python
CGI programs need to process input data from environment variables
and STDIN, depending on the request method
; preferably, the input data would be accessible by the program
in a uniform way
CGI programs need to process input data that is encoded
; preferably, the input data would be available in decoded form
CGI programs need to produce HTML markup/documents as output
; preferably, there would be an easy way to produce HTML markup
In Python, we can use
the cgi module to process inputs
the environ dictionary of the os module
to access environment variables
print statements to produce HTML markup
COMP519 Web Programming Lecture 18 Slide L18 12
Python CGI Programs Python Primer
Python: Basic Syntax
A Python program/script consists of one or more statements and
comments
One-line comments start with # and run to the end of the line
Multi-line comments simply consist of several one-line comments
Statements are delimited by newlines except where a newline is escaped
(by a backslash \)
On Unix/Linux systems, Python scripts begin with #! (called ‘hash
bang’ or ‘she bang’) and the location of the Python interpreter/compiler
#!/ usr / bin / pytho n3
# He llo Wor ld . py
# Our first Pytho n scri pt
print (" Hello World ")
COMP519 Web Programming Lecture 18 Slide L18 13
Python CGI Programs Python Primer
Python: Basic Syntax
Strictly speaking, in Python one assigns a (variable) name to a value,
not the other way round
; a (variable) name does not exist before the first assignment
But, the syntax for an assignment is the same as in JavaScript
age = 23
The first assignment to a variable defines that variable
Python supports the standard binary assignment operators
age += 10
Python uses static scoping
Blocks of statements, called suites are delimited with indentation
; each time the level of indentation is increased, a new block starts
; each time the level of indentation is decreased, a block has ended
A colon : separates the header of block from the rest of the suite
COMP519 Web Programming Lecture 18 Slide L18 14
Python CGI Programs Python Primer
Python: Type System
Python is a dynamically typed language:
a variable declaration does not include a type and
a variable can hold values of different types over time
x = " Hello "
x = 42
is a valid sequence of statements
Python is a (mostly) strongly typed language:
values are not automatically converted from unrelated types
y = " Hello " + 42
will cause an error in Python
However, quite a number of types are considered to be ‘related’
z = 42 and True # z --> True
will not cause an error in Python although a boolean operator is applied
to a number
COMP519 Web Programming Lecture 18 Slide L18 15
Python CGI Programs Python Primer
Python: Type System: Strings
A string literal is a sequence of characters surrounded by single-quotes,
double-quotes, or triple-quotes
'chars' single-quoted string
"chars" double-quoted string
'''chars''' triple-quoted string, can span several lines and
"""chars""" contain single and double quotes, but not at
the start or end
' ' ' This is a triple - quoted ' string ' co nta ini ng " quot es "
and s panning more than one line ' ' '
In all these forms \ acts as escape character
COMP519 Web Programming Lecture 18 Slide L18 16
Python CGI Programs Python Primer
Python: Type System: Dictionaries
A Python dictionary is a mapping of keys to values
(aka associative array or hash table)
A dictionary literal is a comma-separated list of key-value pairs
consisting of a key and a value separated by a colon :
surrounded by curly brackets
{ ' name ' : ' Dave ' , ' age ' : 23 , ' height ' : ' 185 cm ' }
Elements of any immutable type, e.g. strings, can be used as keys
The value associated with a specific key key in a dictionary dict can
be accessed (and modified) using
dict [ key ]
dct = { ' name ' : ' Dave ' , ' age ' : 23 , ' height ' : ' 185 cm ' }
print ( dct [ ' name ' ]) # prints ' Dave '
dct [' height ' ] = ' 190 cm ' # ' height ' now maps to ' 190 cm '
dct [' age ' ] += 1 # ' age ' now maps to 24
dct [' surname ' ] = ' Shield ' # ' surname ' maps to ' Shield '
COMP519 Web Programming Lecture 18 Slide L18 17
Python CGI Programs Python Primer
Python: Conditional Statements
Python conditional statements take the following form
if c o ndi tio n :
suite
elif co ndi tio n :
suite
else :
suite
The else-clause is optional and there can be at most one
The elif-clause is optional and there can be more than one
None of the suite blocks of statements can be empty
; the statement pass can be used as a ‘null statement’
if x == 0:
# We ' ll come up with a soluti on for x == 0 later
pass
else :
y = y / x
COMP519 Web Programming Lecture 18 Slide L18 18
Python CGI Programs Python Primer
Python: Functions
Functions are elements of type function and can be defined as follows:
def ide nti fie r ( param1 , param2 , ...):
doc str ing
suite
The function name identifier is case-sensitive
The function name must be followed by parentheses
A function has zero, one, or more parameters that are variables
Parameters are not typed
docstring is a string describing the function and will be returned by
help(identifier) or identifier.__doc__
suite is a non-empty sequence of statements
A function is called by using the function name followed by a list of
arguments in parentheses
... ide nti fie r ( arg1 , arg2 ,...) ... # Function call
COMP519 Web Programming Lecture 18 Slide L18 19
Python CGI Programs Python Primer
Python: Modules
A lot of functionality of Python is contained in modules
The statement
import mo d ule1 [ , mod ule2 [ ,... m o dule N ]]
makes all functions from modules module1, module2 available, but all
function names must be prefixed by the module name
import math
print math . fact orial (5)
The statement
from mo d ule1 impor t fun1 [ , fun2 [ ,... funN ]]
imports the named functions from module module1 and makes them
available without the need for a prefix
from math import facto rial
print fact ori al (5)
COMP519 Web Programming Lecture 18 Slide L18 20
Python CGI Programs Python Primer
Python: The re module
The re module of Python provides functions that use regular expressions
re.match(regexp,string [,flags])
attempts to find a match for regexp at the start of string
returns a match object when regexp is found and None otherwise
re.match(r'[mM][rs]',"Mr Dobbs") # MatchObject
re.match(r'[mM][rs]',"Hi Mr Dobbs") # None
re.search(regexp,string [,flags])
attempts to find a match for regexp anywhere in string
returns a match object when regexp is found and None otherwise
re.search(r'[mM][rs]',"Hi Mr Dobbs") # MatchObject
re.search(r'[mM][rs]',"Miss Dobbs") # None
COMP519 Web Programming Lecture 18 Slide L18 21
Python CGI Programs Example
Hello World CGI Program
#!/ usr / bin / pytho n3
print ( ' ' ' \
Content - type : text / html
<! D OCTYPE html >
< html lang =" en - GB " >
<head >
< meta charset =" utf -8" >
<title > Hello World </ title >
</ head >
<body >
Hello World
</ body >
</ html > ' ' ' )
COMP519 Web Programming Lecture 18 Slide L18 22
Python CGI Programs Example
User-Defined Functions for an HTML Wrapper
It makes sense to define functions that print out the
initial (up to body start) and final HTML markup (from body end)
#!/ usr / bin / pytho n3
# htmlUH . py placed in
ullrich / pub lic _ht ml / cgi - bin /
def start_html ( title ):
print ( ' ' ' \
Content - type : text / html
<! D OCTYPE html >
< html lang =" en - GB " >
<head >
< meta charset =" utf -8" >
< link rel =" stylesheet " type =" text / css "
href ="/
ullrich / COMP5 19 / e xamp les / table . css ">
<title > ' ' ' + titl e + ' ' ' </ title >
</ head > < body >' ' ' )
def e nd_html ():
print (' ' ' </ body > </ html > ' ' ' )
COMP519 Web Programming Lecture 18 Slide L18 23
Python CGI Programs Processing Environment Variables
Processing Environment Variables
The module os provides the environ dictionary
The environ dictionary maps a script’s environmental variables as keys
to the values of those variables
os . environ [ ' SERVER_AD DR ' ]
os . environ [ ' SERVER_NA ME ' ]
os . environ [ ' S ERVE R_PR OTOC OL ' ]
os . environ [ ' S ERVE R_SO FTWA RE ' ]
os . environ [ ' H TTP_ USER _AGE NT ' ]
os . environ [ ' REMOTE_AD DR ' ]
os . environ [ ' R EQUEST_ME THOD ' ]
os . environ [ ' R EQUEST_SC HEME ' ]
os . environ [ ' SCRIPT_URI ]
os . environ [ ' QUERY_S TRING ' ]
os . environ [ ' SCRIPT_NA ME ' ]
10. 128.0 .10 3
cgi . csc . liv . ac . uk
HTTP /1.1
Apache /2.4.3 4 ... Python
, /3.6 PHP /7.2.10
... Chrome /78.0 .39 04 .85 ...
212.1 59.11 6.53
GET
https
https :// cgi . csc . liv . ac . uk /
,cgi - bin / cgiwr ap / ullric h
,/ pyth on1 4A . py
/ LOCAL / www / html / ullric h /
, py thon 14A . py
COMP519 Web Programming Lecture 18 Slide L18 24
Python CGI Programs Processing Environment Variables
Processing Environment Variables: Example
#!/ usr / bin / python3
im port os , sys , re , codecs , lo cale
from h tmlU H impor t sta rt_html , e nd_htm l
# M ake sure output uses UTF -8 enco d ing
sys . stdout = codecs . ge t w riter (" utf - 8")( sys . stdout . detach ())
sta r t _ html (" Where are you com ing from ?")
us e r_ip = os . e nviro n [" REM O T E _ ADDR "]
print (' < div > Cl ients IP addr e ss : ' + us e r_ip + ' </div > ' )
if re . ma tch (r ' 138\.25 3\. ' , user _ ip ) :
print (' <p > Welcome , u niversity user ! </ p > ' )
print (' ' ' <p > Lots m ore c onten t only avail a b le
to un i v e rsity users </p > ' ' ' )
else :
print (' < div ><b > Sorry , plea se co me back \
when you are on a uni computer </ b > </ div > ' )
en d _html ()
COMP519 Web Programming Lecture 18 Slide L18 25
Python CGI Programs Processing Form Data: The cgi Module
Accessing and Processing Form Data
The module cgi provides methods to access the input data of
HTML forms in a two step process:
1 Create an instance of the FieldStorage class and assign it to a
variable
var i able = cgi . Fi e l d S torage ()
This reads the form data from standard input or the environment
variable QUERY_STRING
2 Then
variable['name'].value
variable.getvalue('name')
variable.getfirst('name', default=None)
variable.getlist('name')
return the value/values entered for the form control with name name
COMP519 Web Programming Lecture 18 Slide L18 26
Python CGI Programs Processing Form Data: The cgi Module
Processing Form Data: Example
We want to create a CGI script that both creates the following form
and validates its inputs
# The form
< form method =" POST ">
<label > User name : < i nput type =" text " name =" user " > </ label >
<label > Em ail address : < input type =" text " name =" email " > </ label >
< i nput type =" s ubmit " name =" submit " / >
</ form >
The form itself will be created by the following function
def pr in tF or m ():
pr int ( ' ' ' < form method =" POST ">
<label > User name : < i nput type =" text " name =" user " > </ label >
<label > Em ail address : < input type =" text " name =" email " > </ label >
< i nput type =" s ubmit " name =" submit " / >
</ form > ' ' ' )
COMP519 Web Programming Lecture 18 Slide L18 27
Python CGI Programs Processing Form Data: The cgi Module
Processing Form Data: Example
Validation will be done by the following two functions
def validate N a me ( field ):
if field == "" or field == None :
ret u rn " No username entered .\ n"
elif len ( field ) < 5:
ret u rn " Username too shor t .\ n "
elif re . sear c h (r ' [^ a - zA - Z0 -9_ -] ' , field ):
ret u rn " Inval i d c h a r ac te r in username .\ n"
else :
ret u rn ""
def valida t e E mail ( field ):
if field == "" or field == None :
ret u rn " No em ail entered .\ n"
elif not (( f ield . find (".") > 0) and ( f ield . find (" @ ") > 0)):
ret u rn " Sy m bol @ or . missing .\ n"
elif re . sear c h (r ' [^ a - zA - Z0 -9\. \ @\_ \-] ' , fiel d ):
ret u rn " Inval i d c h a r ac te r in email .\ n"
else :
ret u rn ""
COMP519 Web Programming Lecture 18 Slide L18 28
Python CGI Programs Processing Form Data: The cgi Module
Accessing and Processing Form Data: Example
#!/ usr / bin / python3
import cgi , os , sys , socket , re , codecs , locale
from htmlUH import start_html , end_html
sys . s t dout = c o d ecs . ge t w riter (" utf -8 ")( sys . stdout . deta c h ())
start_ h tml (" Form Proces s ing ")
inputs = cgi . F i eldSt o rage ()
if inputs . getvalue (' submit ' ):
err = val i dateN a me ( inputs . g e t value ( ' user ' ))
err = err + val i dateE m ail ( inputs . getvalue (' email ' ))
if err :
pri nt (' < div class =" e rror " > E rror : ' , err ,' Try aga in . </ div > ' )
printFo r m ()
else :
pri nt (' < div class =" ok " > All inputs are OK ! </ div >' )
# Some further proce s sing follows now
else :
printFo r m ()
end_html ()
COMP519 Web Programming Lecture 18 Slide L18 29
Revision and Further Reading
Revision and Further Reading
To get familiar with Python start at
Python Software Foundation: Our Documentation.
Python.org, 29 Oct 2019. https://www.python.org/doc/
[accessed 29 Oct 2019]
For information on modules for CGI programming in Python see
cgi: Common Gateway Interface support
https://docs.python.org/3/library/cgi.html
os: Miscellaneous operating system interfaces
https://docs.python.org/3/library/os.html
of Python Software Foundation: The Python Standard Library.
Python.org, 29 October 2019.
https://docs.python.org/3/library
[accessed 29 October 2019]
COMP519 Web Programming Lecture 18 Slide L18 30