COMP284 Scripting Languages
Lecture 2: CGI Programming
Handouts
Ullrich Hustadt
Department of Computer Science
School of Electrical Engineering, Electronics, and Computer Science
University of Liverpool
HTML Primer
Web-based applications: Interfaces
Web-based applications are client-server application where
•
the client software, consisting of user interface and client-side logic,
runs in a web browser
•
static aspects of a user interface are realised using HTML
; form elements with form controls allow a user to enter inputs
•
the client software incorporates user inputs into an HTTP request it
sends to a web server to retrieve information or to update parts of itself
•
the web server uses a program or programs, the server software, to
compute an HTTP response that it sends back to the web browser
Web Browser
Client
Software
Web
Server
Server
Software
HTTP Request
HTTP Response
COMP284 Scripting Languages Lecture 2 Slide L2 – 2
HTML Primer
HTML5 elements
•
An HTML element is an individual component of an HTML document.
•
Most elements consist of a start tag and a matching end tag,
with some content in between
•
The content of an element can consist of other element
; HTML elements can be nested
; HTML documents can be viewed as trees of HTML elements
•
The general form of a start tag
< tagName attrib1 =" value1 " ... a t t r i b N =" val u e N ">
where tagName is a non-empty sequence of alphanumeric ASCII chars,
attrib1,. . . ,attribN, N ≥ 0, are attributes and
value1,. . . ,valueN, N ≥ 0, are attribute values
attribute order is irrelevant
•
A end tag / closing tag takes the form
</ tagName >
COMP284 Scripting Languages Lecture 2 Slide L2 – 3
HTML Primer
HTML5 elements
•
So-called void elements only have a start tag
area base br col embed hr img
input keygen link meta param source track wbr
•
Since void elements have no content they cannot be nested
•
The start tags of void elements can be made self-closing by ending the
tag with /> instead of >, optionally preceded by a space
Examples:
<br > <br / > <br />
COMP284 Scripting Languages Lecture 2 Slide L2 – 4
HTML Primer
HTML5 documents
•
An HTML5 document has a very simple form:
It consists of a DOCTYPE-declaration and an html-element
<! DOCT YPE html >
ht ml -e le me nt
•
An html-element has the form
<html >
he ad -e le me nt
bo dy -e le me nt
</ html >
•
It is recommended that the start tag of an html-element specifies the
language used in the document
< html lang =' en-GB ' >
COMP284 Scripting Languages Lecture 2 Slide L2 – 5
HTML Primer Forms
Forms
•
A form is an HTML element that contains form controls, such as
text fields, buttons, checkboxes, range controls, or color pickers
•
Forms allow users to enter data which can then be sent to a web server
for processing
•
A form element has several (optional) attributes, including
•
action: URL to use for form submission
•
method: HTTP method to use for form submission (get or post)
•
enctype: encoding type to use for form submission
•
novalidate: form is not validated during submission
< form act ion = ' https :// sam . csc . liv .ac . uk / COMP / Calend ar . pl '
method =' post ' enctype =' text / plain ' >
</ form >
•
A form can be submitted and on submission the data entered via the
form controls will be send to the URL in action using the
HTTP request method in method with encoding type enctype
COMP284 Scripting Languages Lecture 2 Slide L2 – 6
HTML Primer Form Controls
Input
•
The input element represents a ‘field’ that allows the user to enter
data of a certain type
•
The input element is void element
•
The type attribute of an input element determine what type of data
can be entered and in what form
Value of Data Form
type type control
text Text with no line breaks Text field
number Floating-point number Text field or spinner
password Password Text field with obscured input
button Button
submit Initiates form submission Button
reset Resets form Button
< input type = ' text ' >
< input type = ' reset ' >
COMP284 Scripting Languages Lecture 2 Slide L2 – 7
HTML Primer Form Controls
Input
•
The input element represents a ‘field’ that allows the user to enter
data of a certain type
•
Common attributes of an input element include
•
id: unique id used to identify the element with the document
•
name: (unique) name used by the form processor to access input
•
autofocus: automatically focus on this form control when the page
is loaded
•
disabled: whether the form control is disabled
•
required: whether the form control is required to have
a non-empty value for form submission
•
Depending on the value of the type attribute, an input element will
have additional attributes that define its behaviour
< input type = ' number ' name =' studentid ' id =' sid '
min = ' 190000000 ' max = ' 999999999 ' >
COMP284 Scripting Languages Lecture 2 Slide L2 – 8
HTML Primer Form Controls
Label
•
In order for a form to be usable, each form control should be
accompanied by an indication of what it is for or how it should be used
Example: ‘Surname’ or ‘Enter your surname’ next to a field into which
the user is meant to enter their surname
•
A label element represents such an indication (a caption)
•
A label element can be associated with a specific form control either
using its for attribute or by putting the form control inside the label
element itself
< label for =' s1 ' > Surname : </ label >
< input type = ' text ' name =' surname ' id =' s1 ' >
<label > First name (s ):
< input type = ' text ' name =' first ' id = ' f1 ' ></ label >
COMP284 Scripting Languages Lecture 2 Slide L2 – 9
HTML Primer Form Controls
Input
< form action = ' pr o cess . py ' >
< l abel for =' s1 ' > S u r name :</ label >
< i nput type =' text ' name =' surname ' id = ' s1 ' >
<label > First name : < in put type = ' text ' name =' first ' id =' f1 ' >
</ label >
<label > St u d e n t I D : < i nput type =' number ' name = ' stu dent id ' id =' sid '
min = ' 190000000 ' max = ' 999999999 ' > </ label >
< i nput type =" submit " >
</form >
•
On submission, the web browser will construct pairs name=value where
name is the value of the name attribute of one of the form controls and
value is the input by the user for that form control
•
A string composed of those pairs will be send to process.py
Example:
Peters, Amy Lee, and 201612345 are entered into the three fields
; surname=Peters, first=Amy+Lee, and studentid=201612345
are sent to process.py
COMP284 Scripting Languages Lecture 2 Slide L2 – 10
HTML Primer Form Controls
Input
< form action = ' pr o cess . py ' >
< l abel for =' s1 ' > S u r name :</ label >
< i nput type = ' text ' n ame = ' surname ' id =' s1 ' >
<label > First name : < in put type = ' text ' name =' first ' id =' f1 ' >
</ label >
<label > St u d e n t I D : < i nput type =' number ' name = ' stu dent id ' id =' sid '
min = ' 190000000 ' max = ' 999999999 ' > </ label >
< i nput type =' submit ' >
</form >
•
This form can be submitted by either activating the ‘Submit’ button or
by pressing the return key within one of the three input fields
•
Form submission is possible even with all input fields being empty,
even the one for the StudentID
; we need to add the required attribute to prevent empty inputs
•
The StudentID field accepts floating point numbers between the
specified min and max values
; 1.9e8 is accepted and studentid=1.9e8 send to process.py
COMP284 Scripting Languages Lecture 2 Slide L2 – 11
HTML Primer Form Controls
Select
•
A select element represents a drop-down menu with pre-defined
options between which the user must select
•
The content of a select element consists of a list of option elements
that represent those options
•
A select-element can have several attributes, including
•
id: unique id used to identify the element with the document
•
name: (unique) name used by the form processor to access input
•
multiple: allow multiple options to be selected
•
required: an option must be selected that has a non-empty value
•
disabled: the current selection can not be changed
•
size: number of options to show to the user
•
An option element can have several attributes, including
•
label: the label used in the drop-down menu
•
value: the value returned for the option
•
selected: the option is selected by default
•
disabled: the option is shown but cannot be selected
COMP284 Scripting Languages Lecture 2 Slide L2 – 12
HTML Primer Form Controls
Select
•
A select element represents a drop-down menu with pre-defined
options between which the user must select
(often preferred over a radio button group)
•
The content of a select element consists of a list of option elements
that represent those options
< label for =' title ' > Select a module : </ label >
< select name =' module ' >
< option value =' COMP201 ' > COMP2 01 : So ftwar e En gi ne er in g I
</ option >
< option value =' COMP207 ' > COMP2 07 : Da tabas e De ve lo pm en t
</ option >
< option value =' COMP211 ' > COMP2 11 : Co mpute r Netwo rk s
</ option >
</ select >
COMP284 Scripting Languages Lecture 2 Slide L2 – 13
HTML Primer Form Controls
Select
< form act ion = ' process .py ' >
< label for =' module ' > Select a module : </ label >
< select name =' module ">
< option value =' COMP201 ' > COMP2 01 : So ftwar e En gi ne er in g I
</ option >
< option value =' COMP207 ' > COMP2 07 : Da tabas e De ve lo pm en t
</ option >
< option value =' COMP211 ' > COMP2 11 : Co mpute r Netwo rk s
</ option >
</ select >
< input type = ' submit ' >
</ form >
•
By default, the first option is selected
•
If the selection is not changed and the user activates the submit button,
then module=COMP201 is sent to process.py
•
In general, the value associated with the selected option will be send
COMP284 Scripting Languages Lecture 2 Slide L2 – 14
HTML Primer Form Controls
Select
< form act ion = ' process .py ' >
< label for =' module ' > Select a module : </ label >
< select name =' module ' >
< option value =' COMP201 ' > COMP2 01 : So ftwar e En gi ne er in g I
</ option >
< option value =' COMP207 ' selected >
COM P20 7 : Dat ab ase Development </ option >
< option value =' COMP211 ' disabled >
COM P21 1 : Com pu ter Net works
</ option >
</ select >
< input type = ' submit ' >
</ form >
•
Adding the attribute selected to an option,
makes it the option that is selected by default
•
Adding the attribute disabled to an option,
makes it impossible to select that option
•
If the selection is not changed and the user activates the submit button,
then module=COMP207 is sent to process.py
COMP284 Scripting Languages Lecture 2 Slide L2 – 15
HTML Primer Form Controls
Select
< form act ion = ' process .py ' >
< select name =" module " required >
< option value =' ' > Select a module </ option >
< option value =' COMP201 ' > COMP2 01 : So ftwar e En gi ne er in g I
</ option >
< option value =' COMP207 ' > COMP2 07 : Da tabas e De ve lo pm en t
</ option >
< option value =' COMP211 ' > COMP2 11 : Co mpute r Netwo rk s
</ option >
</ select >
< input type = ' submit ' >
</ form >
•
That an option with a non-empty value is pre-selected is often not
desirable
; the user does not need to make a conscious choice
•
Adding a default option with empty value and adding the attribute
required to the select element forces the user to make a conscious
choice
COMP284 Scripting Languages Lecture 2 Slide L2 – 16
CGI Overview
Common Gateway Interface — CGI
The Common Gateway Interface (CGI) is a standard method
for web servers to use an external application, a CGI program,
to dynamically generate web pages
1 A web client generates an HTTP request,
for example, from an HTML form, and sends it to a web server
2 The web server selects a CGI program to handle the request,
converts the HTTP request to a CGI request, executes the program
3 The CGI program then processes the CGI request and
the server passes the program’s response back to the client
COMP284 Scripting Languages Lecture 2 Slide L2 – 17
CGI CGI I/O
HTTP Requests and HTML Forms
In the following we focus on HTTP requests that are generated
using HTML forms
< html lang =' en - GB ' >
<head > < title > Usern ame Generator </ title > </ head >
<body >
< form act ion = ' generate . py ' method =' post ' >
<label > Enter your full name :
< input type = ' text ' name =' fullname ' >
</ label >
<label > Select your year of reg istration :
< select name =' year ' >
<option >- - - - </ option >
<option >2010 </ option > ... < option >2020 </ option >
</ select >
</ label >
< input type = ' submit ' name =' submit ' value =' Generate ' >
</ form >
</ body > </ html >
COMP284 Scripting Languages Lecture 2 Slide L2 – 18
CGI CGI I/O
HTTP Requests and HTML Forms
In the following we focus on client requests that are generated
using HTML forms
< html lang = ' en - GB ' >
<head > < title > Username G enerator </ title > </ head >
<body >
< form acti o n =' g e n e r a t e .py ' method =' post ' >
<label > Enter your full name :
< i nput type = ' text ' name =' ful lname ' >
</ label >
<label > Se l ect your year of regis t r a tion :
< s e l ect name =' year ' >
<option > - - - - </ option >
<option >2010 </ option > ... < option >2020 </ option >
</ select >
</ label >
< i nput type = ' submit ' name = ' submit ' val ue =' Generate ' >
</form >
</body > </ html >
COMP284 Scripting Languages Lecture 2 Slide L2 – 19
CGI CGI I/O
HTTP Requests: Encoding of form data
•
Input data from an HTML form is sent URL-encoded as sequence of
name-value pairs: name1=value1&name2=value2&...
ful lname = David %20 Davi dson & year =2 018& submit = Genera te
•
Names may not be unique (for example, in the case of checkboxes)
•
Form controls without name do not appear
•
All characters except A-Z, a-z, 0-9, -, _, .,
∼
(unreserved characters)
are encoded
•
ASCII characters that are not unreserved characters are represented
using ASCII codes (preceded by %)
• A space is represented as %20 or + • % is represented as %25
• + is represented as %2B • ' is represented as %27
COMP284 Scripting Languages Lecture 2 Slide L2 – 20
CGI CGI I/O
HTTP Requests: GET versus POST: Client
The two main request methods used with HTML forms
are GET and POST:
•
GET:
The browser appends form data to the URI in the request
<scheme> "://" <server-name> ":" <server-port>
<script-path> <extra-path> "?" <query-string>
•
Limited to 1KB to 8KB characters depending on both browser and server
•
Requests remain in the browser history and can be bookmarked
•
Requests should not be used for sensitive data, e.g. passwords
•
POST:
The browser appends form data to the body of the request
•
No limit on the length/size of the form data
•
Requests do not remain in the browser history and cannot be bookmarked
•
Requests are suitable for the transfer of sensitive data, e.g. passwords
COMP284 Scripting Languages Lecture 2 Slide L2 – 21
CGI CGI I/O
HTTP Requests: GET versus POST: Server
The two main request methods used with HTML forms
are GET and POST:
•
GET:
•
Form data is provided to CGI programs via the QUERY_STRING
environment variable,
a name/value pair that is part of the environment in which a
process/programs is run by the operating system
•
Standard input is empty
•
POST:
•
The QUERY_STRING environment variable is empty
•
Form data is provided to CGI programs via standard input
COMP284 Scripting Languages Lecture 2 Slide L2 – 22
CGI CGI I/O
HTTP Requests: GET versus POST: Server
The two main request methods used with HTML forms
are GET and POST:
•
Both
Env variable Meaning
REQUEST_METHOD The request method that was used
HTTP_USER_AGENT The browser issuing the request
HTTP_REFERER The URL of the document that the browser
points to before accessing the CGI program
HTTP_ACCEPT A list of the MIME types that the browser can
accept
REMOTE_ADDR The remote IP address of the device from
which the request came
COMP284 Scripting Languages Lecture 2 Slide L2 – 23
Python CGI Programs Motivation
CGI programs and Python
•
CGI programs need to process form data from environment variables
and STDIN, depending on the request method
; preferably, the form data would be accessible by the program
in a uniform way
•
CGI programs need to process form data that is encoded
; preferably, the form data would be available in decoded form
•
CGI programs need to produce HTML markup/documents as output
; preferably, there would be an easy way to produce HTML markup
In Python, we can use
•
the environ dictionary of the os module
to access environment variables
•
the cgi module to process form data
•
print statements to produce HTML markup
COMP284 Scripting Languages Lecture 2 Slide L2 – 24
Python CGI Programs Accessing Environment Variables: The os Module
Accessing Environment Variables
•
The module os provides the environ dictionary
•
The environ dictionary maps a script’s environmental variables as keys
to the values of those variables
os . environ [ ' REQUES T_M ETHOD ' ]
os . environ [ ' HTT P_USER_A GENT ' ]
os . environ [ ' HTTP _REFERER ' ]
os . environ [ ' HTTP_ACCEPT ' ]
os . environ [ ' REMOTE_ADDR ' ]
os . environ [ ' QUER Y_STRING ' ]
GET
... Chr ome /78 .0 .3 904.85 ...
https :// stud ent . csc . liv . ac .
,→uk / cgi - bin / c giwrap /uh /
,→ge nerate . py
text / html , app li ca ti on / xhtml +
,→xml , a pp li ca ti on / xml ;q
,→=0.9 , image / webp , image /
,→apng ,*/*; q =0.8 ,
,→application / signed -
,→ex change ;v = b3 ; q =0.9
21 2. 14 9.127 .4 3
empt y
COMP284 Scripting Languages Lecture 2 Slide L2 – 25
Python CGI Programs Accessing Form Data: The cgi Module
Accessing Form Data
The module cgi provides methods to access the input data of
HTML forms in a two step process:
1 Create an instance of the FieldStorage class and assign it to a
variable
variable = cgi . Fie ld St or age ()
This reads the form data from standard input or the environment
variable QUERY_STRING
2 Then
•
variable['name'].value
•
variable.getvalue('name')
•
variable.getfirst('name', default=None)
•
variable.getlist('name')
return the value/values entered for the form control with name name
COMP284 Scripting Languages Lecture 2 Slide L2 – 26
Python CGI Programs Example
Example
We want to develop a CGI program for a Username Generator with the
following functionality
•
Access is restricted to IP addresses starting with 138.253
(University of Liverpool IP addresses)
•
The program prints out an HTML document with an HTML form that
allows a user to enter the data required for our program
•
The drop-down menu should cover the previous nine years plus the current
year
•
On submission of the completed form, the program
•
generates a username based on the data entered into that form
•
prints out an HTML document with that username
COMP284 Scripting Languages Lecture 2 Slide L2 – 27
Python CGI Programs Example
Example: HTML Wrapper
It makes sense to define functions that print out the
initial (up to body start) and final HTML markup (from body end)
# htmlUH . py
def st ar tHTML ( ti tle ):
print (' ' ' \
Content - type : text / html
<! DOCT YPE html >
< html lang =' en - GB ' >
<head >
< meta charset =' utf -8 ' >
< link rel =' stylesheet ' type =' text /css '
href =' form .css ' >
<title > ' ' ' + title + ' ' ' </ title >
</ head >< body > ' ' ' )
def endHT ML ():
print ( ' ' ' </ body > </ html > ' ' ' )
COMP284 Scripting Languages Lecture 2 Slide L2 – 28
Python CGI Programs Example
Example: HTML Form
def pr in tForm ():
print (' ' ' \
< form act ion = ' generate . py ' method =' post ' >
<label > Enter your full name :
< input type = ' text ' name =' fullname ' >
</ label >
<label > Select your year of reg istration :
< select name =' year ' >
<option >- - - - </ option >' ' ' )
now = d ateti me . dat etime . now ()
for year in range ( now .year -9 , now . year +1):
print ( ' < option > ' , year , ' </ option > ' )
print (' ' ' \
</ select >
</ label >
< input type = ' submit ' name =' submit ' value =' Generate ' >
</ form > ' ' ' )
COMP284 Scripting Languages Lecture 2 Slide L2 – 29
Python CGI Programs Example
Example: Username
Simple function for generating a username from the full name of a user
and the year the user has registered
def genUsername ( name , year ):
# first le tter of given name / names
return (" sg " + name . split () [0][0]. lower ()
# first three letters of s urname
+ name . split ()[ -1][:3]. lower ()
# last two digits of year
+ str ( year )[2:4])
COMP284 Scripting Languages Lecture 2 Slide L2 – 30
Python CGI Programs Example
Example: Putting everything together
#!/ usr / bin / pyth o n3
# ge nerate . py
import os , sys , codecs , datetime , locale
from htmlUH impor t start_html , end_ html
star t _htm l (" Gener a te User n ame ")
if ( os . environ [" R EMOT E_AD DR " ] [0:7] != "1 3 8.25 3"):
pr i n t (' < div > <b > Sorry , p l e ase come back \
when you are on a uni computer </ b > </ div > ' )
else :
inputs = cgi . Fie ldSt orag e ()
if input s . g e tvalue (' submit ' ):
# Form has been com plete d and su b mitt e d
name = input s [' fullname ' ]. value
year = input s [' year ' ]. value
userna me = ge nUse rnam e ( name , year )
pri n t (' < div > The user name for ' + name + ' re gist e red in '
+ year + ' is ' + u sernam e + ' </ div >)
else :
# Show user the form
print Form ()
end_ht ml ()
COMP284 Scripting Languages Lecture 2 Slide L2 – 31
Python CGI Programs Example
Example: Making it work
•
To make the CGI script generate.py
accessible via our departmental web server it must be placed in a
directory
∼
user/public_html/cgi-bin/
•
The directories
∼
user, public_html and cgi-bin must by
world-executable
•
The script must be executable by the user
•
It can then be accessed via
https://student.csc.liv.ac.uk/cgi-bin/cgiwrap/
∼
user/generate.py
•
If we open that URL in a browser we see
•
We can then complete the form and activate ‘Generate’
•
The browser will then show
COMP284 Scripting Languages Lecture 2 Slide L2 – 32
Revision and Further Reading
Revision and Further Reading
•
To get familiar with HTML5, read Chapters 4 to 9 of
J. Niederst Robbins:
Learning Web Design: A Beginner’s Guide to HTML, CSS,
JavaScript, and Web Graphics (5th ed).
O’Reilly, 2018.
E-book https://library.liv.ac.uk/record=b5647021
•
For information on modules for CGI programming in Python see
•
cgi: Common Gateway Interface support
https://docs.python.org/3/library/cgi.html
•
os: Miscellaneous operating system interfaces
https://docs.python.org/3/library/os.html
of Python Software Foundation: The Python Standard Library.
Python.org, 23 December 2019.
https://docs.python.org/3/library
[accessed 24 December 2019]
COMP284 Scripting Languages Lecture 2 Slide L2 – 33
