Organising, modularising, and distributing code

In this session, you will learn about modules including standard and user-defined modules.

You will be introduced to Pythonic way of defining functions, packages, keyword and variables and their scopes. You will learn how to build up a collection of multipurpose code blocks that you can use across many projects to minimize your coding workload.

Let’s start by first introducing some related concepts to Modular programming.

Global, Local Keywords and Variables

A variable is a label, or a name given to a certain location in memory.

Just like other programming languages, Python also uses variables to hold data. A typical Python variable also have a name and a type, but in Python, the data type does not need to be explicitly declared.

To create a new variable in Python, you simply use the assignment operator (=, a single equals sign) and assign the desired value to it.

Example: variable_name = 4

In the above example the variable_name held the value 4, an integer data type. Neither do we have to declare the variable without initializing it nor declare the variable data type.

There are some rules you need to follow when creating variable in Python:

• A variable may only contain letters (uppercase or lowercase), underscore _n character with numbers
• It may not start with a number
• It may not be a keyword (We will discuss what keywords are later)

Variable Scope and Lifetime

Scope of a name in programming generally defines the area of a program in which you can unambiguously access that name, such as variables, functions, objects, and so on. Only the code within the scope of a name will be able to see it and access it.

Scope is used by many programming languages to prevent name collisions and erratic behaviour.

Variable scope is the area of a program where a variable is accessible.

The duration for which the variable exists is its Lifetime.

Python has four different variable scopes:

1. Local
2. Enclosing
3. Global
4. Built in

Let’s look at each one separately.

Local Variable

When a variable is defined inside a function, its scope is LIMITED to the function. The variable is accessible from the point at which it is defined until the end of the function and exists for as long as the function is executing. Its value cannot be altered or even retrieved from outside the function.

If you run the code above, it prints the square area locally but when trying to access and then print the same variable from outside the function, it raises a NameError ( run the code here):

Output:

This is because square_area variable is "local" to the area()- thus, it cannot be reached from outside the function body.

Let’s see another example.

Consider the following nested function:

In the above example, we define another function inner() inside a function outer() to create a nested function. See the output below:

Output:

We got a name error when we attempted to access the second_num from the outer() function.

We were able to print the value of the same variable inside the inner() function because that is where it was defined. outer() function variables have a larger scope and can be accessed from the enclosed function inner().

Run the code below.

Global Variable

A global variable in Python means a variable with a scope throughout the program. As the name suggests, global variables can be accessed anywhere. Any variable defined outside a non-nested function is refered to as a global variable. The scope of such variable is anywhere within the program; thus, it can be used by any function.

Example:

Run the code below.

Output:

If you uncomment the line after the circumference(), where we try to manipulate the global variable side by multiplying it by 2, you will get an error. This is because, when a function tries to manipulate global variables, an UnboundLocal Error will be raised. We can avoid this error by using the global keyword.

Run this example

Built-in Scope

Build-in scope captures all special reserved keywords.

false	Class	for	lambda
none	continue	from	nonlocal
true	Def	global	not
and	Elif	if	or
as	Else	import	pass
assert	except	in	or
break	finally,	is	raise
yield	Why	try	return

The above keywords can be called from any place in our programme without first defining them. Simply said, keywords are special reserved words in Python. They are kept for specific purposes and cannot be used for any other purpose in the program.

Nonlocal Variables

Nonlocal variables are used in nested functions whose local scope is not defined.

When a nested function wants to share the local scope of parent functions, nonlocal keyword is used. Declaring parent function variables as global in such case does not work. When you declare a nested function variable as nonlocal, this implies that the variable can be neither in the local nor the global scope.

Let's see an example of how a nonlocal variable is used in Python.

if you run the above example, it will raise UnboundLocalError: local variable 'area' referenced before assignment.

Even though we declare area as global, we still got error when we tried to modify area in divide(). We can fix this, by declaring area as a nonlocal variable in inside the divide() function:

Run the code below to see how the example works.

See another example below:

We use nonlocal keywords to create nonlocal variables.

Output

In the above code, there is a nested inner() function. We use nonlocal keywords to create a nonlocal variable. The inner() function is defined in the scope of another function outer().

Modular programming is a way of organising programs as they become more complicated.

Modularity is a concept of creating many modules first, and then linking and combining them to form a complete system.

That is, the degree to which a piece of software/web application may be divided into smaller modules is called modularity. The idea behind modularity encourages re-usability and minimize duplication.

If you want to develop programs which are readable, reliable and maintainable without too much effort, you have to use some kind of modular software design. Another key objective of is to minimise dependence.

A modular system is constructed by building several components individually, and largely, independently. By combining them, the executable application will be produced.

Advantages in modularizing code

There are several advantages to modularizing code in a large application:

Reusability

This is one of the key benefits of modular programming. Functionality defined in a single module can be easily reused by other parts of the application. This eliminates the need to duplicate code.

Simplicity

A module typically focuses just on a small portion of the overall problem rather than the whole thing. You will have a more manageable problem domain if you are only working on just a single module. Development is now simpler and less prone to mistakes.

Maintainability

Typically, modules are created to establish logical boundaries across various problem domains.

It is less likely that changes to one module will have an influence on other parts of the programme if modules are constructed in a fashion that reduces interdependency. This increases the likelihood that a group of numerous programmers will be able to collaborate on a huge application.

You might even be able to alter a module without being familiar with the programme outside of it.

Scoping

Modules helps avoid collisions between identifiers in different areas of a program.

Functions, modules and packages

Functions, modules and packages are all constructs in Python that promote code modularization.

In Python, modules are simply Python files with the .py extension, which implement a set of functions. The file name is the module name without the .py suffix.

You can create a Python module, a source file that contains Python definitions and statements to perform some functions, and then import this module into your program so that you can use it. You can also import modules from other modules using the import command.

Let’s see a simple example:

In the above example, we have imported a math module.

The module math provides mathematical constants and functions, e.g. π (math.pi), the sine function (math.sin()) and the cosine function (math.cos()). As you can see in the example, we are able to access the pi attribute and cos() and sin() functions by putting "math." in front of the name.

The math module has so many other functions (see the documentation here). If we are only interested in the 3 objects cos, pi and sin objects, we can import only those by writing the import statement as:

In the above example we explicitly imported what we needed, and we do not have to call the function or attributes without prefixing math. again.

The import statement could be place anywhere in the code (as long as it is placed before we use the imported module)

We can also import everything from a module using the from format:

Avoid using import* to import modules.

Standard or Pre-defined Modules in Python

The math module we used in the above examples is one of the Python’s numerous built-in modules. Python built-in modules come as a package with its interpreter. Some common examples of built-in modules are:

• Random Module
• Sys Module
• OS Module
• Math Module
• Collections Module
• Statistics Module

There are other modules that get loaded automatically as the Python shell starts. This is why we can use functions like print() and input() for input-output operations, and also do data type conversion functions using the int(), complex(), float(); and so on. See various built-in modules in Python here.

User-defined Modules in Python

You can design your own classes and functions, then include them in modules. Latest update include hundreds of lines of code form your modules into any program by simply writing a simple import statement.

As mentioned before, in Python, creating a user-defined module is simple. Simply create you functions in a file and save the file with the .py extension. Any other program can import it using that module name.

Let’s consider an example:

Assuming you want a program that keeps track of various statistics about events that take place while the program is running.

You create a module called stats.py (just a Python file with some of the functions you needed to capture the even stats).

The file contains three independent functions:

1. init() function, which is used to initialize a global variable stats. stats is a container for storing the even stats in form of key:value pair, where key is the event and the value is the stats of the event.
2. event_occurred(event) function, which takes an event and keep counts of the number of times the event has occurred, then store the value in the stats dictionary.
3. The third function is the get_stats() function , which simply returns the sorted items in stats.

We can import stats as a module in another program. See example below importing stats module in another program file named main.py:

In the above example I have just imported my stats module by simply using the import keyword with the name of the file without the .py suffix.

You can run the program here and see how it works.

In the above example, we are not really interested in recording meals and snacks, this is just an example, but the important thing to notice here is how the stats module gets imported, and then how the various functions you defined within the stats.py file get used. For example, consider the following line of code:

Because the event_occurred() function is defined within the stats module, you need to include the name of the module whenever you refer to this function.

You can see that the import statement is used to load modules, and you can tell that the code is referring to something (like a function or class) that is defined within that module whenever you see the name of the module followed by a period.

Python packages enable you to group together several modules in the same way that Python modules let you organize your functions and classes into distinct Python source files. A directory with attributes is a Python package.

Consider the example of animal packages in Figure 1. The __init__.py file is called a package initialization file; the presence of this file tells the Python system that this directory contains a package. The package initialization file can also be used to make importing the package easier.

So how do we use the package? Just like we used the module name.

Using modules and Packages to organize a program

Using modules and packages, you can organise your Python code to reflect the logical structure of what your programme is attempting to do rather than merely distributing it across numerous source files and folders.

^{Figure 2: Using Module and Packages for your Web Application ( Source)}

Consider the scenario in Figure 2, where you are requested to develop a web application to store and report on university test results. Considering the business requirement, you come up with the following general framework for your application:

The program is broken into two main parts:

• a web interface, which interacts with the user (and with other computer programs via an API), and
• a backend, which handles the internal logic of storing information in a database, generating reports, and e-mailing results to students.

As shown in Figure 2, the web interface itself has been broken down into four parts.

• User authentication,
• View/Edit Exam Results,
• Generate report and
• API.

You are beginning to think about the functionality that each logical component of your programme will give as you consider them (i.e., each of the boxes in Figure 2).

You are already using a modular way of thinking as you complete this.

In fact, each of your application's logical parts can be implemented directly as a Python module or package. For instance, you might decide to divide your application into the web and backend major packages where you include modules such as

• User authentication,
• View/Edit Exam Results, reporting and API as the Web package, and then have your database, reportgenerator and emailer as modules in your backend package.

^{Figure 3: Implementing a Cache Using Module and Package}

Consider the illustration of a cache in Figure 3.

A cache operation is like your dictionary.

You can add a new entry based on a given key and you can retrieve the entry using this key.

There is, however, a key difference between a cache and a dictionary.

A dictionary has no limit when it comes to the number of entries you can add or contain.

A cache has a limit on the number of entries that it can contain.

This means that a dictionary can continue to grow forever, and possibly taking up all the computer’s memory if the program keeps running, whereas a cache will never use up all of the RAM because the number of entries are fixed.

Every time a new entry is added, an existing entry must be removed from the cache whenever it reaches its maximum size in order to prevent the cache from expanding further.

While there are several ways to choose which entry to remove, the most typical method is to remove the entry that hasn't been used for the longest amount of time, or the least recently used entry.

Computer programs frequently make use of caches. In fact, even if you haven't used a cache yet in a programme you've written, you've probably seen one before.

Has someone ever advised you to delete the cache from your web browser in order to fix a problem with it?

It's true that web browsers utilise a cache to store previously downloaded files, so they don't have to be fetched again, and deleting the cache is a frequent approach to restore a broken web browser.

Activity 1 - Write a Cache Module

Based on what you have learnt about module, and your understanding of what a cache is, write your own Python module to implement a cache. Consider the following requirements in your implementation:

Cache size limit (You may choose 50 entries as max)

• You will need an init() function to initialize the cache
• You will need a set(key,value) function to store an entry in the cache.
• You will need a get(key) function to retrieve an entry from the cache. If there is no entry for that key, this function should return None.
• You will also need a contains(key) function to check whether a given entry is in the cache.
• You will implement a size() function which returns the number of entries in the cache.
• Put your code inside a file named cache.py ( this will be your cache module)
• At the top of this file import datetime module and set MAX_CACHE_SIZE = 50.
• You will use the datetime built-in module to calculate the least recently used entry in the cache and define MAX_CACHE_SIZE to hold your maximum size value for your cache.
• In order to test the functionalities of your cache, you will need to create a simple test program that will use your cache module. Create a new Python file, call it test_cache.py add the code here.

Compare your code with the simple solution provided here.

How did you do? Try to study the code and modify it to suit your case.

Share your findings in the forum.