Introduction - The Need for Fast Web Applications

If you are a web developer building public web applications, it is very important to consider site performance as one of the primary requirements. A faster web application provides better customer experience and can lead to higher revenues. Site performance is also important for better visibility in Google search index. Google considers site performance as one of the parameters in deciding how high the web content should be in the search results. Hence performance is super important in publicly available websites such as eCommerce storefronts. In this article, I discuss some quick tips you can use while developing your web application for better performance.

Tips to Improve Speed of Web Applications

Here are some tips to improve the speed of your web application,

Tip 1: Use a fast hosting provider which supports hosting your site closer to your target audience. There is a wide range of hosting providers available, but if you use a shared hosting provider, you may not have the desired site speed. The best ones would be the public cloud providers such as Microsoft Azure, Google Cloud Platform or Amazon Web Services who provide cheap and fast servers. If your budget is low, look for a faster shared CPU provider such as Linode.

Tip 2: Use a content delivery network(CDN) for your static files such as images, CSS and JavaScript that rarely changes. This ensures very low latency for such resources since there are a large number of local servers closer to the user that can cache this content.

Tip 3: Minimise the size of files that are sent to the user. This can be achieved by removing anything that is not needed in specific pages/screens and by ensuring compression of the data and files sent to the browser. Most modern browsers natively decompress GZIP files before rendering. For web, the best image format for reduced size is WebP and you can either convert your images to this format or use front-end frameworks(NextJS etc.) that can automatically do this along with progressive enhancement of the image.

Tip 4: Use browser caching where possible using HTTP headers. This ensures that rarely changed files are loaded from the local browser and it will also reduce bandwidth requirements and load on your application server.

Tip 5: Use lazy loading techniques wherever possible. This approach only loads the content that is immediately required and then loads additional content when requested. In some Web applications, this may require doing a balancing act between Web performance and user requirements. This technique can also be used even in cases where full data is needed without intervention. For example, only loading the content as the user scrolls down.

Tip 6: Minimise blocking JavaScript in web pages. Use asynchronous JavaScript to avoid blocking the loading of other elements on the page.

Tip 7: Use server-side rendering. A lot of modern front-end applications are rendered using JavaScript and API response. This leads to highly composable user interfaces, but also leads to low performance since JavaScript needs to be downloaded and then executed on the client machine before content is visible to the user.

By pre-rendering the resulting HTML on the server side, it is possible to substantially improve the performance and responsiveness of a Web application. This is very useful in use cases such as eCommerce journeys where most of the product details page content is the same for all users. In such cases, a hybrid approach can also be taken by enhancing the pre-rendered content with dynamic content on the client side.

Tip 8: Optimise backend API performance - It is important to have all the real-time APIs used in a Web application as fast as possible. This requires careful design of the API, use of appropriate data structures and techniques such as query optimisations or data caching to ensure a fast API. It is also important to verify that API performance is not degraded when real data is loaded in the system or when multiple users are accessing the API. Use volume testing and load testing techniques to ensure API performance.

Tip 9: Minimise multiple HTTP calls wherever possible. Every additional API request requires HTTP handshaking time. Carefully design your API even sometimes creating composite ones to minimise a large number of HTTP calls before the web page is loaded.

Tip 10: Use an appropriate front-end framework. For consumer facing web applications, you can now pick from a large list of front-end frameworks with focus on rendering performance. Some of the popular technologies/frameworks include ReactJS, NextJS and GatsbyJS. These frameworks provide out of the box image optimizations, ability to pre-render a full web page or parts of a web page as needed for the web application. Choose your front-end framework wisely!

Tip 11: Use data caching on the backend. This is closely related to backend API performance. Caching API output or database query results in memory can substantially improve the web application performance. However, you need to be careful that users are not shown stale data.

Tip 12: Use a frontend performance monitoring tool such as Google pagespeed insights. These tools can provide specific inputs or bottleneck areas which you can then prioritise and optimise. Encourage your front-end teams to use browser developer tools and Google pagespeed insights to optimise the application screens during the development itself. Remember any optimizations after site build can be 10x costly than doing it during the development phase!

Tip 13: Use a backend performance profiling tool such as VisualVM for Java. Use an appropriate profiling tool for your backend language to identify possible bottlenecks and optimization opportunities. For example, VisualVM or JProfiler for Java can provide granular performance metrics of your code. Nowadays cloud platforms also provide out of the box profiling tools for cloud native services and language specific workloads (Java, JavaScript, Python etc.). Use them!

Tip 14: Minimise number of page redirects. Use a mobile first page design approach for our application so that there is no longer any need for page redirects. Use browser developer tools or command-line tools to verify that there are no unnecessary redirects for pages.

Tip 15: Minify your site scripts. Most front-end frameworks such as NextJS provide script minimization automatically. Ensure that you also use only the required dependencies in front-end apps.

Tip 16: Avoid third party domain lookups for scripts. You may be using plenty of third party scripts for your application. Usually these scripts are available on third party CDNs and if people are using them in other apps, potentially your site can use the cached one. However, I recommend keeping all scripts including third party scripts on your own server.

Tip 17: Ensure web application has no broken links. Broken links can lead to poor user experience and non-existent file imports unnecessarily increase load on the server. Use browser developer tools to identify and eliminate broken urls or links.

Tip 18: Reduce third party scripts. Many complex web applications contain a large number of third party scripts for various application features. For example, a typical application may include loyalty scripts, analytics scripts, social media scripts etc. Minimise the use of these unless there is a clear business benefit for each of them. Also ensure they are loaded only on pages that require them.

Metrics Used by Google Page Insights to Quantify Site Performance

Following are the metrics used by Google to evaluate web page performance. Use of the above tips appropriately will improve these metrics. Please also note that when Google gives an overall score, the following metrics are given different weights. For example, total blocking time has a much bigger weightage than speed index.

First Contentful Paint(FCP) - This is the time taken for the user to see any content on the browser screen after navigating to the web page. Seeing any content assures the user that something is happening and he can access the full content soon. The lower this time, the better and we should aim to keep this number less than 2 seconds.

Speed Index(SI) - This time metric that shows you how quickly the contents of a page are visibly populated. It is the average time at which visible parts of the page are displayed. It is important to note that this is a mathematically computed time value which represents how the user perceives full visible rendering of the page. Note that since this is measured in terms of the visible view, it is viewport dependent. Try to keep this value under 3.5 seconds.

Largest Contentful Paint(LCP) - This metric reports the render time of the largest image or text block visible within the viewport, relative to when the page first started loading. It is used to measure the user perception on the site speed. Obviously using the load time of the largest image or text block may not always accurately reflect user perception. But Google claims that their research shows for most use cases it does work. Try to keep this value under 2.5 seconds.

Time to Interactive(TTI) - This measures how fast the webpage is ready to accept user input or ready to respond to events. It is measured by looking at when the site is interactive with long running JavaScript events or loading events are finished. Hence this measure can identify issues such as long running JavaScript or large JavaScript file sizes. Try to keep this value under 5 seconds.

Total Blocking Time(TBT) - This is closely related to TTI, but measures the actual blocking time between FCP and TTI. A large value shows the severity of the blocking nature of the website. Try to keep this value under 200 milliseconds.

Cumulative Layout Shift(CLS) - This metric measures the visual stability of the web page as perceived by the user. The intent is to prevent unexpected layout changes which can be confusing to the user and can lead to unintended clicks etc. The recommended value is 0.1 or less. Practically CLS can be minimised by specifying size attributes for media and by ensuring that content is not abruptly introduced above existing content.

Note that Google Page Insights also provides a set of tips in the form of optimisation opportunities. See this page for more details.

Adding Web Performance Requirements in Your Engineering Processes

Please note that for each of the tips above, there are multiple optimization techniques that can be used. I suggest creating these tips as high level technical stories and then adding individual optimization tasks as sub-stories as part of your agile project (for example, in JIRA).

Before starting development on a Web application, it is very important to select appropriate tech stack based on the use case scenario. For example, if you are building an eCommerce storefront where the site performance is super important both for customer experience and SEO, choose a front-end framework such as NextJS or GatsbyJS that supports server side rendering and Google page insight based optimisations. This goes a long way in implementing the tips mentioned above.

Also note that web application performance improvement should be part of the application development cycle and a continuous activity. Initially look for the activities that take only 20% of your effort in achieving 80% improvement. Then gradually address the remaining tips. Also add measurement of site performance as part of your continuous integration and continuous deployment pipeline so that any future changes are immediately optimised if a performance hit is detected. Cloud providers such as Microsoft Azure are now providing cloud native load testing services which you can now integrate within your software release process.

Are you frustrated with increasing Azure cloud costs? For most Azure workloads, the biggest factor affecting the cost is the Azure VM resource. However, it is possible to reduce VM cost by adopting a set of strategies in provisioning and using the VM resources. This article provides 10 tips you can use today to reduce your Azure VM cost.

Overview Azure VM Cost Reduction Tips

Here are the top 10 tips for reducing Azure VM cost,

1. Select the Right VM Type for your Workload
2. Select the Right VM Size for your Workload
3. Use a Low Cost Region for your VMs
4. Use Azure Spot Instance
5. Use Azure Hybrid Benefit
6. Use Reserved Azure VM Instance
7. Reduce Cost by Using a Smaller VM Disk
8. Use Auto Shutdown Feature for VM
9. Use Azure Advisor to Monitor Cost Reduction Opportunities
10. Analyse your VM Cost Breakdown for Optimising Cost

Details on each of the Azure VM cost reduction tips is given below,

1. Select the Right VM Type for your Workload

There are 3 key decisions you need to make before you decide to create an Azure VM. This includes the type of VM you need, the type of OS you want to run on the VM and the region where you want to run your VM. Note that the available options for VMs change based on your region and OS requirements. Use the Azure VM selector tool to narrow down your VM choice.

After specifying the OS and your hosting region requirement (if any), the most important thing to decide is the VM type based on your workload requirement. At a high level, VM types for the following common workload types are available,

• General-purpose workloads
• Compute-intensive workloads
• Storage-intensive workloads
• Memory-intensive workloads
• GPU-enabled workloads
• High-performance computing (HPC) workloads

Once the VM type is selected, you can fine tune the VM selection by using the following requirement filters,

• The CPU brand you need - AMD, Intel or Ampere Altra arm
• Type of CPU usage - continuous or burst usage
• Do you need a GPU in the VM?
• Do you need a storage or data disk attached?

According to this Microsoft blog, by using the newer Ampere Altra arm CPU, you can get up to 50% better price-performance for your cloud workloads!

Here is a sample VM type selection - A memory optimised VM instance in the West Europe region running CentOS 7.9 under AMD CPU with continuous usage and no data disk.

2. Select the Right VM Size for your Workload

After selecting the Azure VM type, the next major decision affecting the cost is the size of the VM. This is a combination of vCPU cores, memory, I/O speed and disk specifications. It is important to choose the lowest configuration that meets your workload requirements. This typically involves starting with a higher VM size, monitoring the key metrics of the provisioned VM and then optimising for the right VM size. Another approach will be to start with the smallest size and then upgrade as per workload requirement.

Here is a sample VM size selection - A memory optimised VM instance in the West Europe region running CentOS 7.9.  Since the workload require at least 32GB, the initial VM model selected is E4bds_V5.

3. Use a Low Cost Region for your VMs

For most workloads, you will require a specific region for your VMs. However sometimes you may not need a specific region. In such cases, ensure that you provision your VM in a region which offers the lowest cost for your VM. For example, the East US region usually has 25% cheaper VM prices than the West Europe region. See this page for a region level price comparison of Azure VMs.

It is very important in this case to ensure that you are not doing any large scale data transfers between your VM in a separate region and your other resources in another region as it will lead to bandwidth cost and slow performance. Also please be aware of compliance and legal requirements when it comes to hosting your VM workloads in a specific region.

4. Use Azure Spot Instance

When you provision a VM, you have an option to specify it as a Spot VM. Spot VMs are up to 85% cheaper than usual VM instances since Azure uses its free available data centre capacity to power them. However, the catch is that your VM can be shutdown abruptly if Azure runs out of free VM instance pool. Hence this is only useful if you want to run some of your workloads using random available instances.

Clearly spot instances are not useful for always on services such as database servers or web applications. However, it is a good choice if you want some kind of background processing that can take its own sweet time to complete.

5. Use Azure Hybrid Benefit

If you are provisioning Windows server based VMs on Azure, you can save up to 49% by using your own Windows server licences. This hybrid benefit can be selected during the creation of Windows server VMs.

6. Use Reserved Azure VM Instance

Azure provides 4 different pricing options for virtual machines. These options ordered from high cost to low cost are pay as you go, 1 year reserved instances, 3 year reserved instances and spot instances. Please note that some options are available only to Azure customers in specific regions. For example, VM reservation for 1 year or 3 years is not available for Indian customers. Following are monthly prices for a sample VM for each option in the West Europe region (since prices also depend on the region). VM Type selected is Standard D4s V3 - 4vCPU and 16GB,

• Pay as you go option - $175
• 1 year VM reservation - $152 (13% discount)
• 3 years VM reservation - $120 (31% discount)
• Azure spot instance - $26 (85% discount)

If you are using a large pool of VMs, it is better to use a mixed reservation approach - you can reserve some for 3 years, some for short term 1 year and keep some unreserved if you plan to do some optimisations to reduce VM instances. See this page for various Azure VM pricing options.

7. Use Auto Shutdown Feature for VM

Azure has a simple shutdown feature which can be configured to enable auto shutdown of a virtual machine at a specified time. This is quite useful for machines that are needed during work hours only. Even if you forget to switch off the VM after your work, the auto shutdown can kick in and turn off the machine. Auto shutdown feature is available for a VM under the Operations section on the left of the VM configuration page.

Sometimes you want your VM to be running for a specific time duration only. For example, you may need your machine running only between 9AM and 11AM. In such cases you can set up automation tasks for starting and stopping VMs. Use the Tasks link under the Automation section available on the left of the VM configuration page.

8. Reduce Cost by Using a Smaller VM OS Disk

When an Azure VM instance is created using the portal, you can only specify the type of disk (HDD or SSD) but not the size of the disk. Many OS images create disks of 128GB size and these also add to the cost of the VM. Hence one way to reduce the cost is to use a lower size OS disk. However, this option is not available on the portal, and hence you need to use PowerShell command or Azure CLI command to use a custom sized OS disk. Alternatively, you can create the VM from the portal and then reduce the OS disk size using the CLI.

9. Use Azure Advisor to Monitor Cost Reduction Opportunities

When you have a large number of resources under your Azure tenant, it is easy to miss cost optimisation options. In such cases, it is important to monitor cost reduction recommendations provided by Azure Advisor. Azure advisor has a separate section on cost recommendations and is capable of providing VM cost recommendations such as reservation options and reduction of VM size based on historic CPU usage. If you are part of a large organisation with a large set of Azure resources, it is important to have a weekly monitoring cost KPI driven by Azure cost Advisor.

10. Analyse your VM Cost Breakdown for Optimising Cost

An Azure VM provisioned usually has the following additional infrastructural components.

• OS Disk and Data disk attached to the VM
• Any public IP attached to the VM
• Networking traffic on VM network interface such as cost of network peering

Each of these components can incur additional cost over and above the VM cost. So it is recommended to analyse the cost incurred by these VM components in Azure portal periodically to further optimise VM cost.

Features of GraalVM

GraalVM is a high performance JDK with a focus on 3 core features - high performance, compiled native app and polyglot programming. It is a drop-in replacement for Java 11 and Java 17. GraalVM community edition is an open source project with over 3.6 million lines of code contributed by the GraalVM team and collaborators.

GraalVM adds the following things to the Java HotSpot VM,

• A GraalVM compiler that is capable of compiling applications to a native image that is built for the underlying operating system. This ensures high performance and quick to start applications suitable for cloud and container platforms. With static linking of system libraries, a static binary can be even run on a bare-metal docker image!
• A language implementation framework codenamed Truffle that makes it possible to run programming languages such as JavaScript, Ruby, Python etc. on top of JVM. Ability to interoperate across languages under a shared runtime makes GraalVM a true polyglot programming environment.
• GraalVM LLVM runtime that can execute programming languages that can be converted to LLVM bitcode. This includes languages such as C, C++ and Fortran.
• A low overhead language agnostic fully dynamic instrumentation support available as part of the VM runtime. This can be used by API clients. It can track very fine-grained VM-level runtime events to inspect, profile and analyse the runtime behaviour of applications running on GraalVM.

The key vision statement of GraalVM is,

"Abstractions should be without performance regret!"

As per Oracle’s initial announcement, GraalVM can provide up to 100x faster startup time and up to 5x reduction in memory usage by using the ahead-of-time native compilation. This makes it a very tempting choice for building cloud native microservices.

GraalVM Architecture Diagram

The following diagram shows the high level layered architecture of the GraalVM ecosystem supporting native image compilation and polyglot features,

GraalVM is a project managed by Oracle and provides both enterprise and community edition distributions.

Goals of GraalVM Project

• Improve the performance of Java VM languages to match the performance of native languages and ensure low memory footprint.
• Reduce the startup time of JVM applications by compiling them ahead-of-time to native code.
• Enable GraalVM to be an embeddable technology, for example inside Oracle database.
• Allow use of multiple programming languages in a single program, the polyglot way.
• Enable application extensibility by supporting reuse of libraries from other languages or building language agnostic tools.

A Brief History of GraalVM

The word Graal comes from the old French for Grail (a thing to be eagerly pursued). Graal project started as a Java compiler research project inside Oracle Labs under the leadership of Thomas Wuerthinger. The original goal was to create a faster compiler for Java but then evolved to build additional features. These included ahead-of-time compilation called native image technology supporting fast native binaries and a language implementation framework enabling polyglot capabilities using non-JVM languages such as Python, Ruby and WebAssembly. All the technologies under Graal were released under the single umbrella name GraalVM.

The GraalVM ecosystem was in development for years in Oracle labs. In June 2018, a release candidate (1.0-RC2) version was released under free community edition. The first production ready version was released by Oracle under GraalVM enterprise edition 19.0 in May 2019 as a drop-in replacement for JDK 8. In October 2022, Oracle announced GraalVM JIT and native image becoming part of OpenJDK.

Following shows the evolution of GraalVM from May 2019 to October 2022. These versions are available either in community edition or enterprise edition with enterprise edition having additional features.

GraalVM 19.x versions (May 2019 to April 2021)

• Added support for JDK 11
• Improvements to GraalVM compiler
• Upgrades to Truffle language runtimes

GraalVM 20.0.x to GraalVM 20.3.x (February 2020 - January 2022)

• Full support for Windows platform
• Improved native images
• Upgrades to Truffle language runtimes
• Improved tooling support

GraalVM 21.0.x to GraalVM 21.3.x (January 2021 - September 2022)

• Java on Truffle
• GraalVM community images from Github container registry
• Linux ARM 64-bit support
• Serialisation support for images
• Tooling and language runtime updates
• Java 17 support

GraalVM 22.0.x to GraalVM 22.3.x (January 2021 - October 2022)

• Dropped support for Java 8. Now supports JDK 11 and 17 only
• Native image optimizations reduced executable size
• Updated to Truffle language implementations
• Added support for Apple silicon ARM M1

Limitations and Disadvantages of GraalVM

There is a lot of hype in the Java ecosystem about GraalVM. However, it is important to be aware of the limitations and disadvantages of GraalVM before adopting it. The ability of GraalVM to create native executables using SubstrateVM while exciting also comes with certain limitations. It is important to know these before deciding to adopt native images as your solution format,

• Java can use reflection to dynamically load classes. This is a problem for native compiler since it needs to know all the classes that may be loaded at runtime to generate the native image. The workaround is to specify all the classes needed using the native image configuration files.
• A lot of third party libraries and frameworks use reflection heavily. Hence those need to support GraalVM explicitly for it to work. See Spring Native.
• The native executables created do not have a JIT compiler. Hence aggressive code optimisation at runtime using profiler and JIT is not possible.
• Community edition of GraalVM provides a serial garbage collector as part of the native image runtime. It is intended for low memory footprint and small Java heap sizes. The runtime behaviour in this case can be different from a JVM based GC.

Language and Runtime Support in GraalVM

GraalVM is designed for the Java ecosystem and hence can execute all languages that compile to the Java bytecode format. These include languages such as Java, Scala, Kotlin etc.

In addition, the Truffle Language Implementation Framework supports the following additional languages,

• JavaScript - ECMA 2021 JavaScript runtime with NodeJS support
• Ruby - Ruby language implementation with support for Ruby on Rails
• R - R language implementation
• Python - Python 3 language implementation
• WASM - WebAssembly implementation
• LLVM - Sulong LLVM bitcode interpreter implementation

Who Uses GraalVM in Production?

Due to GraalVM’s diverse toolset and cloud advantages, some of the big players in the tech industry are currently using GraalVM in production. These include Facebook, Oracle, NVidia, Twitter, Goldman Sachs etc. Some of the examples of GraalVM in production are given below,

• Facebook with Spark workloads - Facebook has huge data sets and Java is used in big data processing using technologies such as Spark. By adopting GraalVM, Facebook was able to get reduced CPU and memory utilisation for spark workloads. Facebook using GraalVM for more details.
• Twitter Microservices Platform - Most of Twitter’s microservices are written in JVM language Scala. Using the GraalVM compiler, the CPU consumption was reduced more than 10%. The key technical features behind this improvement are better code inlining and escape analysis by the new compiler. Twitter using GraalVM in production.
• Goldman Sachs Slang Language Migration - In this interesting use case, Goldman Sachs is using the GraalVM’s Truffle language implementation framework to modernise their in-house programming language called Slang (Securities Language) with ability to interoperate with legacy C++ code. Team found tooling, native interoperate and documentation as key good parts of the GraalVM ecosystem. GoldMan Sachs Using GraalVM in Production for more details.

What You Can Do with GraalVM?

• Use it as a platform for high performance modern Java.
• Use it in container or cloud environments as a low footprint and fast startup Java microservices.
• Use multiple languages such as Java, JavaScript, Ruby etc. for a single application.
• Run native languages such as C, C++ on the JVM.
• Create tools that work across all languages.
• Extend a JVM based application.
• Extend a native application.
• Build Java code as a native library.
• Implement polyglot features in database.
• Create your own language.

GraalVM - Additional Reading

• GraalVM Homepage
• Oracle GraalVM Announcement
• GraalVM Now Part of OpenJDK
• GraalVM Use Cases in Production
• GraalVM and Custom Language Implementation Example (Video)

Python is currently the most popular programming language. Due to its simplicity and expressive power it is being used across the world as the first language to teach programming. It is also now used in advanced programming use cases such as data science, machine learning and statistics. With a powerful and large library of code, applications can be quickly built using Python.

For a beginning programmer starting to learn python, the first recommended resource is the official homepage of Python language itself. The beginner's guide to python provides a guided tour of the language, libraries and various use cases. . If python is your first language, start from here or otherwise start from this page.

Another way to learn python is to start with a book that provides a complete overview of the language and basics of using python. In this article, I will provide a set of book recommendations and what you will find in these books. Some folks like starting with a book than online documentation since it provides direct and focused approach to learning.

A Byte of Python by Swaroop CH - This is a free book on python available online. A PDF copy is also available for download. It is a gentle introduction to python and quickly introduces you to the python ecosystem and fundamentals. It doesn't cover much about programming in general, but as a first introduction to the python language, I recommend this book. The book is around 100 pages and doesn't go into advanced features of python. The book covers installing and getting started, basics, operators, control flow, functions, modules, basic data structures, basics of object oriented programming/problem solving, input/output, exceptions and standard library. This book is available as an open source project here.

Python Crash Course 2nd Edition by Eric Matthes - If you can spend some money, Python crash course is my recommended book for beginners. This book takes a project based approach to teaching python. The first section of the book is a good introduction of the python language starting from basics to object oriented programming using classes. This section is full of code snippets and try it yourself programming problems that gets you quick into the practical usage of python language. The second section shows you how you can build projects using python. It covers 3 major projects consisting of building a game, data visualization and creating web applications using python. The appendix covers installation, use of IDEs and a brief intro to the git version control. All in all, best book for python programmers at the beginner level. With over 500 pages, it may take a while cover the entire material.

Once you have completed these beginner books and has practical experience working with python projects, you may want to learn a bit more about pythonic way of writing code. Following are my recommendations for intermediate to advanced python programmers,

Effective Python by Bret Slatkin - Effective python is a mandatory reading for anyone who uses python in a professional setting or uses it frequently. It consists of 90 important things you need to know when writing python programming in python language organized in 10 chapters. Based on the style of classic effective c++ by Scott Meyers, this is a must have book for any programmer's library. Each advice consists of a brief background followed by plenty of code examples illustrating how the advice is to be applied when writing python code. Reading this book makes you understand how expert programmers write python code. Some examples from the 90 items included in the book are - Prefer raising exceptions to returning None (item 20), Profile before optimizing (item 70) and Consider interactive debugging with pdb (item 80).

Fluent Python by Luciano Ramalho - Fluent python is an advanced book on python that will enable you to understand and write effective and idiomatic python code. As the author himself says in preface, this book was written for practicing Python programmers who want to become proficient in Python 3. With over 700 pages, this book covers python data model, data structures, object oriented idioms, advanced control flows such as generators and coroutines and meta programming. It covers almost all advanced python features an experienced programmer is expected to know such as design patterns, function decorators and closures, ABC, inheritance, operator overloading, context managers, concurrency, class metaprogramming etc.

Python Cookbook by David Beazley and Brian K Jones - David Beazley is a well known software engineer who has made substantial contributions to the python developer community. His video sessions and keynotes are available on youtube and they are notable for the in-depth look at python and its ecosystem. Python cookbook is intended for experienced python programmers looking to deepen their understanding of the language and advanced programming techniques and idioms in python. This book organizes python recipes/techniques under 15 different chapters covering areas from data structures/algorithms, data types(string,numbers, dates etc.), modules, metaprogramming, network programming, c extensions etc. Each recipe is organized in 3 sections - problem, solution and discussion. Some example recipes are sanitizing and cleaning up text, reading and writing binary data and replacing single method classes with functions.

Python in a Nutshell by Alex Martelli- Alex Martelli is a software engineer and fellow of the python software foundation. This book is intended for programmers with some python experience. It is organized in 5 sections - getting started with python, core python language and built-ins, python library and extension modules, network and web programming, extending/distributing and v2/v3 migration. This is an excellent reference to python core and provides an in-depth look at language. However some may find it difficult due to the extreme detailing of the language (book is 700+ pages!).

What is a QR Code?

A QR code (Quick Response code) is a two dimensional barcode used popularly in product tracking, item recognition, document tracking and general marketing. It is also now commonly used in entry tickets since it can be easily scanned using a QR code scanner in a mobile. Standard QR codes can contain upto 3Kb of data. See this page to know how a QR code stores data.

QR codes may be used to display text, encode a web page URL etc. In the following python example, we will encode the URL of this website. Hence by scanning the resulting QR code image, you can navigate to www.quickprogrammingtips.com.

Generating QR Code Using Python

The following example uses pyqrcode module available from python repository. It is an easy to use module written in pure python. Optionally you can use it along with pypng module to render png files.

Before writing and running the following sample python program, ensure you have pyqrcode installed by running the pip command,

If you plan to create png files as qr code output, install pypng as well,

Let us now quickly create a simple QR code which contains the data, "www.quickprogrammingtips". By scanning the QR code, you can visit this site. Type the following code in a file named qrcodegenerator.py,

# Import pyqrcode module. Ensure it is installed using pip3 install pyqrcode
import pyqrcode
# Import png module. Ensure it is installed using pip3 install pypng.
import png
from pyqrcode import QRCode


data = "www.quickprogrammingtips.com"

# create qr code using the alphanumeric encoding of the data (url string in this case)
qrobject = pyqrcode.create(data)

# Create and save svg and png files
qrobject.svg("siteqrcode.svg", scale = 5)
qrobject.png('siteqrcode5.png', scale = 5)
qrobject.png('siteqrcode1.png', scale = 1)
qrobject.png('siteqrcode10.png', scale = 10)

Run the above code using,

The above code generate qr codes in svg and png format in the local folder. We have also created 3 png files with different scale parameters. A scale value of 1 defines the data module size of 1 pixel and hence will be too small. A scale factor of 5 or more creates larger image sizes that can be easily scanned. With a scale factor of 5, qr code images created was around 200px in width and height.

The above code used a simple version of the pyqrcode.create() function. For advanced use it can take additional parameters such as error level, qr version and mode. A typical use case will be - pyqrcode.create('hello', error='L', version=27, mode='binary'). Let me explain what these additional parameters are,

• Error level - QR codes can use one of four possible error correction values. They are referred to by the letters: L, M, Q, and H. The L error correction level corresponds to 7% of the code can be corrected. The M error correction level corresponds to 15% of the code can be corrected. The Q error correction level corresponds to 25% of the code can be corrected. The H error correction level corresponds to 30% of the code can be corrected.
• Version - QR codes have versions 1 to 40. Higher version can hold more data with increased number of dots. When not specified, lowest viable version will be selected based on the data encoding and data correction level.
• Mode - The encoding used to represent the data in a QR code. There are four possible encodings: binary, numeric, alphanumeric, kanji. When not specified, the library will pick one based on the data passed.

The following example creates a QR code for the text "Hello World" using the advanced parameters.

# Import pyqrcode module. Ensure it is installed using pip3 install pyqrcode
import pyqrcode
# Import png module. Ensure it is installed using pip3 install pypng.
import png
from pyqrcode import QRCode


data = 'Hello World'

# create qr code using the alphanumeric encoding of the data
# Since the data contains a space
qrobject = pyqrcode.create(data, error='H', version=10, mode='binary')

# Create and save svg and png files
qrobject.png('messageeqrcode5.png', scale = 5)

If you specify a lower version number that cannot hold the data you want to encode, you will get the follwing error,

The data will not fit inside a version code with the given encoding and error level.