home · about · projects

Image from Wikimedia Commons

Working with Credentials and Configurations in Python

11 Jan 2022

Table of Contents

When writing programs, there is often a large set of configuration and credentials that should not be hard-coded in the program. This also makes the customization of the program much easier and more generally applicable. There are various ways to handle configuration and credentials and you will see here a few of the popular and common ways to do that with Python.

One important note right from the start: When using version control always make sure to not commit credentials and configuration into the repository as this could become a serious security issue. You can add those to .gitignore to avoid pushing those files to version control. Sometimes is useful to have general configuration also in version control, but that depends on your use case.

Python Configuration Files

The first and probably most straight forward way is to have a config.py file somewhere in the project folder that you add to your .gitignore file. A similar pattern can be found in Flask, where you can also structure the configuration based on different contexts like development, production, and testing. The config.py would look something like:

host = 'localhost',
port = 8080,
username = 'user'
password = 'password'

You would simply import it and use it like this:

import config

host = config.host
port = config.port
username = config.username
password = config.password

Environment Variables

You can access environment variables with os.environ:

import os

os.environ['SHELL']

This will throw a KeyError if the variable does not exists. You can check if the variable exists with "SHELL" in os.environ. Sometimes its more elegant to get None or a default value instead of getting an error when a variable does not exist. This can be done like this:

# return None if VAR does not exists
os.environ.get('VAR')

# return "default" if VAR does not exists
os.environ.get('VAR', "default")

You can combine this with the previous way to have a config.py with the following contents:

import os

host = os.environ.get('APP_HOST', 'localhost')
port = os.environ.get('APP_PORT', 8080)
username = os.environ.get('APP_USERNAME')
password = os.environ.get('APP_PASSWORD')

Python Dotenv

Oftentimes you want to have the environment variables in a dedicated .env file outside of version control. One way is to load the file before with:

source .env

This is sometimes error-prone or not possible depending on the setup, so its sometimes better to load the file dynamically with python-dotenv. You can install the package with:

pip install -U python-dotenv

Load the .env file in your program with:

from dotenv import load_dotenv

load_dotenv()

If your environment file is located somewhere else, you can load it with:

load_dotenv("/path/to/.env")

Now, you can use the environment file as you saw before.

JavaScript Object Notation (JSON)

JSON is another handy file format to store your configuration as it has native support. If you are working with frontend code, you are already familiar with its usefulness and ubiquity.

You can prepare your configurations as a JSON (JavaScript Object Notation) in a config.json with the following example configuration:

{
    "host": "localhost",
    "port": 8080,
    "credentials": {
        "username": "user",
        "password": "password"
    }
}

You can load this configuration then with the built-in json package:

import json

with open('config.json', 'r') as f:
    config = json.load(f)

This returns the data as (nested) dictionaries and lists which you can access the way you are used to (config['host'] or config.get('host')).

Yet Another Markup Language (YAML)

Another popular way to store configurations and credentials is the (in)famous YAML format. It is much simpler to use but has some minor quirks when using more complicated formatting. Here is the previous configuration as a YAML file:

host: localhost
port: 8080
credentials:
  username: user
  password: password

There are various packages that you can use. Most commonly PyYAML. You can install it with:

pip install -U PyYAML

To load the configuration, you can type:

with open("config.yml", 'r') as f:
    config = yaml.load(f, Loader=yaml.FullLoader)

The config can be used as previously seen with the JSON example.

Note, that you need to add a Loader in PyYAML 5.1+ because of a vulnerability. Read more about it here. Another common alternative to PyYAML is omegaconf, which includes many other useful parsers for various different file types.

Using a Configuration Parser

The Python standard library includes the configparser module which can work with configuration files similar to the Microsoft Windows INI files. You can prepare the configuration in config.ini with the following contents:

[DEFAULT]
host = localhost
port = 8080

[credentials]
username = user
password = password

The configuration is seperated into sections like [credentials] and within those sections the configuration is stored as key-value pairs like host = localhost.

You can load and use the previous configuration as follows:

import configparser

config = configparser.ConfigParser()
config.read("test.ini")

host = config['DEFAULT']['host']
port = config['DEFAULT']['port']
username = config['credentials']['username']
password = config['credentials']['password']

As you can see, to access the values you have to type config[section][element]. To get all sections as a list, you can type config.sections(). For more information, have a look at the documentation.

Parsing Command-line Options

It is also possible to get credentials and configuration through arguments by using the built-in argparse module.

You can initialize the argument parser with:

import argparse

parser = argparse.ArgumentParser(
    description="Example Program")

# Required arguments
parser.add_argument(action='store',
    dest='username', help="session username")
parser.add_argument(action='store',
    dest='password', help="session password")

# Optional arguments with default values
parser.add_argument("-H", "--host", action='store',
    dest='host', default="localhost",
    help="connection host")
# Allow only arguments of type int
parser.add_argument("-P", "--port", action='store',
    dest='port', default=8080, type=int,
    help="connection port")

Now, you can parse the arguments with:

args = parser.parse_args()

host = args.host
port = args.port
username = args.username
password = args.password

If you save this program in example.py and type python example.py -h, you will receive the following help description:

usage: untitled.py [-h] [-H HOST] [-P PORT] username password

Example Program

positional arguments:
  username              session username
  password              session password

optional arguments:
  -h, --help            show this help message and exit
  -H HOST, --host HOST  connection host
  -P PORT, --port PORT  connection port

Another alternative to argparse is typer which makes some of the parsing easier for complex CLI tools.

Conclusion

Here you saw a few common and popular ways to load configuration and credentials in Python, but there are many more ways if those are not sufficient for your usecase. You can always resort to XML if you really wish. If you miss some way that you particularly find useful, feel free to add it in the comments bellow.

Resources

Related Posts