When developing software, one of the most critical aspects of securing your application is managing sensitive data like API keys, database credentials, and other secrets. In Python, handling these secrets can sometimes be tricky, especially during development, where ease of use can often lead to risky practices like storing sensitive data in source control.

In this blog post, we’ll see how we can securely manage our application secrets in Python using a method similar to .NET’s Secret Manager, ensuring that secrets remain out of your source control while still being easy to use during development.

Keep Secrets Outside of Source Control

One of the most important principles of managing secrets is: never store them in source control. Secrets that are checked into your version control system can be accessed by anyone with access to your repository, leading to potentially devastating security breaches. To avoid this, always ensure sensitive files are excluded from version control.

Ignore Sensitive Files in Source Control

To prevent committing sensitive configuration files, such as secret settings, to your repository, you need to configure your version control system to ignore them. This is typically done by adding an ignore rule for the secrets file (for example, appconfig.secrets.py) in your project’s configuration. The exact method for doing this will depend on your version control system (e.g., Git, Mercurial, etc.).

Structuring Your Configuration Data

A good practice is to structure your configuration data using Python’s dataclass. This keeps the configuration organized and maintainable. Here’s an example of a Config class that holds your database connection settings and a third-party API endpoint:

from dataclasses import dataclass
from typing import Optional

@dataclass
class Connection:
    server: str
    database: str
    user: str
    password: str
    port: Optional[str] = None

@dataclass
class Api:
    endpoint: str
    key: str

@dataclass
class Config:
    connection: Connection
    api: Api

This Config class allows you to neatly organize the configuration values you need, separating them into structured fields for easy access and management.

Example of Configuration Data

In your configuration, you would set the basic values for your app but leave sensitive information like passwords out of it:

config = Config(
    connection=Connection(
        server='my-database-server',
        port=1234,
        database='db_catalogue',
        user='user',
        password='<will be read from secrets>'
    ),
    api = Api(
        third_party_api_endpoint="https://example.com",
        key='<will be read from secrets>'
    )
)

As seen above, the database password and API key are left as placeholders because they will be dynamically loaded from the secrets file later.

Defining the Secrets File

Now, let’s define the structure of the appconfig.secrets.py file, which will contain the actual sensitive data. Here’s what the secrets file might look like:

import lib.configuration as config

config.connection.password = "actual mega-secret password"
config.api.key = "actual mega-secret key"

In this file, we’re directly modifying the configuration by assigning the real values to the placeholders in the config object that was imported from appconfig.py. You can include other sensitive information this file as well.

Loading Configuration and Secrets

Now, let’s look at a simple and secure way to load both regular configuration and sensitive secrets. You can do this by having a primary configuration file (appconfig.py) that contains the non-sensitive values, and a separate secrets file (appconfig.secrets.py) for sensitive data.

Here’s a Python script that loads both configuration and secrets:

import os
import importlib.util

# Load the main config file
spec = importlib.util.spec_from_file_location(
    name="appconfig",
    location="appconfig.py"
)

my_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(my_module)

# Check for the secrets file and load it if present
secrets_file = "appconfig.secrets.py"
if os.path.exists(secrets_file):
    secrets_spec = importlib.util.spec_from_file_location(
        name="appconfig_secrets",
        location=secrets_file
    )

    if secrets_spec.loader is not None:
        secrets_module = importlib.util.module_from_spec(secrets_spec)
        secrets_spec.loader.exec_module(secrets_module)

    # Override sensitive values in the main config with values from the secrets file
    config = my_module.config
    if secrets_module:
        for attr in dir(secrets_module.config):
            if hasattr(secrets_module.config, attr) and hasattr(config, attr):
                setattr(config, attr, getattr(secrets_module.config, attr))

# Continue with your logic

How This Works

1. Main Configuration Loading: The script loads the main configuration from appconfig.py, which contains non-sensitive data and placeholders for sensitive values.
2. Secrets Loading: If the secrets file (appconfig.secrets.py) exists, it loads sensitive information such as passwords or API keys.
3. Secrets Injection: The values from the secrets file are used to override the placeholders in the main configuration.

This way, your sensitive data is dynamically injected into the configuration at runtime, keeping it separate from your main codebase.

This method is easy to implement and requires minimal configuration. It allows you to manage secrets without complex tooling or external dependencies.

This approach provides a simple, effective way to manage secrets during development, following the best practice of never storing secrets in your codebase.