I use the term "environment" quite a lot in this article. If you’re not aware of what that means, or just want to be sure that I use it in the same way as you do, look at the definition I give at the end of the article.

settings.py is a core file in Django projects. It holds all the configuration values that your web app needs to work; database settings, logging configuration, where to find static files, API keys if you work with external APIs, and a bunch of other stuff.

But once you start setting up your Django app on multiple environments; like production, testing, and staging — and on machines for new developers, you are likely to run into a pain point; managing the configuration across the different environments.

This article discusses about this problem, provides the solution that we have used for most of our projects, and also lists other ways in which the Django community tackles this.

The problem

I think one of the reasons that many Django tutorials; including the official one, don’t mention this issue is because honestly, it’s not a huge pain. Compared to how often your make changes to the code for your views, templates, and models – you rarely touch the settings.py file once you have it setup.

But do the same thing over and over again; over the period of years that some projects are in use, and you tend to get tired of it and start wanting a better solution. But what exactly is the problem?

Imagine you are developing a new web application in Django. You have been working on it for a couple of weeks now. Initially it was just something you started as a proof-of-concept, but now your employer sees it going somewhere – and you get assigned a team to help you finish this thing and release it.

When that second developer wants to start working on the project, they clone it from the Git/Mercurial/whatever repo, install the project dependencies, and run python manage.py runserver to test it out. But wait, what’s this.

Errors like these are common when you setup a Django project on a new machine. When you were working on your own, you set some configuration options; like the logging file path, to work on your machine. But now that your colleague wants to run the project on their machine, they don’t have the same file system structure as yours.

The quickest way most beginner Django developers solve this is to change the settings.py file to make the project run on their machines. But this just creates another problem a bit later down the line.

If your colleague pushes their modified settings.py to the code repo and you pull the changes, the project stops working on your machine. Because, wait for it — their file system structure is different than yours.

A first stab at the problem

One solution is to use relative paths instead of absolute ones. So instead of

LOGGING_DIR = "/home/jibran/logs/"

you use

LOGGING_DIR = os.path.join(BASE_DIR, "logs")

This way the logs are now written to a location relative to the BASE_DIR variable, which for Django projects is set to the directory containing the manage.py file.

This solves the problem described above with system specific paths, but it’s not the complete answer. You have many other types of configuration options – the database username and password, authentication keys for APIs, SMTP options on how to send emails…

When you want to deploy your application to multiple environments, API keys are something that are most likely to change. Many of the APIs we use everyday offer testing/sandbox and live environments.

SendWithUs (a service used to send emails) for example allows you to create test API keys that deliver emails to only 1 inbox, ideal when testing as you don’t want test emails going to your customers. When you have production, testing, and local development environments, you will need different keys for all these different setups.

There is also issue of security; you ideally don’t want API keys for your services to be a part of your code repository. Why not?

• A security breach could let anyone have access to your repo. It’s been known to happen!
• You could hire outside contractors or freelancers to work on your code. Ideally they shouldn’t be aware of your API keys.
• You may want to limit what API keys junior developers on your team have access to. If your keys are in your repo, you can’t have that protection.

There are a number of good ways in which you can solve this problem. I’ll first show what the solution that we at Agile Leaf use. Later I’ll also list some other solutions from the Django community.

Our solution

The solution we use here at Agile Leaf is surprisingly simple; both to implement and understand. At the end of settings.py for our Django projects, we put

try:   from local_settings import *except ImportError:    raise Exception("A local_settings.py file is required to run this project")

and then we put all our environment dependent configuration options into a local_settings.py file that sits in the same directory as the settings.py. We make sure that the local settings file isn’t version controlled, and then each environment we deploy to, we create a new local_settings.py file on that machine with settings specific for that environment.

The reason this works is because the settings.py file is just another Python script. When Django needs to read a configuration value, it will import this file. Since we are doing a star import from our local settings into the main settings file, all variables defined in local_settings.py become available inside settings.py. More importantly, these imported variables override any that are already defined in the settings.py, which is why this import goes at the end of the settings file.

Now, instead of defining our API keys in the settings.py file, we define it in local_settings.py, like this.

SENDWITHUS_API_KEY = "<API KEY>"AWS_KEY = "<AWS_KEY>"...

Each developer on our team is then free to modify their own copy of the local_settings.py without needing to touch the main settings.py file. Since the local settings are not pushed to our code repo, we don’t get accidentally break each others development environments.

More importantly, we don’t need to worry about having live API keys on our test environments, or test keys on our production environments. When we create a new environment, we first create a local_settings.py file for that environment.

For example, the first time we create a production server for our projects, we will create the local settings file, which then gets saved on an internal machine that isn’t publicly accessible. The next time we need to create a production server, we can just copy this file over and have the new server up and running quickly.

Which configuration options you put in settings.py and which you put in local_settings.py depends on your project. The best advice I can give is to move settings that definitely change between environments; like the database and logging configurations.

As your project grows and you learn more about this approach, you’ll get an intuitive feel for where a specific configuration option should go to. I plan to write another article later discussing the particular method we use when dividing our configuration options between the main and local settings.

Why this way?

We settled upon this approach because it solves almost all of the pains we felt while developing Django projects.

• We don’t commit our secret API keys to our code repo.
• When the code is setup on a new machine without a local settings file, the developer gets a better error message telling them that a local settings file is needed to run the project, instead of cryptic "Path does not exist" errors.
• Because the local settings are imported at the end, we can add any configuration option to local_settings.py and override whatever settings.py already defines. For example our main settings file can have DEBUG = False but for our local development machines we can put DEBUG = True in the local settings file to override that.
• Because we are overriding the values defined in settings.py, we don’t need to put every configuration option in local_settings.py, only those that we need to change for the machine the project is running on. This also allows us to keep common settings; like the INSTALLED_APPS, MIDDLEWARE and other similar options in our settings.py file.
• Making changes to configuration for each environment is easy. Let’s say we want to change the API key for SendWithUs in our production environment. We modify the local_settings.py we have for our production environment, and then copy that file over to all the production servers.
• This doesn’t change how the configuration is done. It’s just another file that’s imported into the settings.py file. Which means that new Django developers who have just started using Django can directly use the knowledge they gained about configuration from the Django tutorial in our project. They don’t need to learn 1 more thing specific to our projects.

We have been using this technique in our Django projects for a couple of years, and it has served us well. Give it a try. Next I’ll discuss one alternative in detail, and then list some other options available in the community.

A small tweak

A small tweak to our approach is to move the from local_settings import * near the top of settings.py. This has some benefits.

You could define variables like

DATABASE_HOST = 'localhost:5678'DATABASE_USER = 'root'LOGGING_DIR = '/home/jibran/logs'

at the start of settings.py with reasonable defaults for a local development environment. Then import local settings. Finally, instead of using something like

DATABASES = {    'default': {        'ENGINE': 'django.db.backends.postgresql',        'NAME': 'db_name',        'USER': 'jibran'    }}

in the settings.py file, you could use

DATABASES = {    'default': {        'ENGINE': 'django.db.backends.postgresql',        'NAME': DATABASE_NAME,        'USER': DATABASE_USER    }}

This allows you to make it even easier to create a new local settings file. If it’s imported at the bottom; like we do, you need to define that entire DATABASES config dictionary in the local settings to override the configuration in the main settings. By importing near the top, you can override just specific variables and settings.py will use those new values.

We haven’t used this way yet because for us, the tradeoff of making sure that the file is imported in the correct place – after the variables that can be overridden, but before those variables are used vs. needing to write the complete config option to override (like DATABASES in the example above) isn’t worth it. But maybe it’s something you prefer.

Other options

Over the years, people have successfully used a number of ways different than what we have described here. Some of these are listed here.

Version controlled environment specific settings files

One popular method that you can see suggested on Django forums and Stackoverflow is to have multiple settings files. For example, you could have

• settings.dev.py
• settings.prod.py
• settings.test.py

Then when you need to run your project you define the DJANGO_SETTINGS_MODULE env var to point to the correct file.

Let’s say you want to run the development server for your project called myproject. Instead of

python manage.py runserver

You would run

DJANGO_SETTINGS_MODULE=myproject.settings.dev python manage.py runserver

Likewise on the production and test environments, you’d configure uwsgi or gunirocrn to set DJANGO_SETTINGS_MODULE to the correct settings file name for that environment.

I personally don’t like this because:

• You’re still committing API keys to your code repo
• You now have to remember to set the DJANGO_SETTINGS_MODULE env var every time you use manage.py commands. It’s easier for me to define a local settings once instead of needing to remember this.
• Each individual settings file now needs to contain all the configuration options. You’re not overriding anything, instead you have completely different configurations. Given how most projects need to change only some configuration values for different environments, it feels like overkill.
• It’s easy for the files to get out of sync. For example you can add a new application to INSTALLED_APPS for your local environment and forget to add it to the production environment settings. A blog post I found while researching this article discusses a very good way of side stepping this problem, while having environment specific settings files.

While I prefer my way 🙂 I do see the benefits of this approach:

• Updating configuration on servers is now even easier. No need to have two workflows, one for updating the code and one for the settings.
• The project literally works out of the box. Once you put the code on a server, selecting the right settings module is all you need to do before running the code. No need to go messing around figuring out the right version of local_settings.py to use.

Multiple settings files, without needing to explicitly select one

While researching for this article, I found this answer on Stackoverflow. It’s similar to how you would create different settings files for different environments, but instead of selecting one manually, your settings.py file now selects for you based on which machine the project is running on.

Some of the same reservations that I have to the last method apply here.

Django split settings

There’s a nice project I found called django_split_settings. It seems to be a combination of the method we use, and splitting the settings across multiple files. You can include optional local settings files to override default configs.

This seems like a nice option that we might try out in the future. One reservation though is that it adds another thing that new developers will now need to learn.

Django configurations

django-configurations is a Django module I found from the Stackoverflow answer I mentioned above. I haven’t researched it in detail so can’t say anything, but it’s another option for you to consider.

All the other options

I found during research for this article a page on the Django wiki that lists many different ways people handle this and other configuration pains. I’m sure it’s an interesting read to compare different ways. You can see it at https://code.djangoproject.com/wiki/SplitSettings.

Define environment

This article uses the term environment a lot. Some new Django developers might not know what the term means. I’ll try to define it clearly here.

An environment (defined as what I mean when I use the word in this article) is a group of machines (may be just 1 machine) that use the same configuration. Here’s an example.

Once you are done developing your web application, you usually get a cloud server from some provider like AWS, Digital Ocean, Google Cloud, or one of the other hundred options available out there. You then setup your first server to serve live traffic. This machine is your production environment, because it serves production/live traffic.

Eventually, you might want to add more servers to serve more users; by using load balancing. Any other servers you add will usually have the same configuration as your first server. They’ll use the same database, have the same path for logging files, etc. This group of machines that now serves your live traffic is also part of your production environment.

Later you might want to have some servers on the cloud to run a testing environment that you can push new updates to before you update your live environment. Your testing team can use this to find bugs, or you could show new features to clients, without worrying about it effecting your live traffic. This is your testing environment.

You also have your own machine, where you develop the application. Other developers hopefully have their own machines for this task; unless you’re weird and share just 1 machine between yourselves. I’m not judging!

These development machines are your development environment.

Hopefully this clears up what I mean when I use environment in this article.

Signup for our mailing list

We try to publish interesting articles like this on tech, marketing, and other topics based on projects we work on. If you would like to be notified the next time we publish something, please signup for our mailing list. We promise not to spam you!

Do you have an idea for a web site or application you’d like developed and need an experienced team to create it for you? Get in touch with us at [email protected].

And if you’re a fellow developer working on a Django project and get stuck somewhere, reach out to us at [email protected]. We’ll try our best to get back to you and help with whatever you’re having problems with.