A common pattern in scaling production app databases is to keep them as small as possible. Since building production apps is not my forte, I’ll lean on the commentary of experts. I like how Silvia Botros, author of High Performance MySQL, frames it below:
This architecture presents a unique challenge for analytics engineering because you now have many databases with identical schemas, and dbt sources must be enumerated in your YAML files.
I am going to share the three steps that I use to solve this problem. It should be noted that if you are comfortable with jinja, I am sure there are better, more pythonic ways to solves this problem. I have landed on this solution as something that is easy to understand, fast to develop, and fast to run (i.e. performant).
Step 1: leverage YAML anchors and aliases
Anchors and Aliases are YAML constructions that allow you to reduce repeat syntax and extend existing data nodes. You can place Anchors (
https://www.educative.io/blog/advanced-yaml-syntax-cheatsheet&) on an entity to mark a multi-line section. You can then use an Alias (*) call that anchor later in the document to reference that section.
By using anchors and aliases, we can drastically cut down on the amount of duplicate code that we need to write in our YAML file. A simplified version of what I have is below.
- name: BASE_DATABASE
database: CUSTOMER_N
schema: DATA
tables: &SHARD_DATA
- name: table_one
identifier: name_that_makes_sense_to_eng_but_not_data
description: a concise description
- name: table_two
- name: CUSTOMER_DATABASE
database: CUSTOMER_N+1
schema: DATA
tables: *SHARD_DATAUnfortunately with this solution, every time a new shard is added, we have to add a new line to our YAML file. While I don’t have a solution off hand, I am certain that you could generate this file with Python.
Step 2: Persist a list of your sharded databases
This next steps seems pretty obvious, but you need a list of your shards. There are multiple ways to get this data, but I will share two of them. The first is getting the list directly from your information schema.
(SQL SERVER)
SELECT * FROM sys.databases;
(SNOWFLAKE)
SELECT * FROM information_schema.databasesYou can then persist that information in a dbt model that you can query later.
The second way is to create a dbt seed. Since I already have a manual intervention in step 1, I am ok with a little bit of extra work in managing a seed as well. This also gives me the benefit of source control so I can tell when additional shards came online. And of course, this gives a little finer control over what goes into your analytics area since you may have databases that you don’t want to include in the next step. An example seed is below.
Id,SourceName
1,BASE_DATABASE
2,CUSTOMER_DATABASEStep 3: Use jinja + dbt_utils.get_column_values to procedurally generate your SQL
The of magic enabled by dbt here is that you can put a for loop inside your SQL query. This means that instead of writing out hundreds or thousands of lines of code to load your data into one place, dbt will instead generate it. Make sure that you have dbt_utils in your packages.yml file and that you have run ‘dbt deps’ to install it first.
{% set source_names = dbt_utils.get_column_values(table=ref('seed'), column='SourceName') %}
{% for sn in source_names %}
SELECT field_list,
'{{ sn }}' AS source_name
FROM {{ source( sn , 'table_one' ) }} one
INNER JOIN {{ ref( 'table_two' ) }} two ON one.id = two.id
{% if not loop.last %} UNION ALL {% endif %}
{% endfor %}In the case of our example, since we have two records in our ‘seed’ table, this will create two SQL queries with a UNION between them. Perfect!
Now I have scaled this to 25 databases or so, so managing it by hand works fine for me. Obviously if you have thousands of databases in production in this paradigm, running a giant UNION ALL may not be feasible (also I doubt you are reading this article if you have that many databases in prod). In fact, I ran into some internal constraints with parallelization with UNION with some models, so I use pre and post-hooks to handle it in a more scalable manner for those. Again, context matters here, so depending on the shape of your data, this may not work for you. Annoyingly, this doesn’t populate the dbt docs with anything particularly meaningful so you will need to keep that in mind.
(SQL SERVER)
{{ config(
materialized = "table",
pre_hook="
DROP TABLE IF EXISTS #source;
CREATE TABLE #source
(
some_field INT
);
{% set source_names = dbt_utils.get_column_values(table=ref('seed'), column='SourceName') %}
{% for sn in source_names %}
SELECT field_list,
'{{ sn }}' AS source_name
FROM {{ source( sn , 'table_one' ) }} one
INNER JOIN {{ ref( 'table_two' ) }} two ON one.id = two.id
{% endfor %}
DROP TABLE IF EXISTS target;
SELECT * INTO target FROM #source",
post_hook="
DROP TABLE #source;
DROP TABLE target;"
)
}}
SELECT * FROM targetSo there you have it, a few ways to pull multiple tables into one with dbt. Hope you found this helpful!
Alternative methods: using dbt_utils.union_relations
In theory, using dbt_utils.union_relations can also accomplish the same as step 3, but I have not tested it that way.
