DBT governance

Thid markdown document will specify the rules that need to follow by every developer.
Short naming converntion lists that need to follow

model_type Shortcut	Full name
seed	Seed
src	Source
snap	Snapshot
stg	Stage
int	Intermediate
fct	Fact
dim	Dimension
rep	Report
sem	Semantic

model_name Shortcut	Full name	Use case
brg	Brigde	relationship table
log	Log	log data

CIEF General Rule

• Snowflake SQL use to write dbt SQL code
• Star schema is use
• First week of the day is start at Monday, and last week of the day will be Sunday
• Fiscal 1st quarter is start from Febuary
• Timezone: Kuala Lumpur / Malaysia (GMT +8)
• Some status/type is not input in raw database (You should found in seed)

DBT Rule

Structure of the dbt folder model

• Analyses
  1. Analyses model will not create in Snowflake
  2. You can write some adhoc queries
  3. test some SQL code before write in Models

• Macros
  1. Jinja function that reuse in other .sql / .py file

• Models

  • Staging

    1. Materialise: View
    2. 1-to-1 relationship (or mapping) to source tables.
    3. Column renaming
    4. Column remove
    5. Column data renaming, such as (status integer to string)
    6. Data input error cleansing
    7. Check for raw data freshness
    8. Data type transformation

  • Intermediate

    1. Materialise: Ephemerally
    2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want
    3. Be referenced repeatedly in more than one model
    4. Isolating complex operations

  • marts/warehouse

    1. Store fact and dimension models

  • marts/reporting

    1. Store custom reports

• Seeds

• Snapshots

• Tests

File Naming Rules

• File names must be unique
• Each sql/python file must start with [model_type]_[system_name]__[model_name]s.sql/py
• The model should be name as plural, (eg: stg_exchange__orders.sql)

Column Naming Rules

• If array datatype, the column naming must be plural eg (ids, messages)

SQL Coding Rule

• The SQL clause must be UPPERCASE (eg. SELECT, FROM, WHERE, GROUP BY, LIMIT, WITH, AS, SUM, PARTITION OVER, DIV0, LEFT JOIN, ON ....)
• Try to avoid using EXCLUDE in the SQL
• Avoid using multi layer subquery, and try to use CTE, subquery must not more than 1 layer
• Each model structure must contain with

--IMPORT
WITH [table_names] AS (
    SELECT * FROM {{ ref('file_names')}}
),

--LOGIC
[logic_name] AS (
    .....
),

--FINAL
final__[table_names] AS (

)
SELECT * FROM final__[table_names]

Try running the following commands:

• dbt run
• dbt test

Resources:

• Learn more about dbt in the docs
• Check out Discourse for commonly asked questions and answers
• Join the dbt community to learn from other analytics engineers
• Find dbt events near you
• Check out the blog for the latest news on dbt's development and best practices