diff --git a/README.md b/README.md index 13040c0..d24c02a 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,115 @@ -Welcome to your new dbt project! +# DBT governance +Thid markdown document will specify the rules that need to follow by every developer. +
+Short naming converntion lists that need to follow + +| model_type Shortcut | Full name | +| ------------------- | ------ | +| seed | Seed | +| src | Source | +| snap | Snapshot | +| stg | Stage | +| int | Intermediate | +| fct | Fact | +| dim | Dimension | +| rep | Report | +| sem | Semantic | + +| model_name Shortcut | Full name | Use case | +| ------------------- | ------ | ------ | +| brg | Brigde | relationship table | +| log | Log | log data | + + +## CIEF General Rule +- Snowflake SQL use to write dbt SQL code +- Star schema is use +- First week of the day is start at Monday, and last week of the day will be Sunday +- Fiscal 1st quarter is start from Febuary +- Timezone: Kuala Lumpur / Malaysia (GMT +8) +- Some status/type is not input in raw database (You should found in seed) + +
+ +## DBT Rule +### Structure of the dbt folder model +- Analyses + 1. Analyses model will not create in Snowflake + 2. You can write some adhoc queries + 3. test some SQL code before write in Models + +
+ +- Macros + 1. Jinja function that reuse in other .sql / .py file + +
+ +- Models + - Staging + 1. Materialise: `View` + 2. 1-to-1 relationship (or mapping) to source tables. + 3. Column renaming + 4. Column remove + 5. Column data renaming, such as (status integer to string) + 6. Data input error cleansing + 7. Check for raw data freshness + 8. Data type transformation + + - Intermediate + 1. Materialise: `Ephemerally` + 2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want + 3. Be referenced repeatedly in more than one model + 4. Isolating complex operations + + + - marts/warehouse + 1. Store fact and dimension models + + - marts/reporting + 1. Store custom reports + +- Seeds + +- Snapshots + +- Tests + +### File Naming Rules +- File names must be unique +- Each sql/python file must start with [`model_type`]_[`system_name`]__[`model_name`]s.sql/py +- The model should be name as plural, (eg: stg_exchange__orders.sql) + +### Column Naming Rules +- If array datatype, the column naming must be plural eg (ids, messages) + +### SQL Coding Rule +- The SQL clause **must** be UPPERCASE (eg. `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `LIMIT`, `WITH`, `AS`, `SUM`, `PARTITION OVER`, `DIV0`, `LEFT JOIN`, `ON` ....) +- Try to avoid using `EXCLUDE` in the SQL +- Avoid using multi layer subquery, and try to use CTE, subquery must not more than 1 layer +- Each model structure must contain with + +```sql +--IMPORT +WITH [table_names] AS ( + SELECT * FROM {{ ref('file_names')}} +), + +--LOGIC +[logic_name] AS ( + ..... +), + +--FINAL +final__[table_names] AS ( + +) +SELECT * FROM final__[table_names] +``` + + + -### Using the starter project Try running the following commands: - dbt run