README Update

2026-08-19 04:14:00 +00:00 · 2023-03-30 08:33:13 +00:00
parent b8d9903e7f
commit 1d63223a5f
1 changed files with 111 additions and 2 deletions
@@ -1,6 +1,115 @@
-Welcome to your new dbt project!
+# DBT governance
+Thid markdown document will specify the rules that need to follow by every developer.
+<br> 
+Short naming converntion lists that need to follow
+  
+| model_type Shortcut | Full name |
+| ------------------- | ------    |
+| seed                | Seed      |
+| src                 | Source    |
+| snap                | Snapshot  |
+| stg                 | Stage     | 
+| int                 | Intermediate |
+| fct                 | Fact      |
+| dim                 | Dimension |
+| rep                 | Report    |
+| sem                 | Semantic  |
+
+| model_name Shortcut | Full name | Use case |
+| ------------------- | ------    | ------   |
+| brg                 | Brigde    | relationship table |
+| log                 | Log       | log data |
+
+
+## CIEF General Rule
+- Snowflake SQL use to write dbt SQL code
+- Star schema is use
+- First week of the day is start at Monday, and last week of the day will be Sunday
+- Fiscal 1st quarter is start from Febuary 
+- Timezone: Kuala Lumpur / Malaysia (GMT +8)
+- Some status/type is not input in raw database (You should found in seed)
+
+<br/>
+
+## DBT Rule
+### Structure of the dbt folder model
+- Analyses
+    1. Analyses model will not create in Snowflake
+    2. You can write some adhoc queries 
+    3. test some SQL code before write in Models
+
+<br/>
+
+- Macros
+    1. Jinja function that reuse in other .sql / .py file
+
+<br>
+
+- Models 
+    - Staging
+        1. Materialise: `View`  
+        2. 1-to-1 relationship (or mapping) to source tables.
+        3. Column renaming
+        4. Column remove
+        5. Column data renaming, such as (status integer to string)
+        6. Data input error cleansing
+        7. Check for raw data freshness
+        8. Data type transformation
+
+    - Intermediate
+        1. Materialise: `Ephemerally`
+        2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want
+        3. Be referenced repeatedly in more than one model
+        4. Isolating complex operations
+        
+        
+    - marts/warehouse
+        1. Store fact and dimension models
+
+    - marts/reporting
+        1. Store custom reports
+
+- Seeds
+
+- Snapshots
+
+- Tests
+
+### File Naming Rules
+- File names must be unique
+- Each sql/python file must start with [`model_type`]_[`system_name`]__[`model_name`]s.sql/py
+- The model should be name as plural, (eg: stg_exchange__orders.sql)
+
+### Column Naming Rules
+- If array datatype, the column naming must be plural eg (ids, messages)
+
+### SQL Coding Rule
+- The SQL clause **must** be UPPERCASE (eg. `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `LIMIT`, `WITH`, `AS`, `SUM`, `PARTITION OVER`, `DIV0`, `LEFT JOIN`, `ON` ....)
+- Try to avoid using `EXCLUDE` in the SQL
+- Avoid using multi layer subquery, and try to use CTE, subquery must not more than 1 layer
+- Each model structure must contain with 
+
+```sql
+--IMPORT
+WITH [table_names] AS (
+    SELECT * FROM {{ ref('file_names')}}
+),
+
+--LOGIC
+[logic_name] AS (
+    .....
+),
+
+--FINAL
+final__[table_names] AS (
+
+)
+SELECT * FROM final__[table_names]
+```
+
+
+

-### Using the starter project

 Try running the following commands:
 - dbt run