DBT restructure

README.md

@@ -32,56 +32,129 @@ Short naming converntion lists that need to follow
 <br/>
 
 ## DBT Rule
-### Structure of the dbt folder model
+### Structure of the dbt model layer
 - Analyses
-    1. Analyses model will not create in Snowflake
+    1. Analyses layer will not create in Snowflake
     2. You can write some adhoc queries 
-    3. test some SQL code before write in Models
+    3. Test some SQL code before write in Models
 
 <br/>
 
 - Macros
-    1. Jinja function that reuse in other .sql / .py file
+    1. Jinja function that reuse in other models
 
-<br>
+<br/>
 
 - Models 
     - Staging
-        1. Materialise: `View`  
-        2. 1-to-1 relationship (or mapping) to source tables.
-        3. Column renaming
-        4. Column remove
-        5. Column data renaming, such as (status integer to string)
-        6. Data input error cleansing
-        7. Check for raw data freshness
-        8. Data type transformation
+        - System name
+            1. Materialise: `View`  
+            2. 1-to-1 relationship (or mapping) to source tables.
+            3. Column renaming
+            4. Column remove
+            5. Column data renaming, such as (status integer to string)
+            6. Data input error cleansing
+            7. Check for raw data freshness
+            8. Data type transformation
+            9. Data split and merge
+            - Source
+                1. Source will write in YML
+                2. Check Freshness 
+                3. Check duplicate id
+                4. Check null id
 
     - Intermediate
         1. Materialise: `Ephemerally`
         2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want
         3. Be referenced repeatedly in more than one model
-        4. Isolating complex operations
+        4. Isolating complex operations       
         
-        
-    - marts/warehouse
-        1. Store fact and dimension models
+    - Marts/warehouse
+        1. Materialise: `Table`
+        2. Store fact and dimension models
 
-    - marts/reporting
-        1. Store custom reports
+    - Marts/reporting
+        1. Materialise: `Table`
+        2. Store custom reports
+
+<br/>
 
 - Seeds
+    1. Store custom dataset in csv format
+    2. The dataset must not change frequently
+
+<br/>
 
 - Snapshots
+    1. DBT build-in SCD type 2 function
+    2. Prefer to use after the source table and before the staging layer
+
+<br/>
 
 - Tests
+    1. Can write some yml test case
+
+<br/>
 
 ### File Naming Rules
 - File names must be unique
 - Each sql/python file must start with [`model_type`]_[`system_name`]__[`model_name`]s.sql/py
-- The model should be name as plural, (eg: stg_exchange__orders.sql)
+- The model should be name as plural. (eg: stg_exchange__orders.sql)
+
+<br/>
 
 ### Column Naming Rules
-- If array datatype, the column naming must be plural eg (ids, messages)
+- If array datatype, the column naming must be plural (eg: ids, messages)
+- Schema, table and column names should be in `snake_case`.
+
+- Limit use of abbreviations that are related to domain knowledge. An onboarding
+  employee will understand `current_order_status` better than `current_os`.
+
+- Use names based on the _business_ terminology, rather than the source terminology.
+
+- Each model should have a primary key that can identify the unique row, and should be named `<object>_id`, e.g. `account_id` – this makes it easier to know what `id` is being referenced in downstream joined models.
+
+- If a surrogate key is created, it should be named `<object>_sk`.
+
+- For `base` or `staging` models, columns should be ordered in categories, where identifiers are first and date/time fields are at the end.  
+  Example:
+  ```sql
+  transformed as (
+      select
+          -- ids
+          order_id,
+          customer_id,
+
+          -- dimensions
+          order_status,
+          is_shipped,
+
+          -- measures
+          order_total,
+
+          -- date/times
+          created_at,
+          updated_at,
+
+          -- metadata
+          _sdc_batched_at
+      from source
+  )
+  ```
+
+- Date/time columns should be named according to these conventions:
+  - Timestamps: `<event>_datetime`  
+    Example: `created_datetime`
+  
+  - Dates: `<event>_date`  
+    Example: `created_date`
+
+- Booleans should be prefixed with `is_` or `has_`.  
+  Example: `is_active_customer` and `has_admin_access`
+
+- Price/revenue fields should be in decimal currency (e.g. `19.99` for $19.99; many app databases store prices as integers in cents). If non-decimal currency is used, indicate this with suffix, e.g. `price_in_cents`.
+
+<br/>
 
 ### SQL Coding Rule
 - The SQL clause **must** be UPPERCASE (eg. `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `LIMIT`, `WITH`, `AS`, `SUM`, `PARTITION OVER`, `DIV0`, `LEFT JOIN`, `ON` ....)
@@ -96,7 +169,7 @@ WITH [table_names] AS (
 ),
 
 --LOGIC
-[logic_name] AS (
+[logic_names] AS (
     .....
 ),
 
@@ -104,21 +177,8 @@ WITH [table_names] AS (
 final__[table_names] AS (
 
 )
-SELECT * FROM final__[table_names]
+SELECT * FROM final__[final_table_names]
 ```
 
-
-
-
-
-Try running the following commands:
-- dbt run
-- dbt test
-
-
-### Resources:
-- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
-- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
-- Join the [dbt community](http://community.getbdt.com/) to learn from other analytics engineers
-- Find [dbt events](https://events.getdbt.com) near you
-- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
+### Testing
+- At a minimum, `unique` and `not_null` tests should be applied to the expected primary key of each model.