cief/dbt_cloud
1
0
Fork 0
mirror of https://gitlab.com/cief-data/dbt_cloud.git synced 2026-08-19 04:14:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

Readme update

Browse Source
This commit is contained in:
Yam ZhengLim
2023-03-30 10:31:27 +00:00
parent 1d63223a5f
commit ff3cf92d04
1 changed files with 49 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+49 -38
View File
@@ -32,56 +32,82 @@ Short naming converntion lists that need to follow
<br/>
## DBT Rule
### Structure of the dbt folder model
### Structure of the dbt model layer
- Analyses
1. Analyses model will not create in Snowflake
1. Analyses layer will not create in Snowflake
2. You can write some adhoc queries
3. test some SQL code before write in Models
3. Test some SQL code before write in Models
<br/>
- Macros
1. Jinja function that reuse in other .sql / .py file
1. Jinja function that reuse in other models
<br>
<br/>
- Models
- Staging
1. Materialise: `View`
2. 1-to-1 relationship (or mapping) to source tables.
3. Column renaming
4. Column remove
5. Column data renaming, such as (status integer to string)
6. Data input error cleansing
7. Check for raw data freshness
8. Data type transformation
- System name
1. Materialise: `View`
2. 1-to-1 relationship (or mapping) to source tables.
3. Column renaming
4. Column remove
5. Column data renaming, such as (status integer to string)
6. Data input error cleansing
7. Check for raw data freshness
8. Data type transformation
9. Data split and merge
- Source
1. Source will write in YML
2. Check Freshness
3. Check duplicate id
4. Check null id
- Intermediate
1. Materialise: `Ephemerally`
2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want
3. Be referenced repeatedly in more than one model
4. Isolating complex operations
4. Isolating complex operations
- marts/warehouse
1. Store fact and dimension models
- Marts/warehouse
1. Materialise: `Table`
2. Store fact and dimension models
- marts/reporting
1. Store custom reports
- Marts/reporting
1. Materialise: `Table`
2. Store custom reports
<br/>
- Seeds
1. Store custom dataset in csv format
2. The dataset must not change frequently
<br/>
- Snapshots
1. DBT build-in SCD type 2 function
2. Prefer to use after the source table and before the staging layer
<br/>
- Tests
1. Can write some yml test case
<br/>
### File Naming Rules
- File names must be unique
- Each sql/python file must start with [`model_type`]_[`system_name`]__[`model_name`]s.sql/py
- The model should be name as plural, (eg: stg_exchange__orders.sql)
- The model should be name as plural. (eg: stg_exchange__orders.sql)
<br/>
### Column Naming Rules
- If array datatype, the column naming must be plural eg (ids, messages)
- If array datatype, the column naming must be plural (eg: ids, messages)
- all the date & time field will need to name as datetime / date (eg: created_datetime)
<br/>
### SQL Coding Rule
- The SQL clause **must** be UPPERCASE (eg. `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `LIMIT`, `WITH`, `AS`, `SUM`, `PARTITION OVER`, `DIV0`, `LEFT JOIN`, `ON` ....)
@@ -96,7 +122,7 @@ WITH [table_names] AS (
),
--LOGIC
[logic_name] AS (
[logic_names] AS (
.....
),
@@ -104,21 +130,6 @@ WITH [table_names] AS (
final__[table_names] AS (
)
SELECT * FROM final__[table_names]
SELECT * FROM final__[final_table_names]
```
Try running the following commands:
- dbt run
- dbt test
### Resources:
- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
- Join the [dbt community](http://community.getbdt.com/) to learn from other analytics engineers
- Find [dbt events](https://events.getdbt.com) near you
- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices