mirror of
https://gitlab.com/cief-data/dbt_cloud.git
synced 2026-08-19 04:14:00 +00:00
DBT restructure
This commit is contained in:
Yam ZhengLim
@@ -32,56 +32,129 @@ Short naming converntion lists that need to follow
|
||||
<br/>
|
||||
|
||||
## DBT Rule
|
||||
### Structure of the dbt folder model
|
||||
### Structure of the dbt model layer
|
||||
- Analyses
|
||||
1. Analyses model will not create in Snowflake
|
||||
1. Analyses layer will not create in Snowflake
|
||||
2. You can write some adhoc queries
|
||||
3. test some SQL code before write in Models
|
||||
3. Test some SQL code before write in Models
|
||||
|
||||
<br/>
|
||||
|
||||
- Macros
|
||||
1. Jinja function that reuse in other .sql / .py file
|
||||
1. Jinja function that reuse in other models
|
||||
|
||||
<br>
|
||||
<br/>
|
||||
|
||||
- Models
|
||||
- Staging
|
||||
1. Materialise: `View`
|
||||
2. 1-to-1 relationship (or mapping) to source tables.
|
||||
3. Column renaming
|
||||
4. Column remove
|
||||
5. Column data renaming, such as (status integer to string)
|
||||
6. Data input error cleansing
|
||||
7. Check for raw data freshness
|
||||
8. Data type transformation
|
||||
- System name
|
||||
1. Materialise: `View`
|
||||
2. 1-to-1 relationship (or mapping) to source tables.
|
||||
3. Column renaming
|
||||
4. Column remove
|
||||
5. Column data renaming, such as (status integer to string)
|
||||
6. Data input error cleansing
|
||||
7. Check for raw data freshness
|
||||
8. Data type transformation
|
||||
9. Data split and merge
|
||||
- Source
|
||||
1. Source will write in YML
|
||||
2. Check Freshness
|
||||
3. Check duplicate id
|
||||
4. Check null id
|
||||
|
||||
- Intermediate
|
||||
1. Materialise: `Ephemerally`
|
||||
2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want
|
||||
3. Be referenced repeatedly in more than one model
|
||||
4. Isolating complex operations
|
||||
4. Isolating complex operations
|
||||
|
||||
|
||||
- marts/warehouse
|
||||
1. Store fact and dimension models
|
||||
- Marts/warehouse
|
||||
1. Materialise: `Table`
|
||||
2. Store fact and dimension models
|
||||
|
||||
- marts/reporting
|
||||
1. Store custom reports
|
||||
- Marts/reporting
|
||||
1. Materialise: `Table`
|
||||
2. Store custom reports
|
||||
|
||||
<br/>
|
||||
|
||||
- Seeds
|
||||
1. Store custom dataset in csv format
|
||||
2. The dataset must not change frequently
|
||||
|
||||
<br/>
|
||||
|
||||
- Snapshots
|
||||
1. DBT build-in SCD type 2 function
|
||||
2. Prefer to use after the source table and before the staging layer
|
||||
|
||||
<br/>
|
||||
|
||||
- Tests
|
||||
1. Can write some yml test case
|
||||
|
||||
<br/>
|
||||
|
||||
### File Naming Rules
|
||||
- File names must be unique
|
||||
- Each sql/python file must start with [`model_type`]_[`system_name`]__[`model_name`]s.sql/py
|
||||
- The model should be name as plural, (eg: stg_exchange__orders.sql)
|
||||
- The model should be name as plural. (eg: stg_exchange__orders.sql)
|
||||
|
||||
<br/>
|
||||
|
||||
### Column Naming Rules
|
||||
- If array datatype, the column naming must be plural eg (ids, messages)
|
||||
- If array datatype, the column naming must be plural (eg: ids, messages)
|
||||
- Schema, table and column names should be in `snake_case`.
|
||||
|
||||
- Limit use of abbreviations that are related to domain knowledge. An onboarding
|
||||
employee will understand `current_order_status` better than `current_os`.
|
||||
|
||||
- Use names based on the _business_ terminology, rather than the source terminology.
|
||||
|
||||
- Each model should have a primary key that can identify the unique row, and should be named `<object>_id`, e.g. `account_id` – this makes it easier to know what `id` is being referenced in downstream joined models.
|
||||
|
||||
- If a surrogate key is created, it should be named `<object>_sk`.
|
||||
|
||||
- For `base` or `staging` models, columns should be ordered in categories, where identifiers are first and date/time fields are at the end.
|
||||
Example:
|
||||
```sql
|
||||
transformed as (
|
||||
select
|
||||
-- ids
|
||||
order_id,
|
||||
customer_id,
|
||||
|
||||
-- dimensions
|
||||
order_status,
|
||||
is_shipped,
|
||||
|
||||
-- measures
|
||||
order_total,
|
||||
|
||||
-- date/times
|
||||
created_at,
|
||||
updated_at,
|
||||
|
||||
-- metadata
|
||||
_sdc_batched_at
|
||||
from source
|
||||
)
|
||||
```
|
||||
|
||||
- Date/time columns should be named according to these conventions:
|
||||
- Timestamps: `<event>_datetime`
|
||||
Example: `created_datetime`
|
||||
|
||||
- Dates: `<event>_date`
|
||||
Example: `created_date`
|
||||
|
||||
- Booleans should be prefixed with `is_` or `has_`.
|
||||
Example: `is_active_customer` and `has_admin_access`
|
||||
|
||||
- Price/revenue fields should be in decimal currency (e.g. `19.99` for $19.99; many app databases store prices as integers in cents). If non-decimal currency is used, indicate this with suffix, e.g. `price_in_cents`.
|
||||
|
||||
<br/>
|
||||
|
||||
### SQL Coding Rule
|
||||
- The SQL clause **must** be UPPERCASE (eg. `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `LIMIT`, `WITH`, `AS`, `SUM`, `PARTITION OVER`, `DIV0`, `LEFT JOIN`, `ON` ....)
|
||||
@@ -96,7 +169,7 @@ WITH [table_names] AS (
|
||||
),
|
||||
|
||||
--LOGIC
|
||||
[logic_name] AS (
|
||||
[logic_names] AS (
|
||||
.....
|
||||
),
|
||||
|
||||
@@ -104,21 +177,8 @@ WITH [table_names] AS (
|
||||
final__[table_names] AS (
|
||||
|
||||
)
|
||||
SELECT * FROM final__[table_names]
|
||||
SELECT * FROM final__[final_table_names]
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Try running the following commands:
|
||||
- dbt run
|
||||
- dbt test
|
||||
|
||||
|
||||
### Resources:
|
||||
- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
|
||||
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
|
||||
- Join the [dbt community](http://community.getbdt.com/) to learn from other analytics engineers
|
||||
- Find [dbt events](https://events.getdbt.com) near you
|
||||
- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
|
||||
### Testing
|
||||
- At a minimum, `unique` and `not_null` tests should be applied to the expected primary key of each model.
|
||||
@@ -1,5 +1,5 @@
|
||||
-- MARCRO for count the cummulative_activity_number
|
||||
{% set funnel_activity = ['register','email_verified','identity_documents_uploaded','identity_documents_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%}
|
||||
{% set funnel_activity = ['register','email_verified','identity_document_uploaded','identity_document_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%}
|
||||
|
||||
-- IMPORT
|
||||
WITH companies AS (
|
||||
@@ -69,12 +69,13 @@ company_first_time_order_date AS (
|
||||
|
||||
|
||||
|
||||
--start defind funnel
|
||||
--start defind funnel activity
|
||||
repeat_completed_order_after_60days_activities AS (
|
||||
SELECT
|
||||
clean_company_transaction_orders.company_id,
|
||||
0 AS is_data_cleansing_generate_row,
|
||||
'repeat_completed_order_after_60days' AS activity,
|
||||
9 AS activtity_rank,
|
||||
MIN(clean_company_transaction_orders.created_at) AS activity_datetime
|
||||
|
||||
FROM clean_company_transaction_orders
|
||||
@@ -94,6 +95,7 @@ repeat_completed_order_after_30days_activities AS (
|
||||
clean_company_transaction_orders.company_id,
|
||||
0 AS is_data_cleansing_generate_row,
|
||||
'repeat_completed_order_after_30days' AS activity,
|
||||
8 AS activtity_rank,
|
||||
MIN(clean_company_transaction_orders.created_at) AS activity_datetime
|
||||
|
||||
FROM clean_company_transaction_orders
|
||||
@@ -113,6 +115,7 @@ first_time_order_completed_activities AS (
|
||||
clean_company_transaction_orders.company_id,
|
||||
0 AS is_data_cleansing_generate_row,
|
||||
'first_time_order_completed' AS activity,
|
||||
7 AS activtity_rank,
|
||||
MIN(clean_company_transaction_orders.created_at) AS activity_datetime
|
||||
|
||||
FROM clean_company_transaction_orders
|
||||
@@ -129,6 +132,7 @@ first_time_order_activities AS (
|
||||
clean_company_transaction_orders.company_id,
|
||||
0 AS is_data_cleansing_generate_row,
|
||||
'first_time_order' AS activity,
|
||||
6 AS activtity_rank,
|
||||
MIN(clean_company_transaction_orders.created_at) AS activity_datetime
|
||||
|
||||
FROM clean_company_transaction_orders
|
||||
@@ -145,6 +149,7 @@ first_time_booking_activities AS (
|
||||
IFNULL( unclean_first_time_booking_activities.company_id , first_time_order_activities.company_id ) AS company_id,
|
||||
IFF( unclean_first_time_booking_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
|
||||
'first_time_booking' AS activity,
|
||||
5 AS activtity_rank,
|
||||
IFNULL( unclean_first_time_booking_activities.activity_datetime , first_time_order_activities.activity_datetime ) AS activity_datetime
|
||||
|
||||
FROM
|
||||
@@ -168,19 +173,20 @@ first_time_booking_activities AS (
|
||||
),
|
||||
|
||||
|
||||
identity_documents_verified_activities AS (
|
||||
identity_document_verified_activities AS (
|
||||
SELECT
|
||||
IFNULL( unclean_identity_documents_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id,
|
||||
IFF( unclean_identity_documents_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
|
||||
'identity_documents_verified' AS activity,
|
||||
IFNULL( unclean_identity_documents_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime
|
||||
IFNULL( unclean_identity_document_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id,
|
||||
IFF( unclean_identity_document_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
|
||||
'identity_document_verified' AS activity,
|
||||
4 AS activtity_rank,
|
||||
IFNULL( unclean_identity_document_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime
|
||||
|
||||
FROM
|
||||
(
|
||||
|
||||
SELECT
|
||||
company_id,
|
||||
'identity_documents_verified' AS activity,
|
||||
'identity_document_verified' AS activity,
|
||||
MIN(identity_created_datetime) AS activity_datetime
|
||||
|
||||
FROM company_identity_documents
|
||||
@@ -191,27 +197,28 @@ identity_documents_verified_activities AS (
|
||||
GROUP BY
|
||||
company_id
|
||||
|
||||
) AS unclean_identity_documents_verified_activities
|
||||
) AS unclean_identity_document_verified_activities
|
||||
|
||||
|
||||
FULL OUTER JOIN first_time_booking_activities
|
||||
ON (unclean_identity_documents_verified_activities.company_id = first_time_booking_activities.company_id)
|
||||
ON (unclean_identity_document_verified_activities.company_id = first_time_booking_activities.company_id)
|
||||
|
||||
),
|
||||
|
||||
identity_documents_uploaded_activities AS (
|
||||
identity_document_uploaded_activities AS (
|
||||
SELECT
|
||||
IFNULL( unclean_identity_documents_uploaded_activities.company_id , identity_documents_verified_activities.company_id ) AS company_id,
|
||||
IFF( unclean_identity_documents_uploaded_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
|
||||
'identity_documents_uploaded' AS activity,
|
||||
IFNULL( unclean_identity_documents_uploaded_activities.activity_datetime , identity_documents_verified_activities.activity_datetime ) AS activity_datetime
|
||||
IFNULL( unclean_identity_document_uploaded_activities.company_id , identity_document_verified_activities.company_id ) AS company_id,
|
||||
IFF( unclean_identity_document_uploaded_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
|
||||
'identity_document_uploaded' AS activity,
|
||||
3 AS activtity_rank,
|
||||
IFNULL( unclean_identity_document_uploaded_activities.activity_datetime , identity_document_verified_activities.activity_datetime ) AS activity_datetime
|
||||
|
||||
FROM
|
||||
(
|
||||
|
||||
SELECT
|
||||
company_id,
|
||||
'identity_documents_uploaded' AS activity,
|
||||
'identity_document_uploaded' AS activity,
|
||||
MIN(identity_created_datetime) AS activity_datetime
|
||||
|
||||
FROM company_identity_documents
|
||||
@@ -219,19 +226,20 @@ identity_documents_uploaded_activities AS (
|
||||
GROUP BY
|
||||
company_id
|
||||
|
||||
) AS unclean_identity_documents_uploaded_activities
|
||||
) AS unclean_identity_document_uploaded_activities
|
||||
|
||||
|
||||
FULL OUTER JOIN identity_documents_verified_activities
|
||||
ON (unclean_identity_documents_uploaded_activities.company_id = identity_documents_verified_activities.company_id)
|
||||
FULL OUTER JOIN identity_document_verified_activities
|
||||
ON (unclean_identity_document_uploaded_activities.company_id = identity_document_verified_activities.company_id)
|
||||
),
|
||||
|
||||
email_verified_activities AS (
|
||||
SELECT
|
||||
IFNULL( unclean_email_verified_activities.company_id , identity_documents_uploaded_activities.company_id ) AS company_id,
|
||||
IFNULL( unclean_email_verified_activities.company_id , identity_document_uploaded_activities.company_id ) AS company_id,
|
||||
IFF( unclean_email_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
|
||||
'email_verified' AS activity,
|
||||
IFNULL( unclean_email_verified_activities.activity_datetime , identity_documents_uploaded_activities.activity_datetime ) AS activity_datetime
|
||||
2 AS activtity_rank,
|
||||
IFNULL( unclean_email_verified_activities.activity_datetime , identity_document_uploaded_activities.activity_datetime ) AS activity_datetime
|
||||
|
||||
FROM
|
||||
(
|
||||
@@ -249,8 +257,8 @@ email_verified_activities AS (
|
||||
company_id
|
||||
) AS unclean_email_verified_activities
|
||||
|
||||
FULL OUTER JOIN identity_documents_uploaded_activities
|
||||
ON (unclean_email_verified_activities.company_id = identity_documents_uploaded_activities.company_id)
|
||||
FULL OUTER JOIN identity_document_uploaded_activities
|
||||
ON (unclean_email_verified_activities.company_id = identity_document_uploaded_activities.company_id)
|
||||
|
||||
),
|
||||
|
||||
@@ -259,6 +267,7 @@ register_activities AS (
|
||||
IFNULL( unclean_register_activities.company_id , email_verified_activities.company_id ) AS company_id,
|
||||
IFF( unclean_register_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
|
||||
'register' AS activity,
|
||||
1 AS activtity_rank,
|
||||
IFNULL( unclean_register_activities.activity_datetime , email_verified_activities.activity_datetime ) AS activity_datetime
|
||||
|
||||
FROM
|
||||
@@ -310,24 +319,30 @@ union_all_activities_flag_migrated_company AS (
|
||||
|
||||
--FINAL
|
||||
|
||||
sem_exchange__company_funnels AS (
|
||||
sem_exchange__company_conversion_funnels AS (
|
||||
SELECT
|
||||
company_id,
|
||||
is_data_cleansing_generate_row,
|
||||
is_migrated_company,
|
||||
activity,
|
||||
activtity_rank,
|
||||
IFF( ROW_NUMBER() OVER (PARTITION BY company_id ORDER BY activtity_rank DESC) = 1,
|
||||
1,
|
||||
0) AS latest_activity,
|
||||
|
||||
activity_datetime,
|
||||
|
||||
{% for activity in funnel_activity %}
|
||||
{% for activity_type in funnel_activity %}
|
||||
|
||||
count(iff(activity = '{{activity}}', company_id, NULL))
|
||||
OVER (ORDER BY activity_datetime) AS cumulative_{{activity}}_activity,
|
||||
count(iff(activity = '{{activity_type}}', company_id, NULL))
|
||||
OVER (ORDER BY activity_datetime) AS cumulative_{{activity_type}}_activity,
|
||||
|
||||
{% endfor %}
|
||||
|
||||
{% for activity in funnel_activity %}
|
||||
{% for activity_type in funnel_activity %}
|
||||
|
||||
count(iff(activity = '{{activity}}' AND is_migrated_company = 0, company_id, NULL))
|
||||
OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity}}_activity,
|
||||
count(iff(activity = '{{activity_type}}' AND is_migrated_company = 0, company_id, NULL))
|
||||
OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity_type}}_activity,
|
||||
|
||||
{% endfor %}
|
||||
|
||||
@@ -338,4 +353,4 @@ sem_exchange__company_funnels AS (
|
||||
ORDER BY activity_datetime
|
||||
)
|
||||
|
||||
SELECT * FROM sem_exchange__company_funnels
|
||||
SELECT * FROM sem_exchange__company_conversion_funnels
|
||||
@@ -1,46 +0,0 @@
|
||||
-- IMPORTS
|
||||
WITH activity_logs AS (
|
||||
SELECT * FROM {{ ref('base_exchange__activity_logs') }}
|
||||
),
|
||||
|
||||
|
||||
-- LOGIC
|
||||
activity_logs_listout AS (
|
||||
|
||||
SELECT
|
||||
activity_logs.id,
|
||||
activity_logs.log_name,
|
||||
activity_logs.description,
|
||||
activity_logs.subject_id,
|
||||
activity_logs.subject_type,
|
||||
activity_logs.causer_id,
|
||||
activity_logs.causer_type,
|
||||
activity_logs.properties,
|
||||
activity_logs.created_at,
|
||||
activity_logs.updated_at,
|
||||
CURRENT_TIMESTAMP() AS _dbt_ran_at
|
||||
|
||||
FROM activity_logs
|
||||
),
|
||||
|
||||
|
||||
-- FINAL
|
||||
final__stg_exchange__activity_logs AS (
|
||||
|
||||
SELECT
|
||||
id,
|
||||
log_name,
|
||||
description,
|
||||
subject_id,
|
||||
subject_type,
|
||||
causer_id,
|
||||
causer_type,
|
||||
properties,
|
||||
created_at,
|
||||
updated_at,
|
||||
_dbt_ran_at
|
||||
|
||||
FROM activity_logs_listout
|
||||
)
|
||||
|
||||
SELECT * FROM final__stg_exchange__activity_logs
|
||||
@@ -0,0 +1,10 @@
|
||||
-- IMPORTS
|
||||
WITH wallets AS (
|
||||
SELECT * FROM {{ ref('base_exchange__wallets') }}
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
-- FINAL
|
||||
SELECT * FROM wallets
|
||||
@@ -0,0 +1,7 @@
|
||||
-- IMPORTS
|
||||
with wallet_logs as (select * from {{ ref("base_exchange__wallet_logs") }})
|
||||
|
||||
-- FINAL
|
||||
select *
|
||||
from wallet_logs
|
||||
order by wallet_id, updated_at asc
|
||||
@@ -0,0 +1,46 @@
|
||||
-- IMPORTS
|
||||
WITH activity_logs AS (
|
||||
SELECT * FROM {{ source('src_shipping_mysql', 'activity_log') }}
|
||||
),
|
||||
|
||||
|
||||
-- LOGIC
|
||||
activity_logs_rename AS (
|
||||
|
||||
SELECT
|
||||
activity_logs.id,
|
||||
activity_logs.log_name,
|
||||
activity_logs.description,
|
||||
activity_logs.subject_id,
|
||||
activity_logs.subject_type,
|
||||
activity_logs.causer_id,
|
||||
activity_logs.causer_type,
|
||||
activity_logs.properties,
|
||||
activity_logs.created_at AS created_datetime,
|
||||
activity_logs.updated_at AS updated_datetime,
|
||||
current_timestamp() AS _dbt_ran_at
|
||||
|
||||
FROM activity_logs
|
||||
),
|
||||
|
||||
|
||||
-- FINAL
|
||||
final__base_exchange__activity_logs AS (
|
||||
|
||||
SELECT
|
||||
id,
|
||||
log_name,
|
||||
description,
|
||||
subject_id,
|
||||
subject_type,
|
||||
causer_id,
|
||||
causer_type,
|
||||
properties,
|
||||
created_datetime,
|
||||
updated_datetime,
|
||||
_dbt_ran_at
|
||||
|
||||
FROM activity_logs_rename
|
||||
)
|
||||
|
||||
SELECT * FROM final__base_exchange__activity_logs
|
||||
@@ -0,0 +1,23 @@
|
||||
{% docs src_exchange_mysql %}
|
||||
|
||||
Data is extract from CIEF Selfhost MYSQL `exchange_production`, using custom python code.
|
||||
|
||||
- `_source_loaded_at` will add into the raw data when extract
|
||||
```YML
|
||||
loaded_at_field: _source_loaded_at
|
||||
freshness:
|
||||
warn_after: {count: 12, period: hour}
|
||||
error_after: {count: 24, period: hour}
|
||||
```
|
||||
|
||||
- The raw data will use copy into snowflake in 2AM everyday
|
||||
|
||||
- Depend on target.name to determin which snowflake raw schema
|
||||
|
||||
```JINJA
|
||||
{%- if target.name == "prod" -%} PROD_CIEF_RAW_DB
|
||||
{%- else -%} DEV_CIEF_RAW_DB
|
||||
{%- endif -%}
|
||||
```
|
||||
|
||||
{% enddocs %}
|
||||
@@ -0,0 +1,6 @@
|
||||
version: 2
|
||||
|
||||
models:
|
||||
- name: stg_exchange__activity_logs
|
||||
|
||||
- name: stg_exchange__announcments
|
||||
@@ -0,0 +1,619 @@
|
||||
version: 2
|
||||
|
||||
sources:
|
||||
- name: src_exchange_mysql
|
||||
description: '{{ doc("src_exchange_mysql") }}'
|
||||
database: |
|
||||
{%- if target.name == "prod" -%} PROD_CIEF_RAW_DB
|
||||
{%- else -%} DEV_CIEF_RAW_DB
|
||||
{%- endif -%}
|
||||
schema: EXCHANGE_MYSQL
|
||||
loader: custom_python_code
|
||||
loaded_at_field: _source_loaded_at
|
||||
freshness:
|
||||
warn_after: {count: 12, period: hour}
|
||||
error_after: {count: 24, period: hour}
|
||||
meta:
|
||||
owner: "@yam"
|
||||
model_maturity: prod
|
||||
tags:
|
||||
- exchange
|
||||
- daily
|
||||
|
||||
|
||||
tables:
|
||||
- name: activity_log
|
||||
description: Contains most table's activity log that CUD (created, updated, deleted) in www.exchange.cief-malaysia.com
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'activity_log'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
- name: description
|
||||
description: system CUD activity list down
|
||||
tests:
|
||||
- accepted_values:
|
||||
values: ['created', 'updated', 'deleted']
|
||||
|
||||
- name: subject_id
|
||||
description: The exchange CUD table's id, need to refer subject_type column to know what table is CUD in the row
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: subject_type
|
||||
description: What table that CUD using in the row
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: causer_id
|
||||
description: Need to refer causer_type column to know who is CUD in the row. it is not well maintain there is some NULL
|
||||
|
||||
- name: causer_type
|
||||
description: Who is CUD the row
|
||||
|
||||
- name: properties
|
||||
description: Json data format that contain attributes of the activity, it is not well maintain, subject (Country, ServiceType, CurrencyRate. UserEmailVerification, File) have properties only
|
||||
|
||||
- name: created_at
|
||||
description: Datetime that CUD activity occur
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: updated_at
|
||||
description: Exactly same with created_at, redundant column
|
||||
|
||||
tests:
|
||||
- dbt_expectations.expect_column_pair_values_to_be_equal:
|
||||
column_A: updated_at
|
||||
column_B: created_at
|
||||
|
||||
|
||||
- name: addresses
|
||||
description: Addresses that input when user register
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'addresses'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
- name: company_id
|
||||
description: Foreign keys for 'companies'
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: country_id
|
||||
description: Foreign keys for 'countries'
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: state_id
|
||||
description: Foreign keys for 'states'
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: district_id
|
||||
description: Foreign keys for 'districts'
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: postcode
|
||||
description: The number/string that user input
|
||||
tests:
|
||||
- not_null
|
||||
- dbt_expectations.expect_column_values_to_be_of_type:
|
||||
column_type: integer
|
||||
|
||||
- name: street_one
|
||||
description: The string that user input
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: street_two
|
||||
description: The string that user input
|
||||
|
||||
- name: billing
|
||||
description: If mark 1 means it is billing address, 0 means is not billing address, every company should have 1 billing address
|
||||
tests:
|
||||
- dbt_expectations.expect_column_max_to_be_between: #Every company should have a billing address
|
||||
min_value: 1
|
||||
max_value: 1
|
||||
group_by: [company_id]
|
||||
- dbt_expectations.expect_column_sum_to_be_between: #1 company should have 1 billing address only
|
||||
min_value: 1
|
||||
max_value: 1
|
||||
group_by: [company_id]
|
||||
row_condition: "deleted_at is null"
|
||||
|
||||
- name: deleted_at
|
||||
description: System generate datetime that address was deleted
|
||||
|
||||
- name: created_at
|
||||
description: System generate datetime that address was created
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: updated_at
|
||||
description: System generate datetime that address was updated
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
|
||||
- name: announcement_segment
|
||||
description: Superadmin / Operator guys will segment the announcment, but normally they will leave it default. 2 FK combine to becoem a relation table.
|
||||
columns:
|
||||
- name: announcement_id
|
||||
description: Primary key for 'announcement_segment' & Foreign key for 'announcements'
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: segment_id
|
||||
description: Primary key for 'announcement_segment' & Foreign key for 'segments'
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: created_at
|
||||
description: Abandon column
|
||||
tests:
|
||||
- dbt_expectations.expect_column_values_to_be_null
|
||||
|
||||
- name: updated_at
|
||||
description: Abandon column
|
||||
tests:
|
||||
- dbt_expectations.expect_column_values_to_be_null
|
||||
|
||||
|
||||
- name: announcements
|
||||
description: Announcements that show in our website which create by superadmin
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'announcements'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
- name: title
|
||||
description: title of announcement, using string
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: description
|
||||
description: description of announcement, using longtext, have <br> html for break
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: starting_on
|
||||
description: announcement starting datetime
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: ending_on
|
||||
description: announcement ending datetime
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: is_all_day
|
||||
description: column might not in use, system auto generate
|
||||
tests:
|
||||
- not_null
|
||||
- accepted_values:
|
||||
values: [0]
|
||||
|
||||
- name: duration
|
||||
description: column might not in use, system auto generate
|
||||
tests:
|
||||
- not_null
|
||||
- accepted_values:
|
||||
values: [0]
|
||||
|
||||
- name: is_recurring
|
||||
description: column might not in use, system auto generate
|
||||
tests:
|
||||
- not_null
|
||||
- accepted_values:
|
||||
values: [0]
|
||||
|
||||
- name: recurrence_pattern
|
||||
description: column might not in use, system auto generate
|
||||
tests:
|
||||
- dbt_expectations.expect_column_values_to_be_null
|
||||
|
||||
- name: deleted_at
|
||||
description: system generate datetime that announcment was deleted
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: created_at
|
||||
description: system generate datetime that announcment was created
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: updated_at
|
||||
description: system generate datetime that announcment was updated
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
|
||||
- name: bank_logs
|
||||
description: Abandon table
|
||||
config:
|
||||
enabled: false
|
||||
|
||||
|
||||
- name: banks
|
||||
description: Table which store banks details.
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'banks'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
- name: company_id
|
||||
description: Foreign keys for 'companies'
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: reference
|
||||
description: A string which write by user/company for the bank transfer reference or other remind ussage
|
||||
|
||||
- name: holder_name
|
||||
description: Bank holder name
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: account_no
|
||||
description: Bank account number
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
|
||||
|
||||
|
||||
- name: booking_logs
|
||||
description: Raw `bookings` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'booking_logs'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: bookings
|
||||
description: Raw `bookings` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'bookings'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: companies
|
||||
description: Raw `companies` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'companies'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: contacts
|
||||
description: Raw `contacts` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'contacts'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: countries
|
||||
description: Raw `countries` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'countries'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: currencies
|
||||
description: Raw `currencies` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'currencies'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: currency_logs
|
||||
description: Raw `currency_logs` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'currency_logs'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: currency_rate_logs
|
||||
description: Raw `currency_rate_logs` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'currency_rate_logs'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: currency_rates
|
||||
description: Raw `currency_rates` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'currency_rates'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: districts
|
||||
description: Raw `districts` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'districts'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: documents
|
||||
description: Raw `documents` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'documents'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: employees
|
||||
description: Raw `employees` table from Snowflake
|
||||
columns:
|
||||
- name: company_id
|
||||
description: Primary key for 'employees'
|
||||
|
||||
|
||||
- name: failed_jobs
|
||||
description: Raw `failed_jobs` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'failed_jobs'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: files
|
||||
description: Raw `files` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'files'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: group_transactions
|
||||
description: Raw `group_transactions` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'group_transactions'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: groups
|
||||
description: Raw `groups` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'groups'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: jobs
|
||||
description: Raw `jobs` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'jobs'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: migrations
|
||||
description: Raw `migrations` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'migrations'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: model_has_roles
|
||||
description: Raw `model_has_roles` table from Snowflake
|
||||
columns:
|
||||
- name: role_id
|
||||
description: Primary key for 'model_has_roles'
|
||||
|
||||
|
||||
- name: notifications
|
||||
description: Raw `notifications` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'notifications'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: password_resets
|
||||
description: Raw `password_resets` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'password_resets'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: permissions
|
||||
description: Raw `permissions` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'permissions'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: role_has_permissions
|
||||
description: Raw `role_has_permissions` table from Snowflake
|
||||
columns:
|
||||
- name: permission_id
|
||||
description: Primary key for 'role_has_permissions'
|
||||
|
||||
|
||||
- name: roles
|
||||
description: Raw `roles` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'roles'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: segment_companies
|
||||
description: Raw `segment_companies` table from Snowflake
|
||||
columns:
|
||||
- name: segment_id
|
||||
description: Primary key for 'segment_companies'
|
||||
|
||||
|
||||
- name: segment_constants
|
||||
description: Raw `segment_constants` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'segment_constants'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: segments
|
||||
description: Raw `segments` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'segments'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: service_types
|
||||
description: Raw `service_types` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'service_types'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
description: Raw `states` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'states'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: transaction_detail
|
||||
description: Raw `transaction_detail` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'transaction_detail'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: transaction_logs
|
||||
description: Raw `transaction_logs` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'transaction_logs'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: transactions
|
||||
description: Raw `transactions` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'transactions'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: user_email_verifications
|
||||
description: Raw `user_email_verifications` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'user_email_verifications'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: users
|
||||
description: Raw `users` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'users'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: wallet_logs
|
||||
description: Raw `wallet_logs` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'wallet_logs'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
|
||||
- name: wallets
|
||||
description: Raw `wallets` table from Snowflake
|
||||
columns:
|
||||
- name: id
|
||||
description: Primary key for 'wallets'
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
@@ -0,0 +1,94 @@
|
||||
-- IMPORTS
|
||||
WITH activity_logs AS (
|
||||
SELECT * FROM {{ source('src_exchange_mysql', 'activity_log') }}
|
||||
),
|
||||
|
||||
|
||||
-- LOGIC
|
||||
activity_logs_rename AS (
|
||||
|
||||
SELECT
|
||||
activity_logs.id,
|
||||
activity_logs.log_name,
|
||||
activity_logs.description,
|
||||
activity_logs.subject_id,
|
||||
activity_logs.subject_type,
|
||||
activity_logs.causer_id,
|
||||
activity_logs.causer_type,
|
||||
activity_logs.properties,
|
||||
activity_logs.created_at,
|
||||
activity_logs.updated_at,
|
||||
current_timestamp() AS _dbt_ran_at
|
||||
|
||||
FROM activity_logs
|
||||
),
|
||||
|
||||
|
||||
-- FINAL
|
||||
final__base_exchange__activity_logs AS (
|
||||
|
||||
SELECT
|
||||
id,
|
||||
log_name,
|
||||
description,
|
||||
subject_id,
|
||||
subject_type,
|
||||
causer_id,
|
||||
causer_type,
|
||||
properties,
|
||||
created_at,
|
||||
updated_at,
|
||||
_dbt_ran_at
|
||||
|
||||
FROM activity_logs_rename
|
||||
)
|
||||
|
||||
SELECT * FROM final__base_exchange__activity_logs
|
||||
|
||||
|
||||
-- IMPORTS
|
||||
WITH activity_logs AS (
|
||||
SELECT * FROM {{ ref('base_exchange__activity_logs') }}
|
||||
),
|
||||
|
||||
|
||||
-- LOGIC
|
||||
activity_logs_listout AS (
|
||||
|
||||
SELECT
|
||||
activity_logs.id,
|
||||
activity_logs.log_name,
|
||||
activity_logs.description,
|
||||
activity_logs.subject_id,
|
||||
activity_logs.subject_type,
|
||||
activity_logs.causer_id,
|
||||
activity_logs.causer_type,
|
||||
activity_logs.properties,
|
||||
activity_logs.created_at,
|
||||
activity_logs.updated_at,
|
||||
CURRENT_TIMESTAMP() AS _dbt_ran_at
|
||||
|
||||
FROM activity_logs
|
||||
),
|
||||
|
||||
|
||||
-- FINAL
|
||||
final__stg_exchange__activity_logs AS (
|
||||
|
||||
SELECT
|
||||
id,
|
||||
log_name,
|
||||
description,
|
||||
subject_id,
|
||||
subject_type,
|
||||
causer_id,
|
||||
causer_type,
|
||||
properties,
|
||||
created_at,
|
||||
updated_at,
|
||||
_dbt_ran_at
|
||||
|
||||
FROM activity_logs_listout
|
||||
)
|
||||
|
||||
SELECT * FROM final__stg_exchange__activity_logs
|
||||
Reference in New Issue
Block a user