cief/dbt_cloud
1
0
Fork 0
mirror of https://gitlab.com/cief-data/dbt_cloud.git synced 2026-08-19 04:14:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

Undo dbt

Browse Source
This commit is contained in:
Yam ZhengLim
2023-04-11 22:13:15 +08:00
parent 819776bee1
commit b3373e5b58
70 changed files with 147 additions and 1831 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+12 -181
View File
@@ -1,184 +1,15 @@
# DBT governance
Thid markdown document will specify the rules that need to follow by every developer.
<br>
Short naming converntion lists that need to follow
| model_type Shortcut | Full name |
| ------------------- | ------ |
| seed | Seed |
| src | Source |
| snap | Snapshot |
| stg | Stage |
| int | Intermediate |
| fct | Fact |
| dim | Dimension |
| rep | Report |
| sem | Semantic |
Welcome to your new dbt project!
| model_name Shortcut | Full name | Use case |
| ------------------- | ------ | ------ |
| brg | Brigde | relationship table |
| log | Log | log data |
### Using the starter project
Try running the following commands:
- dbt run
- dbt test
## CIEF General Rule
- Snowflake SQL use to write dbt SQL code
- Star schema is use
- First week of the day is start at Monday, and last week of the day will be Sunday
- Fiscal 1st quarter is start from Febuary
- Timezone: Kuala Lumpur / Malaysia (GMT +8)
- Some status/type is not input in raw database (You should found in seed)
<br/>
## DBT Rule
### Structure of the dbt model layer
- Analyses
1. Analyses layer will not create in Snowflake
2. You can write some adhoc queries
3. Test some SQL code before write in Models
<br/>
- Macros
1. Jinja function that reuse in other models
<br/>
- Models
- Staging
- System name
1. Materialise: `View`
2. 1-to-1 relationship (or mapping) to source tables.
3. Column renaming
4. Column remove
5. Column data renaming, such as (status integer to string)
6. Data input error cleansing
7. Check for raw data freshness
8. Data type transformation
9. Data split and merge
- Source
1. Source will write in YML
2. Check Freshness
3. Check duplicate id
4. Check null id
- Intermediate
1. Materialise: `Ephemerally`
2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want
3. Be referenced repeatedly in more than one model
4. Isolating complex operations
- Marts/warehouse
1. Materialise: `Table`
2. Store fact and dimension models
- Marts/reporting
1. Materialise: `Table`
2. Store custom reports
<br/>
- Seeds
1. Store custom dataset in csv format
2. The dataset must not change frequently
<br/>
- Snapshots
1. DBT build-in SCD type 2 function
2. Prefer to use after the source table and before the staging layer
<br/>
- Tests
1. Can write some yml test case
<br/>
### File Naming Rules
- File names must be unique
- Each sql/python file must start with [`model_type`]_[`system_name`]__[`model_name`]s.sql/py
- The model should be name as plural. (eg: stg_exchange__orders.sql)
<br/>
### Column Naming Rules
- If array datatype, the column naming must be plural (eg: ids, messages)
- Schema, table and column names should be in `snake_case`.
- Limit use of abbreviations that are related to domain knowledge. An onboarding
employee will understand `current_order_status` better than `current_os`.
- Use names based on the _business_ terminology, rather than the source terminology.
- Each model should have a primary key that can identify the unique row, and should be named `<object>_id`, e.g. `account_id` – this makes it easier to know what `id` is being referenced in downstream joined models.
- If a surrogate key is created, it should be named `<object>_sk`.
- For `base` or `staging` models, columns should be ordered in categories, where identifiers are first and date/time fields are at the end.
Example:
```sql
transformed as (
select
-- ids
order_id,
customer_id,
-- dimensions
order_status,
is_shipped,
-- measures
order_total,
-- date/times
created_at,
updated_at,
-- metadata
_sdc_batched_at
from source
)
```
- Date/time columns should be named according to these conventions:
- Timestamps: `<event>_datetime`
Example: `created_datetime`
- Dates: `<event>_date`
Example: `created_date`
- Booleans should be prefixed with `is_` or `has_`.
Example: `is_active_customer` and `has_admin_access`
- Price/revenue fields should be in decimal currency (e.g. `19.99` for $19.99; many app databases store prices as integers in cents). If non-decimal currency is used, indicate this with suffix, e.g. `price_in_cents`.
<br/>
### SQL Coding Rule
- The SQL clause **must** be UPPERCASE (eg. `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `LIMIT`, `WITH`, `AS`, `SUM`, `PARTITION OVER`, `DIV0`, `LEFT JOIN`, `ON` ....)
- Try to avoid using `EXCLUDE` in the SQL
- Avoid using multi layer subquery, and try to use CTE, subquery must not more than 1 layer
- Each model structure must contain with
```sql
--IMPORT
WITH [table_names] AS (
SELECT * FROM {{ ref('file_names')}}
),
--LOGIC
[logic_names] AS (
.....
),
--FINAL
final__[table_names] AS (
)
SELECT * FROM final__[final_table_names]
```
### Testing
- At a minimum, `unique` and `not_null` tests should be applied to the expected primary key of each model.
### Resources:
- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
- Join the [dbt community](http://community.getbdt.com/) to learn from other analytics engineers
- Find [dbt events](https://events.getdbt.com) near you
- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
analyses/exchange/company_conversion_v2.sql
+31 -46
View File
@@ -1,5 +1,5 @@
-- MARCRO for count the cummulative_activity_number
{% set funnel_activity = ['register','email_verified','identity_document_uploaded','identity_document_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%}
{% set funnel_activity = ['register','email_verified','identity_documents_uploaded','identity_documents_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%}
-- IMPORT
WITH companies AS (
@@ -69,13 +69,12 @@ company_first_time_order_date AS (
--start defind funnel activity
--start defind funnel
repeat_completed_order_after_60days_activities AS (
SELECT
clean_company_transaction_orders.company_id,
0 AS is_data_cleansing_generate_row,
'repeat_completed_order_after_60days' AS activity,
9 AS activtity_rank,
MIN(clean_company_transaction_orders.created_at) AS activity_datetime
FROM clean_company_transaction_orders
@@ -95,7 +94,6 @@ repeat_completed_order_after_30days_activities AS (
clean_company_transaction_orders.company_id,
0 AS is_data_cleansing_generate_row,
'repeat_completed_order_after_30days' AS activity,
8 AS activtity_rank,
MIN(clean_company_transaction_orders.created_at) AS activity_datetime
FROM clean_company_transaction_orders
@@ -115,7 +113,6 @@ first_time_order_completed_activities AS (
clean_company_transaction_orders.company_id,
0 AS is_data_cleansing_generate_row,
'first_time_order_completed' AS activity,
7 AS activtity_rank,
MIN(clean_company_transaction_orders.created_at) AS activity_datetime
FROM clean_company_transaction_orders
@@ -132,7 +129,6 @@ first_time_order_activities AS (
clean_company_transaction_orders.company_id,
0 AS is_data_cleansing_generate_row,
'first_time_order' AS activity,
6 AS activtity_rank,
MIN(clean_company_transaction_orders.created_at) AS activity_datetime
FROM clean_company_transaction_orders
@@ -149,7 +145,6 @@ first_time_booking_activities AS (
IFNULL( unclean_first_time_booking_activities.company_id , first_time_order_activities.company_id ) AS company_id,
IFF( unclean_first_time_booking_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
'first_time_booking' AS activity,
5 AS activtity_rank,
IFNULL( unclean_first_time_booking_activities.activity_datetime , first_time_order_activities.activity_datetime ) AS activity_datetime
FROM
@@ -173,20 +168,19 @@ first_time_booking_activities AS (
),
identity_document_verified_activities AS (
identity_documents_verified_activities AS (
SELECT
IFNULL( unclean_identity_document_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id,
IFF( unclean_identity_document_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
'identity_document_verified' AS activity,
4 AS activtity_rank,
IFNULL( unclean_identity_document_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime
IFNULL( unclean_identity_documents_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id,
IFF( unclean_identity_documents_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
'identity_documents_verified' AS activity,
IFNULL( unclean_identity_documents_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime
FROM
(
SELECT
company_id,
'identity_document_verified' AS activity,
'identity_documents_verified' AS activity,
MIN(identity_created_datetime) AS activity_datetime
FROM company_identity_documents
@@ -197,28 +191,27 @@ identity_document_verified_activities AS (
GROUP BY
company_id
) AS unclean_identity_document_verified_activities
) AS unclean_identity_documents_verified_activities
FULL OUTER JOIN first_time_booking_activities
ON (unclean_identity_document_verified_activities.company_id = first_time_booking_activities.company_id)
ON (unclean_identity_documents_verified_activities.company_id = first_time_booking_activities.company_id)
),
identity_document_uploaded_activities AS (
identity_documents_uploaded_activities AS (
SELECT
IFNULL( unclean_identity_document_uploaded_activities.company_id , identity_document_verified_activities.company_id ) AS company_id,
IFF( unclean_identity_document_uploaded_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
'identity_document_uploaded' AS activity,
3 AS activtity_rank,
IFNULL( unclean_identity_document_uploaded_activities.activity_datetime , identity_document_verified_activities.activity_datetime ) AS activity_datetime
IFNULL( unclean_identity_documents_uploaded_activities.company_id , identity_documents_verified_activities.company_id ) AS company_id,
IFF( unclean_identity_documents_uploaded_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
'identity_documents_uploaded' AS activity,
IFNULL( unclean_identity_documents_uploaded_activities.activity_datetime , identity_documents_verified_activities.activity_datetime ) AS activity_datetime
FROM
(
SELECT
company_id,
'identity_document_uploaded' AS activity,
'identity_documents_uploaded' AS activity,
MIN(identity_created_datetime) AS activity_datetime
FROM company_identity_documents
@@ -226,20 +219,19 @@ identity_document_uploaded_activities AS (
GROUP BY
company_id
) AS unclean_identity_document_uploaded_activities
) AS unclean_identity_documents_uploaded_activities
FULL OUTER JOIN identity_document_verified_activities
ON (unclean_identity_document_uploaded_activities.company_id = identity_document_verified_activities.company_id)
FULL OUTER JOIN identity_documents_verified_activities
ON (unclean_identity_documents_uploaded_activities.company_id = identity_documents_verified_activities.company_id)
),
email_verified_activities AS (
SELECT
IFNULL( unclean_email_verified_activities.company_id , identity_document_uploaded_activities.company_id ) AS company_id,
IFNULL( unclean_email_verified_activities.company_id , identity_documents_uploaded_activities.company_id ) AS company_id,
IFF( unclean_email_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
'email_verified' AS activity,
2 AS activtity_rank,
IFNULL( unclean_email_verified_activities.activity_datetime , identity_document_uploaded_activities.activity_datetime ) AS activity_datetime
IFNULL( unclean_email_verified_activities.activity_datetime , identity_documents_uploaded_activities.activity_datetime ) AS activity_datetime
FROM
(
@@ -257,8 +249,8 @@ email_verified_activities AS (
company_id
) AS unclean_email_verified_activities
FULL OUTER JOIN identity_document_uploaded_activities
ON (unclean_email_verified_activities.company_id = identity_document_uploaded_activities.company_id)
FULL OUTER JOIN identity_documents_uploaded_activities
ON (unclean_email_verified_activities.company_id = identity_documents_uploaded_activities.company_id)
),
@@ -267,7 +259,6 @@ register_activities AS (
IFNULL( unclean_register_activities.company_id , email_verified_activities.company_id ) AS company_id,
IFF( unclean_register_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row,
'register' AS activity,
1 AS activtity_rank,
IFNULL( unclean_register_activities.activity_datetime , email_verified_activities.activity_datetime ) AS activity_datetime
FROM
@@ -319,30 +310,24 @@ union_all_activities_flag_migrated_company AS (
--FINAL
sem_exchange__company_conversion_funnels AS (
sem_exchange__company_funnels AS (
SELECT
company_id,
is_data_cleansing_generate_row,
is_migrated_company,
activity,
activtity_rank,
IFF( ROW_NUMBER() OVER (PARTITION BY company_id ORDER BY activtity_rank DESC) = 1,
1,
0) AS latest_activity,
activity_datetime,
{% for activity_type in funnel_activity %}
{% for activity in funnel_activity %}
count(iff(activity = '{{activity_type}}', company_id, NULL))
OVER (ORDER BY activity_datetime) AS cumulative_{{activity_type}}_activity,
count(iff(activity = '{{activity}}', company_id, NULL))
OVER (ORDER BY activity_datetime) AS cumulative_{{activity}}_activity,
{% endfor %}
{% for activity_type in funnel_activity %}
{% for activity in funnel_activity %}
count(iff(activity = '{{activity_type}}' AND is_migrated_company = 0, company_id, NULL))
OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity_type}}_activity,
count(iff(activity = '{{activity}}' AND is_migrated_company = 0, company_id, NULL))
OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity}}_activity,
{% endfor %}
@@ -353,4 +338,4 @@ sem_exchange__company_conversion_funnels AS (
ORDER BY activity_datetime
)
SELECT * FROM sem_exchange__company_conversion_funnels
SELECT * FROM sem_exchange__company_funnels
models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_last_transaction_sequence.sql
-8
View File
@@ -1,8 +0,0 @@
with
wallet_log_sequence as (
select * from {{ ref("int_exchange__wallet_transaction_sequence") }}
)
select max(operation_sequence) as last_transaction_id, wallet_id
from wallet_log_sequence
group by wallet_id
models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_log_credit_amount.sql
-56
View File
@@ -1,56 +0,0 @@
-- IMPORTS
with
wallet_logs as (select * from {{ ref("stg_exchange__wallet_logs") }}),
-- LOGIT
wallet_credit_amount as (
select
*,
lag(amount) over (
partition by wallet_id order by updated_at
) as previous_wallet_balance,
amount - lag(amount) over (
partition by wallet_id order by updated_at
) as credited_amount
from wallet_logs
),
wallet_credit_amount_reformed as (
select
id,
wallet_id,
previous_wallet_balance,
amount as current_wallet_balance,
iff(
equal_null(previous_wallet_balance, null), amount, credited_amount
) as credited_amount,
owner_type,
owner_id,
code,
currency_id,
deleted_at,
created_at,
updated_at,
_dbt_ran_at
from wallet_credit_amount
),
final_wallet_credit_amount as (
select
id,
wallet_id,
previous_wallet_balance,
current_wallet_balance,
credited_amount,
iff(credited_amount > 0, 'CREDIT', 'DEBIT') as credit_status,
owner_type,
owner_id,
code,
currency_id,
deleted_at,
created_at,
updated_at,
_dbt_ran_at
from wallet_credit_amount_reformed
)
select *
from final_wallet_credit_amount
models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_transaction_sequence.sql
-13
View File
@@ -1,13 +0,0 @@
with
wallet_credit_log as (
select * from {{ ref("int_exchange__wallet_log_credit_amount") }}
),
wallet_credit_with_sequence as (
select
dense_rank() over (
partition by wallet_id order by updated_at
) as operation_sequence,
*
from wallet_credit_log
)
select * from wallet_credit_with_sequence
models/exchange/intermediate/wallet_log_creadit_amount/int_exchange_wallet_final_log_sequence.sql
-24
View File
@@ -1,24 +0,0 @@
with
wallet_log_sequence as (
select * from {{ ref("int_exchange__wallet_transaction_sequence") }}
),
wallet_last_log as (
select * from {{ ref("int_exchange__wallet_last_transaction_sequence") }}
)
select
s.id,
s.wallet_id,
s.current_wallet_balance,
s.credit_status,
s.owner_type,
s.owner_id,
s.code,
s.currency_id,
s.deleted_at,
s.created_at,
s.updated_at,
s._dbt_ran_at
from wallet_log_sequence s
inner join wallet_last_log l
where s.wallet_id = l.wallet_id and s.operation_sequence = l.last_transaction_id
models/exchange/marts/reporting/rep_exchange__daily_orders/test_exchange__daily_orders.yml
+58 -71
View File
@@ -2,51 +2,27 @@ version: 2
models:
- name: rep_exchange__daily_orders
# @YAM: This is the test code that I write
# @YAM: I already do some space in the YML, but not all the stuff is clean, please clean the code, and makesure it is easy readable.
# columns:
# - name: base_value
# description: test base value combine with dbt_utils
# tests:
# - not_null
# - dbt_utils.expression_is_true:
# expression: " >= 0"
tests:
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: order_completed_datetime
column_B: order_created_datetime
or_equal: True
- dbt_utils.expression_is_true:
expression: "COST_TRANSACTION_VALUE_RM + COST_TRANSACTION_TAX_RM + COST_TRANSACTION_SERVICE_CHARGE_RM < 2*(TRANSACTION_VALUE_RM + TRANSACTION_BASE_TAX + TRANSACTION_TAX_RM)"
severity: warn
columns:
- name: ID
- name: id
description: transaction orders id
tests:
- unique
- not_null
- name: BOOKING_ID
- name: booking_id
description: transaction orders id
tests:
- not_null
- relationships:
to: ref('base_exchange__bookings')
field: id
- name: company_marking_id
description: transaction orders id
tests:
- not_null
- name: SERVICE_TYPE
- name: service_type
description: type of service
tests:
- not_null
- name: SERVICE_TYPE_NAME
- name: service_type_name
description: name of type of service
tests:
- not_null
@@ -59,55 +35,55 @@ models:
- name: seller_company_id
description: seller company id
tests:
- not_null
- not_null
- name: bank_id
description: bank id
tests:
- not_null
- not_null
- name: company_id
description: user id
tests:
- not_null
- name: marking_id
description: marking id
- not_null
- name: company_marking_id
description: company marking id
tests:
- relationships:
to: ref('base_exchange_companies')
field: id
to: ref('base_exchange__companies')
field: marking_id
- name: company_type
description: type of company
tests:
- not_null
- name: COMPANY_TYPE_NAME
- not_null
- name: company_type_name
description: name type of company
tests:
- accepted_values:
values:
- PERSONAL
- CORPORATE
- CORPORATE
- name: company_business_type
description: company business type
tests:
- not_null
- not_null
- name: company_business_type_name
description: company business type name
tests:
- accepted_values:
values:
- IMPORTER
- name: company_segment_id
description: company segment id
tests:
- not_null
- not_null
- name: company_segment_name
description: company segment name
tests:
@@ -116,109 +92,120 @@ models:
- PLATINIUM MEMBER
- GOLD MEMBER
- STANDARD SEGMENT
- name: cost_id
description: cost id
tests:
- relationships:
to: ref('base_exchange__transactions')
field: id
- name: base_value
description: test base value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: quote_value
description: test quote value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: transaction_value_rm
description: transaction value rm combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: transaction_base_tax
description: test transaction base tax combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: base_service_charge
description: test base service charge combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: transaction_tax_rm
description: test transaction tax rm combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: cost_base_value
description: test cost base value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: COST_BASE_TO_QUOTE_CURRENCY_EXCHANGE_RATE
description: test COST BASE TO QUOTE CURRENCY EXCHANGE RATE combine with dbt_utils
- name: cost_base_to_quote_currency_exchange_rate
description: test cost base to quote currency exchange rate combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: COST_QUOTE_VALUE
description: test COST QUOTE VALUE combine with dbt_utils
- name: cost_quote_value
description: test cost quote value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: COST_TRANSACTION_VALUE_RM
- name: cost_transaction_value_rm
description: test COST TRANSACTION VALUE RM combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: COST_TRANSACTION_TAX_RM
- name: cost_transaction_tax_rm
description: test COST TRANSACTION TAX RM combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: COST_BASE_SERVICE_CHARGE
- name: cost_base_service_charge
description: test COST BASE SERVICE CHARGE combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: COST_TRANSACTION_SERVICE_CHARGE_RM
- name: cost_transaction_service_charge_rm
description: test COST TRANSACTION SERVICE CHARGE_RM combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: transaction_service_charge_rm
description: test transaction service charge rm combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
tests:
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: order_completed_datetime
column_B: order_created_datetime
or_equal: True
tests:
- dbt_utils.expression_is_true:
expression: "cost_transaction_value_rm + cost_transaction_tax_rm + cost_transaction_service_charge_rm < 2*(transaction_value_rm + transaction_base_tax + transaction_tax_rm)"
severity: warn
# FRESHNESS SOURCE
models/exchange/marts/warehouse/fct_exchange__wallet_debit_time_duration.sql
-45
View File
@@ -1,45 +0,0 @@
with
wallet_credit_log as (
select * from {{ ref("int_exchange__wallet_log_credit_amount") }}
),
wallet_debit_time_log as (
select
dense_rank() over (
partition by wallet_id order by updated_at
) as operation_sequence,
lag(updated_at) over (
partition by wallet_id order by updated_at
) previous_update_time,
datediff(
second, previous_update_time, updated_at
) as time_between_operation_in_seconds,
*
from wallet_credit_log
),
updated_wallet_debit_time_log as (
select
*,
iff(
(credit_status = 'DEBIT'), time_between_operation_in_seconds, 0
) as seconds_required_to_debit
from wallet_debit_time_log
),
final_wallet_debit_time_log as (
select
* exclude(seconds_required_to_debit),
iff(
equal_null(seconds_required_to_debit, null),
0,
seconds_required_to_debit
) as seconds_required_to_debit
from updated_wallet_debit_time_log
),
final_wallet_debit_time_log_reformed as (
select * exclude(TIME_BETWEEN_OPERATION_IN_SECONDS)
from final_wallet_debit_time_log
where credit_status = 'DEBIT' and seconds_required_to_debit != 0
)
select *
from final_wallet_debit_time_log_reformed
models/exchange/marts/warehouse/fct_exchange__wallet_log_top_10_top_up.sql
-17
View File
@@ -1,17 +0,0 @@
-- IMPORTS
with
wallet_credit_log as (
select * from {{ ref("int_exchange__wallet_log_credit_amount") }}
),
wallet_credit_top_up_frequency_report as (
select credited_amount, count(credited_amount) repetation
from wallet_credit_log
where credit_status = 'CREDIT'
group by credited_amount
order by repetation desc
limit 10
)
select *
from wallet_credit_top_up_frequency_report
models/exchange/marts/warehouse/fct_exchange__wallet_log_wallet_amount_comparison.sql
-24
View File
@@ -1,24 +0,0 @@
with wallet_log as (
select * from {{ref('int_exchange_wallet_final_log_sequence')}}
),
wallet_details as (
select * from {{ref('stg_exchange__wallet')}}
),
wallet_comparison_log as (
select
w.id as wallet_id,
w.amount as wallet_amount,
s.current_wallet_balance as wallet_balance_from_log,
w.owner_type,
s.credit_status,
s.updated_at as wallet_log_update_time,
s._dbt_ran_at,
w.updated_at as wallet_update_time
from wallet_details w
left join
wallet_log s
on w.id = s.wallet_id
)
select * from wallet_comparison_log
models/exchange/marts/warehouse/test_exchange__bookings.yml
-156
View File
@@ -1,156 +0,0 @@
version: 2
models:
- name: fct_exchange__bookings
tests:
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: BOOKING_UPDATED_DATETIME
column_B: BOOKING_CREATED_DATETIME
or_equal: True
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: BOOKING_DELETED_DATETIME
column_B: BOOKING_CREATED_DATETIME
or_equal: True
columns:
- name: ID
description: boking id
tests:
- unique
- not_null
- name: COMPANY_ID
description: company id
tests:
- unique
- not_null
- name: USER_ID
description: user id
tests:
- unique
- not_null
- name: MARKING_ID
description: marking id
tests:
- unique
- not_null
- name: SERVICE_TYPE
description: type of service
tests:
- not_null
- accepted_values:
values:
- 1
- 3
- 4
- name: SERVICE_TYPE_NAME
description: name of type of service
tests:
- not_null
- accepted_values:
values:
- 3 DAYS TRANSFER
- 1 DAY TRANSFER
- 1688 PAYMENT
- name: BANK_ID
description: bank id
tests:
- unique
- not_null
- name: FIX_VALUE
description: test fix value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: FIX_CURRENCY_ID
description: fix currency id
tests:
- not_null
- accepted_values:
values:
- 1
- 3
- 2
- name: FIX_CURRENCY_NAME
description: fix currency name
tests:
- not_null
- accepted_values:
values:
- Malaysian Ringgit
- US Dollar
- Yuan Renminbi
- name: ESTIMATE_BASE_TO_QUOTE_CURRENCY_EXCHANGE_RATE
description: test estimate base to quote currency exchange rate value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: QUOTE_CURRENCY_ID
description: quote currency id
tests:
- not_null
- accepted_values:
values:
- 3
- 2
- name: QUOTE_CURRENCY_NAME
description: quote currency name
tests:
- not_null
- accepted_values:
values:
- US Dollar
- Yuan Renminbi
- name: ESTIMATE_QUOTE_VALUE
description: test estimate quote value value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: ESTIMATE_BASE_VALUE
description: test estimate quote value value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: ESTIMATE_VALUE_RM
description: test estimate quote value value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: STATUS
description: status
tests:
- accepted_values:
values:
- 2
- 3
- 5
- name: STATUS_NAME
description: status name
tests:
- accepted_values:
values:
- SUSPENDED
- APPROVED
- COMPLETED
models/exchange/marts/warehouse/test_exchange__companies.yml
-146
View File
@@ -1,146 +0,0 @@
version: 2
models:
- name: dim_exchange__companies
tests:
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: ADDRESS_DELETED_AT
column_B: ADDRESS_CREATED_AT
or_equal: True
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: ADDRESS_UPDATED_AT
column_B: ADDRESS_CREATED_AT
or_equal: True
columns:
#Gerson todo Why this company id need to ref with addresses?
- name: ID
description: company id
tests:
- unique
- not_null
- relationships:
to: ref('base_exchange__addresses')
field: company_id
- name: NAME
description: company name
tests:
- not_null
#Yam todo there is marking_id in supplier, need mask them
- name: MARKING_ID
description: marking id
tests:
- unique
- not_null
- name: COMPANY_TYPE
description: type of company
tests:
- accepted_values:
values:
- 0
- 1
- 2
config:
where: "COMPANY_TYPE is not null"
- name: COMPANY_TYPE_NAME
description: name of type company
tests:
- accepted_values:
values:
- CORPORATE
- PERSONAL
config:
where: "COMPANY_TYPE_NAME is not null"
- name: BUSINESS_TYPE
description: type of business
tests:
- accepted_values:
values:
- 1
- 2
- 3
config:
where: "BUSINESS_TYPE is not null"
- name: BUSINESS_TYPE_NAME
description: name of type business
tests:
- accepted_values:
values:
- IMPORTER
- CURRENCY_VENDOR
- FREIGHT_FORWARDER
config:
where: "BUSINESS_TYPE_NAME is not null"
- name: SEGMENT_ID
description: segment id
tests:
- accepted_values:
values:
- 1
- 2
- 3
config:
where: "SEGMENT_ID is not null"
- name: SEGMENT_NAME
description: segment name
tests:
- accepted_values:
values:
- STANDARD SEGMENT
- GOLD MEMBER
- PLATINIUM MEMBER
config:
where: "SEGMENT_NAME is not null"
- name: IS_MIGRATED_COMPANY
description: is migrate company
tests:
- accepted_values:
values:
- 1
- 0
- name: STATUS
description: status
tests:
- accepted_values:
values:
- 0
- 1
- 2
- 4
- 5
- name: STATUS_NAME
description: status name
tests:
- accepted_values:
values:
- SUSPENDED
- APPROVED
- PENDING_SUBMISSION
- REJECTED
- PENDING_VERIFICATION
- name: COMPANY_CONTACT_EMAIL
description: company contact email
tests:
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
regex: "[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*@[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*[.][a-zA-Z]{2,5}"
severity: warn
- name: marking_id
description: marking id
tests:
- relationships:
to: ref('base_exchange_companies')
field: id
models/exchange/marts/warehouse/test_exchange__transaction_orders.yml
-163
View File
@@ -1,163 +0,0 @@
version: 2
models:
- name: fct_exchange__transaction_orders
tests:
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: UPDATED_AT
column_B: CREATED_AT
or_equal: True
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: DELETED_AT
column_B: CREATED_AT
or_equal: True
columns:
- name: ID
description: transaction id
tests:
- unique
- not_null
- relationships:
to: ref('int_exchange__transaction_orders_get_user_ids')
field: id
- name: BOOKING_ID
description: booking id
tests:
- unique
- not_null
- relationships:
to: ref('int_exchange__transaction_orders_get_user_ids')
field: BOOKING_ID
- name: SELLER_COMPANY_ID
description: seller company id
tests:
- unique
- not_null
- relationships:
to: ref('int_exchange__transaction_orders_get_user_ids')
field: SELLER_COMPANY_ID
- name: BANK_ID
description: bank id
tests:
- unique
- not_null
- relationships:
to: ref('int_exchange__transaction_orders_get_user_ids')
field: BANK_ID
- name: COMPANY_ID
description: company id
tests:
- unique
- not_null
- relationships:
to: ref('int_exchange__transaction_orders_get_user_ids')
field: COMPANY_ID
- name: STATUS
description: status
tests:
- accepted_values:
values:
- 0
- 1
- 2
- 3
- 4
- 5
- name: STATUS_NAME
description: status name
tests:
- accepted_values:
values:
- COMPLETED
- SUSPENDED
- REJECTED
- PENDING_SUBMISSION
- PENDING_VERIFICATION
- APPROVED
- name: TRANSACTION_TYPE
description: transaction type
tests:
- accepted_values:
values:
- 1
- name: TRANSACTION_TYPE_NAME
description: name transaction type
tests:
- accepted_values:
values:
- PAYMENT
- name: PAYMENT_METHOD
description: method payment
tests:
- accepted_values:
values:
- 1
- 2
- 5
- 4
- name: PAYMENT_METHOD_NAME
description: name method payment
tests:
- accepted_values:
values:
- CASH
- CHEQUE
- PAYMENT_GATEWAY
- WALLET
- name: base_value
description: test base value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: quote_value
description: test quote value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: transaction_value_rm
description: transaction value rm combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: transaction_base_tax
description: test transaction base tax combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: transaction_tax_rm
description: test transaction tax rm combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: base_service_charge
description: test base service charge combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
- name: TRANSACTION_SERVICE_CHARGE_RM
description: test transaction service change rm value combine with dbt_utils
tests:
- not_null
- dbt_utils.expression_is_true:
expression: " >= 0"
models/exchange/marts/warehouse/test_exchange__users.yml
-76
View File
@@ -1,76 +0,0 @@
version: 2
models:
- name: dim_exchange__users
tests:
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: DELETED_AT
column_B: CREATED_AT
or_equal: True
- dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
column_A: EMAIL_VERIFICATION_LAST_EMAIL_CREATED_AT
column_B: EMAIL_VERIFICATION_FIRST_EMAIL_CREATED_AT
or_equal: True
columns:
- name: ID
description: user id
tests:
- unique
- not_null
- name: NAME
description: user name
tests:
- not_null
- name: USER_TYPE
description: type of user
tests:
- not_null
- accepted_values:
values:
- 1
- 2
- name: USER_TYPE_NAME
description: type of user
tests:
- not_null
- accepted_values:
values:
- USER
- ADMIN
- name: STATUS
description: user status
tests:
- not_null
- accepted_values:
values:
- 1
- 2
- name: STATUS_NAME
description: user status name
tests:
- not_null
- accepted_values:
values:
- APPROVED
- PENDING_VERIFICATION
- name: COMPANY_ID
description: company id
tests:
- unique
- not_null
- name: EMAIL
description: user email
tests:
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
regex: "[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*@[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*[.][a-zA-Z]{2,5}"
severity: warn
models/exchange/staging/activity_logs/stg_exchange__activity_logs.sql
+46
View File
@@ -0,0 +1,46 @@
-- IMPORTS
WITH activity_logs AS (
SELECT * FROM {{ ref('base_exchange__activity_logs') }}
),
-- LOGIC
activity_logs_listout AS (
SELECT
activity_logs.id,
activity_logs.log_name,
activity_logs.description,
activity_logs.subject_id,
activity_logs.subject_type,
activity_logs.causer_id,
activity_logs.causer_type,
activity_logs.properties,
activity_logs.created_at,
activity_logs.updated_at,
CURRENT_TIMESTAMP() AS _dbt_ran_at
FROM activity_logs
),
-- FINAL
final__stg_exchange__activity_logs AS (
SELECT
id,
log_name,
description,
subject_id,
subject_type,
causer_id,
causer_type,
properties,
created_at,
updated_at,
_dbt_ran_at
FROM activity_logs_listout
)
SELECT * FROM final__stg_exchange__activity_logs
models/staging/exchange/stg_exchange__announcement_segments.sql → models/exchange/staging/announcement_segments/stg_exchange__announcement_segments.sql
View File
models/staging/exchange/stg_exchange__announcements.sql → models/exchange/staging/announcements/stg_exchange__announcements.sql
View File
models/staging/exchange/stg_exchange__bank_logs.sql → models/exchange/staging/bank_logs/stg_exchange__bank_logs.sql
View File
models/staging/exchange/stg_exchange__banks.sql → models/exchange/staging/banks/stg_exchange__banks.sql
View File
models/staging/exchange/stg_exchange__booking_logs.sql → models/exchange/staging/booking_logs/stg_exchange__booking_logs.sql
View File
models/staging/exchange/stg_exchange__bookings.sql → models/exchange/staging/bookings/stg_exchange__bookings.sql
View File
models/staging/exchange/stg_exchange__companies.sql → models/exchange/staging/companies/stg_exchange__companies.sql
View File
models/staging/exchange/stg_exchange__companies_and_segments_relations.sql → models/exchange/staging/companies_and_segments_relations/stg_exchange__companies_and_segments_relations.sql
View File
models/staging/exchange/stg_exchange__companies_and_users_relations.sql → models/exchange/staging/companies_and_users_relations/stg_exchange__companies_and_users_relations.sql
View File
models/staging/exchange/stg_exchange__company_addresses.sql → models/exchange/staging/company_addresses/stg_exchange__company_addresses.sql
View File
models/staging/exchange/stg_exchange__contacts.sql → models/exchange/staging/contacts/stg_exchange__contacts.sql
View File
models/staging/exchange/stg_exchange__countries.sql → models/exchange/staging/countries/stg_exchange__countries.sql
View File
models/staging/exchange/stg_exchange__currencies.sql → models/exchange/staging/currencies/stg_exchange__currencies.sql
View File
models/staging/exchange/stg_exchange__currency_logs.sql → models/exchange/staging/currency_logs/stg_exchange__currency_logs.sql
View File
models/staging/exchange/stg_exchange__currency_rate_logs.sql → models/exchange/staging/currency_rate_logs/stg_exchange__currency_rate_logs.sql
View File
models/staging/exchange/stg_exchange__current_currency_rates.sql → models/exchange/staging/current_currency_rates/stg_exchange__current_currency_rates.sql
View File
models/staging/exchange/stg_exchange__current_wallets.sql → models/exchange/staging/current_wallets/stg_exchange__current_wallets.sql
View File
models/staging/exchange/stg_exchange__districts.sql → models/exchange/staging/districts/stg_exchange__districts.sql
View File
models/staging/exchange/stg_exchange__documents.sql → models/exchange/staging/documents/stg_exchange__documents.sql
View File
models/staging/exchange/stg_exchange__failed_jobs.sql → models/exchange/staging/failed_jobs/stg_exchange__failed_jobs.sql
View File
models/staging/exchange/stg_exchange__files.sql → models/exchange/staging/files/stg_exchange__files.sql
View File
models/staging/exchange/stg_exchange__group_costs.sql → models/exchange/staging/group_costs/stg_exchange__group_costs.sql
View File
models/staging/exchange/stg_exchange__jobs.sql → models/exchange/staging/jobs/stg_exchange__jobs.sql
View File
models/staging/exchange/stg_exchange__migrations.sql → models/exchange/staging/migrations/stg_exchange__migrations.sql
View File
models/staging/exchange/stg_exchange__model_has_roles.sql → models/exchange/staging/model_has_roles/stg_exchange__model_has_roles.sql
View File
models/staging/exchange/stg_exchange__notifications.sql → models/exchange/staging/notifications/stg_exchange__notifications.sql
View File
models/staging/exchange/stg_exchange__password_resets.sql → models/exchange/staging/password_resets/stg_exchange__password_resets.sql
View File
models/staging/exchange/stg_exchange__permissions.sql → models/exchange/staging/permissions/stg_exchange__permissions.sql
View File
models/staging/exchange/stg_exchange__role_has_permissions.sql → models/exchange/staging/role_has_permissions/stg_exchange__role_has_permissions.sql
View File
models/staging/exchange/stg_exchange__roles.sql → models/exchange/staging/roles/stg_exchange__roles.sql
View File
models/staging/exchange/stg_exchange__segment_constants.sql → models/exchange/staging/segment_constants/stg_exchange__segment_constants.sql
View File
models/staging/exchange/stg_exchange__segments.sql → models/exchange/staging/segments/stg_exchange__segments.sql
View File
models/staging/exchange/stg_exchange__service_types.sql → models/exchange/staging/service_types/stg_exchange__service_types.sql
View File
models/staging/exchange/stg_exchange__states.sql → models/exchange/staging/states/stg_exchange__states.sql
View File
models/staging/exchange/stg_exchange__transaction_costs.sql → models/exchange/staging/transaction_costs/stg_exchange__transaction_costs.sql
View File
models/staging/exchange/stg_exchange__transaction_costs_and_group_costs_relations.sql → models/exchange/staging/transaction_costs_and_group_costs_relations/stg_exchange__transaction_costs_and_group_costs_relations.sql
View File
models/staging/exchange/stg_exchange__transaction_log_costs.sql → models/exchange/staging/transaction_log_costs/stg_exchange__transaction_log_costs.sql
View File
models/staging/exchange/stg_exchange__transaction_log_orders.sql → models/exchange/staging/transaction_log_orders/stg_exchange__transaction_log_orders.sql
View File
models/staging/exchange/stg_exchange__transaction_order_details.sql → models/exchange/staging/transaction_order_details/stg_exchange__transaction_order_details.sql
View File
models/staging/exchange/stg_exchange__transaction_orders.sql → models/exchange/staging/transaction_orders/stg_exchange__transaction_orders.sql
View File
models/staging/exchange/stg_exchange__transaction_wallets.sql → models/exchange/staging/transaction_wallets/stg_exchange__transaction_wallets.sql
View File
models/staging/exchange/stg_exchange__transactions.sql → models/exchange/staging/transactions/stg_exchange__transactions.sql
View File
models/staging/exchange/stg_exchange__user_email_verifications.sql → models/exchange/staging/user_email_verifications/stg_exchange__user_email_verifications.sql
View File
models/staging/exchange/stg_exchange__users.sql → models/exchange/staging/users/stg_exchange__users.sql
View File
models/exchange/staging/wallet/stg_exchange__wallet.sql
-10
View File
@@ -1,10 +0,0 @@
-- IMPORTS
WITH wallets AS (
SELECT * FROM {{ ref('base_exchange__wallets') }}
)
-- FINAL
SELECT * FROM wallets
models/exchange/staging/wallet_logs/stg_exchange__wallet_logs.sql
-7
View File
@@ -1,7 +0,0 @@
-- IMPORTS
with wallet_logs as (select * from {{ ref("base_exchange__wallet_logs") }})
-- FINAL
select *
from wallet_logs
order by wallet_id, updated_at asc
models/shipping/staging/stg_shipping__activity_logs.sql
-46
View File
@@ -1,46 +0,0 @@
-- IMPORTS
WITH activity_logs AS (
SELECT * FROM {{ source('src_shipping_mysql', 'activity_log') }}
),
-- LOGIC
activity_logs_rename AS (
SELECT
activity_logs.id,
activity_logs.log_name,
activity_logs.description,
activity_logs.subject_id,
activity_logs.subject_type,
activity_logs.causer_id,
activity_logs.causer_type,
activity_logs.properties,
activity_logs.created_at AS created_datetime,
activity_logs.updated_at AS updated_datetime,
current_timestamp() AS _dbt_ran_at
FROM activity_logs
),
-- FINAL
final__base_exchange__activity_logs AS (
SELECT
id,
log_name,
description,
subject_id,
subject_type,
causer_id,
causer_type,
properties,
created_datetime,
updated_datetime,
_dbt_ran_at
FROM activity_logs_rename
)
SELECT * FROM final__base_exchange__activity_logs
models/staging/autocount/_autocount__sources.yml
View File
models/staging/crisp/_crisp__sources.yml
View File
models/staging/exchange/_exchange__docs.md
-23
View File
@@ -1,23 +0,0 @@
{% docs src_exchange_mysql %}
Data is extract from CIEF Selfhost MYSQL `exchange_production`, using custom python code.
- `_source_loaded_at` will add into the raw data when extract
```YML
loaded_at_field: _source_loaded_at
freshness:
warn_after: {count: 12, period: hour}
error_after: {count: 24, period: hour}
```
- The raw data will use copy into snowflake in 2AM everyday
- Depend on target.name to determin which snowflake raw schema
```JINJA
{%- if target.name == "prod" -%} PROD_CIEF_RAW_DB
{%- else -%} DEV_CIEF_RAW_DB
{%- endif -%}
```
{% enddocs %}
models/staging/exchange/_exchange__models.yml
-6
View File
@@ -1,6 +0,0 @@
version: 2
models:
- name: stg_exchange__activity_logs
- name: stg_exchange__announcments
models/staging/exchange/_exchange__sources.yml
-619
View File
@@ -1,619 +0,0 @@
version: 2
sources:
- name: src_exchange_mysql
description: '{{ doc("src_exchange_mysql") }}'
database: |
{%- if target.name == "prod" -%} PROD_CIEF_RAW_DB
{%- else -%} DEV_CIEF_RAW_DB
{%- endif -%}
schema: EXCHANGE_MYSQL
loader: custom_python_code
loaded_at_field: _source_loaded_at
freshness:
warn_after: {count: 12, period: hour}
error_after: {count: 24, period: hour}
meta:
owner: "@yam"
model_maturity: prod
tags:
- exchange
- daily
tables:
- name: activity_log
description: Contains most table's activity log that CUD (created, updated, deleted) in www.exchange.cief-malaysia.com
columns:
- name: id
description: Primary key for 'activity_log'
tests:
- unique
- not_null
- name: description
description: system CUD activity list down
tests:
- accepted_values:
values: ['created', 'updated', 'deleted']
- name: subject_id
description: The exchange CUD table's id, need to refer subject_type column to know what table is CUD in the row
tests:
- not_null
- name: subject_type
description: What table that CUD using in the row
tests:
- not_null
- name: causer_id
description: Need to refer causer_type column to know who is CUD in the row. it is not well maintain there is some NULL
- name: causer_type
description: Who is CUD the row
- name: properties
description: Json data format that contain attributes of the activity, it is not well maintain, subject (Country, ServiceType, CurrencyRate. UserEmailVerification, File) have properties only
- name: created_at
description: Datetime that CUD activity occur
tests:
- not_null
- name: updated_at
description: Exactly same with created_at, redundant column
tests:
- dbt_expectations.expect_column_pair_values_to_be_equal:
column_A: updated_at
column_B: created_at
- name: addresses
description: Addresses that input when user register
columns:
- name: id
description: Primary key for 'addresses'
tests:
- unique
- not_null
- name: company_id
description: Foreign keys for 'companies'
tests:
- not_null
- name: country_id
description: Foreign keys for 'countries'
tests:
- not_null
- name: state_id
description: Foreign keys for 'states'
tests:
- not_null
- name: district_id
description: Foreign keys for 'districts'
tests:
- not_null
- name: postcode
description: The number/string that user input
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_of_type:
column_type: integer
- name: street_one
description: The string that user input
tests:
- not_null
- name: street_two
description: The string that user input
- name: billing
description: If mark 1 means it is billing address, 0 means is not billing address, every company should have 1 billing address
tests:
- dbt_expectations.expect_column_max_to_be_between: #Every company should have a billing address
min_value: 1
max_value: 1
group_by: [company_id]
- dbt_expectations.expect_column_sum_to_be_between: #1 company should have 1 billing address only
min_value: 1
max_value: 1
group_by: [company_id]
row_condition: "deleted_at is null"
- name: deleted_at
description: System generate datetime that address was deleted
- name: created_at
description: System generate datetime that address was created
tests:
- not_null
- name: updated_at
description: System generate datetime that address was updated
tests:
- not_null
- name: announcement_segment
description: Superadmin / Operator guys will segment the announcment, but normally they will leave it default. 2 FK combine to becoem a relation table.
columns:
- name: announcement_id
description: Primary key for 'announcement_segment' & Foreign key for 'announcements'
tests:
- not_null
- name: segment_id
description: Primary key for 'announcement_segment' & Foreign key for 'segments'
tests:
- not_null
- name: created_at
description: Abandon column
tests:
- dbt_expectations.expect_column_values_to_be_null
- name: updated_at
description: Abandon column
tests:
- dbt_expectations.expect_column_values_to_be_null
- name: announcements
description: Announcements that show in our website which create by superadmin
columns:
- name: id
description: Primary key for 'announcements'
tests:
- unique
- not_null
- name: title
description: title of announcement, using string
tests:
- not_null
- name: description
description: description of announcement, using longtext, have <br> html for break
tests:
- not_null
- name: starting_on
description: announcement starting datetime
tests:
- not_null
- name: ending_on
description: announcement ending datetime
tests:
- not_null
- name: is_all_day
description: column might not in use, system auto generate
tests:
- not_null
- accepted_values:
values: [0]
- name: duration
description: column might not in use, system auto generate
tests:
- not_null
- accepted_values:
values: [0]
- name: is_recurring
description: column might not in use, system auto generate
tests:
- not_null
- accepted_values:
values: [0]
- name: recurrence_pattern
description: column might not in use, system auto generate
tests:
- dbt_expectations.expect_column_values_to_be_null
- name: deleted_at
description: system generate datetime that announcment was deleted
tests:
- not_null
- name: created_at
description: system generate datetime that announcment was created
tests:
- not_null
- name: updated_at
description: system generate datetime that announcment was updated
tests:
- not_null
- name: bank_logs
description: Abandon table
config:
enabled: false
- name: banks
description: Table which store banks details.
columns:
- name: id
description: Primary key for 'banks'
tests:
- unique
- not_null
- name: company_id
description: Foreign keys for 'companies'
tests:
- not_null
- name: reference
description: A string which write by user/company for the bank transfer reference or other remind ussage
- name: holder_name
description: Bank holder name
tests:
- not_null
- name: account_no
description: Bank account number
tests:
- not_null
- name: booking_logs
description: Raw `bookings` table from Snowflake
columns:
- name: id
description: Primary key for 'booking_logs'
tests:
- unique
- not_null
- name: bookings
description: Raw `bookings` table from Snowflake
columns:
- name: id
description: Primary key for 'bookings'
tests:
- unique
- not_null
- name: companies
description: Raw `companies` table from Snowflake
columns:
- name: id
description: Primary key for 'companies'
tests:
- unique
- not_null
- name: contacts
description: Raw `contacts` table from Snowflake
columns:
- name: id
description: Primary key for 'contacts'
tests:
- unique
- not_null
- name: countries
description: Raw `countries` table from Snowflake
columns:
- name: id
description: Primary key for 'countries'
tests:
- unique
- not_null
- name: currencies
description: Raw `currencies` table from Snowflake
columns:
- name: id
description: Primary key for 'currencies'
tests:
- unique
- not_null
- name: currency_logs
description: Raw `currency_logs` table from Snowflake
columns:
- name: id
description: Primary key for 'currency_logs'
tests:
- unique
- not_null
- name: currency_rate_logs
description: Raw `currency_rate_logs` table from Snowflake
columns:
- name: id
description: Primary key for 'currency_rate_logs'
tests:
- unique
- not_null
- name: currency_rates
description: Raw `currency_rates` table from Snowflake
columns:
- name: id
description: Primary key for 'currency_rates'
tests:
- unique
- not_null
- name: districts
description: Raw `districts` table from Snowflake
columns:
- name: id
description: Primary key for 'districts'
tests:
- unique
- not_null
- name: documents
description: Raw `documents` table from Snowflake
columns:
- name: id
description: Primary key for 'documents'
tests:
- unique
- not_null
- name: employees
description: Raw `employees` table from Snowflake
columns:
- name: company_id
description: Primary key for 'employees'
- name: failed_jobs
description: Raw `failed_jobs` table from Snowflake
columns:
- name: id
description: Primary key for 'failed_jobs'
tests:
- unique
- not_null
- name: files
description: Raw `files` table from Snowflake
columns:
- name: id
description: Primary key for 'files'
tests:
- unique
- not_null
- name: group_transactions
description: Raw `group_transactions` table from Snowflake
columns:
- name: id
description: Primary key for 'group_transactions'
tests:
- unique
- not_null
- name: groups
description: Raw `groups` table from Snowflake
columns:
- name: id
description: Primary key for 'groups'
tests:
- unique
- not_null
- name: jobs
description: Raw `jobs` table from Snowflake
columns:
- name: id
description: Primary key for 'jobs'
tests:
- unique
- not_null
- name: migrations
description: Raw `migrations` table from Snowflake
columns:
- name: id
description: Primary key for 'migrations'
tests:
- unique
- not_null
- name: model_has_roles
description: Raw `model_has_roles` table from Snowflake
columns:
- name: role_id
description: Primary key for 'model_has_roles'
- name: notifications
description: Raw `notifications` table from Snowflake
columns:
- name: id
description: Primary key for 'notifications'
tests:
- unique
- not_null
- name: password_resets
description: Raw `password_resets` table from Snowflake
columns:
- name: id
description: Primary key for 'password_resets'
tests:
- unique
- not_null
- name: permissions
description: Raw `permissions` table from Snowflake
columns:
- name: id
description: Primary key for 'permissions'
tests:
- unique
- not_null
- name: role_has_permissions
description: Raw `role_has_permissions` table from Snowflake
columns:
- name: permission_id
description: Primary key for 'role_has_permissions'
- name: roles
description: Raw `roles` table from Snowflake
columns:
- name: id
description: Primary key for 'roles'
tests:
- unique
- not_null
- name: segment_companies
description: Raw `segment_companies` table from Snowflake
columns:
- name: segment_id
description: Primary key for 'segment_companies'
- name: segment_constants
description: Raw `segment_constants` table from Snowflake
columns:
- name: id
description: Primary key for 'segment_constants'
tests:
- unique
- not_null
- name: segments
description: Raw `segments` table from Snowflake
columns:
- name: id
description: Primary key for 'segments'
tests:
- unique
- not_null
- name: service_types
description: Raw `service_types` table from Snowflake
columns:
- name: id
description: Primary key for 'service_types'
tests:
- unique
- not_null
description: Raw `states` table from Snowflake
columns:
- name: id
description: Primary key for 'states'
tests:
- unique
- not_null
- name: transaction_detail
description: Raw `transaction_detail` table from Snowflake
columns:
- name: id
description: Primary key for 'transaction_detail'
tests:
- unique
- not_null
- name: transaction_logs
description: Raw `transaction_logs` table from Snowflake
columns:
- name: id
description: Primary key for 'transaction_logs'
tests:
- unique
- not_null
- name: transactions
description: Raw `transactions` table from Snowflake
columns:
- name: id
description: Primary key for 'transactions'
tests:
- unique
- not_null
- name: user_email_verifications
description: Raw `user_email_verifications` table from Snowflake
columns:
- name: id
description: Primary key for 'user_email_verifications'
tests:
- unique
- not_null
- name: users
description: Raw `users` table from Snowflake
columns:
- name: id
description: Primary key for 'users'
tests:
- unique
- not_null
- name: wallet_logs
description: Raw `wallet_logs` table from Snowflake
columns:
- name: id
description: Primary key for 'wallet_logs'
tests:
- unique
- not_null
- name: wallets
description: Raw `wallets` table from Snowflake
columns:
- name: id
description: Primary key for 'wallets'
tests:
- unique
- not_null
models/staging/exchange/stg_exchange__activity_logs.sql
-94
View File
@@ -1,94 +0,0 @@
-- IMPORTS
WITH activity_logs AS (
SELECT * FROM {{ source('src_exchange_mysql', 'activity_log') }}
),
-- LOGIC
activity_logs_rename AS (
SELECT
activity_logs.id,
activity_logs.log_name,
activity_logs.description,
activity_logs.subject_id,
activity_logs.subject_type,
activity_logs.causer_id,
activity_logs.causer_type,
activity_logs.properties,
activity_logs.created_at,
activity_logs.updated_at,
current_timestamp() AS _dbt_ran_at
FROM activity_logs
),
-- FINAL
final__base_exchange__activity_logs AS (
SELECT
id,
log_name,
description,
subject_id,
subject_type,
causer_id,
causer_type,
properties,
created_at,
updated_at,
_dbt_ran_at
FROM activity_logs_rename
)
SELECT * FROM final__base_exchange__activity_logs
-- IMPORTS
WITH activity_logs AS (
SELECT * FROM {{ ref('base_exchange__activity_logs') }}
),
-- LOGIC
activity_logs_listout AS (
SELECT
activity_logs.id,
activity_logs.log_name,
activity_logs.description,
activity_logs.subject_id,
activity_logs.subject_type,
activity_logs.causer_id,
activity_logs.causer_type,
activity_logs.properties,
activity_logs.created_at,
activity_logs.updated_at,
CURRENT_TIMESTAMP() AS _dbt_ran_at
FROM activity_logs
),
-- FINAL
final__stg_exchange__activity_logs AS (
SELECT
id,
log_name,
description,
subject_id,
subject_type,
causer_id,
causer_type,
properties,
created_at,
updated_at,
_dbt_ran_at
FROM activity_logs_listout
)
SELECT * FROM final__stg_exchange__activity_logs
models/staging/shipping/_shipping_sources.yml
View File
tests/generic_test__zero_or_positive.sql
View File