diff --git a/README.md b/README.md index ae37e92..13040c0 100644 --- a/README.md +++ b/README.md @@ -1,184 +1,15 @@ -# DBT governance -Thid markdown document will specify the rules that need to follow by every developer. -
-Short naming converntion lists that need to follow - -| model_type Shortcut | Full name | -| ------------------- | ------ | -| seed | Seed | -| src | Source | -| snap | Snapshot | -| stg | Stage | -| int | Intermediate | -| fct | Fact | -| dim | Dimension | -| rep | Report | -| sem | Semantic | +Welcome to your new dbt project! -| model_name Shortcut | Full name | Use case | -| ------------------- | ------ | ------ | -| brg | Brigde | relationship table | -| log | Log | log data | +### Using the starter project + +Try running the following commands: +- dbt run +- dbt test -## CIEF General Rule -- Snowflake SQL use to write dbt SQL code -- Star schema is use -- First week of the day is start at Monday, and last week of the day will be Sunday -- Fiscal 1st quarter is start from Febuary -- Timezone: Kuala Lumpur / Malaysia (GMT +8) -- Some status/type is not input in raw database (You should found in seed) - -
- -## DBT Rule -### Structure of the dbt model layer -- Analyses - 1. Analyses layer will not create in Snowflake - 2. You can write some adhoc queries - 3. Test some SQL code before write in Models - -
- -- Macros - 1. Jinja function that reuse in other models - -
- -- Models - - Staging - - System name - 1. Materialise: `View` - 2. 1-to-1 relationship (or mapping) to source tables. - 3. Column renaming - 4. Column remove - 5. Column data renaming, such as (status integer to string) - 6. Data input error cleansing - 7. Check for raw data freshness - 8. Data type transformation - 9. Data split and merge - - Source - 1. Source will write in YML - 2. Check Freshness - 3. Check duplicate id - 4. Check null id - - - Intermediate - 1. Materialise: `Ephemerally` - 2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want - 3. Be referenced repeatedly in more than one model - 4. Isolating complex operations - - - Marts/warehouse - 1. Materialise: `Table` - 2. Store fact and dimension models - - - Marts/reporting - 1. Materialise: `Table` - 2. Store custom reports - -
- -- Seeds - 1. Store custom dataset in csv format - 2. The dataset must not change frequently - -
- -- Snapshots - 1. DBT build-in SCD type 2 function - 2. Prefer to use after the source table and before the staging layer - -
- -- Tests - 1. Can write some yml test case - -
- -### File Naming Rules -- File names must be unique -- Each sql/python file must start with [`model_type`]_[`system_name`]__[`model_name`]s.sql/py -- The model should be name as plural. (eg: stg_exchange__orders.sql) - -
- -### Column Naming Rules -- If array datatype, the column naming must be plural (eg: ids, messages) -- Schema, table and column names should be in `snake_case`. - -- Limit use of abbreviations that are related to domain knowledge. An onboarding - employee will understand `current_order_status` better than `current_os`. - -- Use names based on the _business_ terminology, rather than the source terminology. - -- Each model should have a primary key that can identify the unique row, and should be named `_id`, e.g. `account_id` – this makes it easier to know what `id` is being referenced in downstream joined models. - -- If a surrogate key is created, it should be named `_sk`. - -- For `base` or `staging` models, columns should be ordered in categories, where identifiers are first and date/time fields are at the end. - Example: - ```sql - transformed as ( - select - -- ids - order_id, - customer_id, - - -- dimensions - order_status, - is_shipped, - - -- measures - order_total, - - -- date/times - created_at, - updated_at, - - -- metadata - _sdc_batched_at - from source - ) - ``` - -- Date/time columns should be named according to these conventions: - - Timestamps: `_datetime` - Example: `created_datetime` - - - Dates: `_date` - Example: `created_date` - -- Booleans should be prefixed with `is_` or `has_`. - Example: `is_active_customer` and `has_admin_access` - -- Price/revenue fields should be in decimal currency (e.g. `19.99` for $19.99; many app databases store prices as integers in cents). If non-decimal currency is used, indicate this with suffix, e.g. `price_in_cents`. - -
- -### SQL Coding Rule -- The SQL clause **must** be UPPERCASE (eg. `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `LIMIT`, `WITH`, `AS`, `SUM`, `PARTITION OVER`, `DIV0`, `LEFT JOIN`, `ON` ....) -- Try to avoid using `EXCLUDE` in the SQL -- Avoid using multi layer subquery, and try to use CTE, subquery must not more than 1 layer -- Each model structure must contain with - -```sql ---IMPORT -WITH [table_names] AS ( - SELECT * FROM {{ ref('file_names')}} -), - ---LOGIC -[logic_names] AS ( - ..... -), - ---FINAL -final__[table_names] AS ( - -) -SELECT * FROM final__[final_table_names] -``` - -### Testing -- At a minimum, `unique` and `not_null` tests should be applied to the expected primary key of each model. \ No newline at end of file +### Resources: +- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction) +- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers +- Join the [dbt community](http://community.getbdt.com/) to learn from other analytics engineers +- Find [dbt events](https://events.getdbt.com) near you +- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices diff --git a/analyses/exchange/company_conversion_v2.sql b/analyses/exchange/company_conversion_v2.sql index e124bcb..af2afe3 100644 --- a/analyses/exchange/company_conversion_v2.sql +++ b/analyses/exchange/company_conversion_v2.sql @@ -1,5 +1,5 @@ -- MARCRO for count the cummulative_activity_number -{% set funnel_activity = ['register','email_verified','identity_document_uploaded','identity_document_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%} +{% set funnel_activity = ['register','email_verified','identity_documents_uploaded','identity_documents_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%} -- IMPORT WITH companies AS ( @@ -69,13 +69,12 @@ company_first_time_order_date AS ( ---start defind funnel activity +--start defind funnel repeat_completed_order_after_60days_activities AS ( SELECT clean_company_transaction_orders.company_id, 0 AS is_data_cleansing_generate_row, 'repeat_completed_order_after_60days' AS activity, - 9 AS activtity_rank, MIN(clean_company_transaction_orders.created_at) AS activity_datetime FROM clean_company_transaction_orders @@ -95,7 +94,6 @@ repeat_completed_order_after_30days_activities AS ( clean_company_transaction_orders.company_id, 0 AS is_data_cleansing_generate_row, 'repeat_completed_order_after_30days' AS activity, - 8 AS activtity_rank, MIN(clean_company_transaction_orders.created_at) AS activity_datetime FROM clean_company_transaction_orders @@ -115,7 +113,6 @@ first_time_order_completed_activities AS ( clean_company_transaction_orders.company_id, 0 AS is_data_cleansing_generate_row, 'first_time_order_completed' AS activity, - 7 AS activtity_rank, MIN(clean_company_transaction_orders.created_at) AS activity_datetime FROM clean_company_transaction_orders @@ -132,7 +129,6 @@ first_time_order_activities AS ( clean_company_transaction_orders.company_id, 0 AS is_data_cleansing_generate_row, 'first_time_order' AS activity, - 6 AS activtity_rank, MIN(clean_company_transaction_orders.created_at) AS activity_datetime FROM clean_company_transaction_orders @@ -149,7 +145,6 @@ first_time_booking_activities AS ( IFNULL( unclean_first_time_booking_activities.company_id , first_time_order_activities.company_id ) AS company_id, IFF( unclean_first_time_booking_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, 'first_time_booking' AS activity, - 5 AS activtity_rank, IFNULL( unclean_first_time_booking_activities.activity_datetime , first_time_order_activities.activity_datetime ) AS activity_datetime FROM @@ -173,20 +168,19 @@ first_time_booking_activities AS ( ), -identity_document_verified_activities AS ( +identity_documents_verified_activities AS ( SELECT - IFNULL( unclean_identity_document_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id, - IFF( unclean_identity_document_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, - 'identity_document_verified' AS activity, - 4 AS activtity_rank, - IFNULL( unclean_identity_document_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime + IFNULL( unclean_identity_documents_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id, + IFF( unclean_identity_documents_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, + 'identity_documents_verified' AS activity, + IFNULL( unclean_identity_documents_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime FROM ( SELECT company_id, - 'identity_document_verified' AS activity, + 'identity_documents_verified' AS activity, MIN(identity_created_datetime) AS activity_datetime FROM company_identity_documents @@ -197,28 +191,27 @@ identity_document_verified_activities AS ( GROUP BY company_id - ) AS unclean_identity_document_verified_activities + ) AS unclean_identity_documents_verified_activities FULL OUTER JOIN first_time_booking_activities - ON (unclean_identity_document_verified_activities.company_id = first_time_booking_activities.company_id) + ON (unclean_identity_documents_verified_activities.company_id = first_time_booking_activities.company_id) ), -identity_document_uploaded_activities AS ( +identity_documents_uploaded_activities AS ( SELECT - IFNULL( unclean_identity_document_uploaded_activities.company_id , identity_document_verified_activities.company_id ) AS company_id, - IFF( unclean_identity_document_uploaded_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, - 'identity_document_uploaded' AS activity, - 3 AS activtity_rank, - IFNULL( unclean_identity_document_uploaded_activities.activity_datetime , identity_document_verified_activities.activity_datetime ) AS activity_datetime + IFNULL( unclean_identity_documents_uploaded_activities.company_id , identity_documents_verified_activities.company_id ) AS company_id, + IFF( unclean_identity_documents_uploaded_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, + 'identity_documents_uploaded' AS activity, + IFNULL( unclean_identity_documents_uploaded_activities.activity_datetime , identity_documents_verified_activities.activity_datetime ) AS activity_datetime FROM ( SELECT company_id, - 'identity_document_uploaded' AS activity, + 'identity_documents_uploaded' AS activity, MIN(identity_created_datetime) AS activity_datetime FROM company_identity_documents @@ -226,20 +219,19 @@ identity_document_uploaded_activities AS ( GROUP BY company_id - ) AS unclean_identity_document_uploaded_activities + ) AS unclean_identity_documents_uploaded_activities - FULL OUTER JOIN identity_document_verified_activities - ON (unclean_identity_document_uploaded_activities.company_id = identity_document_verified_activities.company_id) + FULL OUTER JOIN identity_documents_verified_activities + ON (unclean_identity_documents_uploaded_activities.company_id = identity_documents_verified_activities.company_id) ), email_verified_activities AS ( SELECT - IFNULL( unclean_email_verified_activities.company_id , identity_document_uploaded_activities.company_id ) AS company_id, + IFNULL( unclean_email_verified_activities.company_id , identity_documents_uploaded_activities.company_id ) AS company_id, IFF( unclean_email_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, 'email_verified' AS activity, - 2 AS activtity_rank, - IFNULL( unclean_email_verified_activities.activity_datetime , identity_document_uploaded_activities.activity_datetime ) AS activity_datetime + IFNULL( unclean_email_verified_activities.activity_datetime , identity_documents_uploaded_activities.activity_datetime ) AS activity_datetime FROM ( @@ -257,8 +249,8 @@ email_verified_activities AS ( company_id ) AS unclean_email_verified_activities - FULL OUTER JOIN identity_document_uploaded_activities - ON (unclean_email_verified_activities.company_id = identity_document_uploaded_activities.company_id) + FULL OUTER JOIN identity_documents_uploaded_activities + ON (unclean_email_verified_activities.company_id = identity_documents_uploaded_activities.company_id) ), @@ -267,7 +259,6 @@ register_activities AS ( IFNULL( unclean_register_activities.company_id , email_verified_activities.company_id ) AS company_id, IFF( unclean_register_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, 'register' AS activity, - 1 AS activtity_rank, IFNULL( unclean_register_activities.activity_datetime , email_verified_activities.activity_datetime ) AS activity_datetime FROM @@ -319,30 +310,24 @@ union_all_activities_flag_migrated_company AS ( --FINAL -sem_exchange__company_conversion_funnels AS ( +sem_exchange__company_funnels AS ( SELECT company_id, is_data_cleansing_generate_row, is_migrated_company, - activity, - activtity_rank, - IFF( ROW_NUMBER() OVER (PARTITION BY company_id ORDER BY activtity_rank DESC) = 1, - 1, - 0) AS latest_activity, - activity_datetime, - {% for activity_type in funnel_activity %} + {% for activity in funnel_activity %} - count(iff(activity = '{{activity_type}}', company_id, NULL)) - OVER (ORDER BY activity_datetime) AS cumulative_{{activity_type}}_activity, + count(iff(activity = '{{activity}}', company_id, NULL)) + OVER (ORDER BY activity_datetime) AS cumulative_{{activity}}_activity, {% endfor %} - {% for activity_type in funnel_activity %} + {% for activity in funnel_activity %} - count(iff(activity = '{{activity_type}}' AND is_migrated_company = 0, company_id, NULL)) - OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity_type}}_activity, + count(iff(activity = '{{activity}}' AND is_migrated_company = 0, company_id, NULL)) + OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity}}_activity, {% endfor %} @@ -353,4 +338,4 @@ sem_exchange__company_conversion_funnels AS ( ORDER BY activity_datetime ) -SELECT * FROM sem_exchange__company_conversion_funnels \ No newline at end of file +SELECT * FROM sem_exchange__company_funnels \ No newline at end of file diff --git a/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_last_transaction_sequence.sql b/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_last_transaction_sequence.sql deleted file mode 100644 index 191f581..0000000 --- a/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_last_transaction_sequence.sql +++ /dev/null @@ -1,8 +0,0 @@ -with - wallet_log_sequence as ( - select * from {{ ref("int_exchange__wallet_transaction_sequence") }} - ) - -select max(operation_sequence) as last_transaction_id, wallet_id -from wallet_log_sequence -group by wallet_id diff --git a/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_log_credit_amount.sql b/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_log_credit_amount.sql deleted file mode 100644 index 37ccbe6..0000000 --- a/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_log_credit_amount.sql +++ /dev/null @@ -1,56 +0,0 @@ --- IMPORTS -with - wallet_logs as (select * from {{ ref("stg_exchange__wallet_logs") }}), - -- LOGIT - wallet_credit_amount as ( - select - *, - lag(amount) over ( - partition by wallet_id order by updated_at - ) as previous_wallet_balance, - amount - lag(amount) over ( - partition by wallet_id order by updated_at - ) as credited_amount - from wallet_logs - ), - wallet_credit_amount_reformed as ( - select - id, - wallet_id, - previous_wallet_balance, - amount as current_wallet_balance, - iff( - equal_null(previous_wallet_balance, null), amount, credited_amount - ) as credited_amount, - owner_type, - owner_id, - code, - currency_id, - deleted_at, - created_at, - updated_at, - _dbt_ran_at - from wallet_credit_amount - ), - - final_wallet_credit_amount as ( - select - id, - wallet_id, - previous_wallet_balance, - current_wallet_balance, - credited_amount, - iff(credited_amount > 0, 'CREDIT', 'DEBIT') as credit_status, - owner_type, - owner_id, - code, - currency_id, - deleted_at, - created_at, - updated_at, - _dbt_ran_at - from wallet_credit_amount_reformed - ) - -select * -from final_wallet_credit_amount diff --git a/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_transaction_sequence.sql b/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_transaction_sequence.sql deleted file mode 100644 index 6c909e8..0000000 --- a/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_transaction_sequence.sql +++ /dev/null @@ -1,13 +0,0 @@ -with - wallet_credit_log as ( - select * from {{ ref("int_exchange__wallet_log_credit_amount") }} - ), - wallet_credit_with_sequence as ( - select - dense_rank() over ( - partition by wallet_id order by updated_at - ) as operation_sequence, - * - from wallet_credit_log - ) -select * from wallet_credit_with_sequence diff --git a/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange_wallet_final_log_sequence.sql b/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange_wallet_final_log_sequence.sql deleted file mode 100644 index eac36eb..0000000 --- a/models/exchange/intermediate/wallet_log_creadit_amount/int_exchange_wallet_final_log_sequence.sql +++ /dev/null @@ -1,24 +0,0 @@ -with - wallet_log_sequence as ( - select * from {{ ref("int_exchange__wallet_transaction_sequence") }} - ), - wallet_last_log as ( - select * from {{ ref("int_exchange__wallet_last_transaction_sequence") }} - ) - -select - s.id, - s.wallet_id, - s.current_wallet_balance, - s.credit_status, - s.owner_type, - s.owner_id, - s.code, - s.currency_id, - s.deleted_at, - s.created_at, - s.updated_at, - s._dbt_ran_at -from wallet_log_sequence s -inner join wallet_last_log l -where s.wallet_id = l.wallet_id and s.operation_sequence = l.last_transaction_id diff --git a/models/exchange/marts/reporting/rep_exchange__daily_orders/test_exchange__daily_orders.yml b/models/exchange/marts/reporting/rep_exchange__daily_orders/test_exchange__daily_orders.yml index c7d4451..f1aa2f9 100644 --- a/models/exchange/marts/reporting/rep_exchange__daily_orders/test_exchange__daily_orders.yml +++ b/models/exchange/marts/reporting/rep_exchange__daily_orders/test_exchange__daily_orders.yml @@ -2,51 +2,27 @@ version: 2 models: - name: rep_exchange__daily_orders - - # @YAM: This is the test code that I write - # @YAM: I already do some space in the YML, but not all the stuff is clean, please clean the code, and makesure it is easy readable. - # columns: - # - name: base_value - # description: test base value combine with dbt_utils - # tests: - # - not_null - # - dbt_utils.expression_is_true: - # expression: " >= 0" - tests: - - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: - column_A: order_completed_datetime - column_B: order_created_datetime - or_equal: True - - - dbt_utils.expression_is_true: - expression: "COST_TRANSACTION_VALUE_RM + COST_TRANSACTION_TAX_RM + COST_TRANSACTION_SERVICE_CHARGE_RM < 2*(TRANSACTION_VALUE_RM + TRANSACTION_BASE_TAX + TRANSACTION_TAX_RM)" - severity: warn columns: - - name: ID + - name: id description: transaction orders id tests: - unique - not_null - - name: BOOKING_ID + - name: booking_id description: transaction orders id tests: - not_null - relationships: to: ref('base_exchange__bookings') field: id - - - name: company_marking_id - description: transaction orders id - tests: - - not_null - - name: SERVICE_TYPE + - name: service_type description: type of service tests: - not_null - - name: SERVICE_TYPE_NAME + - name: service_type_name description: name of type of service tests: - not_null @@ -59,55 +35,55 @@ models: - name: seller_company_id description: seller company id tests: - - not_null - + - not_null + - name: bank_id description: bank id tests: - - not_null - + - not_null + - name: company_id description: user id tests: - - not_null - - - name: marking_id - description: marking id + - not_null + + - name: company_marking_id + description: company marking id tests: - relationships: - to: ref('base_exchange_companies') - field: id + to: ref('base_exchange__companies') + field: marking_id - name: company_type description: type of company tests: - - not_null - - - name: COMPANY_TYPE_NAME + - not_null + + - name: company_type_name description: name type of company tests: - accepted_values: values: - PERSONAL - - CORPORATE - + - CORPORATE + - name: company_business_type description: company business type tests: - - not_null - + - not_null + - name: company_business_type_name description: company business type name tests: - accepted_values: values: - IMPORTER - + - name: company_segment_id description: company segment id tests: - - not_null - + - not_null + - name: company_segment_name description: company segment name tests: @@ -116,109 +92,120 @@ models: - PLATINIUM MEMBER - GOLD MEMBER - STANDARD SEGMENT - + - name: cost_id description: cost id tests: - relationships: to: ref('base_exchange__transactions') field: id - + - name: base_value description: test base value combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - + - name: quote_value description: test quote value combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - + - name: transaction_value_rm description: transaction value rm combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - + - name: transaction_base_tax description: test transaction base tax combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - + - name: base_service_charge description: test base service charge combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - + - name: transaction_tax_rm description: test transaction tax rm combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - + - name: cost_base_value description: test cost base value combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - - - name: COST_BASE_TO_QUOTE_CURRENCY_EXCHANGE_RATE - description: test COST BASE TO QUOTE CURRENCY EXCHANGE RATE combine with dbt_utils + + - name: cost_base_to_quote_currency_exchange_rate + description: test cost base to quote currency exchange rate combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - - - name: COST_QUOTE_VALUE - description: test COST QUOTE VALUE combine with dbt_utils + + - name: cost_quote_value + description: test cost quote value combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - - - name: COST_TRANSACTION_VALUE_RM + + - name: cost_transaction_value_rm description: test COST TRANSACTION VALUE RM combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - - - name: COST_TRANSACTION_TAX_RM + + - name: cost_transaction_tax_rm description: test COST TRANSACTION TAX RM combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - - - name: COST_BASE_SERVICE_CHARGE + + - name: cost_base_service_charge description: test COST BASE SERVICE CHARGE combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - - - name: COST_TRANSACTION_SERVICE_CHARGE_RM + + - name: cost_transaction_service_charge_rm description: test COST TRANSACTION SERVICE CHARGE_RM combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" - + - name: transaction_service_charge_rm description: test transaction service charge rm combine with dbt_utils tests: - not_null - dbt_utils.expression_is_true: expression: " >= 0" + + tests: + - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: + column_A: order_completed_datetime + column_B: order_created_datetime + or_equal: True + tests: + - dbt_utils.expression_is_true: + expression: "cost_transaction_value_rm + cost_transaction_tax_rm + cost_transaction_service_charge_rm < 2*(transaction_value_rm + transaction_base_tax + transaction_tax_rm)" + severity: warn + # FRESHNESS SOURCE diff --git a/models/exchange/marts/warehouse/fct_exchange__wallet_debit_time_duration.sql b/models/exchange/marts/warehouse/fct_exchange__wallet_debit_time_duration.sql deleted file mode 100644 index 464b99d..0000000 --- a/models/exchange/marts/warehouse/fct_exchange__wallet_debit_time_duration.sql +++ /dev/null @@ -1,45 +0,0 @@ -with - wallet_credit_log as ( - select * from {{ ref("int_exchange__wallet_log_credit_amount") }} - ), - wallet_debit_time_log as ( - select - dense_rank() over ( - partition by wallet_id order by updated_at - ) as operation_sequence, - lag(updated_at) over ( - partition by wallet_id order by updated_at - ) previous_update_time, - datediff( - second, previous_update_time, updated_at - ) as time_between_operation_in_seconds, - * - from wallet_credit_log - ), - - updated_wallet_debit_time_log as ( - - select - *, - iff( - (credit_status = 'DEBIT'), time_between_operation_in_seconds, 0 - ) as seconds_required_to_debit - from wallet_debit_time_log - ), - final_wallet_debit_time_log as ( - select - * exclude(seconds_required_to_debit), - iff( - equal_null(seconds_required_to_debit, null), - 0, - seconds_required_to_debit - ) as seconds_required_to_debit - from updated_wallet_debit_time_log - ), - final_wallet_debit_time_log_reformed as ( - select * exclude(TIME_BETWEEN_OPERATION_IN_SECONDS) - from final_wallet_debit_time_log - where credit_status = 'DEBIT' and seconds_required_to_debit != 0 - ) -select * -from final_wallet_debit_time_log_reformed diff --git a/models/exchange/marts/warehouse/fct_exchange__wallet_log_top_10_top_up.sql b/models/exchange/marts/warehouse/fct_exchange__wallet_log_top_10_top_up.sql deleted file mode 100644 index e0d7ba1..0000000 --- a/models/exchange/marts/warehouse/fct_exchange__wallet_log_top_10_top_up.sql +++ /dev/null @@ -1,17 +0,0 @@ --- IMPORTS -with - wallet_credit_log as ( - select * from {{ ref("int_exchange__wallet_log_credit_amount") }} - ), - - wallet_credit_top_up_frequency_report as ( - select credited_amount, count(credited_amount) repetation - from wallet_credit_log - where credit_status = 'CREDIT' - group by credited_amount - order by repetation desc - limit 10 - ) - -select * -from wallet_credit_top_up_frequency_report diff --git a/models/exchange/marts/warehouse/fct_exchange__wallet_log_wallet_amount_comparison.sql b/models/exchange/marts/warehouse/fct_exchange__wallet_log_wallet_amount_comparison.sql deleted file mode 100644 index 732a061..0000000 --- a/models/exchange/marts/warehouse/fct_exchange__wallet_log_wallet_amount_comparison.sql +++ /dev/null @@ -1,24 +0,0 @@ -with wallet_log as ( - select * from {{ref('int_exchange_wallet_final_log_sequence')}} -), -wallet_details as ( - select * from {{ref('stg_exchange__wallet')}} -), - -wallet_comparison_log as ( -select - w.id as wallet_id, - w.amount as wallet_amount, - s.current_wallet_balance as wallet_balance_from_log, - w.owner_type, - s.credit_status, - s.updated_at as wallet_log_update_time, - s._dbt_ran_at, - w.updated_at as wallet_update_time -from wallet_details w -left join - wallet_log s - on w.id = s.wallet_id -) - -select * from wallet_comparison_log diff --git a/models/exchange/marts/warehouse/test_exchange__bookings.yml b/models/exchange/marts/warehouse/test_exchange__bookings.yml index 2c5698f..e69de29 100644 --- a/models/exchange/marts/warehouse/test_exchange__bookings.yml +++ b/models/exchange/marts/warehouse/test_exchange__bookings.yml @@ -1,156 +0,0 @@ -version: 2 -models: - - name: fct_exchange__bookings - - tests: - - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: - column_A: BOOKING_UPDATED_DATETIME - column_B: BOOKING_CREATED_DATETIME - or_equal: True - - - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: - column_A: BOOKING_DELETED_DATETIME - column_B: BOOKING_CREATED_DATETIME - or_equal: True - - columns: - - name: ID - description: boking id - tests: - - unique - - not_null - - - name: COMPANY_ID - description: company id - tests: - - unique - - not_null - - - name: USER_ID - description: user id - tests: - - unique - - not_null - - - name: MARKING_ID - description: marking id - tests: - - unique - - not_null - - - name: SERVICE_TYPE - description: type of service - tests: - - not_null - - accepted_values: - values: - - 1 - - 3 - - 4 - - - name: SERVICE_TYPE_NAME - description: name of type of service - tests: - - not_null - - accepted_values: - values: - - 3 DAYS TRANSFER - - 1 DAY TRANSFER - - 1688 PAYMENT - - - name: BANK_ID - description: bank id - tests: - - unique - - not_null - - - name: FIX_VALUE - description: test fix value combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: FIX_CURRENCY_ID - description: fix currency id - tests: - - not_null - - accepted_values: - values: - - 1 - - 3 - - 2 - - - name: FIX_CURRENCY_NAME - description: fix currency name - tests: - - not_null - - accepted_values: - values: - - Malaysian Ringgit - - US Dollar - - Yuan Renminbi - - - name: ESTIMATE_BASE_TO_QUOTE_CURRENCY_EXCHANGE_RATE - description: test estimate base to quote currency exchange rate value combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: QUOTE_CURRENCY_ID - description: quote currency id - tests: - - not_null - - accepted_values: - values: - - 3 - - 2 - - - name: QUOTE_CURRENCY_NAME - description: quote currency name - tests: - - not_null - - accepted_values: - values: - - US Dollar - - Yuan Renminbi - - - name: ESTIMATE_QUOTE_VALUE - description: test estimate quote value value combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: ESTIMATE_BASE_VALUE - description: test estimate quote value value combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: ESTIMATE_VALUE_RM - description: test estimate quote value value combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: STATUS - description: status - tests: - - accepted_values: - values: - - 2 - - 3 - - 5 - - - name: STATUS_NAME - description: status name - tests: - - accepted_values: - values: - - SUSPENDED - - APPROVED - - COMPLETED diff --git a/models/exchange/marts/warehouse/test_exchange__companies.yml b/models/exchange/marts/warehouse/test_exchange__companies.yml index 132503f..e69de29 100644 --- a/models/exchange/marts/warehouse/test_exchange__companies.yml +++ b/models/exchange/marts/warehouse/test_exchange__companies.yml @@ -1,146 +0,0 @@ -version: 2 -models: - - name: dim_exchange__companies - tests: - - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: - column_A: ADDRESS_DELETED_AT - column_B: ADDRESS_CREATED_AT - or_equal: True - - - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: - column_A: ADDRESS_UPDATED_AT - column_B: ADDRESS_CREATED_AT - or_equal: True - - columns: - #Gerson todo Why this company id need to ref with addresses? - - name: ID - description: company id - tests: - - unique - - not_null - - relationships: - to: ref('base_exchange__addresses') - field: company_id - - - name: NAME - description: company name - tests: - - not_null - - #Yam todo there is marking_id in supplier, need mask them - - name: MARKING_ID - description: marking id - tests: - - unique - - not_null - - - name: COMPANY_TYPE - description: type of company - tests: - - accepted_values: - values: - - 0 - - 1 - - 2 - config: - where: "COMPANY_TYPE is not null" - - - name: COMPANY_TYPE_NAME - description: name of type company - tests: - - accepted_values: - values: - - CORPORATE - - PERSONAL - config: - where: "COMPANY_TYPE_NAME is not null" - - - name: BUSINESS_TYPE - description: type of business - tests: - - accepted_values: - values: - - 1 - - 2 - - 3 - config: - where: "BUSINESS_TYPE is not null" - - - name: BUSINESS_TYPE_NAME - description: name of type business - tests: - - accepted_values: - values: - - IMPORTER - - CURRENCY_VENDOR - - FREIGHT_FORWARDER - config: - where: "BUSINESS_TYPE_NAME is not null" - - - name: SEGMENT_ID - description: segment id - tests: - - accepted_values: - values: - - 1 - - 2 - - 3 - config: - where: "SEGMENT_ID is not null" - - - name: SEGMENT_NAME - description: segment name - tests: - - accepted_values: - values: - - STANDARD SEGMENT - - GOLD MEMBER - - PLATINIUM MEMBER - config: - where: "SEGMENT_NAME is not null" - - - name: IS_MIGRATED_COMPANY - description: is migrate company - tests: - - accepted_values: - values: - - 1 - - 0 - - - name: STATUS - description: status - tests: - - accepted_values: - values: - - 0 - - 1 - - 2 - - 4 - - 5 - - - name: STATUS_NAME - description: status name - tests: - - accepted_values: - values: - - SUSPENDED - - APPROVED - - PENDING_SUBMISSION - - REJECTED - - PENDING_VERIFICATION - - - name: COMPANY_CONTACT_EMAIL - description: company contact email - tests: - - not_null - - dbt_expectations.expect_column_values_to_match_regex: - regex: "[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*@[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*[.][a-zA-Z]{2,5}" - severity: warn - - - name: marking_id - description: marking id - tests: - - relationships: - to: ref('base_exchange_companies') - field: id \ No newline at end of file diff --git a/models/exchange/marts/warehouse/test_exchange__transaction_orders.yml b/models/exchange/marts/warehouse/test_exchange__transaction_orders.yml index 8feac32..e69de29 100644 --- a/models/exchange/marts/warehouse/test_exchange__transaction_orders.yml +++ b/models/exchange/marts/warehouse/test_exchange__transaction_orders.yml @@ -1,163 +0,0 @@ -version: 2 -models: - - name: fct_exchange__transaction_orders - - tests: - - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: - column_A: UPDATED_AT - column_B: CREATED_AT - or_equal: True - - - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: - column_A: DELETED_AT - column_B: CREATED_AT - or_equal: True - - columns: - - name: ID - description: transaction id - tests: - - unique - - not_null - - relationships: - to: ref('int_exchange__transaction_orders_get_user_ids') - field: id - - name: BOOKING_ID - description: booking id - tests: - - unique - - not_null - - relationships: - to: ref('int_exchange__transaction_orders_get_user_ids') - field: BOOKING_ID - - name: SELLER_COMPANY_ID - description: seller company id - tests: - - unique - - not_null - - relationships: - to: ref('int_exchange__transaction_orders_get_user_ids') - field: SELLER_COMPANY_ID - - name: BANK_ID - description: bank id - tests: - - unique - - not_null - - relationships: - to: ref('int_exchange__transaction_orders_get_user_ids') - field: BANK_ID - - name: COMPANY_ID - description: company id - tests: - - unique - - not_null - - relationships: - to: ref('int_exchange__transaction_orders_get_user_ids') - field: COMPANY_ID - - - name: STATUS - description: status - tests: - - accepted_values: - values: - - 0 - - 1 - - 2 - - 3 - - 4 - - 5 - - - name: STATUS_NAME - description: status name - tests: - - accepted_values: - values: - - COMPLETED - - SUSPENDED - - REJECTED - - PENDING_SUBMISSION - - PENDING_VERIFICATION - - APPROVED - - - name: TRANSACTION_TYPE - description: transaction type - tests: - - accepted_values: - values: - - 1 - - - name: TRANSACTION_TYPE_NAME - description: name transaction type - tests: - - accepted_values: - values: - - PAYMENT - - - name: PAYMENT_METHOD - description: method payment - tests: - - accepted_values: - values: - - 1 - - 2 - - 5 - - 4 - - - name: PAYMENT_METHOD_NAME - description: name method payment - tests: - - accepted_values: - values: - - CASH - - CHEQUE - - PAYMENT_GATEWAY - - WALLET - - - name: base_value - description: test base value combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: quote_value - description: test quote value combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: transaction_value_rm - description: transaction value rm combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: transaction_base_tax - description: test transaction base tax combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: transaction_tax_rm - description: test transaction tax rm combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: base_service_charge - description: test base service charge combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" - - - name: TRANSACTION_SERVICE_CHARGE_RM - description: test transaction service change rm value combine with dbt_utils - tests: - - not_null - - dbt_utils.expression_is_true: - expression: " >= 0" \ No newline at end of file diff --git a/models/exchange/marts/warehouse/test_exchange__users.yml b/models/exchange/marts/warehouse/test_exchange__users.yml index d0f3682..e69de29 100644 --- a/models/exchange/marts/warehouse/test_exchange__users.yml +++ b/models/exchange/marts/warehouse/test_exchange__users.yml @@ -1,76 +0,0 @@ -version: 2 -models: - - name: dim_exchange__users - tests: - - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: - column_A: DELETED_AT - column_B: CREATED_AT - or_equal: True - - - - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B: - column_A: EMAIL_VERIFICATION_LAST_EMAIL_CREATED_AT - column_B: EMAIL_VERIFICATION_FIRST_EMAIL_CREATED_AT - or_equal: True - - columns: - - name: ID - description: user id - tests: - - unique - - not_null - - - name: NAME - description: user name - tests: - - not_null - - - name: USER_TYPE - description: type of user - tests: - - not_null - - accepted_values: - values: - - 1 - - 2 - - - name: USER_TYPE_NAME - description: type of user - tests: - - not_null - - accepted_values: - values: - - USER - - ADMIN - - - name: STATUS - description: user status - tests: - - not_null - - accepted_values: - values: - - 1 - - 2 - - - name: STATUS_NAME - description: user status name - tests: - - not_null - - accepted_values: - values: - - APPROVED - - PENDING_VERIFICATION - - - name: COMPANY_ID - description: company id - tests: - - unique - - not_null - - - name: EMAIL - description: user email - tests: - - not_null - - dbt_expectations.expect_column_values_to_match_regex: - regex: "[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*@[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*[.][a-zA-Z]{2,5}" - severity: warn \ No newline at end of file diff --git a/models/exchange/staging/activity_logs/stg_exchange__activity_logs.sql b/models/exchange/staging/activity_logs/stg_exchange__activity_logs.sql new file mode 100644 index 0000000..3e83c4a --- /dev/null +++ b/models/exchange/staging/activity_logs/stg_exchange__activity_logs.sql @@ -0,0 +1,46 @@ +-- IMPORTS +WITH activity_logs AS ( + SELECT * FROM {{ ref('base_exchange__activity_logs') }} +), + + +-- LOGIC +activity_logs_listout AS ( + + SELECT + activity_logs.id, + activity_logs.log_name, + activity_logs.description, + activity_logs.subject_id, + activity_logs.subject_type, + activity_logs.causer_id, + activity_logs.causer_type, + activity_logs.properties, + activity_logs.created_at, + activity_logs.updated_at, + CURRENT_TIMESTAMP() AS _dbt_ran_at + + FROM activity_logs +), + + +-- FINAL +final__stg_exchange__activity_logs AS ( + + SELECT + id, + log_name, + description, + subject_id, + subject_type, + causer_id, + causer_type, + properties, + created_at, + updated_at, + _dbt_ran_at + + FROM activity_logs_listout +) + +SELECT * FROM final__stg_exchange__activity_logs \ No newline at end of file diff --git a/models/staging/exchange/stg_exchange__announcement_segments.sql b/models/exchange/staging/announcement_segments/stg_exchange__announcement_segments.sql similarity index 100% rename from models/staging/exchange/stg_exchange__announcement_segments.sql rename to models/exchange/staging/announcement_segments/stg_exchange__announcement_segments.sql diff --git a/models/staging/exchange/stg_exchange__announcements.sql b/models/exchange/staging/announcements/stg_exchange__announcements.sql similarity index 100% rename from models/staging/exchange/stg_exchange__announcements.sql rename to models/exchange/staging/announcements/stg_exchange__announcements.sql diff --git a/models/staging/exchange/stg_exchange__bank_logs.sql b/models/exchange/staging/bank_logs/stg_exchange__bank_logs.sql similarity index 100% rename from models/staging/exchange/stg_exchange__bank_logs.sql rename to models/exchange/staging/bank_logs/stg_exchange__bank_logs.sql diff --git a/models/staging/exchange/stg_exchange__banks.sql b/models/exchange/staging/banks/stg_exchange__banks.sql similarity index 100% rename from models/staging/exchange/stg_exchange__banks.sql rename to models/exchange/staging/banks/stg_exchange__banks.sql diff --git a/models/staging/exchange/stg_exchange__booking_logs.sql b/models/exchange/staging/booking_logs/stg_exchange__booking_logs.sql similarity index 100% rename from models/staging/exchange/stg_exchange__booking_logs.sql rename to models/exchange/staging/booking_logs/stg_exchange__booking_logs.sql diff --git a/models/staging/exchange/stg_exchange__bookings.sql b/models/exchange/staging/bookings/stg_exchange__bookings.sql similarity index 100% rename from models/staging/exchange/stg_exchange__bookings.sql rename to models/exchange/staging/bookings/stg_exchange__bookings.sql diff --git a/models/staging/exchange/stg_exchange__companies.sql b/models/exchange/staging/companies/stg_exchange__companies.sql similarity index 100% rename from models/staging/exchange/stg_exchange__companies.sql rename to models/exchange/staging/companies/stg_exchange__companies.sql diff --git a/models/staging/exchange/stg_exchange__companies_and_segments_relations.sql b/models/exchange/staging/companies_and_segments_relations/stg_exchange__companies_and_segments_relations.sql similarity index 100% rename from models/staging/exchange/stg_exchange__companies_and_segments_relations.sql rename to models/exchange/staging/companies_and_segments_relations/stg_exchange__companies_and_segments_relations.sql diff --git a/models/staging/exchange/stg_exchange__companies_and_users_relations.sql b/models/exchange/staging/companies_and_users_relations/stg_exchange__companies_and_users_relations.sql similarity index 100% rename from models/staging/exchange/stg_exchange__companies_and_users_relations.sql rename to models/exchange/staging/companies_and_users_relations/stg_exchange__companies_and_users_relations.sql diff --git a/models/staging/exchange/stg_exchange__company_addresses.sql b/models/exchange/staging/company_addresses/stg_exchange__company_addresses.sql similarity index 100% rename from models/staging/exchange/stg_exchange__company_addresses.sql rename to models/exchange/staging/company_addresses/stg_exchange__company_addresses.sql diff --git a/models/staging/exchange/stg_exchange__contacts.sql b/models/exchange/staging/contacts/stg_exchange__contacts.sql similarity index 100% rename from models/staging/exchange/stg_exchange__contacts.sql rename to models/exchange/staging/contacts/stg_exchange__contacts.sql diff --git a/models/staging/exchange/stg_exchange__countries.sql b/models/exchange/staging/countries/stg_exchange__countries.sql similarity index 100% rename from models/staging/exchange/stg_exchange__countries.sql rename to models/exchange/staging/countries/stg_exchange__countries.sql diff --git a/models/staging/exchange/stg_exchange__currencies.sql b/models/exchange/staging/currencies/stg_exchange__currencies.sql similarity index 100% rename from models/staging/exchange/stg_exchange__currencies.sql rename to models/exchange/staging/currencies/stg_exchange__currencies.sql diff --git a/models/staging/exchange/stg_exchange__currency_logs.sql b/models/exchange/staging/currency_logs/stg_exchange__currency_logs.sql similarity index 100% rename from models/staging/exchange/stg_exchange__currency_logs.sql rename to models/exchange/staging/currency_logs/stg_exchange__currency_logs.sql diff --git a/models/staging/exchange/stg_exchange__currency_rate_logs.sql b/models/exchange/staging/currency_rate_logs/stg_exchange__currency_rate_logs.sql similarity index 100% rename from models/staging/exchange/stg_exchange__currency_rate_logs.sql rename to models/exchange/staging/currency_rate_logs/stg_exchange__currency_rate_logs.sql diff --git a/models/staging/exchange/stg_exchange__current_currency_rates.sql b/models/exchange/staging/current_currency_rates/stg_exchange__current_currency_rates.sql similarity index 100% rename from models/staging/exchange/stg_exchange__current_currency_rates.sql rename to models/exchange/staging/current_currency_rates/stg_exchange__current_currency_rates.sql diff --git a/models/staging/exchange/stg_exchange__current_wallets.sql b/models/exchange/staging/current_wallets/stg_exchange__current_wallets.sql similarity index 100% rename from models/staging/exchange/stg_exchange__current_wallets.sql rename to models/exchange/staging/current_wallets/stg_exchange__current_wallets.sql diff --git a/models/staging/exchange/stg_exchange__districts.sql b/models/exchange/staging/districts/stg_exchange__districts.sql similarity index 100% rename from models/staging/exchange/stg_exchange__districts.sql rename to models/exchange/staging/districts/stg_exchange__districts.sql diff --git a/models/staging/exchange/stg_exchange__documents.sql b/models/exchange/staging/documents/stg_exchange__documents.sql similarity index 100% rename from models/staging/exchange/stg_exchange__documents.sql rename to models/exchange/staging/documents/stg_exchange__documents.sql diff --git a/models/staging/exchange/stg_exchange__failed_jobs.sql b/models/exchange/staging/failed_jobs/stg_exchange__failed_jobs.sql similarity index 100% rename from models/staging/exchange/stg_exchange__failed_jobs.sql rename to models/exchange/staging/failed_jobs/stg_exchange__failed_jobs.sql diff --git a/models/staging/exchange/stg_exchange__files.sql b/models/exchange/staging/files/stg_exchange__files.sql similarity index 100% rename from models/staging/exchange/stg_exchange__files.sql rename to models/exchange/staging/files/stg_exchange__files.sql diff --git a/models/staging/exchange/stg_exchange__group_costs.sql b/models/exchange/staging/group_costs/stg_exchange__group_costs.sql similarity index 100% rename from models/staging/exchange/stg_exchange__group_costs.sql rename to models/exchange/staging/group_costs/stg_exchange__group_costs.sql diff --git a/models/staging/exchange/stg_exchange__jobs.sql b/models/exchange/staging/jobs/stg_exchange__jobs.sql similarity index 100% rename from models/staging/exchange/stg_exchange__jobs.sql rename to models/exchange/staging/jobs/stg_exchange__jobs.sql diff --git a/models/staging/exchange/stg_exchange__migrations.sql b/models/exchange/staging/migrations/stg_exchange__migrations.sql similarity index 100% rename from models/staging/exchange/stg_exchange__migrations.sql rename to models/exchange/staging/migrations/stg_exchange__migrations.sql diff --git a/models/staging/exchange/stg_exchange__model_has_roles.sql b/models/exchange/staging/model_has_roles/stg_exchange__model_has_roles.sql similarity index 100% rename from models/staging/exchange/stg_exchange__model_has_roles.sql rename to models/exchange/staging/model_has_roles/stg_exchange__model_has_roles.sql diff --git a/models/staging/exchange/stg_exchange__notifications.sql b/models/exchange/staging/notifications/stg_exchange__notifications.sql similarity index 100% rename from models/staging/exchange/stg_exchange__notifications.sql rename to models/exchange/staging/notifications/stg_exchange__notifications.sql diff --git a/models/staging/exchange/stg_exchange__password_resets.sql b/models/exchange/staging/password_resets/stg_exchange__password_resets.sql similarity index 100% rename from models/staging/exchange/stg_exchange__password_resets.sql rename to models/exchange/staging/password_resets/stg_exchange__password_resets.sql diff --git a/models/staging/exchange/stg_exchange__permissions.sql b/models/exchange/staging/permissions/stg_exchange__permissions.sql similarity index 100% rename from models/staging/exchange/stg_exchange__permissions.sql rename to models/exchange/staging/permissions/stg_exchange__permissions.sql diff --git a/models/staging/exchange/stg_exchange__role_has_permissions.sql b/models/exchange/staging/role_has_permissions/stg_exchange__role_has_permissions.sql similarity index 100% rename from models/staging/exchange/stg_exchange__role_has_permissions.sql rename to models/exchange/staging/role_has_permissions/stg_exchange__role_has_permissions.sql diff --git a/models/staging/exchange/stg_exchange__roles.sql b/models/exchange/staging/roles/stg_exchange__roles.sql similarity index 100% rename from models/staging/exchange/stg_exchange__roles.sql rename to models/exchange/staging/roles/stg_exchange__roles.sql diff --git a/models/staging/exchange/stg_exchange__segment_constants.sql b/models/exchange/staging/segment_constants/stg_exchange__segment_constants.sql similarity index 100% rename from models/staging/exchange/stg_exchange__segment_constants.sql rename to models/exchange/staging/segment_constants/stg_exchange__segment_constants.sql diff --git a/models/staging/exchange/stg_exchange__segments.sql b/models/exchange/staging/segments/stg_exchange__segments.sql similarity index 100% rename from models/staging/exchange/stg_exchange__segments.sql rename to models/exchange/staging/segments/stg_exchange__segments.sql diff --git a/models/staging/exchange/stg_exchange__service_types.sql b/models/exchange/staging/service_types/stg_exchange__service_types.sql similarity index 100% rename from models/staging/exchange/stg_exchange__service_types.sql rename to models/exchange/staging/service_types/stg_exchange__service_types.sql diff --git a/models/staging/exchange/stg_exchange__states.sql b/models/exchange/staging/states/stg_exchange__states.sql similarity index 100% rename from models/staging/exchange/stg_exchange__states.sql rename to models/exchange/staging/states/stg_exchange__states.sql diff --git a/models/staging/exchange/stg_exchange__transaction_costs.sql b/models/exchange/staging/transaction_costs/stg_exchange__transaction_costs.sql similarity index 100% rename from models/staging/exchange/stg_exchange__transaction_costs.sql rename to models/exchange/staging/transaction_costs/stg_exchange__transaction_costs.sql diff --git a/models/staging/exchange/stg_exchange__transaction_costs_and_group_costs_relations.sql b/models/exchange/staging/transaction_costs_and_group_costs_relations/stg_exchange__transaction_costs_and_group_costs_relations.sql similarity index 100% rename from models/staging/exchange/stg_exchange__transaction_costs_and_group_costs_relations.sql rename to models/exchange/staging/transaction_costs_and_group_costs_relations/stg_exchange__transaction_costs_and_group_costs_relations.sql diff --git a/models/staging/exchange/stg_exchange__transaction_log_costs.sql b/models/exchange/staging/transaction_log_costs/stg_exchange__transaction_log_costs.sql similarity index 100% rename from models/staging/exchange/stg_exchange__transaction_log_costs.sql rename to models/exchange/staging/transaction_log_costs/stg_exchange__transaction_log_costs.sql diff --git a/models/staging/exchange/stg_exchange__transaction_log_orders.sql b/models/exchange/staging/transaction_log_orders/stg_exchange__transaction_log_orders.sql similarity index 100% rename from models/staging/exchange/stg_exchange__transaction_log_orders.sql rename to models/exchange/staging/transaction_log_orders/stg_exchange__transaction_log_orders.sql diff --git a/models/staging/exchange/stg_exchange__transaction_order_details.sql b/models/exchange/staging/transaction_order_details/stg_exchange__transaction_order_details.sql similarity index 100% rename from models/staging/exchange/stg_exchange__transaction_order_details.sql rename to models/exchange/staging/transaction_order_details/stg_exchange__transaction_order_details.sql diff --git a/models/staging/exchange/stg_exchange__transaction_orders.sql b/models/exchange/staging/transaction_orders/stg_exchange__transaction_orders.sql similarity index 100% rename from models/staging/exchange/stg_exchange__transaction_orders.sql rename to models/exchange/staging/transaction_orders/stg_exchange__transaction_orders.sql diff --git a/models/staging/exchange/stg_exchange__transaction_wallets.sql b/models/exchange/staging/transaction_wallets/stg_exchange__transaction_wallets.sql similarity index 100% rename from models/staging/exchange/stg_exchange__transaction_wallets.sql rename to models/exchange/staging/transaction_wallets/stg_exchange__transaction_wallets.sql diff --git a/models/staging/exchange/stg_exchange__transactions.sql b/models/exchange/staging/transactions/stg_exchange__transactions.sql similarity index 100% rename from models/staging/exchange/stg_exchange__transactions.sql rename to models/exchange/staging/transactions/stg_exchange__transactions.sql diff --git a/models/staging/exchange/stg_exchange__user_email_verifications.sql b/models/exchange/staging/user_email_verifications/stg_exchange__user_email_verifications.sql similarity index 100% rename from models/staging/exchange/stg_exchange__user_email_verifications.sql rename to models/exchange/staging/user_email_verifications/stg_exchange__user_email_verifications.sql diff --git a/models/staging/exchange/stg_exchange__users.sql b/models/exchange/staging/users/stg_exchange__users.sql similarity index 100% rename from models/staging/exchange/stg_exchange__users.sql rename to models/exchange/staging/users/stg_exchange__users.sql diff --git a/models/exchange/staging/wallet/stg_exchange__wallet.sql b/models/exchange/staging/wallet/stg_exchange__wallet.sql deleted file mode 100644 index 60fdb61..0000000 --- a/models/exchange/staging/wallet/stg_exchange__wallet.sql +++ /dev/null @@ -1,10 +0,0 @@ --- IMPORTS -WITH wallets AS ( - SELECT * FROM {{ ref('base_exchange__wallets') }} -) - - - - --- FINAL -SELECT * FROM wallets \ No newline at end of file diff --git a/models/exchange/staging/wallet_logs/stg_exchange__wallet_logs.sql b/models/exchange/staging/wallet_logs/stg_exchange__wallet_logs.sql deleted file mode 100644 index 2e0ad6f..0000000 --- a/models/exchange/staging/wallet_logs/stg_exchange__wallet_logs.sql +++ /dev/null @@ -1,7 +0,0 @@ --- IMPORTS -with wallet_logs as (select * from {{ ref("base_exchange__wallet_logs") }}) - --- FINAL -select * -from wallet_logs -order by wallet_id, updated_at asc diff --git a/models/shipping/staging/stg_shipping__activity_logs.sql b/models/shipping/staging/stg_shipping__activity_logs.sql deleted file mode 100644 index 8f30a63..0000000 --- a/models/shipping/staging/stg_shipping__activity_logs.sql +++ /dev/null @@ -1,46 +0,0 @@ --- IMPORTS -WITH activity_logs AS ( - SELECT * FROM {{ source('src_shipping_mysql', 'activity_log') }} -), - - --- LOGIC -activity_logs_rename AS ( - - SELECT - activity_logs.id, - activity_logs.log_name, - activity_logs.description, - activity_logs.subject_id, - activity_logs.subject_type, - activity_logs.causer_id, - activity_logs.causer_type, - activity_logs.properties, - activity_logs.created_at AS created_datetime, - activity_logs.updated_at AS updated_datetime, - current_timestamp() AS _dbt_ran_at - - FROM activity_logs -), - - --- FINAL -final__base_exchange__activity_logs AS ( - - SELECT - id, - log_name, - description, - subject_id, - subject_type, - causer_id, - causer_type, - properties, - created_datetime, - updated_datetime, - _dbt_ran_at - - FROM activity_logs_rename -) - -SELECT * FROM final__base_exchange__activity_logs \ No newline at end of file diff --git a/models/staging/autocount/_autocount__sources.yml b/models/staging/autocount/_autocount__sources.yml deleted file mode 100644 index e69de29..0000000 diff --git a/models/staging/crisp/_crisp__sources.yml b/models/staging/crisp/_crisp__sources.yml deleted file mode 100644 index e69de29..0000000 diff --git a/models/staging/exchange/_exchange__docs.md b/models/staging/exchange/_exchange__docs.md deleted file mode 100644 index ed501d3..0000000 --- a/models/staging/exchange/_exchange__docs.md +++ /dev/null @@ -1,23 +0,0 @@ -{% docs src_exchange_mysql %} - -Data is extract from CIEF Selfhost MYSQL `exchange_production`, using custom python code. - -- `_source_loaded_at` will add into the raw data when extract -```YML -loaded_at_field: _source_loaded_at -freshness: - warn_after: {count: 12, period: hour} - error_after: {count: 24, period: hour} -``` - -- The raw data will use copy into snowflake in 2AM everyday - -- Depend on target.name to determin which snowflake raw schema - -```JINJA -{%- if target.name == "prod" -%} PROD_CIEF_RAW_DB -{%- else -%} DEV_CIEF_RAW_DB -{%- endif -%} -``` - -{% enddocs %} \ No newline at end of file diff --git a/models/staging/exchange/_exchange__models.yml b/models/staging/exchange/_exchange__models.yml deleted file mode 100644 index cc0f92b..0000000 --- a/models/staging/exchange/_exchange__models.yml +++ /dev/null @@ -1,6 +0,0 @@ -version: 2 - -models: - - name: stg_exchange__activity_logs - - - name: stg_exchange__announcments \ No newline at end of file diff --git a/models/staging/exchange/_exchange__sources.yml b/models/staging/exchange/_exchange__sources.yml deleted file mode 100644 index d511863..0000000 --- a/models/staging/exchange/_exchange__sources.yml +++ /dev/null @@ -1,619 +0,0 @@ -version: 2 - -sources: - - name: src_exchange_mysql - description: '{{ doc("src_exchange_mysql") }}' - database: | - {%- if target.name == "prod" -%} PROD_CIEF_RAW_DB - {%- else -%} DEV_CIEF_RAW_DB - {%- endif -%} - schema: EXCHANGE_MYSQL - loader: custom_python_code - loaded_at_field: _source_loaded_at - freshness: - warn_after: {count: 12, period: hour} - error_after: {count: 24, period: hour} - meta: - owner: "@yam" - model_maturity: prod - tags: - - exchange - - daily - - - tables: - - name: activity_log - description: Contains most table's activity log that CUD (created, updated, deleted) in www.exchange.cief-malaysia.com - columns: - - name: id - description: Primary key for 'activity_log' - tests: - - unique - - not_null - - - name: description - description: system CUD activity list down - tests: - - accepted_values: - values: ['created', 'updated', 'deleted'] - - - name: subject_id - description: The exchange CUD table's id, need to refer subject_type column to know what table is CUD in the row - tests: - - not_null - - - name: subject_type - description: What table that CUD using in the row - tests: - - not_null - - - name: causer_id - description: Need to refer causer_type column to know who is CUD in the row. it is not well maintain there is some NULL - - - name: causer_type - description: Who is CUD the row - - - name: properties - description: Json data format that contain attributes of the activity, it is not well maintain, subject (Country, ServiceType, CurrencyRate. UserEmailVerification, File) have properties only - - - name: created_at - description: Datetime that CUD activity occur - tests: - - not_null - - - name: updated_at - description: Exactly same with created_at, redundant column - - tests: - - dbt_expectations.expect_column_pair_values_to_be_equal: - column_A: updated_at - column_B: created_at - - - - name: addresses - description: Addresses that input when user register - columns: - - name: id - description: Primary key for 'addresses' - tests: - - unique - - not_null - - - name: company_id - description: Foreign keys for 'companies' - tests: - - not_null - - - name: country_id - description: Foreign keys for 'countries' - tests: - - not_null - - - name: state_id - description: Foreign keys for 'states' - tests: - - not_null - - - name: district_id - description: Foreign keys for 'districts' - tests: - - not_null - - - name: postcode - description: The number/string that user input - tests: - - not_null - - dbt_expectations.expect_column_values_to_be_of_type: - column_type: integer - - - name: street_one - description: The string that user input - tests: - - not_null - - - name: street_two - description: The string that user input - - - name: billing - description: If mark 1 means it is billing address, 0 means is not billing address, every company should have 1 billing address - tests: - - dbt_expectations.expect_column_max_to_be_between: #Every company should have a billing address - min_value: 1 - max_value: 1 - group_by: [company_id] - - dbt_expectations.expect_column_sum_to_be_between: #1 company should have 1 billing address only - min_value: 1 - max_value: 1 - group_by: [company_id] - row_condition: "deleted_at is null" - - - name: deleted_at - description: System generate datetime that address was deleted - - - name: created_at - description: System generate datetime that address was created - tests: - - not_null - - - name: updated_at - description: System generate datetime that address was updated - tests: - - not_null - - - - name: announcement_segment - description: Superadmin / Operator guys will segment the announcment, but normally they will leave it default. 2 FK combine to becoem a relation table. - columns: - - name: announcement_id - description: Primary key for 'announcement_segment' & Foreign key for 'announcements' - tests: - - not_null - - - name: segment_id - description: Primary key for 'announcement_segment' & Foreign key for 'segments' - tests: - - not_null - - - name: created_at - description: Abandon column - tests: - - dbt_expectations.expect_column_values_to_be_null - - - name: updated_at - description: Abandon column - tests: - - dbt_expectations.expect_column_values_to_be_null - - - - name: announcements - description: Announcements that show in our website which create by superadmin - columns: - - name: id - description: Primary key for 'announcements' - tests: - - unique - - not_null - - - name: title - description: title of announcement, using string - tests: - - not_null - - - name: description - description: description of announcement, using longtext, have
html for break - tests: - - not_null - - - name: starting_on - description: announcement starting datetime - tests: - - not_null - - - name: ending_on - description: announcement ending datetime - tests: - - not_null - - - name: is_all_day - description: column might not in use, system auto generate - tests: - - not_null - - accepted_values: - values: [0] - - - name: duration - description: column might not in use, system auto generate - tests: - - not_null - - accepted_values: - values: [0] - - - name: is_recurring - description: column might not in use, system auto generate - tests: - - not_null - - accepted_values: - values: [0] - - - name: recurrence_pattern - description: column might not in use, system auto generate - tests: - - dbt_expectations.expect_column_values_to_be_null - - - name: deleted_at - description: system generate datetime that announcment was deleted - tests: - - not_null - - - name: created_at - description: system generate datetime that announcment was created - tests: - - not_null - - - name: updated_at - description: system generate datetime that announcment was updated - tests: - - not_null - - - - name: bank_logs - description: Abandon table - config: - enabled: false - - - - name: banks - description: Table which store banks details. - columns: - - name: id - description: Primary key for 'banks' - tests: - - unique - - not_null - - - name: company_id - description: Foreign keys for 'companies' - tests: - - not_null - - - name: reference - description: A string which write by user/company for the bank transfer reference or other remind ussage - - - name: holder_name - description: Bank holder name - tests: - - not_null - - - name: account_no - description: Bank account number - tests: - - not_null - - - - - - name: booking_logs - description: Raw `bookings` table from Snowflake - columns: - - name: id - description: Primary key for 'booking_logs' - tests: - - unique - - not_null - - - - name: bookings - description: Raw `bookings` table from Snowflake - columns: - - name: id - description: Primary key for 'bookings' - tests: - - unique - - not_null - - - - name: companies - description: Raw `companies` table from Snowflake - columns: - - name: id - description: Primary key for 'companies' - tests: - - unique - - not_null - - - - name: contacts - description: Raw `contacts` table from Snowflake - columns: - - name: id - description: Primary key for 'contacts' - tests: - - unique - - not_null - - - - name: countries - description: Raw `countries` table from Snowflake - columns: - - name: id - description: Primary key for 'countries' - tests: - - unique - - not_null - - - - name: currencies - description: Raw `currencies` table from Snowflake - columns: - - name: id - description: Primary key for 'currencies' - tests: - - unique - - not_null - - - - name: currency_logs - description: Raw `currency_logs` table from Snowflake - columns: - - name: id - description: Primary key for 'currency_logs' - tests: - - unique - - not_null - - - - name: currency_rate_logs - description: Raw `currency_rate_logs` table from Snowflake - columns: - - name: id - description: Primary key for 'currency_rate_logs' - tests: - - unique - - not_null - - - - name: currency_rates - description: Raw `currency_rates` table from Snowflake - columns: - - name: id - description: Primary key for 'currency_rates' - tests: - - unique - - not_null - - - - name: districts - description: Raw `districts` table from Snowflake - columns: - - name: id - description: Primary key for 'districts' - tests: - - unique - - not_null - - - - name: documents - description: Raw `documents` table from Snowflake - columns: - - name: id - description: Primary key for 'documents' - tests: - - unique - - not_null - - - - name: employees - description: Raw `employees` table from Snowflake - columns: - - name: company_id - description: Primary key for 'employees' - - - - name: failed_jobs - description: Raw `failed_jobs` table from Snowflake - columns: - - name: id - description: Primary key for 'failed_jobs' - tests: - - unique - - not_null - - - - name: files - description: Raw `files` table from Snowflake - columns: - - name: id - description: Primary key for 'files' - tests: - - unique - - not_null - - - - name: group_transactions - description: Raw `group_transactions` table from Snowflake - columns: - - name: id - description: Primary key for 'group_transactions' - tests: - - unique - - not_null - - - - name: groups - description: Raw `groups` table from Snowflake - columns: - - name: id - description: Primary key for 'groups' - tests: - - unique - - not_null - - - - name: jobs - description: Raw `jobs` table from Snowflake - columns: - - name: id - description: Primary key for 'jobs' - tests: - - unique - - not_null - - - - name: migrations - description: Raw `migrations` table from Snowflake - columns: - - name: id - description: Primary key for 'migrations' - tests: - - unique - - not_null - - - - name: model_has_roles - description: Raw `model_has_roles` table from Snowflake - columns: - - name: role_id - description: Primary key for 'model_has_roles' - - - - name: notifications - description: Raw `notifications` table from Snowflake - columns: - - name: id - description: Primary key for 'notifications' - tests: - - unique - - not_null - - - - name: password_resets - description: Raw `password_resets` table from Snowflake - columns: - - name: id - description: Primary key for 'password_resets' - tests: - - unique - - not_null - - - - name: permissions - description: Raw `permissions` table from Snowflake - columns: - - name: id - description: Primary key for 'permissions' - tests: - - unique - - not_null - - - - name: role_has_permissions - description: Raw `role_has_permissions` table from Snowflake - columns: - - name: permission_id - description: Primary key for 'role_has_permissions' - - - - name: roles - description: Raw `roles` table from Snowflake - columns: - - name: id - description: Primary key for 'roles' - tests: - - unique - - not_null - - - - name: segment_companies - description: Raw `segment_companies` table from Snowflake - columns: - - name: segment_id - description: Primary key for 'segment_companies' - - - - name: segment_constants - description: Raw `segment_constants` table from Snowflake - columns: - - name: id - description: Primary key for 'segment_constants' - tests: - - unique - - not_null - - - - name: segments - description: Raw `segments` table from Snowflake - columns: - - name: id - description: Primary key for 'segments' - tests: - - unique - - not_null - - - - name: service_types - description: Raw `service_types` table from Snowflake - columns: - - name: id - description: Primary key for 'service_types' - tests: - - unique - - not_null - - - description: Raw `states` table from Snowflake - columns: - - name: id - description: Primary key for 'states' - tests: - - unique - - not_null - - - - name: transaction_detail - description: Raw `transaction_detail` table from Snowflake - columns: - - name: id - description: Primary key for 'transaction_detail' - tests: - - unique - - not_null - - - - name: transaction_logs - description: Raw `transaction_logs` table from Snowflake - columns: - - name: id - description: Primary key for 'transaction_logs' - tests: - - unique - - not_null - - - - name: transactions - description: Raw `transactions` table from Snowflake - columns: - - name: id - description: Primary key for 'transactions' - tests: - - unique - - not_null - - - - name: user_email_verifications - description: Raw `user_email_verifications` table from Snowflake - columns: - - name: id - description: Primary key for 'user_email_verifications' - tests: - - unique - - not_null - - - - name: users - description: Raw `users` table from Snowflake - columns: - - name: id - description: Primary key for 'users' - tests: - - unique - - not_null - - - - name: wallet_logs - description: Raw `wallet_logs` table from Snowflake - columns: - - name: id - description: Primary key for 'wallet_logs' - tests: - - unique - - not_null - - - - name: wallets - description: Raw `wallets` table from Snowflake - columns: - - name: id - description: Primary key for 'wallets' - tests: - - unique - - not_null \ No newline at end of file diff --git a/models/staging/exchange/stg_exchange__activity_logs.sql b/models/staging/exchange/stg_exchange__activity_logs.sql deleted file mode 100644 index 0040c52..0000000 --- a/models/staging/exchange/stg_exchange__activity_logs.sql +++ /dev/null @@ -1,94 +0,0 @@ --- IMPORTS -WITH activity_logs AS ( - SELECT * FROM {{ source('src_exchange_mysql', 'activity_log') }} -), - - --- LOGIC -activity_logs_rename AS ( - - SELECT - activity_logs.id, - activity_logs.log_name, - activity_logs.description, - activity_logs.subject_id, - activity_logs.subject_type, - activity_logs.causer_id, - activity_logs.causer_type, - activity_logs.properties, - activity_logs.created_at, - activity_logs.updated_at, - current_timestamp() AS _dbt_ran_at - - FROM activity_logs -), - - --- FINAL -final__base_exchange__activity_logs AS ( - - SELECT - id, - log_name, - description, - subject_id, - subject_type, - causer_id, - causer_type, - properties, - created_at, - updated_at, - _dbt_ran_at - - FROM activity_logs_rename -) - -SELECT * FROM final__base_exchange__activity_logs - - --- IMPORTS -WITH activity_logs AS ( - SELECT * FROM {{ ref('base_exchange__activity_logs') }} -), - - --- LOGIC -activity_logs_listout AS ( - - SELECT - activity_logs.id, - activity_logs.log_name, - activity_logs.description, - activity_logs.subject_id, - activity_logs.subject_type, - activity_logs.causer_id, - activity_logs.causer_type, - activity_logs.properties, - activity_logs.created_at, - activity_logs.updated_at, - CURRENT_TIMESTAMP() AS _dbt_ran_at - - FROM activity_logs -), - - --- FINAL -final__stg_exchange__activity_logs AS ( - - SELECT - id, - log_name, - description, - subject_id, - subject_type, - causer_id, - causer_type, - properties, - created_at, - updated_at, - _dbt_ran_at - - FROM activity_logs_listout -) - -SELECT * FROM final__stg_exchange__activity_logs \ No newline at end of file diff --git a/models/staging/shipping/_shipping_sources.yml b/models/staging/shipping/_shipping_sources.yml deleted file mode 100644 index e69de29..0000000 diff --git a/tests/generic_test__zero_or_positive.sql b/tests/generic_test__zero_or_positive.sql deleted file mode 100644 index e69de29..0000000