Undo dbt

README.md

@@ -1,184 +1,15 @@
-# DBT governance
+Welcome to your new dbt project!
-Thid markdown document will specify the rules that need to follow by every developer.
-<br> 
-Short naming converntion lists that need to follow
-  
-| model_type Shortcut | Full name |
-| ------------------- | ------    |
-| seed                | Seed      |
-| src                 | Source    |
-| snap                | Snapshot  |
-| stg                 | Stage     | 
-| int                 | Intermediate |
-| fct                 | Fact      |
-| dim                 | Dimension |
-| rep                 | Report    |
-| sem                 | Semantic  |
 
-| model_name Shortcut | Full name | Use case |
+### Using the starter project
-| ------------------- | ------    | ------   |
+
-| brg                 | Brigde    | relationship table |
+Try running the following commands:
-| log                 | Log       | log data |
+- dbt run
+- dbt test
 
 
-## CIEF General Rule
+### Resources:
-- Snowflake SQL use to write dbt SQL code
+- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
-- Star schema is use
+- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
-- First week of the day is start at Monday, and last week of the day will be Sunday
+- Join the [dbt community](http://community.getbdt.com/) to learn from other analytics engineers
-- Fiscal 1st quarter is start from Febuary 
+- Find [dbt events](https://events.getdbt.com) near you
-- Timezone: Kuala Lumpur / Malaysia (GMT +8)
+- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
-- Some status/type is not input in raw database (You should found in seed)
-
-<br/>
-
-## DBT Rule
-### Structure of the dbt model layer
-- Analyses
-    1. Analyses layer will not create in Snowflake
-    2. You can write some adhoc queries 
-    3. Test some SQL code before write in Models
-
-<br/>
-
-- Macros
-    1. Jinja function that reuse in other models
-
-<br/>
-
-- Models 
-    - Staging
-        - System name
-            1. Materialise: `View`  
-            2. 1-to-1 relationship (or mapping) to source tables.
-            3. Column renaming
-            4. Column remove
-            5. Column data renaming, such as (status integer to string)
-            6. Data input error cleansing
-            7. Check for raw data freshness
-            8. Data type transformation
-            9. Data split and merge
-            - Source
-                1. Source will write in YML
-                2. Check Freshness 
-                3. Check duplicate id
-                4. Check null id
-
-    - Intermediate
-        1. Materialise: `Ephemerally`
-        2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want
-        3. Be referenced repeatedly in more than one model
-        4. Isolating complex operations       
-        
-    - Marts/warehouse
-        1. Materialise: `Table`
-        2. Store fact and dimension models
-
-    - Marts/reporting
-        1. Materialise: `Table`
-        2. Store custom reports
-
-<br/>
-
-- Seeds
-    1. Store custom dataset in csv format
-    2. The dataset must not change frequently
-
-<br/>
-
-- Snapshots
-    1. DBT build-in SCD type 2 function
-    2. Prefer to use after the source table and before the staging layer
-
-<br/>
-
-- Tests
-    1. Can write some yml test case
-
-<br/>
-
-### File Naming Rules
-- File names must be unique
-- Each sql/python file must start with [`model_type`]_[`system_name`]__[`model_name`]s.sql/py
-- The model should be name as plural. (eg: stg_exchange__orders.sql)
-
-<br/>
-
-### Column Naming Rules
-- If array datatype, the column naming must be plural (eg: ids, messages)
-- Schema, table and column names should be in `snake_case`.
-
-- Limit use of abbreviations that are related to domain knowledge. An onboarding
-  employee will understand `current_order_status` better than `current_os`.
-
-- Use names based on the _business_ terminology, rather than the source terminology.
-
-- Each model should have a primary key that can identify the unique row, and should be named `<object>_id`, e.g. `account_id` – this makes it easier to know what `id` is being referenced in downstream joined models.
-
-- If a surrogate key is created, it should be named `<object>_sk`.
-
-- For `base` or `staging` models, columns should be ordered in categories, where identifiers are first and date/time fields are at the end.  
-  Example:
-  ```sql
-  transformed as (
-      select
-          -- ids
-          order_id,
-          customer_id,
-
-          -- dimensions
-          order_status,
-          is_shipped,
-
-          -- measures
-          order_total,
-
-          -- date/times
-          created_at,
-          updated_at,
-
-          -- metadata
-          _sdc_batched_at
-      from source
-  )
-  ```
-
-- Date/time columns should be named according to these conventions:
-  - Timestamps: `<event>_datetime`  
-    Example: `created_datetime`
-  
-  - Dates: `<event>_date`  
-    Example: `created_date`
-
-- Booleans should be prefixed with `is_` or `has_`.  
-  Example: `is_active_customer` and `has_admin_access`
-
-- Price/revenue fields should be in decimal currency (e.g. `19.99` for $19.99; many app databases store prices as integers in cents). If non-decimal currency is used, indicate this with suffix, e.g. `price_in_cents`.
-
-<br/>
-
-### SQL Coding Rule
-- The SQL clause **must** be UPPERCASE (eg. `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `LIMIT`, `WITH`, `AS`, `SUM`, `PARTITION OVER`, `DIV0`, `LEFT JOIN`, `ON` ....)
-- Try to avoid using `EXCLUDE` in the SQL
-- Avoid using multi layer subquery, and try to use CTE, subquery must not more than 1 layer
-- Each model structure must contain with 
-
-```sql
---IMPORT
-WITH [table_names] AS (
-    SELECT * FROM {{ ref('file_names')}}
-),
-
---LOGIC
-[logic_names] AS (
-    .....
-),
-
---FINAL
-final__[table_names] AS (
-
-)
-SELECT * FROM final__[final_table_names]
-```
-
-### Testing
-- At a minimum, `unique` and `not_null` tests should be applied to the expected primary key of each model.

analyses/exchange/company_conversion_v2.sql

@@ -1,5 +1,5 @@
 -- MARCRO for count the cummulative_activity_number
-{% set funnel_activity = ['register','email_verified','identity_document_uploaded','identity_document_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%}
+{% set funnel_activity = ['register','email_verified','identity_documents_uploaded','identity_documents_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%}
 
 -- IMPORT
 WITH companies AS (
@@ -69,13 +69,12 @@ company_first_time_order_date AS (
 
 
 
---start defind funnel activity
+--start defind funnel
 repeat_completed_order_after_60days_activities AS (
     SELECT 
         clean_company_transaction_orders.company_id,
         0 AS is_data_cleansing_generate_row,
         'repeat_completed_order_after_60days' AS activity,
-        9 AS activtity_rank,
         MIN(clean_company_transaction_orders.created_at) AS activity_datetime
 
     FROM clean_company_transaction_orders
@@ -95,7 +94,6 @@ repeat_completed_order_after_30days_activities AS (
         clean_company_transaction_orders.company_id,
         0 AS is_data_cleansing_generate_row,
         'repeat_completed_order_after_30days' AS activity,
-        8 AS activtity_rank,
         MIN(clean_company_transaction_orders.created_at) AS activity_datetime
 
     FROM clean_company_transaction_orders
@@ -115,7 +113,6 @@ first_time_order_completed_activities AS (
         clean_company_transaction_orders.company_id,
         0 AS is_data_cleansing_generate_row,
         'first_time_order_completed' AS activity,
-        7 AS activtity_rank,
         MIN(clean_company_transaction_orders.created_at) AS activity_datetime
 
     FROM clean_company_transaction_orders
@@ -132,7 +129,6 @@ first_time_order_activities AS (
         clean_company_transaction_orders.company_id,
         0 AS is_data_cleansing_generate_row,
         'first_time_order' AS activity,
-        6 AS activtity_rank,
         MIN(clean_company_transaction_orders.created_at) AS activity_datetime
 
     FROM clean_company_transaction_orders
@@ -149,7 +145,6 @@ first_time_booking_activities AS (
         IFNULL( unclean_first_time_booking_activities.company_id , first_time_order_activities.company_id ) AS company_id,
         IFF( unclean_first_time_booking_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
         'first_time_booking' AS activity,
-        5 AS activtity_rank,
         IFNULL( unclean_first_time_booking_activities.activity_datetime , first_time_order_activities.activity_datetime ) AS activity_datetime
 
     FROM
@@ -173,20 +168,19 @@ first_time_booking_activities AS (
 ),
 
 
-identity_document_verified_activities AS (
+identity_documents_verified_activities AS (
     SELECT
-        IFNULL( unclean_identity_document_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id,
+        IFNULL( unclean_identity_documents_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id,
-        IFF( unclean_identity_document_verified_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
+        IFF( unclean_identity_documents_verified_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
-        'identity_document_verified' AS activity,
+        'identity_documents_verified' AS activity,
-        4 AS activtity_rank,
+        IFNULL( unclean_identity_documents_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime
-        IFNULL( unclean_identity_document_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime
     
     FROM 
         (
 
         SELECT
             company_id,
-            'identity_document_verified' AS activity,
+            'identity_documents_verified' AS activity,
             MIN(identity_created_datetime) AS activity_datetime
         
         FROM company_identity_documents
@@ -197,28 +191,27 @@ identity_document_verified_activities AS (
         GROUP BY 
             company_id
 
-        ) AS unclean_identity_document_verified_activities
+        ) AS unclean_identity_documents_verified_activities
     
     
     FULL OUTER JOIN first_time_booking_activities 
-        ON (unclean_identity_document_verified_activities.company_id = first_time_booking_activities.company_id)
+        ON (unclean_identity_documents_verified_activities.company_id = first_time_booking_activities.company_id)
 
 ),
 
-identity_document_uploaded_activities AS (
+identity_documents_uploaded_activities AS (
     SELECT
-        IFNULL( unclean_identity_document_uploaded_activities.company_id , identity_document_verified_activities.company_id ) AS company_id,
+        IFNULL( unclean_identity_documents_uploaded_activities.company_id , identity_documents_verified_activities.company_id ) AS company_id,
-        IFF( unclean_identity_document_uploaded_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
+        IFF( unclean_identity_documents_uploaded_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
-        'identity_document_uploaded' AS activity,
+        'identity_documents_uploaded' AS activity,
-        3 AS activtity_rank,
+        IFNULL( unclean_identity_documents_uploaded_activities.activity_datetime , identity_documents_verified_activities.activity_datetime ) AS activity_datetime
-        IFNULL( unclean_identity_document_uploaded_activities.activity_datetime , identity_document_verified_activities.activity_datetime ) AS activity_datetime
     
     FROM 
         (
 
         SELECT
             company_id,
-            'identity_document_uploaded' AS activity,
+            'identity_documents_uploaded' AS activity,
             MIN(identity_created_datetime) AS activity_datetime
         
         FROM company_identity_documents
@@ -226,20 +219,19 @@ identity_document_uploaded_activities AS (
         GROUP BY  
             company_id
 
-        ) AS unclean_identity_document_uploaded_activities
+        ) AS unclean_identity_documents_uploaded_activities
     
     
-    FULL OUTER JOIN identity_document_verified_activities 
+    FULL OUTER JOIN identity_documents_verified_activities 
-        ON (unclean_identity_document_uploaded_activities.company_id = identity_document_verified_activities.company_id)
+        ON (unclean_identity_documents_uploaded_activities.company_id = identity_documents_verified_activities.company_id)
 ),
 
 email_verified_activities AS (
     SELECT
-        IFNULL( unclean_email_verified_activities.company_id , identity_document_uploaded_activities.company_id ) AS company_id,
+        IFNULL( unclean_email_verified_activities.company_id , identity_documents_uploaded_activities.company_id ) AS company_id,
         IFF( unclean_email_verified_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
         'email_verified' AS activity,
-        2 AS activtity_rank,
+        IFNULL( unclean_email_verified_activities.activity_datetime , identity_documents_uploaded_activities.activity_datetime ) AS activity_datetime
-        IFNULL( unclean_email_verified_activities.activity_datetime , identity_document_uploaded_activities.activity_datetime ) AS activity_datetime
     
     FROM 
         (
@@ -257,8 +249,8 @@ email_verified_activities AS (
             company_id
         ) AS unclean_email_verified_activities
 
-    FULL OUTER JOIN identity_document_uploaded_activities
+    FULL OUTER JOIN identity_documents_uploaded_activities
-        ON (unclean_email_verified_activities.company_id = identity_document_uploaded_activities.company_id)
+        ON (unclean_email_verified_activities.company_id = identity_documents_uploaded_activities.company_id)
 
 ),
 
@@ -267,7 +259,6 @@ register_activities AS (
         IFNULL( unclean_register_activities.company_id , email_verified_activities.company_id ) AS company_id,
         IFF( unclean_register_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
         'register' AS activity,
-        1 AS activtity_rank,
         IFNULL( unclean_register_activities.activity_datetime , email_verified_activities.activity_datetime ) AS activity_datetime
 
     FROM 
@@ -319,30 +310,24 @@ union_all_activities_flag_migrated_company AS (
 
 --FINAL
 
-sem_exchange__company_conversion_funnels AS (
+sem_exchange__company_funnels AS (
     SELECT
         company_id,
         is_data_cleansing_generate_row,
         is_migrated_company,
-        activity,
-        activtity_rank,
-        IFF( ROW_NUMBER() OVER (PARTITION BY company_id ORDER BY activtity_rank DESC) = 1,
-            1,
-            0)  AS latest_activity,
-            
         activity_datetime,
 
-    {% for activity_type in funnel_activity %}
+    {% for activity in funnel_activity %}
 
-        count(iff(activity = '{{activity_type}}', company_id, NULL)) 
+        count(iff(activity = '{{activity}}', company_id, NULL)) 
-            OVER (ORDER BY activity_datetime) AS cumulative_{{activity_type}}_activity,
+            OVER (ORDER BY activity_datetime) AS cumulative_{{activity}}_activity,
 
     {% endfor %}
 
-    {% for activity_type in funnel_activity %}
+    {% for activity in funnel_activity %}
 
-        count(iff(activity = '{{activity_type}}' AND is_migrated_company = 0, company_id, NULL)) 
+        count(iff(activity = '{{activity}}' AND is_migrated_company = 0, company_id, NULL)) 
-            OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity_type}}_activity,
+            OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity}}_activity,
 
     {% endfor %}
 
@@ -353,4 +338,4 @@ sem_exchange__company_conversion_funnels AS (
   ORDER BY activity_datetime
 )
 
-SELECT * FROM sem_exchange__company_conversion_funnels
+SELECT * FROM sem_exchange__company_funnels

models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_last_transaction_sequence.sql

@@ -1,8 +0,0 @@
-with
-    wallet_log_sequence as (
-        select * from {{ ref("int_exchange__wallet_transaction_sequence") }}
-    )
-
-select max(operation_sequence) as last_transaction_id, wallet_id
-from wallet_log_sequence
-group by wallet_id

models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_log_credit_amount.sql

@@ -1,56 +0,0 @@
--- IMPORTS
-with
-    wallet_logs as (select * from {{ ref("stg_exchange__wallet_logs") }}),
-    -- LOGIT
-    wallet_credit_amount as (
-        select
-            *,
-            lag(amount) over (
-                partition by wallet_id order by updated_at
-            ) as previous_wallet_balance,
-            amount - lag(amount) over (
-                partition by wallet_id order by updated_at
-            ) as credited_amount
-        from wallet_logs
-    ),
-    wallet_credit_amount_reformed as (
-        select
-            id,
-            wallet_id,
-            previous_wallet_balance,
-            amount as current_wallet_balance,
-            iff(
-                equal_null(previous_wallet_balance, null), amount, credited_amount
-            ) as credited_amount,
-            owner_type,
-            owner_id,
-            code,
-            currency_id,
-            deleted_at,
-            created_at,
-            updated_at,
-            _dbt_ran_at
-        from wallet_credit_amount
-    ),
-
-    final_wallet_credit_amount as (
-        select
-            id,
-            wallet_id,
-            previous_wallet_balance,
-            current_wallet_balance,
-            credited_amount,
-            iff(credited_amount > 0, 'CREDIT', 'DEBIT') as credit_status,
-            owner_type,
-            owner_id,
-            code,
-            currency_id,
-            deleted_at,
-            created_at,
-            updated_at,
-            _dbt_ran_at
-        from wallet_credit_amount_reformed
-    )
-
-select *
-from final_wallet_credit_amount

models/exchange/intermediate/wallet_log_creadit_amount/int_exchange__wallet_transaction_sequence.sql

@@ -1,13 +0,0 @@
-with
-    wallet_credit_log as (
-        select * from {{ ref("int_exchange__wallet_log_credit_amount") }}
-    ),
-    wallet_credit_with_sequence as (
-        select
-            dense_rank() over (
-                partition by wallet_id order by updated_at
-            ) as operation_sequence,
-            *
-        from wallet_credit_log
-    )
-select * from wallet_credit_with_sequence

models/exchange/intermediate/wallet_log_creadit_amount/int_exchange_wallet_final_log_sequence.sql

@@ -1,24 +0,0 @@
-with
-    wallet_log_sequence as (
-        select * from {{ ref("int_exchange__wallet_transaction_sequence") }}
-    ),
-    wallet_last_log as (
-        select * from {{ ref("int_exchange__wallet_last_transaction_sequence") }}
-    )
-
-select
-    s.id,
-    s.wallet_id,
-    s.current_wallet_balance,
-    s.credit_status,
-    s.owner_type,
-    s.owner_id,
-    s.code,
-    s.currency_id,
-    s.deleted_at,
-    s.created_at,
-    s.updated_at,
-    s._dbt_ran_at
-from wallet_log_sequence s
-inner join wallet_last_log l
-where s.wallet_id = l.wallet_id and s.operation_sequence = l.last_transaction_id

models/exchange/marts/reporting/rep_exchange__daily_orders/test_exchange__daily_orders.yml

@@ -2,51 +2,27 @@ version: 2
 
 models:
   - name: rep_exchange__daily_orders
-
-    # @YAM: This is the test code that I write
-    # @YAM: I already do some space in the YML, but not all the stuff is clean, please clean the code, and makesure it is easy readable.
-    # columns: 
-    #   - name: base_value
-    #     description: test base value combine with dbt_utils
-    #     tests:
-    #       - not_null
-    #       - dbt_utils.expression_is_true:
-    #           expression: " >= 0"
-    tests:
-          - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
-              column_A: order_completed_datetime
-              column_B: order_created_datetime
-              or_equal: True
-    
-          - dbt_utils.expression_is_true:
-              expression: "COST_TRANSACTION_VALUE_RM + COST_TRANSACTION_TAX_RM + COST_TRANSACTION_SERVICE_CHARGE_RM  < 2*(TRANSACTION_VALUE_RM + TRANSACTION_BASE_TAX + TRANSACTION_TAX_RM)" 
-              severity: warn 
     columns: 
-      - name: ID
+      - name: id
         description: transaction orders id
         tests:
           - unique
           - not_null
 
-      - name: BOOKING_ID
+      - name: booking_id
         description: transaction orders id
         tests:
           - not_null
           - relationships:
               to: ref('base_exchange__bookings')
               field: id
-
-      - name: company_marking_id
-        description: transaction orders id
-        tests:
-          - not_null
              
-      - name: SERVICE_TYPE
+      - name: service_type
         description: type of service
         tests:
           - not_null
 
-      - name: SERVICE_TYPE_NAME
+      - name: service_type_name
         description: name of type of service
         tests:
           - not_null
@@ -59,55 +35,55 @@ models:
       - name: seller_company_id
         description:  seller company id
         tests:
-          - not_null
+          - not_null  
-      
+
       - name: bank_id
         description:  bank id
         tests:
-          - not_null
+          - not_null  
-      
+          
       - name: company_id
         description:  user  id
         tests:
-          - not_null
+          - not_null  
-      
+
-      - name: marking_id
+      - name: company_marking_id
-        description: marking id
+        description: company marking id
         tests:
           - relationships:
-              to: ref('base_exchange_companies')
+              to: ref('base_exchange__companies')
-              field: id
+              field: marking_id
 
       - name: company_type
         description:  type of company
         tests:
-          - not_null
+          - not_null 
-      
+
-      - name: COMPANY_TYPE_NAME
+      - name: company_type_name
         description:  name type of company
         tests: 
           - accepted_values:
               values:
                 - PERSONAL
-                - CORPORATE
+                - CORPORATE 
-       
+
       - name: company_business_type
         description:   company business type
         tests:
-          - not_null
+          - not_null 
-      
+
       - name: company_business_type_name
         description:   company business type name
         tests: 
           - accepted_values:
               values:
                 - IMPORTER
-      
+
       - name: company_segment_id
         description:  company segment id
         tests: 
-          - not_null
+          - not_null 
-      
+
       - name: company_segment_name
         description:  company segment name
         tests: 
@@ -116,109 +92,120 @@ models:
                 - PLATINIUM MEMBER
                 - GOLD MEMBER
                 - STANDARD SEGMENT
-      
+
       - name: cost_id
         description: cost id 
         tests:
           - relationships:
               to: ref('base_exchange__transactions')
               field: id
-      
+
       - name: base_value
         description: test base value combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
       - name: quote_value
         description: test quote value combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
       - name: transaction_value_rm
         description: transaction  value rm combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
       - name: transaction_base_tax
         description: test transaction base tax combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
       - name: base_service_charge
         description: test base service charge combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
       - name: transaction_tax_rm
         description: test transaction tax rm combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
       - name: cost_base_value
         description: test cost base value combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
-      - name: COST_BASE_TO_QUOTE_CURRENCY_EXCHANGE_RATE
+      - name: cost_base_to_quote_currency_exchange_rate
-        description: test COST BASE TO QUOTE CURRENCY EXCHANGE RATE combine with dbt_utils
+        description: test cost base to quote currency exchange rate combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
-      - name: COST_QUOTE_VALUE
+      - name: cost_quote_value
-        description: test COST QUOTE VALUE combine with dbt_utils
+        description: test cost quote value combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+              
-      - name: COST_TRANSACTION_VALUE_RM
+      - name: cost_transaction_value_rm
         description: test COST TRANSACTION VALUE RM combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
-      - name: COST_TRANSACTION_TAX_RM
+      - name: cost_transaction_tax_rm
         description: test COST TRANSACTION TAX RM combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
-      - name: COST_BASE_SERVICE_CHARGE
+      - name: cost_base_service_charge
         description: test COST BASE SERVICE CHARGE combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
-      - name: COST_TRANSACTION_SERVICE_CHARGE_RM
+      - name: cost_transaction_service_charge_rm
         description: test COST TRANSACTION SERVICE CHARGE_RM combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
-      
+
       - name: transaction_service_charge_rm
         description: test transaction service charge rm combine with dbt_utils
         tests:
           - not_null
           - dbt_utils.expression_is_true:
               expression: " >= 0"
+
+    tests:
+      - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
+          column_A: order_completed_datetime
+          column_B: order_created_datetime
+          or_equal: True
+    tests:
+      - dbt_utils.expression_is_true:
+          expression: "cost_transaction_value_rm + cost_transaction_tax_rm + cost_transaction_service_charge_rm  < 2*(transaction_value_rm + transaction_base_tax + transaction_tax_rm)" 
+          severity: warn 
+          
 # FRESHNESS SOURCE 

models/exchange/marts/warehouse/fct_exchange__wallet_debit_time_duration.sql

@@ -1,45 +0,0 @@
-with
-    wallet_credit_log as (
-        select * from {{ ref("int_exchange__wallet_log_credit_amount") }}
-    ),
-    wallet_debit_time_log as (
-        select
-            dense_rank() over (
-                partition by wallet_id order by updated_at
-            ) as operation_sequence,
-            lag(updated_at) over (
-                partition by wallet_id order by updated_at
-            ) previous_update_time,
-            datediff(
-                second, previous_update_time, updated_at
-            ) as time_between_operation_in_seconds,
-            *
-        from wallet_credit_log
-    ),
-
-    updated_wallet_debit_time_log as (
-
-        select
-            *,
-            iff(
-                (credit_status = 'DEBIT'), time_between_operation_in_seconds, 0
-            ) as seconds_required_to_debit
-        from wallet_debit_time_log
-    ),
-    final_wallet_debit_time_log as (
-        select
-            * exclude(seconds_required_to_debit),
-            iff(
-                equal_null(seconds_required_to_debit, null),
-                0,
-                seconds_required_to_debit
-            ) as seconds_required_to_debit
-        from updated_wallet_debit_time_log
-    ),
-    final_wallet_debit_time_log_reformed as (
-        select * exclude(TIME_BETWEEN_OPERATION_IN_SECONDS)
-        from final_wallet_debit_time_log
-        where credit_status = 'DEBIT' and seconds_required_to_debit != 0
-    )
-select *
-from final_wallet_debit_time_log_reformed

models/exchange/marts/warehouse/fct_exchange__wallet_log_top_10_top_up.sql

@@ -1,17 +0,0 @@
--- IMPORTS
-with
-    wallet_credit_log as (
-        select * from {{ ref("int_exchange__wallet_log_credit_amount") }}
-    ),
-
-    wallet_credit_top_up_frequency_report as (
-        select credited_amount, count(credited_amount) repetation
-        from wallet_credit_log
-        where credit_status = 'CREDIT'
-        group by credited_amount
-        order by repetation desc
-        limit 10
-    )
-
-select *
-from wallet_credit_top_up_frequency_report

models/exchange/marts/warehouse/fct_exchange__wallet_log_wallet_amount_comparison.sql

@@ -1,24 +0,0 @@
-with wallet_log as (
-    select * from {{ref('int_exchange_wallet_final_log_sequence')}}
-),
-wallet_details as (
-    select * from {{ref('stg_exchange__wallet')}}
-),
-
-wallet_comparison_log as (
-select
-    w.id as wallet_id,
-    w.amount as wallet_amount,
-    s.current_wallet_balance as wallet_balance_from_log,
-    w.owner_type,
-    s.credit_status,
-    s.updated_at as wallet_log_update_time,
-    s._dbt_ran_at,
-    w.updated_at as wallet_update_time
-from wallet_details w
-left join
-    wallet_log s
-    on w.id = s.wallet_id
-)
-
-select * from wallet_comparison_log

models/exchange/marts/warehouse/test_exchange__bookings.yml

@@ -1,156 +0,0 @@
-version: 2
-models:
-  - name: fct_exchange__bookings
-
-    tests:
-          - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
-              column_A: BOOKING_UPDATED_DATETIME
-              column_B: BOOKING_CREATED_DATETIME
-              or_equal: True
-    
-          - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
-              column_A: BOOKING_DELETED_DATETIME
-              column_B: BOOKING_CREATED_DATETIME
-              or_equal: True
-
-    columns:  
-      - name: ID
-        description: boking  id
-        tests:
-          - unique  
-          - not_null
-      
-      - name: COMPANY_ID
-        description: company  id
-        tests:
-          - unique  
-          - not_null
-      
-      - name: USER_ID
-        description: user  id
-        tests:
-          - unique  
-          - not_null
-      
-      - name: MARKING_ID
-        description: marking  id
-        tests:
-          - unique  
-          - not_null
-
-      - name: SERVICE_TYPE
-        description:  type of service
-        tests:
-          - not_null
-          - accepted_values:
-              values:
-                - 1
-                - 3
-                - 4
-
-      - name: SERVICE_TYPE_NAME
-        description: name of type of service
-        tests:
-          - not_null
-          - accepted_values:
-              values:
-                - 3 DAYS TRANSFER
-                - 1 DAY TRANSFER
-                - 1688 PAYMENT
-
-      - name: BANK_ID
-        description: bank  id
-        tests:
-          - unique  
-          - not_null
-
-      - name: FIX_VALUE
-        description: test fix value combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"                
-
-      - name: FIX_CURRENCY_ID
-        description:  fix currency id
-        tests:
-          - not_null
-          - accepted_values:
-              values:
-                - 1
-                - 3
-                - 2
-
-      - name: FIX_CURRENCY_NAME
-        description: fix currency name
-        tests:
-          - not_null
-          - accepted_values:
-              values:
-                - Malaysian Ringgit
-                - US Dollar
-                - Yuan Renminbi
-      
-      - name: ESTIMATE_BASE_TO_QUOTE_CURRENCY_EXCHANGE_RATE
-        description: test estimate base to quote currency exchange rate value combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"   
-
-      - name: QUOTE_CURRENCY_ID
-        description:  quote currency id
-        tests:
-          - not_null
-          - accepted_values:
-              values:
-                - 3
-                - 2
-
-      - name: QUOTE_CURRENCY_NAME
-        description: quote  currency name
-        tests:
-          - not_null
-          - accepted_values:
-              values:
-                - US Dollar
-                - Yuan Renminbi    
-
-      - name: ESTIMATE_QUOTE_VALUE
-        description: test estimate  quote value  value combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"   
-
-      - name: ESTIMATE_BASE_VALUE
-        description: test estimate  quote value  value combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"   
-
-      - name: ESTIMATE_VALUE_RM
-        description: test estimate  quote value  value combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"  
-
-      - name: STATUS
-        description:  status
-        tests:
-          - accepted_values:
-              values:
-                - 2
-                - 3
-                - 5
-
-      - name: STATUS_NAME
-        description:  status name
-        tests:
-          - accepted_values:
-              values:
-                - SUSPENDED
-                - APPROVED
-                - COMPLETED

models/exchange/marts/warehouse/test_exchange__companies.yml

@@ -1,146 +0,0 @@
-version: 2
-models:
-  - name: dim_exchange__companies
-    tests:
-      - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
-              column_A: ADDRESS_DELETED_AT
-              column_B: ADDRESS_CREATED_AT
-              or_equal: True
-
-      - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
-              column_A: ADDRESS_UPDATED_AT
-              column_B: ADDRESS_CREATED_AT
-              or_equal: True
-
-    columns:  
-      #Gerson todo Why this company id need to ref with addresses?
-      - name: ID
-        description: company id
-        tests:
-          - unique  
-          - not_null
-          - relationships:
-              to: ref('base_exchange__addresses')
-              field: company_id
-
-      - name: NAME
-        description: company name
-        tests:
-          - not_null
-
-      #Yam todo there is marking_id in supplier, need mask them
-      - name: MARKING_ID
-        description: marking id
-        tests:
-          - unique
-          - not_null
-
-      - name: COMPANY_TYPE
-        description:  type of company
-        tests:
-          - accepted_values:
-              values:
-                - 0
-                - 1
-                - 2
-              config:
-                where: "COMPANY_TYPE is not null"
-
-      - name: COMPANY_TYPE_NAME
-        description:  name of type company
-        tests:
-          - accepted_values:
-              values:
-                - CORPORATE
-                - PERSONAL
-              config:
-                where: "COMPANY_TYPE_NAME is not null"
-
-      - name: BUSINESS_TYPE
-        description:  type of business
-        tests:
-          - accepted_values:
-              values:
-                - 1
-                - 2
-                - 3
-              config:
-                where: "BUSINESS_TYPE is not null"
-
-      - name: BUSINESS_TYPE_NAME
-        description:  name of type business
-        tests:
-          - accepted_values:
-              values:
-                - IMPORTER
-                - CURRENCY_VENDOR
-                - FREIGHT_FORWARDER
-              config:
-                where: "BUSINESS_TYPE_NAME is not null"
-
-      - name: SEGMENT_ID
-        description:  segment id
-        tests:
-          - accepted_values:
-              values:
-                - 1
-                - 2
-                - 3
-              config:
-                where: "SEGMENT_ID is not null"
-
-      - name: SEGMENT_NAME
-        description:  segment name
-        tests:
-          - accepted_values:
-              values:
-                - STANDARD SEGMENT
-                - GOLD MEMBER
-                - PLATINIUM MEMBER
-              config:
-                where: "SEGMENT_NAME is not null"
-
-      - name: IS_MIGRATED_COMPANY
-        description:  is migrate company
-        tests:
-          - accepted_values:
-              values:
-                - 1
-                - 0
-
-      - name: STATUS
-        description:  status
-        tests:
-          - accepted_values:
-              values:
-                - 0
-                - 1
-                - 2
-                - 4
-                - 5
-
-      - name: STATUS_NAME
-        description:  status name
-        tests:
-          - accepted_values:
-              values:
-                - SUSPENDED
-                - APPROVED
-                - PENDING_SUBMISSION
-                - REJECTED
-                - PENDING_VERIFICATION
-
-      - name: COMPANY_CONTACT_EMAIL
-        description: company contact email
-        tests:
-          - not_null
-          - dbt_expectations.expect_column_values_to_match_regex:
-               regex: "[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*@[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*[.][a-zA-Z]{2,5}"
-               severity: warn 
-               
-      - name: marking_id
-        description: marking id
-        tests:
-          - relationships:
-              to: ref('base_exchange_companies')
-              field: id

models/exchange/marts/warehouse/test_exchange__transaction_orders.yml

@@ -1,163 +0,0 @@
-version: 2
-models:
-  - name: fct_exchange__transaction_orders
-
-    tests:
-          - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
-              column_A: UPDATED_AT
-              column_B: CREATED_AT
-              or_equal: True
-    
-          - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
-              column_A: DELETED_AT
-              column_B: CREATED_AT
-              or_equal: True
-
-    columns:  
-      - name: ID
-        description: transaction  id
-        tests:
-          - unique  
-          - not_null
-          - relationships:
-              to: ref('int_exchange__transaction_orders_get_user_ids')
-              field: id
-      - name: BOOKING_ID
-        description: booking  id
-        tests:
-          - unique  
-          - not_null
-          - relationships:
-              to: ref('int_exchange__transaction_orders_get_user_ids')
-              field: BOOKING_ID
-      - name: SELLER_COMPANY_ID
-        description: seller company  id
-        tests:
-          - unique  
-          - not_null
-          - relationships:
-              to: ref('int_exchange__transaction_orders_get_user_ids')
-              field: SELLER_COMPANY_ID
-      - name: BANK_ID
-        description: bank  id
-        tests:
-          - unique  
-          - not_null
-          - relationships:
-              to: ref('int_exchange__transaction_orders_get_user_ids')
-              field: BANK_ID
-      - name: COMPANY_ID
-        description: company  id
-        tests:
-          - unique  
-          - not_null
-          - relationships:
-              to: ref('int_exchange__transaction_orders_get_user_ids')
-              field: COMPANY_ID
-              
-      - name: STATUS
-        description:  status
-        tests:
-          - accepted_values:
-              values:
-                - 0
-                - 1
-                - 2
-                - 3
-                - 4
-                - 5
-
-      - name: STATUS_NAME
-        description:  status name
-        tests:
-          - accepted_values:
-              values:
-                - COMPLETED
-                - SUSPENDED
-                - REJECTED
-                - PENDING_SUBMISSION
-                - PENDING_VERIFICATION
-                - APPROVED
-
-      - name: TRANSACTION_TYPE
-        description:  transaction type
-        tests:
-          - accepted_values:
-              values:
-                - 1
-
-      - name: TRANSACTION_TYPE_NAME
-        description:   name transaction type
-        tests:
-          - accepted_values:
-              values:
-                - PAYMENT
-
-      - name: PAYMENT_METHOD
-        description:  method payment
-        tests:
-          - accepted_values:
-              values:
-                - 1
-                - 2
-                - 5
-                - 4
-
-      - name: PAYMENT_METHOD_NAME
-        description:   name method payment
-        tests:
-          - accepted_values:
-              values:
-                - CASH
-                - CHEQUE
-                - PAYMENT_GATEWAY
-                - WALLET
-             
-      - name: base_value
-        description: test base value combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"  
-      
-      - name: quote_value
-        description: test quote value combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"
-      
-      - name: transaction_value_rm
-        description: transaction  value rm combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"
-      
-      - name: transaction_base_tax
-        description: test transaction base tax combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"
-      
-      - name: transaction_tax_rm
-        description: test transaction tax rm combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"
-
-      - name: base_service_charge
-        description: test base service charge combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"
-              
-      - name: TRANSACTION_SERVICE_CHARGE_RM
-        description: test transaction service change rm value combine with dbt_utils
-        tests:
-          - not_null
-          - dbt_utils.expression_is_true:
-              expression: " >= 0"

models/exchange/marts/warehouse/test_exchange__users.yml

@@ -1,76 +0,0 @@
-version: 2
-models:
-  - name: dim_exchange__users
-    tests:
-      - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
-              column_A: DELETED_AT
-              column_B: CREATED_AT
-              or_equal: True
-              
-    
-      - dbt_expectations.expect_column_pair_values_A_to_be_greater_than_B:
-              column_A: EMAIL_VERIFICATION_LAST_EMAIL_CREATED_AT
-              column_B: EMAIL_VERIFICATION_FIRST_EMAIL_CREATED_AT
-              or_equal: True
-    
-    columns: 
-      - name: ID
-        description: user id
-        tests:
-          - unique
-          - not_null
-
-      - name: NAME
-        description: user name
-        tests:
-          - not_null
-
-      - name: USER_TYPE
-        description:  type of user
-        tests:
-          - not_null
-          - accepted_values:            
-              values:
-                - 1
-                - 2
-
-      - name: USER_TYPE_NAME
-        description:  type of user
-        tests:
-          - not_null
-          - accepted_values:
-              values:
-                - USER
-                - ADMIN
-
-      - name: STATUS
-        description:  user status
-        tests:
-          - not_null
-          - accepted_values:
-              values:
-                - 1
-                - 2
-
-      - name: STATUS_NAME
-        description:  user status name
-        tests:
-          - not_null
-          - accepted_values:
-              values:
-                - APPROVED
-                - PENDING_VERIFICATION
-
-      - name: COMPANY_ID
-        description: company id
-        tests:
-          - unique
-          - not_null
-
-      - name: EMAIL
-        description: user email
-        tests:
-          - not_null
-          - dbt_expectations.expect_column_values_to_match_regex:
-               regex: "[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*@[a-zA-Z0-9_]+([.][a-zA-Z0-9_]+)*[.][a-zA-Z]{2,5}"
-               severity: warn 

models/exchange/staging/activity_logs/stg_exchange__activity_logs.sql

@@ -0,0 +1,46 @@
+-- IMPORTS
+WITH activity_logs AS (
+    SELECT * FROM {{ ref('base_exchange__activity_logs') }}
+),
+
+
+-- LOGIC
+activity_logs_listout AS (
+
+    SELECT
+        activity_logs.id,
+        activity_logs.log_name,
+        activity_logs.description,
+        activity_logs.subject_id,
+        activity_logs.subject_type,
+        activity_logs.causer_id,
+        activity_logs.causer_type,
+        activity_logs.properties,
+        activity_logs.created_at,
+        activity_logs.updated_at,
+        CURRENT_TIMESTAMP()                                                                         AS _dbt_ran_at
+
+    FROM activity_logs
+),
+
+
+-- FINAL
+final__stg_exchange__activity_logs AS (
+
+    SELECT
+        id,
+        log_name,
+        description,
+        subject_id,
+        subject_type,
+        causer_id,
+        causer_type,
+        properties,
+        created_at,
+        updated_at,
+        _dbt_ran_at
+
+    FROM activity_logs_listout
+) 
+
+SELECT * FROM final__stg_exchange__activity_logs

models/exchange/staging/wallet/stg_exchange__wallet.sql

@@ -1,10 +0,0 @@
--- IMPORTS
-WITH wallets AS (
-    SELECT * FROM {{ ref('base_exchange__wallets') }}
-)
-
-
-
-
--- FINAL 
-SELECT * FROM wallets

models/exchange/staging/wallet_logs/stg_exchange__wallet_logs.sql

@@ -1,7 +0,0 @@
--- IMPORTS
-with wallet_logs as (select * from {{ ref("base_exchange__wallet_logs") }})
-
--- FINAL 
-select *
-from wallet_logs
-order by wallet_id, updated_at asc

models/shipping/staging/stg_shipping__activity_logs.sql

@@ -1,46 +0,0 @@
--- IMPORTS
-WITH activity_logs AS (
-    SELECT * FROM {{ source('src_shipping_mysql', 'activity_log') }}
-),
-
-
--- LOGIC
-activity_logs_rename AS (
-
-    SELECT
-        activity_logs.id,
-        activity_logs.log_name,
-        activity_logs.description,
-        activity_logs.subject_id,
-        activity_logs.subject_type,
-        activity_logs.causer_id,
-        activity_logs.causer_type,
-        activity_logs.properties,
-        activity_logs.created_at                                                                        AS created_datetime,
-        activity_logs.updated_at                                                                        AS updated_datetime,
-        current_timestamp()                                                                             AS _dbt_ran_at
-
-    FROM activity_logs
-),
-
-
--- FINAL
-final__base_exchange__activity_logs AS (
-
-    SELECT
-        id,
-        log_name,
-        description,
-        subject_id,
-        subject_type,
-        causer_id,
-        causer_type,
-        properties,
-        created_datetime,
-        updated_datetime,
-        _dbt_ran_at
-
-    FROM activity_logs_rename
-) 
-
-SELECT * FROM final__base_exchange__activity_logs

models/staging/exchange/_exchange__docs.md

@@ -1,23 +0,0 @@
-{% docs src_exchange_mysql %}
-
-Data is extract from CIEF Selfhost MYSQL `exchange_production`, using custom python code.
-
-- `_source_loaded_at` will add into the raw data when extract
-```YML
-loaded_at_field: _source_loaded_at
-freshness:
-    warn_after: {count: 12, period: hour}
-    error_after: {count: 24, period: hour}
-```
-
-- The raw data will use copy into snowflake in 2AM everyday
-
-- Depend on target.name to determin which snowflake raw schema
-
-```JINJA
-{%- if target.name == "prod" -%} PROD_CIEF_RAW_DB
-{%- else -%} DEV_CIEF_RAW_DB
-{%- endif -%}
-```
-
-{% enddocs %}

models/staging/exchange/_exchange__models.yml

@@ -1,6 +0,0 @@
-version: 2
-
-models:
-  - name: stg_exchange__activity_logs
-
-  - name: stg_exchange__announcments  

models/staging/exchange/_exchange__sources.yml

@@ -1,619 +0,0 @@
-version: 2
-
-sources:
-  - name: src_exchange_mysql
-    description: '{{ doc("src_exchange_mysql") }}'
-    database: |
-              {%- if target.name == "prod" -%} PROD_CIEF_RAW_DB
-              {%- else -%} DEV_CIEF_RAW_DB
-              {%- endif -%}
-    schema: EXCHANGE_MYSQL
-    loader: custom_python_code
-    loaded_at_field: _source_loaded_at
-    freshness:
-      warn_after: {count: 12, period: hour}
-      error_after: {count: 24, period: hour}
-    meta: 
-      owner: "@yam"
-      model_maturity: prod
-    tags:
-      - exchange
-      - daily 
-
-
-    tables: 
-      - name: activity_log
-        description: Contains most table's activity log that CUD (created, updated, deleted) in www.exchange.cief-malaysia.com
-        columns:
-          - name: id
-            description: Primary key for 'activity_log'
-            tests:
-              - unique
-              - not_null
-
-          - name: description
-            description: system CUD activity list down
-            tests:
-              - accepted_values:
-                  values: ['created', 'updated', 'deleted']
-
-          - name: subject_id
-            description: The exchange CUD table's id, need to refer subject_type column to know what table is CUD in the row
-            tests:
-              - not_null     
-
-          - name: subject_type
-            description: What table that CUD using in the row
-            tests:
-              - not_null        
-
-          - name: causer_id
-            description: Need to refer causer_type column to know who is CUD in the row. it is not well maintain there is some NULL     
-
-          - name: causer_type
-            description: Who is CUD the row
-         
-         - name: properties
-            description: Json data format that contain attributes of the activity, it is not well maintain, subject (Country, ServiceType, CurrencyRate. UserEmailVerification, File) have properties only            
-            
-         - name: created_at
-            description: Datetime that CUD activity occur
-            tests:
-              - not_null   
-
-         - name: updated_at
-            description: Exactly same with created_at, redundant column   
-
-        tests:
-          - dbt_expectations.expect_column_pair_values_to_be_equal:
-              column_A: updated_at
-              column_B: created_at
-
-
-      - name: addresses
-        description: Addresses that input when user register 
-        columns:
-          - name: id
-            description: Primary key for 'addresses'
-            tests:
-              - unique
-              - not_null
-
-          - name: company_id
-            description: Foreign keys for 'companies'
-            tests:
-              - not_null
-
-          - name: country_id
-            description: Foreign keys for 'countries'
-            tests:
-              - not_null
-
-          - name: state_id
-            description: Foreign keys for 'states'
-            tests:
-              - not_null
-
-          - name: district_id
-            description: Foreign keys for 'districts'
-            tests:
-              - not_null
-
-          - name: postcode
-            description: The number/string that user input
-            tests:
-              - not_null
-              - dbt_expectations.expect_column_values_to_be_of_type:
-                  column_type: integer
-
-          - name: street_one
-            description: The string that user input
-            tests:
-              - not_null      
-
-          - name: street_two
-            description: The string that user input
-
-          - name: billing
-            description: If mark 1 means it is billing address, 0 means is not billing address, every company should have 1 billing address
-            tests:
-              - dbt_expectations.expect_column_max_to_be_between:       #Every company should have a billing address
-                  min_value: 1
-                  max_value: 1
-                  group_by: [company_id]
-              - dbt_expectations.expect_column_sum_to_be_between:       #1 company should have 1 billing address only
-                  min_value: 1
-                  max_value: 1
-                  group_by: [company_id]
-                  row_condition: "deleted_at is null"
-
-          - name: deleted_at
-            description: System generate datetime that address was deleted
-
-          - name: created_at
-            description: System generate datetime that address was created
-            tests:
-              - not_null
-
-          - name: updated_at
-            description: System generate datetime that address was updated 
-            tests:
-              - not_null               
-
-
-      - name: announcement_segment
-        description: Superadmin / Operator guys will segment the announcment, but normally they will leave it default. 2 FK combine to becoem a relation table.
-        columns:
-          - name: announcement_id
-            description: Primary key for 'announcement_segment' & Foreign key for 'announcements'
-            tests:
-              - not_null
-          
-          - name: segment_id
-            description: Primary key for 'announcement_segment' & Foreign key for 'segments'
-            tests:
-              - not_null
-
-          - name: created_at
-            description: Abandon column
-            tests:
-              - dbt_expectations.expect_column_values_to_be_null
-
-          - name: updated_at
-            description: Abandon column
-            tests:
-              - dbt_expectations.expect_column_values_to_be_null              
-
-
-      - name: announcements
-        description: Announcements that show in our website which create by superadmin
-        columns:
-          - name: id
-            description: Primary key for 'announcements'
-            tests:
-              - unique
-              - not_null
-          
-          - name: title
-            description: title of announcement, using string
-            tests:
-              - not_null
-
-          - name: description
-            description: description of announcement, using longtext, have <br> html for break
-            tests:
-              - not_null
-
-          - name: starting_on
-            description: announcement starting datetime
-            tests:
-              - not_null  
-
-          - name: ending_on
-            description: announcement ending datetime
-            tests:
-              - not_null 
-
-          - name: is_all_day
-            description: column might not in use, system auto generate
-            tests:
-              - not_null
-              - accepted_values:
-                  values: [0]
-
-          - name: duration
-            description: column might not in use, system auto generate
-            tests:
-              - not_null 
-              - accepted_values:
-                  values: [0]
-
-          - name: is_recurring
-            description: column might not in use, system auto generate
-            tests:
-              - not_null 
-              - accepted_values:
-                  values: [0]
-
-          - name: recurrence_pattern
-            description: column might not in use, system auto generate
-            tests:
-              - dbt_expectations.expect_column_values_to_be_null
-
-          - name: deleted_at
-            description: system generate datetime that announcment was deleted
-            tests:
-              - not_null
-
-          - name: created_at
-            description: system generate datetime that announcment was created
-            tests:
-              - not_null
-
-          - name: updated_at
-            description: system generate datetime that announcment was updated
-            tests:
-              - not_null
-
-            
-      - name: bank_logs
-        description: Abandon table
-        config:
-          enabled: false
-
-
-      - name: banks
-        description: Table which store banks details.
-        columns:
-          - name: id
-            description: Primary key for 'banks'
-            tests:
-              - unique
-              - not_null
-
-          - name: company_id
-            description: Foreign keys for 'companies'
-            tests:
-              - not_null            
-
-          - name: reference
-            description: A string which write by user/company for the bank transfer reference or other remind ussage
-
-          - name: holder_name
-            description: Bank holder name  
-            tests:
-              - not_null 
-
-           - name: account_no
-            description: Bank account number
-            tests:
-              - not_null            
-      
-
-
-
-      - name: booking_logs
-        description: Raw `bookings` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'booking_logs'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: bookings
-        description: Raw `bookings` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'bookings'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: companies
-        description: Raw `companies` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'companies'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: contacts
-        description: Raw `contacts` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'contacts'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: countries
-        description: Raw `countries` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'countries'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: currencies
-        description: Raw `currencies` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'currencies'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: currency_logs
-        description: Raw `currency_logs` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'currency_logs'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: currency_rate_logs
-        description: Raw `currency_rate_logs` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'currency_rate_logs'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: currency_rates
-        description: Raw `currency_rates` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'currency_rates'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: districts
-        description: Raw `districts` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'districts'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: documents
-        description: Raw `documents` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'documents'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: employees
-        description: Raw `employees` table from Snowflake
-        columns:
-          - name: company_id
-            description: Primary key for 'employees'
-
-
-      - name: failed_jobs
-        description: Raw `failed_jobs` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'failed_jobs'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: files
-        description: Raw `files` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'files'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: group_transactions
-        description: Raw `group_transactions` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'group_transactions'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: groups
-        description: Raw `groups` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'groups'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: jobs
-        description: Raw `jobs` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'jobs'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: migrations
-        description: Raw `migrations` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'migrations'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: model_has_roles
-        description: Raw `model_has_roles` table from Snowflake
-        columns:
-          - name: role_id
-            description: Primary key for 'model_has_roles'
-
-
-      - name: notifications
-        description: Raw `notifications` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'notifications'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: password_resets
-        description: Raw `password_resets` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'password_resets'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: permissions
-        description: Raw `permissions` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'permissions'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: role_has_permissions
-        description: Raw `role_has_permissions` table from Snowflake
-        columns:
-          - name: permission_id
-            description: Primary key for 'role_has_permissions'
-
-
-      - name: roles
-        description: Raw `roles` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'roles'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: segment_companies
-        description: Raw `segment_companies` table from Snowflake
-        columns:
-          - name: segment_id
-            description: Primary key for 'segment_companies'
-
-
-      - name: segment_constants
-        description: Raw `segment_constants` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'segment_constants'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: segments
-        description: Raw `segments` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'segments'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: service_types
-        description: Raw `service_types` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'service_types'
-            tests:
-              - unique
-              - not_null
-
-
-        description: Raw `states` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'states'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: transaction_detail
-        description: Raw `transaction_detail` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'transaction_detail'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: transaction_logs
-        description: Raw `transaction_logs` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'transaction_logs'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: transactions
-        description: Raw `transactions` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'transactions'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: user_email_verifications
-        description: Raw `user_email_verifications` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'user_email_verifications'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: users
-        description: Raw `users` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'users'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: wallet_logs
-        description: Raw `wallet_logs` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'wallet_logs'
-            tests:
-              - unique
-              - not_null
-
-
-      - name: wallets
-        description: Raw `wallets` table from Snowflake
-        columns:
-          - name: id
-            description: Primary key for 'wallets'
-            tests:
-              - unique
-              - not_null

models/staging/exchange/stg_exchange__activity_logs.sql

@@ -1,94 +0,0 @@
--- IMPORTS
-WITH activity_logs AS (
-    SELECT * FROM {{ source('src_exchange_mysql', 'activity_log') }}
-),
-
-
--- LOGIC
-activity_logs_rename AS (
-
-    SELECT
-        activity_logs.id,
-        activity_logs.log_name,
-        activity_logs.description,
-        activity_logs.subject_id,
-        activity_logs.subject_type,
-        activity_logs.causer_id,
-        activity_logs.causer_type,
-        activity_logs.properties,
-        activity_logs.created_at,
-        activity_logs.updated_at,
-        current_timestamp()                                                                             AS _dbt_ran_at
-
-    FROM activity_logs
-),
-
-
--- FINAL
-final__base_exchange__activity_logs AS (
-
-    SELECT
-        id,
-        log_name,
-        description,
-        subject_id,
-        subject_type,
-        causer_id,
-        causer_type,
-        properties,
-        created_at,
-        updated_at,
-        _dbt_ran_at
-
-    FROM activity_logs_rename
-) 
-
-SELECT * FROM final__base_exchange__activity_logs
-
-
--- IMPORTS
-WITH activity_logs AS (
-    SELECT * FROM {{ ref('base_exchange__activity_logs') }}
-),
-
-
--- LOGIC
-activity_logs_listout AS (
-
-    SELECT
-        activity_logs.id,
-        activity_logs.log_name,
-        activity_logs.description,
-        activity_logs.subject_id,
-        activity_logs.subject_type,
-        activity_logs.causer_id,
-        activity_logs.causer_type,
-        activity_logs.properties,
-        activity_logs.created_at,
-        activity_logs.updated_at,
-        CURRENT_TIMESTAMP()                                                                         AS _dbt_ran_at
-
-    FROM activity_logs
-),
-
-
--- FINAL
-final__stg_exchange__activity_logs AS (
-
-    SELECT
-        id,
-        log_name,
-        description,
-        subject_id,
-        subject_type,
-        causer_id,
-        causer_type,
-        properties,
-        created_at,
-        updated_at,
-        _dbt_ran_at
-
-    FROM activity_logs_listout
-) 
-
-SELECT * FROM final__stg_exchange__activity_logs