DBT restructure

README.md

@@ -32,56 +32,129 @@ Short naming converntion lists that need to follow
 <br/>
 
 ## DBT Rule
-### Structure of the dbt folder model
+### Structure of the dbt model layer
 - Analyses
-    1. Analyses model will not create in Snowflake
+    1. Analyses layer will not create in Snowflake
     2. You can write some adhoc queries 
-    3. test some SQL code before write in Models
+    3. Test some SQL code before write in Models
 
 <br/>
 
 - Macros
-    1. Jinja function that reuse in other .sql / .py file
+    1. Jinja function that reuse in other models
 
-<br>
+<br/>
 
 - Models 
     - Staging
-        1. Materialise: `View`  
+        - System name
-        2. 1-to-1 relationship (or mapping) to source tables.
+            1. Materialise: `View`  
-        3. Column renaming
+            2. 1-to-1 relationship (or mapping) to source tables.
-        4. Column remove
+            3. Column renaming
-        5. Column data renaming, such as (status integer to string)
+            4. Column remove
-        6. Data input error cleansing
+            5. Column data renaming, such as (status integer to string)
-        7. Check for raw data freshness
+            6. Data input error cleansing
-        8. Data type transformation
+            7. Check for raw data freshness
+            8. Data type transformation
+            9. Data split and merge
+            - Source
+                1. Source will write in YML
+                2. Check Freshness 
+                3. Check duplicate id
+                4. Check null id
 
     - Intermediate
         1. Materialise: `Ephemerally`
         2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want
         3. Be referenced repeatedly in more than one model
-        4. Isolating complex operations
+        4. Isolating complex operations       
         
-        
+    - Marts/warehouse
-    - marts/warehouse
+        1. Materialise: `Table`
-        1. Store fact and dimension models
+        2. Store fact and dimension models
 
-    - marts/reporting
+    - Marts/reporting
-        1. Store custom reports
+        1. Materialise: `Table`
+        2. Store custom reports
+
+<br/>
 
 - Seeds
+    1. Store custom dataset in csv format
+    2. The dataset must not change frequently
+
+<br/>
 
 - Snapshots
+    1. DBT build-in SCD type 2 function
+    2. Prefer to use after the source table and before the staging layer
+
+<br/>
 
 - Tests
+    1. Can write some yml test case
+
+<br/>
 
 ### File Naming Rules
 - File names must be unique
 - Each sql/python file must start with [`model_type`]_[`system_name`]__[`model_name`]s.sql/py
-- The model should be name as plural, (eg: stg_exchange__orders.sql)
+- The model should be name as plural. (eg: stg_exchange__orders.sql)
+
+<br/>
 
 ### Column Naming Rules
-- If array datatype, the column naming must be plural eg (ids, messages)
+- If array datatype, the column naming must be plural (eg: ids, messages)
+- Schema, table and column names should be in `snake_case`.
+
+- Limit use of abbreviations that are related to domain knowledge. An onboarding
+  employee will understand `current_order_status` better than `current_os`.
+
+- Use names based on the _business_ terminology, rather than the source terminology.
+
+- Each model should have a primary key that can identify the unique row, and should be named `<object>_id`, e.g. `account_id` – this makes it easier to know what `id` is being referenced in downstream joined models.
+
+- If a surrogate key is created, it should be named `<object>_sk`.
+
+- For `base` or `staging` models, columns should be ordered in categories, where identifiers are first and date/time fields are at the end.  
+  Example:
+  ```sql
+  transformed as (
+      select
+          -- ids
+          order_id,
+          customer_id,
+
+          -- dimensions
+          order_status,
+          is_shipped,
+
+          -- measures
+          order_total,
+
+          -- date/times
+          created_at,
+          updated_at,
+
+          -- metadata
+          _sdc_batched_at
+      from source
+  )
+  ```
+
+- Date/time columns should be named according to these conventions:
+  - Timestamps: `<event>_datetime`  
+    Example: `created_datetime`
+  
+  - Dates: `<event>_date`  
+    Example: `created_date`
+
+- Booleans should be prefixed with `is_` or `has_`.  
+  Example: `is_active_customer` and `has_admin_access`
+
+- Price/revenue fields should be in decimal currency (e.g. `19.99` for $19.99; many app databases store prices as integers in cents). If non-decimal currency is used, indicate this with suffix, e.g. `price_in_cents`.
+
+<br/>
 
 ### SQL Coding Rule
 - The SQL clause **must** be UPPERCASE (eg. `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `LIMIT`, `WITH`, `AS`, `SUM`, `PARTITION OVER`, `DIV0`, `LEFT JOIN`, `ON` ....)
@@ -96,7 +169,7 @@ WITH [table_names] AS (
 ),
 
 --LOGIC
-[logic_name] AS (
+[logic_names] AS (
     .....
 ),
 
@@ -104,21 +177,8 @@ WITH [table_names] AS (
 final__[table_names] AS (
 
 )
-SELECT * FROM final__[table_names]
+SELECT * FROM final__[final_table_names]
 ```
 
-
+### Testing
-
+- At a minimum, `unique` and `not_null` tests should be applied to the expected primary key of each model.
-
-
-Try running the following commands:
-- dbt run
-- dbt test
-
-
-### Resources:
-- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
-- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
-- Join the [dbt community](http://community.getbdt.com/) to learn from other analytics engineers
-- Find [dbt events](https://events.getdbt.com) near you
-- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices

analyses/exchange/company_conversion_v2.sql

@@ -1,5 +1,5 @@
 -- MARCRO for count the cummulative_activity_number
-{% set funnel_activity = ['register','email_verified','identity_documents_uploaded','identity_documents_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%}
+{% set funnel_activity = ['register','email_verified','identity_document_uploaded','identity_document_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%}
 
 -- IMPORT
 WITH companies AS (
@@ -69,12 +69,13 @@ company_first_time_order_date AS (
 
 
 
---start defind funnel
+--start defind funnel activity
 repeat_completed_order_after_60days_activities AS (
     SELECT 
         clean_company_transaction_orders.company_id,
         0 AS is_data_cleansing_generate_row,
         'repeat_completed_order_after_60days' AS activity,
+        9 AS activtity_rank,
         MIN(clean_company_transaction_orders.created_at) AS activity_datetime
 
     FROM clean_company_transaction_orders
@@ -94,6 +95,7 @@ repeat_completed_order_after_30days_activities AS (
         clean_company_transaction_orders.company_id,
         0 AS is_data_cleansing_generate_row,
         'repeat_completed_order_after_30days' AS activity,
+        8 AS activtity_rank,
         MIN(clean_company_transaction_orders.created_at) AS activity_datetime
 
     FROM clean_company_transaction_orders
@@ -113,6 +115,7 @@ first_time_order_completed_activities AS (
         clean_company_transaction_orders.company_id,
         0 AS is_data_cleansing_generate_row,
         'first_time_order_completed' AS activity,
+        7 AS activtity_rank,
         MIN(clean_company_transaction_orders.created_at) AS activity_datetime
 
     FROM clean_company_transaction_orders
@@ -129,6 +132,7 @@ first_time_order_activities AS (
         clean_company_transaction_orders.company_id,
         0 AS is_data_cleansing_generate_row,
         'first_time_order' AS activity,
+        6 AS activtity_rank,
         MIN(clean_company_transaction_orders.created_at) AS activity_datetime
 
     FROM clean_company_transaction_orders
@@ -145,6 +149,7 @@ first_time_booking_activities AS (
         IFNULL( unclean_first_time_booking_activities.company_id , first_time_order_activities.company_id ) AS company_id,
         IFF( unclean_first_time_booking_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
         'first_time_booking' AS activity,
+        5 AS activtity_rank,
         IFNULL( unclean_first_time_booking_activities.activity_datetime , first_time_order_activities.activity_datetime ) AS activity_datetime
 
     FROM
@@ -168,19 +173,20 @@ first_time_booking_activities AS (
 ),
 
 
-identity_documents_verified_activities AS (
+identity_document_verified_activities AS (
     SELECT
-        IFNULL( unclean_identity_documents_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id,
+        IFNULL( unclean_identity_document_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id,
-        IFF( unclean_identity_documents_verified_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
+        IFF( unclean_identity_document_verified_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
-        'identity_documents_verified' AS activity,
+        'identity_document_verified' AS activity,
-        IFNULL( unclean_identity_documents_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime
+        4 AS activtity_rank,
+        IFNULL( unclean_identity_document_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime
     
     FROM 
         (
 
         SELECT
             company_id,
-            'identity_documents_verified' AS activity,
+            'identity_document_verified' AS activity,
             MIN(identity_created_datetime) AS activity_datetime
         
         FROM company_identity_documents
@@ -191,27 +197,28 @@ identity_documents_verified_activities AS (
         GROUP BY 
             company_id
 
-        ) AS unclean_identity_documents_verified_activities
+        ) AS unclean_identity_document_verified_activities
     
     
     FULL OUTER JOIN first_time_booking_activities 
-        ON (unclean_identity_documents_verified_activities.company_id = first_time_booking_activities.company_id)
+        ON (unclean_identity_document_verified_activities.company_id = first_time_booking_activities.company_id)
 
 ),
 
-identity_documents_uploaded_activities AS (
+identity_document_uploaded_activities AS (
     SELECT
-        IFNULL( unclean_identity_documents_uploaded_activities.company_id , identity_documents_verified_activities.company_id ) AS company_id,
+        IFNULL( unclean_identity_document_uploaded_activities.company_id , identity_document_verified_activities.company_id ) AS company_id,
-        IFF( unclean_identity_documents_uploaded_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
+        IFF( unclean_identity_document_uploaded_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
-        'identity_documents_uploaded' AS activity,
+        'identity_document_uploaded' AS activity,
-        IFNULL( unclean_identity_documents_uploaded_activities.activity_datetime , identity_documents_verified_activities.activity_datetime ) AS activity_datetime
+        3 AS activtity_rank,
+        IFNULL( unclean_identity_document_uploaded_activities.activity_datetime , identity_document_verified_activities.activity_datetime ) AS activity_datetime
     
     FROM 
         (
 
         SELECT
             company_id,
-            'identity_documents_uploaded' AS activity,
+            'identity_document_uploaded' AS activity,
             MIN(identity_created_datetime) AS activity_datetime
         
         FROM company_identity_documents
@@ -219,19 +226,20 @@ identity_documents_uploaded_activities AS (
         GROUP BY  
             company_id
 
-        ) AS unclean_identity_documents_uploaded_activities
+        ) AS unclean_identity_document_uploaded_activities
     
     
-    FULL OUTER JOIN identity_documents_verified_activities 
+    FULL OUTER JOIN identity_document_verified_activities 
-        ON (unclean_identity_documents_uploaded_activities.company_id = identity_documents_verified_activities.company_id)
+        ON (unclean_identity_document_uploaded_activities.company_id = identity_document_verified_activities.company_id)
 ),
 
 email_verified_activities AS (
     SELECT
-        IFNULL( unclean_email_verified_activities.company_id , identity_documents_uploaded_activities.company_id ) AS company_id,
+        IFNULL( unclean_email_verified_activities.company_id , identity_document_uploaded_activities.company_id ) AS company_id,
         IFF( unclean_email_verified_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
         'email_verified' AS activity,
-        IFNULL( unclean_email_verified_activities.activity_datetime , identity_documents_uploaded_activities.activity_datetime ) AS activity_datetime
+        2 AS activtity_rank,
+        IFNULL( unclean_email_verified_activities.activity_datetime , identity_document_uploaded_activities.activity_datetime ) AS activity_datetime
     
     FROM 
         (
@@ -249,8 +257,8 @@ email_verified_activities AS (
             company_id
         ) AS unclean_email_verified_activities
 
-    FULL OUTER JOIN identity_documents_uploaded_activities
+    FULL OUTER JOIN identity_document_uploaded_activities
-        ON (unclean_email_verified_activities.company_id = identity_documents_uploaded_activities.company_id)
+        ON (unclean_email_verified_activities.company_id = identity_document_uploaded_activities.company_id)
 
 ),
 
@@ -259,6 +267,7 @@ register_activities AS (
         IFNULL( unclean_register_activities.company_id , email_verified_activities.company_id ) AS company_id,
         IFF( unclean_register_activities.company_id IS NULL, 1, 0)  AS is_data_cleansing_generate_row,
         'register' AS activity,
+        1 AS activtity_rank,
         IFNULL( unclean_register_activities.activity_datetime , email_verified_activities.activity_datetime ) AS activity_datetime
 
     FROM 
@@ -310,24 +319,30 @@ union_all_activities_flag_migrated_company AS (
 
 --FINAL
 
-sem_exchange__company_funnels AS (
+sem_exchange__company_conversion_funnels AS (
     SELECT
         company_id,
         is_data_cleansing_generate_row,
         is_migrated_company,
+        activity,
+        activtity_rank,
+        IFF( ROW_NUMBER() OVER (PARTITION BY company_id ORDER BY activtity_rank DESC) = 1,
+            1,
+            0)  AS latest_activity,
+            
         activity_datetime,
 
-    {% for activity in funnel_activity %}
+    {% for activity_type in funnel_activity %}
 
-        count(iff(activity = '{{activity}}', company_id, NULL)) 
+        count(iff(activity = '{{activity_type}}', company_id, NULL)) 
-            OVER (ORDER BY activity_datetime) AS cumulative_{{activity}}_activity,
+            OVER (ORDER BY activity_datetime) AS cumulative_{{activity_type}}_activity,
 
     {% endfor %}
 
-    {% for activity in funnel_activity %}
+    {% for activity_type in funnel_activity %}
 
-        count(iff(activity = '{{activity}}' AND is_migrated_company = 0, company_id, NULL)) 
+        count(iff(activity = '{{activity_type}}' AND is_migrated_company = 0, company_id, NULL)) 
-            OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity}}_activity,
+            OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity_type}}_activity,
 
     {% endfor %}
 
@@ -338,4 +353,4 @@ sem_exchange__company_funnels AS (
   ORDER BY activity_datetime
 )
 
-SELECT * FROM sem_exchange__company_funnels
+SELECT * FROM sem_exchange__company_conversion_funnels

models/exchange/staging/activity_logs/stg_exchange__activity_logs.sql

@@ -1,46 +0,0 @@
--- IMPORTS
-WITH activity_logs AS (
-    SELECT * FROM {{ ref('base_exchange__activity_logs') }}
-),
-
-
--- LOGIC
-activity_logs_listout AS (
-
-    SELECT
-        activity_logs.id,
-        activity_logs.log_name,
-        activity_logs.description,
-        activity_logs.subject_id,
-        activity_logs.subject_type,
-        activity_logs.causer_id,
-        activity_logs.causer_type,
-        activity_logs.properties,
-        activity_logs.created_at,
-        activity_logs.updated_at,
-        CURRENT_TIMESTAMP()                                                                         AS _dbt_ran_at
-
-    FROM activity_logs
-),
-
-
--- FINAL
-final__stg_exchange__activity_logs AS (
-
-    SELECT
-        id,
-        log_name,
-        description,
-        subject_id,
-        subject_type,
-        causer_id,
-        causer_type,
-        properties,
-        created_at,
-        updated_at,
-        _dbt_ran_at
-
-    FROM activity_logs_listout
-) 
-
-SELECT * FROM final__stg_exchange__activity_logs

models/exchange/staging/wallet/stg_exchange__wallet.sql

@@ -0,0 +1,10 @@
+-- IMPORTS
+WITH wallets AS (
+    SELECT * FROM {{ ref('base_exchange__wallets') }}
+)
+
+
+
+
+-- FINAL 
+SELECT * FROM wallets

models/exchange/staging/wallet_logs/stg_exchange__wallet_logs.sql

@@ -0,0 +1,7 @@
+-- IMPORTS
+with wallet_logs as (select * from {{ ref("base_exchange__wallet_logs") }})
+
+-- FINAL 
+select *
+from wallet_logs
+order by wallet_id, updated_at asc

models/shipping/staging/stg_shipping__activity_logs.sql

@@ -0,0 +1,46 @@
+-- IMPORTS
+WITH activity_logs AS (
+    SELECT * FROM {{ source('src_shipping_mysql', 'activity_log') }}
+),
+
+
+-- LOGIC
+activity_logs_rename AS (
+
+    SELECT
+        activity_logs.id,
+        activity_logs.log_name,
+        activity_logs.description,
+        activity_logs.subject_id,
+        activity_logs.subject_type,
+        activity_logs.causer_id,
+        activity_logs.causer_type,
+        activity_logs.properties,
+        activity_logs.created_at                                                                        AS created_datetime,
+        activity_logs.updated_at                                                                        AS updated_datetime,
+        current_timestamp()                                                                             AS _dbt_ran_at
+
+    FROM activity_logs
+),
+
+
+-- FINAL
+final__base_exchange__activity_logs AS (
+
+    SELECT
+        id,
+        log_name,
+        description,
+        subject_id,
+        subject_type,
+        causer_id,
+        causer_type,
+        properties,
+        created_datetime,
+        updated_datetime,
+        _dbt_ran_at
+
+    FROM activity_logs_rename
+) 
+
+SELECT * FROM final__base_exchange__activity_logs

models/staging/exchange/_exchange__docs.md

@@ -0,0 +1,23 @@
+{% docs src_exchange_mysql %}
+
+Data is extract from CIEF Selfhost MYSQL `exchange_production`, using custom python code.
+
+- `_source_loaded_at` will add into the raw data when extract
+```YML
+loaded_at_field: _source_loaded_at
+freshness:
+    warn_after: {count: 12, period: hour}
+    error_after: {count: 24, period: hour}
+```
+
+- The raw data will use copy into snowflake in 2AM everyday
+
+- Depend on target.name to determin which snowflake raw schema
+
+```JINJA
+{%- if target.name == "prod" -%} PROD_CIEF_RAW_DB
+{%- else -%} DEV_CIEF_RAW_DB
+{%- endif -%}
+```
+
+{% enddocs %}

models/staging/exchange/_exchange__models.yml

@@ -0,0 +1,6 @@
+version: 2
+
+models:
+  - name: stg_exchange__activity_logs
+
+  - name: stg_exchange__announcments  

models/staging/exchange/_exchange__sources.yml

@@ -0,0 +1,619 @@
+version: 2
+
+sources:
+  - name: src_exchange_mysql
+    description: '{{ doc("src_exchange_mysql") }}'
+    database: |
+              {%- if target.name == "prod" -%} PROD_CIEF_RAW_DB
+              {%- else -%} DEV_CIEF_RAW_DB
+              {%- endif -%}
+    schema: EXCHANGE_MYSQL
+    loader: custom_python_code
+    loaded_at_field: _source_loaded_at
+    freshness:
+      warn_after: {count: 12, period: hour}
+      error_after: {count: 24, period: hour}
+    meta: 
+      owner: "@yam"
+      model_maturity: prod
+    tags:
+      - exchange
+      - daily 
+
+
+    tables: 
+      - name: activity_log
+        description: Contains most table's activity log that CUD (created, updated, deleted) in www.exchange.cief-malaysia.com
+        columns:
+          - name: id
+            description: Primary key for 'activity_log'
+            tests:
+              - unique
+              - not_null
+
+          - name: description
+            description: system CUD activity list down
+            tests:
+              - accepted_values:
+                  values: ['created', 'updated', 'deleted']
+
+          - name: subject_id
+            description: The exchange CUD table's id, need to refer subject_type column to know what table is CUD in the row
+            tests:
+              - not_null     
+
+          - name: subject_type
+            description: What table that CUD using in the row
+            tests:
+              - not_null        
+
+          - name: causer_id
+            description: Need to refer causer_type column to know who is CUD in the row. it is not well maintain there is some NULL     
+
+          - name: causer_type
+            description: Who is CUD the row
+         
+         - name: properties
+            description: Json data format that contain attributes of the activity, it is not well maintain, subject (Country, ServiceType, CurrencyRate. UserEmailVerification, File) have properties only            
+            
+         - name: created_at
+            description: Datetime that CUD activity occur
+            tests:
+              - not_null   
+
+         - name: updated_at
+            description: Exactly same with created_at, redundant column   
+
+        tests:
+          - dbt_expectations.expect_column_pair_values_to_be_equal:
+              column_A: updated_at
+              column_B: created_at
+
+
+      - name: addresses
+        description: Addresses that input when user register 
+        columns:
+          - name: id
+            description: Primary key for 'addresses'
+            tests:
+              - unique
+              - not_null
+
+          - name: company_id
+            description: Foreign keys for 'companies'
+            tests:
+              - not_null
+
+          - name: country_id
+            description: Foreign keys for 'countries'
+            tests:
+              - not_null
+
+          - name: state_id
+            description: Foreign keys for 'states'
+            tests:
+              - not_null
+
+          - name: district_id
+            description: Foreign keys for 'districts'
+            tests:
+              - not_null
+
+          - name: postcode
+            description: The number/string that user input
+            tests:
+              - not_null
+              - dbt_expectations.expect_column_values_to_be_of_type:
+                  column_type: integer
+
+          - name: street_one
+            description: The string that user input
+            tests:
+              - not_null      
+
+          - name: street_two
+            description: The string that user input
+
+          - name: billing
+            description: If mark 1 means it is billing address, 0 means is not billing address, every company should have 1 billing address
+            tests:
+              - dbt_expectations.expect_column_max_to_be_between:       #Every company should have a billing address
+                  min_value: 1
+                  max_value: 1
+                  group_by: [company_id]
+              - dbt_expectations.expect_column_sum_to_be_between:       #1 company should have 1 billing address only
+                  min_value: 1
+                  max_value: 1
+                  group_by: [company_id]
+                  row_condition: "deleted_at is null"
+
+          - name: deleted_at
+            description: System generate datetime that address was deleted
+
+          - name: created_at
+            description: System generate datetime that address was created
+            tests:
+              - not_null
+
+          - name: updated_at
+            description: System generate datetime that address was updated 
+            tests:
+              - not_null               
+
+
+      - name: announcement_segment
+        description: Superadmin / Operator guys will segment the announcment, but normally they will leave it default. 2 FK combine to becoem a relation table.
+        columns:
+          - name: announcement_id
+            description: Primary key for 'announcement_segment' & Foreign key for 'announcements'
+            tests:
+              - not_null
+          
+          - name: segment_id
+            description: Primary key for 'announcement_segment' & Foreign key for 'segments'
+            tests:
+              - not_null
+
+          - name: created_at
+            description: Abandon column
+            tests:
+              - dbt_expectations.expect_column_values_to_be_null
+
+          - name: updated_at
+            description: Abandon column
+            tests:
+              - dbt_expectations.expect_column_values_to_be_null              
+
+
+      - name: announcements
+        description: Announcements that show in our website which create by superadmin
+        columns:
+          - name: id
+            description: Primary key for 'announcements'
+            tests:
+              - unique
+              - not_null
+          
+          - name: title
+            description: title of announcement, using string
+            tests:
+              - not_null
+
+          - name: description
+            description: description of announcement, using longtext, have <br> html for break
+            tests:
+              - not_null
+
+          - name: starting_on
+            description: announcement starting datetime
+            tests:
+              - not_null  
+
+          - name: ending_on
+            description: announcement ending datetime
+            tests:
+              - not_null 
+
+          - name: is_all_day
+            description: column might not in use, system auto generate
+            tests:
+              - not_null
+              - accepted_values:
+                  values: [0]
+
+          - name: duration
+            description: column might not in use, system auto generate
+            tests:
+              - not_null 
+              - accepted_values:
+                  values: [0]
+
+          - name: is_recurring
+            description: column might not in use, system auto generate
+            tests:
+              - not_null 
+              - accepted_values:
+                  values: [0]
+
+          - name: recurrence_pattern
+            description: column might not in use, system auto generate
+            tests:
+              - dbt_expectations.expect_column_values_to_be_null
+
+          - name: deleted_at
+            description: system generate datetime that announcment was deleted
+            tests:
+              - not_null
+
+          - name: created_at
+            description: system generate datetime that announcment was created
+            tests:
+              - not_null
+
+          - name: updated_at
+            description: system generate datetime that announcment was updated
+            tests:
+              - not_null
+
+            
+      - name: bank_logs
+        description: Abandon table
+        config:
+          enabled: false
+
+
+      - name: banks
+        description: Table which store banks details.
+        columns:
+          - name: id
+            description: Primary key for 'banks'
+            tests:
+              - unique
+              - not_null
+
+          - name: company_id
+            description: Foreign keys for 'companies'
+            tests:
+              - not_null            
+
+          - name: reference
+            description: A string which write by user/company for the bank transfer reference or other remind ussage
+
+          - name: holder_name
+            description: Bank holder name  
+            tests:
+              - not_null 
+
+           - name: account_no
+            description: Bank account number
+            tests:
+              - not_null            
+      
+
+
+
+      - name: booking_logs
+        description: Raw `bookings` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'booking_logs'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: bookings
+        description: Raw `bookings` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'bookings'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: companies
+        description: Raw `companies` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'companies'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: contacts
+        description: Raw `contacts` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'contacts'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: countries
+        description: Raw `countries` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'countries'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: currencies
+        description: Raw `currencies` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'currencies'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: currency_logs
+        description: Raw `currency_logs` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'currency_logs'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: currency_rate_logs
+        description: Raw `currency_rate_logs` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'currency_rate_logs'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: currency_rates
+        description: Raw `currency_rates` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'currency_rates'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: districts
+        description: Raw `districts` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'districts'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: documents
+        description: Raw `documents` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'documents'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: employees
+        description: Raw `employees` table from Snowflake
+        columns:
+          - name: company_id
+            description: Primary key for 'employees'
+
+
+      - name: failed_jobs
+        description: Raw `failed_jobs` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'failed_jobs'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: files
+        description: Raw `files` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'files'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: group_transactions
+        description: Raw `group_transactions` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'group_transactions'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: groups
+        description: Raw `groups` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'groups'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: jobs
+        description: Raw `jobs` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'jobs'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: migrations
+        description: Raw `migrations` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'migrations'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: model_has_roles
+        description: Raw `model_has_roles` table from Snowflake
+        columns:
+          - name: role_id
+            description: Primary key for 'model_has_roles'
+
+
+      - name: notifications
+        description: Raw `notifications` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'notifications'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: password_resets
+        description: Raw `password_resets` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'password_resets'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: permissions
+        description: Raw `permissions` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'permissions'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: role_has_permissions
+        description: Raw `role_has_permissions` table from Snowflake
+        columns:
+          - name: permission_id
+            description: Primary key for 'role_has_permissions'
+
+
+      - name: roles
+        description: Raw `roles` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'roles'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: segment_companies
+        description: Raw `segment_companies` table from Snowflake
+        columns:
+          - name: segment_id
+            description: Primary key for 'segment_companies'
+
+
+      - name: segment_constants
+        description: Raw `segment_constants` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'segment_constants'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: segments
+        description: Raw `segments` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'segments'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: service_types
+        description: Raw `service_types` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'service_types'
+            tests:
+              - unique
+              - not_null
+
+
+        description: Raw `states` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'states'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: transaction_detail
+        description: Raw `transaction_detail` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'transaction_detail'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: transaction_logs
+        description: Raw `transaction_logs` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'transaction_logs'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: transactions
+        description: Raw `transactions` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'transactions'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: user_email_verifications
+        description: Raw `user_email_verifications` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'user_email_verifications'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: users
+        description: Raw `users` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'users'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: wallet_logs
+        description: Raw `wallet_logs` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'wallet_logs'
+            tests:
+              - unique
+              - not_null
+
+
+      - name: wallets
+        description: Raw `wallets` table from Snowflake
+        columns:
+          - name: id
+            description: Primary key for 'wallets'
+            tests:
+              - unique
+              - not_null

models/staging/exchange/stg_exchange__activity_logs.sql

@@ -0,0 +1,94 @@
+-- IMPORTS
+WITH activity_logs AS (
+    SELECT * FROM {{ source('src_exchange_mysql', 'activity_log') }}
+),
+
+
+-- LOGIC
+activity_logs_rename AS (
+
+    SELECT
+        activity_logs.id,
+        activity_logs.log_name,
+        activity_logs.description,
+        activity_logs.subject_id,
+        activity_logs.subject_type,
+        activity_logs.causer_id,
+        activity_logs.causer_type,
+        activity_logs.properties,
+        activity_logs.created_at,
+        activity_logs.updated_at,
+        current_timestamp()                                                                             AS _dbt_ran_at
+
+    FROM activity_logs
+),
+
+
+-- FINAL
+final__base_exchange__activity_logs AS (
+
+    SELECT
+        id,
+        log_name,
+        description,
+        subject_id,
+        subject_type,
+        causer_id,
+        causer_type,
+        properties,
+        created_at,
+        updated_at,
+        _dbt_ran_at
+
+    FROM activity_logs_rename
+) 
+
+SELECT * FROM final__base_exchange__activity_logs
+
+
+-- IMPORTS
+WITH activity_logs AS (
+    SELECT * FROM {{ ref('base_exchange__activity_logs') }}
+),
+
+
+-- LOGIC
+activity_logs_listout AS (
+
+    SELECT
+        activity_logs.id,
+        activity_logs.log_name,
+        activity_logs.description,
+        activity_logs.subject_id,
+        activity_logs.subject_type,
+        activity_logs.causer_id,
+        activity_logs.causer_type,
+        activity_logs.properties,
+        activity_logs.created_at,
+        activity_logs.updated_at,
+        CURRENT_TIMESTAMP()                                                                         AS _dbt_ran_at
+
+    FROM activity_logs
+),
+
+
+-- FINAL
+final__stg_exchange__activity_logs AS (
+
+    SELECT
+        id,
+        log_name,
+        description,
+        subject_id,
+        subject_type,
+        causer_id,
+        causer_type,
+        properties,
+        created_at,
+        updated_at,
+        _dbt_ran_at
+
+    FROM activity_logs_listout
+) 
+
+SELECT * FROM final__stg_exchange__activity_logs