diff --git a/README.md b/README.md index d24c02a..ae37e92 100644 --- a/README.md +++ b/README.md @@ -32,56 +32,129 @@ Short naming converntion lists that need to follow
## DBT Rule -### Structure of the dbt folder model +### Structure of the dbt model layer - Analyses - 1. Analyses model will not create in Snowflake + 1. Analyses layer will not create in Snowflake 2. You can write some adhoc queries - 3. test some SQL code before write in Models + 3. Test some SQL code before write in Models
- Macros - 1. Jinja function that reuse in other .sql / .py file + 1. Jinja function that reuse in other models -
+
- Models - Staging - 1. Materialise: `View` - 2. 1-to-1 relationship (or mapping) to source tables. - 3. Column renaming - 4. Column remove - 5. Column data renaming, such as (status integer to string) - 6. Data input error cleansing - 7. Check for raw data freshness - 8. Data type transformation + - System name + 1. Materialise: `View` + 2. 1-to-1 relationship (or mapping) to source tables. + 3. Column renaming + 4. Column remove + 5. Column data renaming, such as (status integer to string) + 6. Data input error cleansing + 7. Check for raw data freshness + 8. Data type transformation + 9. Data split and merge + - Source + 1. Source will write in YML + 2. Check Freshness + 3. Check duplicate id + 4. Check null id - Intermediate 1. Materialise: `Ephemerally` 2. Stacking layers of logic with clear and specific purposes to prepa our staging models to join into the entities we want 3. Be referenced repeatedly in more than one model - 4. Isolating complex operations + 4. Isolating complex operations - - - marts/warehouse - 1. Store fact and dimension models + - Marts/warehouse + 1. Materialise: `Table` + 2. Store fact and dimension models - - marts/reporting - 1. Store custom reports + - Marts/reporting + 1. Materialise: `Table` + 2. Store custom reports + +
- Seeds + 1. Store custom dataset in csv format + 2. The dataset must not change frequently + +
- Snapshots + 1. DBT build-in SCD type 2 function + 2. Prefer to use after the source table and before the staging layer + +
- Tests + 1. Can write some yml test case + +
### File Naming Rules - File names must be unique - Each sql/python file must start with [`model_type`]_[`system_name`]__[`model_name`]s.sql/py -- The model should be name as plural, (eg: stg_exchange__orders.sql) +- The model should be name as plural. (eg: stg_exchange__orders.sql) + +
### Column Naming Rules -- If array datatype, the column naming must be plural eg (ids, messages) +- If array datatype, the column naming must be plural (eg: ids, messages) +- Schema, table and column names should be in `snake_case`. + +- Limit use of abbreviations that are related to domain knowledge. An onboarding + employee will understand `current_order_status` better than `current_os`. + +- Use names based on the _business_ terminology, rather than the source terminology. + +- Each model should have a primary key that can identify the unique row, and should be named `_id`, e.g. `account_id` – this makes it easier to know what `id` is being referenced in downstream joined models. + +- If a surrogate key is created, it should be named `_sk`. + +- For `base` or `staging` models, columns should be ordered in categories, where identifiers are first and date/time fields are at the end. + Example: + ```sql + transformed as ( + select + -- ids + order_id, + customer_id, + + -- dimensions + order_status, + is_shipped, + + -- measures + order_total, + + -- date/times + created_at, + updated_at, + + -- metadata + _sdc_batched_at + from source + ) + ``` + +- Date/time columns should be named according to these conventions: + - Timestamps: `_datetime` + Example: `created_datetime` + + - Dates: `_date` + Example: `created_date` + +- Booleans should be prefixed with `is_` or `has_`. + Example: `is_active_customer` and `has_admin_access` + +- Price/revenue fields should be in decimal currency (e.g. `19.99` for $19.99; many app databases store prices as integers in cents). If non-decimal currency is used, indicate this with suffix, e.g. `price_in_cents`. + +
### SQL Coding Rule - The SQL clause **must** be UPPERCASE (eg. `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `LIMIT`, `WITH`, `AS`, `SUM`, `PARTITION OVER`, `DIV0`, `LEFT JOIN`, `ON` ....) @@ -96,7 +169,7 @@ WITH [table_names] AS ( ), --LOGIC -[logic_name] AS ( +[logic_names] AS ( ..... ), @@ -104,21 +177,8 @@ WITH [table_names] AS ( final__[table_names] AS ( ) -SELECT * FROM final__[table_names] +SELECT * FROM final__[final_table_names] ``` - - - - -Try running the following commands: -- dbt run -- dbt test - - -### Resources: -- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction) -- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers -- Join the [dbt community](http://community.getbdt.com/) to learn from other analytics engineers -- Find [dbt events](https://events.getdbt.com) near you -- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices +### Testing +- At a minimum, `unique` and `not_null` tests should be applied to the expected primary key of each model. \ No newline at end of file diff --git a/analyses/exchange/company_conversion_v2.sql b/analyses/exchange/company_conversion_v2.sql index af2afe3..e124bcb 100644 --- a/analyses/exchange/company_conversion_v2.sql +++ b/analyses/exchange/company_conversion_v2.sql @@ -1,5 +1,5 @@ -- MARCRO for count the cummulative_activity_number -{% set funnel_activity = ['register','email_verified','identity_documents_uploaded','identity_documents_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%} +{% set funnel_activity = ['register','email_verified','identity_document_uploaded','identity_document_verified','first_time_booking','first_time_order','first_time_order_completed','repeat_completed_order_after_30days','repeat_completed_order_after_60days']%} -- IMPORT WITH companies AS ( @@ -69,12 +69,13 @@ company_first_time_order_date AS ( ---start defind funnel +--start defind funnel activity repeat_completed_order_after_60days_activities AS ( SELECT clean_company_transaction_orders.company_id, 0 AS is_data_cleansing_generate_row, 'repeat_completed_order_after_60days' AS activity, + 9 AS activtity_rank, MIN(clean_company_transaction_orders.created_at) AS activity_datetime FROM clean_company_transaction_orders @@ -94,6 +95,7 @@ repeat_completed_order_after_30days_activities AS ( clean_company_transaction_orders.company_id, 0 AS is_data_cleansing_generate_row, 'repeat_completed_order_after_30days' AS activity, + 8 AS activtity_rank, MIN(clean_company_transaction_orders.created_at) AS activity_datetime FROM clean_company_transaction_orders @@ -113,6 +115,7 @@ first_time_order_completed_activities AS ( clean_company_transaction_orders.company_id, 0 AS is_data_cleansing_generate_row, 'first_time_order_completed' AS activity, + 7 AS activtity_rank, MIN(clean_company_transaction_orders.created_at) AS activity_datetime FROM clean_company_transaction_orders @@ -129,6 +132,7 @@ first_time_order_activities AS ( clean_company_transaction_orders.company_id, 0 AS is_data_cleansing_generate_row, 'first_time_order' AS activity, + 6 AS activtity_rank, MIN(clean_company_transaction_orders.created_at) AS activity_datetime FROM clean_company_transaction_orders @@ -145,6 +149,7 @@ first_time_booking_activities AS ( IFNULL( unclean_first_time_booking_activities.company_id , first_time_order_activities.company_id ) AS company_id, IFF( unclean_first_time_booking_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, 'first_time_booking' AS activity, + 5 AS activtity_rank, IFNULL( unclean_first_time_booking_activities.activity_datetime , first_time_order_activities.activity_datetime ) AS activity_datetime FROM @@ -168,19 +173,20 @@ first_time_booking_activities AS ( ), -identity_documents_verified_activities AS ( +identity_document_verified_activities AS ( SELECT - IFNULL( unclean_identity_documents_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id, - IFF( unclean_identity_documents_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, - 'identity_documents_verified' AS activity, - IFNULL( unclean_identity_documents_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime + IFNULL( unclean_identity_document_verified_activities.company_id , first_time_booking_activities.company_id ) AS company_id, + IFF( unclean_identity_document_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, + 'identity_document_verified' AS activity, + 4 AS activtity_rank, + IFNULL( unclean_identity_document_verified_activities.activity_datetime , first_time_booking_activities.activity_datetime ) AS activity_datetime FROM ( SELECT company_id, - 'identity_documents_verified' AS activity, + 'identity_document_verified' AS activity, MIN(identity_created_datetime) AS activity_datetime FROM company_identity_documents @@ -191,27 +197,28 @@ identity_documents_verified_activities AS ( GROUP BY company_id - ) AS unclean_identity_documents_verified_activities + ) AS unclean_identity_document_verified_activities FULL OUTER JOIN first_time_booking_activities - ON (unclean_identity_documents_verified_activities.company_id = first_time_booking_activities.company_id) + ON (unclean_identity_document_verified_activities.company_id = first_time_booking_activities.company_id) ), -identity_documents_uploaded_activities AS ( +identity_document_uploaded_activities AS ( SELECT - IFNULL( unclean_identity_documents_uploaded_activities.company_id , identity_documents_verified_activities.company_id ) AS company_id, - IFF( unclean_identity_documents_uploaded_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, - 'identity_documents_uploaded' AS activity, - IFNULL( unclean_identity_documents_uploaded_activities.activity_datetime , identity_documents_verified_activities.activity_datetime ) AS activity_datetime + IFNULL( unclean_identity_document_uploaded_activities.company_id , identity_document_verified_activities.company_id ) AS company_id, + IFF( unclean_identity_document_uploaded_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, + 'identity_document_uploaded' AS activity, + 3 AS activtity_rank, + IFNULL( unclean_identity_document_uploaded_activities.activity_datetime , identity_document_verified_activities.activity_datetime ) AS activity_datetime FROM ( SELECT company_id, - 'identity_documents_uploaded' AS activity, + 'identity_document_uploaded' AS activity, MIN(identity_created_datetime) AS activity_datetime FROM company_identity_documents @@ -219,19 +226,20 @@ identity_documents_uploaded_activities AS ( GROUP BY company_id - ) AS unclean_identity_documents_uploaded_activities + ) AS unclean_identity_document_uploaded_activities - FULL OUTER JOIN identity_documents_verified_activities - ON (unclean_identity_documents_uploaded_activities.company_id = identity_documents_verified_activities.company_id) + FULL OUTER JOIN identity_document_verified_activities + ON (unclean_identity_document_uploaded_activities.company_id = identity_document_verified_activities.company_id) ), email_verified_activities AS ( SELECT - IFNULL( unclean_email_verified_activities.company_id , identity_documents_uploaded_activities.company_id ) AS company_id, + IFNULL( unclean_email_verified_activities.company_id , identity_document_uploaded_activities.company_id ) AS company_id, IFF( unclean_email_verified_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, 'email_verified' AS activity, - IFNULL( unclean_email_verified_activities.activity_datetime , identity_documents_uploaded_activities.activity_datetime ) AS activity_datetime + 2 AS activtity_rank, + IFNULL( unclean_email_verified_activities.activity_datetime , identity_document_uploaded_activities.activity_datetime ) AS activity_datetime FROM ( @@ -249,8 +257,8 @@ email_verified_activities AS ( company_id ) AS unclean_email_verified_activities - FULL OUTER JOIN identity_documents_uploaded_activities - ON (unclean_email_verified_activities.company_id = identity_documents_uploaded_activities.company_id) + FULL OUTER JOIN identity_document_uploaded_activities + ON (unclean_email_verified_activities.company_id = identity_document_uploaded_activities.company_id) ), @@ -259,6 +267,7 @@ register_activities AS ( IFNULL( unclean_register_activities.company_id , email_verified_activities.company_id ) AS company_id, IFF( unclean_register_activities.company_id IS NULL, 1, 0) AS is_data_cleansing_generate_row, 'register' AS activity, + 1 AS activtity_rank, IFNULL( unclean_register_activities.activity_datetime , email_verified_activities.activity_datetime ) AS activity_datetime FROM @@ -310,24 +319,30 @@ union_all_activities_flag_migrated_company AS ( --FINAL -sem_exchange__company_funnels AS ( +sem_exchange__company_conversion_funnels AS ( SELECT company_id, is_data_cleansing_generate_row, is_migrated_company, + activity, + activtity_rank, + IFF( ROW_NUMBER() OVER (PARTITION BY company_id ORDER BY activtity_rank DESC) = 1, + 1, + 0) AS latest_activity, + activity_datetime, - {% for activity in funnel_activity %} + {% for activity_type in funnel_activity %} - count(iff(activity = '{{activity}}', company_id, NULL)) - OVER (ORDER BY activity_datetime) AS cumulative_{{activity}}_activity, + count(iff(activity = '{{activity_type}}', company_id, NULL)) + OVER (ORDER BY activity_datetime) AS cumulative_{{activity_type}}_activity, {% endfor %} - {% for activity in funnel_activity %} + {% for activity_type in funnel_activity %} - count(iff(activity = '{{activity}}' AND is_migrated_company = 0, company_id, NULL)) - OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity}}_activity, + count(iff(activity = '{{activity_type}}' AND is_migrated_company = 0, company_id, NULL)) + OVER (ORDER BY activity_datetime) AS exclude_migrated_cumulative_{{activity_type}}_activity, {% endfor %} @@ -338,4 +353,4 @@ sem_exchange__company_funnels AS ( ORDER BY activity_datetime ) -SELECT * FROM sem_exchange__company_funnels \ No newline at end of file +SELECT * FROM sem_exchange__company_conversion_funnels \ No newline at end of file diff --git a/models/exchange/staging/activity_logs/stg_exchange__activity_logs.sql b/models/exchange/staging/activity_logs/stg_exchange__activity_logs.sql deleted file mode 100644 index 3e83c4a..0000000 --- a/models/exchange/staging/activity_logs/stg_exchange__activity_logs.sql +++ /dev/null @@ -1,46 +0,0 @@ --- IMPORTS -WITH activity_logs AS ( - SELECT * FROM {{ ref('base_exchange__activity_logs') }} -), - - --- LOGIC -activity_logs_listout AS ( - - SELECT - activity_logs.id, - activity_logs.log_name, - activity_logs.description, - activity_logs.subject_id, - activity_logs.subject_type, - activity_logs.causer_id, - activity_logs.causer_type, - activity_logs.properties, - activity_logs.created_at, - activity_logs.updated_at, - CURRENT_TIMESTAMP() AS _dbt_ran_at - - FROM activity_logs -), - - --- FINAL -final__stg_exchange__activity_logs AS ( - - SELECT - id, - log_name, - description, - subject_id, - subject_type, - causer_id, - causer_type, - properties, - created_at, - updated_at, - _dbt_ran_at - - FROM activity_logs_listout -) - -SELECT * FROM final__stg_exchange__activity_logs \ No newline at end of file diff --git a/models/exchange/staging/wallet/stg_exchange__wallet.sql b/models/exchange/staging/wallet/stg_exchange__wallet.sql new file mode 100644 index 0000000..60fdb61 --- /dev/null +++ b/models/exchange/staging/wallet/stg_exchange__wallet.sql @@ -0,0 +1,10 @@ +-- IMPORTS +WITH wallets AS ( + SELECT * FROM {{ ref('base_exchange__wallets') }} +) + + + + +-- FINAL +SELECT * FROM wallets \ No newline at end of file diff --git a/models/exchange/staging/wallet_logs/stg_exchange__wallet_logs.sql b/models/exchange/staging/wallet_logs/stg_exchange__wallet_logs.sql new file mode 100644 index 0000000..2e0ad6f --- /dev/null +++ b/models/exchange/staging/wallet_logs/stg_exchange__wallet_logs.sql @@ -0,0 +1,7 @@ +-- IMPORTS +with wallet_logs as (select * from {{ ref("base_exchange__wallet_logs") }}) + +-- FINAL +select * +from wallet_logs +order by wallet_id, updated_at asc diff --git a/models/shipping/staging/stg_shipping__activity_logs.sql b/models/shipping/staging/stg_shipping__activity_logs.sql new file mode 100644 index 0000000..8f30a63 --- /dev/null +++ b/models/shipping/staging/stg_shipping__activity_logs.sql @@ -0,0 +1,46 @@ +-- IMPORTS +WITH activity_logs AS ( + SELECT * FROM {{ source('src_shipping_mysql', 'activity_log') }} +), + + +-- LOGIC +activity_logs_rename AS ( + + SELECT + activity_logs.id, + activity_logs.log_name, + activity_logs.description, + activity_logs.subject_id, + activity_logs.subject_type, + activity_logs.causer_id, + activity_logs.causer_type, + activity_logs.properties, + activity_logs.created_at AS created_datetime, + activity_logs.updated_at AS updated_datetime, + current_timestamp() AS _dbt_ran_at + + FROM activity_logs +), + + +-- FINAL +final__base_exchange__activity_logs AS ( + + SELECT + id, + log_name, + description, + subject_id, + subject_type, + causer_id, + causer_type, + properties, + created_datetime, + updated_datetime, + _dbt_ran_at + + FROM activity_logs_rename +) + +SELECT * FROM final__base_exchange__activity_logs \ No newline at end of file diff --git a/models/staging/autocount/_autocount__sources.yml b/models/staging/autocount/_autocount__sources.yml new file mode 100644 index 0000000..e69de29 diff --git a/models/staging/crisp/_crisp__sources.yml b/models/staging/crisp/_crisp__sources.yml new file mode 100644 index 0000000..e69de29 diff --git a/models/staging/exchange/_exchange__docs.md b/models/staging/exchange/_exchange__docs.md new file mode 100644 index 0000000..ed501d3 --- /dev/null +++ b/models/staging/exchange/_exchange__docs.md @@ -0,0 +1,23 @@ +{% docs src_exchange_mysql %} + +Data is extract from CIEF Selfhost MYSQL `exchange_production`, using custom python code. + +- `_source_loaded_at` will add into the raw data when extract +```YML +loaded_at_field: _source_loaded_at +freshness: + warn_after: {count: 12, period: hour} + error_after: {count: 24, period: hour} +``` + +- The raw data will use copy into snowflake in 2AM everyday + +- Depend on target.name to determin which snowflake raw schema + +```JINJA +{%- if target.name == "prod" -%} PROD_CIEF_RAW_DB +{%- else -%} DEV_CIEF_RAW_DB +{%- endif -%} +``` + +{% enddocs %} \ No newline at end of file diff --git a/models/staging/exchange/_exchange__models.yml b/models/staging/exchange/_exchange__models.yml new file mode 100644 index 0000000..cc0f92b --- /dev/null +++ b/models/staging/exchange/_exchange__models.yml @@ -0,0 +1,6 @@ +version: 2 + +models: + - name: stg_exchange__activity_logs + + - name: stg_exchange__announcments \ No newline at end of file diff --git a/models/staging/exchange/_exchange__sources.yml b/models/staging/exchange/_exchange__sources.yml new file mode 100644 index 0000000..d511863 --- /dev/null +++ b/models/staging/exchange/_exchange__sources.yml @@ -0,0 +1,619 @@ +version: 2 + +sources: + - name: src_exchange_mysql + description: '{{ doc("src_exchange_mysql") }}' + database: | + {%- if target.name == "prod" -%} PROD_CIEF_RAW_DB + {%- else -%} DEV_CIEF_RAW_DB + {%- endif -%} + schema: EXCHANGE_MYSQL + loader: custom_python_code + loaded_at_field: _source_loaded_at + freshness: + warn_after: {count: 12, period: hour} + error_after: {count: 24, period: hour} + meta: + owner: "@yam" + model_maturity: prod + tags: + - exchange + - daily + + + tables: + - name: activity_log + description: Contains most table's activity log that CUD (created, updated, deleted) in www.exchange.cief-malaysia.com + columns: + - name: id + description: Primary key for 'activity_log' + tests: + - unique + - not_null + + - name: description + description: system CUD activity list down + tests: + - accepted_values: + values: ['created', 'updated', 'deleted'] + + - name: subject_id + description: The exchange CUD table's id, need to refer subject_type column to know what table is CUD in the row + tests: + - not_null + + - name: subject_type + description: What table that CUD using in the row + tests: + - not_null + + - name: causer_id + description: Need to refer causer_type column to know who is CUD in the row. it is not well maintain there is some NULL + + - name: causer_type + description: Who is CUD the row + + - name: properties + description: Json data format that contain attributes of the activity, it is not well maintain, subject (Country, ServiceType, CurrencyRate. UserEmailVerification, File) have properties only + + - name: created_at + description: Datetime that CUD activity occur + tests: + - not_null + + - name: updated_at + description: Exactly same with created_at, redundant column + + tests: + - dbt_expectations.expect_column_pair_values_to_be_equal: + column_A: updated_at + column_B: created_at + + + - name: addresses + description: Addresses that input when user register + columns: + - name: id + description: Primary key for 'addresses' + tests: + - unique + - not_null + + - name: company_id + description: Foreign keys for 'companies' + tests: + - not_null + + - name: country_id + description: Foreign keys for 'countries' + tests: + - not_null + + - name: state_id + description: Foreign keys for 'states' + tests: + - not_null + + - name: district_id + description: Foreign keys for 'districts' + tests: + - not_null + + - name: postcode + description: The number/string that user input + tests: + - not_null + - dbt_expectations.expect_column_values_to_be_of_type: + column_type: integer + + - name: street_one + description: The string that user input + tests: + - not_null + + - name: street_two + description: The string that user input + + - name: billing + description: If mark 1 means it is billing address, 0 means is not billing address, every company should have 1 billing address + tests: + - dbt_expectations.expect_column_max_to_be_between: #Every company should have a billing address + min_value: 1 + max_value: 1 + group_by: [company_id] + - dbt_expectations.expect_column_sum_to_be_between: #1 company should have 1 billing address only + min_value: 1 + max_value: 1 + group_by: [company_id] + row_condition: "deleted_at is null" + + - name: deleted_at + description: System generate datetime that address was deleted + + - name: created_at + description: System generate datetime that address was created + tests: + - not_null + + - name: updated_at + description: System generate datetime that address was updated + tests: + - not_null + + + - name: announcement_segment + description: Superadmin / Operator guys will segment the announcment, but normally they will leave it default. 2 FK combine to becoem a relation table. + columns: + - name: announcement_id + description: Primary key for 'announcement_segment' & Foreign key for 'announcements' + tests: + - not_null + + - name: segment_id + description: Primary key for 'announcement_segment' & Foreign key for 'segments' + tests: + - not_null + + - name: created_at + description: Abandon column + tests: + - dbt_expectations.expect_column_values_to_be_null + + - name: updated_at + description: Abandon column + tests: + - dbt_expectations.expect_column_values_to_be_null + + + - name: announcements + description: Announcements that show in our website which create by superadmin + columns: + - name: id + description: Primary key for 'announcements' + tests: + - unique + - not_null + + - name: title + description: title of announcement, using string + tests: + - not_null + + - name: description + description: description of announcement, using longtext, have
html for break + tests: + - not_null + + - name: starting_on + description: announcement starting datetime + tests: + - not_null + + - name: ending_on + description: announcement ending datetime + tests: + - not_null + + - name: is_all_day + description: column might not in use, system auto generate + tests: + - not_null + - accepted_values: + values: [0] + + - name: duration + description: column might not in use, system auto generate + tests: + - not_null + - accepted_values: + values: [0] + + - name: is_recurring + description: column might not in use, system auto generate + tests: + - not_null + - accepted_values: + values: [0] + + - name: recurrence_pattern + description: column might not in use, system auto generate + tests: + - dbt_expectations.expect_column_values_to_be_null + + - name: deleted_at + description: system generate datetime that announcment was deleted + tests: + - not_null + + - name: created_at + description: system generate datetime that announcment was created + tests: + - not_null + + - name: updated_at + description: system generate datetime that announcment was updated + tests: + - not_null + + + - name: bank_logs + description: Abandon table + config: + enabled: false + + + - name: banks + description: Table which store banks details. + columns: + - name: id + description: Primary key for 'banks' + tests: + - unique + - not_null + + - name: company_id + description: Foreign keys for 'companies' + tests: + - not_null + + - name: reference + description: A string which write by user/company for the bank transfer reference or other remind ussage + + - name: holder_name + description: Bank holder name + tests: + - not_null + + - name: account_no + description: Bank account number + tests: + - not_null + + + + + - name: booking_logs + description: Raw `bookings` table from Snowflake + columns: + - name: id + description: Primary key for 'booking_logs' + tests: + - unique + - not_null + + + - name: bookings + description: Raw `bookings` table from Snowflake + columns: + - name: id + description: Primary key for 'bookings' + tests: + - unique + - not_null + + + - name: companies + description: Raw `companies` table from Snowflake + columns: + - name: id + description: Primary key for 'companies' + tests: + - unique + - not_null + + + - name: contacts + description: Raw `contacts` table from Snowflake + columns: + - name: id + description: Primary key for 'contacts' + tests: + - unique + - not_null + + + - name: countries + description: Raw `countries` table from Snowflake + columns: + - name: id + description: Primary key for 'countries' + tests: + - unique + - not_null + + + - name: currencies + description: Raw `currencies` table from Snowflake + columns: + - name: id + description: Primary key for 'currencies' + tests: + - unique + - not_null + + + - name: currency_logs + description: Raw `currency_logs` table from Snowflake + columns: + - name: id + description: Primary key for 'currency_logs' + tests: + - unique + - not_null + + + - name: currency_rate_logs + description: Raw `currency_rate_logs` table from Snowflake + columns: + - name: id + description: Primary key for 'currency_rate_logs' + tests: + - unique + - not_null + + + - name: currency_rates + description: Raw `currency_rates` table from Snowflake + columns: + - name: id + description: Primary key for 'currency_rates' + tests: + - unique + - not_null + + + - name: districts + description: Raw `districts` table from Snowflake + columns: + - name: id + description: Primary key for 'districts' + tests: + - unique + - not_null + + + - name: documents + description: Raw `documents` table from Snowflake + columns: + - name: id + description: Primary key for 'documents' + tests: + - unique + - not_null + + + - name: employees + description: Raw `employees` table from Snowflake + columns: + - name: company_id + description: Primary key for 'employees' + + + - name: failed_jobs + description: Raw `failed_jobs` table from Snowflake + columns: + - name: id + description: Primary key for 'failed_jobs' + tests: + - unique + - not_null + + + - name: files + description: Raw `files` table from Snowflake + columns: + - name: id + description: Primary key for 'files' + tests: + - unique + - not_null + + + - name: group_transactions + description: Raw `group_transactions` table from Snowflake + columns: + - name: id + description: Primary key for 'group_transactions' + tests: + - unique + - not_null + + + - name: groups + description: Raw `groups` table from Snowflake + columns: + - name: id + description: Primary key for 'groups' + tests: + - unique + - not_null + + + - name: jobs + description: Raw `jobs` table from Snowflake + columns: + - name: id + description: Primary key for 'jobs' + tests: + - unique + - not_null + + + - name: migrations + description: Raw `migrations` table from Snowflake + columns: + - name: id + description: Primary key for 'migrations' + tests: + - unique + - not_null + + + - name: model_has_roles + description: Raw `model_has_roles` table from Snowflake + columns: + - name: role_id + description: Primary key for 'model_has_roles' + + + - name: notifications + description: Raw `notifications` table from Snowflake + columns: + - name: id + description: Primary key for 'notifications' + tests: + - unique + - not_null + + + - name: password_resets + description: Raw `password_resets` table from Snowflake + columns: + - name: id + description: Primary key for 'password_resets' + tests: + - unique + - not_null + + + - name: permissions + description: Raw `permissions` table from Snowflake + columns: + - name: id + description: Primary key for 'permissions' + tests: + - unique + - not_null + + + - name: role_has_permissions + description: Raw `role_has_permissions` table from Snowflake + columns: + - name: permission_id + description: Primary key for 'role_has_permissions' + + + - name: roles + description: Raw `roles` table from Snowflake + columns: + - name: id + description: Primary key for 'roles' + tests: + - unique + - not_null + + + - name: segment_companies + description: Raw `segment_companies` table from Snowflake + columns: + - name: segment_id + description: Primary key for 'segment_companies' + + + - name: segment_constants + description: Raw `segment_constants` table from Snowflake + columns: + - name: id + description: Primary key for 'segment_constants' + tests: + - unique + - not_null + + + - name: segments + description: Raw `segments` table from Snowflake + columns: + - name: id + description: Primary key for 'segments' + tests: + - unique + - not_null + + + - name: service_types + description: Raw `service_types` table from Snowflake + columns: + - name: id + description: Primary key for 'service_types' + tests: + - unique + - not_null + + + description: Raw `states` table from Snowflake + columns: + - name: id + description: Primary key for 'states' + tests: + - unique + - not_null + + + - name: transaction_detail + description: Raw `transaction_detail` table from Snowflake + columns: + - name: id + description: Primary key for 'transaction_detail' + tests: + - unique + - not_null + + + - name: transaction_logs + description: Raw `transaction_logs` table from Snowflake + columns: + - name: id + description: Primary key for 'transaction_logs' + tests: + - unique + - not_null + + + - name: transactions + description: Raw `transactions` table from Snowflake + columns: + - name: id + description: Primary key for 'transactions' + tests: + - unique + - not_null + + + - name: user_email_verifications + description: Raw `user_email_verifications` table from Snowflake + columns: + - name: id + description: Primary key for 'user_email_verifications' + tests: + - unique + - not_null + + + - name: users + description: Raw `users` table from Snowflake + columns: + - name: id + description: Primary key for 'users' + tests: + - unique + - not_null + + + - name: wallet_logs + description: Raw `wallet_logs` table from Snowflake + columns: + - name: id + description: Primary key for 'wallet_logs' + tests: + - unique + - not_null + + + - name: wallets + description: Raw `wallets` table from Snowflake + columns: + - name: id + description: Primary key for 'wallets' + tests: + - unique + - not_null \ No newline at end of file diff --git a/models/staging/exchange/stg_exchange__activity_logs.sql b/models/staging/exchange/stg_exchange__activity_logs.sql new file mode 100644 index 0000000..0040c52 --- /dev/null +++ b/models/staging/exchange/stg_exchange__activity_logs.sql @@ -0,0 +1,94 @@ +-- IMPORTS +WITH activity_logs AS ( + SELECT * FROM {{ source('src_exchange_mysql', 'activity_log') }} +), + + +-- LOGIC +activity_logs_rename AS ( + + SELECT + activity_logs.id, + activity_logs.log_name, + activity_logs.description, + activity_logs.subject_id, + activity_logs.subject_type, + activity_logs.causer_id, + activity_logs.causer_type, + activity_logs.properties, + activity_logs.created_at, + activity_logs.updated_at, + current_timestamp() AS _dbt_ran_at + + FROM activity_logs +), + + +-- FINAL +final__base_exchange__activity_logs AS ( + + SELECT + id, + log_name, + description, + subject_id, + subject_type, + causer_id, + causer_type, + properties, + created_at, + updated_at, + _dbt_ran_at + + FROM activity_logs_rename +) + +SELECT * FROM final__base_exchange__activity_logs + + +-- IMPORTS +WITH activity_logs AS ( + SELECT * FROM {{ ref('base_exchange__activity_logs') }} +), + + +-- LOGIC +activity_logs_listout AS ( + + SELECT + activity_logs.id, + activity_logs.log_name, + activity_logs.description, + activity_logs.subject_id, + activity_logs.subject_type, + activity_logs.causer_id, + activity_logs.causer_type, + activity_logs.properties, + activity_logs.created_at, + activity_logs.updated_at, + CURRENT_TIMESTAMP() AS _dbt_ran_at + + FROM activity_logs +), + + +-- FINAL +final__stg_exchange__activity_logs AS ( + + SELECT + id, + log_name, + description, + subject_id, + subject_type, + causer_id, + causer_type, + properties, + created_at, + updated_at, + _dbt_ran_at + + FROM activity_logs_listout +) + +SELECT * FROM final__stg_exchange__activity_logs \ No newline at end of file diff --git a/models/exchange/staging/announcement_segments/stg_exchange__announcement_segments.sql b/models/staging/exchange/stg_exchange__announcement_segments.sql similarity index 100% rename from models/exchange/staging/announcement_segments/stg_exchange__announcement_segments.sql rename to models/staging/exchange/stg_exchange__announcement_segments.sql diff --git a/models/exchange/staging/announcements/stg_exchange__announcements.sql b/models/staging/exchange/stg_exchange__announcements.sql similarity index 100% rename from models/exchange/staging/announcements/stg_exchange__announcements.sql rename to models/staging/exchange/stg_exchange__announcements.sql diff --git a/models/exchange/staging/bank_logs/stg_exchange__bank_logs.sql b/models/staging/exchange/stg_exchange__bank_logs.sql similarity index 100% rename from models/exchange/staging/bank_logs/stg_exchange__bank_logs.sql rename to models/staging/exchange/stg_exchange__bank_logs.sql diff --git a/models/exchange/staging/banks/stg_exchange__banks.sql b/models/staging/exchange/stg_exchange__banks.sql similarity index 100% rename from models/exchange/staging/banks/stg_exchange__banks.sql rename to models/staging/exchange/stg_exchange__banks.sql diff --git a/models/exchange/staging/booking_logs/stg_exchange__booking_logs.sql b/models/staging/exchange/stg_exchange__booking_logs.sql similarity index 100% rename from models/exchange/staging/booking_logs/stg_exchange__booking_logs.sql rename to models/staging/exchange/stg_exchange__booking_logs.sql diff --git a/models/exchange/staging/bookings/stg_exchange__bookings.sql b/models/staging/exchange/stg_exchange__bookings.sql similarity index 100% rename from models/exchange/staging/bookings/stg_exchange__bookings.sql rename to models/staging/exchange/stg_exchange__bookings.sql diff --git a/models/exchange/staging/companies/stg_exchange__companies.sql b/models/staging/exchange/stg_exchange__companies.sql similarity index 100% rename from models/exchange/staging/companies/stg_exchange__companies.sql rename to models/staging/exchange/stg_exchange__companies.sql diff --git a/models/exchange/staging/companies_and_segments_relations/stg_exchange__companies_and_segments_relations.sql b/models/staging/exchange/stg_exchange__companies_and_segments_relations.sql similarity index 100% rename from models/exchange/staging/companies_and_segments_relations/stg_exchange__companies_and_segments_relations.sql rename to models/staging/exchange/stg_exchange__companies_and_segments_relations.sql diff --git a/models/exchange/staging/companies_and_users_relations/stg_exchange__companies_and_users_relations.sql b/models/staging/exchange/stg_exchange__companies_and_users_relations.sql similarity index 100% rename from models/exchange/staging/companies_and_users_relations/stg_exchange__companies_and_users_relations.sql rename to models/staging/exchange/stg_exchange__companies_and_users_relations.sql diff --git a/models/exchange/staging/company_addresses/stg_exchange__company_addresses.sql b/models/staging/exchange/stg_exchange__company_addresses.sql similarity index 100% rename from models/exchange/staging/company_addresses/stg_exchange__company_addresses.sql rename to models/staging/exchange/stg_exchange__company_addresses.sql diff --git a/models/exchange/staging/contacts/stg_exchange__contacts.sql b/models/staging/exchange/stg_exchange__contacts.sql similarity index 100% rename from models/exchange/staging/contacts/stg_exchange__contacts.sql rename to models/staging/exchange/stg_exchange__contacts.sql diff --git a/models/exchange/staging/countries/stg_exchange__countries.sql b/models/staging/exchange/stg_exchange__countries.sql similarity index 100% rename from models/exchange/staging/countries/stg_exchange__countries.sql rename to models/staging/exchange/stg_exchange__countries.sql diff --git a/models/exchange/staging/currencies/stg_exchange__currencies.sql b/models/staging/exchange/stg_exchange__currencies.sql similarity index 100% rename from models/exchange/staging/currencies/stg_exchange__currencies.sql rename to models/staging/exchange/stg_exchange__currencies.sql diff --git a/models/exchange/staging/currency_logs/stg_exchange__currency_logs.sql b/models/staging/exchange/stg_exchange__currency_logs.sql similarity index 100% rename from models/exchange/staging/currency_logs/stg_exchange__currency_logs.sql rename to models/staging/exchange/stg_exchange__currency_logs.sql diff --git a/models/exchange/staging/currency_rate_logs/stg_exchange__currency_rate_logs.sql b/models/staging/exchange/stg_exchange__currency_rate_logs.sql similarity index 100% rename from models/exchange/staging/currency_rate_logs/stg_exchange__currency_rate_logs.sql rename to models/staging/exchange/stg_exchange__currency_rate_logs.sql diff --git a/models/exchange/staging/current_currency_rates/stg_exchange__current_currency_rates.sql b/models/staging/exchange/stg_exchange__current_currency_rates.sql similarity index 100% rename from models/exchange/staging/current_currency_rates/stg_exchange__current_currency_rates.sql rename to models/staging/exchange/stg_exchange__current_currency_rates.sql diff --git a/models/exchange/staging/current_wallets/stg_exchange__current_wallets.sql b/models/staging/exchange/stg_exchange__current_wallets.sql similarity index 100% rename from models/exchange/staging/current_wallets/stg_exchange__current_wallets.sql rename to models/staging/exchange/stg_exchange__current_wallets.sql diff --git a/models/exchange/staging/districts/stg_exchange__districts.sql b/models/staging/exchange/stg_exchange__districts.sql similarity index 100% rename from models/exchange/staging/districts/stg_exchange__districts.sql rename to models/staging/exchange/stg_exchange__districts.sql diff --git a/models/exchange/staging/documents/stg_exchange__documents.sql b/models/staging/exchange/stg_exchange__documents.sql similarity index 100% rename from models/exchange/staging/documents/stg_exchange__documents.sql rename to models/staging/exchange/stg_exchange__documents.sql diff --git a/models/exchange/staging/failed_jobs/stg_exchange__failed_jobs.sql b/models/staging/exchange/stg_exchange__failed_jobs.sql similarity index 100% rename from models/exchange/staging/failed_jobs/stg_exchange__failed_jobs.sql rename to models/staging/exchange/stg_exchange__failed_jobs.sql diff --git a/models/exchange/staging/files/stg_exchange__files.sql b/models/staging/exchange/stg_exchange__files.sql similarity index 100% rename from models/exchange/staging/files/stg_exchange__files.sql rename to models/staging/exchange/stg_exchange__files.sql diff --git a/models/exchange/staging/group_costs/stg_exchange__group_costs.sql b/models/staging/exchange/stg_exchange__group_costs.sql similarity index 100% rename from models/exchange/staging/group_costs/stg_exchange__group_costs.sql rename to models/staging/exchange/stg_exchange__group_costs.sql diff --git a/models/exchange/staging/jobs/stg_exchange__jobs.sql b/models/staging/exchange/stg_exchange__jobs.sql similarity index 100% rename from models/exchange/staging/jobs/stg_exchange__jobs.sql rename to models/staging/exchange/stg_exchange__jobs.sql diff --git a/models/exchange/staging/migrations/stg_exchange__migrations.sql b/models/staging/exchange/stg_exchange__migrations.sql similarity index 100% rename from models/exchange/staging/migrations/stg_exchange__migrations.sql rename to models/staging/exchange/stg_exchange__migrations.sql diff --git a/models/exchange/staging/model_has_roles/stg_exchange__model_has_roles.sql b/models/staging/exchange/stg_exchange__model_has_roles.sql similarity index 100% rename from models/exchange/staging/model_has_roles/stg_exchange__model_has_roles.sql rename to models/staging/exchange/stg_exchange__model_has_roles.sql diff --git a/models/exchange/staging/notifications/stg_exchange__notifications.sql b/models/staging/exchange/stg_exchange__notifications.sql similarity index 100% rename from models/exchange/staging/notifications/stg_exchange__notifications.sql rename to models/staging/exchange/stg_exchange__notifications.sql diff --git a/models/exchange/staging/password_resets/stg_exchange__password_resets.sql b/models/staging/exchange/stg_exchange__password_resets.sql similarity index 100% rename from models/exchange/staging/password_resets/stg_exchange__password_resets.sql rename to models/staging/exchange/stg_exchange__password_resets.sql diff --git a/models/exchange/staging/permissions/stg_exchange__permissions.sql b/models/staging/exchange/stg_exchange__permissions.sql similarity index 100% rename from models/exchange/staging/permissions/stg_exchange__permissions.sql rename to models/staging/exchange/stg_exchange__permissions.sql diff --git a/models/exchange/staging/role_has_permissions/stg_exchange__role_has_permissions.sql b/models/staging/exchange/stg_exchange__role_has_permissions.sql similarity index 100% rename from models/exchange/staging/role_has_permissions/stg_exchange__role_has_permissions.sql rename to models/staging/exchange/stg_exchange__role_has_permissions.sql diff --git a/models/exchange/staging/roles/stg_exchange__roles.sql b/models/staging/exchange/stg_exchange__roles.sql similarity index 100% rename from models/exchange/staging/roles/stg_exchange__roles.sql rename to models/staging/exchange/stg_exchange__roles.sql diff --git a/models/exchange/staging/segment_constants/stg_exchange__segment_constants.sql b/models/staging/exchange/stg_exchange__segment_constants.sql similarity index 100% rename from models/exchange/staging/segment_constants/stg_exchange__segment_constants.sql rename to models/staging/exchange/stg_exchange__segment_constants.sql diff --git a/models/exchange/staging/segments/stg_exchange__segments.sql b/models/staging/exchange/stg_exchange__segments.sql similarity index 100% rename from models/exchange/staging/segments/stg_exchange__segments.sql rename to models/staging/exchange/stg_exchange__segments.sql diff --git a/models/exchange/staging/service_types/stg_exchange__service_types.sql b/models/staging/exchange/stg_exchange__service_types.sql similarity index 100% rename from models/exchange/staging/service_types/stg_exchange__service_types.sql rename to models/staging/exchange/stg_exchange__service_types.sql diff --git a/models/exchange/staging/states/stg_exchange__states.sql b/models/staging/exchange/stg_exchange__states.sql similarity index 100% rename from models/exchange/staging/states/stg_exchange__states.sql rename to models/staging/exchange/stg_exchange__states.sql diff --git a/models/exchange/staging/transaction_costs/stg_exchange__transaction_costs.sql b/models/staging/exchange/stg_exchange__transaction_costs.sql similarity index 100% rename from models/exchange/staging/transaction_costs/stg_exchange__transaction_costs.sql rename to models/staging/exchange/stg_exchange__transaction_costs.sql diff --git a/models/exchange/staging/transaction_costs_and_group_costs_relations/stg_exchange__transaction_costs_and_group_costs_relations.sql b/models/staging/exchange/stg_exchange__transaction_costs_and_group_costs_relations.sql similarity index 100% rename from models/exchange/staging/transaction_costs_and_group_costs_relations/stg_exchange__transaction_costs_and_group_costs_relations.sql rename to models/staging/exchange/stg_exchange__transaction_costs_and_group_costs_relations.sql diff --git a/models/exchange/staging/transaction_log_costs/stg_exchange__transaction_log_costs.sql b/models/staging/exchange/stg_exchange__transaction_log_costs.sql similarity index 100% rename from models/exchange/staging/transaction_log_costs/stg_exchange__transaction_log_costs.sql rename to models/staging/exchange/stg_exchange__transaction_log_costs.sql diff --git a/models/exchange/staging/transaction_log_orders/stg_exchange__transaction_log_orders.sql b/models/staging/exchange/stg_exchange__transaction_log_orders.sql similarity index 100% rename from models/exchange/staging/transaction_log_orders/stg_exchange__transaction_log_orders.sql rename to models/staging/exchange/stg_exchange__transaction_log_orders.sql diff --git a/models/exchange/staging/transaction_order_details/stg_exchange__transaction_order_details.sql b/models/staging/exchange/stg_exchange__transaction_order_details.sql similarity index 100% rename from models/exchange/staging/transaction_order_details/stg_exchange__transaction_order_details.sql rename to models/staging/exchange/stg_exchange__transaction_order_details.sql diff --git a/models/exchange/staging/transaction_orders/stg_exchange__transaction_orders.sql b/models/staging/exchange/stg_exchange__transaction_orders.sql similarity index 100% rename from models/exchange/staging/transaction_orders/stg_exchange__transaction_orders.sql rename to models/staging/exchange/stg_exchange__transaction_orders.sql diff --git a/models/exchange/staging/transaction_wallets/stg_exchange__transaction_wallets.sql b/models/staging/exchange/stg_exchange__transaction_wallets.sql similarity index 100% rename from models/exchange/staging/transaction_wallets/stg_exchange__transaction_wallets.sql rename to models/staging/exchange/stg_exchange__transaction_wallets.sql diff --git a/models/exchange/staging/transactions/stg_exchange__transactions.sql b/models/staging/exchange/stg_exchange__transactions.sql similarity index 100% rename from models/exchange/staging/transactions/stg_exchange__transactions.sql rename to models/staging/exchange/stg_exchange__transactions.sql diff --git a/models/exchange/staging/user_email_verifications/stg_exchange__user_email_verifications.sql b/models/staging/exchange/stg_exchange__user_email_verifications.sql similarity index 100% rename from models/exchange/staging/user_email_verifications/stg_exchange__user_email_verifications.sql rename to models/staging/exchange/stg_exchange__user_email_verifications.sql diff --git a/models/exchange/staging/users/stg_exchange__users.sql b/models/staging/exchange/stg_exchange__users.sql similarity index 100% rename from models/exchange/staging/users/stg_exchange__users.sql rename to models/staging/exchange/stg_exchange__users.sql diff --git a/models/staging/shipping/_shipping_sources.yml b/models/staging/shipping/_shipping_sources.yml new file mode 100644 index 0000000..e69de29