{% docs macro_address_to_coordinates %}

# Procedure to define E-UDF in GCP and calling the function in DBT
## 1.0 Information required to create E-UDF
As you complete the tasks to create an external function in the Google Cloud Console, you are required to record specific values (e.g. Cloud Function Trigger URL) during each step in the process. Whereby, the values you entered, will be required in subsequent steps.

To facilitate recording/tracking of this information, we have provided a template with fields for each of the required values. Save the template in a text editor for easier referencing.

Reference Source: [Snowflake Planning](https://docs.snowflake.com/en/sql-reference/external-functions-creating-gcp-planning)



```
=================== Tracking Worksehet: Google Cloud Console ===================


Step 1: Cloud Function (Remote Service) Info

Cloud Function Trigger URL: ____________________________________________________


Step 2: API Config File Info 

Path Suffix: ___________________________________________________________________

Configuration File Name: _______________________________________________________


Step 3: API Gateway (Proxy Service) Info

Managed Service Identifier: ____________________________________________________

Gateway Base URL : _____________________________________________________________


Steps 4: API Integration & External Function Info

API Integration Name: __________________________________________________________

API_GCP_SERVICE_ACCOUNT: _______________________________________________________

External Function Name: ________________________________________________________


Step 5: Security Info 

Security Definition Name: ______________________________________________________
```
<br>

## 2.0 Config E-UDF in GCP & Snowflake
### Step 1. Create the Cloud Function in GCP
Reference Source: [Snowflake Create the Remote Service](https://docs.snowflake.com/en/sql-reference/external-functions-creating-gcp-ui-remote-service)

1. Create function from **GCP Cloud functions service**
    
    🔍 GCP Search: Cloud Function

2. Configuration to set up the cloud function are as followed:
    - **Basics**
        - Environment: 1st gen
        - Function name: `<function_name_of_your_choice>`
        - Region: us-central1

    - **Trigger**
        - Trigger type: HTTP
        - Authentication: Require authentication

    - **Runtime**
        - Memory allocated: 256MB
        - Timeout: 240
3. Save the configuration.
4. Proceed by clicking next and work on inputing the function program in the window.
5. Deploy the function when ready.
6. Under Google Cloud Function UI, select the deployed function and select the trigger tab.
7. Copy the trigger URL of the function and record into the template field <mark>Step 1: Cloud function trigger URL</mark>. For example: `https://us-central1-data-management-123456.cloudfunctions.net/address_to_geo_coordinate`

<br>

### Step 2. Create Proxy Service in GCP
Reference Source: [Snowflake Create the Proxy Service](https://docs.snowflake.com/en/sql-reference/external-functions-creating-gcp-ui-proxy-service)

1. On your local file system (desktop), create a YAML config file with the `unique name` of your choice and end with the extension `.yml`. For example, `config_address_to_geo_coordinate.yml`
2. Copy the following configuration template into the YAML file.

    ```yml
    swagger: '2.0'
    info:
    title: API Gateway config for Snowflake external function.
    description: This configuration file connects the API Gateway resource to the remote service (Cloud Function).
    version: 1.0.0
    schemes:
    - https
    produces:
    - application/json
    paths:
    /<PATH>:
        post:
        summary: Echo the input.
        operationId: echo
        x-google-backend:
            address: <HTTP ENDPOINT TO ROUTE REQUEST TO>
            protocol: h2
        responses:
            '200':
            description: <DESCRIPTION>
            schema:
                type: string
    ```

3. Under paths, replace `<PATH>` with a unique name. For example, `func-address-to-geo`. Do not remove the forward slash.
4. Save the path name into the template field <mark>Step 2: Path Suffix</mark>. For example, `func-address-to-geo`
5. Replace `<HTTP ENDPOINT TO ROUTE REQUEST TO>` with the value previously recorded in the template field <mark>Step 1: Cloud Function Trigger URL</mark> ***The URL should not be enclosed in "quotation marks"***. For example, `https://us-central1-data-management-123456.cloudfunctions.net/address_to_geo_coordinate`
6. Save the file and record the yaml file name into the template field <mark>Step 3: Configuration File Name</mark>. For example, `config_address_to_geo_coordinate.yml`
7. Search for Google Cloud API Gateway service in GCP.

    🔍 GCP Search: API Gateway

8. Create a Gateway and configure as followed:
    - **API**
        - Select an API: Create new API
        - Display Name: `<display_name_of_your_choice>`
        - API ID: `<API_ID_of_your_choice>`
        - Region: us-central1

    - **API Config**
        - Select a Config: Create new API config
        - Upload an API Spec: Upload the config yaml file as configured in previous step from your local machine.
        - Display Name: `<display_name_of_your_choice>`
        - Select a Service Account: App Engine default service account

    - **Gateway details**
        - Display Name: `<display_name_of_your_choice>`
        - Location: us-central1

9. Click create gateway and wait several minutes for it to create.
10. In the API gateway page, copy the value from the managed service for the newly created API. Record this value into the template field <mark>Step 3: Managed Service Identifier</mark>. For example, `api-address-to-geo-coordinates-external-function-123456abcde.apigateway.data-management-123456.cloud.goog`
11. In the API gateway page, select the API and click on the Gateways tab.
12. Copy Gateway URL and record this information into the template file <mark>Step 3: Gateway Base URL</mark>. For example, `https://gateway-address-to-coordinates-12345abc.uc.gateway.dev`

### Step 3. Create API Integration in Snowflake
Reference Source: [Snowflake Create the API Integration](https://docs.snowflake.com/en/sql-reference/external-functions-creating-gcp-common-api-integration)

1. Open a new Snowflake sql worksheet.
2. Ensure current snowflake account has accountadmin privileges.
3. Enter following sql statement template:

```sql
create or replace api integration <integration_name>
    api_provider = google_api_gateway
    google_audience = '<google_audience_claim>'
    api_allowed_prefixes = ('<url>')
    enabled = true;
```

4. Replace `<integration_name>` with `<integration_name_of_your_choice>` and record the newly defined name into the template field <mark>Step 4: API Integration Name</mark>. For example, `gcp_api_integration_address_to_coordinates`
5. Replace `<google_audience_claim>` with value previously recorded from template field <mark>Step 3: Managed Service Identifier</mark>
6. Execute the statement.
7. Run the following sql statement:

    ```sql
    describe integration <newly_defined_integration_name_in_previous_step>;
    ```

8. Record `API_GCP_SERVICE_ACCOUNT` value obtained from the execution into the template field <mark>Step 4: API_GCP_SERVICE_ACCOUNT</mark>. For example, `abc1234567@gcpuscentral1-abc1.iam.gserviceaccount.com`


### Step 4. Create External Function in Snowflake
Reference Source: [Snowflake Create the External Function](https://docs.snowflake.com/en/sql-reference/external-functions-creating-gcp-common-ext-function)

1. In the snowflake sql worksheet, enter the following statement:

    ```sql
    create or replace external function <external_function_name>(<parameters>)
        returns variant
        api_integration = <api_integration_name>
        as '<function_url>';
    ```

2. Replace `<external_function_name>` with a unique function name of your choice. For example, `echo_address_to_coordinates`
3. Replace `<parameters>` based on your external function configuration. For example, `id VARCHAR, full_address VARCHAR`
4. Replace `<api_integration_name>` with the value previously recorded from template field <mark>Step 4: API Integration Name</mark> 
5. Replace `<function_URL>` with the value previously recorded from template field <mark>Step 3: Gateway Base URL</mark> and <mark>Step 2: Path Suffix</mark>. Combine the string by a forward slash ( / ) between them. For example: `'https://gateway-address-to-coordinates-12345abc.uc.gateway.dev/func-address-to-geo'`
6. Execute the statement.

<br>

## Security Policy

<br>

## 3.0 Calling E-UDF Function in Snowflake
1. Calling E-UDF in the schema by the name of the E-UDF defined previously. For example:
```sql
SELECT echo_address_to_coordinates(full_address) FROM address;
```
2. If the E-UDF is not stored in the same directory as your current working directory, you need to specify the database and schema name of the E-UDF. For example:
```sql
SELECT dev_data_warehouse.dev_schema.echo_address_to_coordinates(full_address) FROM address;
```

<br>

## 4.0 Defining and calling macro and global variable for the E-UDF in DBT
### Step 1: Defining macro for geocoding address

This macro converts an address string into a set of geographical coordinates. The macro will filter the rows, excluding masked data rows based on data mask pattern defined in a global variable. It accepts five input parameters representing address components and returns the parsed JSON response from the E-UDF. This set of instructions are for used in dbt.

1. Define a macro in the /macros folder directing towards the E-UDF. Remember to specify the database and schema name that store the E-UDF.

The name of the macro utilized here is address_to_coordinates()

Parameters:

address_line_one: VARCHAR or STRING - Represents ADDRESS_LINE_ONE.

address_line_two: VARCHAR or STRING - Represents ADDRESS_LINE_TWO.

postcode        : VARCHAR or STRING - Represents POSTCODE.

district_name   : VARCHAR or STRING - Represents DISTRICT_NAME.

state_name      : VARCHAR or STRING - Represents STATE_NAME.

```sql
-- Remeber to add { } braces at the start and end of the jinja statement %
-- Add double { } on the following locations
-- Around the global variable var('mask_data_pattern')
%- macro address_to_coordinates(address_line_one, address_line_two, postcode, district_name, state_name) -%
    CASE
        WHEN 
            ( address_line_one IN var('mask_data_pattern') OR address_line_two in var('mask_data_pattern')
            OR
            address_line_one IS NULL AND address_line_two IS NULL ) 
                THEN NULL
        ELSE 
            PARSE_JSON(global_function.eudf.echo_address_to_coordinates(address_line_one, address_line_two, postcode, district_name, state_name))
    END
%- endmacro -%
```

### Step 2: Defining global variable
A global variable can be defined in `dbt_project.yml` file. The variable here is called `mask_data_pattern` which is to store a list of string patterns used for masking data.
```yml
vars:
    mask_data_pattern: ('*MASK*', '****')
```

### Step 3: Implement incremental model
Include a materialize configuration block to the model to specify the model as 'incremental' and input the unique_key value(s).
- Snowflake by default uses the merge method for incremental strategy. Thus this model adopts the default method.
- The unique_key must not have null and duplicates. It is suggested to use the primary key of the table instead.
- One or more unique_key can be selected.
- Add double { } to wrap around the configuration block
```sql
config(
    materialized='incremental',
    unique_key=['COMPANY_ID', 'COMPANY_ADDRESS_ID']
)
```

Inlcude a filter block using is_incremental() function for the model to define how to update the model.

The following example uses the 'UPDATED_DATETIME' as the filter to identify new rows to undergo transformation.
- Add double { } to wrap around the following locations:
- ref('test_address__mock_data')
- this
- Remeber to add { } braces at the start and end of the jinja statement %

```sql
WITH events as (
    SELECT * FROM ref('test_address__mock_data')

    % if is_incremental() %
        WHERE UPDATED_DATETIME > (SELECT max(UPDATED_DATETIME) FROM this )
    %- endif %
),
```

### Step 4: Calling the macro
To call the defined macro, the following syntax can be utilized, take note to **include** the **double curly braces**:
- In addition, underlying values can be derived from the results of the macro. You can access the values in the list by referring to the name of the key.
- Shown in the second code block for example, the latitude and longitude of the location is retrieved and stored in a new column.

```sql
-- Add double { } on the following locations
-- Around the E-UDF address_to_coordinates()
-- Around the ref('test_address__mock_data')
WITH geocoded_address AS (
    SELECT 
        *,
        address_to_coordinates( 'ADDRESS_LINE_ONE', 'ADDRESS_LINE_TWO', 'POSTCODE', 'DISTRICT_NAME', 'STATE_NAME' ) AS FUNCTION_JSON_RETURN

    FROM events
),
json_data_extraction AS (
    SELECT
        FUNCTION_JSON_RETURN:result:geometry:location:lat::float AS LATITUDE,
        FUNCTION_JSON_RETURN:result:geometry:location:lng::float AS LONGITUDE,
        FUNCTION_JSON_RETURN:geocoding_api_returns AS API_GEOCODING_JSON_RETURN,
        FUNCTION_JSON_RETURN:result AS API_PLACE_DETAILS_JSON_RETURN,
        *

    FROM geocoded_address
)
```
## 5.0 Local Variables

Several local variables are defined using Jinja templating:

- `days_of_week`: A list of strings representing the days of the week.
- `masked_address_sql`: A SQL condition statement to check if the address line is masked. This variable also refers to the global variable 'mask_data_pattern'.
- `dummy_value`: A dummy value used for placeholders.
- `dummy_long_string`: A long dummy string used for placeholders.
- `dummy_short_string`: A short dummy string used for placeholders.
- `dummy_phone`: A dummy phone number string used for placeholders.
- `dummy_operating_hours`: A dummy operating hours string used for placeholders.
- `dummy_website`: A dummy website URL string used for placeholders.

## 6.0 CTEs (Common Table Expressions)

`company_addresses_with_row_num` <br>
This CTE assigns a row number to each row in the company_addresses table and filters the data based on the incremental update condition if the job is incremental.

`company_addressess_join_companies_countries_states_districts` <br>
This CTE joins the company_addresses_with_row_num CTE with the countries, states, and districts tables to form a fuller table.

`batched_data_batch` <br>
This CTE splits the data rows into batches for processing. Each batch contains up to 10 rows. This is to avoid timeout and resource exhaustion experienced in snowflake caused by concurrency.

`address_enrich_using_google_api_batch` <br>
This CTE enriches the address data by using the address_to_coordinates E-UDF, which calls the Google Maps API to obtain geocoding and location information for each address in the batch.

`union_all_batch_address_enriched` <br>
This CTE combines the results from all the batches into a single table.

`operation_hours_json` <br>
This CTE extracts the operating hour range for each day of the week from the JSON data returned by the API.

`operation_hour_range_extraction` <br>
This CTE further processes the extracted operating hour ranges and removes the day prefixes.

`operation_hour_to_24_hour_transform` <br>
This CTE converts the operating hour ranges to a 24-hour format.

`special_single_operation_hour_to_24_hour_transform` <br>
This CTE handles cases where there are only single operating hours within a day but do not have an AM/PM indicator and converts them to a 24-hour format.

`special_double_operation_hour_to_24_hour_transform` <br>
This CTE handles cases where there are two operating hours within a day and converts them to a 24-hour format.

`concatenate_operation_hour` <br>
This CTE concatenates the first and second transformed time ranges for each day of the week. For scenarios with only single operating hours within a day, the original time range will be returned.

`google_place_api_details_extraction` <br>
This CTE further extracts potentially valuable information returned by the API and flatten them.

`masked_data_transformation` <br>
This CTE applies transformations to the selected metrics and dimensions. It replaces sensitive information with dummy values if the address is masked.


{% enddocs %}