CIEF Extract Load

Built With

• Python 3.9.12
• mysql-connector-python 8.0.30
• google-cloud-storage 2.5.0

✨ Code-base structure

How To Run CMD:

###### pip3 install -r requirements.txt
###### Create "EXCHANGE_CONFIG.json" & "EXCHANGE_EXTRACT_SQL.json" in the config file and input the config value
###### python3 main.py

Example:- EXCHANGE_CONFIG.json

{
    "host"      :   "127.0.0.1",
    "user"      :   "admin",
    "password"  :   "admin",
    "database"  :   "classicmodels"
}

Example:- EXCHANGE_EXTRACT_SQL.json

{
    "customers"    :   "SELECT * FROM customers;",
    "employees"    :   "SELECT * FROM employees;",
    "offices"      :   "SELECT * FROM offices;",
    "orderdetails" :   "SELECT * FROM orderdetails;",
    "orders"       :   "SELECT * FROM orders;",
    "payments"     :   "SELECT * FROM payments;",
    "productlines" :   "SELECT * FROM productlines;",
    "products"     :   "SELECT * FROM products;"
}

Project Tree

data-management
│
├─ .gitignore                                               # Ignore the file to push into gitlab
│
├─ config                                                   # Folder store all the config file to use in the code
│  ├─ CLOUD-STORAGE-ADMIN-CREDENTIAL.json                   # Get from GCP Service Account cloud-storage-admin-service-account
│  ├─ EXCHANGE_CONFIG.json                                  # Manually config the MySQL ID, Password, etc... 
│  └─ EXCHANGE_EXTRACT_SQL.json                             # SQL use in MYSQL to extract the data we need to save in temp's csv
│
├─ exchange                                                 # Exchange web application
│  ├─ extract                                               # All extract needed code is in here
│  │  ├─ exchange_extract_function.py                       # Exchange extract code function to import in run_extract_load.py
│  │  └─ exchange_temp_extract_csv                          # Temp folder to store all the csv extract from MYSQL    
│  │
│  └─ load                                                  # All load needed code is in here
│     └─ exchange_load_function.py                          # Exchange load code function to import in run_extract_load.py
│
├─ Dockerfile                                               # Still have bug, under development (23/8/2022)
├─ general_function.py                                      # All customize general code function to import in run_extract_load.py
├─ README.md                                                # Project guide
├─ requirements.txt                                         # Install needed library
└─ main.py                                                  # Main code to run the extract load process

Manage By

Name:- Francis Yam Zheng Lim

Contact:- 0129747749

Email:- yamzhenglim0427@gmail.com