mirror of
https://gitlab.com/cief-data/data-management.git
synced 2026-08-19 04:13:57 +00:00
Yam ZhengLim
78 lines
3.2 KiB
Markdown
78 lines
3.2 KiB
Markdown
# CIEF Extract Load
|
|
|
|
### Built With
|
|
|
|
* [Python 3.9.12](https://www.python.org/downloads/)
|
|
* [mysql-connector-python 8.0.30](https://pypi.org/project/mysql-connector-python/)
|
|
* [google-cloud-storage 2.5.0](https://pypi.org/project/google-cloud-storage/)
|
|
|
|
## ✨ Code-base structure
|
|
|
|
### How To Run CMD:
|
|
```
|
|
###### pip3 install -r requirements.txt
|
|
###### Create "EXCHANGE_CONFIG.json" & "EXCHANGE_EXTRACT_SQL.json" in the config file and input the config value
|
|
###### python3 main.py
|
|
```
|
|
|
|
### Some VM Tips
|
|
```
|
|
###### Host: data-managmenet project
|
|
###### Image: Debian 11
|
|
###### Region: US
|
|
###### Directory: /home/cief_data/business_intelligence/data-management
|
|
###### Crontab config: crontab -e
|
|
###### Crontab log: grep CRON /var/log/syslog
|
|
###### Python log: grep 'mycmd' /var/log/syslog
|
|
###### Remove file cmd: rm -rf
|
|
```
|
|
|
|
###### Example:- EXCHANGE_CONFIG.json
|
|
```
|
|
{
|
|
"host" : "127.0.0.1",
|
|
"user" : "admin",
|
|
"password" : "admin",
|
|
"database" : "classicmodels"
|
|
}
|
|
```
|
|
###### Example:- EXCHANGE_EXTRACT_SQL.json
|
|
```
|
|
{
|
|
"customers" : "SELECT *, now() AS '_source_loaded_at' FROM customers;",
|
|
"employees" : "SELECT *, now() AS '_source_loaded_at' FROM employees;",
|
|
"offices" : "SELECT *, now() AS '_source_loaded_at' FROM offices;"
|
|
}
|
|
```
|
|
### Project Tree
|
|
|
|
```
|
|
data-management
|
|
│
|
|
├─ .gitignore # Ignore the file to push into gitlab
|
|
│
|
|
├─ config # Folder store all the config file to use in the code
|
|
│ ├─ CLOUD-STORAGE-ADMIN-CREDENTIAL.json # Get from GCP Service Account cloud-storage-admin-service-account
|
|
│ ├─ EXCHANGE_CONFIG.json # Manually config the MySQL ID, Password, etc...
|
|
│ └─ EXCHANGE_EXTRACT_SQL.json # SQL use in MYSQL to extract the data we need to save in temp's csv
|
|
│
|
|
├─ exchange # Exchange web application
|
|
│ ├─ extract # All extract needed code is in here
|
|
│ │ ├─ exchange_extract_function.py # Exchange extract code function to import in run_extract_load.py
|
|
│ │ └─ exchange_temp_extract_csv # Temp folder to store all the csv extract from MYSQL (Auto Create)
|
|
│ │ └─ .gitkeep # Keep empty folder
|
|
│ │
|
|
│ └─ load # All load needed code is in here
|
|
│ └─ exchange_load_function.py # Exchange load code function to import in run_extract_load.py
|
|
│
|
|
├─ general_function.py # All customize general code function to import in run_extract_load.py
|
|
├─ README.md # Project guide
|
|
├─ requirements.txt # Install needed library
|
|
└─ main.py # Main code to run the extract load process
|
|
|
|
```
|
|
|
|
### Manage By
|
|
##### Name:- Francis Yam Zheng Lim
|
|
##### Contact:- 0129747749
|
|
##### Email:- yamzhenglim0427@gmail.com |