mirror of
https://gitlab.com/cief-data/data-management.git
synced 2026-08-19 04:13:57 +00:00
Yam ZhengLim
44 lines
2.3 KiB
Markdown
44 lines
2.3 KiB
Markdown
# CIEF Extract Load
|
|
|
|
### Built With
|
|
|
|
* [Python 3.9.12](https://www.python.org/downloads/)
|
|
* [mysql-connector-python 8.0.30](https://pypi.org/project/mysql-connector-python/)
|
|
* [google-cloud-storage 2.5.0](https://pypi.org/project/google-cloud-storage/)
|
|
|
|
## ✨ Code-base structure
|
|
|
|
### How To Run CMD:
|
|
###### pip3 install -r requirements.txt
|
|
###### python3 main.py
|
|
|
|
```
|
|
data-management
|
|
│
|
|
├─ .gitignore # Ignore the file to push into gitlab
|
|
│
|
|
├─ config # Folder store all the config file to use in the code
|
|
│ ├─ CLOUD-STORAGE-ADMIN-CREDENTIAL.json # Get from GCP Service Account cloud-storage-admin-service-account
|
|
│ ├─ EXCHANGE_CONFIG.json # Manually config the MySQL ID, Password, etc...
|
|
│ └─ EXCHANGE_EXTRACT_SQL.json # SQL use in MYSQL to extract the data we need to save in temp's csv
|
|
│
|
|
├─ exchange # Exchange web application
|
|
│ ├─ extract # All extract needed code is in here
|
|
│ │ ├─ exchange_extract_function.py # Exchange extract code function to import in run_extract_load.py
|
|
│ │ └─ exchange_temp_extract_csv # Temp folder to store all the csv extract from MYSQL
|
|
│ │
|
|
│ └─ load # All load needed code is in here
|
|
│ └─ exchange_load_function.py # Exchange load code function to import in run_extract_load.py
|
|
│
|
|
├─ Dockerfile # Still have bug, under development (23/8/2022)
|
|
├─ general_function.py # All customize general code function to import in run_extract_load.py
|
|
├─ README.md # Project guide
|
|
├─ requirements.txt # Install needed library
|
|
└─ main.py # Main code to run the extract load process
|
|
|
|
```
|
|
|
|
### Manage By
|
|
##### Name:- Francis Yam Zheng Lim
|
|
##### Contact:- 0129747749
|
|
##### Email:- yamzhenglim0427@gmail.com |