cief/data-management
1
0
Fork 0
mirror of https://gitlab.com/cief-data/data-management.git synced 2026-08-19 04:13:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update documentation

Browse Source
This commit is contained in:
Yam ZhengLim
2022-12-06 22:34:32 +08:00
parent f60b4f25a0
commit 7adb86fab9
3 changed files with 27 additions and 1813 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+27 -5
View File
@@ -5,13 +5,14 @@
* [Python 3.9.12](https://www.python.org/downloads/)
* [mysql-connector-python 8.0.29](https://pypi.org/project/mysql-connector-python/)
* [google-cloud-storage 2.5.0](https://pypi.org/project/google-cloud-storage/)
* [crisp-api 1.1.13](https://github.com/crisp-im/python-crisp-api)
* [pandas 1.5.2](https://pandas.pydata.org/)
## ✨ Code-base structure
### How To Run CMD:
```
###### pip3 install -r requirements.txt
###### Create "EXCHANGE_CONFIG.json" & "EXCHANGE_EXTRACT_SQL.json" in the config file and input the config value
###### python3 main.py
```
@@ -27,6 +28,7 @@
###### Python log: grep 'mycmd' /var/log/syslog
###### Remove file cmd: rm -rf
###### git clone https://gitlab.com/CIEFWorldwideSdnBhd/data-management.git
###### pip3 install -r requirements.txt
```
###### Example:- EXCHANGE_CONFIG.json
@@ -55,19 +57,39 @@ data-management
│
├─ config # Folder store all the config file to use in the code
│ ├─ CLOUD-STORAGE-ADMIN-CREDENTIAL.json # Get from GCP Service Account cloud-storage-admin-service-account
│ ├─ CRISP_CONFIG.json # Config for CRISP API key, secret & web-id
│ ├─ EXCHANGE_CONFIG.json # Manually config the MySQL ID, Password, etc...
│ └─ EXCHANGE_EXTRACT_SQL.json # SQL use in MYSQL to extract the data we need to save in temp's csv
│ ├─ EXCHANGE_EXTRACT_SQL.json # SQL use in MYSQL to extract the data we need to save in temp's csv
│ ├─ SHIPPING_CONFIG.json # Manually config the MySQL ID, Password, etc...
│ └─ SHIPPING_EXTRACT_SQL.json # SQL use in MYSQL to extract the data we need to save in temp's csv
│
├─ crisp # CRISP-CRM
│ ├─ extract # All crisp extract needed code is in here
│ │ ├─ crisp_extract_function.py # Crisp extract code function to import in main.py
│ │ └─ crisp_temp_extract_csv # Temp folder to store all the csv extract from API (Auto Create)
│ │ └─ .gitkeep # Keep empty folder
│ └─ load # All load needed code is in here
│ └─ crisp_load_function.py # CRISP load code function to import in main.py
│
├─ exchange # Exchange web application
│ ├─ extract # All extract needed code is in here
│ │ ├─ exchange_extract_function.py # Exchange extract code function to import in run_extract_load.py
│ │ ├─ exchange_extract_function.py # Exchange extract code function to import in main.py
│ │ └─ exchange_temp_extract_csv # Temp folder to store all the csv extract from MYSQL (Auto Create)
│ │ └─ .gitkeep # Keep empty folder
│ │
│ └─ load # All load needed code is in here
│ └─ exchange_load_function.py # Exchange load code function to import in run_extract_load.py
│ └─ exchange_load_function.py # Exchange load code function to import in main.py
│
├─ general_function.py # All customize general code function to import in run_extract_load.py
├─ shipping # Shipping web application
│ ├─ extract # All extract needed code is in here
│ │ ├─ shipping_extract_function.py # Shipping extract code function to import in main.py
│ │ └─ shipping_temp_extract_csv # Temp folder to store all the csv extract from MYSQL (Auto Create)
│ │ └─ .gitkeep # Keep empty folder
│ │
│ └─ load # All load needed code is in here
│ └─ Ssipping_load_function.py # Shipping load code function to import in main.py
│
├─ general_function.py # All customize general code function to import in main.py
├─ README.md # Project guide
├─ requirements.txt # Install needed library
└─ main.py # Main code to run the extract load process