cief/data-management
1
0
Fork 0
mirror of https://gitlab.com/cief-data/data-management.git synced 2026-08-19 04:13:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Readme update

Browse Source
This commit is contained in:
Yam ZhengLim
2022-09-26 19:03:10 +08:00
parent 471fb3a53a
commit cd06db9aeb
1 changed files with 14 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+14 -1
View File
@@ -14,6 +14,19 @@
###### Create "EXCHANGE_CONFIG.json" & "EXCHANGE_EXTRACT_SQL.json" in the config file and input the config value ###### Create "EXCHANGE_CONFIG.json" & "EXCHANGE_EXTRACT_SQL.json" in the config file and input the config value
###### python3 main.py ###### python3 main.py
``` ```
### Some VM Tips
```
###### Host: data-managmenet project
###### Image: Debian 11
###### Region: US
###### Directory: /home/cief_data/business_intelligence/data-management
###### Crontab config: crontab -e
###### Crontab log: grep CRON /var/log/syslog
###### Python log: grep 'mycmd' /var/log/syslog
###### Remove file cmd: rm -rf
```
###### Example:- EXCHANGE_CONFIG.json ###### Example:- EXCHANGE_CONFIG.json
``` ```
{ {
@@ -52,11 +65,11 @@ data-management
│ ├─ extract # All extract needed code is in here │ ├─ extract # All extract needed code is in here
│ │ ├─ exchange_extract_function.py # Exchange extract code function to import in run_extract_load.py │ │ ├─ exchange_extract_function.py # Exchange extract code function to import in run_extract_load.py
│ │ └─ exchange_temp_extract_csv # Temp folder to store all the csv extract from MYSQL (Auto Create) │ │ └─ exchange_temp_extract_csv # Temp folder to store all the csv extract from MYSQL (Auto Create)
│ │ └─ .gitkeep # Keep empty folder
│ │ │ │
│ └─ load # All load needed code is in here │ └─ load # All load needed code is in here
│ └─ exchange_load_function.py # Exchange load code function to import in run_extract_load.py │ └─ exchange_load_function.py # Exchange load code function to import in run_extract_load.py
│ │
├─ Dockerfile # Still have bug, under development (23/8/2022)
├─ general_function.py # All customize general code function to import in run_extract_load.py ├─ general_function.py # All customize general code function to import in run_extract_load.py
├─ README.md # Project guide ├─ README.md # Project guide
├─ requirements.txt # Install needed library ├─ requirements.txt # Install needed library