cief/data-management
1
0
Fork 0
mirror of https://gitlab.com/cief-data/data-management.git synced 2026-08-19 04:13:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

C2M Scrape add

Browse Source
This commit is contained in:
Yam ZhengLim
2024-01-27 14:46:57 +08:00
parent e53a9817a6
commit 4ff602f2c6
9 changed files with 280 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+12
View File
@@ -57,6 +57,7 @@ data-management
├─ .gitignore # Ignore the file to push into gitlab
│
├─ config # Folder store all the config file to use in the code
│ ├─ C2M_CONFIG.json # C2M webscraping configuration
│ ├─ CLOUD-STORAGE-ADMIN-CREDENTIAL.json # Get from GCP Service Account cloud-storage-admin-service-account
│ ├─ CRISP_CONFIG.json # Config for CRISP API key, secret & web-id
│ ├─ EXCHANGE_CONFIG.json # Manually config the MySQL ID, Password, etc...
@@ -64,6 +65,17 @@ data-management
│ ├─ SHIPPING_CONFIG.json # Manually config the MySQL ID, Password, etc...
│ └─ SHIPPING_EXTRACT_SQL.json # SQL use in MYSQL to extract the data we need to save in temp's csv
│
├─ c2m # C2M webscrapping
│ ├─ scrape # All script use to scrape the competitor data is in here
│ │ ├─ c2m_scrape_app.py # C2M webscrap, call in main.py or seperate run it in your local environment
│ │ ├─ c2m_temp_scrape_files # Temp folder to store all the csv or error.png from webscraper (Auto Create)
│ │ │ └─ .gitkeep # Keep empty folder
│ │ └─ local_run_files # Folder to store all the data or error from webscraper which run at local pc
│ │ └─ error # Folder to store error.png from webscraper which run at local pc
│ │ └─ .gitkeep # Keep empty folder
│ └─ load
│ └─ c2m_load_function.py # C2M load code function to import in main.py
│
├─ crisp # CRISP-CRM
│ ├─ extract # All crisp extract needed code is in here
│ │ ├─ crisp_extract_function.py # Crisp extract code function to import in main.py