addition: main_server_code, scripts, docs

Created by: Pushkar191098

Addition:

1. main_server
2. scripts
3. docs
4. training content

deploy/dev.env

+BASIC_HTTP_USERNAME=test
+BASIC_HTTP_PASSWORD=generic@123#
+ELASTIC_DB_URL=https://localhost:9200
+BLOB_SAS_TOKEN=XXXXX
+BLOB_ACCOUNT_URL=YYYYY
+BLOB_CONTAINER_NAME=ZZZZZ
+MAX_RUNNING_JOBS=4

deploy/web-scraping.yml

+version: '3.7'
+services:
+  web-scraping-project:
+    deploy:
+      replicas: 1
+      update_config:
+        parallelism: 3
+        delay: 10s
+      restart_policy:
+        condition: on-failure
+    ports:
+      - "5001:5001"
+    env_file:
+    - ./dev.env
+
+    networks:
+      - frontend
+
+networks:
+  frontend:
+    driver: overlay
+    external: true

docs/README.md

+# Configuration README
+
+[Configure config.py](config.md)
+
+[Configure agents](agents.md)
+
+[Configure azure](azure.md)
+
+[Configure Environment Variables](env-variables.md)
+
+[Configure ElasticSearch Log](eslog.md)
+
+[Configure scripts.py](scripts.md)
+
+[docker deployment](docker.md)
\ No newline at end of file

docs/agents.md

+
+# Agent Configurations
+
+To include new agents, add the agent_data to `/static/agents.json`
+
+format: 
+
+```
+    {
+        "agentId": "MY-AGENT-1",
+        "description": "Crawler For my_agent_1",
+        "provider": "AGENT-PROVIDER-X",
+        "URL": "https://www.my-agent.com",
+        "scripts": {
+            "scriptType1": "myAgentScript1",
+            "scriptType2": "myAgentScript2",
+            "scriptType3": "myAgentScript3",
+            ...
+        }
+    }
+```
+
+example: 
+
+```
+    [
+        {
+            "agentId": "APPLIED-SELENIUM",
+            "description": "Crawler For Applied",
+            "provider": "Applied",
+            "URL": "https://www.applied.com",
+            "scripts": {
+                "info": "AppliedSelenium",
+                "pdf": "AppliedSelenium"
+            }
+        },
+        {
+            "agentId": "GRAINGER-SELENIUM",
+            "description": "Crawler For Grainger",
+            "provider": "Grainger",
+            "URL": "https://www.grainger.com",
+            "scripts": {
+                "info": "GraingerSelenium",
+                "pdf": "GraingerSelenium"
+            }
+        }
+    ]
+```
\ No newline at end of file

docs/azure.md

+# Azure
+
+1. Initialize BlobStorage object.
+```
+blob_storage = BlobStorage()
+```
+
+2. Set the folder for storage.
+```
+blob_storage.set_agent_folder(folder_name)
+```
+arguments:
+    
+* folder_name : Name of the folder.
+
+
+3. Upload it to BlobStorage.
+
+```
+b_status, b_str = blob_storage.upload_file(file_name, data, overwrite)
+```
+arguments:
+    
+* file_name : Name of the file.
+* data : data to be uploaded.
+* overwrite : (boolean), flag for overwriting the data to BlobStorage.
+
+return:
+
+* b_status : (boolean), if the data has uploaded or not.
+* b_str : Exception if the data is not uploaded.

docs/config.md

+
+# Configure config.py
+
+* [Server configuration](#Server-configuration)
+* [Agent configuration](#Agent-configuration)
+* [AzureBlob configuration](#AzureBlob-configuration)
+* [ElasticSearch variables](#ElasticSearch-DB-variables)
+* [Logging configuration](#Logging-configuration)
+
+## Server configuration
+
+| Variables             | Type      | Description                       |
+| :--------             | :-------  | :-------------------------        |
+| `SERVER_HOST`         | `string`  |  host for Server                  |
+| `SERVER_PORT`         | `string`  |  port for Server                  |
+| `SERVER_DEBUG`        | `bool`    |  debugging for Server             |
+| `SERVER_CORS`         | `bool`    |  CORS policy for Server           |
+| `SERVER_STATIC_PATH`  | `string`  |  static folder path for Server    |
+| `API_URL_PREFIX`      | `string`  |  url prefix for Server            |
+| `API_MANDATORY_PARAMS`| `list`    |  mandatory parameters for request |
+| `BASIC_HTTP_USERNAME` | `string`  |  username to access Server        |
+| `BASIC_HTTP_PASSWORD` | `string`  |  password to access Server        |
+
+## Agent configuration
+
+| Variables              | Type      | Description                                     |
+| :--------              | :-------  | :-------------------------                      |
+| `AGENT_SCRIPT_TYPES`   | `dict`    |  types of scraping_scripts                      |
+| `AGENT_CONFIG_PATH`    | `string`  |  file_path for agent_configuration(json file)   |
+| `AGENT_CONFIG_PKL_PATH`| `string`  |  file_path for agent_configuration(pickle file) |
+
+## AzureBlob configuration
+
+| Variables             | Type     | Description                       |
+| :--------             | :------- | :-------------------------        |
+| `BLOB_INTIGRATION`    | `bool`   |  enable/disable AzureBlob Storage |
+| `BLOB_SAS_TOKEN`      | `string` |  SAS Token for AzureBlob Storage  |
+| `BLOB_ACCOUNT_URL`    | `string` |  Account URL for AzureBlob Storage|
+| `BLOB_CONTAINER_NAME` | `string` |  Container for AzureBlob Storage  |
+
+## ElasticSearch DB variables
+
+| Variables        | Type     | Description                          |
+| :--------        | :------- | :-------------------------           |
+| `ELASTIC_DB_URL` | `string` |  URL of ElasticSearch Server         |
+| `ES_LOG_INDEX`   | `string` |  Info Logging Index in ElasticSearch |
+| `ES_JOB_INDEX`   | `string` |  Job  Logging Index in ElasticSearch |
+| `ES_DATA_INDEX`  | `string` |  Data Logging Index in ElasticSearch |
+
+## Logging configuration
+
+| Variables                     | Type     | Description                    |
+| :--------                     | :------- | :-------------------------     |
+| `JOB_OUTPUT_PATH`             | `string` |  folder_path for JOB output    |
+| `MAX_RUNNING_JOBS`            | `int`    |  Max No. of Running Jobs       |
+| `MAX_WAITING_JOBS`            | `int`    |  Max No. of Waiting Jobs       |
+| `JOB_RUNNING_STATUS`          | `string` |  Status for Running Jobs       |
+| `JOB_COMPLETED_SUCCESS_STATUS`| `string` |  Status for Successfull Jobs   |
+| `JOB_COMPLETED_FAILED_STATUS` | `string` |  Status for Failed Jobs        |

docs/docker.md

+# Docker Deployment
+
+* Stop and remove existing containers with name `web-scraping-project`.
+```
+docker stop web-scraping-project 
+docker rm web-scraping-project
+```
+
+* Build Docker image: `web-scraping-project`
+```
+docker build -t web-scraping-project ./src/
+```
+_Note: ./src/ contains Dockerfile_
+
+
+* Spawn: `web-scraping-project`.
+```
+docker run --name web-scraping-project -p 5001:5001 --env-file ./deploy/dev.env -it web-scraping-project
+```
+
+_Note: Here environment file (--env-file) refers from local storage_
\ No newline at end of file

docs/env-variables.md

+
+# Environment-Variables
+
+The Following are supported Environment-Variables
+
+| Variables             | Type      | Description                       |
+| :--------             | :-------  | :-------------------------        |
+| `BASIC_HTTP_USERNAME` | `string`  |  username for server              |
+| `BASIC_HTTP_PASSWORD` | `string`  |  password for server              |
+| `ELASTIC_DB_URL`      | `string`  |  URL of elasticsearch_DB          |
+| `BLOB_SAS_TOKEN`      | `string`  |  azure blob_storage SAS token     |
+| `BLOB_ACCOUNT_URL`    | `string`  |  azure blob_storage account_URL   |
+| `BLOB_CONTAINER_NAME` | `string`  |  azure blob_storage container_name|
+| `MAX_RUNNING_JOBS`    | `int`     |  maximum jobs running at a time   |
+| `MAX_WAITING_JOBS`    | `int`     |  maximum jobs waiting at a time   |

docs/eslog.md

+
+# ElasticSearch Log
+
+* Initialize Log object.
+```
+log = Log(agentRunContext)
+```
+
+* Types of logs:
+    
+    1. log.job : it shows the job status, logs are added to `config.ES_JOB_INDEX`.
+        
+        Syntax:
+        ```
+        log.job(status, message)
+        ```
+        
+        Examples:
+        ```
+        log.job(config.JOB_RUNNING_STATUS, 'job Started')
+        # your code goes here
+        try:
+            log.job(config.JOB_COMPLETED_SUCCESS_STATUS, 'Job Completed')
+        except:
+            log.job(config.JOB_COMPLETED_FAILED_STATUS, 'Job Failed')
+        ```
+
+    2. log.info : it shows the job info, logs are added to `config.ES_LOG_INDEX`.
+
+        Syntax:
+        ```
+        log.info(info_type, message)
+        ```
+        Examples:
+        ```
+        log.info('info', 'This is generalization project')
+        log.info('warning', 'Script is taking more than usual time')
+        log.info('exception', 'No Products Available')
+        ```
+    3. log.data : it shows the job data, logs are added to `config.ES_DATA_INDEX`.
+        
+        Syntax:
+        ```
+        log.data(data)
+        ```
+        Example:
+        ```
+        data = {
+            "A" : "123",
+            "B" : "Generic Project"
+        }
+        log.data(data)
+        ```

docs/scripts.md

+
+# Scripts
+
+1. Create `python_file` in the respective scriptType folder in `./src/scripts`.
+
+2. Format of the script `my_agent_script.py`.
+```
+# imports
+
+# create a function
+def myAgentScript(agentRunContext):
+    log = Log(agentRunContext)
+    try:
+    
+        log.job(config.JOB_RUNNING_STATUS, Job Started')
+
+        # Your script
+        # Goes here
+
+        log.job(config.JOB_COMPLETED_SUCCESS_STATUS, Successfully Scraped Dats')
+
+    except Exception as e:
+        log.job(config.JOB_COMPLETED_FAILED_STATUS, str(e))
+        log.info('exception', traceback.format_exc())
+
+```
+3. Add script to `init.py` as 
+```
+from .my_agent_script import myAgentScript
+```
+
+

src/common/__init__.py

+from .scraping_utils import get_driver
+from .elastic_wrapper import Log
+from .errors import ValueMissing, FormatError, BadRequestError
+from .blob_storage import BlobStorage