Unverified Commit df95d0bc authored by Pushkar Chauhan's avatar Pushkar Chauhan Committed by GitHub
Browse files

Update README.md

added Setup, Documentation, API Reference, Authors
1 merge request!1addition: main_server_code, scripts, docs
Showing with 77 additions and 1 deletion
+77 -1
# python-crawler-quickstart
Python based Web crawler Quick Start Project
Python based Web crawler Quick Start Project.
For Scraping the project uses Selenium & Scrapy framework.
## Setup
* Clone this repositary
```
git clone "https://github.com/dileep-gadiraju/python-webscraping-quickstart"
```
* After cloning, Install python packages by running the following command from `./src`.
```
pip install -r "requirements.txt"
```
* Start ElasticSearch,Kibana services as docker-containers.
(refer: https://www.elastic.co/guide/en/kibana/current/docker.html)
* Import API-collections from `./test` for REST client tool.
* Set required global variables
* Run below command from `./src` to start the Server.
```
python app.py
```
Successful local deployment should show Server is up on port 5001.
## Documentation
For Scripting and configuration documentation, refer `./docs` folder
## API Reference
#### Get all Agents
```
GET /general/agents
```
_No paramenters Required_
#### Start a Scraping Job
```
POST /general/run
```
_The following are mandatory Request Body Parameters_
| Parameter | Type | Description |
| :-------- | :------- | :-------------------------------- |
| `agentId` | `string` | `Valid AGENT-ID` |
| `type` | `string` | `Valid Type Of JOB` |
| `search` | `string` | `my search query` |
#### Get Job Status
```
GET /general/status
```
| Parameter | Type | Description |
| :-------- | :------- | :-------------------------------- |
| `JobId` | `string` | `(required) uuid of a job` |
## Authors
- [@dileep-gadiraju](https://github.com/dileep-gadiraju)
- [@Pushkar-Chauhan](https://github.com/Pushkar191098)
- [@dhiru579](https://github.com/dhiru579)
- [@ArchakGAmruth](https://github.com/ArchakGAmruth)
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment