POST
/
v1
/
crawl
{
  "url": "https://example.com",
  "options": {
    "max_pages": 10,
    "max_depth": 3,
    "timeout": 30000,
    "scroll": true,
    "screenshot": false,
    "actions": [
      {
        "type": "scroll",
        "duration": 1000
      }
    ]
  }
}

{
  "success": true,
  "data": {
    "pages": [
      {
        "url": "https://example.com",
        "content": "Extracted content...",
        "metadata": {
          "depth": 0,
          "timestamp": "2024-03-19T12:00:00Z"
        }
      }
    ],
    "metadata": {
      "total_pages": 1,
      "max_depth_reached": 0,
      "duration": 1200
    }
  }
}
The Crawl endpoint allows you to crawl an entire website with advanced options. It follows links and extracts content from multiple pages according to your specified parameters.

​
Authentication

​
Authorization
string
required
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

​
Body Parameters

​
url
string
required
The URL of the website to crawl. Must be a valid URI with maximum length of 2083 characters.
​
options
object

​
Response

​
success
boolean
Indicates if the crawl was successful
​
data
object
The crawled data from the website

​
Error Responses

​
error
object
{
  "url": "https://example.com",
  "options": {
    "max_pages": 10,
    "max_depth": 3,
    "timeout": 30000,
    "scroll": true,
    "screenshot": false,
    "actions": [
      {
        "type": "scroll",
        "duration": 1000
      }
    ]
  }
}

{
  "success": true,
  "data": {
    "pages": [
      {
        "url": "https://example.com",
        "content": "Extracted content...",
        "metadata": {
          "depth": 0,
          "timestamp": "2024-03-19T12:00:00Z"
        }
      }
    ],
    "metadata": {
      "total_pages": 1,
      "max_depth_reached": 0,
      "duration": 1200
    }
  }
}

Authorizations

​
Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

Response

200
application/json

Successful Response

The response is of type object.