Documentation
Authentication
You need to pass a bearer token authorization header with every request. Please create the token in your
profile. You can regenerate your token. If you have changed your plan, please make sure you use the newly created token.
-H 'Authorization: Bearer YOUR_API_TOKEN'
Basic request
For a basic request all you have to do is pass the url you want to scrape as a query param:
curl -X GET 'https://api.scrapethat.site/?url=https://en.wikipedia.org/wiki/application_programming_interface'
-H 'Authorization: Bearer YOUR_API_TOKEN'
-H 'Authorization: Bearer YOUR_API_TOKEN'
You'll get a JSON response with three properties:
Depending on your website they will be populated differently. The 'meta' property for example will most likely always be populated. The other two depend on weather the website has a jsonld or itemscop/mircodata tags and which tags are being used. No matter what is used on the site, the object returned will be standardized within each property.
If you didn't get the result your where looking for please check the options below.
{jsonld:{}, mircodata:{}, meta:{}}.Depending on your website they will be populated differently. The 'meta' property for example will most likely always be populated. The other two depend on weather the website has a jsonld or itemscop/mircodata tags and which tags are being used. No matter what is used on the site, the object returned will be standardized within each property.
If you didn't get the result your where looking for please check the options below.
Options
The following options are optional query params you can pass on your request. If it's a boolean param your can either set
&example=true or just add
&example.| Param | Default | Description |
|---|---|---|
render_js | false | Sometimes websites are Javascript rendered and only fully appear when loaded via a browser. Try this option if you get an empty result with the normal request. Since this request is more ressource intensive it's more limited. |
keep_params | false | By default we'll strip the query params from the url. This will cause unexpected results in some cases. Set to true if zou need the query params to get the correct website result. |
jsonld | false | Set to true if you only want the 'jsonld' property in the result. This can speed up the response time and can be used in conjunction with 'microdata' and 'meta'. |
microdata | false | Set to true if you only want the 'microdata' property in the result. This can speed up the response time and can be used in conjunction with 'jsonld' and 'meta'. |
meta | false | Set to true if you only want the 'meta' property in the result. This can speed up the response time and can be used in conjunction with 'jsonld' and 'microdata'. |
no_parse | false | You can also skip parsing all together and just request the html body by setting this to true. |