You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Differentiate URLs sent from the crawl API and others sent by users (when its query is not found). In this case, the priority is to download the robots file of the URL sent by the user.
We can have multiple levels of priority to allow the development of more use cases and/or business requirements.
The text was updated successfully, but these errors were encountered:
Differentiate URLs sent from the crawl API and others sent by users (when its query is not found). In this case, the priority is to download the robots file of the URL sent by the user.
We can have multiple levels of priority to allow the development of more use cases and/or business requirements.
The text was updated successfully, but these errors were encountered: