You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Trying to generate a sitemap for my site, BotMap.io goes to a bunch of urls before switching to facebook, urls all starting with http://www.facebook.com/share.php?u=...
This is surprising, shouldn't you filter out urls unrelated to the domain of the site being crawled ?
I tried to disallow www.facebook.com, but didn't work.
The text was updated successfully, but these errors were encountered:
Trying to generate a sitemap for my site, BotMap.io goes to a bunch of urls before switching to facebook, urls all starting with
http://www.facebook.com/share.php?u=...
This is surprising, shouldn't you filter out urls unrelated to the domain of the site being crawled ?
I tried to disallow www.facebook.com, but didn't work.
The text was updated successfully, but these errors were encountered: