Randomize user agent

[richardvenneman]

I'm getting the idea that Instagram might not be serving content based on our headless user agent. Investigate if it's possible to randomise the user agent to increase success rate.

• https://github.com/alisterscott/webdriver-user-agent

[mattnwa]

Hey Richard, Great work! I'm working through this precise problem on our daily collection. It seems like my lambda service was working fine using a mobile user agent until this last weekend. Now 6 of 10 calls are redirecting to the home login page.

Maybe Instagram is catching the server IP? I'm not familiar enough to know that proxies would help in this situation but figure maybe?

[richardvenneman]

Hi @mattnwa thanks for the report!

I suspected the same thing. Instagram could be blacklisting or rate-limiting the IP addresses.

I think a proxy-kind of setup could definitely be a solution to this issue. I've also briefly looked at utilising the Tor network, however I not really familiar with both myself either :)

Will have to investigate the possible options. This gem looks pretty interesting: https://github.com/joshweir/scraypa.