We have an issue while developing our new site. Our SEO expert strongly advises to use htpasswd protection. Typical crawl disallows were too weak and only a recommendation for crawlers. Hence, we established a htpasswd – but we had to lift it recently, as Gumlet needed access to send image files.
And as things that can happen will happen, we leaked a link to our staging site to customers. Of course, they checked out the still unofficial site. Maybe someone even shared the link on social media? While our site still isn’t indexed yet – we saw hundreds of visitors – some people even placed orders.
We are worried that continuous hits attract crawlers so that they index our inventory that is still on a staging domain (with the production site still in place). This would cause quite some damage.
Long story short – could you guys share the IP range you use for Gumlet? That way, we can put up the password protection again and only whitelist your service. I’d be happy for a quick reaction – maybe you should share them in the docs anyway?