That's not a use case I've put any thought into, honestly, but if I assume the "good spiders" are well-behaved, you should be able to identify them based on their user agent string. Or perhaps you can nail down what IP ranges the good spiders are coming from and allow them based on that.