Detect crawlers and bots

To detect crawlers or bots, first pass the user agent to Hypertune.

Next, add a rule to your flag that checks whether the user agent matches a crawler or bot using a regular expression:

regex

(?i)(bot|crawler|crawl|spider|slurp|fetch|search|monitor|scraper|python|perl|php|java|wget|curl|httpclient|libwww|bingpreview|mediapartners\-google)

In the example above, the rule runs before the default rule that contains the Experiment expression. As a result, crawlers and bots will exit the flag logic immediately. They won’t enter the experiment, which keeps exposures and analytics clean.

For this reason, we recommend keeping all targeting logic within a single flag rather than spreading it across multiple flags or hardcoding it in your application.

Hypertune makes this possible via its flexible configuration language, Hyperlang.