> For the complete documentation index, see [llms.txt](https://docs.hypertune.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.hypertune.com/guides/detect-crawlers-and-bots.md).

# Detect crawlers and bots

To detect crawlers or bots, first [pass the user agent to Hypertune](/guides/pass-the-user-agent-to-hypertune.md).

Next, add a rule to your flag that checks whether the user agent matches a crawler or bot using a regular expression:

{% code title="regex" %}

```regex
(?i)(bot|crawler|crawl|spider|slurp|fetch|search|monitor|scraper|python|perl|php|java|wget|curl|httpclient|libwww|bingpreview|mediapartners\-google)
```

{% endcode %}

<figure><img src="/files/ccFuYqZ2HH6a4i4Mw5Xy" alt=""><figcaption></figcaption></figure>

In the example above, the rule runs before the default rule that contains the `Experiment` expression. As a result, crawlers and bots will exit the flag logic immediately. They won’t enter the experiment, which keeps exposures and analytics clean.

For this reason, we recommend keeping all targeting logic within a single flag rather than spreading it across multiple flags or hardcoding it in your application.

Hypertune makes this possible via its flexible configuration language, [Hyperlang](/concepts/logic.md#hyperlang).