Robots.txt Guide: How Malaysian Businesses Control Google and AI Crawlers

Learn how robots.txt works, how to control Google and AI crawlers like GPTBot, and which mistakes to avoid on your Malaysian business website.

Your robots.txt file is tiny. Yet it decides which bots can crawl your website, including Googlebot and the new wave of AI crawlers. One wrong line can hide your best pages from Google. On the other hand, a smart setup helps search engines focus on the pages that actually bring you leads.

In this guide, you will learn what robots.txt does, how to write one, and how Malaysian businesses can control AI bots like GPTBot. Let’s keep it simple and practical.

Server network cables showing how robots.txt guides crawlers through a website
Photo by Taylor Vick on Unsplash

What Is Robots.txt and Why Does It Matter?

A robots.txt file is a plain text file that sits at the root of your domain. For example, ours lives at creativesite.com.my/robots.txt. It tells crawlers which parts of your site they may visit.

Think of it like the security guard at a condo in Mont Kiara. The guard doesn’t lock every door. Instead, he tells visitors which floors they can enter. Generally, polite visitors follow the rules. Rude ones, though, may ignore them.

That’s an important point. Robots.txt is a set of instructions, not a lock. Google and other major search engines respect it. However, bad bots and scrapers often don’t.

Why Your Business Should Care

Every website has pages that don’t need crawling. Think of admin areas, internal search results, cart pages and filter URLs. When bots waste time on these, they may crawl your key service pages less often.

For a small Malaysian business site, crawl budget is rarely a crisis. Still, a clean robots.txt file keeps things tidy. More importantly, it stops you from blocking pages by accident. We see that mistake in audits more often than you’d think.

How Robots.txt Works: The Basic Rules

Google only reads a handful of fields. According to Google’s robots.txt specification, it supports user-agent, allow, disallow and sitemap. Here’s what each one does:

  • User-agent: names the bot the rules apply to. An asterisk (*) means all bots.
  • Disallow: tells the bot not to crawl a path.
  • Allow: opens up a path inside a blocked folder.
  • Sitemap: points bots to your XML sitemap.

Here’s a simple example for a typical WordPress site:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com.my/sitemap_index.xml

This tells every bot to skip the admin area. At the same time, it keeps admin-ajax.php open, because some themes and plugins need it. Finally, it shows bots where your sitemap lives.

Technical Facts Google Confirms

Google’s documentation spells out a few rules that trip people up:

  • The file must sit in the top-level directory. A file inside a subfolder won’t count.
  • Google reads up to 500 KiB. It ignores anything after that limit.
  • Google usually caches your file for up to 24 hours. So changes don’t apply instantly.
  • Paths are case-sensitive. /Blog/ and /blog/ are different paths.
  • Google supports two wildcards: * for any characters and $ for the end of a URL.
  • Google doesn’t support crawl-delay. Other search engines may still read it.

The Biggest Robots.txt Myth: Blocking Keeps Pages Out of Google

This myth causes real damage. Many site owners think a disallow rule removes a page from Google. It doesn’t.

In its introduction to robots.txt, Google says the file is not a way to keep a page out of its results. A blocked page can still appear in search if other sites link to it. In that case, the result simply shows no description.

What to Use Instead

If you want a page gone from Google, use a noindex tag. Alternatively, protect the page with a password. Here’s the catch: Google must crawl the page to see the noindex tag. So don’t block that same page in robots.txt, or Google will never see your instruction.

For example, a Penang clinic might want to hide its “thank you” page after a booking form. The right move is a noindex tag, not a disallow rule. If your pages are vanishing for other reasons, read our guide on why pages are not indexed on Google.

White robot representing AI crawlers controlled by robots.txt rules
Photo by Alex Knight on Unsplash

How to Control AI Crawlers With Robots.txt

This is where robots.txt gets interesting in 2026. AI companies now run their own crawlers. For instance, some collect content to train models. Others fetch pages to answer questions in AI search tools.

As a result, you need to decide what you want. Do you want visibility in AI answers? Or do you want to keep your content out of AI training? You can often choose one without the other.

Google-Extended

Google-Extended is a special token, not a separate crawler. Based on Google’s crawler documentation, it controls whether your content may help train future Gemini models. It also covers grounding in Gemini Apps and Vertex AI.

Here’s the good news. Google states that Google-Extended does not affect your inclusion in Google Search. It isn’t a ranking signal either. So blocking it won’t hurt your normal rankings.

OpenAI and Other AI Bots

OpenAI runs more than one bot. GPTBot collects content for model training. OAI-SearchBot, on the other hand, helps show websites in ChatGPT search results. Therefore, blocking OAI-SearchBot can reduce your visibility in ChatGPT answers.

Here’s an example that blocks AI training but keeps AI search open:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Before you copy this, think about your goals. Most Malaysian SMEs want more exposure, not less. If people ask ChatGPT for “the best aircond service in Shah Alam”, you want your brand in that answer. That’s why we often recommend leaving AI search bots open. You can pair this with an llms.txt file, which we explain in our llms.txt guide for Malaysian websites.

How to Create or Edit Robots.txt in WordPress

Most of our clients use WordPress, so this part matters. Luckily, editing the file there takes only a few minutes. Follow these steps.

Step 1: Check Your Current File

Type yourdomain.com.my/robots.txt into your browser. WordPress creates a virtual file by default, so you’ll usually see something. Next, take a screenshot before you change anything. That way, you can roll back easily.

Step 2: Edit It With Yoast SEO

If you use Yoast SEO, go to Yoast SEO, then Tools, then File editor. From there, you can create or edit the file without touching your server. Other SEO plugins offer similar editors. Alternatively, upload a robots.txt file to your root folder through your hosting file manager.

Step 3: Add Your Rules and Sitemap

First, start small. Block only what you truly need to block, such as /wp-admin/ or internal search URLs. Then add your sitemap line at the bottom. Also, keep a short comment with a # sign to explain each rule for your team.

Step 4: Test in Google Search Console

Google retired its old robots.txt tester. Instead, Search Console now offers a robots.txt report, as Search Engine Land reported. The report shows which files Google found, when it last crawled them and any errors. Next, use the URL Inspection tool to check that key pages aren’t blocked.

Computer screen with code while editing robots.txt in WordPress
Photo by Chris Ried on Unsplash

7 Common Robots.txt Mistakes to Avoid

We run many audits for businesses across Malaysia. These are the mistakes we spot again and again:

  1. Blocking the whole site by accident. A single “Disallow: /” under User-agent: * blocks everything. This often happens when a staging site goes live.
  2. Blocking CSS and JavaScript. Google needs these files to render your pages properly.
  3. Using disallow to remove pages. As we covered, use noindex instead.
  4. Forgetting the sitemap line. It’s free, simple and helps discovery.
  5. Wrong letter case. Remember that paths are case-sensitive.
  6. Blocking AI search bots without a plan. You may lose visibility in AI answers.
  7. Ignoring server errors. Google’s spec says a 5xx error on the file makes Google pause crawling your site for up to 12 hours.

Interestingly, a 404 for the file is less scary. Google treats most 4xx errors as if there are no crawl rules at all. Even so, a proper file gives you more control.

A Quick Robots.txt Checklist for Malaysian Businesses

Use this list whenever you launch a site, migrate a domain or redesign your pages:

  • The file loads at yourdomain.com.my/robots.txt.
  • No rule blocks your homepage, service pages or blog.
  • CSS, JavaScript and image folders stay open.
  • Your sitemap URL appears at the bottom.
  • You made a clear decision about AI training and AI search bots.
  • The Search Console robots.txt report shows no errors.

Robots.txt is just one piece of technical SEO, of course. For the bigger picture, check our technical SEO guide. It covers speed, indexing, structure and more.

Conclusion: Small File, Big Impact

Your robots.txt file only takes a few lines. Even so, it shapes how Google and AI tools see your business. Keep it simple, block only what you must, and never use it to hide pages from search results.

Most importantly, make a conscious choice about AI crawlers. Visibility in AI answers is becoming a real source of leads in Malaysia. You don’t want a forgotten line to shut that door.

Not sure if your file is helping or hurting? Get a free SEO audit from Creative Site. We’ll review your robots.txt, indexing and technical setup, then show you what to fix first. You can also see real results in our SEO case studies, or explore our SEO services in Malaysia.

Frequently Asked Questions

Does every website need a robots.txt file?

No, it’s not mandatory. Without one, Google simply assumes it can crawl everything. However, a robots.txt file gives you control and lets you list your sitemap.

Can robots.txt remove a page from Google?

No. A blocked page can still appear in results if other sites link to it. Use a noindex tag or password protection instead.

Will blocking Google-Extended hurt my rankings?

No. Google says Google-Extended does not affect inclusion in Google Search and isn’t a ranking signal. It only covers Gemini training and grounding.

How long do robots.txt changes take to work?

Google generally caches the file for up to 24 hours. So most changes take effect within a day. Server errors can stretch that time.

Should I block AI crawlers on my business website?

It depends on your goals. Blocking training bots is fine for many brands. Still, most Malaysian SMEs should keep AI search bots open to stay visible in AI answers.

Don’t Miss the Next SEO Tip

One practical SEO read every week to help more customers find your business on Google.

We don’t spam! Read more in our privacy policy

Aira
Aira

Aira is part of the SEO content team at Creative Site Digital Solution, covering SEO, AI search, and digital growth strategies for Malaysian SMEs.

Leave a Reply

Your email address will not be published. Required fields are marked *

Get FREE SEO Checklist Now! Get FREE SEO Checklist Now!