Learn SEO • Lesson 03

Technical SEO: Make Your Website Search-Engine Friendly.

Learn how crawling, indexing, robots.txt, XML sitemaps, canonical tags, page speed, structured data and other technical foundations help search engines access, understand and index a website correctly.

Crawling Indexing Robots.txt XML Sitemap Canonical Core Web Vitals Schema Markup
How Search Engines Process a Website
01
Discover Search engines discover URLs through links, sitemaps and previously known pages.
02
Crawl Bots request and inspect accessible pages.
03
Render & Understand Search systems process content, HTML and page signals.
04
Index Eligible pages may be stored in the search index.
05
Rank Indexed pages may be selected for relevant searches.
Start Here

What is Technical SEO?

Technical SEO focuses on the technical aspects of a website that affect how search engines discover, crawl, understand and index its pages.

Imagine you have excellent content, but Google cannot access the page because it is accidentally blocked. That content may never receive the visibility it deserves.

Technical SEO therefore creates the foundation that allows your On-Page SEO and content to work effectively.

Technical SEO = Make important content accessible, understandable and indexable. A technically healthy website does not automatically rank at the top of Google, but technical problems can prevent otherwise useful pages from performing properly.
The Search Process

Crawling, indexing and ranking are not the same thing.

Understanding this difference makes technical SEO much easier.

01

Discovery

Google discovers URLs through links, XML sitemaps, previous crawls and other signals.

02

Crawling

Googlebot requests accessible URLs and retrieves the content it can access.

03

Indexing

Google evaluates the page and may include it in its searchable index.

04

Ranking

Indexed pages may then compete for visibility when relevant searches are performed.

robots.txt file example for Technical SEO
Technical Element 01

Robots.txt

A robots.txt file provides crawling instructions to search engine bots. It is normally located at:

https://example.com/robots.txt

It can help prevent crawlers from accessing certain sections that do not need to be crawled.

User-agent Identifies which crawler the rule applies to.
Disallow Requests that the crawler does not crawl a specific path.
Allow Can permit access to a path inside an otherwise restricted area.
Sitemap Can point crawlers towards the location of the XML sitemap.
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.phpSitemap: https://example.com/sitemap_index.xml
Important: robots.txt controls crawling — it is not a reliable method for removing a URL from Google's index. Also avoid blocking important CSS, JavaScript or pages Google needs to understand.
Technical Element 02

XML Sitemap

An XML sitemap is a machine-readable file that lists important URLs you want search engines to discover.

For many WordPress websites, an SEO plugin automatically creates a sitemap. Common examples may look like:

https://example.com/sitemap_index.xmlorhttps://example.com/wp-sitemap.xml
Include important indexable URLs Pages included in the sitemap should generally be pages you actually want search engines to discover and index.
Avoid broken URLs Do not intentionally fill the sitemap with 404 pages, redirects or irrelevant URLs.
Keep it updated Modern CMS platforms and SEO plugins often handle this automatically.
Submit Your Sitemap

How to submit a sitemap to Google Search Console

1
Open Google Search Console Sign in and select the correct website property.
2
Open “Sitemaps” Find the Sitemaps section in the Search Console menu.
3
Enter the sitemap URL Example: sitemap_index.xml
4
Click Submit Google can then process the sitemap and report its status.
Open Google Search Console →
Canonical URL technical SEO
Technical Element 03

Canonical URLs

Sometimes multiple URLs contain the same or very similar content. A canonical tag can indicate which URL you prefer search engines to treat as the representative version.

<link rel="canonical" href="https://example.com/running-shoes/">
Duplicate or similar URLs Canonicalisation can help consolidate signals where multiple versions of similar content exist.
Self-referencing canonical Many normal indexable pages use a canonical pointing to their own preferred URL.
Canonical is a hint Search engines may choose a different canonical if other signals strongly suggest another URL.
Example: If several product URLs show effectively the same product, the canonical can help communicate which version should be treated as the main page.
HTTP Status Codes

Know what the server is telling search engines.

HTTP status codes communicate whether a page loads normally, redirects somewhere else, cannot be found or has a server problem.

200

OK

The page loaded successfully and can normally be processed.

301

Permanent Redirect

The old URL permanently redirects users and search engines to another URL.

404

Not Found

The requested page does not exist at that URL.

5xx

Server Error

The server is temporarily unable to successfully return the page.

Website Performance

Fast, responsive pages generally provide a better experience for visitors and make websites easier to use.

Technical Element 04

Page Speed & Core Web Vitals

Page performance can be affected by large images, heavy JavaScript, slow hosting, unnecessary plugins, third-party scripts and inefficient frontend code.

LCP — Largest Contentful Paint Measures how quickly the main visible content is presented to the user.
INP — Interaction to Next Paint Measures responsiveness when a user interacts with the page.
CLS — Cumulative Layout Shift Measures unexpected movement of page elements while loading.

Use Google PageSpeed Insights to test individual URLs and review both laboratory and available real-user data.

Check Website Speed →
Structured Data Examples
Organization Information about an organisation
Article Information about article content
Product Product-related structured information
Breadcrumb Page position in site hierarchy
Local Business Business details where applicable
Event Structured event information
Technical Element 05

Schema Markup & Structured Data

Structured data provides machine-readable information about entities and content on a webpage.

One common format is JSON-LD. For example, a business could provide structured information describing its organisation.

<script type="application/ld+json"> { "@context": "https://schema.org", "@type": "Organization", "name": "Example Company", "url": "https://example.com/", "logo": "https://example.com/logo.png" } </script>
Important: Do not add schema simply to chase rich results. The markup should accurately represent content that exists on the page. Eligible structured data also does not guarantee that Google will show a rich result.

After implementing supported structured data, test the page using Google's Rich Results Test.

Test Structured Data →
More Technical SEO

Other technical areas you should understand.

Technical SEO goes beyond sitemaps and page speed. These areas can also affect how efficiently a website is crawled, indexed and used.

01

HTTPS

Use secure HTTPS connections and avoid insecure mixed-content resources where possible.

02

Mobile Usability

Pages should work properly across mobile devices without tiny text, broken layouts or difficult interactions.

03

Site Architecture

Important pages should be reachable through a logical navigation and internal linking structure.

04

Broken Links

Regularly review internal links that point to deleted, incorrect or inaccessible URLs.

05

Redirect Chains

Avoid unnecessary sequences such as URL A → URL B → URL C when a direct redirect is possible.

06

Duplicate Content

Manage duplicate or near-duplicate URLs through sensible architecture, canonicalisation and indexing decisions.

07

Orphan Pages

Important pages should normally receive relevant internal links instead of existing without connections to the site.

08

Noindex

A noindex directive can request that a page is not included in search results when indexing is not desired.

09

JavaScript SEO

Important content and links should remain accessible and understandable when websites depend heavily on JavaScript.

Practical Workflow

How to check Technical SEO step by step.

Use this simplified workflow when reviewing a website.

01

Check Whether Important Pages Are Indexed

Use Google Search Console and URL Inspection to understand whether key URLs can be indexed and whether Google has indexed them.

02

Review Robots.txt

Confirm that important pages and resources are not being blocked accidentally.

03

Check the XML Sitemap

Make sure the sitemap loads properly and contains the important canonical URLs you want search engines to discover.

04

Submit Sitemap to Search Console

Submit the sitemap and monitor whether Google can process it successfully.

05

Review Status Codes & Redirects

Check for broken pages, unnecessary redirect chains and server errors affecting important URLs.

06

Check Canonical Tags

Verify that important pages point to the intended preferred URL and are not accidentally canonicalised elsewhere.

07

Test Website Speed

Review PageSpeed Insights and investigate heavy images, scripts, plugins and server-side performance issues.

08

Review Mobile Experience

Test important templates on smaller screens and make sure content remains readable and usable.

09

Validate Structured Data

Test relevant schema markup and correct invalid or inaccurate structured data where necessary.

10

Monitor Search Console

Technical SEO is ongoing. Continue monitoring indexing, crawl issues, performance and changes after website updates.

Key Takeaways

Remember these five things.

01 Google needs to discover and access a page before it can consider indexing it.
02 robots.txt controls crawling, while indexing decisions are separate.
03 Sitemaps help search engines discover important URLs more efficiently.
04 Speed, mobile usability and site health contribute to a better technical foundation.
05 Schema should accurately describe content, not be added simply to chase rich results.
Knowledge Check

Technical SEO Quiz

Answer all five questions and check your score.

Question 01

What is the primary purpose of robots.txt?

Question 02

What is an XML sitemap mainly used for?

Question 03

What does a 301 status code normally indicate?

Question 04

What is the purpose of a canonical tag?

Question 05

Which Google tool can test supported structured data for rich results?

Next Lesson

Your technical foundation is ready. Now build authority beyond the website.

Next, learn Off-Page SEO — including backlinks, referring domains, link quality, brand mentions, digital PR and how authority signals can support organic search visibility.