There are many situations for which a website offers PDF files on its pages or products , whether they are reports, catalogs or simply downloadable guides, for the use and enjoyment of its users.

Now, you may not want PDF files to be indexed by search engines and appear on the SERPs , and you would prefer your other content to be indexed rather than the files themselves.

If this is your case, and you do not want your PDF files to appear in search engine results, you have 2 ways to achieve it…

Do not index PDF files from robots.txt file

The first, very simple method is to add a policy to the file robots.txtthat PDF files should not be indexed.

To do this, locate the file robots.txt, which will normally be in the root folder of your installation, to edit it and add this line:

Disallow: /*.pdf$

Normally, a robots.txtWordPress default file would look like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://ayudawp.com/wp-sitemap.xml

And, after adding the line, it should look like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*.pdf$
Sitemap: https://ayudawp.com/wp-sitemap.xml

If the file does not exist, it is possible that you have a virtual filerobots.txt , in which case you should create it manually.

Now, you should know that not all search engines will listen to this instruction, Google will, but it is not certain that all of them will, so you still have to opt for the second option…

Do not index PDF files using the X-Robots tag

The most effective method to prevent your PDF files from being indexed is to add an X-Robots tag to the headers of all the pages on your website. To do this you must add the following to .htaccessyour installation file on an Apache server:

<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>

In case your server is NGINX, then you must add the nginx.conffollowing to the file:

location ~* \.pdf$
{
add_header X-Robots-Tag "noindex, nofollow";
}

Save the changes to the file you’ve modified and the tag will be added X-Robotsto the header of your entire website, telling search engine bots not to index or follow PDF files.