====== robot404 Plugin ======

---- plugin ----
description: Send 404 (page-not-found) to webcrawlers (aka robots) for disabled actions and hidden pages
author     : Ahmet Sacan
type       : action
lastupdate : 2024-09-20
compatible : Anteater, Rincewind, Angua, Adora Belle, Weatherwax, Binky, Kaos
similar    : nftr
tags       : header, meta, robots, seo, spiders

downloadurl: https://github.com/ahmetsacan/dokuwiki-plugin-robot404/zipball/master
bugtracker : https://github.com/ahmetsacan/dokuwiki-plugin-robot404/issues
sourcerepo : https://github.com/ahmetsacan/dokuwiki-plugin-robot404/
donationurl: https://www.paypal.com/donate/?hosted_button_id=2LXDWN3YBZL52
----

I see no reason for webcrawlers to visit and index certain actions, such as login and search. The plugin configuration allows you to select which actions should be considered as disallowed for robots. Actions that are disabled in dokuwiki are already considered disallowed for robots. 

Disallowed actions will produce 404 code (page not found) when the user agent is detected to be a robot. Actions that are disabled in dokuwiki normally just display a message; webcrawlers will still index the resulting page and also visit these pages again in the future. 

Whether the client is a robot is determined using HTTP_USER_AGENT header. For testing purposes, you can add ?isrobot404=1 to the URL to simulate a robot visit.


===== Installation =====

Search and install the plugin using the [[plugin:extension|Extension Manager]]. Refer to [[:Plugins]] on how to install plugins manually.