To become so good, ChatGPT has trained on an astronomical amount of resources, and your website may be one of them. Be aware that there are solutions to protect yourself from the exploitation of your texts by OpenAI.
Based on the GPT-4 language model, ChatGPT is a conversational agent that has the answer to almost everything. To ensure its proper functioning, the company that developed it, OpenAI, used a gigantic database of content published and available online. To feed the knowledge of ChatGPT, the firm used books, Wikipedia, press sites, Reddit publications and various web pages of all kinds… which potentially includes your own website.
But OpenAI never asked your permission to exploit your content to improve its language models, and therefore, ChatGPT. The company also does not pay you for your participation in the evolution of the capabilities of its product. You may have several reasons for not wanting the content of your site to be siphoned off by ChatGPT for its training purposes, whether practical or moral: to avoid overloading your site with requests, to refuse as a matter of principle to participate to the emergence of artificial intelligence…
How to prevent ChatGPT from exploiting the content of your website?
That’s why we present some tips that should allow you to block access to the contents of your site to ChatGPT.
1. Block Common Crawl on your entire site using the robots.txt file
One of the methods OpenAI uses to power its language models is Common Crawl. This is a non-profit organization that has developed a robot called CCBot. Its purpose is to explore and analyze the web to provide free archives and datasets of content published on the Internet.
CCBot uses the robots.txt protocol to perform its so-called “crawling” operations. This file is located at the root of your website, and its role is to tell crawlers the URLs it is allowed to access on your domain.
To block CCBot, which ChatGPT uses, on your entire website, add the following instructions to your robots.txt file:
User-agent: CCBot
Disallow: /
2. Use a nofollow tag in your website’s metadata
Still to limit access to your content by CCBot, you have the option of adding a nofollow tag within the metadata of your HTML code, which is used to structure and lay out your website. This will let the robots know that you do not want the content of your site to be indexed. The tag in question is this:
<meta name="CCBot" content="nofollow">
3. Use captchas against bots
Captchas are a very popular security technique on the Internet, and their main purpose is to prevent the overloading of websites caused by robots by imposing a test on them that they are not supposed to pass. This solution is only recommended if you are also concerned about malicious bots attacking your site, especially in the context of denial of service (DDoS) attacks. Indeed, these captchas will also be imposed on your human visitors, which can deteriorate their experience on your website. Therefore, favor the two other solutions that we describe above.
These few methods should be enough to prevent ChatGPT (and all platforms that use databases created by crawlers) from using your website and the content it hosts as a playground. you are of course not immune that the chatbot has already had access to your resources in the past. Also keep in mind that it can in some cases be interesting to let the robots scan your site, not all mean you harm.
.