GPTZero, developed by Princeton University student Edward Tian, is an artificial intelligence (AI) content detection tool specifically designed for educators and teachers. Its creation was prompted by the growing concern over “AI plagiarism,” which refers to the improper use of AI tools to generate content without appropriate attribution. Educators and university lecturers have expressed worries about students employing AI content creation tools like ChatGPT during exams.
Tian positions GPTZero as a solution to this problem by analyzing texts based on their perplexity and burstiness. Perplexity measures the level of randomness in a text according to a language model’s preference for it. Burstiness, on the other hand, evaluates the consistency of perplexity over time compared to the varied nature of human-written text.
Currently in beta mode, GPTZero is set to release a paper with accuracy statistics based on data from student journalism pieces. With the increasing prevalence of AI content creation, tools like GPTZero play a crucial role in regulating AI usage and safeguarding the integrity of students’ work. Exploring the features of this AI detection tool will provide insight into its viability as a solution to AI plagiarism.
Features
- Perplexity score: Perplexity serves as a measure of randomness, evaluating how effectively a language model, such as ChatGPT, can predict subsequent words. Higher scores indicate greater unpredictability, suggesting that a text is more likely to be human-generated. GPTZero currently utilizes the perplexity score based on the GPT-2 model.
- Burstiness score: This metric examines the distribution of sentences to determine whether there is uniformity or pattern. It operates on the principle that human writers tend to incorporate both short and long sentences in their writing.
- Highlighted AI-written portions: GPTZero’s recent update includes phrase detection functionality. The tool identifies and highlights the phrases within a text that are most likely generated by an AI. Furthermore, GPTZero indicates the sentence with the highest perplexity score.
- API integration: GPTZero’s creator has made its API easily accessible for integration into various systems. Users can also reach out to the app’s engineers via email to request customization support.
Testing GPTZero’s Accuracy
To assess the accuracy of GPTZero, we conducted a test using seven text samples generated by Jasper.ai, a widely used AI writing tool. In order to provide a comparison, we will display the results from GPTZero alongside those from Chatgptdemo.net. However, before revealing the results, it is essential to understand how GPTZero presents its findings.
Unlike many other AI content detection tools that use a probability format, GPTZero offers a general statement indicating whether the text is likely generated by AI or not. Additionally, the app displays the perplexity score and the burstiness score.
On the other hand, Chatgptdemo.net presents results in percentage form. It evaluates the degree of originality in a text and the percentage that is likely to have been created by AI. Unlike GPTZero, Chatgptdemo.net is trained on multiple language models, including GPT-3, GPT-3.5, GPT-4 and ChatGPT.
During our testing, we observed perplexity scores reaching as high as 30,000. This metric forms the basis of our analysis alongside the general statements provided by GPTZero.
None of the samples achieved a 100% detection rate with GPTZero. The app would display the statement “Your text is likely to be written entirely by AI” if it detected 100% AI-generated content. Conversely, Chatgptdemo.net achieved 100% detection on samples #1 and #2, and it detected over 90% AI-generated content in three additional samples.
In general, GPTZero has the ability to detect instances of AI-generated content. However, there is room for improvement in how the scores are presented, particularly for educators to fully understand the extent of AI plagiarism in a text. In terms of accuracy and clarity, Chatgptdemo.net emerged as the winner in this test.
GPTZero’s Potential
As the GPTZero tool is currently in beta mode, it would be premature to highlight its drawbacks at this stage. Instead, let’s focus on its potential future developments. Firstly, GPTZero is currently offered as a free tool, making it easily accessible for teachers.
Moreover, the availability of its API allows other developers to utilize and integrate GPTZero into their own systems.
From the outset, Edward Tian emphasized that GPTZero was specifically designed with academics in mind, distinguishing it from many AI content detection tools that cater to blog posts and marketing copy.
As the GPTZero team continues to test the tool on more student work data, its accuracy is expected to improve. When used in conjunction with online plagiarism checkers, GPTZero can contribute to enhancing the integrity of student essays and theses.
For now, educators should exercise caution in their use of GPTZero during its ongoing development. As stated on the tool’s page, AI detection is just one component of a comprehensive assessment of student work. Additionally, educators can also utilize plagiarism detection tools that scan documents against vast amounts of publicly available online content.
Conclusion
While GPTZero is a free tool that shows promise in detecting AI plagiarism, it currently lags behind competitors like Chatgptdemo.net, which not only identifies AI-generated content but also detects plagiarism.
It is essential to acknowledge that GPTZero is still in the beta phase. As the Princeton team continues to enhance the application, we can anticipate improvements in accuracy and features. In the interim, GPTZero’s metrics serve as a useful starting point for teachers seeking to identify AI plagiarism.
