The results have been so good that educators have warned it could lead to widespread cheating and even signal the end of traditional classroom teaching methods.
In a white paper titled “ChatGPT goes to law school” published on Monday, he and his coauthors reported that the bot scored a C+ overall.
While this was enough for a pass, the bot was near the bottom of the class in most subjects and “bombed” at multiple-choice questions involving mathematics.
“In writing essays, ChatGPT displayed a strong grasp of basic legal rules and had consistently solid organisation and composition,” the authors wrote. But the bot “often struggled to spot issues when given an open-ended prompt, a core skill on law school exams”.Officials in New York and other jurisdictions have banned the use of ChatGPT in schools, but Choi suggested it could be a valuable teaching aide.
“Overall, ChatGPT wasn’t a great law student acting alone,” he wrote on Twitter. “But we expect that collaborating with humans, language models like ChatGPT would be very useful to law students taking exams and to practicing lawyers.”