Nov 16 (Reuters) – Popular AI chatbot GPT-4 outperforms most aspiring lawyers on the legal ethics exam required by nearly every state in order to practice law, a new study has found.
GPT-4 answered 74% of the questions correctly on a simulated Multistate Professional Responsibility Exam (MPRE), compared with an estimated 68% average among human test takers nationwide, according to a report released on Thursday by LegalOn Technologies — which sells AI software that reviews contracts.
“Our study indicates that in the future it may be possible to develop AI to assist lawyers with ethical compliance and operate, where relevant, in alignment with lawyers’ professional responsibilities,” the study reads.
It joins a growing body of research examining AI within legal education and attorney licensure. An earlier study found that the previous version of GPT-4 earned passing but not stellar scores on law school final exams. Another more recent study found that GPT-4 can pass the bar exam. Earlier this month, researchers found that access to GPT-4 improved speed on legal writing assignments but didn’t bolster the quality law students’ work.
GPT-4 is a large language model from Microsoft-backed OpenAI that generates human-like text based on user queries.
A spokesperson for the National Conference of Bar Examiners, which develops the MPRE, said that it could not assess the LegalOn report’s claims that GPT-4 can pass its ethics test.
“The legal profession is always evolving in its use of technology, and will continue to do so,” said National Conference spokesperson Sophie Martin. She added that “attorneys have a unique set of skills that AI cannot currently match.”
Every state besides Wisconsin requires law students to pass the 60-multiple-choice MPRE before they are admitted to practice, in addition to passing the bar exam. Tested subjects include conflicts of interest, lawyer-client relationships and confidentiality. Most people take the MPRE while in law school.
GPT-4 performed particularly well on questions about conflicts of interest, with a 91% rate of correct responses. It also answered 88% of questions about the client-lawyer relationship correctly. But its accuracy fell with questions on communications about legal services and safekeeping funds and other property, which it answered correctly a respective 71% and 72% of the time.
“This research demonstrates for the first time that top-performing generative AI models can apply black-letter ethical rules as effectively as aspiring lawyers,” the study reads.
Read more:
AI improves legal writing speed, not quality – study
Bar exam score shows AI can keep up with ‘human lawyers,’ researchers say
Get the latest legal news of the day delivered straight to your inbox with The Afternoon Docket.
Reporting by Karen Sloan
Our Standards: The Thomson Reuters Trust Principles.
