ANSPRE: A Method that Boosts the Accuracy Performance of Large Language Models
en-GBde-DEes-ESfr-FR

ANSPRE: A Method that Boosts the Accuracy Performance of Large Language Models


Researchers develop a new method that enables large language models to answer questions more concisely and accurately

Ishikawa, Japan -- Large language models (LLMs) are machine-learning models designed to understand and generate human language. State-of-the-art LLMs have demonstrated outstanding potential in open-domain question answering (ODQA), where the model is tasked with providing answers to factual questions. This is particularly useful in fields such as finance, healthcare, and education. However, LLMs typically rely on their pre-trained knowledge to answer questions, which can become outdated in a constantly changing world.

This limitation can be addressed by using Retrieval-Augmented Generation (RAG) with a pre-trained LLM. In this approach, the question is augmented with documents from a knowledge base. Despite these advancements, LLMs often produce lengthy responses, providing contextual information that can make it difficult and time-consuming to identify the exact answer phrase.

Another important aspect of LLMs is their ability to produce confidence scores, which reflect how certain the model is about the correctness of its answer. These scores are especially crucial in high-risk fields such as finance, law, and healthcare. Although LLMs can generate sequence probabilities for a specific response, this probability is often unreliable in terms of calibration. This means the predicted confidence may not accurately correlate with the probability of correctness and should not be used as a confidence score. The inability to identify the exact answer phrase and produce a reliable confidence score limits the practical application of LLMs.

To address these limitations, a team of researchers from the Japan Advanced Institute of Science and Technology, led by Professor Nguyen Le Minh and including doctoral students Nguyen-Khang Le, Dieu-Hien Nguyen introduced a novel method called Answer-prefix Generation (ANSPRE). “ANSPRE can improve the generation quality of LLMs, allow them to output the exact answer phrase, and produce reliable confidence scores. Additionally, it can be incorporated into any LLM and complex architecture” says Prof. Nguyen. Their study will be presented at ECAI-2024, the 27th European Conference on Artificial Intelligence held on October 19-24­.

The main idea of ANSPRE is to add a sequence of text to the LLM prompt that leads to the answer phrase. This sequence of text is called the ‘answer prefix’. Prof. Nguyen explains, “Consider the example question, ‘What gambling game, requiring two coins to play, was popular in World War I?’ An answer prefix for this question could be, ‘The gambling game requiring two coins to play that was popular in World War I was ___.’ As most LLMs are trained with causal language modeling, using the answer prefix would allow the LLM to generate the exact answer phrase in place of the blank.”

Given a question, ANSPRE first generates an answer prefix using selected few-shot examples. The researchers demonstrated that only a few handcrafted examples were sufficient to generate a high-quality answer prefix. ANSPRE then uses an existing retriever to gather relevant documents from the knowledge base, similar to RAG. It combines the document, the question, and the answer prefix, and prompts the LLM to generate the answer phrase. Finally, ANSPRE aggregates the answer phrases and confidence scores across different documents used to answer the question, to produce the final answer.

The researchers demonstrated ANSPRE’s versatility by constructing Self-Reflective Answer-Prefix Generation (SELF-ANSPRE), which combines ANSPRE with Self-Reflective RAG (SEFT-RAG). SEFT-RAG improves LLM generation by introducing reflection tokens to decide when and what to retrieve from the knowledge base and rank the responses based on the utility of the documents and the answer. In SELF-ANSPRE the confidence scores from ANSPRE and scores from reflection tokens are combined to generate the final ranking score.

The researchers tested ANSPRE on three ODQA benchmarks and various LLM architectures. The results showed that ANSPRE significantly improves pre-trained and instruction-tuned LLMS, producing high-quality answers and confidence scores that strongly correlate with correctness. Moreover, SELF-ANSPRE significantly enhanced SEFT-RAG. Their analysis also highlighted the importance of each ANSPRE component.

“Our method can lead to more concise and accurate question answering in critical fields like medical diagnosis, legal assistance, and education, and improve customer support. Furthermore, in the long term, our research could foster widespread human-artificial intelligence collaboration by increasing trust in AI systems ,” remarks Prof. Nguyen.

Overall, this innovative method marks a significant step forward for LLMs and can lead to their broader application, even in sensitive domains.


###

Reference
Title of original paper: ANSPRE: Improving Question-Answering in Large Language Models with Answer-Prefix Generation
Authors: Nguyen-Khang Le, Dieu-Hien Nguyen, and Nguyen Le Minh
Conference: ECAI-2024, the 27th European Conference on Artificial Intelligence
DOI: 10.3233/FAIA392



About Japan Advanced Institute of Science and Technology, Japan
Founded in 1990 in Ishikawa prefecture, the Japan Advanced Institute of Science and Technology (JAIST) was the first independent national graduate school in Japan. Now, after 30 years of steady progress, JAIST has become one of Japan’s top-ranking universities. JAIST counts with multiple satellite campuses and strives to foster capable leaders with a state-of-the-art education system where diversity is key; about 40% of its alumni are international students. The university has a unique style of graduate education based on a carefully designed coursework-oriented curriculum to ensure that its students have a solid foundation on which to carry out cutting-edge research. JAIST also works closely both with local and overseas communities by promoting industry–academia collaborative research.


About Professor Nguyen Le Minh from Japan Advanced Institute of Science and Technology, Japan
Nguyen Le Minh is a Professor at the Japan Advanced Institute of Science and Technology (JAIST). He received his M.S. from Vietnam National University, Hanoi, in 2001 and his Ph.D. from JAIST in 2004. He also leads the Nguyen laboratory at JAIST. He has over 370 publications with over 5000 citations. His research interests include artificial intelligence, natural language processing, deep learning, and machine learning.
Title: ANSPRE: Improving Question-Answering in Large Language Models with Answer-Prefix Generation
Authors: Nguyen-Khang Le, Dieu-Hien Nguyen, and Nguyen Le Minh
Conference: ECAI-2024, the 27th European Conference on Artificial Intelligence
DOI: 10.3233/FAIA392
Angehängte Dokumente
  • Figure 1. Overview of the proposed method, Answer-prefix Generation compared to ordinary Retrieval-Augmented Generation.  Image caption: Answer-prefix Generation (ANSPRE) generates an answer prefix for the prompt question and then retrieves relevant information from the knowledge base like Retrieval-Augmented Generation (RAG) to generate accurate and concise answer phrases.  Image credit: Nguyen Le Minh from JAIST.
  • Figure 2. SELF-ANSPRE: Adaptation of the proposed method to Self-Reflective Retrieval-Augmented Generation.   Image Caption: Self-Reflective Answer-prefix Generation (SELF- ANSPRE) demonstrates that the proposed ANSPRE can be incorporated into any large language moddel and even into sophisticated systems.  Image credit: Nguyen Le Minh from JAIST.
Regions: Asia, Japan
Keywords: Applied science, Artificial Intelligence, Computing, Engineering, Humanities, Linguistics

Disclaimer: AlphaGalileo is not responsible for the accuracy of news releases posted to AlphaGalileo by contributing institutions or for the use of any information through the AlphaGalileo system.

Referenzen

We have used AlphaGalileo since its foundation but frankly we need it more than ever now to ensure our research news is heard across Europe, Asia and North America. As one of the UK’s leading research universities we want to continue to work with other outstanding researchers in Europe. AlphaGalileo helps us to continue to bring our research story to them and the rest of the world.
Peter Dunn, Director of Press and Media Relations at the University of Warwick
AlphaGalileo has helped us more than double our reach at SciDev.Net. The service has enabled our journalists around the world to reach the mainstream media with articles about the impact of science on people in low- and middle-income countries, leading to big increases in the number of SciDev.Net articles that have been republished.
Ben Deighton, SciDevNet
AlphaGalileo is a great source of global research news. I use it regularly.
Robert Lee Hotz, LA Times

Wir arbeiten eng zusammen mit...


  • BBC
  • The Times
  • National Geographic
  • The University of Edinburgh
  • University of Cambridge
  • iesResearch
Copyright 2024 by DNN Corp Terms Of Use Privacy Statement