Xinyuan Wang

Generating accurate step-by-step reasoning is essential for Large Language Models (LLMs) to address complex problems and enhance robustness and interpretability. Despite the flux of research on developing advanced reasoning approaches, systematically analyzing the diverse LLMs and reasoning strategies in generating reasoning chains remains a significant challenge. The difficulties stem from the lack of two key elements: (1) an automatic method for evaluating the generated reasoning chains on different tasks, and (2) a unified formalism and implementation of the diverse reasoning approaches for systematic comparison. This paper aims to close the gap: (1) We introduce AutoRace for fully automated reasoning chain evaluation. Existing metrics rely on expensive human annotations or pre-defined LLM prompts not adaptable to different tasks. In contrast, AutoRace automatically creates detailed evaluation criteria tailored for each task, and uses GPT-4 for accurate evaluation following the criteria. (2) We develop LLM Reasoners, a library for standardized modular implementation of existing and new reasoning algorithms, under a unified formulation of the search, reward, and world model components. With the new evaluation and library, (3) we conduct extensive study of different reasoning approaches (e.g., CoT, ToT, RAP). The analysis reveals interesting findings about different factors contributing to reasoning, including the reward-guidance, breadth-vs-depth in search, world model, and prompt formats, etc.

Highly effective, task-specific prompts are often heavily engineered by experts to integrate detailed instructions and domain insights based on a deep understanding of both instincts of large language models (LLMs) and the intricacies of the target task. However, automating the generation of such expert-level prompts remains elusive. Existing prompt optimization methods tend to overlook the depth of domain knowledge and struggle to efficiently explore the vast space of expert-level prompts. Addressing this, we present PromptAgent, an optimization method that autonomously crafts prompts equivalent in quality to those handcrafted by experts. At its core, PromptAgent views prompt optimization as a strategic planning problem and employs a principled planning algorithm, rooted in Monte Carlo tree search, to strategically navigate the expert-level prompt space. Inspired by human-like trial-and-error exploration, PromptAgent induces precise expert-level insights and in-depth instructions by reflecting on model errors and generating constructive error feedback. Such a novel framework allows the agent to iteratively examine intermediate prompts (states), refine them based on error feedbacks (actions), simulate future rewards, and search for high-reward paths leading to expert prompts. We apply PromptAgent to 12 tasks spanning three practical domains: BIG-Bench Hard (BBH), as well as domain-specific and general NLP tasks, showing it significantly outperforms strong Chain-of-Thought and recent prompt optimization baselines. Extensive analyses emphasize its capability to craft expert-level, detailed, and domain-insightful prompts with great efficiency and generalizability.

To reduce the pressure of medical triage in the hospitals, this paper proposes a medical triage system that could classify patients’ questions or texts about their symptoms into several given categories to give suggestions on which kind of consulting room patients could choose. First, we have done extensive research on the medical care situation and the hospitals’ problems in China and conclude that reducing the triage pressure is of great importance for hospitals. We then collect the medical Question Answering datasets, including questions and answers with symptom tags. According to the form of our data, we use BERT, a mainstream model in Natural Language Processing, as the base of our system and modify it with additional components specified to our task. We developed two models based on different datasets. One is trained by data from the five most frequent symptom tags. And for the other one, we use the whole dataset by identifying the appearance of special words, measuring the overlap of all the tags, and merging them into 20 categories. Both of them utilize several training techniques and result in relatively high accuracy: top1 85% accuracy and top2 accuracy 96% on the smaller dataset, top1 accuracy 66.2%, and top2 accuracy 78.3% on the other one. Then we analyze the results and build up our web system for medical use. If given real-world data with similar data distribution, our system could help patients judge diseases and alleviate the triage problem in medical treatment to a certain extent. Moreover, a similar strategy of our model could also be adapted for use in different fields like book searching in libraries. Therefore, our system has a broad application prospect.