publications

2026

1. 

   PhoneBuddy: Training Open Models for Agentic Phone Use

   Zhengyang Tang, Xin Lai, Pengyuan Lyu, Xinyuan Wang, Tianyi Bai, and 5 more authors

   2026

   Abs PDF Website

   Phones are becoming an important execution surface for general-purpose agents, but training open models for reliable phone use remains difficult because the deployment environment is slow, stateful, side-effectful, and hard to reset or verify, while scalable mock environments only approximate real behavior. We present PhoneBuddy, a training recipe and open-model line that combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs runnable mock apps from real GUI usage structure. PhoneBuddy first builds a shared supervised fine-tuning stage, then compares real-app RL against mixed RL across both environments. On a 150-task human evaluation, success improves from 36.67% (SFT) to 40.67% (real-app RL) to 45.33% (mixed RL); on AndroidWorld, from 60.3% to 77.2% to 83.2%. Mock-app training is not a replacement for real-app RL, but a complementary source of scalable, resettable, and automatically checked interaction.

2. 

   Do Phone-Use Agents Respect Your Privacy?

   Zhengyang Tang, Ke Ji, Xidong Wang, Zihan Ye, Xinyuan Wang, and 5 more authors

   2026

   Abs PDF

   We investigate whether phone-use agents protect user privacy when carrying out routine mobile tasks. This has been hard to assess because privacy-compliant behavior was not well-defined for such agents, and standard apps do not expose what data agents enter into forms. We introduce MyPhoneBench, a verifiable evaluation framework for privacy behavior in mobile agents. It defines privacy-respecting phone use through three principles—permissioned access, minimal disclosure, and user-controlled memory—via a privacy contract called iMy, combined with instrumented mock apps and rule-based auditing. Evaluating five frontier models across 10 mobile apps and 300 tasks, we find task success, privacy-compliant completion, and reuse of saved preferences are separate capabilities, with no single model excelling at all three. The most common failure is poor data minimization. Privacy issues stem from over-helpful execution of benign tasks, and evaluating success alone overstates how deployment-ready these agents are.

3. 

   GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

   Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, and 6 more authors

   2026

   Abs PDF

   We formalize end-to-end game generation, where coding agents must turn natural-language specifications into playable games within a game engine. Unlike traditional coding, scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We propose three desiderata—Engine Grounding, Artifact Completeness, and Interactive Verification—and introduce an evaluation framework using replayed demonstrations and rubric-guided multimodal judging. This is instantiated as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. The task remains difficult: the top agent reaches only 41.46%, with most below 40%. Agents implement recognizable mechanics but struggle with complete content, visual feedback, and presentation.

2025

1. 

   VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos

   Dunjie Lu, Yiheng Xu, Junli Wang, Haoyuan Wu, Xinyuan Wang, and 10 more authors

   2025

   PDF Website

2. 

   Opencua: Open foundations for computer-use agents

   Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, and 6 more authors

   NeurIPS 2025 (Spotlight), COLM 2025 Workshop AIA (Best Paper), 2025

   Abs PDF Code Poster Website

   Vision-language models have demonstrated impressive capabilities as computer-use agents (CUAs) capable of automating diverse computer tasks. As their commercial potential grows, critical details of the most capable CUA systems remain closed. As these agents will increasingly mediate digital interactions and execute consequential decisions on our behalf, the research community needs access to open CUA frameworks to study their capabilities, limitations, and risks. To bridge this gap, we propose OpenCUA, a comprehensive open-source framework for scaling CUA data and foundation models. Our framework consists of: (1) an annotation infrastructure that seamlessly captures human computer-use demonstrations; (2) AgentNet, the first large-scale computer-use task dataset spanning 3 operating systems and 200+ applications and websites; (3) a scalable pipeline that transforms demonstrations into state-action pairs with reflective long Chain-of-Thought reasoning that sustain robust performance gains as data scales. Our end-to-end agent models demonstrate strong performance across CUA benchmarks. In particular, OpenCUA-32B achieves an average success rate of 34.8% on OSWorld-Verified, establishing a new state-of-the-art (SOTA) among open-source models and surpassing OpenAI CUA (GPT-4o). Further analysis confirms that our approach generalizes well across domains and benefits significantly from increased test-time computation. We release our annotation tool, datasets, code, and models to build open foundations for further CUA research.

3. 

   Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis

   Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, and 6 more authors

   NeurIPS 2025 (Spotlight), 2025

   Abs PDF Code Website

   GUI Grounding, the ability to map natural language instructions to specific actions on graphical user interfaces, remains a critical bottleneck in computer use agent development. Current benchmarks oversimplify grounding tasks as short referring expressions, failing to capture the complexity of real-world interactions that require software commonsense, layout understanding, and fine-grained manipulation capabilities.To address these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types including text matching, element recognition, layout understanding, and precise manipulation. Additionally, we synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples through multi-perspective decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its effectiveness by outperforming existing approaches on ScreenSpot-v2, ScreenSpot-Pro, and our new benchmark. Furthermore, we demonstrate that improved grounding with Jedi directly enhances agentic capabilities of general foundation models on complex computer tasks, improving from 5% to 27% in OSWorld. Through detailed ablation studies, we identify key factors contributing to grounding performance and verify that combining specialized data for different interface elements enables compositional generalization to novel interfaces. All data, checkpoints, and code are open-sourced and available for future research.

4. 

   Introducing OSWorld-Verified

   Tianbao Xie, Mengqi Yuan, Danyang Zhang, Xinzhuang Xiong, Zhennan Shen, and 12 more authors

   xlang.ai, Jul 2025

   Website

5. 

   Computer Agent Arena: Compare & Test Computer Use Agents on Crowdsourced Real-World Tasks

   Bowen Wang, Xinyuan Wang, Jiaqi Deng, Tianbao Xie, Ryan Li, and 11 more authors

   Jul 2025

   Website

2024

1. 

   LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models

   Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, and 6 more authors

   COLM 2024, Jul 2024

   Abs PDF Code

   Generating accurate step-by-step reasoning is essential for Large Language Models (LLMs) to address complex problems and enhance robustness and interpretability. Despite the flux of research on developing advanced reasoning approaches, systematically analyzing the diverse LLMs and reasoning strategies in generating reasoning chains remains a significant challenge. The difficulties stem from the lack of two key elements: (1) an automatic method for evaluating the generated reasoning chains on different tasks, and (2) a unified formalism and implementation of the diverse reasoning approaches for systematic comparison. This paper aims to close the gap: (1) We introduce AutoRace for fully automated reasoning chain evaluation. Existing metrics rely on expensive human annotations or pre-defined LLM prompts not adaptable to different tasks. In contrast, AutoRace automatically creates detailed evaluation criteria tailored for each task, and uses GPT-4 for accurate evaluation following the criteria. (2) We develop LLM Reasoners, a library for standardized modular implementation of existing and new reasoning algorithms, under a unified formulation of the search, reward, and world model components. With the new evaluation and library, (3) we conduct extensive study of different reasoning approaches (e.g., CoT, ToT, RAP). The analysis reveals interesting findings about different factors contributing to reasoning, including the reward-guidance, breadth-vs-depth in search, world model, and prompt formats, etc.

2. 

   PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization

   Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, and 4 more authors

   ICLR 2024, Jul 2024

   Abs PDF Code Poster

   Highly effective, task-specific prompts are often heavily engineered by experts to integrate detailed instructions and domain insights based on a deep understanding of both instincts of large language models (LLMs) and the intricacies of the target task. However, automating the generation of such expert-level prompts remains elusive. Existing prompt optimization methods tend to overlook the depth of domain knowledge and struggle to efficiently explore the vast space of expert-level prompts. Addressing this, we present PromptAgent, an optimization method that autonomously crafts prompts equivalent in quality to those handcrafted by experts. At its core, PromptAgent views prompt optimization as a strategic planning problem and employs a principled planning algorithm, rooted in Monte Carlo tree search, to strategically navigate the expert-level prompt space. Inspired by human-like trial-and-error exploration, PromptAgent induces precise expert-level insights and in-depth instructions by reflecting on model errors and generating constructive error feedback. Such a novel framework allows the agent to iteratively examine intermediate prompts (states), refine them based on error feedbacks (actions), simulate future rewards, and search for high-reward paths leading to expert prompts. We apply PromptAgent to 12 tasks spanning three practical domains: BIG-Bench Hard (BBH), as well as domain-specific and general NLP tasks, showing it significantly outperforms strong Chain-of-Thought and recent prompt optimization baselines. Extensive analyses emphasize its capability to craft expert-level, detailed, and domain-insightful prompts with great efficiency and generalizability.

2021

1. 

   Reduce the medical burden: An automatic medical triage system using text classification BERT based on Transformer structure

   Xinyuan Wang, Make Tao, Runpu Wang, and Likui Zhang

   In 2021 2nd International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE), Jul 2021

   Abs PDF

   To reduce the pressure of medical triage in the hospitals, this paper proposes a medical triage system that could classify patients’ questions or texts about their symptoms into several given categories to give suggestions on which kind of consulting room patients could choose. First, we have done extensive research on the medical care situation and the hospitals’ problems in China and conclude that reducing the triage pressure is of great importance for hospitals. We then collect the medical Question Answering datasets, including questions and answers with symptom tags. According to the form of our data, we use BERT, a mainstream model in Natural Language Processing, as the base of our system and modify it with additional components specified to our task. We developed two models based on different datasets. One is trained by data from the five most frequent symptom tags. And for the other one, we use the whole dataset by identifying the appearance of special words, measuring the overlap of all the tags, and merging them into 20 categories. Both of them utilize several training techniques and result in relatively high accuracy: top1 85% accuracy and top2 accuracy 96% on the smaller dataset, top1 accuracy 66.2%, and top2 accuracy 78.3% on the other one. Then we analyze the results and build up our web system for medical use. If given real-world data with similar data distribution, our system could help patients judge diseases and alleviate the triage problem in medical treatment to a certain extent. Moreover, a similar strategy of our model could also be adapted for use in different fields like book searching in libraries. Therefore, our system has a broad application prospect.