Repository: xlang-ai/xlang-paper-reading
Branch: main
Commit: 2c68370d3fe4
Files: 7
Total size: 68.6 KB
Directory structure:
gitextract_yex7uxkh/
├── .gitignore
├── README.md
├── llm-code-generation.md
├── llm-robotics-and-embodied-ai.md
├── llm-tool-use.md
├── llm-web-grounding.md
└── retrieval_embedding.md
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
.idea/
================================================
FILE: README.md
================================================
# XLang Paper Reading
[](https://twitter.com/XLangNLP)
[](https://join.slack.com/t/xlanggroup/shared_invite/zt-20zb8hxas-eKSGJrbzHiPmrADCDX3_rQ)
[](https://discord.gg/4Gnw7eTEZR)
## Introduction
**Exe**cutable **Lang**uage **G**rounding ([XLANG](https://xlang.ai)) focuses on building language model agents that transform (“grounding”) language instructions into code or actions executable in real-world environments, including databases (data agent), web applications (plugins/web agent), and the physical world (robotic agent) etc,. It lies at the heart of language model agents or natural language interfaces that can interact with and learn from these real-world environments to facilitate human interaction with data analysis, web applications, and robotic instruction through conversation. Recent advances in XLang incorporate techniques such as LLM + external tools, code generation, semantic parsing, and dialog or interactive systems.
<div align="center">
<a href="https://xlang.ai">
<img src="https://docs.xlang.ai/assets/images/xlang_overview-89a754ae588aaa568c2294058489ec18.jpg" width="600" />
</a>
</div>
Here we make a paper list for you to keep track of the research in this track. Stay tuned and have fun!
### Paper Group
- [LLM code generation](https://github.com/xlang-ai/xlang-paper-reading/blob/main/llm-code-generation.md)
- [LLM agents (with tool use)](https://github.com/xlang-ai/xlang-paper-reading/blob/main/llm-tool-use.md)
- [LLM web grounding](https://github.com/xlang-ai/xlang-paper-reading/blob/main/llm-web-grounding.md)
- [LLM robotics](https://github.com/xlang-ai/xlang-paper-reading/blob/main/llm-robotics-and-embodied-ai.md)
================================================
FILE: llm-code-generation.md
================================================
## Paper collection for LLM code generation
## Introduction
## Papers
1. **DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning** PLDI 2021
*Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, oshua B. Tenenbaum* [[pdf](https://dl.acm.org/doi/pdf/10.1145/3453483.3454080)]
1. **On-the-Fly Adaptation of Source Code Models using Meta-Learning** NeurIPS 2020 CAP Workshop
*Disha Shrivastava, Hugo Larochelle, Daniel Tarlow* [[pdf](https://arxiv.org/abs/2003.11768v1)], 2020.5.26
1. **Competition-Level Code Generation with AlphaCode.** Science 2022
*Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, Oriol Vinyals* [[pdf](https://arxiv.org/abs/2203.07814)], 2022.2.8
1. **An Exploratory Study on Code Attention in BERT.** ICPC 2022
*Rishab Sharma, Fuxiang Chen, Fatemeh Fard, David Lo* [[pdf](https://arxiv.org/abs/2204.10200)], 2022.4.5
1. **CoCoSoDa: Effective Contrastive Learning for Code Search.** ICSE 2023
*Ensheng Shi, Yanlin Wang, Wenchao Gu, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, Hongbin Sun* [[pdf](https://arxiv.org/abs/2204.03293)], 2022.4.7
1. **Natural Language to Code Translation with Execution**, EMNLP 2022
*Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, Sida I. Wang* [[pdf](https://arxiv.org/abs/2204.11454)], 2022.4.25
1. **Fault-Aware Neural Code Rankers.** NeurIPS 2022
*Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shuvendu K Lahiri, Madanlal Musuvathi, Jianfeng Gao* [[pdf](https://arxiv.org/abs/2206.03865)], 2022.6.4
1. **Making Large Language Models Better Reasoners with Step-Aware Verifier.** ACL 2023
*Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, Weizhu Chen* [[pdf](https://arxiv.org/abs/2206.02336)], 2022.6.6
1. **CodeS: Towards Code Model Generalization Under Distribution Shift.** ICSE-NIER 2023
*Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Lei Ma, Mike Papadakis, Yves Le Traon* [[pdf](https://arxiv.org/abs/2206.05480)], 2022.6.11
1. **NatGen: Generative pre-training by "Naturalizing" source code.** ESEC/FSE 2022
*Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar Devanbu, Baishakhi Ray* [[pdf](https://arxiv.org/abs/2206.07585)], 2022.6.15
1. **Repository-Level Prompt Generation for Large Language Models of Code.** ICML 2023
*Disha Shrivastava, Hugo Larochelle, Daniel Tarlow* [[pdf](https://arxiv.org/abs/2206.12839)], 2022.6.26
1. **DocPrompting: Generating Code by Retrieving the Docs.** ICLR 2023
*Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, Graham Neubig* [[pdf](https://arxiv.org/abs/2207.05987)], 2022.7.13
1. **CodeT: Code Generation with Generated Tests.** ICLR 2023
*Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, Weizhu Chen* [[pdf](https://arxiv.org/abs/2207.10397)], 2022.7.21
1. **Neurosymbolic Repair for Low-Code Formula Languages.** OOPSLA 2022
*Rohan Bavishi, Harshit Joshi, José Pablo Cambronero Sánchez, Anna Fariha, Sumit Gulwani, Vu Le, Ivan Radicek, Ashish Tiwari* [[pdf](https://arxiv.org/abs/2207.11765)], 2022.7.24
1. **Language Models Can Teach Themselves to Program Better.** ICLR 2023
*Patrick Haluptzok, Matthew Bowers, Adam Tauman Kalai* [[pdf](https://arxiv.org/abs/2207.14502)], 2022.7.29
1. **CSSAM: Code Search via Attention Matching of Code Semantics and Structures.** SANER 2023
*Yi Hu, Bo Cai, Yaoxiang Yu* [[pdf](https://arxiv.org/abs/2208.03922)], 2022.8.8
1. **Incorporating Domain Knowledge through Task Augmentation for Front-End JavaScript Code Generation.** ESEC/FSE 2022
*Sijie Shen, Xiang Zhu, Yihong Dong, Qizhi Guo, Yankun Zhen, Ge Li* [[pdf](https://arxiv.org/abs/2208.10091)], 2022.8.22
1. **Code4Struct: Code Generation for Few-Shot Event Structure Prediction.** ACL 2023
*Xingyao Wang, Sha Li, Heng Ji* [[pdf](https://arxiv.org/abs/2210.12810)], 2022.10.23
1. **DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation.** ICML 2023
*Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang, Tao Yu* [[pdf](https://arxiv.org/abs/2211.11501)], 2022.11.18
1. **Coder Reviewer Reranking for Code Generation.** ICML 2023
*Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, Sida I. Wang* [[pdf](https://arxiv.org/abs/2211.16490)], 2022.11.29
1. **Natural Language to Code Generation in Interactive Data Science Notebooks.** ACL 2023
*Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, Charles Sutton* [[pdf](https://arxiv.org/abs/2212.09248)], 2022.12.19
1. **Python Code Generation by Asking Clarification Questions.** ACL 2023
*Haau-Sing Li, Mohsen Mesgar, André F. T. Martins, Iryna Gurevych* [[pdf](https://arxiv.org/abs/2212.09885)], 2022.12.19
1. **Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments.** ACL 2023
*Yu Gu, Xiang Deng, Yu Su* [[pdf](https://arxiv.org/abs/2212.09736)], 2022.12.19
1. **Execution-Based Evaluation for Open-Domain Code Generation**
*Zhiruo Wang, Shuyan Zhou, Daniel Fried, Graham Neubig* [[pdf](https://arxiv.org/abs/2212.10481)], 2023.12.20
1. **Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning.** SIGIR 2023
*Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, Yongbin Li* [[pdf](https://arxiv.org/abs/2301.13808)], 2023.1.31
1. **Learning Performance-Improving Code Edits** Arxiv
*Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, Amir Yazdanbakhsh* [[pdf](https://arxiv.org/abs/2302.07867)], 2023.2.15
1. **LEVER: Learning to Verify Language-to-Code Generation with Execution.** ICML 2023
*Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, Xi Victoria Lin* [[pdf](https://arxiv.org/abs/2302.08468)], 2023.2.16
1. **EvoPrompting: Language Models for Code-Level Neural Architecture Search.** Arxiv
*Angelica Chen, David M. Dohan, David R. So* [[pdf](https://arxiv.org/abs/2302.14838)], 2023.2.28
1. **Planning with Large Language Models for Code Generation.** ICLR 2023
*Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, Chuang Gan.* [[pdf](https://arxiv.org/abs/2303.05510)], 2023.3.9
1. **Self-planning Code Generation with Large Language Model.** Arxiv
*Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, Ge Li* [[pdf](https://arxiv.org/abs/2303.06689)], 2023.3.12
1. **Reflexion: Language Agents with Verbal Reinforcement Learning.** Arxiv
*Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao* [[pdf](https://arxiv.org/abs/2303.11366)], 2023.3.20
1. **Teaching Large Language Models to Self-Debug.** Arxiv
*Xinyun Chen, Maxwell Lin, Nathanael Schärli, Denny Zhou* [[pdf](https://arxiv.org/abs/2304.05128)], 2023.4.11
1. **WizardLM: Empowering Large Language Models to Follow Complex Instructions.** Arxiv
*Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Daxin Jiang* [[pdf](https://arxiv.org/abs/2304.12244)], 2023.4.24
1. **Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation.** ICML 2023
*Wenqing Zheng, S P Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, Zhangyang Wang* [[pdf](https://arxiv.org/abs/2305.00909)], 2023.4.28
1. **From Words to Code: Harnessing Data for Program Synthesis from Natural Language.** Arxiv
*Anirudh Khatry, Joyce Cahoon, Jordan Henkel, Shaleen Deep, Venkatesh Emani, Avrilia Floratou, Sumit Gulwani, Vu Le, Mohammad Raza, Sherry Shi, Mukul Singh, Ashish Tiwari* [[pdf](https://arxiv.org/abs/2305.01598)], 2023.5.2
1. **Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs.** Arxiv
*Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, Yongbin Li* [[pdf](https://arxiv.org/abs/2305.03111)], 2023.5
1. **Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.** Arxiv
*Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming Zhang* [[pdf](https://arxiv.org/abs/2305.01210)], 2023.5.2
1. **On Contrastive Learning of Semantic Similarity forCode to Code Search.** Arxiv
*Anthony Saieva, Saikat Chakraborty, Gail Kaiser* [[pdf](https://arxiv.org/abs/2305.03843)], 2023.5.5
1. **Self-Edit: Fault-Aware Code Editor for Code Generation.** ACL 2023
*Kechi Zhang, Zhuo Li, Jia Li, Ge Li, Zhi Jin* [[pdf](https://arxiv.org/abs/2305.04087)], 2023.5.6
1. **ToolCoder: Teach Code Generation Models to use API search tools** Arxiv
*Kechi Zhang, Huangzhao Zhang, Ge Li, Jia Li, Zhuo Li, Zhi Jin* [[pdf](https://arxiv.org/abs/2305.04032)], 2023.5.6
1. **Code Execution with Pre-trained Language Models.** ACL 2023 Findings
*Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, Nan Duan* [[pdf](https://arxiv.org/abs/2305.05383)], 2023.5.8
1. **StarCoder: may the source be with you!** Arxiv
*Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, Harm de Vries* [[pdf](https://arxiv.org/abs/2305.06161)], 2023.5.9
1. **SelfzCoT: a Self-Prompt Zero-shot CoT from Semantic-level to Code-level for a Better Utilization of LLMs.** Arxiv
*IokTong Lei, ZhiDong Deng* [[pdf](https://arxiv.org/abs/2305.11461)], 2023.5.19
1. **Text-to-SQL Error Correction with Language Models of Code.** ACL 2023
*Ziru Chen, Shijie Chen, Michael White, Raymond Mooney, Ali Payani, Jayanth Srinivasa, Yu Su, Huan Sun* [[pdf](https://arxiv.org/abs/2305.13073)], 2023.5.22
1. **ALGO: Synthesizing Algorithmic Programs with Generated Oracle Verifiers.** Arxiv
*Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, Lei Li* [[pdf](https://arxiv.org/abs/2305.14591)], 2023.5.24
1. **Tuning Models of Code with Compiler-Generated Reinforcement Learning Feedback.** Arxiv
*Abhinav Jain, Chima Adiole, Swarat Chaudhuri, Thomas Reps, Chris Jermaine* [[pdf](https://arxiv.org/abs/2305.18341)], 2023.5.25
1. **SQL-PaLM: Improved Large Language ModelAdaptation for Text-to-SQL.** Arxiv
*Ruoxi Sun, Sercan O. Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, Tomas Pfister* [[pdf](https://arxiv.org/abs/2306.00739)], 2023.5.26
1. **Grammar Prompting for Domain-Specific Language Generation with Large Language Models.** Arxiv
*Bailin Wang, Zi Wang, Xuezhi Wang, Yuan Cao, Rif A. Saurous, Yoon Kim* [[pdf](https://arxiv.org/abs/2305.19234)], 2023.5.30
1. **SELFEVOLVE: A Code Evolution Framework via Large Language Models.** Arxiv
*Shuyang Jiang, Yuhao Wang, Yu Wang* [[pdf](https://arxiv.org/abs/2306.02907)], 2023.6.5
1. **WizardCoder: Empowering Code Large Language Models with Evol-Instruct.** Arxiv
*Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, Daxin Jiang* [[pdf](https://arxiv.org/abs/2306.08568)], 2023.6.14
1. **Demystifying GPT Self-Repair for Code Generation.** Arxiv
*Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, Armando Solar-Lezama* [[pdf](https://arxiv.org/abs/2306.09896)], 2023.6.16
1. **RepoFusion: Training Code Models to Understand Your Repository.** Arxiv
*Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, Torsten Scholak* [[pdf](https://arxiv.org/abs/2306.10998)], 2023.6.19
1. **Guiding Language Models of Code with Global Context using Monitors.** Arxiv
*Lakshya A Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, Sriram K. Rajamani* [[pdf](https://arxiv.org/abs//2306.10763)], 2023.6.19
1. **Textbooks Are All You Need.** Arxiv
*Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, Yuanzhi Li* [[pdf](https://arxiv.org/abs/2306.11644)], 2023.6.20
1. **Language models are weak learners.** Arxiv
*Hariharan Manikandan, Yiding Jiang, J Zico Kolter* [[pdf](https://arxiv.org/abs/2306.14101)], 2023.6.25
1. **LongCoder: A Long-Range Pre-trained Language Model for Code Completion.** Arxiv
*Daya Guo, Canwen Xu, Nan Duan, Jian Yin, Julian McAuley* [[pdf](https://arxiv.org/abs/2306.14893)], 2023.6.26
1. **InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback.** Arxiv
*John Yang, Akshara Prabhakar, Karthik Narasimhan, Shunyu Yao* [[pdf](https://arxiv.org/abs/2306.14898)], 2023.6.26
1. **A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis** Arxiv
*Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, Aleksandra Faust* [[pdf](https://arxiv.org/abs/2307.12856)], 2023.7.24
1. **Predicting Code Coverage without Execution** Arxiv
*Michele Tufano, Shubham Chandel, Anisha Agarwal, Neel Sundaresan, Colin Clement* [[pdf](https://arxiv.org/abs/2307.13383)], 2023.7.25
1. **ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis** Arxiv
*Kensen Shi, Joey Hong, Manzil Zaheer, Pengcheng Yin, Charles Sutton* [[pdf](https://arxiv.org/abs/2307.13883)], 2023.7.26
1. **Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models** Arxiv
*Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, Tomas Pfister* [[pdf](https://arxiv.org/abs/2308.00675)], 2023.8.1
1. **Symmetry-Preserving Program Representations for Learning Code Semantics** Arxiv
*Kexin Pei, Weichen Li, Qirui Jin, Shuyang Liu, Scott Geng, Lorenzo Cavallaro, Junfeng Yang, Suman Jana* [[pdf](https://arxiv.org/abs/2308.03312)], 2023.8.7
1. **OctoPack: Instruction Tuning Code Large Language Models**
*Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, Shayne Longpre* [[pdf](https://arxiv.org/abs/2308.07124)], 2023.8.14
1. **Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification**
*Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, Hongsheng Li* [[pdf](https://arxiv.org/abs/2308.07921)], 2023.8.15
1. **Does Asking Clarifying Questions Increases Confidence in Generated Code? On the Communication Skills of Large Language Models**
*Jie JW Wu* [[pdf](https://arxiv.org/abs/2308.13507)], 2023.8.25
1. **BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual Pragmatic Knowledge**
*Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, Mark Gerstein* [[pdf](https://arxiv.org/abs/2308.16458)], 2023.8.31
1. **CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models**
*Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, Yifan Liu, Jingkuan Wang, Siyuan Qi, Kangning Zhang, Weinan Zhang, Yong Yu* [[pdf](https://arxiv.org/abs/2309.01940)], 2023.9.5
1. **Improving Code Generation by Dynamic Temperature Sampling**
*Yuqi Zhu, Jia Li, Ge Li, YunFei Zhao, Jia Li, Zhi Jin, Hong Mei* [[pdf](https://arxiv.org/abs/2309.02772)], 2023.9.6
1. **RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair**
*Weishi Wang, Yue Wang, Shafiq Joty, Steven C.H. Hoi* [[pdf](https://arxiv.org/abs/2309.06057)], 2023.9.12
1. **Safurai 001: New Qualitative Approach for Code LLM Evaluation**
*Davide Cifarelli, Leonardo Boiardi, Alessandro Puppo* [[pdf](https://arxiv.org/abs/2309.11385)], 2023.9.20
1. **Program Repair with Minimal Edits Using CodeT5**
*Atsushi Shirafuji, Md. Mostafizer Rahman, Md Faizul Ibne Amin, Yutaka Watanobe* [[pdf](https://arxiv.org/abs/2309.14760)], 2023.9.26
1. **L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models**
*Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, Shafiq Joty, Yingbo Zhou, Dragomir Radev, Arman Cohan* [[pdf](https://arxiv.org/abs/2309.17446)], 2023.9.29
1. **Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency**
*Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, Nan Duan* [[pdf](https://arxiv.org/abs/2309.17272)], 2023.9.29
1. **L2MAC: Large Language Model Automatic Computer for Unbounded Code Generation**
*Samuel Holt, Max Ruiz Luyten, Mihaela van der Schaar* [[pdf](https://arxiv.org/abs/2310.02003)], 2023.10.2
1. **Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation**
*Eric Zelikman, Eliana Lorch, Lester Mackey, Adam Tauman Kalai* [[pdf](https://arxiv.org/abs/2310.02304)], 2023.10.3
1. **$\mathcal{B}$-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis**
*Zishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun, Hongxia Yang* [[pdf](https://arxiv.org/abs/2310.03173)], 2023.10.4
1. **Can Language Models Employ the Socratic Method? Experiments with Code Debugging**
*Erfan Al-Hossami, Razvan Bunescu, Justin Smith, Ryan Teehan* [[pdf](https://arxiv.org/abs/2310.03210)], 2023.10.4
1. **MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning**
*Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, Hongsheng Li* [[pdf](https://arxiv.org/abs/2310.03731)], 2023.10.5
1. **The Program Testing Ability of Large Language Models for Code**
*Weimin Xiong, Yiwen Guo, Hao Chen* [[pdf](https://arxiv.org/abs/2310.05727)], 2023.10.9
1. **SWE-bench: Can Language Models Resolve Real-World GitHub Issues?**
*Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan* [[pdf](https://arxiv.org/abs/2310.06770)], 2023.10.10
1. **CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules**
*Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, Shafiq Joty* [[pdf](https://arxiv.org/abs/2310.08992)], 2023.10.13
1. **Large Language Model-Aware In-Context Learning for Code Generation**
*Jia Li, Ge Li, Chongyang Tao, Jia Li, Huangzhao Zhang, Fang Liu, Zhi Jin* [[pdf](https://arxiv.org/abs/2310.09748)], 2023.10.15
1. **CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion** NeurIPS 2023
*Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, Bing Xiang* [[pdf](https://arxiv.org/abs/2310.11248)], 2023.10.17
1. **Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis**
*Philip John Gorinski, Matthieu Zimmer, Gerasimos Lampouras, Derrick Goh Xin Deik, Ignacio Iacobacci* [[pdf](https://arxiv.org/abs/2310.13669)], 10.20
1. **API-Assisted Code Generation for Question Answering on Varied Table Structures**
*Yihan Cao, Shuyi Chen, Ryan Liu, Zhiruo Wang, Daniel Fried* [[pdf](https://arxiv.org/abs/2310.14687)], 2023.10.23
1. **Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation**
*Jiexin Wang, Liuwen Cao, Xitong Luo, Zhiping Zhou, Jiayuan Xie, Adam Jatowt, Yi Cai* [[pdf](https://arxiv.org/abs/2310.16263)], 2023.10.25
1. **Symbolic Planning and Code Generation for Grounded Dialogue** EMNLP 2023
*Justin T. Chiu, Wenting Zhao, Derek Chen, Saujas Vaduguru, Alexander M. Rush, Daniel Fried* [[pdf](https://arxiv.org/abs/2310.17140)], 2023.10.26
1. **Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation** EMNLP 2023
*Hailin Chen, Amrita Saha, Steven Hoi, Shafiq Joty* [[pdf](https://arxiv.org/abs/2310.18628)], 2023.10.28
1. **LILO: LEARNING INTERPRETABLE LIBRARIES BY COMPRESSING AND DOCUMENTING CODE** Arxiv
*Gabriel Grand, Lionel Wong, Matthew Bowers, Theo X. Olausson, Muxin Liu, Joshua B. Tenenbaum, Jacob Andreas* [[pdf](https://arxiv.org/abs/2310.19791)], 2023.10.30
1. **InstructCoder: Empowering Language Models for Code Editing** Arxiv
*Qisheng Hu, Kaixin Li, Xu Zhao, Yuxi Xie, Tiedong Liu, Hui Chen, Qizhe Xie, Junxian He* [[pdf](https://arxiv.org/abs/2310.20329)], 2023.10.31
1. **Data Augmentation for Code Translation with Comparable Corpora and Multiple References**, EMNLP 2023 Findings
*Yiqing Xie, Atharva Naik, Daniel Fried, Carolyn Rose* [[pdf](https://arxiv.org/abs/2311.00317)], 2023.11.1
1. **Safurai-Csharp: Harnessing Synthetic Data to improve language-specific Code LLM**
*Davide Cifarelli, Leonardo Boiardi, Alessandro Puppo, Leon Jovanovic* [[pdf](https://arxiv.org/abs/2311.03243)], 2023.11.6
1. **Retrieval-Augmented Code Generation for Universal Information Extraction**
*Yucan Guo, Zixuan Li, Xiaolong Jin, Yantao Liu, Yutao Zeng, Wenxuan Liu, Xiang Li, Pan Yang, Long Bai, Jiafeng Guo, Xueqi Cheng* [[pdf](https://arxiv.org/abs/2311.02962)], 2023.11.6
1. **Past as a Guide: Leveraging Retrospective Learning for Python Code Completion**, Neurips 2023 Workshop
*Seunggyoon Shin, Seunggyu Chang, Sungjoon Choi* [[pdf](https://arxiv.org/abs/2311.07635)], 2023.11.13
1. **Coffee: Boost Your Code LLMs by Fixing Bugs with Feedback**
*Seungjun Moon, Yongho Song, Hyungjoo Chae, Dongjin Kang, Taeyoon Kwon, Kai Tzu-iunn Ong, Seung-won Hwang, Jinyoung Yeo* [[pdf](https://arxiv.org/abs/2311.07215)], 2023.11.13
1. **Explain-then-Translate: An Analysis on Improving Program Translation with Self-generated Explanations**
*Zilu Tang, Mayank Agarwal, Alex Shypula, Bailin Wang, Derry Wijaya, Jie Chen, Yoon Kim*, 2023.11.13
1. **CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation**
*Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Shuiguang Deng, Hari Sundaram* [[pdf](https://arxiv.org/abs/2311.08588)], 2023.11.14
1. **ML-Bench: Large Language Models Leverage Open-source Libraries for Machine Learning Tasks**
*Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Zhengliang Li, Liang Chen, Yiming Zong, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, Mark Gerstein* [[pdf](https://arxiv.org/abs/2311.09835)], 2023.11.16
1. **GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding**
*Andor Diera, Abdelhalim Dahou, Lukas Galke, Fabian Karl, Florian Sihler, Ansgar Scherp* [[pdf](https://arxiv.org/abs/2311.09707)], 2023.11.16
1. **Evaluating In-Context Learning of Libraries for Code Generation**
*Arkil Patel, Siva Reddy, Dzmitry Bahdanau, Pradeep Dasigi* [[pdf](https://arxiv.org/abs/2311.09635)], 2023.11.16
1. **Function-constrained Program Synthesis**, 2023 NeurIPS R0-Fomo Workshop
*Patrick Hajali, Ignas Budvytis* [[pdf](https://arxiv.org/abs/2311.15500)], 2023.11.27
1. **Applications of Large Language Models in Data Processing: Innovative Approaches to Segmenting and Renewing Information**
*Yu-Chen Lin, Akhilesh Kumar, Wen-Liang Zhang, Norman Chang, Muhammad Zakir, Rucha Apte, Chao Wang, Jyh-Shing Roger Jang* [[pdf](https://arxiv.org/abs/2311.16267)], 2023.11.27
1. **Self-Infilling Code Generation**
*Lin Zheng, Jianbo Yuan, Zhi Zhang, Hongxia Yang, Lingpeng Kong* [[pdf](https://arxiv.org/abs/2311.17972)], 2023.11.29
1. **Competition-Level Problems are Effective LLM Evaluators**
*Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, Weizhu Chen* [[pdf](https://arxiv.org/abs/2312.02143)], 2023.12.4
1. **Magicoder: Source Code Is All You Need**
*Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, Lingming Zhang* [[pdf](https://arxiv.org/abs/2312.02120)], 2023.12.4
1. **Chain of Code: Reasoning with a Language Model-Augmented Code Emulator**
*Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, Brian Ichter* [[pdf](https://arxiv.org/abs/2312.04474)], 2023.12.7
================================================
FILE: llm-robotics-and-embodied-ai.md
================================================
## Paper collection of LLM + tool use for robotics and embodied AI
## Introduction
## Papers
1. **CLIPort: What and Where Pathways for Robotic Manipulation.** CoRL 2021
*Mohit Shridhar, Lucas Manuelli, Dieter Fox* [[pdf](https://arxiv.org/abs/2109.12098)], 2021.9
2. **Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents.** ICML 2022
*Wenlong Huang, Pieter Abbeel, Deepak Pathak, Igor Mordatch* [[pdf](https://arxiv.org/abs/2201.07207)], 2022.1
3. **Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language.** ICLR 2023
*Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, Pete Florence* [[pdf](https://arxiv.org/abs/2204.00598)], 2022.4
4. **Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.** CoRL 2022
*Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, Andy Zeng* [[pdf](https://arxiv.org/abs/2204.01691)], 2022.4
5. **Inner Monologue: Embodied Reasoning through Planning with Language Models.** CoRL 2022
*Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, Brian Ichter* [[pdf](https://arxiv.org/abs/2207.05608)], 2022.7
6. **JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents.** SoCal NLP 2022
*Kaizhi Zheng, Kaiwen Zhou, Jing Gu, Yue Fan, Jialu Wang, Zonglin Di, Xuehai He, Xin Eric Wang* [[pdf](https://arxiv.org/abs/2208.13266)], 2022.8
7. **ProgPrompt: Generating Situated Robot Task Plans using Large Language Models.** ICRA 2023
*Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg* [[pdf](https://arxiv.org/abs/2209.11302)], 2022.9
8. **Code as Policies: Language Model Programs for Embodied Control.** ICRA 2023
*Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng* [[pdf](https://arxiv.org/abs/2209.07753)], 2022.9
9. **VIMA: General Robot Manipulation with Multimodal Prompts.** ICML 2023
*Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, Linxi Fan* [[pdf](https://arxiv.org/abs/2210.03094)], 2022.10
10. **LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models.** Arxiv
*Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, Yu Su* [[pdf](https://arxiv.org/abs/2212.04088)], 2022.12
11. **RT-1: Robotics Transformer for Real-World Control at Scale.** Arxiv
*Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich* [[pdf](https://arxiv.org/abs/2212.06817)], 2022.12
12. **Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling** ICML 2023
*Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, Roy Fox* [[pdf](https://arxiv.org/abs/2301.12050)], 2023.1
13. **Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents.** Arxiv
*Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, Yitao Liang* [[pdf](https://arxiv.org/abs/2302.01560)], 2023.2
14. **Grounded Decoding: Guiding Text Generation with Grounded Models for Robot Control** Arxiv
*Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, Brian Ichter* [[pdf](https://arxiv.org/abs/2303.00855)], 2023.3
15. **PaLM-E: An Embodied Multimodal Language Model** ICML 2023
*Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, Pete Florence* [[pdf](https://arxiv.org/abs/2303.03378)], 2023.3
16. **Text2Motion: From Natural Language Instructions to Feasible Plans** ICRA 2023 PT4R Workshop
*Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, Jeannette Bohg* [[pdf](https://arxiv.org/abs/2303.12153)], 2023.3
17. **Programmatically Grounded, Compositionally Generalizable Robotic Manipulation** ICLR 2023
*Renhao Wang, Jiayuan Mao, Joy Hsu, Hang Zhao, Jiajun Wu, Yang Gao* [[pdf](https://arxiv.org/abs/2304.13826)], 2023.4
18. **TidyBot: Personalized Robot Assistance with Large Language Models** IROS 2023
*Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, Thomas Funkhouser* [[pdf](https://arxiv.org/abs/2305.05658)], 2023.5
19. **EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought** Arxiv
*Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, Ping Luo* [[pdf](https://arxiv.org/abs/2305.15021)], 2023.5
20. **SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and Reasoning** Arxiv
*Yue Wu, Shrimai Prabhumoye, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, Yuanzhi Li* [[pdf](https://arxiv.org/abs/2305.15486)], 2023.5
21. **Voyager: An Open-Ended Embodied Agent with Large Language Models** Arxiv
*Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar* [[pdf](https://arxiv.org/abs/2305.16291)], 2023.5
22. **Mindstorms in Natural Language-Based Societies of Mind** Arxiv
*Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R. Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, Louis Kirsch, Bing Li, Guohao Li, Shuming Liu, Jinjie Mai, Piotr Piękos, Aditya Ramesh, Imanol Schlag, Weimin Shi, Aleksandar Stanić, Wenyi Wang, Yuhui Wang, Mengmeng Xu, Deng-Ping Fan, Bernard Ghanem, Jürgen Schmidhuber* [[pdf](https://arxiv.org/abs/2305.17066)], 2023.5
23. **Embodied Executable Policy Learning with Language-based Scene Summarization** Arxiv
*Jielin Qiu, Mengdi Xu, William Han, Seungwhan Moon, Ding Zhao* [[pdf](https://arxiv.org/abs/2306.05696)], 2023.6
24. **Generating Language Corrections for Teaching Physical Control Tasks** ICML 2023
*Megha Srivastava, Noah Goodman, Dorsa Sadigh* [[pdf](https://arxiv.org/abs/2306.07012)], 2023.6
25. **SayTap: Language to Quadrupedal Locomotion** Arxiv
*Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, Tatsuya Harada* [[pdf](https://arxiv.org/abs/2306.07580)], 2023.6
26. **Language to Rewards for Robotic Skill Synthesis** Arxiv
*Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, Fei Xia* [[pdf](https://arxiv.org/abs/2306.08647)], 2023.6
27. **REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction** Arxiv
*Zeyi Liu, Arpit Bahety, Shuran Song* [[pdf](https://arxiv.org/abs/2306.15724)], 2023.6
28. **ChatGPT for Robotics: Design Principles and Model Abilities** Arxiv
*Sai Vemprala, Rogerio Bonatti, Arthur Bucker, Ashish Kapoor* [[pdf](https://arxiv.org/abs//2306.17582)], 2023.6
29. **Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Theory of Mind.** Arxiv
*Swarnadeep Saha, Peter Hase, Mohit Bansal* [[pdf](https://arxiv.org/abs/2306.09299)], 2023.6
30. **RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation.** Arxiv
*Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, Antoine Laurens, Claudio Fantacci, Valentin Dalibard, Martina Zambelli, Murilo Martins, Rugile Pevceviciute, Michiel Blokzijl, Misha Denil, Nathan Batchelor, Thomas Lampe, Emilio Parisotto, Konrad Żołna, Scott Reed, Sergio Gómez Colmenarejo, Jon Scholz, Abbas Abdolmaleki, Oliver Groth, Jean-Baptiste Regli, Oleg Sushkov, Tom Rothörl, José Enrique Chen, Yusuf Aytar, Dave Barker, Joy Ortiz, Martin Riedmiller, Jost Tobias Springenberg, Raia Hadsell, Francesco Nori, Nicolas Heess* [[pdf](https://arxiv.org/abs/2306.11706)], 2023.6
31. **Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners** Arxiv
*Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, Anirudha Majumdar* [[pdf](https://arxiv.org/abs/2307.01928)], 2023.7
32. **Building Cooperative Embodied Agents Modularly with Large Language Models** Arxiv
*Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, Chuang Gan* [[pdf](https://arxiv.org/abs/2307.02485)], 2023.7
33. **VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models** Arxiv
*Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Li Fei-Fei* [[pdf](https://arxiv.org/abs/2307.05973)], 2023.7
34. **Demonstrating Large Language Models on Robots** RSS 2023 Demo Track
*Google DeepMind* [[pdf](https://roboticsconference.org/program/papers/024)], 2023.7
35. **GenSim: Generative Models for Supersizing Robotic Simulation Tasks** Github
*Lirui Wang* [[pdf](https://github.com/liruiw/GenSim)], 2023.7
36. **Large Language Models as General Pattern Machines** Arxiv
*Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, Andy Zeng* [[pdf](https://arxiv.org/abs/2307.04721)], 2023.7
37. **SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Task Planning** Arxiv
*Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, Niko Suenderhauf* [[pdf](https://arxiv.org/abs/2307.06135)], 2023.7
38. **RoCo: Dialectic Multi-Robot Collaboration with Large Language Models** Arxiv
*Zhao Mandi, Shreeya Jain, Shuran Song* [[pdf](https://arxiv.org/abs/2307.04738)], 2023.7
39. **RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control** Arxiv
*Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich* [[pdf](https://robotics-transformer2.github.io/assets/rt2.pdf)], 2023.7
40. **Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition** Arxiv
*Huy Ha, Pete Florence, Shuran Song* [[pdf](https://arxiv.org/abs/2307.14535)], 2023.7
41. **Learning to Model the World with Language** Arxiv
*Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, Anca Dragan* [[pdf](https://arxiv.org/abs/2308.01399)], 2023.8
42. **Physically Grounded Vision-Language Models for Robotic Manipulation** Arxiv
*Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, Dorsa Sadigh* [[pdf](https://arxiv.org/abs/2309.02561)], 2023.9
43. **Text2Reward: Automated Dense Reward Function Generation for Reinforcement Learning** Arxiv
*Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, Tao Yu* [[pdf](https://arxiv.org/abs/2309.11489)], 2023.9
================================================
FILE: llm-tool-use.md
================================================
## Paper collection for LLM tool use
## Introduction
## Papers
1. **TALM: Tool Augmented Language Models.** Arxiv
*Aaron Parisi, Yao Zhao, Noah Fiedel* [[pdf](https://arxiv.org/abs/2205.12255)], 2022.5
2. **Binding Language Models in Symbolic Languages.** ICLR 2023
*Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, Tao Yu* [[pdf](https://arxiv.org/abs/2210.02875)], 2022.10
3. **Synergizing Reasoning and Acting in Language Models.** ICLR 2023
*Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao* [[pdf](https://arxiv.org/abs/2210.03629)], 2022.10
4. **PAL: Program-aided Language Models.** ICML 2023
*Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, Graham Neubig* [[pdf](https://arxiv.org/abs/2211.10435)], 2022.11
5. **Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks.** Arxiv
*Wenhu Chen, Xueguang Ma, Xinyi Wang, William W. Cohen* [[pdf](https://arxiv.org/abs/2211.12588)], 2022.11
6. **Planning with Large Language Models via Corrective Re-prompting.** Neurips 2023 workshop
*Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, Stefanie Tellex* [[pdf](https://arxiv.org/abs/2211.09935)], 2022.11
7. **Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning.** SIGIR 2023
*Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, Yongbin Li* [[pdf](https://arxiv.org/abs/2301.13808)], 2023.1
8. **Augmented Language Models: a Survey.** Arxiv
*Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, Thomas Scialom* [[pdf](https://arxiv.org/abs/2302.07842)], 2023.2
9. **Collaborating with language models for embodied reasoning.** NeurIPS 2022 LaReL workshop
*Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill, Rob Fergus* [[pdf](https://arxiv.org/abs/2302.00763)], 2023.2
10. **Toolformer: Language Models Can Teach Themselves to Use Tools.** Arxiv
*Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom* [[pdf](https://arxiv.org/abs/2302.04761)], 2023.2
11. **Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models.** Arxiv
*Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, Nan Duan* [[pdf](https://arxiv.org/abs/2303.04671)], 2023.3
12. **ViperGPT: Visual Inference via Python Execution for Reasoning.** Arxiv
*Dídac Surís, Sachit Menon, Carl Vondrick* [[pdf](https://arxiv.org/abs/2303.08128)], 2023.3
13. **ART: Automatic multi-step reasoning and tool-use for large language models.** Arxiv
*Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, Marco Tulio Ribeiro* [[pdf](http://arxiv.org/abs/2303.09014)], 2023.3
14. **TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs.** Arxiv
*Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, Nan Duan* [[pdf](https://arxiv.org/abs/2303.16434)], 2023.3
15. **HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace.** Arxiv
*Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang* [[pdf](https://arxiv.org/abs/2303.17580)], 2023.3
16. **OpenAGI: When LLM Meets Domain Experts.** Arxiv
*Yingqiang Ge, Wenyue Hua, Kai Mei, Jianchao Ji, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang* [[pdf](https://arxiv.org/abs/2304.04370)], 2023.4
17. **API-Bank: A Benchmark for Tool-Augmented LLMs.** Arxiv
*Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li* [[pdf](https://arxiv.org/abs/2304.08244)], 2023.4
18. **Tool Learning with Foundation Models.** Arxiv
*Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, Maosong Sun* [[pdf](https://arxiv.org/abs/2304.08354)], 2023.4
19. **GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information.** Arxiv
*Qiao Jin, Yifan Yang, Qingyu Chen, Zhiyong Lu* [[pdf](http://arxiv.org/abs/2304.09667)], 2023.4
20. **Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models.** Arxiv
*Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Jianfeng Gao* [[pdf](https://arxiv.org/abs/2304.09842)], 2023.4
21. **LLM+P: Empowering Large Language Models with Optimal Planning Proficiency.** Arxiv
*Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, Peter Stone* [[pdf](https://arxiv.org/abs/2304.11477)], 2023.4
22. **Search-in-the-Chain: Towards Accurate, Credible and Traceable Large Language Models for Knowledge-intensive Tasks.** Arxiv
*Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, Tat-Seng Chua* [[pdf](https://arxiv.org/abs/2304.14732)], 2023.4
23. **ToolCoder: Teach Code Generation Models to use API search tools.** Arxiv
*Kechi Zhang, Ge Li, Jia Li, Zhuo Li, Zhi Jin* [[pdf](https://arxiv.org/abs/2305.04032)], 2023.5
24. **Small models are valuable plug-ins for large language models.** Arxiv
*Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, Julian McAuley* [[pdf](https://arxiv.org/abs/2305.08848)], 2023.5
25. **ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings.** Arxiv
*Shibo Hao, Tianyang Liu, Zhen Wang, Zhiting Hu* [[pdf](https://arxiv.org/abs/2305.11554)], 2023.5
26. **CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing.** Arxiv
*Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, Weizhu Chen* [[pdf](https://arxiv.org/abs/2305.11738)], 2023.5
27. **Making Language Models Better Tool Learners with Execution Feedback.** Arxiv
*Shuofei Qiao, Honghao Gui, Huajun Chen, Ningyu Zhang* [[pdf](https://arxiv.org/abs/2305.13068)], 2023.5
28. **PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents.** Arxiv
*Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, Mohit Iyyer* [[pdf](https://arxiv.org/abs/2305.14564)], 2023.5
29. **ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models.** Arxiv
*Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, Dongkuan Xu* [[pdf](https://arxiv.org/abs/2305.18323)], 2023.5
30. **Gorilla: Large Language Model Connected with Massive APIs.** Arxiv
*Shishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. Gonzalez* [[pdf](https://arxiv.org/abs/2305.15334)], 2023.5
31. **On the Tool Manipulation Capability of Open-source Large Language Models.** Arxiv
*Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, Jian Zhang* [[pdf](https://arxiv.org/abs/2305.16504)], 2023.5
32. **Large Language Models as Tool Makers.** Arxiv
*Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, Denny Zhou* [[pdf](https://arxiv.org/abs/2305.17126)], 2023.5
33. **GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction.** Arxiv
*Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, Ying Shan* [[pdf](https://arxiv.org/abs/2305.18752)], 2023.5
34. **CREATOR: Disentangling Abstract and Concrete Reasonings of Large Language Models through Tool Creation.** Arxiv
*Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, Heng Ji* [[pdf](https://arxiv.org/abs/2305.14318)], 2023.5
35. **Modular Visual Question Answering via Code Generation.** ACL 2023
*Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein* [[pdf](https://arxiv.org/abs/2306.05392)], 2023.6
36. **ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases.** Arxiv
*Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Le Sun* [[pdf](https://arxiv.org/abs/2306.05301)], 2023.6
37. **Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow.** Arxiv
*Wenqi Zhang, Yongliang Shen, Weiming Lu, Yueting Zhuang* [[pdf](https://arxiv.org/abs/2306.07209)], 2023.6
38. **ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs** Arxiv
*Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, Maosong Sun* [[pdf](https://arxiv.org/abs/2307.16789)], 2023.7
39. **LLM Powered Autonomous Agents** Blog
*Lilian Weng*, [[blog](https://lilianweng.github.io/posts/2023-06-23-agent/)] 2023.7
40. **Language Agents in the Digital World: Opportunities and Risks** Blog
*Shunyu Yao and Karthik Narasimhan*, [[blog](https://princeton-nlp.github.io/language-agent-impact/)] 2023.7
41. **ExpeL: LLM Agents Are Experiential Learners** Arxiv
*Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, Gao Huang*, [[pdf](https://arxiv.org/abs/2308.10144)] 2023.8
42. **Agents: An Open-source Framework for Autonomous Language Agents** Arxiv
*Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Ningyu Zhang, Huajun Chen, Peng Cui, Mrinmaya Sachan*, [[pdf](https://arxiv.org/abs/2309.07870)] 2023.9
43. **MindAgent: Emergent Gaming Interaction** Arxiv
*Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, Jianfeng Gao*, [[pdf](https://arxiv.org/abs/2309.09971)] 2023.9
44. **Multimodal Foundation Models: From Specialists to General-Purpose Assistants** Arxiv
*Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao*, [[pdf](https://arxiv.org/abs/2309.10020)] 2023.9
45. **Identifying the Risks of LM Agents with an LM-Emulated Sandbox** Arxiv
*Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, Tatsunori Hashimoto*, [[pdf](https://arxiv.org/abs/2309.15817)] 2023.9
46. **Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models** Arxiv
*Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, Yu-Xiong Wang*, [[pdf](https://arxiv.org/abs/2309.15817)] 2023.9
================================================
FILE: llm-web-grounding.md
================================================
## Paper collection for LLM GUI autonomous agent
## Introduction
## Papers
1. **World of Bits: An Open-Domain Platform for Web-Based Agents.** ICML 2017
*Tianlin (Tim) Shi, Andrej Karpathy, Linxi (Jim) Fan, Jonathan Hernandez, Percy Liang* [[pdf](http://proceedings.mlr.press/v70/shi17a/shi17a.pdf)], 2017
2. **Rico: A Mobile App Dataset for Building Data-Driven Design Applications** 2017
*Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, Ranjitha Kumar* [[pdf](https://dl.acm.org/doi/pdf/10.1145/3126594.3126651)], 2017
4. **Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration.** ICLR 2018
*Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, Percy Liang* [[pdf](https://arxiv.org/abs/1802.08802)], 2018.2
5. **Mapping Natural Language Instructions to Mobile UI Action Sequences.** ACL 2020
*Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, Jason Baldridge* [[pdf](https://arxiv.org/abs/2005.03776)], 2020.5
6. **AndroidEnv: A Reinforcement Learning Platform for Android.** ViGIL at NAACL 2021
*Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, Doina Precup* [[pdf](https://arxiv.org/abs/2105.13231)], 2021.5
7. **Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments.** ViGIL at NAACL 2021
*Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, Bryan A. Plummer* [[pdf](https://arxiv.org/abs/2104.08560)], 2021.4
8. **A data-driven approach for learning to control computers.** PLMR
*Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, Timothy Lillicrap* [[pdf](https://arxiv.org/abs/2202.08137)], 2022.2
9. **META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI.** Arxiv
*Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, Kai Yu* [[pdf](https://arxiv.org/abs/2205.11029)], 2022.5
10. **WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents.** Arxiv
*Shunyu Yao, Howard Chen, John Yang, Karthik Narasimhan* [[pdf](https://arxiv.org/abs/2207.01206)], 2022.7
11. **Enabling Conversational Interaction with Mobile UI using Large Language Models.** CHI 2023
*Bryan Wang, Gang Li, Yang Li* [[pdf](https://arxiv.org/abs/2209.08655)], 2022.9
12. **UGIF: UI Grounded Instruction Following.** Arxiv
*Sagar Gubbi Venkatesh, Partha Talukdar, Srini Narayanan* [[pdf](https://arxiv.org/abs/2211.07615)], 2022.11
13. **Multimodal Web Navigation with Instruction-Finetuned Foundation Models.** ICLR 2023 Workshop ME-FoMo
*Hiroki Furuta, Ofir Nachum, Kuang-Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, Izzeddin Gur* [[pdf](https://arxiv.org/abs/2305.11854)], 2023.5
14. **Hierarchical Prompting Assists Large Language Model on Web Navigation.** ACL 2023 NLRSE workshop
*Abishek Sridhar, Robert Lo, Frank F. Xu, Hao Zhu, Shuyan Zhou* [[pdf](https://arxiv.org/abs/2305.14257)], 2023.5
15. **From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces.** Arxiv
*Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, Kristina Toutanova* [[pdf](https://arxiv.org/abs/2306.00245)], 2023.6
16. **Mind2Web: Towards a Generalist Agent for the Web.** Arxiv
*Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, Yu Su* [[pdf](https://arxiv.org/abs/2306.06070)], 2023.6
17. **A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis** Arxiv
*Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, Aleksandra Faust* [[pdf](https://arxiv.org/abs/2307.12856)], 2023.7
18. **WebArena: A Realistic Web Environment for Building Autonomous Agents** Arxiv
*Shuyan Zhou, Frank F. Xu, Hao Zh+, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, Graham Neubig* [[pdf](https://webarena.dev/static/paper.pdf)], 2023.7
19. **Empowering LLM to use Smartphone for Intelligent Task Automation** Arxiv
*Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, Yunxin Liu* [[pdf](https://arxiv.org/abs/2308.15272)], 2023.8
20. **Android in the Wild: A Large-Scale Dataset for Android Device Control** Arxiv
*Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, Timothy Lillicrap* [[pdf](https://arxiv.org/abs/2307.10088)], 2023.7
21. **An Empirical Study & Evaluation of Modern CAPTCHAs** Arxiv
*Andrew Searles, Yoshimichi Nakatsuka, Ercan Ozturk, Andrew Paverd, Gene Tsudik, Ai Enkoji* [[pdf](https://arxiv.org/abs/2307.12108)], 2023.7
19. **LASER: LLM Agent with State-Space Exploration for Web Navigation** Arxiv
*Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Dong Yu* [[pdf](https://arxiv.org/abs/2309.08172)], 2023.9
20. **You Only Look at Screens: Multimodal Chain-of-Action Agents** Arxiv
*Zhuosheng Zhang, Aston Zhang* [[pdf](https://arxiv.org/abs/2309.11436)], 2023.9
21. **HeaP: Hierarchical Policies for Web Actions using LLMs** Arxiv
*Paloma Sodhi, S.R.K. Branavan, Ryan McDonald* [[pdf](https://arxiv.org/abs/2310.03720)], 2023.10
22. **The Unsolved Challenges of LLMs as Generalist Web Agents: A Case Study** Arxiv
*Rim_Assouel1, Tom Marty, Massimo Caccia, Issam H. Laradji, Alexandre Drouin, Sai Rajeswar, Hector Palacios, Quentin Cappart, David Vazquez, Nicolas Chapados, Maxime Gasse, Alexandre Lacoste* [[pdf](https://openreview.net/forum?id=jt3il4fC5B)], 2023.12
23. **"What's important here?": Opportunities and Challenges of Using LLMs in Retrieving Information from Web Interfaces** Arxiv
*Faria Huq, Jeffrey P. Bigham, Nikolas Martelaro* [[pdf](https://arxiv.org/abs/2312.06147)], 2023.12
24. **ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation** Arxiv
*Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, Mike Zheng Shou* [[pdf](https://arxiv.org/abs/2312.13108)], 2023.12
25. **GPT-4V(ision) is a Generalist Web Agent, if Grounded** Arxiv
*Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su* [[pdf](https://arxiv.org/abs/2401.01614)], 2024.1
26. **SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents** Arxiv
*Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, Zhiyong Wu* [[pdf](https://arxiv.org/abs/2401.10935)], 2024.1
27. **ScreenAgent: A Vision Language Model-driven Computer Control Agent** Arxiv
*Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, Qi Wang* [[pdf](https://arxiv.org/abs/2402.07945)], 2024.2
28. **OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web** Arxiv
*Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, Ruslan Salakhutdinov* [[pdf](https://arxiv.org/abs/2402.17553)], 2024.2
29. **WebLINX: Real-World Website Navigation with Multi-Turn Dialogue** Arxiv
*Xing Han Lù, Zdeněk Kasner, Siva Reddy* [[pdf](https://arxiv.org/abs/2402.05930)], 2024.2
30. **Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as a Case Study** Arxiv
*Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, Yifei Bi, Pengjie Gu, Xinrun Wang, Börje F. Karlsson, Bo An, Zongqing Lu* [[pdf](https://arxiv.org/abs/2403.03186)], 2024.3
31. **AgentStudio: A Toolkit for Building General Virtual Agents** Arxiv
*Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, Shuicheng Yan* [[pdf](https://arxiv.org/abs/2403.17918)], 2024.3
32. **VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?** Arxiv
*Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, Xiang Yue* [[pdf](https://arxiv.org/abs/2404.05955)], 2024.4
33. **OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments** Arxiv
*Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu* [[pdf](https://arxiv.org/abs/2404.07972)], 2024.4
34. **WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents** Arxiv
*Michael Lutz, Arth Bohra, Manvel Saroyan, Artem Harutyunyan, Giovanni Campagna* [[pdf](https://arxiv.org/abs/2404.05902)], 2024.4
35. **Autonomous Evaluation and Refinement of Digital Agents** Arxiv
*Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, Alane Suhr* [[pdf](https://arxiv.org/abs/2404.06474)], 2024.4
36. **MMInA: Benchmarking Multihop Multimodal Internet Agents** Arxiv
*Ziniu Zhang, Shulin Tian, Liangyu Chen, Ziwei Liu* [[pdf](https://arxiv.org/abs/2404.09992)], 2024.4
37. **SteP: Stacked LLM Policies for Web Actions** Arxiv
*Paloma Sodhi, S.R.K. Branavan, Yoav Artzi, Ryan McDonald* [[pdf](https://arxiv.org/abs/2310.03720)], 2024.4
================================================
FILE: retrieval_embedding.md
================================================
1. **Retrieval Augmented Code Generation and Summarization**, EMNLP-Findings 2021
*Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang* [[pdf](https://arxiv.org/abs/2108.11601)], 2021.8.26
1. **Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledge**, AAAI-22
*Yoonna Jang, Jungwoo Lim, Yuna Hur, Dongsuk Oh, Suhyune Son, Yeonsoo Lee, Donghoon Shin, Seungryong Kim, Heuiseok Lim* [[pdf](https://arxiv.org/abs/2112.08619)], 2021.12.16
1. **Retrieving Multimodal Information for Augmented Generation: A Survey**, Arxiv
*Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty* [[pdf](https://arxiv.org/abs/2303.10868)], 2023.3.20
1. **Combining Contexts from Multiple Sources for Documentation-Specific Code Example Generation**, Arxiv
*Junaed Younus Khan, Gias Uddin* [[pdf](https://arxiv.org/abs/2303.14542)], 2023.3.25
1. **LeanDojo: Theorem Proving with Retrieval-Augmented Language Models**, NeurIPS 2023
*Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, Anima Anandkumar* [[pdf](https://arxiv.org/abs/2306.15626)], 2023.6.27
1. **METATOOL BENCHMARK FOR LARGE LANGUAGE MODELS: DECIDING WHETHER TO USE TOOLS AND WHICH TO USE**
*Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, Lichao Sun* [[pdf](https://arxiv.org/abs/2310.03128)], 2023.10.4
1. **Reverse Chain: A Generic-Rule for LLMs to Master Multi-API Planning**
*Yinger Zhang, Hui Cai, Yicheng Chen, Rui Sun, Jing Zheng* [[pdf](https://arxiv.org/abs/2310.04474)], 2023.10.6
1. **Retrieve Anything To Augment Large Language Models**
*Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, Jian-Yun Nie* [[pdf](https://arxiv.org/abs/2310.07554)], 2023.10.11
1. **Language Agnostic Code Embeddings**
*Saiteja Utpala, Alex Gu, Pin Yu Chen* [[pdf](https://arxiv.org/abs/2310.16803)], 2023.10.25
1. **Outlier Dimensions Encode Task-Specific Knowledge**, EMNLP 2023
*William Rudman, Catherine Chen, Carsten Eickhoff* [[pdf](https://arxiv.org/abs/2310.17715)], 2023.10.26
1. **Uncovering Meanings of Embeddings via Partial Orthogonality**
*Yibo Jiang, Bryon Aragam, Victor Veitch* [[pdf](https://arxiv.org/abs/2310.17611)], 2023.10.26
1. **TSTR: Target Similarity Tuning Meets the Real World**, EMNLP-Findings 2023
*Anirudh Khatry, Sumit Gulwani, Priyanshu Gupta, Vu Le, Ananya Singha, Mukul Singh, Gust Verbruggen** [[pdf](https://arxiv.org/abs/2310.17228)], 2023.10.26
1. **VoyageAI embedding**
*Tengyu Ma, etc.* [[blog](https://blog.voyageai.com/2023/10/29/voyage-embeddings/)], 2023.10.30
1. **Generative retrieval-augmented ontologic graph and multi-agent strategies for interpretive large language model-based materials design**
*Markus J. Buehler* [[pdf](https://arxiv.org/abs/2310.19998)], 2023.10.30
1. **On Surgical Fine-tuning for Language Encoders** EMNLP 2023
*Abhilasha Lodha, Gayatri Belapurkar, Saloni Chalkapurkar, Yuanming Tao, Reshmi Ghosh, Samyadeep Basu, Dmitrii Petrov, Soundararajan Srinivasan* [[pdf](https://arxiv.org/abs/2310.17041)], 2023.10.25
1. **Efficient Neural Ranking using Forward Indexes and Lightweight Encoders**
*Jurek Leonhardt, Henrik Müller, Koustav Rudra, Megha Khosla, Abhijit Anand, Avishek Anand* [[pdf](https://arxiv.org/abs/2311.01263)], 2023.11.2
1. **CRUSH4SQL: Collective Retrieval Using Schema Hallucination For Text2SQL**, EMNLP 2023
*Mayank Kothyari, Dhruva Dhingra, Sunita Sarawagi, Soumen Chakrabarti* [[pdf](https://arxiv.org/abs/2311.01173)], 2023.11.2
1. **Multi-Step Dialogue Workflow Action Prediction**
*Ramya Ramakrishnan, Ethan Elenberg, Hashan Narangodage, Ryan McDonald* [[pdf](https://arxiv.org/abs/2311.09593)], 2023.11.16
1. **Sequencing Matters: A Generate-Retrieve-Generate Model for Building Conversational Agents**
*Quinn Patwardhan, Grace Hui Yang* [[pdf](https://arxiv.org/abs/2311.09513)], 2023.11.16
1. **Effective Large Language Model Adaptation for Improved Grounding**
*Xi Ye, Ruoxi Sun, Sercan Ö. Arik, Tomas Pfister* [[pdf](https://arxiv.org/abs/2311.09533)]
1. **ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Science**
*Sai Munikoti, Anurag Acharya, Sridevi Wagle, Sameera Horawalavithana* [[pdf](https://arxiv.org/abs/2311.12289)], 2023.11.21
1. **UniIR: Training and Benchmarking Universal Multimodal Information Retrievers**
*Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, Wenhu Chen* [[pdf](https://arxiv.org/abs/2311.17136)], 2023.11.28
1. **How to Build an AI Tutor that Can Adapt to Any Course and Provide Accurate Answers Using Large Language Model and Retrieval-Augmented Generation**
*Chenxi Dong* [[pdf](https://arxiv.org/abs/2311.17696)], 2023.11.29
1. **A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrieval**
*Matthew Gwilliam, Michael Cogswell, Meng Ye, Karan Sikka, Abhinav Shrivastava, Ajay Divakaran* [[pdf](https://arxiv.org/abs/2312.00115)], 2023.11.30
1. **Context Retrieval via Normalized Contextual Latent Interaction for Conversational Agent**, 2023 IEEE International Conference on Data Mining Workshops (ICDMW)
*Junfeng Liu, Zhuocheng Mei, Kewen Peng, Ranga Raju Vatsavai* [[pdf](https://arxiv.org/abs/2312.00774)], 2023.12.1
1. **Explanatory Argument Extraction of Correct Answers in Resident Medical Exams**
*Iakes Goenaga, Aitziber Atutxa, Koldo Gojenola, Maite Oronoz, Rodrigo Agerri* [[pdf](https://arxiv.org/abs/2312.00567)], 2023.12.1
1. **Event-driven Real-time Retrieval in Web Search**
*Nan Yang, Shusen Zhang, Yannan Zhang, Xiaoling Bai, Hualong Deng, Tianhua Zhou, Jin Ma* [[pdf](https://arxiv.org/abs/2312.00372)], 2023.12.1
1. **D-Bot: Database Diagnosis System using Large Language Models**
*Xuanhe Zhou, Guoliang Li, Zhaoyan Sun, Zhiyuan Liu, Weize Chen, Jianming Wu, Jiesi Liu, Ruohang Feng, Guoyang Zeng* [[pdf](https://arxiv.org/abs/2312.01454)], 2023.12.3
1. **Opportunities for Retrieval and Tool Augmented Large Language Models in Scientific Facilities**
*Michael H. Prince, Henry Chan, Aikaterini Vriza, Tao Zhou, Varuni K. Sastry, Matthew T. Dearing, Ross J. Harder, Rama K. Vasudevan, Mathew J. Cherukara* [[pdf](https://arxiv.org/abs/2312.01291)], 2023.12.3
1. **ProTIP: Progressive Tool Retrieval Improves Planning**
*Raviteja Anantha, Bortik Bandyopadhyay, Anirudh Kashi, Sayantan Mahinder, Andrew W Hill, Srinivas Chappidi* [[pdf](https://arxiv.org/abs/2312.10332)], 2023.12.16
1. **ControlLLM: Augment Language Models with Tools by Searching on Graphs**
*Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Ziheng Li, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, Wenhai Wang* [[pdf](https://arxiv.org/abs/2310.17796)], 2023.12.18
1. **TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems**
*Yilun Kong, Jingqing Ruan, Yihong Chen, Bin Zhang, Tianpeng Bao, Shiwei Shi, Guoqing Du, Xiaoru Hu, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao* [[pdf](https://arxiv.org/abs/2311.11315)], 2023.12.19
1. **RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval**
*Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, Christopher D. Manning* [[pdf](https://arxiv.org/abs/2401.18059)], 2024.01.31
gitextract_yex7uxkh/ ├── .gitignore ├── README.md ├── llm-code-generation.md ├── llm-robotics-and-embodied-ai.md ├── llm-tool-use.md ├── llm-web-grounding.md └── retrieval_embedding.md
Condensed preview — 7 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (72K chars).
[
{
"path": ".gitignore",
"chars": 7,
"preview": ".idea/\n"
},
{
"path": "README.md",
"chars": 2036,
"preview": "# XLang Paper Reading\n\n![](https:"
},
{
"path": "llm-code-generation.md",
"chars": 26437,
"preview": "## Paper collection for LLM code generation\n\n## Introduction\n\n## Papers\n\n1. **DreamCoder: Bootstrapping Inductive Progra"
},
{
"path": "llm-robotics-and-embodied-ai.md",
"chars": 13563,
"preview": "## Paper collection of LLM + tool use for robotics and embodied AI\n\n## Introduction\n\n## Papers\n\n1. **CLIPort: What and W"
},
{
"path": "llm-tool-use.md",
"chars": 11154,
"preview": "## Paper collection for LLM tool use\n\n## Introduction\n\n## Papers\n\n1. **TALM: Tool Augmented Language Models.** Arxiv\n\n "
},
{
"path": "llm-web-grounding.md",
"chars": 9474,
"preview": "## Paper collection for LLM GUI autonomous agent\n\n## Introduction\n\n## Papers\n\n1. **World of Bits: An Open-Domain Platfor"
},
{
"path": "retrieval_embedding.md",
"chars": 7593,
"preview": "\n1. **Retrieval Augmented Code Generation and Summarization**, EMNLP-Findings 2021\n\n *Md Rizwan Parvez, Wasi Uddin Ahm"
}
]
About this extraction
This page contains the full source code of the xlang-ai/xlang-paper-reading GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 7 files (68.6 KB), approximately 23.4k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.