Welcome! I am currently a second year PhD student at the Computer Sciences Department of UW-Madison, where I am fortunate enough to be working with Prof. Dimitris Papailiopoulos. My current research interests revolve around Large Language Models, and both theoretical and practical aspects of Machine Learning and Deep Learning.
Before coming to Madison, I earned my BSc and integrated MEng from the ECE department of the Technical University of Crete. During this time, I had the opportunity to work with Prof. Aggelos Bletsas, focusing on asynchronous inference algorithms for ambiently powered wireless sensor networks [Diploma Thesis].
Recent studies have shown that Large Language Models (LLMs) struggle to accurately retrieve information and maintain reasoning capabilities when processing long-context inputs. To address these limitations, we propose a finetuning approach utilizing a carefully designed synthetic dataset comprising numerical key-value retrieval tasks. Our experiments on models like GPT-3.5 Turbo and Mistral 7B demonstrate that finetuning LLMs on this dataset significantly improves LLMs’ information retrieval and reasoning capabilities in longer-context settings. We present an analysis of the finetuned models, illustrating the transfer of skills from synthetic to real task evaluations (e.g., 10.5% improvement on 20 documents MDQA at position 10 for GPT-3.5 Turbo). We also find that finetuned LLMs’ performance on general benchmarks remains almost constant while LLMs finetuned on other baseline long-context augmentation data can encourage hallucination (e.g., on TriviaQA, Mistral 7B finetuned on our synthetic data cause no performance drop while other baseline data can cause a drop that ranges from 2.33% to 6.19%). Our study highlights the potential of finetuning on synthetic data for improving the performance of LLMs on longer-context tasks.
@article{xiong2024artificialneedlesrealhaystacks,title={From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data},author={Xiong, Zheyang and Papageorgiou, Vasilis and Lee, Kangwook and Papailiopoulos, Dimitris},journal={arXiv preprint arXiv:2406.19292},year={2024},}
