German Summarization with Large Language Models

Abstract

In this thesis, we analyze the effectiveness of Large Language Models (LLMs) on zero-shot Single-Document Summarization (SDS) of German news articles, evaluate the performance of LLMs in Multi-Document Summarization (MDS), and propose methods to improve MDS using LLMs by applying In-Context Learning (ICL) with LLMs with a limited context window. Our results show that while larger LLMs have advantages for effective SDS, effective SDS performance does not depend solely on model size. Rather, it depends on a combination of factors, including prompt design, properly tuned models, and understanding the inherent tendencies of the models used. Our results show that current open-source models can match the performance of state-of-the-art proprietary models, including GPT-4 and PaLM2, in the SDS task. A central focus of this thesis is to evaluate and propose methods to improve MDS using LLMs by applying ICL. In order to be able to apply ICL using LLMs with a limited context window size, we propose a novel use of existing extractive summarization techniques in combination with ICL to improve over zero-shot summarization for MDS. It exploits the capabilities of extractive summarization techniques to identify salient information not to produce short summaries but to identify and assemble salient information into more detailed compressed documents. These documents are then summarized using LLMs. The proposed methods allow some control over the summarization through the selection of examples and are not limited to the German language but have the potential to be generalized to other languages. Based on the evaluation of both single-document and multi-document summarization using LLMs, we show the effective- ness of LLMs in solving both tasks for German news articles.