SANGWON YOON

We study the economic consequences of managers' vocal delivery quality during earnings conference calls. We introduce a novel measure, vocal delivery quality, that captures the acoustic comprehensibility of audio information for an average listener. Our measure relies on a deep-learning algorithm applied to a large sample of earnings call audio files. Consistent with predictions from the psychology and accounting literatures, we find evidence that the quality of managers' vocal delivery deteriorates when they deliver negative news, such as a decrease in earnings or negative narrative information, and positive but transitory earnings news. We show that the stock market reacts in real time to managers' vocal delivery quality. We also document that the vocal delivery quality has an effect on information intermediaries such as analysts and the media. Overall, our findings underscore the role of vocal dimensions in financial communication.

As natural language generation (NLG) models have become prevalent, systematically assessing the quality of machine-generated texts has become increasingly important. Recent studies introduce LLM-based evaluators that operate as reference-free metrics, demonstrating their capability to adeptly handle novel tasks. However, these models generally rely on a single-agent approach, which, we argue, introduces an inherent limit to their performance. This is because there exist biases in LLM agent's responses, including preferences for certain text structure or content. In this work, we propose DEBATE, an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil's Advocate. Within the framework, one agent is instructed to criticize other agents' arguments, potentially resolving the bias in LLM agent's answers. DEBATE substantially outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat. We also show that the extensiveness of debates among agents and the persona of an agent can influence the performance of evaluators.