Fetching the paper…

Reading the bibliography…

2023

Benchmarking Large Language Model Volatility

Yu, Boyang

Original
PDF
Post on X

Understand

The impact of non-deterministic outputs from Large Language Models (LLMs) is not well examined for financial text understanding tasks.

Through a compelling case study on investing in the US equity market via news sentiment analysis, we uncover substantial variability in sentence-level sentiment classification results, underscoring the innate volatility of LLM outputs.
These uncertainties cascade downstream, leading to more significant variations in portfolio construction and return.
While tweaking the temperature parameter in the language model decoder presents a potential remedy, it comes at the expense of stifled creativity.

Benchmarking Large Language Model Volatility · Around

Built on

Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., and Ghahramani, Z · 2016
Earlier work this paper cites.
Simple and scalable predictive uncertainty estimation using deep ensembles
Lakshminarayanan, B., Pritzel, A., and Blundell, C · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Earlier work this paper cites.
Deep probability estimation
Original
Liu, S., Kaku, A., Zhu, W., Leibovich, M., Mohan, S., Yu, B., Huang, H., Zanna, L., Razavian, N., Niles-Weed, J., et al · 2021
Earlier work this paper cites.
Quantifying social biases using templates is unreliable
Original
Seshadri, P., Pezeshkpour, P., and Singh, S · 2022
Earlier work this paper cites.

Similar

Leancontext: Cost-efficient domain-specific question answering using llms
Original
Arefeen, M. A., Debnath, B., and Chakradhar, S · 2023
Cited alongside, same era.
What comes next? evaluating uncertainty in neural text generators against human production variability
Original
Giulianelli, M., Baan, J., Aziz, W., Fernández, R., and Plank, B · 2023
Cited alongside, same era.
News signals: An nlp library for text and time series
Hokamp, C., Ghalandari, D. G., and Ghaffari, P · 2023
Cited alongside, same era.
Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation
Original
Kuhn, L., Gal, Y., and Farquhar, S · 2023
Cited alongside, same era.
Generating with confidence: Uncertainty quantification for black-box large language models
Original

Then

Can chatgpt forecast stock price movements? return predictability and large language models
Original
Lopez-Lira, A., and Tang, Y · 2023
Closest in time.
Gpt-4 technical report, 2023
OpenAI · 2023
Closest in time.
Llama: Open and efficient foundation language models
Original
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al · 2023
Closest in time.
Can chatgpt predict future interest rate decisions?
Woodhouse, D., and Charlesworth, A · 2023
Closest in time.
Large language models can rate news outlet credibility
Original
Yang, K.-C., and Menczer, F · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Lin, Z., Trivedi, S., and Sun, J · 2023

Cited alongside, same era.