Do LLMs Agree on the Creativity Evaluation of Alternative Uses?

E. P. Torrance, “Torrance tests of creative thinking,” Educational and Psychological Measurement , 1966. [Online]. Available: https://doi.org/10.1037/t05532-000

1966

Earlier work this paper cites.

J. P. Guilford, “Creativity: Yesterday, today and tomorrow,” The Journal of Creative Behavior , vol. 1, no. 1, pp. 3–14, 1967. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/j.2162-6057.1967.tb00002.x

1967

Earlier work this paper cites.

A. Jordanous, “A standardised procedure for evaluating creative systems: Computational creativity evaluation based on what it is to be creative,” Cognitive Computation , vol. 4, no. 3, pp. 246–279, 2012. [Online]. Available: https://kar.kent.ac.uk/42379/

2012

Earlier work this paper cites.

C. França, L. F. W. Góes, A. Amorim, R. Rocha, and A. R. Da Silva, “Regent-dependent creativity: A domain independent metric for the assessment of creative artifacts,” in Proceedings of the Seventh International Conference on Computational Creativity . Citeseer, 2016, pp. 68–75

2016

Earlier work this paper cites.

R. E. Beaty and D. R. Johnson, “Automating creativity assessment with semdis: An open platform for computing semantic distance,” Behavior Research Methods , vol. 53, pp. 757–780, 2021. [Online]. Available: https://doi.org/10.3758/s13428-020-01453-w

2021

Earlier work this paper cites.

C. Stevenson, I. Smal, M. Baas, R. Grasman, and H. van der Maas, “Putting gpt-3’s creativity to the (alternative uses) test,” in Proceedings of the International Conference on Computational Creativity 2022 . Association for Computational Creativity (ACC), 2022, pp. 164–168. [Online]. Available: https://arxiv.org/abs/2206.08932

Original

2022

Earlier work this paper cites.

T. Yang, Q. Zhang, Z. Sun, and Y. Hou, “Automatic assessment of divergent thinking in chinese language with transdis: A transformer-based language model approach,” Behavior Research Methods , vol. 56, no. 6, p. 5798–5819, Dec. 2023. [Online]. Available: http://dx.doi.org/10.3758/s13428-023-02313-z

2023

Earlier work this paper cites.

F. Góes, P. Sawicki, M. Grzes, M. Volpe, and J. Watson, “Pushing gpt’s creativity to its limits: Alternative uses and torrance tests,” in Proceedings of the 14th International Conference on Computational Creativity, Ontario, Canada, June 19-23, 2023 , A. Pease, J. M. Cunha, M. Ackerman, and D. G. Brown, Eds. Association for Computational Creativity (ACC), 2023, pp. 342–346. [Online]. Available: https://computationalcreativity.net/iccc23/papers/ICCC-2023_paper_90.pdf

2023

Earlier work this paper cites.

L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” 2023. [Online]. Available: https://arxiv.org/abs/2306.05685

Original

2023

Earlier work this paper cites.

F. Góes, P. Sawicki, M. Grzes, M. Volpe, and D. Brown, “Is GPT-4 good enough to evaluate jokes?” in Proceedings of the 14th International Conference on Computational Creativity, Ontario, Canada, June 19-23, 2023 , A. Pease, J. M. Cunha, M. Ackerman, and D. G. Brown, Eds. Association for Computational Creativity (ACC), 2023, pp. 367–371. [Online]. Available: https://computationalcreativity.net/iccc23/papers/ICCC-2023_paper_89.pdf

2023

Earlier work this paper cites.

P. Sawicki, M. Grzes, F. Góes, A. Jordanous, D. Brown, S. Paraskevopoulou, M. Peeperkorn, and A. Khatun, “On the power of special-purpose GPT models to create and evaluate new poetry in old styles,” in Proceedings of the 14th International Conference on Computational Creativity, Ontario, Canada, June 19-23, 2023 , A. Pease, J. M. Cunha, M. Ackerman, and D. G. Brown, Eds. Association for Computational Creativity (ACC), 2023, pp. 10–19. [Online]. Available: https://computationalcreativity.net/iccc23/papers/ICCC-2023_paper_18.pdf

2023

Earlier work this paper cites.

P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui, “Large language models are not fair evaluators,” 2023. [Online]. Available: https://arxiv.org/abs/2305.17926

Original

2023

Cited alongside, same era.

C.-H. Chiang and H. yi Lee, “Can large language models be an alternative to human evaluations?” 2023. [Online]. Available: https://arxiv.org/abs/2305.01937

Original

2023

Cited alongside, same era.

J. Fu, S.-K. Ng, Z. Jiang, and P. Liu, “Gptscore: Evaluate as you desire,” 2023. [Online]. Available: https://arxiv.org/abs/2302.04166

Original

2023

Cited alongside, same era.

C.-H. Chiang and H. yi Lee, “A closer look into automatic evaluation using large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2310.05657

Original

2023

Cited alongside, same era.

G. Franceschelli and M. Musolesi, “Creative beam search: Llm-as-a-judge for improving response generation,” 2024. [Online]. Available: https://arxiv.org/abs/2405.00099

Original

2024

Closest in time.

M. Desmond, Z. Ashktorab, Q. Pan, C. Dugan, and J. M. Johnson, “Evalullm: Llm assisted evaluation of generative outputs,” in Companion Proceedings of the 29th International Conference on Intelligent User Interfaces , ser. IUI ’24 Companion. New York, NY, USA: Association for Computing Machinery, 2024, p. 30–32. [Online]. Available: https://doi.org/10.1145/3640544.3645216

2024

Closest in time.

B. Goecke, P. V. DiStefano, W. Aschauer, K. Haim, R. Beaty, and B. Forthmann, “Automated scoring of scientific creativity in german,” The Journal of Creative Behavior , 2024. [Online]. Available: https://doi.org/10.1002/jocb.658

2024

Closest in time.

T. Raz, S. Luchini, R. Beaty, and Y. Kenett, “Automated scoring of open-ended question complexity: A large language model approach,” 2024. [Online]. Available: https://doi.org/10.21203/rs.3.rs-3890828/v1