Fetching the paper…

Reading the bibliography…

2024

PhysBERT: A Text Embedding Model for Physics Scientific Literature

Hellert, Thorsten, Montenegro, João, Pollastro, Andrea

Original
PDF
Post on X

Understand

The specialized language and complex concepts in physics pose significant challenges for information extraction through Natural Language Processing (NLP).

Central to effective NLP applications is the text embedding model, which converts text into dense vector representations for efficient information retrieval and semantic analysis.
In this work, we introduce PhysBERT, the first physics-specific text embedding model.
Pre-trained on a curated corpus of 1.2 million arXiv physics papers and fine-tuned with supervised data, PhysBERT outperforms leading general-purpose models on physics-specific tasks including the effectiveness in fine-tuning for specific physics subdomains.

PhysBERT: A Text Embedding Model for Physics Scientific Literature · Around

Built on

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, Roberta: A robustly optimized bert pretraining approach (2019), arXiv:1907.11692 [cs.CL]
Original
1907
Earlier work this paper cites.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, Huggingface’s transformers: State-of-the-art natural language processing (2020), arXiv:1910.03771 [cs.CL]
Original
1910
Earlier work this paper cites.
A. Singhal et al. , Modern information retrieval: A brief overview, IEEE Data Eng. Bull. 24
2001
Earlier work this paper cites.
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, and D. Kiela, Retrieval-augmented generation for knowledge-intensive nlp tasks (2021), arXiv:2005.11401 [cs.CL]
Original
2005
Earlier work this paper cites.
S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala, Pytorch distributed: Experiences on accelerating data parallel training (2020), arXiv:2006.15704 [cs.DC]
Original
2006
Earlier work this paper cites.
A. Rosenberg and J. Hirschberg, V-measure: A conditional entropy-based external cluster evaluation measure, in Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) , edited by J. Eisner (Association for Computational Linguistics, Prague, Czech Republic, 2007) pp. 410–420
2007
Earlier work this paper cites.
A. Rosenberg and J. Hirschberg, V-measure: A conditional entropy-based external cluster evaluation measure, in Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) , edited by J. Eisner (Association for Computational Linguistics, Prague, Czech Republic, 2007) pp. 410–420
2007
Earlier work this paper cites.
C. D. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval (Cambridge University Press, Cambridge, UK, 2008)
2008
Earlier work this paper cites.
T. Mikolov, K. Chen, G. Corrado, and J. Dean, Efficient estimation of word representations in vector space (2013), arXiv:1301.3781 [cs.CL]
Original
2013
Earlier work this paper cites.
Y. Wang, L. Wang, Y. Li, D. He, T.-Y. Liu, and W. Chen, A theoretical analysis of ndcg type ranking measures (2013), arXiv:1304.6480 [cs.LG]
Original
2013
Earlier work this paper cites.
Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, nature 521
2015
Earlier work this paper cites.
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, Deep contextualized word representations, in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , edited by M. Walker, H. Ji, and A. Stent (Association for Computational Linguistics, New Orleans, Louisiana, 2018) pp. 2227–2237
2018
Earlier work this paper cites.
J. Howard and S. Ruder, Universal language model fine-tuning for text classification, in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , edited by I. Gurevych and Y. Miyao (Association for Computational Linguistics, Melbourne, Australia, 2018) pp. 328–339
2018
Earlier work this paper cites.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, A. Müller, J. Nothman, G. Louppe, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Édouard Duchesnay, Scikit-learn: Machine learning in python (2018), arXiv:1201.0490 [cs.LG]
Original
2018
Earlier work this paper cites.
N. Reimers and I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (Association for Computational Linguistics, 2019)
Original
2019
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding (2019), arXiv:1810.04805 [cs.CL]
Original
2019
Earlier work this paper cites.
I. Beltagy, K. Lo, and A. Cohan, Scibert: A pretrained language model for scientific text, arXiv preprint arXiv:1903.10676 (2019)
Original
2019
Earlier work this paper cites.
J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics 36
2019
Earlier work this paper cites.
National energy research scientific computing center (2020)
2020
Earlier work this paper cites.

Similar

B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heinz, and D. Roth, Recent advances in natural language processing via large pre-trained language models: A survey (2021), arXiv:2111.01243 [cs.CL]
Original
2021
Cited alongside, same era.
N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych, Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models (2021), arXiv:2104.08663 [cs.IR]
Original
2021
Cited alongside, same era.
R. van Dinter, B. Tekinerdogan, and C. Catal, Automation of systematic literature reviews: A systematic literature review, Information and Software Technology 136
2021
Cited alongside, same era.
L. Gao, Y. Zhang, J. Han, and J. Callan, Scaling deep contrastive learning batch size under memory limited setup (2021), arXiv:2101.06983 [cs.LG]
Original
2021
Cited alongside, same era.

Then

Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang, Retrieval-augmented generation for large language models: A survey (2024), arXiv:2312.10997 [cs.CL]
Original
2024
Closest in time.
H. Face, Massively multilingual text embedding benchmark (mteb) leaderboard, https://huggingface.co/spaces/mteb/leaderboard (2024), accessed: 2024-07-28
2024
Closest in time.
A. Hexemer, Exploration of a beamline chatbot (2024), unpublished
2024
Closest in time.
A. Sulc et al. , Towards Unlocking Insights from Logbooks Using AI, in 15th International Particle Accelerator Conference (2024) arXiv:2406.12881 [physics.acc-ph]
Original
2024
Closest in time.
F. Rehm, AccGPT - The Current Vision for AI Assistance at CERN’s Accelerator Control and Beyond, Talk at the 1st Large Language Models in Physics Symposium (2024)

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

A. Singh, M. D’Arcy, A. Cohan, D. Downey, and S. Feldman, Scirepeval: A multi-format benchmark for scientific document representations, in Conference on Empirical Methods in Natural Language Processing (2022)

2022

Cited alongside, same era.

T. Gupta, M. Zaki, N. M. A. Krishnan, and Mausam, Matscibert: A materials domain language model for text mining and information extraction, npj Computational Materials 8

2022

Cited alongside, same era.

T. Gao, X. Yao, and D. Chen, Simcse: Simple contrastive learning of sentence embeddings (2022), arXiv:2104.08821 [cs.CL]

2022

Cited alongside, same era.

M. Grootendorst, Bertopic: Neural topic modeling with a class-based tf-idf procedure (2022), arXiv:2203.05794 [cs.CL]

2022

Cited alongside, same era.

Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, Survey of hallucination in natural language generation, ACM Computing Surveys 55

2023

Cited alongside, same era.

N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, Mteb: Massive text embedding benchmark (2023), arXiv:2210.07316 [cs.CL]

2023

Cited alongside, same era.

S. Shermukhamedov, D. Mamurjonova, and M. Probst, Structure to property: Chemical element embeddings and a deep learning approach for accurate prediction of chemical properties (2023), arXiv:2309.09355 [physics.chem-ph]

2023

Cited alongside, same era.

N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen, J. Yi, W. Zhao, X. Wang, Z. Liu, H. Zheng, J. Chen, Y. Liu, J. Tang, J. Li, and M. Sun, Parameter-efficient fine-tuning of large-scale pre-trained language models, Nature Machine Intelligence 5

2023

Cited alongside, same era.

2024

Closest in time.

D. Murnane, G. Facini, R. Li, D. D. Santo, and C. Randazzo, chATLAS - An AI Assistant for the ATLAS Collaboration, Talk at the 1st Large Language Models in Physics Symposium (2024)

2024

Closest in time.

P. Steinbach, T. Niehoff, and T. Gottschall, Extracting Measurements from (legacy) Publications, Talk at the 1st Large Language Models in Physics Symposium (2024)

2024

Closest in time.

arXiv, https://arxiv.org (a), accessed: 2024-07-12

2024

Closest in time.

Physbert model and dataset collection, https://huggingface.co/collections/thellert/physbert-66c21ee8e61ccd71d7d4414a (2024)

2024

Closest in time.

E. Chagnon, R. Pandolfi, J. Donatelli, and D. Ushizima, Benchmarking topic models on scientific articles using berteley, Natural Language Processing Journal 6

2024

Closest in time.

Semantic Scholar, https://www.semanticscholar.org/ , accessed: 2024-07-23

2024

Closest in time.

arXiv, arxiv category taxonomy, https://arxiv.org/category_taxonomy (b), accessed: 2024-07-12

2024

Closest in time.

arXiv, arxiv category taxonomy, https://info.arxiv.org/help/bulk_data_s3.html (c), accessed: 2024-07-12

2024

Closest in time.

R. Liu, J. Wei, F. Liu, C. Si, Y. Zhang, J. Rao, S. Zheng, D. Peng, D. Yang, D. Zhou, and A. M. Dai, Best practices and lessons learned on synthetic data for language models (2024), arXiv:2404.07503 [cs.CL]

2024

Closest in time.

L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei, Text embeddings by weakly-supervised contrastive pre-training (2024), arXiv:2212.03533 [cs.CL]

2024

Closest in time.

N. Reimers and I. Gurevych, Sentencetransformers/all-minilm-l6-v2, https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 (2020a), accessed: 2024-07-21

2024

Closest in time.

N. Reimers and I. Gurevych, Sentencetransformers/all-mpnet-base-v2, https://huggingface.co/sentence-transformers/all-mpnet-base-v2 (2020b), accessed: 2024-07-21

2024

Closest in time.

X. Li and J. Li, Angle-optimized text embeddings (2024), arXiv:2309.12871 [cs.CL]

2024

Closest in time.

S. Lee, A. Shakir, D. Koenig, and J. Lipp, Open source strikes bread - new fluffy embeddings model (2024)

2024

Closest in time.