Fetching the paper…

Reading the bibliography…

2023

ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning

Liu, Shangqing, Wu, Bozhi, Xie, Xiaofei et al.

Original
PDF
Post on X

Understand

Large-scale pre-trained models such as CodeBERT, GraphCodeBERT have earned widespread attention from both academia and industry.

Attributed to the superior ability in code representation, they have been further applied in multiple downstream tasks such as clone detection, code search and code translation.
However, it is also observed that these state-of-the-art pre-trained models are susceptible to adversarial attacks.
The performance of these pre-trained models drops significantly with simple perturbations such as renaming variable names.

ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning · Around

Built on

S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.
L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research , vol. 9, no. 11, 2008
2008
Earlier work this paper cites.
J. Wang and J. Zhu, “Portfolio theory of information retrieval,” in Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval , 2009, pp. 115–122
2009
Earlier work this paper cites.
M. M. Palatucci, D. A. Pomerleau, G. E. Hinton, and T. Mitchell, “Zero-shot learning with semantic output codes,” 2009
2009
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems , vol. 25, pp. 1097–1105, 2012
2012
Earlier work this paper cites.
F. Yamaguchi, N. Golde, D. Arp, and K. Rieck, “Modeling and discovering vulnerabilities with code property graphs,” in 2014 IEEE Symposium on Security and Privacy . IEEE, 2014, pp. 590–604
2014
Earlier work this paper cites.
J. Svajlenko, J. F. Islam, I. Keivanloo, C. K. Roy, and M. M. Mia, “Towards a big data curated benchmark of inter-project code clones,” in 2014 IEEE International Conference on Software Maintenance and Evolution . IEEE, 2014, pp. 476–480
2014
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473 , 2014
Original
2014
Earlier work this paper cites.
Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel, “Gated graph sequence neural networks,” arXiv preprint arXiv:1511.05493 , 2015
Original
2015
Earlier work this paper cites.
R. Sennrich, B. Haddow, and A. Birch, “Improving neural machine translation models with monolingual data,” arXiv preprint arXiv:1511.06709 , 2015
Original
2015
Earlier work this paper cites.
L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin, “Convolutional neural networks over tree structures for programming language processing,” in Thirtieth AAAI Conference on Artificial Intelligence , 2016
2016
Earlier work this paper cites.
M. Allamanis, M. Brockschmidt, and M. Khademi, “Learning to represent programs with graphs,” arXiv preprint arXiv:1711.00740 , 2017
Original
2017
Earlier work this paper cites.
A. V. M. Barone and R. Sennrich, “A parallel corpus of python functions and documentation strings for automated code documentation and code generation,” arXiv preprint arXiv:1707.02275 , 2017
Original
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems , 2017, pp. 5998–6008
2017
Earlier work this paper cites.
B. Dai and D. Lin, “Contrastive learning for image captioning,” arXiv preprint arXiv:1710.02534 , 2017
Original
2017
Earlier work this paper cites.
M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, “A survey of machine learning for big code and naturalness,” ACM Computing Surveys (CSUR) , vol. 51, no. 4, pp. 1–37, 2018
2018
Earlier work this paper cites.
“The github blog,” https://github.blog/2018-11-08-100m-repos/
2018
Earlier work this paper cites.
U. Alon, S. Brody, O. Levy, and E. Yahav, “code2seq: Generating sequences from structured representations of code,” arXiv preprint arXiv:1808.01400 , 2018
Original
2018
Earlier work this paper cites.
X. Gu, H. Zhang, and S. Kim, “Deep code search,” in 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE) . IEEE, 2018, pp. 933–944
2018
Earlier work this paper cites.
R. Russell, L. Kim, L. Hamilton, T. Lazovich, J. Harer, O. Ozdemir, P. Ellingwood, and M. McConley, “Automated vulnerability detection in source code using deep representation learning,” in 2018 17th IEEE international conference on machine learning and applications (ICMLA) . IEEE, 2018, pp. 757–762
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018
Original
2018
Earlier work this paper cites.
Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 3733–3742
2018
Earlier work this paper cites.
X. Chen, C. Liu, and D. Song, “Tree-to-tree neural networks for program translation,” arXiv preprint arXiv:1802.03691 , 2018
Original
2018
Earlier work this paper cites.
Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” arXiv preprint arXiv:1801.01681 , 2018
Original
2018
Earlier work this paper cites.

Similar

Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks,” arXiv preprint arXiv:1909.03496 , 2019
Original
2019
Cited alongside, same era.
U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “code2vec: Learning distributed representations of code,” Proceedings of the ACM on Programming Languages , vol. 3, no. POPL, pp. 1–29, 2019
2019
Cited alongside, same era.
H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019
Original
2019
Cited alongside, same era.
M. Allamanis, “The adverse effects of code duplication in machine learning models of code,” in Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software , 2019, pp. 143–153
2019
Cited alongside, same era.

Then

M. Kim, J. Tack, and S. J. Hwang, “Adversarial self-supervised contrastive learning,” arXiv preprint arXiv:2006.07589 , 2020
Original
2020
Later among the works it cites.
H. Fang, S. Wang, M. Zhou, J. Ding, and P. Xie, “Cert: Contrastive self-supervised learning for language understanding,” arXiv preprint arXiv:2005.12766 , 2020
Original
2020
Later among the works it cites.
D. Shen, M. Zheng, Y. Shen, Y. Qu, and W. Chen, “A simple but tough-to-beat data augmentation approach for natural language understanding and generation,” arXiv preprint arXiv:2009.13818 , 2020
Original
2020
Later among the works it cites.
N. Yefet, U. Alon, and E. Yahav, “Adversarial examples for models of code,” Proceedings of the ACM on Programming Languages , vol. 4, no. OOPSLA, pp. 1–30, 2020
2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi, “Pre-trained contextual embedding of source code,” 2019

2019

Cited alongside, same era.

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

2019

Cited alongside, same era.

E. Ma, “Nlp augmentation,” https://github.com/makcedward/nlpaug, 2019

2019

Cited alongside, same era.

M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,” ACM Transactions on Software Engineering and Methodology (TOSEM) , vol. 28, no. 4, pp. 1–29, 2019

2019

Cited alongside, same era.

Z. Yang, Y. Cheng, Y. Liu, and M. Sun, “Reducing word omission errors in neural machine translation: A contrastive learning approach,” 2019

2019

Cited alongside, same era.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

2019

Cited alongside, same era.

S. Liu, Y. Chen, X. Xie, J. K. Siow, and Y. Liu, “Retrieval-augmented generation for code summarization via hybrid gnn,” in International Conference on Learning Representations , 2020

2020

Cited alongside, same era.

U. Alon, R. Sadaka, O. Levy, and E. Yahav, “Structural language models of code,” in International Conference on Machine Learning . PMLR, 2020, pp. 245–256

2020

Cited alongside, same era.

G. Ramakrishnan, J. Henkel, Z. Wang, A. Albarghouthi, S. Jha, and T. Reps, “Semantic robustness of models of source code,” arXiv preprint arXiv:2002.03043 , 2020

2020

Later among the works it cites.

P. Bielik and M. Vechev, “Adversarial robustness for code,” in International Conference on Machine Learning . PMLR, 2020, pp. 896–907

2020

Later among the works it cites.

H. Zhang, Z. Li, G. Li, L. Ma, Y. Liu, and Z. Jin, “Generating adversarial examples for holding robustness of source code processing models,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 34, no. 01, 2020, pp. 1169–1176

2020

Later among the works it cites.

S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al. , “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664 , 2021

2021

Later among the works it cites.

M. Allamanis, H. Jackson-Flux, and M. Brockschmidt, “Self-supervised bug detection and repair,” arXiv preprint arXiv:2105.12787 , 2021

2021

Later among the works it cites.

S. Liu, X. Xie, L. Ma, J. Siow, and Y. Liu, “Graphsearchnet: Enhancing gnns via capturing global dependency for semantic code search,” arXiv preprint arXiv:2111.02671 , 2021

2021

Later among the works it cites.

A. Svyatkovskiy, S. Lee, A. Hadjitofi, M. Riechert, J. V. Franco, and M. Allamanis, “Fast and memory-efficient neural code completion,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) . IEEE, 2021, pp. 329–340

2021

Later among the works it cites.

Y. Zhou, J. K. Siow, C. Wang, S. Liu, and Y. Liu, “Spi: Automated identification of security patches via commits,” ACM Transactions on Software Engineering and Methodology (TOSEM) , vol. 31, no. 1, pp. 1–27, 2021

2021

Later among the works it cites.

Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021

Original

2021

Later among the works it cites.

W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” arXiv preprint arXiv:2103.06333 , 2021

2021

Later among the works it cites.

N. D. Bui, Y. Yu, and L. Jiang, “Self-supervised contrastive learning for code retrieval and summarization via semantic-preserving transformations,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2021, pp. 511–521

2021

Later among the works it cites.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” arXiv preprint arXiv:2103.00020 , 2021

2021

Later among the works it cites.

T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” arXiv preprint arXiv:2104.08821 , 2021

2021

Later among the works it cites.

Q. Chen, J. Lacomis, E. J. Schwartz, G. Neubig, B. Vasilescu, and C. L. Goues, “Varclr: Variable semantic representation pre-training via contrastive learning,” arXiv preprint arXiv:2112.02650 , 2021

2021

Later among the works it cites.

S. Srikant, S. Liu, T. Mitrovska, S. Chang, Q. Fan, G. Zhang, and U.-M. O’Reilly, “Generating adversarial computer programs using optimized obfuscations,” arXiv preprint arXiv:2103.11882 , 2021

2021

Later among the works it cites.

B. Wu, S. Liu, R. Feng, X. Xie, J. Siow, and S.-W. Lin, “Enhancing security patch identification by capturing structures in commits,” IEEE Transactions on Dependable and Secure Computing , 2022

2022

Later among the works it cites.

X. Li, S. Liu, R. Feng, G. Meng, X. Xie, K. Chen, and Y. Liu, “Transrepair: Context-aware program repair for compilation errors,” arXiv preprint arXiv:2210.03986 , 2022

2022

Later among the works it cites.

Z. Yang, J. Shi, J. He, and D. Lo, “Natural attack for pre-trained models of code,” arXiv preprint arXiv:2201.08698 , 2022

2022

Later among the works it cites.

S. Liu, Y. Li, and Y. Liu, “Commitbart: A large pre-trained model for github commits,” arXiv preprint arXiv:2208.08100 , 2022

2022

Later among the works it cites.

S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2016, pp. 2073–2083

2083

Closest in time.