2020

Exploring Software Naturalness through Neural Language Models

Buratti, Luca, Pujar, Saurabh, Bornea, Mihaela et al.

Understand

The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing.

We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks.
Present approaches to code analysis depend heavily on features derived from the Abstract Syntax Tree (AST) while our transformer-based language models work on raw source code.
This work is the first to investigate whether such language models can discover AST features automatically.

Built on

The C Programming Language
Kernighan, B. W., and Ritchie, D. M · 1988
Earlier work this paper cites.
Foundations of Statistical Natural Language Processing
Manning, C. D., and Schütze, H · 1999
Earlier work this paper cites.
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., and Devanbu, P · 2012
Earlier work this paper cites.
Rough Audit Tool for Security
Inc., S. S · 2013
Earlier work this paper cites.
Why don’t software developers use static analysis tools to find bugs?
Johnson, B., Song, Y., Murphy-Hill, E., and Bowdidge, R · 2013
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J · 2013
Earlier work this paper cites.
Code completion with statistical language models
Raychev, V., Vechev, M., and Yahav, E · 2014
Earlier work this paper cites.
Modeling and discovering vulnerabilities with code property graphs
Yamaguchi, F., Golde, N., Arp, D., and Rieck, K · 2014
Earlier work this paper cites.
github.com/microsoft/gated-graph-neural-network-samples , 2015
Microsoft · 2015
Earlier work this paper cites.
Predicting program properties from “big code”
Raychev, V., Vechev, M., and Krause, A · 2015
Earlier work this paper cites.
Gated graph sequence neural networks
Li, Y., Tarlow, D., Brockschmidt, M., and Zemel, R · 2016
Earlier work this paper cites.
On the "naturalness" of buggy code
Ray, B., Hellendoorn, V., Godhane, S., Tu, Z., Bacchelli, A., and Devanbu, P · 2016
Earlier work this paper cites.
"neural machine translation of rare words with subword units"
"Sennrich, R., Haddow, B., and Birch, A · 2016
Earlier work this paper cites.
Are deep neural networks the best choice for modeling source code?
Hellendoorn, V. J., and Devanbu, P · 2017
Earlier work this paper cites.
Juliet test suite v1.3
NIST · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I · 2017
Earlier work this paper cites.

Similar

Analyzing false positive source code vulnerabilities using static analysis tools
Cheirdari, F., and Karabatis, G · 2018
Cited alongside, same era.
Deep learning type inference
Hellendoorn, V. J., Bird, C., Barr, E. T., and Allamanis, M · 2018
Cited alongside, same era.
SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Kudo, T., and Richardson, J · 2018
Cited alongside, same era.
Syntax and sensibility: Using language models to detect and correct syntax errors
Santos, E. A., Campbell, J. C., Patel, D., Hindle, A., and Amaral, J. N · 2018
Cited alongside, same era.
Flawfinder
Wheeler, D.A · 2018
Cited alongside, same era.
The adverse effects of code duplication in machine learning models of code
Allamanis, M · 2019
Cited alongside, same era.
Capturing source code semantics via tree-based convolution over api-enhanced ast
Chen, L., Ye, W., and Zhang, S · 2019
Cited alongside, same era.
Sequencer: Sequence-to-sequence learning for end-to-end program repair
Chen, Z., Kommrusch, S. J., Tufano, M., Pouchet, L., Poshyvanyk, D., and Monperrus, M · 2019
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2019
Cited alongside, same era.
Smt-based refutation of spurious bug reports in the clang static analyzer
Gadelha, M. R., Steffinlongo, E., Cordeiro, L. C., Fischer, B., and Nicole, D. A · 2019
Cited alongside, same era.
Deep transfer learning for source code modeling
Original
Hussain, Y., Huang, Z., Zhou, Y., and Wang, S · 2019
Cited alongside, same era.
Pre-trained contextual embedding of source code
Original
Kanade, A., Maniatis, P., Balakrishnan, G., and Shi, K · 2019
Cited alongside, same era.

Then

Typewriter: Neural type prediction with search-based validation
Original
Pradel, M., Gousios, G., Liu, J., and Chandra, S · 2019
Later among the works it cites.
Neural program repair by jointly learning to localize and repair
Original
Vasic, M., Kanade, A., Maniatis, P., Bieber, D., and Singh, R · 2019
Later among the works it cites.
Superglue: A stickier benchmark for general-purpose language understanding systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R · 2019
Later among the works it cites.
Huggingface’s transformers: State-of-the-art natural language processing
Original
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J · 2019
Later among the works it cites.
A novel neural source code representation based on abstract syntax tree
Zhang, J., Wang, X., Zhang, H., Sun, H., Wang, K., and Liu, X · 2019
Later among the works it cites.
A novel neural source code representation based on abstract syntax tree
Zhang, J., Wang, X., Zhang, H., Sun, H., Wang, K., and Liu, X · 2019
Later among the works it cites.
Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks
Zhou, Y., Liu, S., Siow, J., Du, X., and Liu, Y · 2019
Later among the works it cites.
Codebert: A pre-trained model for programming and natural languages
Original
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al · 2020
Closest in time.
Spanbert: Improving pre-training by representing and predicting spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., and Levy, O · 2020
Closest in time.
Big code != big vocabulary: Open-vocabulary models for source code
Original
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., and Janes, A · 2020
Closest in time.
Draper VDISC Dataset - Vulnerability Detection in Source Code
Louis Kim, Rebecca Russell · 2020
Closest in time.
Lambdanet: Probabilistic type inference using graph neural networks
Wei, J., Goyal, M., Durrett, G., and Dillig, I · 2020
Closest in time.
https://github.com/FFmpeg/FFmpeg , 2020
FFMPeg · 2020
Closest in time.
https://github.com/quemu/qemu , 2020
QEMU · 2020
Closest in time.
https://clang-analyzer.llvm.org , 2020
Clang Static Analyzer · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…