A popular language model that can solve introductory programming problems, OpenAI’s Codex, has drawn much attention not only in the natural language processing field but also in the software engineering field. It supports programmers by suggesting the next tokens to write, and it can even generate a whole function definition from a document string. We focus on its capability of automatically solving programming problems through code generation from problem descriptions. We investigate the model’s sensitivity to problem descriptions by formatting and modifying them. The experimental results show that the more explicitly formatted problem description enhances the code generation performance from 30.9% (raw) to 39.9% (formatted). Additionally, we observe that code generation relies on information specified in the problem description, such as variable names and constant values, as anonymizing them reduces the performance significantly. Moreover, statistical biases in code generation are identified, such as the generated programs ignoring the problem modification and answering the exact opposite problem. The changes in accuracy across formats suggest that the model does not correctly understand the natural language explaining the problem specification even if the model could solve the programming problems with high accuracy.
IOS Press, Inc.
6751 Tepper Drive
Clifton, VA 20124
Tel.: +1 703 830 6300
Fax: +1 703 830 2300 firstname.lastname@example.org
(Corporate matters and books only) IOS Press c/o Accucoms US, Inc.
For North America Sales and Customer Service
West Point Commons
Lansdale PA 19446
Tel.: +1 866 855 8967
Fax: +1 215 660 5042 email@example.com