{"title": "Neural Code Comprehension: A Learnable Representation of Code Semantics", "book": "Advances in Neural Information Processing Systems", "page_first": 3585, "page_last": 3597, "abstract": "With the recent success of embeddings in natural language processing, research has been conducted into applying similar methods to code analysis. Most works attempt to process the code directly or use a syntactic tree representation, treating it like sentences written in a natural language. However, none of the existing methods are sufficient to comprehend program semantics robustly, due to structural features such as function calls, branching, and interchangeable order of statements. In this paper, we propose a novel processing technique to learn code semantics, and apply it to a variety of program analysis tasks. In particular, we stipulate that a robust distributional hypothesis of code applies to both human- and machine-generated programs. Following this hypothesis, we define an embedding space, inst2vec, based on an Intermediate Representation (IR) of the code that is independent of the source programming language. We provide a novel definition of contextual flow for this IR, leveraging both the underlying data- and control-flow of the program. We then analyze the embeddings qualitatively using analogies and clustering, and evaluate the learned representation on three different high-level tasks. We show that even without fine-tuning, a single RNN architecture and fixed inst2vec embeddings outperform specialized approaches for performance prediction (compute device mapping, optimal thread coarsening); and algorithm classification from raw code (104 classes), where we set a new state-of-the-art.", "full_text": "Neural Code Comprehension: A Learnable\n\nRepresentation of Code Semantics\n\nTal Ben-Nun\nETH Zurich\n\nZurich 8092, Switzerland\n\nAlice Shoshana Jakobovits\n\nETH Zurich\n\nZurich 8092, Switzerland\n\nTorsten Hoe\ufb02er\n\nETH Zurich\n\nZurich 8092, Switzerland\n\ntalbn@inf.ethz.ch\n\nalicej@student.ethz.ch\n\nhtor@inf.ethz.ch\n\nAbstract\n\nWith the recent success of embeddings in natural language processing, research\nhas been conducted into applying similar methods to code analysis. Most works\nattempt to process the code directly or use a syntactic tree representation, treating it\nlike sentences written in a natural language. However, none of the existing methods\nare suf\ufb01cient to comprehend program semantics robustly, due to structural features\nsuch as function calls, branching, and interchangeable order of statements. In this\npaper, we propose a novel processing technique to learn code semantics, and apply\nit to a variety of program analysis tasks. In particular, we stipulate that a robust\ndistributional hypothesis of code applies to both human- and machine-generated\nprograms. Following this hypothesis, we de\ufb01ne an embedding space, inst2vec,\nbased on an Intermediate Representation (IR) of the code that is independent of the\nsource programming language. We provide a novel de\ufb01nition of contextual \ufb02ow\nfor this IR, leveraging both the underlying data- and control-\ufb02ow of the program.\nWe then analyze the embeddings qualitatively using analogies and clustering, and\nevaluate the learned representation on three different high-level tasks. We show that\neven without \ufb01ne-tuning, a single RNN architecture and \ufb01xed inst2vec embeddings\noutperform specialized approaches for performance prediction (compute device\nmapping, optimal thread coarsening); and algorithm classi\ufb01cation from raw code\n(104 classes), where we set a new state-of-the-art.\n\n1\n\nIntroduction\n\nThe emergence of the \u201cBig Data era\u201d manifests in the form of a dramatic increase in accessible code.\nIn the year 2017 alone, GitHub reports [25] approximately 1 billion git commits (code modi\ufb01cation\nuploads) written in 337 different programming languages. Sifting through, categorizing, and under-\nstanding code thus becomes an essential task for a variety of \ufb01elds. Applications include identifying\ncode duplication, performance prediction, algorithm detection for alternative code suggestion (guided\nprogramming), vulnerability analysis, and malicious code detection. These tasks are challenging,\nas code can be modi\ufb01ed such that it syntactically differs (for instance, via different or reordered\noperations, or written in a different language altogether), but remains semantically equivalent (i.e.,\nproduces the same result). However, these tasks are also ideal for machine learning, since they can be\nrepresented as classic regression and classi\ufb01cation problems.\nIn order to mechanize code comprehension, the research community typically employs reinforcement\nlearning and stochastic compilation for super-optimization [13, 56]; or borrows concepts from Natural\nLanguage Processing (NLP) for human-authored code, relying on the following hypothesis:\nThe naturalness hypothesis [3]. Software is a form of human communication;\nsoftware corpora have similar statistical properties to natural language corpora;\nand these properties can be exploited to build better software engineering tools.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: Component overview of the Neural Code Comprehension pipeline.\n\nFor NLP-based approaches, input code is usually processed into tokens (e.g., keywords, braces) [18]\nor other representations [4, 7, 53], and optionally undergoes embedding in a continuous lower-\ndimensional space. In the spirit of the successful word2vec model [47, 48], the mapping to the\nembedding space is learned by pairing a token with its surrounding tokens. Following this process,\nRNNs [23] are trained on sequences of such tokens. This model has been successfully used for NLP-\nlike tasks, such as summarization [6], function name prediction [7], and algorithm classi\ufb01cation [49].\nAlthough the results for stochastic code optimization and NLP embeddings are promising, two issues\narise. Firstly, in prior works, the source programming language (or machine code for optimization) is\n\ufb01xed, which does not re\ufb02ect the plethora of languages, nor generalizes to future languages. Secondly,\nexisting methods process tokens (or instructions) sequentially, targeting function- and loop-free code.\nSuch codes, however, do not represent the majority of the applications.\nThis paper presents Neural Code Comprehension1: a general-purpose processing pipeline geared\ntowards representing code semantics in a robust and learnable manner. The pipeline, depicted in\nFig. 1, accepts code in various source languages and converts it to statements in an Intermediate\nRepresentation (IR), using the LLVM Compiler Infrastructure [39]. The LLVM IR, which is explained\nin detail in Section 4, is then processed to a robust representation that we call conteXtual Flow Graphs\n(XFGs). XFGs are constructed from both the data- and control-\ufb02ow of the code, thus inherently\nsupporting loops and function calls. In turn, the XFG structure is used to train an embedding space\nfor individual statements, called inst2vec (from the word \u201cinstruction\u201d), which is fed to RNNs for\na variety of high-level tasks.\nNeural Code Comprehension is evaluated on multiple levels, using clustering and analogies for\ninst2vec, as well as three different code comprehension tasks for XFGs: algorithm classi\ufb01cation;\nheterogeneous compute device (e.g., CPU, GPU) mapping; and optimal thread coarsening factor\nprediction, which model the runtime of an application without running it. Our datasets contain CPU\nand GPU code written in C, C++, OpenCL, and FORTRAN, though LLVM supports additional\nlanguages such as Python, Rust, Swift, Go, and CUDA. Our work makes the following contributions:\n\u2022 We formulate a robust distributional hypothesis for code, from which we draw a novel\n\u2022 We detail the construction of the XFG, the \ufb01rst representation designed speci\ufb01cally for\n\u2022 We evaluate the representation using clustering, analogies, semantic tests, and three funda-\n\u2022 Using one simple LSTM architecture and \ufb01xed pre-trained embeddings, we match or surpass\n\ndistributed representation of code statements based on contextual \ufb02ow and LLVM IR.\n\nthe best-performing approaches in each task, including specialized DNN architectures.\n\nstatement embeddings that combines data and control \ufb02ow.\n\nmentally different high-level code learning tasks.\n\n2 Related Work\n\nDistributed representations of code were \ufb01rst suggested by Allamanis et al. [2], followed by several\nworks leveraging embeddings to apply NLP techniques to programming languages [3, 61].\n\nCode Representation Previous research focuses on embedding high-level programming languages\nsuch as Java [20, 30], C [41], or OpenCL [18] in the form of tokens or statements, as well as lower\n\n1Code, datasets, trained embeddings, and results available at https://www.github.com/spcl/ncc\n\n2\n\nC/C++FORTRANPythonJavaCUDAOpenCL. . .SSA Representation(LLVM IR)Neural Code ComprehensionRNNRNNRNNRNNinst2vecContextual Flow Graph (XFG)Malicious Code DetectionGuided ProgrammingCode OptimizationHardware MappingAnti-VirusIDECompilerHigh-Level TasksFrontendSourceCode\flevel representations such as object code [41]. To the best of our knowledge, however, no attempt\nhas been made to train embeddings for compiler IRs prior to this work. As for representing the\ncontext of a token, which is necessary for training embeddings, some works rely on lexicographical\nlocality [2, 18, 20], whereas others exploit the structural nature of code, using Data Flow Graphs [4],\nControl Flow Graphs [51, 53, 64], Abstract Syntax Trees (ASTs) [12, 30], paths in the AST [8],\nor an augmented AST, for instance with additional edges connecting different uses and updates of\nsyntax tokens corresponding to variables [5]. We differ from all previous approaches by introducing\ncontextual \ufb02ow, a graph representation that captures both data and control dependencies. In compiler\nresearch, similar graphs exist but have not been successfully exploited for machine learning. Examples\ninclude the Program Dependence Graph (PDG) [24] and the IR known as Sea of Nodes [15, 16].\nUnlike these representations, our graphs are not designed to be optimized by a compiler nor translated\nto machine code, which allows us to introduce ambiguity (e.g., ignoring parameter order) in favor\nof preserving context. Other works applying Machine Learning techniques to PDGs exist: Hsiao et\nal. [34] use PDGs to compute n-gram models for program analysis, and Wang et al. [62] use them for\ndetecting copy direction among programs using Extreme Learning Machines. However, our work is\nthe \ufb01rst to leverage a hybrid of control and data \ufb02ow for the training of embeddings.\n\nAutomated Tasks on Code Learned representations of code are commonly used for two types of\ntasks: uncovering program semantics or optimizing programs. For the former task, code embeddings\nhave been used to perform function or variable naming [2, 7], clone detection [63], code comple-\ntion [54, 65], summarization [6], and algorithm classi\ufb01cation [49]. As for program optimization,\nresearch has been conducted on automatic feature generation for code [40, 50]; and Cummins et\nal. [18] notably leverage embeddings of OpenCL code to predict optimal device mapping and thread\ncoarsening factors. Their work differs from ours in that the method is restricted to the OpenCL\nlanguage, and that they process programs in a sequential order, which does not capture complex code\nstructures. Furthermore, the state-of-the-art in automatic tuning for program optimization [10] uses\nsurrogate performance models and active learning, and does not take code semantics into account.\n\nEmbedding Evaluation Previous works that use code embeddings do not evaluate the quality of\nthe trained space on its own merit, but rather through the performance of subsequent (downstream)\ntasks. One exception is Allamanis et al. [2], who present empirical evidence of vector similarities\nfor similar method names. To the best of our knowledge, we are the \ufb01rst to quantify the quality of a\ncode embedding space itself in the form of clustering, syntactic analogies, semantic analogies, and\ncategorical distance tests.\n\n3 A Robust Distributional Hypothesis of Code\n\nThe linguistic Distributional Hypothesis [32, 52] is given by: Words that occur in the same contexts\ntend to have similar meanings. We stipulate that code, which describes a sequence of operations to a\nprocessor, behaves similarly, and paraphrase this hypothesis to:\n\nStatements that occur in the same contexts tend to have similar semantics.\n\nHowever, the above wording is vague, due to the possible meanings of the highlighted elements. Below\nwe attempt to provide adequate de\ufb01nitions, upon which we build a learnable code representation.\n\nStatements To choose the right abstraction for statements, we take two concerns into account:\nuniversality and uniformity. As stated above, source code comes in many languages and thus\n\ufb01xating on a single one would hinder universality. At the other extreme, machine code (assembly)\nis target-speci\ufb01c, containing specialized instructions and relying on hardware characteristics, such\nas registers and memory architectures. As for uniformity, in a high-level language one statement\nmay represent simple arithmetics, multiple operations, or even class de\ufb01nitions (for example, the\nJava statement button.setOnClickListener(new View.OnClickListener(){...})). On the\nother hand, assembly is too limited, since instructions are reused for different purposes. We thus wish\nto choose statements that are independent of the source language, as well as the hardware architecture.\n\nContext The de\ufb01nition of a context for code statements should also be carefully considered. We\nde\ufb01ne context as statements whose execution directly depends on each other. Learning from consec-\nutive statements in code does not necessarily ful\ufb01ll this de\ufb01nition, as, for example, a programmer\n\n3\n\n\f(a) Source code\n\n(b) LLVM IR\n\n(c) Data\ufb02ow basic blocks\n\n(d) Contextual Flow Graph\n\nFigure 2: Contextual \ufb02ow processing scheme.\n\nmay use a variable in the \ufb01rst line of a function, but only use it again in the last line. Moreover,\nsuch long-term relationships may vanish when using RNNs and attention learning. It is possible to\ndetermine the data dependencies of each statement by analyzing data\ufb02ow, however, branches and\nfunction calls do not necessarily generate such dependencies. Another way of representing execution\ndependence is through the notion of causality (i.e., the \u201chappens-before\u201d relation [38]), which can\nbe used to complement data\ufb02ow. In our representation, context is the union of data dependence and\nexecution dependence, thereby capturing both relations.\n\nSimilarity To de\ufb01ne similarity, one \ufb01rst needs to de\ufb01ne the semantics of a statement. We draw the\nde\ufb01nition of semantics from Operational Semantics in programming language theory, which refers\nto the effects (e.g., preconditions, postconditions) of each computational step in a given program.\nIn this paper, we speci\ufb01cally assume that each statement modi\ufb01es the system state in a certain way\n(e.g., adds two numbers) and consumes resources (e.g., uses registers and \ufb02oating-point units). It\nfollows that semantic similarity can be de\ufb01ned by two statements consuming the same resources or\nmodifying the system state in a similar way. Using this de\ufb01nition, two versions of the same algorithm\nwith different variable types would be synonymous.\n\n4 Contextual Flow Processing\n\nThe aforementioned statements and contexts cannot be directly extracted from source code, but rather\nrequire processing akin to partial compilation (e.g., data\ufb02ow extraction). In this section, we brie\ufb02y\ndescribe a popular compilation pipeline and proposed modi\ufb01cations to create a learnable vocabulary\nof statements and their context.\n\n4.1 Compilation, Static Single Assignment, and LLVM IR\n\nMajor contemporary compilers, such as GCC and LLVM, support multiple programming languages\nand hardware targets. To avoid duplication in code optimization techniques, they enforce a strict\nseparation between the source language (frontend), an Intermediate Representation (IR) that can be\noptimized, and the target machine code (backend) that should be mapped to a speci\ufb01c hardware. In\nparticular, the LLVM IR [45] supports various architectures (e.g., GPUs), and can represent optimized\ncode (e.g., using vector registers) inherently. Figures 2a and 2b depict an example code and its LLVM\nIR equivalent, and the structure of an LLVM IR statement is shown in Fig. 3.\n\nx0 = a * b;\nx1 = c * c;\nx2 = x0 + x1;\nx3 = x + x2;\n\nFigure 3: Anatomy of an LLVM IR statement.\n\nFigure 4: SSA of x += (a*b)+(c*c).\n\nIn the LLVM infrastructure, the IR is given in Static Single Assignment (SSA) form [19]. Brie\ufb02y,\nan SSA IR ensures that every variable is assigned only once, which makes it easy to track data\ufb02ow\nbetween IR statements, as shown in Fig. 4. To overcome analysis issues resulting from control-\ufb02ow,\nsuch as loops, SSA de\ufb01nes \u03c6-expressions. These expressions enumerate all possible outcomes that\n\n4\n\ndoublethres= 5.0;if(x < thres)x = y * y;elsex = 2.0 * y;x += 1.0;%cmp= fcmpoltdouble %x, 5.0bri1 %cmp, label%LT, label%GELT:%2 = fmuldouble%y, %yGE:%3 = fmuldouble2.0, %yAFTER:%4 = phidouble[%2,%LT], [%3,%GE]%5 = fadddouble%4, 1.0%05.0%cmp%LT%GE%2%3%AFTER%y2.0%y1.0%5%3%2%4%x%x%y%LT%cmp%AFTER%5%4faddphi%3%2%GEfmulbrbrfcmpphi%5 = loadfloat, float*%a1, align 4, !tbaa!1; commentOutput IdentifierInstructionTypesInputIdentifierOtherParametersMetadata\fcan lead to a variable (depending on the runtime control-\ufb02ow), and can be used to optimize code\nacross branches. In Fig. 2b, the identi\ufb01er %4 is constructed from a \u03c6-expression that can take either\nthe value of %2 or %3, depending on the value of x.\n\n4.2 Contextual Flow Graphs\n\nTo analyze data\ufb02ow for optimization, LLVM divides the IR statements into \u201cbasic blocks\u201d, which\ncontain no control-\ufb02ow divergence, illustrated in Fig. 2c. Within a basic block, statements naturally\ncreate traceable data\ufb02ow as SSA lists data dependencies in the form of input identi\ufb01ers (even if\nconditional), and assigns the results to a single identi\ufb01er. However, as shown in Section 3, data\ufb02ow\nalone does not suf\ufb01ce to provide context for a given statement, e.g., when in the vicinity of a branch.\nTherefore, we de\ufb01ne a representation that incorporates both the relative data- and control-\ufb02ow of a\nstatement, which we call the conteXtual Flow Graph (XFG).\nXFGs (e.g., Fig. 2d) are directed multigraphs, where two nodes can be connected by more than\none edge. XFG nodes can either be variables or label identi\ufb01ers (e.g., basic block, function name),\nappearing in the \ufb01gure as ovals or rectangles respectively. Correspondingly, an edge either represents\ndata-dependence (in black), carrying an LLVM IR statement; or execution dependence (light blue).\nXFG Construction We generate XFGs incrementally from LLVM IR, as follows:\n\n1. Read LLVM IR statements once, storing function names and return statements.\n2. Second pass over the statements, adding nodes and edges according to the following rule-set:\n\n(a) Data dependencies within a basic block are connected.\n(b) Inter-block dependencies (e.g., \u03c6-expressions) are both connected directly and through\n\nthe label identi\ufb01er (statement-less edges).\n\n(c) Identi\ufb01ers without a data\ufb02ow parent are connected to their root (label or program root).\n\nIt follows that XFGs create paths through data\ufb02ow as well as branches, loops, and functions (including\nrecursion). Owing to the two passes, as well as the linear-time construction of LLVM IR [58], XFGs\nare constructed in O(n) for a program with n SSA statements. This is especially valuable when\nlearning over large code corpora, such as Tensor\ufb02ow.\nExternal Code Calls to external code (e.g., libraries, frameworks) can be divided into two cate-\ngories: statically- and dynamically-linked. If the code is accessible during compilation (header-only\nframeworks and static libraries), LLVM IR is available and the statements are traversed as part of the\nXFG. In the dynamic case, the library code is not included and is represented as a call statement.\n\n5 inst2vec: Embedding Statements in Continuous Space\n\nWith XFGs providing a notion of context, we can now train an embedding space for individual\nstatements. To support learnability, desiderata for such a space include: (a) statements that are in\nclose proximity should have similar artifacts on a system (i.e., use the same resources); and (b)\nchanging the same attributes (e.g., data type) for different instructions should result in a similar offset\nin the space. We train LLVM IR statement embeddings using the skip-gram model [48], following\npreprocessing to limit the vocabulary size.\n\n5.1 Statement Preprocessing and Training\n\nPreprocessing First, we \ufb01lter out comments and metadata from statements. Then, identi\ufb01ers and\nimmediate values (numeric constants, strings) are replaced with %ID and <INT/FLOAT/STRING>\nrespectively, where immediate values are fed separately to downstream RNNs. Lastly, data structures\nare \u201cinlined\u201d, that is, their contents are encoded within the statement. Fig. 5 lists statements before\nand after preprocessing.\n\nstore float %250, float* %82, align 4, !tbaa !1\n%10 = fadd fast float %9, 1.3\n%8 = load %\"struct.aaa\"*, %\"struct.aaa\"** %2\n\nstore float %ID, float* %ID, align 4\n%ID = fadd fast float %ID, <FLOAT>\n%ID = load { float, float }*, { float, float }** %ID\n\n(a) LLVM IR\n\n(b) inst2vec statements\n\nFigure 5: Before and after preprocessing LLVM IR to inst2vec statements.\n\n5\n\n\fTable 1: inst2vec training dataset statistics\n\nDiscipline\n\nDataset\n\nFiles\n\nLLVM IR\nLines\n\nVocabulary\nSize\n\nXFG Stmt.\nPairs\n\nMachine Learning\n\nTensor\ufb02ow [1]\n\n2,492\n\n16,943,893\n\n220,554\n\n260,250,973\n\nHigh-Performance Computing\n\nBenchmarks\n\nAMD APP SDK [9]\nBLAS [22]\n\nNAS [57]\nParboil [59]\nPolybenchGPU [27]\nRodinia [14]\nSHOC [21]\n\nScienti\ufb01c Computing\n\nCOSMO [11]\n\n123\n300\n\n268\n151\n40\n92\n112\n\n161\n\nOperating Systems\n\nComputer Vision\n\nLinux kernel [42]\n\nOpenCV [36]\nNVIDIA samples [17]\n\n1,988\n\n442\n60\n\n1,304,669\n280,782\n\n572,521\n118,575\n33,601\n103,296\n399,287\n\n152,127\n\n2,544,245\n\n1,908,683\n43,563\n\n4,146\n566\n\n1,793\n2,175\n577\n3,861\n3,381\n\n2,344\n\n136,545\n\n39,920\n2,467\n\n45,081,359\n283,856\n\n1,701,968\n151,916\n40,975\n266,354\n12,096,508\n\n2,338,153\n\n5,271,179\n\n10,313,451\n74,915\n\nSynthetic\n\nSynthetic\n\n17,801\n\n26,045,547\n\n113,763\n\n303,054,685\n\nTotal (Combined)\n\n\u2014\n\n24,030\n\n50,450,789\n\n8,565\n\n640,926,292\n\nDataset Table 1 summarizes the code corpora and vocabulary statistics of the inst2vec dataset.\nWe choose corpora from different disciplines, including high-performance computing, benchmarks,\noperating systems, climate sciences, computer vision, machine learning (using Tensor\ufb02ow\u2019s own\nsource code), and synthetically-generated programs. The code in the dataset is written in C, C++,\nFORTRAN, and OpenCL, and is compiled for Intel CPUs as well as NVIDIA and AMD GPUs. The\n\ufb01les in the dataset were compiled to LLVM IR with Clang [44] and Flang [43], using compilation \ufb02ags\nfrom the original code (if available) and randomly chosen compiler optimization (e.g., -ffast-math)\nand target architecture \ufb02ags.\nFor the synthetic corpus, we use both C code and the Eigen [31] C++ library. In particular, ran-\ndom linear algebra operations are procedurally generated from high-level templates using different\nparameters, such as data types, operations, and dimensions.\n\nSetup and Training Given a set of XFGs created from the LLVM IR \ufb01les, we generate neighboring\nstatement pairs up to a certain context size, following the skip-gram model [48]. A context of size\nN includes all statement pairs that are connected by a path shorter or equal to N. To obtain the\npairs, we construct a dual graph in which statements are nodes, omitting duplicate edges. Following\nthis process, we discard statements that occur less than 300 times in the dataset, pairs of identical\nstatements, and perform subsampling of frequent pairs, similarly to Mikolov et al. [48]. We train\ninst2vec with an embedding dimension of 200 for 5 epochs using Tensor\ufb02ow [1]. The Adam\noptimizer [37] is used with the default published hyperparameters and softmax cross-entropy loss.\n\n5.2 Evaluation\n\nClustering Fig. 6 depicts the t-SNE [60] plots for trained inst2vec spaces with different XFG\ncontext sizes, colored by statement and data type (legend in Appendix A). In the plots, we see that\nboth a context size of 1 statement in each direction (Fig. 6a) or 3 statements (Fig. 6c) generate large,\nmulti-type clusters, as well as outliers. This phenomenon eventually contributes to a lower \ufb01nal\nanalogy score, due to inappropriate representation of inter-statement relations, as can be seen below.\nOwing to these results, we choose a context size of 2 statements (Fig. 6b), which mostly consists of\nseparate, monochromatic clusters, indicating strong clustering w.r.t. instruction and data types. While\ndata type syntactic clusters are unsurprising, their existence is not trivial, since the dataset contains\ndiverse codebases rather than copies of the same functions with different types.\nAn example of a semantically-similar statement cluster can be found in data structures. In particular,\nthe top-5 nearest neighbors of operations on the complex data type \u201cstd::complex<float>\u201d include\n\u201c2 x float\u201d (i.e., a vector type). In fact, LLVM IR represents the complex data type as {float,\nfloat}, so this property is generalized to any user-de\ufb01ned data structure (struct) with two \ufb02oats.\n\nAnalogies and Tests We also evaluate inst2vec by automatically generating a list of statement\nanalogies (\u201ca\u201d is to \u201cb\u201d as \u201cc\u201d is to \u201c?\u201d, or \u201ca:b; c:?\u201d) that appear in our vocabulary using the LLVM\n\n6\n\n\f(a) Context size = 1\n\n(b) Context size = 2\n\n(c) Context size = 3\n\nFigure 6: Two-dimensional t-SNE plots for learned embeddings (best viewed in color).\n\nIR syntax. We then use the embeddings to \ufb01nd the result by computing a-b+c and asking whether\nthe result is in the top-5 neighbors (cosine distance). Additionally, we automatically create relative\ndistance expressions using the LLVM IR reference categories [45] of the form d(a, b) < d(a, c) to\ntest whether statements that use different resources are further away than those who use the same.\nTable 2 shows the analogy and test results for inst2vec trained on XFG as well as on CFG (control\n\ufb02ow-only) and DFG (data \ufb02ow-only) for different context sizes. The analogies are divided into\ndifferent categories, including data types (i.e., transitions between types), options (e.g., fast math),\nconversions (e.g., bit casting, extension, truncation), and data structures (e.g., vector-type equivalents\nof structures). Below are examples of a type analogy:\n\n%ID = add i64 %ID, %ID\n\n:\n\n%ID = fadd float %ID, %ID;\n\n%ID = sub i64 %ID, %ID\nand a data structure analogy:\n\n:?\n\n%ID = fsub float %ID, %ID\n\n%ID = extractvalue { double, double } %ID, 0\n\n:\n\n%ID = extractelement <2 x double> %ID, <TYP> 0;\n\n%ID = extractvalue { double, double } %ID, 1\n\n:? %ID = extractelement <2 x double> %ID, <TYP> 1\n\nThe results con\ufb01rm that over all scores, a context size of 2 is the best-performing con\ufb01guration, and\nshow that the XFG representation is more complete and leads to better embeddings than taking into\naccount control or data \ufb02ow alone.\n\nContext\ntype\n\nCFG\n\nDFG\n\nXFG\n\nContext\nSize\n\n1\n2\n3\n\n1\n2\n3\n\n1\n2\n3\n\nTable 2: Analogy and test scores for inst2vec\n\nSyntactic Analogies\n\nSemantic Analogies\n\nSemantic Distance Test\n\nTypes\n\nOptions\n\nConversions\n\nData Structures\n\n0 (0 %)\n1 (0.18 %)\n0 (0 %)\n\n53 (9.46 %)\n71 (12.68 %)\n67 (22.32 %)\n\n101 (18.04 %)\n226 (40.36 %)\n125 (22.32 %)\n\n1 (1.89 %)\n1 (1.89 %)\n1 (1.89 %)\n\n12 (22.64 %)\n12 (22.64 %)\n18 (33.96 %)\n\n13 (24.53 %)\n45 (84.91 %)\n24 (45.28 %)\n\n1 (0.07 %)\n0 (0 %)\n4 (0.27 %)\n\n2 (0.13 %)\n12 (0.80 %)\n40 (2.65 %)\n\n100 (6.63 %)\n134 (8.89 %)\n48 (3.18 %)\n\n0 (0 %)\n0 (0 %)\n0 (0 %)\n\n4 (50.00 %)\n3 (37.50 %)\n4 (50.00 %)\n\n3 (37.50 %)\n7 (87.50 %)\n7 (87.50 %)\n\n51.59 %\n50.47 %\n53.79 %\n\n56.79 %\n57.44 %\n60.38 %\n\n60.98 %\n79.12 %\n62.56 %\n\n6 Code Comprehension Experiments\n\nIn this section, we evaluate inst2vec on three different tasks, comparing with manually-extracted\nfeatures and state-of-the-art specialized deep learning approaches. Throughout all tasks, we use\nthe same neural network architecture and our pre-trained embedding matrix from Section 5, which\nremains \ufb01xed during training.\nTraining Our recurrent network (see schematic description in the Appendix B) consists of an\ninst2vec input with an XFG context size of 2, followed by two stacked LSTM [33] layers with 200\nunits in each layer, batch normalization [35], a dense 32-neuron layer with ReLU activations, and\noutput units matching the number of classes. The loss function is a categorical cross-entropy trained\n\n7\n\n50-500-50050500-50-100100500-50801006040200-20-60-40-80-60-400-2020406080\fusing Adam [37] with the default hyperparameters. Additionally, for the compute device mapping\nand optimal thread coarsening factor prediction tasks, we train the LLVM IR statements with the\nimmediate values that were stripped from them during preprocessing (see Section 5). Further details\nare given in Appendix C.\nDatasets The algorithm classi\ufb01cation task uses the POJ-104 [49] dataset2, collected from a Peda-\ngogical Open Judge system. The dataset contains 104 program classes written by 500 different people\n(randomly selected subset per class). For the compute device mapping and optimal thread coarsening\nfactor prediction tasks, we use an OpenCL code dataset3 provided by Cummins et al. [18].\n\n6.1 Algorithm Classi\ufb01cation\n\nUsing inst2vec, we construct an RNN that reads embedded source code and outputs a predicted\nprogram class. We compare our approach with Tree-Based CNNs (TBCNN) [49], the best-performing\nalgorithm classi\ufb01er in the POJ-104 dataset. TBCNN constructs embeddings from Astract Syntax Tree\nnodes of source code, and employs two specialized layers: tree convolutions and dynamic pooling.\nTheir network comprises 5 layers, where convolution and fully connected layers are 600-dimensional.\nOur data preparation follows the experiment conducted by Mou et al. [49], splitting the dataset 3:1:1\nfor training, validation, and testing. To compile the programs successfully, we prepend #include\nstatements to each \ufb01le. Data augmentation is then applied on the training set by compiling each \ufb01le 8\ntimes with different \ufb02ags (-O{0-3}, -ffast-math).\n\nTable 3: Algorithm classi\ufb01cation test accuracy\nTBCNN [49]\n\nSurface Features [49]\n\nMetric\n\n(RBF SVM + Bag-of-Trees)\n\nRNN [49]\n\ninst2vec\n\nTest Accuracy [%]\n\n88.2\n\n84.8\n\n94.0\n\n94.83\n\nTable 3 compares inst2vec (trained for 100 epochs) with the reported results of Mou et al. [49],\nwhich contain TBCNN as well as a 600-cell RNN and a manual feature extraction approach (Surface\nFeatures). The results show that inst2vec sets a new state-of-the-art with a 13.8 % decrease in error,\neven though the dataset used to generate the embeddings does not include POJ-104 (see Table 1).\n\n6.2 Heterogeneous Compute Device Mapping\n\nNext, we use Neural Code Comprehension to predict whether a given OpenCL program will run\nfaster on a CPU (Intel Core i7-3820) or a GPU (AMD Tahiti 7970 and NVIDIA GTX 970) given\nits code, input data size, and work-group size (i.e., number of threads that work in a group with\nshared memory). To achieve that, we use the same experimental methodology presented by Cummins\net al. [18], removing their specialized OpenCL source rewriter and replacing their code token\nembeddings with our XFGs and inst2vec. We concatenate the data and work-group sizes to the\nnetwork inputs, and train with strati\ufb01ed 10-fold cross-validation. We repeat the training 10 times with\nrandom initialization of the network\u2019s weights and report the best result.\nIn Table 4, inst2vec and inst2vec-imm (i.e., with immediate value handling) are compared with a\nmanual code feature extraction approach by Grewe et al. [29] and DeepTune [18], in terms of runtime\nprediction accuracies and resulting speedup. The baseline for the speedup is a static mapping, which\nselects the device that yields the best average case performance over all programs in the data set:\nin the case of AMD Tahiti versus Intel i7-3820, that is the CPU and in the case of NVIDIA GTX\nversus Intel i7-3820, it is the GPU. The results indicate that inst2vec outperforms Grewe et al. and\nis on-par with DeepTune. We believe that the better predictions in DeepTune are the result of training\nthe embedding matrix in tandem with the high-level task, thereby specializing it to the dataset. This\nspecialized training is, however, surpassed by taking immediate values into account during training.\nWe present the result of the best immediate value handling method in Table 4 (inst2vec-imm), and\nthe exhaustive results can be found in Appendix D.\n\n2https://sites.google.com/site/treebasedcnn/\n3https://www.github.com/ChrisCummins/paper-end2end-dl\n\n8\n\n\fTable 4: Heterogeneous device mapping results\n\nArchitecture\n\nPrediction Accuracy [%]\n\nAMD Tahiti 7970\nNVIDIA GTX 970\n\nAMD Tahiti 7970\nNVIDIA GTX 970\n\nGPU\n\n41.18\n56.91\n\nGPU\n\n3.26\n1.00\n\nGrewe et al. [29]\n\nDeepTune [18]\n\ninst2vec\n\ninst2vec-imm\n\n73.38\n72.94\n\n83.68\n80.29\n\nSpeedup\n\n82.79\n82.06\n\n88.09\n86.62\n\nGrewe et al.\n\nDeepTune\n\ninst2vec\n\ninst2vec-imm\n\n2.91\n1.26\n\n3.34\n1.41\n\n3.42\n1.42\n\n3.47\n1.44\n\n6.3 Optimal Thread Coarsening Factor Prediction\n\nOur third example predicts the best-performing thread coarsening factor, a measure of the amount of\nwork done per GPU thread, for a given OpenCL code. We again compare the achieved speedups of\ninst2vec with manual features [46], DeepTune, and DeepTune with transfer learning applied from\nthe task in Section 6.2 (denoted by DeepTune-TL). Possible values for the coarsening factor are 1\n(baseline for speedups), 2, 4, 8, 16, and 32. The results in Table 5 show that while inst2vec yields\nbetter speedups than DeepTune-TL in only half of the cases (possibly due to the embedding special-\nization in DeepTune), the manually-extracted features are consistently outperformed by inst2vec.\nMoreover, inst2vec-imm is consistently on-par with DeepTune, but improves inconsistently on\ninst2vec (on the AMD Tahiti and the NVIDIA GTX only), and fails to outperform DeepTune-TL.\nThis can be explained by the small size of the training data for this task (17 programs with 6 different\nthread coarsening factors for each hardware platform). The optimal device mapping task (Section\n6.2), on the other hand, features 680 programs for each platform.\n\nTable 5: Speedups achieved by coarsening threads\n\nComputing Platform\n\nMagni et al. [46]\n\nDeepTune [18]\n\nDeepTune-TL [18]\n\ninst2vec\n\ninst2vec-imm\n\nAMD Radeon HD 5900\nAMD Tahiti 7970\nNVIDIA GTX 480\nNVIDIA Tesla K20c\n\n1.21\n1.01\n0.86\n0.94\n\n1.10\n1.05\n1.10\n0.99\n\n1.17\n1.23\n1.14\n0.93\n\n1.37\n1.10\n1.07\n1.06\n\n1.28\n1.18\n1.11\n1.00\n\n7 Conclusion\n\nIn this paper, we have empirically shown that semantics of statements can be successfully recovered\nfrom their context alone. This recovery relies both on proper granularity, where we propose to use\n\ufb01ltered LLVM IR instructions; and on the grouping of statements, for which we use a mixture of data-\nand control-\ufb02ow. We use our proposed representation to perform three high-level classi\ufb01cation and\nprediction tasks, outperforming all manually-extracted features and achieving results that are on-par\nwith (and better than) two inherently different state-of-the-art specialized DNN solutions.\nWith this work, we attempt to pave the way towards mechanized code comprehension via machine\nlearning, whether the code was authored by a human or automatically-generated. Further research\ncould be conducted in various directions. Rather than directly using statements, the representation\nmay be re\ufb01ned using part-based models, which have already been applied successfully in language\nmodels [55]. inst2vec can also be used as a basis for neural code interpretation, using a modi\ufb01ed\nDifferentiable Neural Computer [28] to enable execution of arbitrary code over DNNs.\n\nAcknowledgments\n\nWe wish to thank Theodoros Theodoridis, K\ufb01r Levy, Tobias Grosser, and Yunyan Guo for fruitful\ndiscussions. The authors also acknowledge MeteoSwiss, and thank Hussein Harake, Colin McMurtrie,\nand the whole CSCS team for granting access to the Greina machines, and for their excellent technical\nsupport. TBN is supported by the ETH Postdoctoral Fellowship and Marie Curie Actions for People\nCOFUND program.\n\n9\n\n\fReferences\n[1] Mart\u00edn Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro,\nGreg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow,\nAndrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser,\nManjunath Kudlur, Josh Levenberg, Dandelion Man\u00e9, Rajat Monga, Sherry Moore, Derek\nMurray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal\nTalwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Vi\u00e9gas, Oriol Vinyals, Pete\nWarden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-\nscale machine learning on heterogeneous systems, 2015. Software available from tensor\ufb02ow.org.\n\n[2] Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. Suggesting accurate\nmethod and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of\nSoftware Engineering, ESEC/FSE 2015, pages 38\u201349, New York, NY, USA, 2015. ACM.\n\n[3] Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles A. Sutton. A survey of\n\nmachine learning for big code and naturalness. CoRR, abs/1709.06182, 2017.\n\n[4] Miltiadis Allamanis and Marc Brockschmidt. Smartpaste: Learning to adapt source code. CoRR,\n\nabs/1705.07867, 2017.\n\n[5] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent\n\nprograms with graphs. CoRR, abs/1711.00740, 2017.\n\n[6] Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. A convolutional attention network for\n\nextreme summarization of source code. CoRR, abs/1602.03001, 2016.\n\n[7] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed\n\nrepresentations of code. CoRR, abs/1803.09473, 2018.\n\n[8] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A general path-based representation\n\nfor predicting program properties. CoRR, abs/1803.09544, 2018.\n\n[9] AMD. AMD OpenCL accelerated parallel processing SDK. https://developer.amd.com/\n\namd-accelerated-parallel-processing-app-sdk/.\n\n[10] P. Balaprakash, R. B. Gramacy, and S. M. Wild. Active-learning-based surrogate models for\nempirical performance tuning. In 2013 IEEE International Conference on Cluster Computing\n(CLUSTER), pages 1\u20138, Sept 2013.\n\n[11] M. Baldauf, A. Seifert, J. F\u00f6rstner, D. Majewski, M. Raschendorfer, and T. Reinhardt. Opera-\ntional Convective-Scale Numerical Weather Prediction with the COSMO Model: Description\nand Sensitivities. Monthly Weather Review, 139:3887\u20133905, December 2011.\n\n[12] Pavol Bielik, Veselin Raychev, and Martin Vechev. PHOG: Probabilistic model for code. In\nMaria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International\nConference on Machine Learning, volume 48 of Proceedings of Machine Learning Research,\npages 2933\u20132942, New York, New York, USA, June 2016. PMLR.\n\n[13] Rudy Bunel, Alban Desmaison, M. Pawan Kumar, Philip H. S. Torr, and Pushmeet Kohli.\nLearning to superoptimize programs. International Conference on Learning Representations,\n2017.\n\n[14] Shuai Che, Michael Boyer, Jiayuan Meng, David Tarjan, Jeremy W. Sheaffer, Sang-Ha Lee, and\nKevin Skadron. Rodinia: A benchmark suite for heterogeneous computing. In Proceedings of\nthe 2009 IEEE International Symposium on Workload Characterization (IISWC), IISWC \u201909,\npages 44\u201354, Washington, DC, USA, 2009. IEEE Computer Society.\n\n[15] Cliff Click and Keith D. Cooper. Combining analyses, combining optimizations. ACM Transac-\n\ntions on Programming Languages and Systems, 17, 1995.\n\n[16] Cliff Click and Michael Paleczny. A simple graph-based intermediate representation. SIGPLAN\n\nNot., 30(3):35\u201349, March 1995.\n\n10\n\n\f[17] NVIDIA Corporation. CUDA. http://developer.nvidia.com/object/cuda.html.\n\n[18] Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. End-to-end deep learning\n\nof optimization heuristics. In PACT. ACM, 2017.\n\n[19] Ron Cytron, Jeanne Ferrante, Barry K. Rosen, Mark N. Wegman, and F. Kenneth Zadeck.\nEf\ufb01ciently computing static single assignment form and the control dependence graph. ACM\nTrans. Program. Lang. Syst., 13(4):451\u2013490, October 1991.\n\n[20] Hoa Khanh Dam, Truyen Tran, and Trang Pham. A deep language model for software code.\n\nCoRR, abs/1608.02715, 2016.\n\n[21] Anthony Danalis, Gabriel Marin, Collin McCurdy, Jeremy S. Meredith, Philip Roth, Kyle\nSpafford, Vinod Tipparaju, and Jeffrey Vetter. The Scalable HeterOgeneous Computing (SHOC)\nbenchmark suite. pages 63\u201374, January 2010.\n\n[22] Jack Dongarra. Basic linear algebra subprograms technical forum standard. page 1 \u2014 111,\n\n2002.\n\n[23] Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179 \u2013 211, 1990.\n\n[24] Jeanne Ferrante, Karl J. Ottenstein, and Joe D. Warren. The program dependence graph and its\n\nuse in optimization. ACM Trans. Program. Lang. Syst., 9(3):319\u2013349, July 1987.\n\n[25] GitHub. GitHub Octoverse. https://octoverse.github.com/, 2017.\n\n[26] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse recti\ufb01er neural networks.\n\n2011.\n\n[27] Scott Grauer-Gray, Lifan Xu, Robert Searles, Sudhee Ayalasomayajula, and John Cavazos.\n\nAuto-tuning a high-level language targeted to GPU codes. 2012.\n\n[28] Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-\nBarwi\u00b4nska, Sergio G\u00f3mez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou,\net al. Hybrid computing using a neural network with dynamic external memory. Nature,\n538(7626):471\u2013476, 2016.\n\n[29] Dominik Grewe, Zheng Wang, and Michael O\u2019Boyle. Portable mapping of data parallel\n\nprograms to OpenCL for heterogeneous systems. pages 1\u201310, February 2013.\n\n[30] Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. Deep API learning. In\nProceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of\nSoftware Engineering, FSE 2016, pages 631\u2013642, New York, NY, USA, 2016. ACM.\n\n[31] Ga\u00ebl Guennebaud and Beno\u00eet Jacob et al. Eigen v3. http://eigen.tuxfamily.org, 2010.\n\n[32] Zellig S. Harris. Distributional Structure, pages 3\u201322. Springer Netherlands, Dordrecht, 1981.\n\n[33] Sepp Hochreiter and J\u00fcrgen Schmidhuber. Long short-term memory. Neural Computation,\n\n9(8):1735\u20131780, 1997.\n\n[34] Chun-Hung Hsiao, Michael Cafarella, and Satish Narayanasamy. Using web corpus statistics\nfor program analysis. In Proceedings of the 2014 ACM International Conference on Object\nOriented Programming Systems Languages & Applications, OOPSLA \u201914, pages 49\u201365, New\nYork, NY, USA, 2014. ACM.\n\n[35] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training\nby reducing internal covariate shift. In Proceedings of the 32Nd International Conference\non International Conference on Machine Learning - Volume 37, ICML\u201915, pages 448\u2013456.\nJMLR.org, 2015.\n\n[36] Itseez. Open source computer vision library. https://github.com/itseez/opencv, 2015.\n\n[37] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR,\n\nabs/1412.6980, 2014.\n\n11\n\n\f[38] Leslie Lamport. Time, clocks, and the ordering of events in a distributed system. Commun.\n\nACM, 21(7):558\u2013565, July 1978.\n\n[39] Chris Lattner and Vikram Adve. LLVM: a compilation framework for lifelong program analysis\ntransformation. In International Symposium on Code Generation and Optimization, 2004. CGO\n2004., pages 75\u201386, March 2004.\n\n[40] Hugh Leather, Edwin Bonilla, and Michael O\u2019Boyle. Automatic feature generation for machine\nlearning based optimizing compilation. In Proceedings of the 7th Annual IEEE/ACM Interna-\ntional Symposium on Code Generation and Optimization, CGO \u201909, pages 81\u201391, Washington,\nDC, USA, 2009. IEEE Computer Society.\n\n[41] Dor Levy and Lior Wolf. Learning to align the source code to the compiled object code. In\nProceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney,\nNSW, Australia, 6-11 August 2017, pages 2043\u20132051, 2017.\n\n[42] Linux. Linux kernel source code (version 4.15.1). https://www.kernel.org/.\n\n[43] LLVM.\n\nFlang: a FORTRAN compiler frontend for LLVM.\n\nflang-compiler/flang.\n\nhttps://github.com/\n\n[44] LLVM. Clang: a C language family frontend for LLVM v4.0.0. http://clang.llvm.org/,\n\n2017.\n\n[45] LLVM. LLVM language reference manual. https://llvm.org/docs/LangRef.html, 2018.\n\n[46] Alberto Magni, Christophe Dubach, and Michael O\u2019Boyle. Automatic optimization of thread-\ncoarsening for graphics processors. In Proceedings of the 23rd International Conference on\nParallel Architectures and Compilation, PACT \u201914, pages 455\u2013466, New York, NY, USA, 2014.\nACM.\n\n[47] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Ef\ufb01cient estimation of word\n\nrepresentations in vector space. CoRR, abs/1301.3781, 2013.\n\n[48] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed\nrepresentations of words and phrases and their compositionality. In Proceedings of the 26th\nInternational Conference on Neural Information Processing Systems - Volume 2, NIPS\u201913,\npages 3111\u20133119, USA, 2013. Curran Associates Inc.\n\n[49] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over\ntree structures for programming language processing. In Proceedings of the Thirtieth AAAI\nConference on Arti\ufb01cial Intelligence, AAAI\u201916, pages 1287\u20131293. AAAI Press, 2016.\n\n[50] Mircea Namolaru, Albert Cohen, Grigori Fursin, Ayal Zaks, and Ari Freund. Practical aggrega-\ntion of semantical program properties for machine learning based optimization. In Proceedings\nof the 2010 International Conference on Compilers, Architectures and Synthesis for Embedded\nSystems, CASES \u201910, pages 197\u2013206, New York, NY, USA, 2010. ACM.\n\n[51] Ricardo Nobre, Luiz G. A. Martins, and Jo\u00e3o M. P. Cardoso. A graph-based iterative compiler\npass selection and phase ordering approach. In Proceedings of the 17th ACM SIGPLAN/SIGBED\nConference on Languages, Compilers, Tools, and Theory for Embedded Systems, LCTES 2016,\npages 21\u201330, New York, NY, USA, 2016. ACM.\n\n[52] Patrick Pantel. Inducing ontological co-occurrence vectors. In Proceedings of the 43rd Annual\nMeeting on Association for Computational Linguistics, ACL \u201905, pages 125\u2013132, Stroudsburg,\nPA, USA, 2005. Association for Computational Linguistics.\n\n[53] Eunjung Park, John Cavazos, and Marco A. Alvarez. Using graph-based program characteriza-\ntion for predictive modeling. In Proceedings of the Tenth International Symposium on Code\nGeneration and Optimization, CGO \u201912, pages 196\u2013206, New York, NY, USA, 2012. ACM.\n\n[54] Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language\n\nmodels. SIGPLAN Not., 49(6):419\u2013428, June 2014.\n\n12\n\n\f[55] Cicero Dos Santos and Bianca Zadrozny. Learning character-level representations for part-of-\nspeech tagging. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International\nConference on Machine Learning, volume 32 of Proceedings of Machine Learning Research,\npages 1818\u20131826, Bejing, China, June 2014. PMLR.\n\n[56] Eric Schkufza, Rahul Sharma, and Alex Aiken. Stochastic superoptimization. In Proceedings of\nthe Eighteenth International Conference on Architectural Support for Programming Languages\nand Operating Systems, ASPLOS \u201913, pages 305\u2013316, New York, NY, USA, 2013. ACM.\n\n[57] Sangmin Seo, Gangwon Jo, and Jaejin Lee. Performance characterization of the nas parallel\nbenchmarks in opencl. In Proceedings of the 2011 IEEE International Symposium on Workload\nCharacterization, IISWC \u201911, pages 137\u2013148, Washington, DC, USA, 2011. IEEE Computer\nSociety.\n\n[58] Vugranam C. Sreedhar and Guang R. Gao. A linear time algorithm for placing phi-nodes. In\nProceedings of the 22Nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming\nLanguages, POPL \u201995, pages 62\u201373, New York, NY, USA, 1995. ACM.\n\n[59] John A. Stratton, Christopher Rodrigues, I-Jui Sung, Nady Obeid, Li-Wen Chang, Nasser\nAnssari, Geng Daniel Liu, and Wen-mei W. Hwu. Parboil: A revised benchmark suite for\nscienti\ufb01c and commercial throughput computing. Center for Reliable and High-Performance\nComputing, 2012.\n\n[60] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine\n\nLearning Research, 9:2579\u20132605, 2008.\n\n[61] Martin Vechev and Eran Yahav. Programming with \"big code\". Found. Trends Program. Lang.,\n\n3(4):231\u2013284, December 2016.\n\n[62] Baoezeng Wang, Xiaochun Yang, and Guoren Wang. Detecting copy directions among programs\n\nusing extreme learning machines. 2015:1\u201315, 05 2015.\n\n[63] Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. Deep learning\ncode fragments for code clone detection. In Proceedings of the 31st IEEE/ACM International\nConference on Automated Software Engineering, ASE 2016, pages 87\u201398, New York, NY, USA,\n2016. ACM.\n\n[64] Xiaojun Xu, Chang Liu, Qian Feng, Heng Yin, Le Song, and Dawn Song. Neural network-based\ngraph embedding for cross-platform binary code similarity detection. CoRR, abs/1708.06525,\n2017.\n\n[65] Yixiao Yang, Yu Jiang, Ming Gu, Jiaguang Sun, Jian Gao, and Han Liu. A language model for\nstatements of software code. In Proceedings of the 32Nd IEEE/ACM International Conference\non Automated Software Engineering, ASE 2017, pages 682\u2013687, Piscataway, NJ, USA, 2017.\nIEEE Press.\n\n13\n\n\f", "award": [], "sourceid": 1819, "authors": [{"given_name": "Tal", "family_name": "Ben-Nun", "institution": "ETH Zurich"}, {"given_name": "Alice Shoshana", "family_name": "Jakobovits", "institution": "ETH Zurich"}, {"given_name": "Torsten", "family_name": "Hoefler", "institution": "ETH Zurich"}]}