{"title": "Learning to Solve SMT Formulas", "book": "Advances in Neural Information Processing Systems", "page_first": 10317, "page_last": 10328, "abstract": "We present a new approach for learning to solve SMT formulas. We phrase the challenge of solving SMT formulas as a tree search problem where at each step a transformation is applied to the input formula until the formula is solved. Our approach works in two phases: first, given a dataset of unsolved formulas we learn a policy that for each formula selects a suitable transformation to apply at each step in order to solve the formula, and second, we synthesize a strategy in the form of a loop-free program with branches. This strategy is an interpretable representation of the policy decisions and is used to guide the SMT solver to decide formulas more efficiently, without requiring any modification to the solver itself and without needing to evaluate the learned policy at inference time. We show that our approach is effective in practice - it solves 17% more formulas over a range of benchmarks and achieves up to 100x runtime improvement over a state-of-the-art SMT solver.", "full_text": "Learning to Solve SMT Formulas\n\nMislav Balunovi\u00b4c, Pavol Bielik, Martin Vechev\n\nDepartment of Computer Science\n\nbmislav@ethz.ch, {pavol.bielik, martin.vechev}@inf.ethz.ch\n\nETH Z\u00fcrich\nSwitzerland\n\nAbstract\n\nWe present a new approach for learning to solve SMT formulas. We phrase the\nchallenge of solving SMT formulas as a tree search problem where at each step\na transformation is applied to the input formula until the formula is solved. Our\napproach works in two phases: \ufb01rst, given a dataset of unsolved formulas we learn\na policy that for each formula selects a suitable transformation to apply at each step\nin order to solve the formula, and second, we synthesize a strategy in the form of\na loop-free program with branches. This strategy is an interpretable representation\nof the policy decisions and is used to guide the SMT solver to decide formulas\nmore ef\ufb01ciently, without requiring any modi\ufb01cation to the solver itself and without\nneeding to evaluate the learned policy at inference time. We show that our approach\nis effective in practice \u2013 it solves 17% more formulas over a range of benchmarks\nand achieves up to 100\u00d7 runtime improvement over a state-of-the-art SMT solver.\n\n1\n\nIntroduction\n\nSatis\ufb01ability Modulo Theories (SMT) solvers are a powerful class of automated theorem provers\nwhich can deduce satis\ufb01ability and validity of \ufb01rst-order formulas in particular logical theories (e.g.,\nreal numbers, arrays, bit vectors). SMT solvers are more general than SAT solvers and have been\nused in a variety of application domains including veri\ufb01cation (e.g., neural networks [28]), program\nsynthesis, static analysis, scheduling, and others [16].\nTo ef\ufb01ciently solve complex real world problems, state-of-the-art SMT solvers (e.g., Z3 [15],\nYices [18], CVC4 [4], MathSAT5 [13], Boolector [35]) contain hand-crafted heuristics combining\nalgorithmic proof methods and satis\ufb01ability search techniques. Indeed, crafting the right heuristic\nis critical and can be the difference between solving a complex formula in seconds or not at all. To\nenable end users to create suitable heuristics, several modern SMT solvers such as Z3 [15] provide a so\ncalled tactic language for expressing heuristics. Typically, such tactics are combined and performed\nin a sequence (speci\ufb01ed by the user), forming an interpretable program called a strategy. However,\nthe resulting strategies can often end up being quite complex (e.g., containing loops and conditionals).\nCombined with the vast search space, this means that manually \ufb01nding a well-performing strategy\ncan be very dif\ufb01cult, even for experts.\n\nOur Contributions We present a new approach, based on a combination of learning and synthesis\ntechniques, which addresses the problem of automatically \ufb01nding the right SMT strategy. Given\na dataset of formulas whose solution is initially unknown, we \ufb01rst train a policy that searches for\nstrategies that are fast at solving the formulas in this dataset. Then, we synthesize a single strategy\n(a program with branches) that captures the policy decision making in an interpretable manner. The\nresulting strategy is then passed on to the SMT solver which uses it to make more effective decisions\nwhen solving a given formula. We note that our approach does not require any changes to the solver\u2019s\ninternals and thus can work with any decision procedure which cleanly exports its tactics.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fWe performed an extensive experimental evaluation of our approach on formulas of varying complex-\nity across 3 different theories (QF_NRA, QF_BV and QF_NIA). We show that our synthesized strategies\nsolve 17% more formulas and are up to 100\u00d7 faster when compared to the default strategy used in\nthe state-of-the-art Z3 solver. Further, our learned strategies generalize well and can solve formulas\nwhich are much more challenging than those seen during training. Finally, we make our tool, called\nfastSMT, datasets and experiments available at http://fastsmt.ethz.ch/.\n\n2 Related work\n\nGiven the importance and wide range of SMT solver applications, a number of approaches have been\ndeveloped to improve both, their runtime as well as the range of formulas they can solve.\n\nPortfolio based approaches The most common approach of tools such as SATzilla [44],\nCPhydra [12], SUNNY [2], Proteus [22], ISAC [27] is a portfolio based method. The key idea\nis that different SMT solvers use different heuristics and hence work well for different types of\nformulas. Then, given several SMT solvers and a dataset of formulas, the goal is to learn a classi\ufb01er\nwhich uses features extracted from the given formula to select the right SMT solver (or alternatively\nde\ufb01nes order in which to run the solvers). In comparison, we address a harder problem - learn how to\ninstantiate an SMT solver with a strategy that ef\ufb01ciently solves the given dataset of formulas.\n\nEvolutionary search The work most closely related to ours is StratEVO [39] which also studies\nthe task of generating SMT strategies. However, StratEVO has several limitations \u2013 it performs\nsearch using an evolutionary algorithm which does not incorporate any form of learning, the search\ndoes not depend on the actual formula, and local mutations tend to get stuck in local minima (as we\nshow in our experiments in Section 5). As a result, for many tasks where the suitable strategy cannot\nbe trivially found (e.g., is very short) their approach reduces to random search. Instead, we leverage\nmodels that learn from previously explored strategies as well as the current formula. As we show, this\nenables discovery of complex strategies that are out of reach for approaches not based on learning.\n\nLearning branching heuristics and premise selection Recently, learning techniques have been\napplied to improving performance of SAT solvers [32, 42], constraint programming solvers [34],\nsolving quanti\ufb01ed boolean formulas [41], solving mixed integer programming problems [29] as\nwell as premise selection in interactive theorem provers [1, 43, 33]. At a high level, these are\ncomplementary to us \u2013 we learn to search across many tactics and combine them into high level\nstrategies while they optimize a single tactic (e.g., by learning which variable to branch on in SAT).\nOur work also supports formulas from a wide range of theories (as long as there is a corresponding\ntactic language) where selecting a suitable high level strategy leads to higher speedups compared\nto optimizing a single tactic in isolation. However, there are also common challenges such as\nde\ufb01ning a suitable formula representation. This representation can range from a set of hand-crafted\nfeatures [32, 36], to recursive and convolutional neural networks [33, 1], to graph embeddings [43].\nWe extend this line of work by considering fast to compute representations based on bag of words,\nsyntactic features, and features extracted from a graph representation of the formula.\n\nParameter tuning Finally, a number of approaches exist for \ufb01nding good parameter con\ufb01gurations\nfrom a vast space of both discrete and continuous parameters, including ParamILS [25], GGA [3],\nTB \u2212 SPO [24] or SMAC [23]. An interesting application of such approaches to the domain of SAT/SMT\nsolvers is proposed by SATenstein [30] which \ufb01rst designs a highly parameterized solver framework\nto be subsequently optimized by off-the shelf tools. Although such tools are not applicable for the\ntask of searching for strategy programs (that include loops and conditionals) considered in our work,\nthey can be used to \ufb01ne-tune the strategy parameters once a candidate strategy is found.\n\n3 SMT strategies: preliminaries\n\nAt a high level, an SMT solver takes as input a \ufb01rst-order logical formula and then tries to decide if\nthe formula is satis\ufb01able. In the process, solvers employ various heuristics that \ufb01rst transform the\ninput formula into a suitable representation and then use search procedures to check for satis\ufb01ability.\nExisting heuristics include: simplify which applies basic simpli\ufb01cation rules such as constant\n\n2\n\n\fTable 1: Formalization of the Strategy language used to express SMT strategies in Z3 [15].\n\n(Strategy)\n\nq\n\n(Tactics)\n(Predicates)\n(Expressions)\n(Constants)\n(Probes)\n(AOperators)\n(BOperators)\n(Parameter)\n(Parameters)\n\nt\np\nexpr\nc\nprobe\n\u2295\n(cid:46)(cid:47)\nparam\nparams\n\nt | q; q | if p then q else q | q else q |\n::=\nrepeat q, c | try q for c | using t with params\nTactics = { bit_blast, solve_eqs, elim_uncnstr . . .}\n\u2208\np \u2227 p | p \u2228 p | expr (cid:46)(cid:47) expr\n::=\nc | probe | expr \u2295 expr\n::=\n\u2208\nConsts = Q\nProbe \u2192 Q,\n::=\n::= + | \u2212 | \u2217 | /\n::= > | < | \u2265 | \u2264 | = | (cid:54)=\n::=\n::=\n\nParam = { hoist_mul, flat, som, . . .}\n\nProbe = { num_consts, is_pb, . . .}\n\n(Param, Q),\n\u0001 | param; params\n\nfolding (x + 0 \u2192 x), removal of redundant expressions (x \u2212 x \u2192 0), gaussian_elim which\neliminates variables (x = 1\u2227 y \u2265 x + z \u2192 y \u2265 1 + z) using Gaussian elimination, elim_term_ite\nwhich replaces the term if-then-else with fresh auxiliary declarations (if x > y then x else y) >\nz \u2192 k > z \u2227 (x > y \u21d2 k = x) \u2227 (x \u2264 y \u21d2 k = y), or bit_blast which reduces bit-vector\nexpressions by introducing fresh variables, one for each bit of the original bit-vector (e.g., a bit\nvector of size 4 is expanded into four fresh variables). In total, the Z3 SMT solver de\ufb01nes more than\n100 such heuristic transformations (called tactics) that can be combined together to de\ufb01ne a custom\nstrategy. For example, a strategy for integer arithmetic can be de\ufb01ned as1:\nusing simplify with (som : true); normalize_bounds; lia2pb; pb2bv; bit_blast; sat\nAlthough the above sequence of transformations (tactics) works well for some types of input formulas\n(e.g., in case every variable has a lower and an upper bound), for other formulas a different set of\ntactics is more suited. In some cases, the suitable set of tactics can be obtained by a small modi\ufb01cation\nof the original tactic while in others, a completely different set of tactics needs to be de\ufb01ned. As a\nconcrete example, consider the following strategy implemented in the Yices SMT solver [18, 17]:\n\nif (\u00acdiff \u2227 num_atoms\n\ndim\n\n< k) then simplex else floyd_warshall\n\nHere, two high level tactics can be applied to solve an input formula \u2013 the Simplex algorithm or\nthe algorithm based on Floyd-Warshall all-pairs shortest distance. The Simplex algorithm is used\nif the input formula is not in the difference logic fragment (denoted using \u00acdiff) and the ratio of\ninequalities divided by the number of uninterpreted constants is smaller than a threshold k.\n\nLanguage for expressing SMT solver strategies\nIn Fig. 1 we formalize the language used by\nZ3 to enable control over the core solver heuristics. The basic building blocks of the language\nare called tactics and represent various heuristic transformations that might be applied during the\nsearch. Optionally, each tactic can de\ufb01ne a set of parameters that affect its behavior. For example, the\nsimplify tactic contains > 50 different parameters that control which simpli\ufb01cations are performed\n(e.g., if som : true then simplify puts polynomials in som-of-monomials form). The tactics are\ncombined into larger programs either sequentially or using one of the following control structures:\n\u2022 if p then q1 else q2: If the predicate p evaluates to true apply q1, otherwise apply q2.\nThe predicate can contain arithmetic expressions as well as Probes which collect statistics\nof the current formula (e.g., num_consts returns the number of non-boolean constants).\n\n\u2022 q1 else q2: First apply q1 and if it fails then apply q2 on the original input.\n\u2022 repeat q, c: Keep applying q until it does not modify the input formula any more or the\n\u2022 try q for c: Apply q to the input and if it does not return in c millisenconds then fail.\n\u2022 using t with params: Apply the given tactic t with the given parameters params.\n\nnumber of iterations is greater than the constant c.\n\n1For more information and examples we refer the reader to the online tutorial available at:\n\nhttps://rise4fun.com/z3/tutorial/strategies\n\n3\n\n\f4 Learning SMT strategies\n\nN(cid:88)\n\ni=1\n\n(cid:26)runtime(q(fi))\n\ntimeout_penalty otherwise\n\nif q solves fi\n\n(1)\n\nWe now present our approach for synthesizing SMT strategies. Formally, the task we are interested in\nsolving is de\ufb01ned as follows:\nProblem statement Given a dataset of SMT formulas F = {fi}N\n\ni=1, our goal is to \ufb01nd a strategy:\n\nqbest = arg min\nq\u2208Strategy\n\ncost(fi, q) where cost(fi, q) def=\n\nHere, runtime(q(fi)) \u2208 Q denotes the runtime required for strategy q to solve formula fi and\ntimeout_penalty \u2208 Q is a constant denoting the penalty for not solving fi (either due to timeout\nor because the strategy q is not powerful enough). Our goal is to \ufb01nd a strategy that minimizes the\ntime required to solve the formulas in the dataset F. Note that our cost function re\ufb02ects the fact that\nwe aim to synthesize a strategy that solves as many formulas as possible yet one that is also the fastest.\nGenerally, optimizing for Equation 1 directly is problematic as using runtime makes the optimization\nproblem inherently noisy and non-deterministic. It also makes learning hard to parallelize due to\nsigni\ufb01cant impact of hardware and environment on overall runtime. Thus, instead of runtime, we use\nthe amount of work measured as the number of basic operations performed by the solver required to\nsolve a formula (e.g., implemented via the rlimit counter in Z3).\n\nChallenges To solve the problem of learning SMT strategies, one has to address two challenges:\n\u2022 Interpretability. We are interested in learning a model that is not only ef\ufb01cient at solving\na given set of formulas but also expressible as programs in the Strategy language. This is\nimportant as the learned strategies can then be directly used an input to existing solvers.\n\n\u2022 No prior domain knowledge. Our learning does not assume any prior knowledge about\nthe dataset F and which strategies work best in general \u2013 initially the solution to all the\nformulas in the dataset is unknown, no existing strategies are used to boostrap the learning\n(not even the default strategies already written by the SMT solver developers) and we do\nnot rely on any heuristics (and their combination) that may be useful in solving formulas\nfrom F. Indeed, this represents the most challenging setting for learning.\n\nOur approach The key idea behind learning a program qbest \u2208 Strategy ef\ufb01ciently is to take\nadvantage of the fact that each program q can be decomposed into a set of smaller branch-free\nprograms q1,...,k, each qi corresponding to one execution path of q. This is possible because programs\nin the Stategy language do not contain state (all state is implicitly captured in the formula being\nsolved). As a result, in our approach we learn a program qbest \u2208 Strategy via a two step process:\n\u2022 Learn candidate strategies. First, we learn a policy which \ufb01nds a set of candidate strategies\nconsisting of only sequences of tactics where each strategy performs well on a different\nsubset of the dataset F. This allows us to phrase the learning problem as a tree search over\ntactics for which state-of-the-art models can be used. This step is described in Section 4.1.\n\u2022 Synthesize a combined strategy. Then, given a set of candidate strategies, we combine these\ninto a single best strategy qbest by synthesizing control structures such as branches and loops\nas supported by the Strategy language. This step is described in Section 4.2.\n\n4.1 Learning candidate strategies\n\nWe phrase the problem of learning candidate strategies as a tree search problem. We start from an\ninitial state s0 and at each timestep t we choose an action at that expands the tree by exploring\na single edge. In our setting, a state corresponds to a tuple consisting of an SMT formula and\na strategy used to compute the formula. An action is described by a tactic and its parameters (if any)\na \u2208 Tactics \u00d7 Params. Applying an action transforms the formula into another one. Terminal\nstates are those that decide the initial formula f, that is, f was reduced to a form that is either trivially\nsatis\ufb01able or unsatis\ufb01able. Further, for practical reasons, terminal states also include those to which\napplying an action (tactic) leads to a timeout.\n\n4\n\n\fStrategy\n\n[simplify, sat, PAD, . . . ]\n\nEmbedding\n\nZ3 Probes\n\n[num_consts, is_pb, . . . ]\n\neval. on f\n\n[1733, 0, . . . ]\n\nFormula Representation\nBOW | Skip-Gram | AST\n\neval. on f\n\nEmbedding\n\nR\ne\nL\nU\n\nR\ne\nL\nU\n\nR\ne\nL\nU\n\nS\no\nf\nt\n\nM\na\nx\n\nS\ni\ng\nm\no\ni\nd\n\nTactics\n0.70 : pb2bv\n0.15 : smt\n0.01 : sat\n. . .\nParams\n0.20 : flat\n0.75 : som\n0.95 : factor\n. . .\n\nFigure 1: Architecture of the neural network model which predicts next tactic and arguments.\n\nA terminal state de\ufb01nes a strategy q (i.e., a sequence of tactics starting from the tree root) with\nan associated cost(f, q) (as de\ufb01ned in Equation 1) that we aim to minimize. During the search\nwe keep a priority queue of tuples (s, a, p) consisting of a state, possible action and its associated\nprobability, initialized with (s0, \u0001, 1). At each step, we remove the tuple with highest probability\nfrom the queue, apply its action to obtain a new state si and update the priority queue with new tuples\n(si, a, p \u00b7 \u03c0(a | si)) for all actions a \u2208 Tactics \u00d7 Params capturing the possible transitions from si.\nOur goal is to learn a policy, denoted as \u03c0(a | s), that represents a probability distribution of actions\nconditioned on a state s. Ideally, the learned distribution should be such that selecting the most likely\naction at that state minimizes the expected cost. In what follows we \ufb01rst describe the models used to\nrepresent the policy \u03c0 considered in our work and then describe how the models are trained, including\nhow to construct a suitable training dataset.\n\nBilinear model Based on matrix factor models used in unsupervised learning of word embeddings\n[11, 38] as well as supervised text classi\ufb01cation [26] we de\ufb01ne a bilinear model as follows:\n\n\u03c0(a | s) = \u03c3(U V \u03c6(s, a))\n\nwhere \u03c6(s, a) is a bag of features computed for action a taken in state s, U \u2208 Rk\u00d7h and V \u2208 Rh\u00d7|V |\nare the learnable weight matrices, V is the input vocabulary and \u03c3 is softmax computing the probability\ndistribution over output classes (in our case actions to be taken in a given state s). As the set of all\npossible actions is too large to consider, we randomly generate a subset of parameters for each tactic\nbefore training starts, thus obtaining the overall set of actions S \u2286 Tactics \u00d7 Params. We de\ufb01ne\n\u03c6(s, a) to be all n-grams constructed from the strategy of state s. For example, for strategy a1; a2; a3,\nthe extracted n-grams (features) are described by the vector: (cid:104)a1, a2, a3, a1a2, a2a3, a1a2a3(cid:105). Then,\nthe vocabulary set V is simply the collection of all possible n-grams formed over S.\n\nNeural network Our second model is based on a neural network and improves over the bilinear\nmodel in two key aspects \u2013 it considers a richer state when making the predictions and predicts tactics\nas well as their corresponding parameters. The architecture of the neural network model is illustrated\nin Fig. 1 and uses two inputs: (i) the strategy in the current state (as in the bilinear model), and (ii)\nthe formula f to be solved in the current state. The strategy is encoded by \ufb01rst padding it to be\nof \ufb01xed size, concatenating the embedding of each action in the strategy into a single vector and\n\ufb01nally feeding the resulting vector into a fully-connected layer. We encode the input formula in two\nways: \ufb01rst, by computing a set of formula measures such as the number of expressions and constants,\nand second, by learning a representation of the formula itself. The formula measures are computed\nusing probes supported by Z3 and are a subset of the possibilities one could de\ufb01ne. For the learned\nrepresentation of the formula we experimented with three different approaches:\n\nlanguage. We obtain its bag-of-words and use it as the formula embedding.\n\n\u2022 Bag-of-words (BOW): The formula is treated as a sequence of tokens from the SMT-LIB\n\u2022 Abstract Syntax Tree (AST): From the formulas\u2019s AST, we extract all subtrees of depth at\nmost two. The bag-of-words over such subtrees is used as the formula embedding. Note\nthat subtrees which appear in less than 5% of formulas in F are discarded.\n\u2022 Skip-gram: Each formula in the dataset is treated as a sequence of tokens over which\nwe learn a skip-gram model. We de\ufb01ne embedding of the formula as the average of all\nembeddings of its tokens.\n\n5\n\n\fAlgorithm 1: Iterative algorithm used to train policy \u03c0\nData: Formulas F, Number of iterations N, Number of formulas to sample K, Exploration\nResult: Trained policy \u03c0, Explored strategies Q\n1 D \u2190 \u2205; Q \u2190 \u2205; \u03c0 \u2190 policy initialization\n2 for i = 1 to N do\n3\n4\n5\n6\n\n\u02c6\u03c0 \u2190 \u03b2i\u03c0 + (1 \u2212 \u03b2i)\u03c0explore\nQ \u2190 Q \u222a Top K most likely strategies for each formula in F according to \u02c6\u03c0\nD \u2190 D \u222a Extract training dataset from strategies Q\n\u03c0 \u2190 Retrain model \u03c0 on D\n\nrates \u03b2, Exploration policy \u03c0explore (e.g., random policy)\n\n(cid:46) policy \u02c6\u03c0 explores with probability (1 \u2212 \u03b2i)\n\nThe network output consists of two parts \u2013 a probability distribution over tactics to apply next and an\nassignment to each tactic parameter. The possible tactic parameters are captured by the set Param\nfrom Fig. 1. We provide a full list of tactics and their parameters in the extended version of our paper.\nTo compute arguments for the tactic parameters, the network introduces a separate output layer for\neach parameter type. The layer performs regression and outputs normalized values in the range [0, 1].\nFor boolean parameters, values 1 and 0 correspond to true and false, respectively. For integer\nparameters, the output of the network is mapped and discretized into the range of allowed values.\n\nModel training At a high level, our training is based on the DAgger method [40] and is shown in\nAlgorithm 1. The training starts with a randomly initialized policy used to sample the top K most\nlikely strategies for each formula in the training dataset F (line 4). Selected strategies are evaluated\nand used to create a training dataset (line 5, described below) on which the policy \u03c0 is retrained\n(line 6). As the model is initially untrained the strategies are sampled at random and only as the\ntraining progresses the policy will learn to select strategies that perform well on formulas from F. As\nan alternative, one could pre-train the initial policy using strategies supplied by an expert in which\ncase the algorithm would correspond to imitation learning. However, in our work we assume such\nexpert strategies are not available and therefore we start with a model that is initialized randomly.\nBuilding the training dataset Each sample in our training dataset D = {(cid:104)(ti, pi), si(cid:105)}M\ni=1 for the\nneural network consists of a state and its associated training label. Here, the label is a tuple consisting\nof a probability distribution over tactics t \u2208 R|Tactics| and the values of all tactic parameters\np \u2208 R|Param|. The intuition behind this choice is that for a state s, the vector t encodes the likelihood\nthat a given tactic is fast at solving the formula whereas p contains the best parameter values found\nso far during training. Generating the dataset in this way encodes the preference for strategies that\nare most ef\ufb01cient in contrast to \ufb01nding any strategy that solves the input formula. To train the neural\nnetwork model using such a dataset, the loss is constructed as a weighted average of the cross-entropy\nloss for tactic prediction and mean-squared-error for parameter prediction.\nWe build the dataset as follows. First, we evaluate each strategy in Q on the formula for which it\nwas generated and keep only those that succeeded in deciding the formula. Second, let us denote\nwith r(t, si) the best runtime (or timeout) achieved from state si by \ufb01rst applying tactic t, with\nrbest(si) the best runtime achieved from state si, and with v(p, si) the value assigned to parameter p\nin tactics with the best runtime from state si. We obtain r, rbest and v by considering all the states\nand actions performed when evaluating the strategies in Q. Then, for each non-terminal state si that\neventually succeeded we create one training sample (cid:104)(\u03c3( \u02dcti), pi), si(cid:105) where \u02dcti = [1/r(t, si)]t\u2208Tactics,\n\u03c3 normalizes \u02dcti to a valid probability distribution and pi = [v(p, si)]p\u2208Param. To generate the\ntraining dataset for bilinear model we follow a similar procedure with the exception that we use\n\u02dcti = 1[rbest(si) = r(t, si)]t\u2208Tactics which assigns probability 1 to the best tactic and 0 to others.\n\nPruning via equivalence classes A challenge in training the models presented above is that whether\na strategy solves the formula is known only at terminal states. This is especially problematic for\ndatasets where majority of the effort is spent on \ufb01nding any successful strategy. To address this\nissue we take advantage of the fact that some information can be learned also from partial strategies\n\u2013 namely their current runtime and their transformed formula. This allows us to check if multiple\ntransformed formulas are equivalent (we consider two formulas equivalent if their abstract syntax\ntress are identical) and keep only the one which was fastest to reach (and prune the others).\n\n6\n\n\fa1\n\na2\n\na4\n\na3\n\na5\n\ntree representation of q1 and q2\n\nq1 = a1; a2; a3\nq2 = a1; a4; a5\n\nf1\n10\n30\n\nf2\n100\n\nTIMEOUT\n\nf3\n100\n20\n\nf1\n200\n\nf2\n250\n\nf3\n30\n\nnum_expr\n\nH(F) = \u2212(cid:88)\n\nif true then a2 else _\nif num_expr > 100 then a2 else a4\n\nformulas f1, f2, f3 and their associated costs\nbranch with cost 0.276 = (\u22121/3 \u00b7 log(1/3) \u2212 2/3 \u00b7 log(2/3)) + 0\nbranch with cost 0.2 = \u22122/3 \u00b7 log(1/2) + 1/3 \u00b7 0\n\nformula measure\n\nFigure 2: Illustration of decision tree representation of strategies, scored formulas and branches.\n\n4.2 Synthesizing a combined, interpretable strategy\n\nThe policy \u03c0 learned in Section 4.1 can be used to speed-up SMT solving as follows: invoke the\nsolver with the current formula f and the action a0 selected by \u03c0, obtain a new intermediate formula\nf1 from the solver, then again invoke the solver from scratch with f1 and a new action a1, obtaining\nanother intermediate formula f2, and so on. Unfortunately, even if the \ufb01nal formula fk is solved (e.g.,\ndetermined to be SAT), we would have lost the connection with the original formula f as at each\nstep we are making a new, fresh invocation of the SMT solver. This is problematic for tasks (e.g.,\nplanning) that require more information from SMT solvers, beyond satis\ufb01ability of f, for instance,\nthe model itself. To address this challenge without changing the internals of the SMT solver, we\npropose to synthesize an interpretable policy qbest that follows \u03c0 and can be directly expressed in the\nStrategy language from Fig. 1 (and thus be given as input to the SMT solver).\nRecall from Fig. 1 that the Strategy language de\ufb01nes two types of statements that can be used to\ncombine programs: if-then-else and or-else. However, notice that the or-else statement is in\nfact a special version of the if statement with a condition that checks if a given tactic terminates\nwithin c milliseconds. As a result, we can reduce the problem of synthesizing programs in the\nStrategy language to the problem of synthesizing branches with predicates over a set of candidate\nstrategies. We can obtain the set of candidate strategies by either evaluating \u03c0 over the training\nformulas or by extracting successful strategies from the set Q explored during policy learning. Next,\nwe discuss how to synthesize qbest.\n\nDecision tree learning To synthesize branches with predicates, we adapt techniques based on\ndecision tree learning. Consider the tree illustrated in Fig. 2 (top left) with the same structure as\nused during search (i.e., edges denoting actions and nodes denoting states) but formed from the\ntwo candidate strategies q1 and q2. Each SMT formula is evaluated by taking a path in the tree and\napplying all the corresponding actions. Intuitively, at each node with more than one outgoing edge\na decision needs to be taken to determine which path to take. To encode such decisions, our goal is to\nintroduce one or more branches at each such node (denoted as orange square in Fig. 2).\nMore formally, let Q = {q1, . . . , qn} be a set of candidate strategies, F be our dataset of formulas\nand b ::= if p then qtrue else qfalse a branch that partitions F into two parts \u2013 Ftrue are formulas\non which predicate p evaluates to true and Ffalse for the rest. We de\ufb01ne a notion of multi-label\nentropy of a dataset of formulas, denoted as H(F) [14]:\n\nq\u2208Q p(q) log(p(q)) + (1 \u2212 p(q)) log(1 \u2212 p(q))\n\nwhere p(q) denotes the ratio of formulas solved by strategy q in F. The goal of synthesis is then to\ndiscover branches that partition F into smaller sets, each of which has small entropy (i.e. either none\nor all formulas are solved). Using the entropy, we de\ufb01ne a cost associated with a branch b as:\n\ncost(b,Ftrue,Ffalse) = (|Ftrue|/|F|)H(Ftrue) + (|Ffalse|/|F|)H(Ffalse)\n\nThat is, branch cost is a weighted sum of entropies in each of the resulting branches. With this scoring\nfunction we build the decision tree in a top-down fashion \u2013 for each node with multiple outgoing\nedges we recursively synthesize predicates that minimize the cost. If dataset size for some node is\nbelow a certain threshold we greedily select the strategy which can solve the most formulas, breaking\nties using runtimes. To express branches, we consider the following types of predicates: (i) true\nwhich allows expressing a default choice, (ii) Probes with arithmetic expression as de\ufb01ned in Fig. 1,\nand (iii) try s for c which allows checking whether tactics terminate within c ms.\n\n7\n\n\fModels Comparison\n\nEffect of State Representation\n\nFigure 3: Comparison of proposed models and baselines for learning SMT solver strategies. Each line\n(higher is better) denotes the quality of the best learned strategy among top x most likely strategies\nfound by a given model (x axis) proportional to the best known strategy overall (y axis).\n\n5 Experiments\n\nWe implemented our method in a system called fastSMT and evaluated its effectiveness on 5 different\ndatasets \u2013 AProVE [19, 5], hycomp [7], core [6], leipzig [8] and Sage2 [9, 21]. These contain\nformulas of varying complexity across 3 logics QF_NRA, QF_BV and QF_NIA. The formulas have on\naverage 336, 35, 228, 153, 345 assertions, 929, 10690, 1672, 886, 1887 expressions and 118, 49,\n46, 20, 79 variables for leipzig, core, hycomp, AProVE and Sage2 benchmarks, respectively and\nrequire up to 60 MB for each formula to be stored in the SMT2-lib format. All datasets are part\nof the of\ufb01cial SMT-LIB benchmarks [10] and encode formulas from both academic and industrial\ntools (i.e. Sage and AProVE). For all datasets we split formulas in training, validation and test set in\npredetermined ratios. To train our models we use 10 iterations of Algorithm 1 with exponentially\ndecaying exploration rate to choose between policy and random action. We train the bilinear model\nusing FastText [26] for 5 epochs with learning rate 0.1 and 20 hidden units. We implemented the\nneural network model in PyTorch [37] and trained using a learning rate 10\u22124, mini batch size 32,\nAdam optimizer [31] with Xavier initialization [20] and early stopping. All our datasets, experiments,\nimplementation and extended version of this paper are available at http://fastsmt.ethz.ch/.\n\nSearch models comparison To compare different learned models we used them to obtain the 100\nmost likely strategies for each formula in our test dataset. The results are shown in Fig. 3 (left) and\nadditionally include three baseline models that perform random search, breadth-\ufb01rst search and search\nusing an evolutionary algorithm. The x-axis shows the number of most likely strategies sampled\nfrom each model and y-axis their runtime proportional to runtime of the best known strategy (i.e.,\nbest strategy from top 100 strategies across all models). Here, score one denotes that the best known\nstrategy was found whereas score zero denotes that no strategy was found that solves the formula.\nEven though baselines perform poorly, they are already able to \ufb01nd simpler strategies that can solve\nsome of the formulas. Note that in our experiment the evolutionary algorithm performed similarly to\na random model as it got easily stuck in local minima without enough exploration. Overall, the best\nmodel is the neural network which is also most complex and considers the richest set of features.\n\nEffect of state representation In Fig. 3 (right) we evaluate the effect of instantiating the neural\nnetwork model with a different set of input features capturing the current state. For our task, the\nrepresentation at the right level of complexity is bag-of-words \u2013 if the formula representation is\n\ufb02attened using pre-trained embeddings it loses the relevant information and with more complex AST\nfeatures the model suffers due to data sparsity. Further, we note that for our task the most important\nfeatures are those capturing sequences of tactics applied so far, which is illustrated by the strong\nperformance of Strategy only model in Fig. 3 (right).\n\nComparison to state-of-the-art SMT solvers We evaluate the effectiveness of the learned strate-\ngies compared to the hand-crafted strategies in Z3 4.6.2 on two metrics \u2013 number of solved formulas\nand runtime. Unless stated otherwise, we use time limit of 10 seconds allocated for solving each\nformula and instead of runtime, we use the amount of executed basic operations (using rlimit\ncounter in Z3) as a deterministic and machine independent measure of the work required to solve the\nformula. We include additional experiments with runtime in the extended version of the paper.\n\n8\n\n020406080100Numberofsampledstrategies0.00.20.40.60.81.0PerformanceEvolutionaryRandomBilinearBFSNN01020304050Numberofsampledstrategies0.00.20.40.60.81.0PerformanceStrategy+Probes+BOWStrategy+Probes+Skip-gramStrategy+Probes+ASTStrategy+ProbesStrategy\fTable 2: Comparison of best strategy found by any of our models (Section 4.1) against Z3.\n\nFormulas solved\n\nSpeedup percentile\n\nOnly Z3 Only Learned None\n\nBoth\n57\n332\n270\n273\n283\n68.3%\n\nleipzig\nSage2\ncore\nhycomp\nAProVE\nTotal\n\n3\n2\n0\n0\n0\n\n0\n246\n0\n34\n18\n\n0.3%\n\n16.8%\n\n50th\n\n90th\n10th\n5.8\u00d7 60.7\u00d7 191.5\u00d7\n8\n35.5\u00d7\n1.2\u00d7 2.7\u00d7\n220\n1.2\u00d7 1.3\u00d7\n1.9\u00d7\n0\n1.0\u00d7 1.8\u00d7\n4.0\u00d7\n18\n3.9\u00d7 87.8\u00d7 1314.0\u00d7\n14\n14.6% 2.6\u00d7 30.9\u00d7 309.4\u00d7\n\nTable 3: Comparison of the combined strategy synthesized by our approach (Section 4.2) against Z3.\n\nFormulas solved\n\nSpeedup percentile\n\nOnly Z3 Only Learned None\n\nBoth\n55\n2488\n270\n1547\n1365\n54.6%\n\nleipzig\nSage2\ncore\nhycomp\nAProVE\nTotal\n\n5\n200\n0\n93\n76\n3.6%\n\n1\n705\n0\n112\n112\n8.9%\n\n7\n\n3051\n\n10th\n90th\n50th\n21.7\u00d7\n1.4\u00d7 9.9\u00d7\n3.1\u00d7\n0.8\u00d7 1.2\u00d7\n1.2\u00d7 1.3\u00d7\n1.8\u00d7\n0\n2.3\u00d7\n0.4\u00d7 1.1\u00d7\n230\n3.2\u00d7 65.1\u00d7 860.8\u00d7\n159\n32.9% 1.4\u00d7 15.7\u00d7 178.0\u00d7\n\nTable 2 shows the number of formulas solved by Z3 compared to the best strategy found by any of\nour methods. We also measure speedups of our strategies over Z3 on all formulas which were solved\nby both methods. For example, the 90th percentile in the AProVE benchmark denotes that for 90% of\nformulas the speedup is at least 3.9\u00d7. The learned strategies signi\ufb01cantly outperform Z3 across all\nbenchmarks \u2013 solving 17% more formulas, often with up to 3 orders of magnitude speedups and with\nonly 5 formulas not solved by any of the learned strategies even though they can be solved by Z3.\nThis shows that, for all our benchmarks, the strategies found during training generalize well to other\nformulas from the same dataset.\nTable 3 shows performance of the single combined strategy synthesized as described in Section 4.2.\nHere, the result of synthesis is a program in the Strategy language that is used as input to Z3\ntogether with the formula to solve. Naturally, as the Strategy language has limited expressiveness\n(i.e., restricting the kind of expressions that can be used as branch predicates) the performance\nimprovement is smaller compared to best strategy found by any of our methods for each formula as\nshown in Table 2. However, more importantly, the improvement over the default Z3 strategy is still\nsigni\ufb01cant allowing us to solve 8.9% more formulas compared to Z3.\n\nGeneralization to harder to solve formulas So far, in all our experiments the time limit for both\ntraining and evaluation was set to 10 seconds. To evaluate how our learned strategies generalize to\nharder to solve formulas we kept the 10 seconds time limit for training but used 600 seconds time\nlimit for the evaluation. Then, our learned strategies can solve 97.7% formulas (up from 85.1%)\nacross all the benchmarks with even slightly better speedups than those shown in Table 2.\n\n6 Conclusion\n\nWe presented a new approach for solving SMT formulas based on a combination of training a policy\nthat learns to discover strategies that solve formulas ef\ufb01ciently and a synthesizer that produces\ninterpretable strategies based on this model. The synthesized strategies are represented as programs\nwith branches and are directly usable by state-of-the-art SMT solvers to guide its search. This avoids\nthe need to evaluate the learned policy at inference time and enables close integration with existing\nSMT solvers. Our technique is practically effective \u2013 it solves 17% more formulas over a range of\nbenchmarks and achieves up to 100\u00d7 runtime improvements over state-of-the-art SMT solver.\n\n9\n\n\fReferences\n[1] A. A. Alemi, F. Chollet, N. Een, G. Irving, C. Szegedy, and J. Urban. DeepMath - Deep\nSequence Models for Premise Selection. In Proceedings of the 30th International Conference\non Neural Information Processing Systems, NIPS\u201916, pages 2243\u20132251, 2016.\n\n[2] R. Amadini, M. Gabbrielli, and J. Mauro. SUNNY: a Lazy Portfolio Approach for Constraint\n\nSolving. Theory and Practice of Logic Programming, 14(4-5):509\u2013524, 2014.\n\n[3] C. Ans\u00f3tegui, M. Sellmann, and K. Tierney. A Gender-Based Genetic Algorithm for the\nAutomatic Con\ufb01guration of Algorithms. In Principles and Practice of Constraint Programming\n- CP 2009, pages 142\u2013157, 2009.\n\n[4] C. Barrett, C. L. Conway, M. Deters, L. Hadarean, D. Jovanovi\u2019c, T. King, A. Reynolds, and\nC. Tinelli. CVC4. In Proceedings of the 23rd International Conference on Computer Aided\nVeri\ufb01cation (CAV \u201911), volume 6806 of Lecture Notes in Computer Science, pages 171\u2013177,\nJuly 2011.\n\n[5] C. Barrett, P. Fontaine, and C. Tinelli. AProVE Benchmarks. https://clc-gitlab.cs.\n\nuiowa.edu:2443/SMT-LIB-benchmarks/QF_NIA/tree/master/AProVE, 2016.\n\n[6] C. Barrett, P. Fontaine, and C. Tinelli. core Benchmarks. https://clc-gitlab.cs.uiowa.\n\nedu:2443/SMT-LIB-benchmarks/QF_BV/tree/master/bruttomesso/core, 2016.\n\n[7] C. Barrett, P. Fontaine, and C. Tinelli.\n\nhycomp Benchmarks. https://clc-gitlab.\n\ncs.uiowa.edu:2443/SMT-LIB-benchmarks/QF_NRA/tree/master/hycomp, 2016.\n\n[8] C. Barrett, P. Fontaine, and C. Tinelli.\n\nleipzig Benchmarks. https://clc-gitlab.cs\n\n.uiowa.edu:2443/SMT-LIB-benchmarks/QF_NIA/tree/master/leipzig, 2016.\n\n[9] C. Barrett, P. Fontaine, and C. Tinelli. Sage2 Benchmarks. https://clc-gitlab.cs.\n\nuiowa.edu:2443/SMT-LIB-benchmarks/Sage2, 2016.\n\n[10] C. Barrett, P. Fontaine, and C. Tinelli. The Satis\ufb01ability Modulo Theories Library (SMT-LIB).\n\nwww.SMT-LIB.org, 2016.\n\n[11] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov. Enriching Word Vectors with Subword\nInformation. Transactions of the Association for Computational Linguistics, 5:135\u2013146, 2017.\n\n[12] D. Bridge, E. O\u2019Mahony, and B. O\u2019Sullivan. Case-Based Reasoning for Autonomous Constraint\n\nSolving. In Autonomous Search, pages 73\u201395. 2012.\n\n[13] A. Cimatti, A. Griggio, B. Schaafsma, and R. Sebastiani. The MathSAT5 SMT Solver. In\n\nProceedings of TACAS, volume 7795 of LNCS, 2013.\n\n[14] A. Clare and R. D. King. Knowledge discovery in multi-label phenotype data. In European\n\nConference on Principles of Data Mining and Knowledge Discovery, pages 42\u201353, 2001.\n\n[15] L. De Moura and N. Bj\u00f8rner. Z3: An Ef\ufb01cient SMT Solver. In Proceedings of the Theory\nand Practice of Software, 14th International Conference on Tools and Algorithms for the\nConstruction and Analysis of Systems, TACAS\u201908/ETAPS\u201908, pages 337\u2013340, 2008.\n\n[16] L. De Moura and N. Bj\u00f8rner. Satis\ufb01ability Modulo Theories: Introduction and Applications.\n\nCommun. ACM, 54(9):69\u201377, Sept. 2011.\n\n[17] L. de Moura and G. O. Passmore. The Strategy Challenge in SMT Solving. In Automated\n\nReasoning and Mathematics, pages 15\u201344. 2013.\n\n[18] B. Dutertre. Yices 2.2. In Computer Aided Veri\ufb01cation, pages 737\u2013744, 2014.\n\n[19] J. Giesl, C. Aschermann, M. Brockschmidt, F. Emmes, F. Frohn, C. Fuhs, J. Hensel, C. Otto,\nM. Pl\u00fccker, P. Schneider-Kamp, T. Str\u00f6der, S. Swiderski, and R. Thiemann. Analyzing Program\nTermination and Complexity Automatically with AProVE. Journal of Automated Reasoning,\n58(1):3\u201331, 2017.\n\n10\n\n\f[20] X. Glorot and Y. Bengio. Understanding the dif\ufb01culty of training deep feedforward neural\nnetworks. In In Proceedings of the International Conference on Arti\ufb01cial Intelligence and\nStatistics (AISTATS\u201910). Society for Arti\ufb01cial Intelligence and Statistics, 2010.\n\n[21] P. Godefroid, M. Y. Levin, and D. Molnar. Automated Whitebox Fuzz Testing. 2008.\n\n[22] B. Hurley, L. Kotthoff, Y. Malitsky, and B. O\u2019Sullivan. Proteus: A Hierarchical Portfolio of\nSolvers and Transformations. In Integration of AI and OR Techniques in Constraint Program-\nming, pages 301\u2013317, 2014.\n\n[23] F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential Model-based Optimization for General\nAlgorithm Con\ufb01guration. In Proceedings of the 5th International Conference on Learning and\nIntelligent Optimization, LION\u201905, pages 507\u2013523, 2011.\n\n[24] F. Hutter, H. H. Hoos, K. Leyton-Brown, and K. Murphy. Time-Bounded Sequential Parameter\n\nOptimization. In Learning and Intelligent Optimization, pages 281\u2013298, 2010.\n\n[25] F. Hutter, H. H. Hoos, K. Leyton-Brown, and T. St\u00fctzle. ParamILS: An Automatic Algorithm\n\nCon\ufb01guration Framework. J. Artif. Int. Res., 36(1):267\u2013306, 2009.\n\n[26] A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov. Bag of Tricks for Ef\ufb01cient Text Classi\ufb01ca-\ntion. In Proceedings of the 15th Conference of the European Chapter of the Association for\nComputational Linguistics: Volume 2, Short Papers, pages 427\u2013431, April 2017.\n\n[27] S. Kadioglu, Y. Malitsky, M. Sellmann, and K. Tierney. ISAC \u2013Instance-Speci\ufb01c Algorithm Con-\n\ufb01guration. In Proceedings of the 2010 Conference on ECAI 2010: 19th European Conference\non Arti\ufb01cial Intelligence, pages 751\u2013756, 2010.\n\n[28] G. Katz, C. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer. Reluplex: An Ef\ufb01cient SMT\nSolver for Verifying Deep Neural Networks. In Computer Aided Veri\ufb01cation, pages 97\u2013117,\n2017.\n\n[29] E. B. Khalil, P. L. Bodic, L. Song, G. Nemhauser, and B. Dilkina. Learning to Branch in\nMixed Integer Programming. In Proceedings of the Thirtieth AAAI Conference on Arti\ufb01cial\nIntelligence, AAAI\u201916, pages 724\u2013731, 2016.\n\n[30] A. R. KhudaBukhsh, L. Xu, H. H. Hoos, and K. Leyton-Brown. SATenstein: Automatically\nBuilding Local Search SAT Solvers from Components. In Proceedings of the 21st International\nJont Conference on Arti\ufb01cal Intelligence, IJCAI\u201909, pages 517\u2013524, 2009.\n\n[31] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. CoRR, abs/1412.6980,\n\n2014.\n\n[32] M. G. Lagoudakis and M. L. Littman. Learning to Select Branching Rules in the DPLL\nProcedure for Satis\ufb01ability. Electronic Notes in Discrete Mathematics, 9:344 \u2013 359, 2001. LICS\n2001 Workshop on Theory and Applications of Satis\ufb01ability Testing (SAT 2001).\n\n[33] S. M. Loos, G. Irving, C. Szegedy, and C. Kaliszyk. Deep Network Guided Proof Search. In\n\nLPAR, volume 46 of EPiC Series in Computing, pages 85\u2013105, 2017.\n\n[34] M. Loth, M. Sebag, Y. Hamadi, and M. Schoenauer. Bandit-based Search for Constraint\nProgramming. In Proceedings of the 19th International Conference on Principles and Practice\nof Constraint Programming, CP\u201913, pages 464\u2013480, 2013.\n\n[35] A. Niemetz, M. Preiner, and A. Biere. Boolector 2.0 system description. Journal on Satis\ufb01ability,\n\nBoolean Modeling and Computation, 9:53\u201358, 2014 (published 2015).\n\n[36] E. Nudelman, K. Leyton-Brown, H. H. Hoos, A. Devkar, and Y. Shoham. Understanding\nRandom SAT: Beyond the Clauses-to-variables Ratio. In Proceedings of the 10th International\nConference on Principles and Practice of Constraint Programming, CP\u201904, pages 438\u2013452,\n2004.\n\n[37] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison,\n\nL. Antiga, and A. Lerer. Automatic differentiation in PyTorch. 2017.\n\n11\n\n\f[38] J. Pennington, R. Socher, and C. Manning. Glove: Global Vectors for Word Representation. In\nProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing\n(EMNLP), pages 1532\u20131543, 2014.\n\n[39] N. G. Ram\u00edrez, Y. Hamadi, E. Monfroy, and F. Saubion. Evolving SMT Strategies. In IEEE\n28th International Conference on Tools with Arti\ufb01cial Intelligence (ICTAI), pages 247\u2013254,\n2016.\n\n[40] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction\nto no-regret online learning. In Proceedings of the fourteenth international conference on\narti\ufb01cial intelligence and statistics, pages 627\u2013635, 2011.\n\n[41] H. Samulowitz and R. Memisevic. Learning to Solve QBF. In Proceedings of the 22nd National\n\nConference on Arti\ufb01cial Intelligence - Volume 1, AAAI\u201907, pages 255\u2013260, 2007.\n\n[42] P. Somol, P. Pudil, and J. Kittler. Fast Branch & Bound Algorithms for Optimal Feature\n\nSelection. IEEE Trans. Pattern Anal. Mach. Intell., 26(7):900\u2013912, July 2004.\n\n[43] M. Wang, Y. Tang, J. Wang, and J. Deng. Premise Selection for Theorem Proving by Deep Graph\nEmbedding. In Advances in Neural Information Processing Systems 30, pages 2786\u20132796.\n2017.\n\n[44] L. Xu, F. Hutter, H. H. Hoos, and K. Leyton-Brown. SATzilla: Portfolio-based Algorithm\n\nSelection for SAT. J. Artif. Int. Res., 32(1):565\u2013606, June 2008.\n\n12\n\n\f", "award": [], "sourceid": 6603, "authors": [{"given_name": "Mislav", "family_name": "Balunovic", "institution": "ETH Zurich"}, {"given_name": "Pavol", "family_name": "Bielik", "institution": "ETH Zurich"}, {"given_name": "Martin", "family_name": "Vechev", "institution": "DeepCode and ETH Zurich, Switzerland"}]}