{"title": "VLSI Implementation of a High-Capacity Neural Network Associative Memory", "book": "Advances in Neural Information Processing Systems", "page_first": 793, "page_last": 800, "abstract": null, "full_text": "VLSI Implementation of a High-Capacity Neural Network \n\n793 \n\nVLSI Implementation of a High-Capacity \n\nNeural Network Associative Memory \n\nTzi-Dar Chiueh 1 and Rodney M. Goodman \n\nDepartment of Electrical Engineering (116-81) \n\nCalifornia Institute of Technology \n\nPasadena, CA 91125, USA \n\nABSTRACT \n\nIn this paper we describe the VLSI design and testing of a high \ncapacity associative memory which we call the exponential cor(cid:173)\nrelation associative memory (ECAM). The prototype 3J.'-CMOS \nprogrammable chip is capable of storing 32 memory patterns of \n24 bits each. The high capacity of the ECAM is partly due to the \nuse of special exponentiation neurons, which are implemented via \nsub-threshold MOS transistors in this design. The prototype chip \nis capable of performing one associative recall in 3 J.'S. \n\n1 ARCHITECTURE \n\nPreviously (Chiueh, 1989), we have proposed a general model for correlation-based \nassociative memories, which includes a variant of the Hopfield memory and high(cid:173)\norder correlation memories as special cases. This new exponential correlation as(cid:173)\nsociative memory (ECAM) possesses a very large storage capacity, which scales \nexponentially with the length of memory patterns (Chiueh, 1988). Furthermore, it \nhas been shown that the ECAM is asymptotically stable in both synchronous and \n\n1 Tzi-Dar Chiueh is now with the Department of Electrical Engineering, National Taiwan Uni(cid:173)\n\nversity, Taipei, Taiwan 10764. \n\n\f794 \n\nChiueh and Goodman \n\nasynchronous updating modes (Chiueh, 1989). The model is based on an archi(cid:173)\ntecture consisting of binary connection weights, simple hard-limiter neurons, and \nspecialized nonlinear circuits as shown in Figure 1. The evolution equation of this \ngeneral model is \n\n(1) \n\nwhere u(1), u(2), ... , u(M) are the M memory patterns. x and x, are the current and \nthe next state patterns of the system respectively, and sgn is the threshold function, \nwhich takes on the value +1 if its argument is nonnegative, and -1 otherwise. \nWe addressed, in particular, the case where f(\u00b7) is in the form of an exponentiation, \nnamely, when the evolution equation is given by \n\n(2) \n\nand a is a constant greater than unity. \n\nThe ECAM chip we have designed is programmable; that is, one can change the \nstored memory patterns at will. To perform an associative recall, one first loads a \nset of memory patterns into the chip. The chip is then switched to the associative \nrecall mode, an input pattern is presented to the ECAM chip, and the ECAM chip \nthen computes the next state pattern according to Equation (2). The components \nof the next state pattern appear at the output in parallel after the internal circuits \nhave settled. Feedback is easily incorporated by connecting the output port to the \ninput port, in which case the chip will cycle until a fixed point is reached. \n\n2 DESIGN OF THE ECAM CIRCUITS \n\nFrom the evolution equation of the ECAM, we notice that there are essentially three \ncircuits that need to be designed in order to build an ECAM chip. They are: \n\n\u2022 < u(1:), x >, the correlation computation circuit; \n\u00b7 I: a__ u(1:), the exponentiation, multiplication and summing circuit; \n\nM \n\n1:=1 \n\n\u2022 sgn( .), the threshold circuit. \n\nWe now describe each circuit, present its design, and finally integrate all these \ncircuits to get the complete design of the ECAM chip. \n\n\fVLSI Implementation ora High-Capacity Neural Network \n\n795 \n\n2.1 CORRELATION COMPUTATION \n\nIn Figure 2, we illustrate a voltage-divider type circuit consisting of NMOS transis(cid:173)\ntors working as controlled resistors (linear resistors or open circuits). This circuit \ncomputes the correlation between the input pattern x and a memory pattern u(l:). \nIf the ith components of these two patterns are the same, the corresponding XOR \ngate outputs a \"0\" and there is a connection from the node V~~ to VBB; other(cid:173)\nwise, there is a connection from V~~ to GND. Hence the output voltage will be \nproportional to the number of positions at which x and u(l:) match. The maximum \noutput voltage is controlled by an externally supplied bias voltage VBB. Normally, \nVBB is set to a voltage lower than the threshold voltage of NMOS transistors (VTH) \nfor a reason that will be explained later. Note that the conductance of an NMOS \ntransistor in the ON mode is not fixed, but rather depends on its gate-to-source \nvoltage and its drain-to-source voltage. Thus, some nonlinearity is bound to occur \nin the correlation computation circuit, however, simulation shows that this effect is \nsmall. \n\n2.2 EXPONENTIATION, MULTIPLICATION, AND SUMMATION \n\nFigure 4 shows a circuit that computes the exponentiation of V~~, the product of \nthe u~l:) and the exponential, and the sum of all M products. \nThe exponentiation function is implemented by an NMOS transistor whose gate \nvoltage is V~~. Since VBB, the maximum value that V~~ can assume, is set to be \nlower than the threshold voltage (VTH); the NMOS transistor is in the subthreshold \nregion, where its drain current depends exponentially on its gate-to-source voltage \n(Mead, 1989). If we temporarily ignore the transistors controlled by u~l:) or the \ncomplement of u~l:), the current flowing through the exponentiation transistor asso(cid:173)\nciated with V~~ will scale exponentially with V~~. Therefore, the exponentiation \nfunction is properly computed. \nSince the multiplier u~l:) assumes either +1 or -1, the multiplication can be easily \ndone by forming two branches, each made up of a transmission gate in series with an \nexponentiation transistor whose gate voltage is V~~. One of the two transmission \ngates is controlled by u~l:), and the other by the complement of u~l:). Consequently, \nwhen u~l:) = 1, the positive branch will carry a current that scales exponentially \nwith the correlation of the input x and the ph memory pattern u(l:) , while the \nnegative branch is essentially an open circuit, and vice versa. \n\nSummation of the M terms in the evolution equation is done by current summing. \nThe final results are two currents It and Ii, which need to be compared by a \nthreshold circuit to determine the sign of the ith bit of the next state pattern x~. \nIn the ECAM a simple differential amplifier (Figure 3) performs the comparison. \n\n\f796 \n\nCbiueb and Goodman \n\n2.3 THE BASIC ECAM CELL \n\nThe above computational circuits are then combined with a simple static RAM cell, \nto make up a basic ECAM cell as illustrated in Figure 5. The final design of an \nECAM that stores M N-bit memory patterns can be obtained by replicating the \nbasic ECAM cell M times in the horizontal direction and N times in the vertical \ndirection, together with read/write circuits, sense amplifiers, address decoders, and \nI/O multiplexers. The prototype ECAM chip is made up of 32 x 24 ECAM cells, \nand stores 32 memory patterns each 24 bits wide. \n\n3 ECAM CHIP TEST RESULTS \n\nThe test procedure for the ECAM is to first generate 32 memory patterns at random \nand then program the ECAM chip with these 32 patterns. We then pick a memory \npattern at random, flip a specified number of bits randomly, and feed the resulting \npattern to the ECAM as an input pattern (x). The output pattern (x') can then be \nfed back to the inputs of the ECAM chip. This iteration continues until the pattern \nat the input is the same as that at the the output, at which time the ECAM chip \nis said to have reached a stable state. We select 10 sets of 32 memory patterns and \nfor each set we run the ECAM chip on 100 trial input patterns with a fixed number \nof errors. Altogether, the test consists of 1000 trials. \n\nIn Figure 6, we illustrate the ECAM chip test results. The number of successes is \nplotted against the number of errors in the input patterns for the following four \ncases: 1) The ECAM chip with VBB = 5V; 2) VBB = 2V; 3) VBB = IV; and 4) a \nsimulated ECAM in which the exponentiation constant a, equals 2. It is apparent \nfrom Figure 6 that as the number of errors increases, the number of successes \ndecreases, which is expected. Also, one notices that the simulated ECAM is by far \nthe best one, which is again not unforeseen because the ECAM chip is, after all, \nonly an approximation of the ideal ECAM model. \nWhat is really unexpected is that the best performance occurs for VBB = 2V rather \nthan VBB = IV (VTH in this CMOS process). This phenomenon arises because \nof two contradictory effects brought about by increasing VBB. On the one hand, \nincreasing VBB increases the dynamic range of the exponentiation transistors in the \nECAM chip. Suppose that the correlations of two memory patterns u(l) and u(k) \nwith the input pattern x are tJ and tk, respectively, where tJ > tk; then \n\nV(I) _ (tJ + N) VBB \n\n(k) _ (tk + N) VBB \n\nux -\n\n2N \n\n,V ux -\n\n2N \n\n. \n\nTherefore, as VBB increases, so does the difference between V~I~ and V~~, and u(l) \nbecomes more dominant than u(k) in the weighted sum of the evolution equation. \n\n\fVLSI Implementation or a High\u00b7Capacity Neural Network \n\n797 \n\nHence, as VBB increases, the error correcting ability of the ECAM chip should \nimprove. On the other hand, as VBB increases beyond the threshold voltage, the \nexponentiation transistors leave the subthreshold region and may enter saturation, \nwhere the drain current is approximately proportional to the square of the gate(cid:173)\nto-source voltage. Since a second-order correlation associative memory in general \npossesses a smaller storage capacity than an ECAM, one would expect that with \na fixed number of loaded memory patterns, the ECAM should do better than the \nsecond-order correlation associative memory. Thus one effect tends to enhance the \nperformance of the ECAM chip, while the other tends to degrade it. A compromise \nbetween these two effects is reached, and the best performance is achieved when \nVBB = 2V. \nFor the case when VBB = 2V, the drain current versus gate-to-source voltage char(cid:173)\nacteristic of the exponentiation transistors is actually a hybrid of a square function \nand an exponentiation function. At the bottom it is of an exponential form, and \nit gradually flattens out to a square function, once the gate-to-source voltage be(cid:173)\ncomes larger than the threshold voltage. Therefore, the ECAM chip with VBB = \n2V is a mixture of the second-order correlation associative memory and the pure \nECAM . According to the convergence theorem for correlation associative memories \n(Chiueh, 1989) and the fact that f(\u00b7) in the ECAM chip with VBB = 2V is still \nmonotonically nondecreasing, the ECAM chip is still asymptotically stable when \nVBB = 2V. \nWe have tested the speed of the ECAM chip using binary image vector quantization \nas an example problem. The speed at which the ECAM chip can vector-quantize \nbinary images is of interest. We find experimentally that the ECAM chip is capable \nof doing one associative recall operation, in less than 3 j.ts, ' n 4 x 4 blocks. This \nprojects to approximately 49 ms for a 512 x 512 binary image, or more than 20 \nimages per second . \n\n4 CONCLUSIONS \n\nIn this paper, we have presented a VLSI circuit design for implementing a high \ncapacity correlation associative memory. The performance of the ECAM chip is \nshown to be almost as good as a computer-simulated ECAM . Furthermore, we \nbelieve that the ECAM chip is more robust than an associative memory using a \nwinner-take-all function, because it obtains its result via iteration, as opposed to \none shot. In conclusion, we believe that the ECAM chip provides a fast and efficient \nway for solving many associative recall problems, such as vector quantization and \noptical character recognition. \n\nAcknowledgement \n\nThis work was supported in part by NSF grant No . MIP - 8711568. \n\n\f798 \n\nChiueh and Goodman \n\nReferences \n\nT. D. Chiueh and R. M. Goodman. (1988) \"High Capacity Exponential Associative \nMemory,\" in Proc. of IEEE IeNN, Vol. I, pp. 153-160. \n\nT. D. Chiueh. \nNetworks,\" Ph. D. dissertation, California Institute of Technology. \n\n(1989) \"Pattern Classification and Associative Recall by Neural \n\nC. A. Mead. (1989) Analog VLSI and Neural Systems. Reading, MA : Addison(cid:173)\nWesley. \n\nFigure 1: Architecture of the General Correlation-Based Associative Memory \n\n(I<) \nu \nN-l N-l \n\nX \n\nFigure 2: The Correlation Computation Circuit \n\n\fVLSI Implementation or a High-Capacity Neural Network \n\n799 \n\nVoo \n\nX'. \nI \n\nI \n\n+ \nI \n\nFigure 3: The Threshold Circuit \n\n-\nI i \n\n\u2022 \u2022 \u2022 \n\nV (1) \nux \n\n(1) \nU. \nI \n\nV (2) \nux \n\n(2) \nU. \nI \n\nV(M) \nux \n\n(M) \n\nU. \nI \n\nFigure 4: The Exponentiation, Multiplication, and Summation Circuit \n\n\f800 \n\nChiueh and Goodman \n\nI . \n1 \n\nI . \n1 \n\nV(k) \nux \n\n(1<) \nU. \n1 \n\nRAM \ncell \n\nr., \n\n(1<) \nu. \n1 \n\n(1<) u. \n\n1 \n\nFigure 5: Circuit Diagram of the Basic ECAM Cell \n\n1000 G \n\n900 \n\n\u2022 \n\n..... \nfIl \nres \n.... \n.~ \n0 \n8 \n..... \n..... \n\nZ \n\n.. Simulation (a=2) \n... Vbb=5V \n0- Vbb =2V \n-I- Vbb = lV \n\n800 \n700 \n\n0 .... g 600 \n~ fIl \n500 \n~ \n~ 400 \n..... \n0 \n'\"' \nQ) ] 200 \n\n300 \n\n100 \n\n0 \n\n0 \n\n1 \n\n2 \n\n3 \n\n4 \n\n5 \n\n6 \n\n7 \n\nNumber of errors in input patterns \n\nFigure 6: Error Correcting Ability of the ECAM Chip with Different VBB compared \nwith a Simulated ECAM with a = 2 \n\n\f", "award": [], "sourceid": 217, "authors": [{"given_name": "Tzi-Dar", "family_name": "Chiueh", "institution": null}, {"given_name": "Rodney", "family_name": "Goodman", "institution": null}]}__