{"title": "Neural Networks: The Early Days", "book": "Advances in Neural Information Processing Systems", "page_first": 828, "page_last": 842, "abstract": null, "full_text": "828 \n\nCowan \n\nNeural networks: the early days \n\nJ.D. Cowan \n\nDepartment of Mathematics, Committee on \nNeurobiology, and Brain Research Institute, \n\nThe University of Chicago, 5734  S. Univ. Ave., \n\nChicago, Illinois 60637 \n\nABSTRACT \n\nA short account is  given of various  investigations of neural  network \nproperties,  beginning  with  the  classic  work of McCulloch  & Pitts. \nEarly work on neurodynamics and statistical mechanics, analogies with \nmagnetic materials, fault tolerance via parallel distributed processing, \nmemory, learning,  and pattern recognition,  is described. \n\n1  INTRODUCTION \n\nIn this brief account of the early days in neural network research, it is not possible to be \ncomprehensive.  This article then is a somewhat subjective survey of some, but not all, of \nthe developments in the theory of neural networks in the twent-five year period, from \n1943 to  1968, when  many of the ideas and concepts were formulated,  which define the \nfield  of neural network research.  This comprises work on  connections with automata \ntheory and computability; neurodynamics,  both deterministic  and statistical;  analogies \nwith magnetic materials and spin systems; reliability via parallel and parallel distributed \nprocessing; modifiable synapses and conditioning; associative memory; and supervised \nand unsupervised learning. \n\n2  McCULLOCH-PITTS NETWORKS \n\nThe modem era may be said to have begun with the work of McCulloch and Pitts (1943). \nThis  is  too  well-known  to  need commenting on. Let me just make  some historical re(cid:173)\nmarks.  McCulloch, who was by training a psychiatrist and neuroanatomist, spent some \ntwenty years thinking about the representation of event in the nervous system. From 1941 \nto 1951 he worked in Chicago. Chicago at that time was one of the centers of neural of \n\n\fNeural Networks:  The Early Days \n\n829 \n\nFigure1: Warren McCulloch circa  1962 \n\nnetwork research,  mainly through the work of the Rashevsky group in the Committee on \nMathematical Biology at the University of Chicago. Rashevsky, Landahl, Rapaport and \nShim bel,  among others, carried out many early investigations of the dynamics of neural \nnetworks, using a mixture of calculus and algebra. In  1942 McCulloch was introduced to \nWalter Pitts, then a 17 year old student of Rashevsky's. Pitts was a mathematical prodigy \nwho had joined the  Committee  sometime in  1941. There is  an  (apocryphal)  story  that \nPitts  was  led  to  the  Rashevsky  group  after  a  chance  meeting  with  the  philosopher \nBertrand Russell,  at that time a  visitor to  the University of Chicago.  In  any event Pitts \nwas already working on algebraic aspects of neural networks, and it did not take him long \nto  see  the  point  behind  McCulloch's  quest  for  the  embodiment  of mind. \nIn  one  of \nMcCulloch later essays (McCulloch 1961) he describes the history of his efforts thus: \n\nMy  object, as a psychologist, was to invent a least psychic event, or \n\"psychon\", that would have the following properties: First, it was to be \nso simple an event that it either happened or else it did  not  happen. \nSecond, it was to happen only if its bound cause had happened-shades \nof Duns  Scotus!-that  is,  it  was  to  imply  its  temporal  antecedent. \nThird  it  was  to  propose  this  to  subsequent psychons.  Fourth,  these \nwere  to  be  compounded  to  produce  the  equivalents  of  more \ncomplicated  propositions  concerning  their  antecedents .. .In  1921  it \ndawned  on  me  that  these  events  might  be  regarded  as  the  all-or(cid:173)\nnothing impulses of neurons, combined by convergence upon the next \nneuron to yield complexes of propositional events. \n\nTheir subsequent  1943  paper  was  remarkable  in  many respects.  It is  best appreciated \nwithin  the  zeitgeist of the  era when  it was written.  As  Papert has  documented in  his \nintroduction to a collection of McCulloch's papers (McCulloch 1967), 1943 was a semi-\n\n\f830 \n\nCowan \n\nnal year  for the development of the science of the mind.  Craik's monograph The Nature \nof Explanation  and  the  paper  \"Behavior,  Purpose  and  Teleology,  by  Rosenbleuth, \nWiener  and  Bigelow,  were  also  published  in  1943.  As  Papert  noted,  \"The  common \nfeature  [of  these  publications]  is  their  recognition  that  the  laws  governing  the \nembodiment of mind should be sought among the laws governing information rather than \nenergy  or  matter\".  The  paper  by  McCulloch  and  Pitts  certainly  lies  within  this \nframework. \n\nFigure 2: Walter Pitts circa 1952 \n\nMcCulloch-Pitts  networks  (hence-forth  referred  to  as  MP  networks),  are  finite  state \nautomata embodying  the  logic of propositions, with quantifiers, as McCulloch wished; \nand permit the framing of sharp hypotheses about the nature of brain mechanisms, in a \nform  equivalent  to  computer  programs.  This  was  a  remarkable  achievement.  It \nestablished once and for all, the validity of making formal  models of brain mechanisms, \nif not their veridicality. It also established the possibility of a rigorous theory of mind, in \nthat  neural  networks  with  feedback  loops  can  exhibit  purposive  behavior,  or  as \nMcCulloch and Pitts put it: \n\nboth  the  formal  and the  final  aspects of that  activity  which  we are \nwont  to  call  mental  are  rigorously  deducible  from  present \nneurophysiology ... [and]  that in  [imaginable  networks] ... \"Mind\"  no \nlonger \"goes more ghostly than a ghost\". \n\n2.1  FAULT TOLERANCE \n\n:MP networks were the first designed to perform specific logical tasks; and of course logic \ncan  be  mapped  into arithmetic.  Landahl,  McCulloch  and  Pitts  (1943),  for  example, \nnoted that the arithmetical operations +, 1-, and x can be obtained in MP networks via the \nlogical operations OR. NOT, and AND. Thus the arithmetical expression a-a.b = a.(l-b) \n\n\f", "award": [], "sourceid": 198, "authors": [{"given_name": "Jack", "family_name": "Cowan", "institution": null}]}