Submitted by
Assigned_Reviewer_1
Q1: Comments to author(s).
First provide a summary of the paper, and then address the following
criteria: Quality, clarity, originality and significance. (For detailed
reviewing guidelines, see
http://nips.cc/PaperInformation/ReviewerInstructions)
The paper investigates how correlation among synaptic
weights, not correlation among neural activity, influences the retrieval
performance of autoassociative memory. Authors studied two types of
wellknown learning rules, additive learning rule (e.g., Hebbian learning)
and palimpsest learning rule (e.g., cascade learning), and showed that
synaptic correlations are induced in most of the cases. They also
investigated optimal retrieval dynamics and showed that there exists a
local version of dynamics that can be implemented in neural networks
(except for an XOR cascade model).
 Quality Theoretical
background of the paper is solid and sound. Their claims are well
supported by theoretical analysis and experimental data.
 Clarity
The paper was organized well and is written clearly.

Originality Statistical dependencies among neuronal activity have been
studied extensively. However, statistical relationship between synaptic
weights has not attracted much attention. To the reviewer’s knowledge,
this is the first attempt to treat this important problem with solid
theoretical analysis. The review considers that the paper is highly
original.
 Significance The paper has provided important
results to a neuroscience community. The paper will have significant
influence on how we look at statistical properties of neural dynamics.
Finally, here is one comment for improving the manuscript.
 Lines 42 & 409 The authors compare their results with
Song et al. (2005), but they described the recording area incorrectly;
Song et al. recorded pyramidal neurons in rat “visual” cortex, not
somatosensory cortex. Please revise the manuscript accordingly.
Q2: Please summarize your review in 12
sentences
The paper investigates correlations among synaptic
weights that have been ignored in previous computational studies. It
elucidated important and nontrivial results and is expected to have major
impacts to a neuroscience community. Submitted by
Assigned_Reviewer_4
Q1: Comments to author(s).
First provide a summary of the paper, and then address the following
criteria: Quality, clarity, originality and significance. (For detailed
reviewing guidelines, see
http://nips.cc/PaperInformation/ReviewerInstructions)
Presents a nice analysis of learning rules and recall
dynamics in associative memory models. The paper addresses the problem of
correlations in weights that arises from additive or other types of
learning rules, and proposes recall dynamics  which can be implemented
through local interactions  that can help mitigate the destructive
interference that results from these correlations.
Q2: Please summarize your review in 12
sentences
Presents a useful and insightful analysis of
associative memory models and a derivation of more optimal recall
dynamics.
Submitted by
Assigned_Reviewer_5
Q1: Comments to author(s).
First provide a summary of the paper, and then address the following
criteria: Quality, clarity, originality and significance. (For detailed
reviewing guidelines, see
http://nips.cc/PaperInformation/ReviewerInstructions)
The authors study the role of correlations between
synaptic connections in a recurrent network induced by synaptic learning
rules. The authors separate the contribution of learning rules by making
the simplifying assumption that the activity patterns which drive learning
do not themselves have correlations. They proceed to discuss two classes
of learning rules, additive and cascade models, derive and examine the
differences in the correlations induced by them and their effect on the
accuracy, simplicity and plausibility of recall dynamics. The paper
follows in the path of several recent papers discussing recall dynamics as
probabilistic inference, is generally clearly written, interesting and has
substantial results that should be of interest to the general NIPS
community.
Major comments The first part of the paper overlaps
with previously published papers. Though this introduction is important
and some relevant papers are referenced by the authors, it must be made
much clearer where the novel parts are found and which results are already
published.
Minor comments In general, one can certainly
appreciate trying to write in a more engaging, elevated form but some
sentences are a bit too poetical. For instance, “assuaging an old enemy
leads to…” is a bit much.
Lines 3941: this sentence is too vague
and imprecise. As the authors themselves will state in a few pages, the
perfect symmetry of Hopfield networks (an obvious synaptic correlation) is
well known, and heavily studied issue. The authors should be more precise
with what kinds of correlations have not been explored and in what
context, or at least tone down this statement.
Line 7778: “recall
is inherently a probabilistic inference…” that is not the only approach,
rather the authors can claim that “recall can be seen as a probabilistic…”
Footnote page 2 “the the”, one “the” is a typo
Line
390392: “Statistical dependencies between synaptic efficacies are a
natural consequence of activity dependent synaptic plasticity, and yet
their implications for network function have been unexplored.” Again, this
statement has to be made more specific considering the Hopfield network
literature.
Footnote #10: this statement should be put into the
discussion at the point where the footnote was referenced.
Line
414415: “the deeper is the cascade” should be “the deeper the cascade”
Postfeedback response: My comments were mostly minor, the
authors addressed them in their response and I am sure will be able to
correct everything for the final version. Q2: Please
summarize your review in 12 sentences
This paper studying the effect of correlation in
synaptic weights induced by synaptic learning on memory recall is clear,
well written and has a number of interesting results.
Q1:Author
rebuttal: Please respond to any concerns raised in the reviews. There are
no constraints on how you want to argue your case, except for the fact
that your text should be limited to a maximum of 6000 characters. Note
however that reviewers and area chairs are very busy and may not read long
vague rebuttals. It is in your own interest to be concise and to the
point.
We thank the reviewers for their comments.
R1:
Sorry for the inaccuracy in our description of the Song et al
experiment. We will fix this.
R3:  The novel part of the
paper starts after Eq.2 (p.2). We will edit the text there to make it very
explicit that the general formulation of autoassociative recall as
probabilistic inference, and deriving recall dynamics from this stance, is
not new (citing all the relevant refs), but the explicit treatment of
correlations in P(Wx) is.
 We will make sure the style is not
overly poetic. Sorry for the imprecision.
 We will make it clear
upfront (in the 1st sentence of para 2, p.1, and again in the para after
Eq.2, p.2) that the nontrivial correlations we are focussing on are those
that go beyond the perfect correlations or anticorrelations emerging
between reciprocal synapses with precisely symmetric or antisymmetric
learning rules (eg. in the Hopfield network), as has been noted.

We will incorporate Footnote 10 into the main text.
 We will
correct the other typographical / stylistic errors pointed out by the
Reviewer.
