The paper investigates how correlation among synaptic weights, not correlation among neural activity, influences the retrieval performance of auto-associative memory. Authors studied two types of well-known learning rules, additive learning rule (e.g., Hebbian learning) and palimpsest learning rule (e.g., cascade learning), and showed that synaptic correlations are induced in most of the cases. They also investigated optimal retrieval dynamics and showed that there exists a local version of dynamics that can be implemented in neural networks (except for an XOR cascade model).

- Quality
Theoretical background of the paper is solid and sound. Their claims are well supported by theoretical analysis and experimental data.

- Clarity
The paper was organized well and is written clearly.

- Originality
Statistical dependencies among neuronal activity have been studied extensively. However, statistical relationship between synaptic weights has not attracted much attention. To the reviewer’s knowledge, this is the first attempt to treat this important problem with solid theoretical analysis. The review considers that the paper is highly original.

- Significance
The paper has provided important results to a neuroscience community. The paper will have significant influence on how we look at statistical properties of neural dynamics.

Finally, here is one comment for improving the manuscript.

- Lines 42 & 409
The authors compare their results with Song et al. (2005), but they described the recording area incorrectly; Song et al. recorded pyramidal neurons in rat “visual” cortex, not somatosensory cortex. Please revise the manuscript accordingly.

The paper investigates correlations among synaptic weights that have been ignored in previous computational studies. It elucidated important and non-trivial results and is expected to have major impacts to a neuroscience community.

Presents a nice analysis of learning rules and recall dynamics in associative memory models. The paper addresses the problem of correlations in weights that arises from additive or other types of learning rules, and proposes recall dynamics - which can be implemented through local interactions - that can help mitigate the destructive interference that results from these correlations.

Presents a useful and insightful analysis of associative memory models and a derivation of more optimal recall dynamics.

The authors study the role of correlations between synaptic connections in a recurrent network induced by synaptic learning rules. The authors separate the contribution of learning rules by making the simplifying assumption that the activity patterns which drive learning do not themselves have correlations. They proceed to discuss two classes of learning rules, additive and cascade models, derive and examine the differences in the correlations induced by them and their effect on the accuracy, simplicity and plausibility of recall dynamics.
The paper follows in the path of several recent papers discussing recall dynamics as probabilistic inference, is generally clearly written, interesting and has substantial results that should be of interest to the general NIPS community.

Major comments
The first part of the paper overlaps with previously published papers. Though this introduction is important and some relevant papers are referenced by the authors, it must be made much clearer where the novel parts are found and which results are already published.

Minor comments
In general, one can certainly appreciate trying to write in a more engaging, elevated form but some sentences are a bit too poetical. For instance, “assuaging an old enemy leads to…” is a bit much.

Lines 39-41: this sentence is too vague and imprecise. As the authors themselves will state in a few pages, the perfect symmetry of Hopfield networks (an obvious synaptic correlation) is well known, and heavily studied issue. The authors should be more precise with what kinds of correlations have not been explored and in what context, or at least tone down this statement.

Line 77-78: “recall is inherently a probabilistic inference…” that is not the only approach, rather the authors can claim that “recall can be seen as a probabilistic…”

Footnote page 2 “the the”, one “the” is a typo

Line 390-392: “Statistical dependencies between synaptic efficacies are a natural consequence of activity dependent synaptic plasticity, and yet their implications for network function have been unexplored.” Again, this statement has to be made more specific considering the Hopfield network literature.

Footnote #10: this statement should be put into the discussion at the point where the footnote was referenced.

Line 414-415: “the deeper is the cascade” should be “the deeper the cascade”


Post-feedback response:
My comments were mostly minor, the authors addressed them in their response and I am sure will be able to correct everything for the final version.

This paper studying the effect of correlation in synaptic weights induced by synaptic learning on memory recall is clear, well written and has a number of interesting results.

We thank the reviewers for their comments.

R1:
Sorry for the inaccuracy in our description of the Song et al experiment. We will fix this.

R3:
- The novel part of the paper starts after Eq.2 (p.2). We will edit the text there to make it very explicit that the general formulation of autoassociative recall as probabilistic inference, and deriving recall dynamics from this stance, is not new (citing all the relevant refs), but the explicit treatment of correlations in P(W|x) is.

- We will make sure the style is not overly poetic. Sorry for the imprecision.

- We will make it clear upfront (in the 1st sentence of para 2, p.1, and again in the para after Eq.2, p.2) that the non-trivial correlations we are focussing on are those that go beyond the perfect correlations or anti-correlations emerging between reciprocal synapses with precisely symmetric or anti-symmetric learning rules (eg. in the Hopfield network), as has been noted.

- We will incorporate Footnote 10 into the main text.

- We will correct the other typographical / stylistic errors pointed out by the Reviewer.