Modelling and inference for dynamics in complex and disordered systems

11 Jun 2014, 09:00 → 13 Jun 2014, 18:00 Europe/Stockholm

FA32 (Albanova University Centre)

Manfred Opper, Peter Sollich, Yasser Roudi (NTNU/Nordita)

A NETADIS-NORDITA meeting in honor of John Hertz' 70th birthday

Organizers: Yasser Roudi, Manfred Opper and Peter Sollich

Studying the dynamics of large systems with complex dynamics, e.g. arising
from disorder in the form of competing interactions, is important for a wide
range of problems from spin glasses to machine learning, neuroscience, protein
interaction networks, metabolic networks and systemic risk in finance.

The dynamics in these systems are typically stochastic and time irreversible,
raising many technical and conceptual challenges: How do we describe the
dynamics accurately without full stochastic simulation? How can we solve
"dynamical inverse problems", i.e. infer properties of the underlying network
from observations of the dynamics? How do we extend inference approaches
for complete observations of trajectories to partial (and noisy) observations?

In this workshop we bring together experts in statistical physics approaches
suited to large networks together with experts in interdisciplinary application
areas to address these questions.

Invited Speakers

• Sara A. Solla (Northwestern Univ)
• David Sherrington (Oxford Univ)
• Matteo Marsili (ICTP)
• Giorgio Parisi (La Sapienza)
• Bert Kappen (Radboud Univ Nijmegen)
• Frederico Ricci-Tersenghi (La Sapienza)
• Peter Latham (Univ College London)
• Ramon Grima (Univ of Edinburg)
• Guido Sanguinetti (Univ of Edinburg)
• Ole Winther (Technical Univ of Denmark)
• David Saad (Aston Univ)
• Enzo Marinari (La Sapienza)
• Nicolas Sourlas (École normale supérieure)
• Heinz Horner (Univ of Heidelberg)
• Reimer Kuhn (Kings College London)
• Theo Nieuwenhuizen (Univ of Amsterdam)

Sponsored by:

Wednesday, 11 June

09:00 → 10:00 Registration

10:00 → 11:00 Relaxor ferroelectrics as pseudo-Stoner glasses

In 1979 John Hertz introduced the concept of the Stoner Glass to describe spin glass ordering in itinerant transition metal magnetic alloys. Two decades earlier an interesting behaviour (of distinctly frequency-dependent but sharpish peaks in the dynamical permittivity) was observed in some ceramic ionic alloys, now known as relaxor ferroelectrics. They have received a great deal of attention and practical application, but, even over half a century later, their microscopic their underlying physics has remained ill-understood and controversial. In this talk I shall argue that they are actually displacive analogues of John’s Stoner glass, the peaking corresponding to the onset of the quasi-Stoner spin glass phase. Displacive relaxor ferroelectrics have also been shown to exhibit polar nano-regions persisting for measurable times at temperatures higher than those of the permittivity peaks. I shall argue that this behaviour and both the pseudo-Stoner glass and pseudo-Stoner ferro-order can be understood in terms of a mapping to the problem of Anderson localization, analogous to (but extended beyond) one I devised in 1973 to understand the same experimental systems as inspired John, based on a similar disordered Hubbard model to that he used (but with a complementary methodology). Recent experimental measurements corroborate the argument for PNRs. These observations extrapolate to several further suggestions and questions for both relaxors and itinerant spin glasses and the light of new knowledge could stream again through the window that John opened half a lifetime ago. Reference 1. J.A.Hertz: The Stoner Glass; PRB 19, 4796 (1979) 2. D.Sherrington and K.Mihill: Effects of Clustering on the Magnetic Properties of Transition Metal Alloys: J.Physique Colloque 35, C4-199 (1974) 3. D.Sherrington: BZT: A Soft Pseudospin Glass; PRL 111, 227601 (2013) 4. D.Sherrington: Pb(Mg1/3Nb2/3)O3: A minimal induced-moment soft pseudospin glass perspective; PRB 89, 064105 (2014) 5. M.E.Manley et al.: Phonon localization drives polar nanoregions in a relaxor ferroelectric; Nature Comm. 5:3683 DOI:10.1038/ncomms4683 (online April 2014)

Speaker: David Sherrington (Rudolf Peierls Centre for Theoretical Physics, University of Oxford, UK)

11:00 → 12:00 The puzzle of time-reparametrization-invariance in aging glassy systems

Time-reparametrization-invariance has been discussed in the context of glasses as well as different models of spin glasses. In particular I have investigated this question within the framework of the spherical p-spin-glass-model, since in this model at least the equations of motion can be written down in a closed form. They are closely related to the mode coupling approach for glasses. At first sight time-reparametrization-invariant solutions exist in the aging regime. Investigating, however, short waiting times in the two-time correlation- and response- functions a unique solution is selected.

Speaker: Heinz Horner (Inst. für Theor. Physik, University of Heidelberg)

12:00 → 13:00 Lunch break

13:00 → 14:00 On the phase transition of the random field Ising model

New numerical simulations of the random field Ising model at zero temperature have been performed recently. I will review the results of these simulations which add new hindsight to the nature of the phase transition in this model.

Speaker: Nicolas Sourlas (Ecole Normale Superieure, Paris)

14:00 → 14:30 Coffee break

14:30 → 15:30 Spectra of large random stochastic matrices and relaxation in complex systems

We compute spectra of large random stochastic matrices, i.e. Markov matrices defined on random graphs, where each edge (ij) in a (sparse) random graph is given a positive random weight W_{ij}>0 in such a fashion that the each column sum of the matrix W is normalized to one, \sum_i W_{ij}= 1. We use the replica method to compute spectra in the thermodynamic limit, and the cavity method to obtain results for very large single instances. The stucture of the graphs and the distribution of the non-zeo weights W_{ij} are largely arbitrary, as long as the mean degree remains finite and the column sum constraint are satisfied. Knowing the spectra of stochastic matrices is tantamount to knowing the complete spectrum of relaxation times of stochastic processes described by them, so our results should have many interesting applications for the description of relaxation in complex systems.

Speaker: Reimer Kuhn (Kings College, London)

15:30 → 16:30 The hydrogen ground state in stochastic electrodynamics

Stochastic electrodynamics assumes a classical world in a background of stochastic electromagnetic fields having the zero-point Planck energy $\hbar\omega/2$ per mode. The theory explains various linear problems, such as harmonic oscillators and the Casimir effect. Under certain assumptions, the Heisenberg and Schrodinger equations have been derived. Likewise, entanglement has been demonstrated in this local theory. To test these results on a nonlinear problem, I reconsider the hydrogen ground state. Simulations by Cole and Zou 2003 yielded an encouraging comparison with the quantum result. In this talk I recall an analytical conjecture for the phase space density, presented in Vaxjo 2005 for the relativistic case. After discussing the question of stability, I compare the conjecture with recent simulations carried out in Amsterdam.

Speaker: Theo Nieuwenhuizen (University of Amsterdam, The Netherlands)

16:30 → 19:00 Poster Session

Thursday, 12 June

09:00 → 10:00 Zipf’s law arises naturally from hidden structure

Both natural and artificial systems often exhibit a surprising degree of statistical regularity. One such regularity is Zipf's law. Originally formulated for word frequency, Zipf's law has since been observed in a broad range of phenomena, including city size, firm size, mutual fund size, amino acid sequences, and neural activity. Partly because it is so unexpected, a great deal of effort has gone into explaining it. So far, almost all explanations are either domain specific or require fine-tuning. Here we propose an alternative explanation, which exploits the fact that most real-world datasets can be understood as being generated from a latent variable model. We show that data generated from a such a model exhibits Zipf's law under very mild conditions. We provide the theoretical underpinnings of this result, illustrate it on words and neural data, and point out examples of Zipf's law in the literature for which we can identify a latent variable model.

Speaker: Peter Latham (UCL, UK)

10:00 → 11:00 Generalized mean field approximation for parallel dynamics of the Ising model

The dynamics of non-equilibrium Ising model with parallel updates is investigated using a generalized mean field approximation that incorporates multiple two-site correlations at any two time steps, which can be obtained recursively. The proposed method shows significant improvement in predicting local system properties compared to other mean field approximation techniques, particularly in systems with symmetric interactions. Results are also evaluated against those obtained from Monte Carlo simulations. The method is also employed to obtain parameter values for the kinetic inverse Ising modelling problem, where couplings and local fields values of a fully connected spin system are inferred from data. Hamed Mahmoudi, David Saad, Generalized mean field approximation for parallel dynamics of the Ising model, Jour. Stat. Mech., in press (2014)

Speaker: David Saad (Aston University, Birmingham, UK)

11:00 → 11:15 Coffee break

11:15 → 12:15 Learning from qualitative properties of stochastic dynamical systems

Stochastic processes are widely used mathematical models in disciplines ranging from biology to physics and economics. Consequently, there has been considerable interest in the statistics and machine learning communities in devising approximate Bayesian inference methods for specific classes of stochastic processes. The general scenario considered is that the data consists of noisy observations of the state of the system at discrete time points. While this is clearly an important scenario, I will argue that it is natural to also consider another type of observations which globally characterise trajectories of the system. These "phenotypic" observations are naturally expressed as constraints which must hold for a continuous subset of the observation interval, i.e. they are "continuous time observations". I will consider two approaches for learning in such systems: a general purpose Gaussian Process optimisation method for maximum likelihood parameter estimation, and a message passing approximate inference algorithm for posterior inference for diffusion processes.

Speaker: Guido Sanguinetti (University of Edinburg)

12:15 → 13:45 Lunch break

13:45 → 14:45 Beyond the linear-noise approximation of stochastic biochemical networks

Exact solutions of the chemical master equation are only known for a handful of simple chemical systems. In the past decade, the linear-noise approximation (LNA) has become a popular means to systematically approximate the master equation and to hence obtain insight into the effect of noise on the dynamics of biochemical systems. However a number of assumptions underlying the LNA considerably limit its application to realistic biochemical networks; these are the assumptions that molecule numbers are not too small and that the probability distribution is unimodal. In this talk, I will discuss recent theoretical developments which (i) extend the LNA to multimodal systems, and (ii) correct the LNA estimates of mean concentrations and variances by consideration of higher-order terms in the system-size expansion. The usefulness of these methods to obtaining a more complete picture of stochastic biochemical dynamics will be showcased on various biochemical systems involving gene expression, feedback control, enzyme-mediated catalysis and circadian rhythms.

Speaker: Ramon Grima (University of Edinburg)

14:45 → 15:15 Coffee break

15:15 → 16:15 Optimal Sampling for Optimal Control

Intelligent systems, whether natural or artificial, must act in a world that is highly unpredictable. To plan actions with uncertainty is a stochastic optimal control problem. However, there are two fundamental problems: the optimal control solution is intractable to compute and intractable to represent due the non-trivial state dependence of the optimal control. This has prevented large scale application of stochastic optimal control theory sofar. The path integral control theory describes a class of control problems whose solution can be computed as an inference computation. In this presentation we formalize the intuitive notion that the efficiency of the inference computation is related to the proximity of the sampling control to the optimal control. Secondly, we show new results that allow approximate computation of state dependent optimal controls in terms of basis functions. We illustrate the results on a few examples.

Speaker: Bert Kappen (University of Nijmegen, The Netherlands)

16:15 → 17:15 Constrained Allocation Flux Balance Analysis of Metabolic Networks

New experimental results on bacterial growth inspire a novel top-down approach to study cell metabolism, combining flux balance and proteomics constraints. Flux Balance Analysis has been proven a very useful tool in modeling cellular metabolism, but it is unable to capture features such as the switch to anaerobic metabolism at high growth rates observed in bacteria. We introduce the Constrained Allocation Flux Bal-ance Analysis framework, CAFBA, which is able, among others, to correctly reproduce the anabolic switch

Speaker: Enzo Marinari (Universita di Roma, La Sapienza, Rome)

17:15 → 18:30 Poster Session

18:30 → 21:00 Conference Dinner

Friday, 13 June

10:00 → 11:00 A modified kinetic inverse Ising method for the inference of synaptic spatial structure and characteristic times

A long standing problem in neuroscience, both in modeling and in data analysis, is the one of inferring synaptic couplings from correlations of the sampled neural activities. The recent availability of techniques allowing simultaneous recording from several tens electrodes, in-vitro as well as in-vivo, gave new momentum to research in this direction. In particular, much effort has been devoted to develop and refine inference methods inspired by the statistical mechanics of spin systems (so called ‘inverse Ising’ methods). In the original proposal (Schneidman E, Berry MJ, Segev R and Bialek W, Nature 440, 1007-1012, 2006), simultaneously recorded data are binned in time, discretized and interpreted as successive configurations of a spin system with pairwise interactions at equilibrium. Inference proceeds then as the solution of a constrained optimization problem: determine the spin couplings providing the maximum entropy (Gibbs) distribution compatible with the observed mean activities and pair spatial correlations, used as constraints. ‘Brute-force’ solutions can be obtained by iterative procedures akin to learning algorithms in Boltzmann machines; the need to reduce the computational load for large networks motivated the use of various forms of mean-field estimates of the correlations from the measured mean activities. The interest in relaxing the assumption of equilibrium later led to the development of inference methods based on kinetic Ising models; for a review of the state of the art see Hertz J, Roudi Y, Tyrcha J, in "Principle of Neural Coding" S. Panzeri and R. Q. Quiroga eds, CRC Press 2013. In this work, using simulations of networks of integrate-and-fire neurons, we incorporate in kinetic inverse Ising inference methods the important notion that spikes are transmitted between neurons with delays, which are estimated from the profile of the cross-correlation function prior to the inference procedure, and suggest the right choice of the time bin used in the inference algorithm. A method is also developed to take into account a finite time of integration of the synaptic input. Finally, we analytically and numerically study the relationship between the inferred and the real synaptic efficacies, and how the choice of the time bin affects it. Such relationships turns out to be quadratic both for excitatory and inhibitory synapses, but it depends critically on the time bin for the excitatory synapses only, while being essentially independent of the time bin for the inhibitory ones. Work in collaboration with C. Capone, P. Del Giudice, C. Filosa, G. Gigante.

Speaker: Federico RICCI-TERSENGHI (Universita di Roma, La Sapienza, Roma)

11:00 → 11:15 Coffee break

11:45 → 12:45 A “new” dynamical transition in mean field disordered systems

In this talk I will describe line phase transitions in the dynamics of disordered systems. I will elucidate the relations between the fate of this transition at low temperature and the equilibrium properties of the system. As an example I will discuss standard mean field models, as the p-spin models, and new realistic mean field models as the Mari-Kurchan model for hard spheres.

Speaker: Giorgio Parisi (Universita di Roma, La Sapienza, Rome)

12:45 → 14:00 Lunch break

14:00 → 15:00 Cumulant Corrections for TAP equations

Expectation Propagation (EP) provides a framework for approximate inference closely related to the TAP equations. When the model under consideration is over a latent Gaussian field, with the approximation being Gaussian, we show how these approximations can systematically be corrected. A perturbative expansion is made of the exact but intractable correction, and can be applied to the model's partition function and other moments of interest. The correction is expressed over the higher-order cumulants which are neglected by EP's local matching of moments. Through the expansion, we see that EP is correct to first order. By considering higher orders, corrections of increasing polynomial complexity can be applied to the approximation. The second order provides a correction in quadratic time, which we apply to an array of Gaussian process and Ising models. The corrections generalize to arbitrarily complex approximating families, which we illustrate on tree-structured Ising model approximations. Furthermore, they provide a polynomial-time assessment of the approximation error. We also provide both theoretical and practical insights on the exactness of the EP solution. Reference: M. Opper, U. Paquet and O. Winther, Perturbative Corrections for Approximate Inference in Gaussian Latent Variable Models, JMLR 14(Sep):2857−2898, 2013.

Speaker: Ole Winther (DTU, Copenhagen)

15:00 → 16:00 Why do complex systems look critical?

Speaker: Matteo Marsili (ICTP, Trieste)