new

Get trending papers in your email inbox!

Subscribe

Daily Papers

byAK and the research community

Nov 11

GPU Based Parallel Ising Computing for Combinatorial Optimization Problems in VLSI Physical Design

In VLSI physical design, many algorithms require the solution of difficult combinatorial optimization problems such as max/min-cut, max-flow problems etc. Due to the vast number of elements typically found in this problem domain, these problems are computationally intractable leading to the use of approximate solutions. In this work, we explore the Ising spin glass model as a solution methodology for hard combinatorial optimization problems using the general purpose GPU (GPGPU). The Ising model is a mathematical model of ferromagnetism in statistical mechanics. Ising computing finds a minimum energy state for the Ising model which essentially corresponds to the expected optimal solution of the original problem. Many combinatorial optimization problems can be mapped into the Ising model. In our work, we focus on the max-cut problem as it is relevant to many VLSI physical design problems. Our method is inspired by the observation that Ising annealing process is very amenable to fine-grain massive parallel GPU computing. We will illustrate how the natural randomness of GPU thread scheduling can be exploited during the annealing process to create random update patterns and allow better GPU resource utilization. Furthermore, the proposed GPU-based Ising computing can handle any general Ising graph with arbitrary connections, which was shown to be difficult for existing FPGA and other hardware based implementation methods. Numerical results show that the proposed GPU Ising max-cut solver can deliver more than 2000X speedup over the CPU version of the algorithm on some large examples, which shows huge performance improvement for addressing many hard optimization algorithms for practical VLSI physical design.

  • 5 authors
·
Mar 13, 2019

Pairing-channel reconstruction by Ising spin-orbit coupling in a multiorbital triangular-lattice model

Ising spin-orbit coupling (SOC) is widely known for locking electronic spins and protecting superconductivity against in-plane paramagnetic pair breaking. Here we show that, in a correlated multiorbital system, Ising SOC can play a qualitatively different role: it reconstructs the hierarchy of spin-fluctuation-mediated pairing channels by reshaping the low-energy Fermi surfaces, orbital-spin coherence factors, and momentum-resolved scattering phase space. We study a noncentrosymmetric two-orbital triangular-lattice model with Rashba SOC, Ising SOC, and onsite SOC. We solve the pairing problem within a multiorbital random-phase-approximation framework. In the single-orbital limit, an E-representation state with a dominant S_z=0 component emerges only at relatively strong Ising SOC. Orbital degrees of freedom facilitate the stabilization of this pairing state at a smaller nominal Ising SOC strength and generate a broad competition between E and A_1 pairing channels. The E state can maintain the same S_z=0 triplet character, whereas the A_1 state becomes a fully gapped sign-changing splusminus superconductor. The latter also realizes a topologically nontrivial phase in the time-reversal-invariant weak-pairing regime, with a nonzero Z_2 index, Wilson-loop spectral flow, and boundary modes. Its topology is not carried by a single orbital pairing component, but emerges cooperatively from the dominant intra- and inter-orbital pairing components. These results establish a microscopic route by which Ising SOC and orbital hybridization can reconstruct interaction-driven superconductivity, producing both Ising-triplet and topological singlet-dominated phases in correlated non-centrosymmetric systems.

  • 5 authors
·
Oct 14

Statistical Landscape Exploration and Weak-Field Defects in Selective Oscillator Ising Machines

Oscillator Ising machines (OIMs) provide an analog dynamical approach to combinatorial optimization, but comparatively little is known about the statistical structure of the ensembles they generate. Using the Sherrington-Kirkpatrick (SK) spin-glass model, we introduce a selective oscillator scheme, OIM+, in which successive annealing epochs are combined with energy-based elimination and repopulation of replicas. We characterize the resulting states through overlap distributions, Hamming distances, hierarchical clustering, and local-field statistics. OIM+ explores structured low-energy regions while retaining a nontrivial ensemble of configurations despite repeated selection. We identify sparse weak-local-field residual defects as a characteristic limitation of the dynamics. A short Metropolis-Hastings refinement preferentially corrects these defects, indicating that the oscillator dynamics performs the dominant collective exploration while the remaining corrections are predominantly local. The weak fields arise mainly from cancellation among competing interaction terms rather than uniformly weak couplings. The same behavior persists for an antiferromagnetically biased SK ensemble, where the hybrid method reaches a pooled mean energy comparable to a substantially longer pure Metropolis anneal. The complete protocol also achieves competitive performance on standard G-set Max-Cut benchmarks. These results suggest a natural division of labor between collective analog search and targeted local refinement.

  • 2 authors
·
Oct 12

Fisher Curvature Scaling at Critical Points: An Exact Information-Geometric Exponent from Periodic Boundary Conditions

We study the scalar curvature of the Fisher information metric on the microscopic coupling-parameter manifold of lattice spin models at criticality. For a d-dimensional lattice with periodic boundary conditions and n = L^d sites, the Fisher manifold has m = d cdot n dimensions (one per bond), and we find |R(J_c)| sim n^{d_R} with d_R = (dν+ 2η)/(dν+ η), where ν and η are the correlation-length and anomalous-dimension critical exponents. For 2D Ising (ν= 1, η= 1/4), this predicts d_R = 10/9, confirmed by exact transfer-matrix computations (L = 6--9: d_R = 1.1115 pm 0.0002) and multi-seed MCMC through L = 24. For 3D Ising (ν= 0.630, η= 0.0363), the prediction d_R = 1.019 is consistent with MCMC on L^3 tori up to L = 10 (power-law fit: d_R = 1.040). For 2D Potts q = 3 (predicted 33/29 approx 1.138), FFT-MCMC through L = 40 shows d_eff oscillating non-monotonically around sim 1.20, consistent with O(1/(ln L)^2) logarithmic corrections. For q = 4 (predicted 22/19), effective exponents oscillate with strong logarithmic corrections. The Ricci decomposition identity R_3 = -R_1/2, R_4 = -R_2/2 holds to 5--6 digits for all models. This exponent is distinct from Ruppeiner thermodynamic curvature and reflects the collective geometry of the growing Fisher manifold. We provide falsification criteria and predictions for additional universality classes.

  • 1 authors
·
Mar 8

LLM Compression by Block Removal with Constrained Binary Optimization

In this paper, we formulate the compression of large language models (LLMs) by optimally deleting transformer blocks (``block removal'') as a constrained binary optimization (CBO) problem that can be mapped to a physical system (Ising glass), whose energies are a strong proxy for downstream model performance. This formulation enables an efficient ranking of a large number of candidate block-removal configurations yielding many high-quality, non-trivial solutions beyond those only removing consecutive regions. Our method performs strongly in the deep compression regime, such as for 50% compression of Llama-3.3-70B-Instruct, where we achieve an almost 23 percentage point increase on the MMLU benchmark compared to other state-of-the-art (SOTA) block-removal methods. For lighter compression, it performs on par with those methods across several benchmarks for Llama-3.1-8B-Instruct, Qwen3-14B (both before and after retraining), as well as Llama-3.3-70B-Instruct. The approach is computationally efficient and requires only forward and backward passes on a calibration dataset for a few active parameters. Additionally, we demonstrate that using good heuristic solvers for the CBO problem provides solutions that perform well on downstream tasks in negligible runtime when it is unfeasible to solve the problem exactly. The method can be readily applied to any architecture. We illustrate this generality on the recent NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 model, which exhibits a highly inhomogeneous and challenging block structure, and where we outperform SOTA for AIME25 and GPQA when removing either 2 attention layers or 3 mixture-of-experts layers.

Onset of a quantum Fisher-Selke sequence in the transverse-field ANNNI model and its bilayer

We study the transverse-field axial next-nearest-neighbor Ising (ANNNI) model on the square lattice and on a bilayer. At the classical multiphase point κ=J_2/J_0=1/2, every configuration of straight domain walls that separates domains of at least two sites is a ground state. The walls are rigid, so perturbation theory in the transverse field g=Γ/J_0 can be carried out for each configuration separately, and we take it to order g^8. Quantum fluctuations select langle 3rangle (uparrowuparrowuparrowdownarrowdownarrowdownarrow) at order g^2, because spins next to a wall gain the most energy from virtual flips. The structures langle 23rangle, langle 223rangle and langle 2223rangle then open between langle 3rangle and the antiphase langle 2rangle at orders g^4, g^6 and g^8, in the order of the Fisher-Selke sequence of the classical three-dimensional model. A second layer suppresses each successive phase more strongly, beyond the stiffening of the local fields. We test the two widest phases with stochastic series expansion quantum Monte Carlo, comparing the free energies of the competing ordered states by thermodynamic integration, since annealing freezes in the modulation that forms at the ordering transition. The simulations reproduce the langle 3rangle wedge to about 1%, and in a pre-registered test langle 23rangle is stable at six of seven points, with the seventh resolved by post-hoc runs.

  • 4 authors
·
Oct 3

Markov length can diverge in systems whose universal physics is spatially Markovian

Conditional mutual information (CMI) probes spatial non-Markovianity and plays a central role in a recently developed framework for defining mixed-state phases based on local reversibility. This framework sharply distinguishes exponentially decaying from algebraically decaying CMI, since the latter obstructs equivalence to a state with finite Markov length. Here we show that states that flow to the same renormalization-group (RG) fixed point can nevertheless exhibit qualitatively different CMI. Starting from a critical Gibbs state of a finite-range local classical Hamiltonian, whose CMI vanishes beyond the interaction range, we show that an RG-irrelevant perturbation that breaks detailed balance generically generates algebraically decaying CMI. Crucially, these power-law tails are cutoff-suppressed: although they lead to an infinite Markov length at every fixed lattice cutoff, they vanish as a positive power of the cutoff in the continuum limit. We construct solvable models that exhibit this phenomenon and predict that the Ising-symmetric critical point of Toom's cellular automata generically exhibits such behavior. We also find analogous cutoff-suppressed CMI in the high-temperature paramagnetic phase of the long-range Ising paramagnet, in contrast with the genuine power-law CMI at the finite-temperature critical point in this same model that survives the continuum limit. We also derive a general result that in one spatial dimension, CMI decays faster than the inverse square of the buffer length if and only if the long-distance physics is Markovian.

  • 2 authors
·
Oct 7

Variational Monte Carlo (VMC) with row-update Projected Entangled-Pair States (PEPS) and its applications in quantum spin glasses

Solving the quantum many-body ground state problem remains a central challenge in computational physics. In this context, the Variational Monte Carlo (VMC) framework based on Projected Entangled Pair States (PEPS) has witnessed rapid development, establishing itself as a vital approach for investigating strongly correlated two-dimensional systems. However, standard PEPS-VMC algorithms predominantly rely on sequential local updates. This conventional approach often suffers from slow convergence and critical slowing down, particularly in the vicinity of phase transitions or within frustrated landscapes. To address these limitations, we propose an efficient autoregressive row-wise sampling algorithm for PEPS that enables direct, rejection-free sampling via single-layer contractions. By utilizing autoregressive single-layer row updates to generate collective, non-local configuration proposals, our method significantly reduces temporal correlations compared to local Metropolis moves. We benchmark the algorithm on the two-dimensional transverse-field Ising model and the quantum spin glass. Our results demonstrate that the row-wise scheme effectively mitigates critical slowing down near the Ising critical point. Furthermore, in the rugged landscape of the quantum spin glass, it yields improved optimization stability and lower ground-state energies. These findings indicate that single-layer autoregressive row updates provide a flexible and robust improvement to local PEPS-VMC sampling and may serve as a basis for more advanced sampling schemes.

  • 5 authors
·
Jan 27

Ferromagnetic ordering in mazelike stripe liquid of a dipolar six-state clock model

We present a comprehensive numerical study of a six-state clock model with a long-range dipolar type interaction. This model is motivated by the ferroelectric orders in the multiferroic hexagonal manganites. At low temperatures, trimerization of local atomic structures leads to six distinct but energetically degenerate structural distortion, which can be modeled by a six-state clock model. Moreover, the atomic displacements in the trimerized state further produce a local electric polarization whose sign depends on whether the clock variable is even or odd. These induced electric dipoles, which can be modeled by emergent Ising degrees of freedom, interact with each other via long-range dipolar interactions. Extensive Monte Carlo simulations are carried out to investigate low temperature phases resulting from the competing interactions. Upon lowering temperature, the system undergoes two Berezinskii-Kosterlitz-Thouless (BKT) transitions, characteristic of the standard six-state clock model in two dimensions. The dipolar interaction between emergent Ising spins induces a first-order transition into a ground state characterized by a three-fold degenerate stripe order. The intermediate phase between the discontinuous and the second BKT transition corresponds to a maze-like hexagonal liquid with short-range stripe ordering. Moreover, this intermediate phase also exhibits an unusual ferromagnetic order with two adjacent clock variables occupying the two types of stripes of the labyrinthine pattern.

  • 3 authors
·
Dec 12, 2024

CADAQUES: A Cost-Aware Dual Architecture for Query-Efficient Autonomous Discovery

Autonomous discovery systems couple a resource that answers queries (a simulator, instrument, or analytic model) to an algorithm that selects what to query next. Most software frameworks for this loop inherit the control structure of numerical optimization: campaigns run for a fixed number of iterations, query costs are absent from the programming interface, and decision-making is treated as free. In practice, queries may differ in cost by orders of magnitude, and planners built on large language models or expensive surrogates consume resources of their own. Here we present CADAQUES, an open-source Python framework built on one architectural principle: cost is a first-class primitive of the discovery loop. CADAQUES separates the loop into two structural protocols, an Oracle that answers queries and a Driver that proposes them, and charges both evaluations and decisions against a common vector-valued budget spanning wall time, CPU hours, monetary cost, and language model tokens. An append-only ledger records, for each transaction, the cost declared before execution and the cost settled afterwards, making their discrepancy an observable property of the campaign. We evaluate the architecture by locating the critical temperature of the two-dimensional Ising model from noisy finite-size estimates against the exact thermodynamic-limit reference. In this noisy setting, strategies that concentrate around the best observed result can be misled by noise-induced peaks, whereas a schedule that explores with cheap low-fidelity queries and refines with higher-fidelity ones yields lower and less variable errors than high fidelity throughout, at the studied budget scale. Metering adds tens of microseconds per iteration, three orders of magnitude below the cheapest oracle query. The framework is MIT-licensed and archived at Zenodo (doi:10.5281/zenodo.21293589).

  • 1 authors
·
Jul 16

Information Theory and Statistical Mechanics Revisited

The statistical mechanics of Gibbs is a juxtaposition of subjective, probabilistic ideas on the one hand and objective, mechanical ideas on the other. In this paper, we follow the path set out by Jaynes, including elements added subsequently to that original work, to explore the consequences of the purely statistical point of view. We show how standard methods in the equilibrium theory could have been derived simply from a description of the available problem information. In addition, our presentation leads to novel insights into questions associated with symmetry and non-equilibrium statistical mechanics. Two surprising consequences to be explored in further work are that (in)distinguishability factors are automatically predicted from the problem formulation and that a quantity related to the thermodynamic entropy production is found by considering information loss in non-equilibrium processes. Using the problem of ion channel thermodynamics as an example, we illustrate the idea of building up complexity by successively adding information to create progressively more complex descriptions of a physical system. Our result is that such statistical mechanical descriptions can be used to create transparent, computable, experimentally-relevant models that may be informed by more detailed atomistic simulations. We also derive a theory for the kinetic behavior of this system, identifying the nonequilibrium `process' free energy functional. The Gibbs relation for this functional is a fluctuation-dissipation theorem applicable arbitrarily far from equilibrium, that captures the effect of non-local and time-dependent behavior from transient driving forces. Based on this work, it is clear that statistical mechanics is a general tool for constructing the relationships between constraints on system information.

  • 3 authors
·
May 27, 2011

Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics

Transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse. We show that weight decay acts as a scalar empirical control parameter for these regimes, and introduce two cheap online diagnostics, mean pairwise attention-head cosine similarity and entropy standard deviation, that track training dynamics from attention activations alone and complement loss-landscape diagnostics at lower compute cost. Across eleven experimental conditions and three model scales (0.82M to 85M parameters), the weight-decay axis separates memorization, developmental grokking, and collapse. A near-transition logistic fit localizes the memorization-to-developmental boundary at λ_c=0.0158 (95% CI [0.0109, 0.0200], N=210); a power-law fit gives an empirical exponent ν=0.757 (CI [0.725, 0.799]). Reference exponents ν=1/2 and 3D Ising νapprox 0.63 lie outside this empirical CI under our four-bin grid, so we report ν as empirical and defer universality-class identification to denser finite-size-scaling work. A horizon-matched multi-task replication (n=280, four modular operations) preserves the weight-decay control pattern; a paired attention-head re-initialization experiment at λ=0.05 changes Phase-2 amplitude (Cohen's d=-1.190, n=10, p_t=4.5 times 10^{-3}), while matched weight-norm clipping does not. Three cross-architecture probes (4L MLP, 4L LSTM, and 4L Mamba; each n=70) replicate the weight-decay-controlled transition with architecture-specific λ_c values. Main diagnostic claims are scoped to modular arithmetic in small transformer attention models; the non-attention experiments are scope probes, and architecture-wide, language-model, and universality-class claims are out of scope.

  • 1 authors
·
May 18

Ergotropy and Capacity Optimization in Heisenberg Spin Chain Quantum Batteries

This study examines the performance of finite spin quantum batteries (QBs) using Heisenberg spin models with Dzyaloshinsky-Moriya (DM) and Kaplan--Shekhtman--Entin-Wohlman--Aharony (KSEA) interactions. The QBs are modeled as interacting quantum spins in local inhomogeneous magnetic fields, inducing variable Zeeman splitting. We derive analytical expressions for the maximal extractable work, ergotropy and the capacity of QBs, as recently examined by Yang et al. [Phys. Rev. Lett. 131, 030402 (2023)]. These quantities are analytically linked through certain quantum correlations, as posited in the aforementioned study. Different Heisenberg spin chain models exhibit distinct behaviors under varying conditions, emphasizing the importance of model selection for optimizing QB performance. In antiferromagnetic (AFM) systems, maximum ergotropy occurs with a Zeeman splitting field applied to either spin, while ferromagnetic (FM) systems benefit from a uniform Zeeman field. Temperature significantly impacts QB performance, with ergotropy in the AFM case being generally more robust against temperature increases compared to the FM case. Incorporating DM and KSEA couplings can significantly enhance the capacity and ergotropy extraction of QBs. However, there exists a threshold beyond which additional increases in these interactions cause a sharp decline in capacity and ergotropy. This behavior is influenced by temperature and quantum coherence, which signal the occurrence of a sudden phase transition. The resource theory of quantum coherence proposed by Baumgratz et al. [Phys. Rev. Lett. 113, 140401 (2014)] plays a crucial role in enhancing ergotropy and capacity. However, ergotropy is limited by both the system's capacity and the amount of coherence. These findings support the theoretical framework of spin-based QBs and may benefit future research on quantum energy storage devices.

  • 8 authors
·
Jul 31, 2024

Replica symmetry breaking in dense neural networks

Understanding the glassy nature of neural networks is pivotal both for theoretical and computational advances in Machine Learning and Theoretical Artificial Intelligence. Keeping the focus on dense associative Hebbian neural networks, the purpose of this paper is two-fold: at first we develop rigorous mathematical approaches to address properly a statistical mechanical picture of the phenomenon of {\em replica symmetry breaking} (RSB) in these networks, then -- deepening results stemmed via these routes -- we aim to inspect the {\em glassiness} that they hide. In particular, regarding the methodology, we provide two techniques: the former is an adaptation of the transport PDE to the case, while the latter is an extension of Guerra's interpolation breakthrough. Beyond coherence among the results, either in replica symmetric and in the one-step replica symmetry breaking level of description, we prove the Gardner's picture and we identify the maximal storage capacity by a ground-state analysis in the Baldi-Venkatesh high-storage regime. In the second part of the paper we investigate the glassy structure of these networks: in contrast with the replica symmetric scenario (RS), RSB actually stabilizes the spin-glass phase. We report huge differences w.r.t. the standard pairwise Hopfield limit: in particular, it is known that it is possible to express the free energy of the Hopfield neural network as a linear combination of the free energies of an hard spin glass (i.e. the Sherrington-Kirkpatrick model) and a soft spin glass (the Gaussian or "spherical" model). This is no longer true when interactions are more than pairwise (whatever the level of description, RS or RSB): for dense networks solely the free energy of the hard spin glass survives, proving a huge diversity in the underlying glassiness of associative neural networks.

  • 4 authors
·
Nov 25, 2021

Competition of Exchange and Correlation Energies in Two-Dimensional N-component Electron Gas Ferromagnetism

Motivated by recent observations of symmtry broken phases in lightly-doped multilayer graphene, we investigate magnetic phase transitions in a generalized electron gas model with four-component electron spin. This model simplifies the problem with a parabolic dispersion band, abstracting away the details of the graphene band structure to focus solely on the effects of the Coulomb interaction. We report four findings: 1) In the Hartree-Fock approximation, we observe that the paramagnetic state undergoes a sequence of density-driven, first-order phase transitions, progressively depopulating electrons from each spin component until achieving complete polarization within a very narrow density window where 1.2<r_s<2 (r_s being the electron gas parameter). 2) Further incorporating the correlation energy via the Bohm-Pines random-phase approximation shows that the cascade of transitions obtained within Hartree-Fock approximation is replaced by a single ferromagnetic phase transition at r_s = 6.12. 3) The disappearance of cascade is due to the correlation energy difference between the four-component paramagnetic state and symmetry-broken phases, which is nearly an order of magnitude more negative than the corresponding Hartree-Fock energy difference for 1.2 < r_s < 2. 4) The transition from the paramagnetic state to the fully polarized state at r_s=6.12 is governed by the balance between exchange and correlation energies, a competition that cannot be captured by mean-field approximations to models featuring effective (density-dependent) delta-function interactions, such as the Stoner model. We use the insights from our model to comment on the phase diagram of multilayer graphene electron gas.

  • 3 authors
·
Jul 1, 2024

Equivariant Neural Networks for Force-Field Models of Lattice Systems

Machine-learning (ML) force fields enable large-scale simulations with near-first-principles accuracy at substantially reduced computational cost. Recent work has extended ML force-field approaches to adiabatic dynamical simulations of condensed-matter lattice models with coupled electronic and structural or magnetic degrees of freedom. However, most existing formulations rely on hand-crafted, symmetry-aware descriptors, whose construction is often system-specific and can hinder generality and transferability across different lattice Hamiltonians. Here we introduce a symmetry-preserving framework based on equivariant neural networks (ENNs) that provides a general, data-driven mapping from local configurations of dynamical variables to the associated on-site forces in a lattice Hamiltonian. In contrast to ENN architectures developed for molecular systems -- where continuous Euclidean symmetries dominate -- our approach aims to embed the discrete point-group and internal symmetries intrinsic to lattice models directly into the neural-network representation of the force field. As a proof of principle, we construct an ENN-based force-field model for the adiabatic dynamics of the Holstein Hamiltonian on a square lattice, a canonical system for electron-lattice physics. The resulting ML-enabled large-scale dynamical simulations faithfully capture mesoscale evolution of the symmetry-breaking phase, illustrating the utility of lattice-equivariant architectures for linking microscopic electronic processes to emergent dynamical behavior in condensed-matter lattice systems.

  • 2 authors
·
Jan 7

CHGNet: Pretrained universal neural network potential for charge-informed atomistic modeling

The simulation of large-scale systems with complex electron interactions remains one of the greatest challenges for the atomistic modeling of materials. Although classical force fields often fail to describe the coupling between electronic states and ionic rearrangements, the more accurate ab-initio molecular dynamics suffers from computational complexity that prevents long-time and large-scale simulations, which are essential to study many technologically relevant phenomena, such as reactions, ion migrations, phase transformations, and degradation. In this work, we present the Crystal Hamiltonian Graph neural Network (CHGNet) as a novel machine-learning interatomic potential (MLIP), using a graph-neural-network-based force field to model a universal potential energy surface. CHGNet is pretrained on the energies, forces, stresses, and magnetic moments from the Materials Project Trajectory Dataset, which consists of over 10 years of density functional theory static and relaxation trajectories of sim 1.5 million inorganic structures. The explicit inclusion of magnetic moments enables CHGNet to learn and accurately represent the orbital occupancy of electrons, enhancing its capability to describe both atomic and electronic degrees of freedom. We demonstrate several applications of CHGNet in solid-state materials, including charge-informed molecular dynamics in Li_xMnO_2, the finite temperature phase diagram for Li_xFePO_4 and Li diffusion in garnet conductors. We critically analyze the significance of including charge information for capturing appropriate chemistry, and we provide new insights into ionic systems with additional electronic degrees of freedom that can not be observed by previous MLIPs.

  • 7 authors
·
Feb 27, 2023

Thermodynamic Isomorphism of Transformers: A Lagrangian Approach to Attention Dynamics

We propose an effective field-theoretic framework for analyzing Transformer attention through a thermodynamic lens. By constructing a Lagrangian on the information manifold equipped with the Fisher metric, we show that, within the Shannon--Boltzmann entropy framework, the Softmax function arises as a stationary solution minimizing a Helmholtz free energy functional. This establishes a formal correspondence between scaled dot-product attention and canonical ensemble statistics. Extending this mapping to macroscopic observables, we define an effective specific heat associated with fluctuations of the attention energy landscape. In controlled experiments on the modular addition task (p = 19--113), we observe a robust peak in this fluctuation measure that consistently precedes the onset of generalization. While no asymptotic power-law divergence is detected in this finite-depth regime, the reproducible enhancement of energy variance suggests a critical-like crossover accompanying representational reorganization. Our framework provides a unified statistical-mechanical perspective on attention scaling, training dynamics, and positional encoding, interpreting the phenomena as emergent properties of an effective thermodynamic system rather than isolated heuristics. Although the present results indicate finite-size crossover behavior rather than a strict phase transition, they motivate further investigation into scaling limits of deep architectures through fluctuation-based observables.

  • 1 authors
·
Feb 12

Simulating 2+1D Lattice Quantum Electrodynamics at Finite Density with Neural Flow Wavefunctions

We present a neural flow wavefunction, Gauge-Fermion FlowNet, and use it to simulate 2+1D lattice compact quantum electrodynamics with finite density dynamical fermions. The gauge field is represented by a neural network which parameterizes a discretized flow-based transformation of the amplitude while the fermionic sign structure is represented by a neural net backflow. This approach directly represents the U(1) degree of freedom without any truncation, obeys Guass's law by construction, samples autoregressively avoiding any equilibration time, and variationally simulates Gauge-Fermion systems with sign problems accurately. In this model, we investigate confinement and string breaking phenomena in different fermion density and hopping regimes. We study the phase transition from the charge crystal phase to the vacuum phase at zero density, and observe the phase seperation and the net charge penetration blocking effect under magnetic interaction at finite density. In addition, we investigate a magnetic phase transition due to the competition effect between the kinetic energy of fermions and the magnetic energy of the gauge field. With our method, we further note potential differences on the order of the phase transitions between a continuous U(1) system and one with finite truncation. Our state-of-the-art neural network approach opens up new possibilities to study different gauge theories coupled to dynamical matter in higher dimensions.

  • 4 authors
·
Dec 14, 2022

The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domains

Scaling has been critical in improving model performance and generalization in machine learning. It involves how a model's performance changes with increases in model size or input data, as well as how efficiently computational resources are utilized to support this growth. Despite successes in other areas, the study of scaling in Neural Network Interatomic Potentials (NNIPs) remains limited. NNIPs act as surrogate models for ab initio quantum mechanical calculations. The dominant paradigm here is to incorporate many physical domain constraints into the model, such as rotational equivariance. We contend that these complex constraints inhibit the scaling ability of NNIPs, and are likely to lead to performance plateaus in the long run. In this work, we take an alternative approach and start by systematically studying NNIP scaling strategies. Our findings indicate that scaling the model through attention mechanisms is efficient and improves model expressivity. These insights motivate us to develop an NNIP architecture designed for scalability: the Efficiently Scaled Attention Interatomic Potential (EScAIP). EScAIP leverages a multi-head self-attention formulation within graph neural networks, applying attention at the neighbor-level representations. Implemented with highly-optimized attention GPU kernels, EScAIP achieves substantial gains in efficiency--at least 10x faster inference, 5x less memory usage--compared to existing NNIPs. EScAIP also achieves state-of-the-art performance on a wide range of datasets including catalysts (OC20 and OC22), molecules (SPICE), and materials (MPTrj). We emphasize that our approach should be thought of as a philosophy rather than a specific model, representing a proof-of-concept for developing general-purpose NNIPs that achieve better expressivity through scaling, and continue to scale efficiently with increased computational resources and training data.

Berkeley UC Berkeley
·
Oct 31, 2024

MatterGen: a generative model for inorganic materials design

The design of functional materials with desired properties is essential in driving technological advances in areas like energy storage, catalysis, and carbon capture. Generative models provide a new paradigm for materials design by directly generating entirely novel materials given desired property constraints. Despite recent progress, current generative models have low success rate in proposing stable crystals, or can only satisfy a very limited set of property constraints. Here, we present MatterGen, a model that generates stable, diverse inorganic materials across the periodic table and can further be fine-tuned to steer the generation towards a broad range of property constraints. To enable this, we introduce a new diffusion-based generative process that produces crystalline structures by gradually refining atom types, coordinates, and the periodic lattice. We further introduce adapter modules to enable fine-tuning towards any given property constraints with a labeled dataset. Compared to prior generative models, structures produced by MatterGen are more than twice as likely to be novel and stable, and more than 15 times closer to the local energy minimum. After fine-tuning, MatterGen successfully generates stable, novel materials with desired chemistry, symmetry, as well as mechanical, electronic and magnetic properties. Finally, we demonstrate multi-property materials design capabilities by proposing structures that have both high magnetic density and a chemical composition with low supply-chain risk. We believe that the quality of generated materials and the breadth of MatterGen's capabilities represent a major advancement towards creating a universal generative model for materials design.

  • 21 authors
·
Dec 6, 2023

A new quantum hydrodynamic description of ferroelectricity in spiral magnets

The strong coupling between magnetism and ferroelectricity was found in rare earth manganites, where the electric polarization could be induced by special magnetic ordering. There is no theoretical model that would allow us to study the static and dynamic properties of electric polarization in strongly correlated magnetic dielectrics. In the presented research, we have taken the main step towards the construction of such a fundamental model, and made a direct connection between the microscopic Katsura-Nagaosa-Balatsky theory and Mostovoy's phenomenological model for magnetically induced polarization. A novel description of the ferroelectricity of spin origin is proposed within the framework of the many-particle quantum hydrodynamics method. It is applied to the study of cells of magnetic ions, where the electric dipole moment is proportional to the vector product of spins. Our approach is based on the many-particle Pauli equation, where the influence of an external magnetic field is considered. We define the electric dipole moment operator of the ion cell and introduce the macroscopic polarization as the quantum mechanical average of that operator. We formulate a model for the description of nonequilibrium polarization and derive a new polarization evolution equation. The polarization switching in ferroelectric magnets with the spiral spin-density-wave state is considered, and we demonstrate that the proposed model yields known results and can predict novel effects. The dynamic magnetoelectric effect can be investigated by employing this novel equation to study the evolution of polarization.

  • 3 authors
·
Nov 6, 2023

Time evolution of the Boltzmann entropy for a nonequilibrium dilute gas

We investigate the time evolution of the Boltzmann entropy of a dilute gas of N particles, N>>1, as it undergoes a free expansion doubling its volume. The microstate of the system, a point in the 4N dimensional phase space, changes in time via Hamiltonian dynamics. Its entropy, at any time t, is given by the logarithm of the phase space volume of all the microstates giving rise to its macrostate at time t. The macrostates that we consider are defined by coarse graining the one-particle phase space into cells Δ_α. The initial and final macrostates of the system are equilibrium states in volumes V and 2V, with the same energy E and particle number N. Their entropy per particle is given, for sufficiently large systems, by the thermodynamic entropy as a function of the particle and energy density, whose leading term is independent of the size of the Δ_α. The intermediate (non-equilibrium) entropy does however depend on the size of the cells Δ_α. Its change with time is due to (i) dispersal in physical space from free motion and to (ii) the collisions between particles which change their velocities. The former depends strongly on the size of the velocity coarse graining Δv: it produces entropy at a rate proportional to Δv. This dependence is investigated numerically and analytically for a dilute two-dimensional gas of hard discs. It becomes significant when the mean free path between collisions is of the same order or larger than the length scale of the initial spatial inhomogeneity. In the opposite limit, the rate of entropy production is essentially independent of Δv and is given by the Boltzmann equation for the limit Δvrightarrow 0. We show that when both processes are active the time dependence of the entropy has a scaling form involving the ratio of the rates of its production by the two processes.

  • 4 authors
·
Mar 12, 2024

Deep learning probability flows and entropy production rates in active matter

Active matter systems, from self-propelled colloids to motile bacteria, are characterized by the conversion of free energy into useful work at the microscopic scale. These systems generically involve physics beyond the reach of equilibrium statistical mechanics, and a persistent challenge has been to understand the nature of their nonequilibrium states. The entropy production rate and the magnitude of the steady-state probability current provide quantitative ways to do so by measuring the breakdown of time-reversal symmetry and the strength of nonequilibrium transport of measure. Yet, their efficient computation has remained elusive, as they depend on the system's unknown and high-dimensional probability density. Here, building upon recent advances in generative modeling, we develop a deep learning framework that estimates the score of this density. We show that the score, together with the microscopic equations of motion, gives direct access to the entropy production rate, the probability current, and their decomposition into local contributions from individual particles, spatial regions, and degrees of freedom. To represent the score, we introduce a novel, spatially-local transformer-based network architecture that learns high-order interactions between particles while respecting their underlying permutation symmetry. We demonstrate the broad utility and scalability of the method by applying it to several high-dimensional systems of interacting active particles undergoing motility-induced phase separation (MIPS). We show that a single instance of our network trained on a system of 4096 particles at one packing fraction can generalize to other regions of the phase diagram, including systems with as many as 32768 particles. We use this observation to quantify the spatial structure of the departure from equilibrium in MIPS as a function of the number of particles and the packing fraction.

  • 2 authors
·
Sep 22, 2023

A Model Zoo on Phase Transitions in Neural Networks

Using the weights of trained Neural Network (NN) models as data modality has recently gained traction as a research field - dubbed Weight Space Learning (WSL). Multiple recent works propose WSL methods to analyze models, evaluate methods, or synthesize weights. Weight space learning methods require populations of trained models as datasets for development and evaluation. However, existing collections of models - called `model zoos' - are unstructured or follow a rudimentary definition of diversity. In parallel, work rooted in statistical physics has identified phases and phase transitions in NN models. Models are homogeneous within the same phase but qualitatively differ from one phase to another. We combine the idea of `model zoos' with phase information to create a controlled notion of diversity in populations. We introduce 12 large-scale zoos that systematically cover known phases and vary over model architecture, size, and datasets. These datasets cover different modalities, such as computer vision, natural language processing, and scientific ML. For every model, we compute loss landscape metrics and validate full coverage of the phases. With this dataset, we provide the community with a resource with a wide range of potential applications for WSL and beyond. Evidence suggests the loss landscape phase plays a role in applications such as model training, analysis, or sparsification. We demonstrate this in an exploratory study of the downstream methods like transfer learning or model weights averaging.

  • 6 authors
·
Apr 25, 2025 2

Predict before you train: Scaling Laws for particle physics foundation models

The largest machine learning models in particle physics are also the most expensive to train, yet the return on scaling a given architecture cannot be estimated before that compute is spent. Scaling laws have been fit for jets, but none has yet been shown to predict the performance of models it was not fit on. We show that, for a generic transformer pretrained on collider jets, it can be forecast. Fitting a joint model-and-data scaling law on small models alone, spanning three orders of magnitude of training compute, we predict the loss of models trained afterward with more than one hundred times more compute to within one percent. We then connect the forecast to downstream physics performance: across two standard tagging benchmarks, lower pretraining loss yields systematically lower fine-tuning loss and higher background rejection after fine-tuning. Within this model family and these tasks, a compute budget can therefore be translated into expected physics performance before any large model is trained. The final frontier model is consistent with the published numbers for current state-of-the-art physics-aware foundation models trained on the same corpus, on accuracy, AUC, and quark/gluon rejection, with a residual edge for the physics-aware model only in the high-purity tail of top tagging. We release five pretrained models spanning multiple sizes, together with the complete training recipe and code.

  • 3 authors
·
Aug 24

Achieving the quantum field theory limit in far-from-equilibrium quantum link models

Realizations of gauge theories in setups of quantum synthetic matter open up the possibility of probing salient exotic phenomena in condensed matter and high-energy physics, along with potential applications in quantum information and science technologies. In light of the impressive ongoing efforts to achieve such realizations, a fundamental question regarding quantum link model regularizations of lattice gauge theories is how faithfully they capture the quantum field theory limit of gauge theories. Recent work [Zache, Van Damme, Halimeh, Hauke, and Banerjee, at https://journals.aps.org/prd/abstract/10.1103/PhysRevD.106.L091502 has shown through analytic derivations, exact diagonalization, and infinite matrix product state calculations that the low-energy physics of 1+1D U(1) quantum link models approaches the quantum field theory limit already at small link spin length S. Here, we show that the approach to this limit also lends itself to the far-from-equilibrium quench dynamics of lattice gauge theories, as demonstrated by our numerical simulations of the Loschmidt return rate and the chiral condensate in infinite matrix product states, which work directly in the thermodynamic limit. Similar to our findings in equilibrium that show a distinct behavior between half-integer and integer link spin lengths, we find that criticality emerging in the Loschmidt return rate is fundamentally different between half-integer and integer spin quantum link models in the regime of strong electric-field coupling. Our results further affirm that state-of-the-art finite-size ultracold-atom and NISQ-device implementations of quantum link lattice gauge theories have the real potential to simulate their quantum field theory limit even in the far-from-equilibrium regime.

  • 5 authors
·
Dec 8, 2021

Divide-and-Conquer Neural Network Surrogates for Quantum Sampling: Accelerating Markov Chain Monte Carlo in Large-Scale Constrained Optimization Problems

Sampling problems are promising candidates for demonstrating quantum advantage, and one approach known as quantum-enhanced Markov chain Monte Carlo [Layden, D. et al., Nature 619, 282-287 (2023)] uses quantum samples as a proposal distribution to accelerate convergence to a target distribution. On the other hand, many practical problems are large-scale and constrained, making it difficult to construct efficient proposal distributions in classical methods and slowing down MCMC mixing. In this work, we propose a divide-and-conquer neural network surrogate framework for quantum sampling to accelerate MCMC under fixed Hamming weight constraints. Our method divides the interaction graph for an Ising problem into subgraphs, generates samples using QAOA for those subproblems with an XY mixer, and trains neural network surrogates conditioned on the Hamming weight to provide proposal distributions for each subset while preserving the constraint. In numerical experiments of Boltzmann sampling on 3-regular graphs, our method consistently accelerated mixing as the system size N increased, with average improvements in the autocorrelation decay rate constant by speedup factors of about 20.3 and 7.6 over classical pair-flip methods based on nearest-neighbor and non-nearest-neighbor exchanges, respectively. We also applied the method to an MNIST feature mask optimization problem with N=784, obtaining faster energy convergence and a 2.03% higher classification accuracy. These results show that our method enables efficient and scalable MCMC and can outperform classical methods for practical applications on NISQ devices.

  • 3 authors
·
Apr 21

Single-shot thermometry of simulated Bose--Einstein condensates using artificial intelligence

Precise determination of thermodynamic parameters in ultracold Bose gases remains challenging due to the destructive nature of conventional measurement techniques and inherent experimental uncertainties. We demonstrate an artificial intelligence approach for rapid, non-destructive estimation of the chemical potential and temperature from single-shot, in situ imaged density profiles of finite-temperature Bose gases. Our convolutional neural network is trained exclusively on quasi-2D `pancake' condensates in harmonic trap configurations. It achieves parameter extraction within fractions of a second. The model also demonstrates zero-shot generalisation across both trap geometry and thermalisation dynamics, successfully estimating thermodynamic parameters for toroidally trapped condensates with errors of only a few nanokelvin despite no prior exposure to such geometries during training, and maintaining predictive accuracy during dynamic thermalisation processes after a relatively brief evolution without explicit training on non-equilibrium states. These results suggest that supervised learning can overcome traditional limitations in ultracold atom thermometry, with extension to broader geometric configurations, temperature ranges, and additional parameters potentially enabling comprehensive real-time analysis of quantum gas experiments. Such capabilities could significantly streamline experimental workflows whilst improving measurement precision across a range of quantum fluid systems.

  • 3 authors
·
Jun 20, 2025

CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers

Large language models (LLMs) have shown remarkable progress in coding and math problem-solving, but evaluation on advanced research-level problems in hard sciences remains scarce. To fill this gap, we present CMT-Benchmark, a dataset of 50 problems covering condensed matter theory (CMT) at the level of an expert researcher. Topics span analytical and computational approaches in quantum many-body, and classical statistical mechanics. The dataset was designed and verified by a panel of expert researchers from around the world. We built the dataset through a collaborative environment that challenges the panel to write and refine problems they would want a research assistant to solve, including Hartree-Fock, exact diagonalization, quantum/variational Monte Carlo, density matrix renormalization group (DMRG), quantum/classical statistical mechanics, and model building. We evaluate LLMs by programmatically checking solutions against expert-supplied ground truth. We developed machine-grading, including symbolic handling of non-commuting operators via normal ordering. They generalize across tasks too. Our evaluations show that frontier models struggle with all of the problems in the dataset, highlighting a gap in the physical reasoning skills of current LLMs. Notably, experts identified strategies for creating increasingly difficult problems by interacting with the LLMs and exploiting common failure modes. The best model, GPT5, solves 30\% of the problems; average across 17 models (GPT, Gemini, Claude, DeepSeek, Llama) is 11.4pm2.1\%. Moreover, 18 problems are solved by none of the 17 models, and 26 by at most one. These unsolved problems span Quantum Monte Carlo, Variational Monte Carlo, and DMRG. Answers sometimes violate fundamental symmetries or have unphysical scaling dimensions. We believe this benchmark will guide development toward capable AI research assistants and tutors.

  • 19 authors
·
Oct 6, 2025

An Introduction to Electrocatalyst Design using Machine Learning for Renewable Energy Storage

Scalable and cost-effective solutions to renewable energy storage are essential to addressing the world's rising energy needs while reducing climate change. As we increase our reliance on renewable energy sources such as wind and solar, which produce intermittent power, storage is needed to transfer power from times of peak generation to peak demand. This may require the storage of power for hours, days, or months. One solution that offers the potential of scaling to nation-sized grids is the conversion of renewable energy to other fuels, such as hydrogen or methane. To be widely adopted, this process requires cost-effective solutions to running electrochemical reactions. An open challenge is finding low-cost electrocatalysts to drive these reactions at high rates. Through the use of quantum mechanical simulations (density functional theory), new catalyst structures can be tested and evaluated. Unfortunately, the high computational cost of these simulations limits the number of structures that may be tested. The use of machine learning may provide a method to efficiently approximate these calculations, leading to new approaches in finding effective electrocatalysts. In this paper, we provide an introduction to the challenges in finding suitable electrocatalysts, how machine learning may be applied to the problem, and the use of the Open Catalyst Project OC20 dataset for model training.

  • 17 authors
·
Oct 14, 2020

Nonlinear Spectral Computing

Many machine learning algorithms rely on the Fourier transform, which is also the inner core of many analog computing devices and, specifically, of optical computing. The square of the modulus of the Fourier spectrum naturally maps onto pairwise interactions, as in the renowned Ising model, making it applicable to combinatorial optimization. Here we show how the nonlinear Fourier transform, related to integrable nonlinear partial differential equations, significantly widens the computational capability and also enables a wide variety of computing gates. We study specifically the case of the nonlinear Schrödinger equation; we introduce an input composed by ``tokens'', i.e., batches of information encoded in a multi-step potential, and we show how the nonlinear spectral computing arising in the wave propagation not only generalizes combinatorial optimization to higher order problems, but also realizes boolean gates. In the weakly nonlinear regime, nonlinearity generates higher-order interactions as perturbative corrections to the pairwise terms. In the solitonic regime, the discrete nonlinear spectrum performs digital computation through soliton generation. By using four input tokens, we demonstrate the programmability of all 256 Boolean functions of three variables, including 3-SAT instances. Our results show that integrable nonlinear propagation provides an unprecedented single physical platform for higher-order and logical computation, opening the way to a rich variety of new algorithms for analog computing, machine learning, and classical and quantum information processing.

  • 3 authors
·
Oct 5