Algorithms & Architectures

We develop algorithms and architectures that put probabilistic computing to work in generative AI. We start from the computation a model needs and investigate which parts can benefit from fast sampling. This guides the design of p-computers and motivates algorithms that use their capabilities.

Conceptual division of work in correlated diffusion: a neural network estimates the clean state, while a probabilistic sampler generates the next state.
Conceptual division of work in correlated diffusion: a neural network estimates the clean state, while a probabilistic sampler generates the next state.

Generative AI with probabilistic computing

Diffusion models combine neural-network evaluation with stochastic sampling. We investigate how the sampling process itself can carry information about the system being modeled. In our work on correlated diffusion, known interactions guide the stochastic dynamics, while a neural network estimates the underlying clean state. Our initial demonstrations generate configurations of Ising models and spin glasses. This work explores how neural networks and p-computers could share the computational work of generative modeling.

Figure: Singh et al., From Independent to Correlated Diffusion, Fig. 1.

SANTA and SANTA++ for efficient attention

We also develop probabilistic methods that reduce memory traffic during large language model inference. SANTA samples from attention probabilities and combines only the selected value vectors. SANTA++ uses representative keys to select groups of cached tokens without first reading every key. It evaluates attention within the sampled groups and reweights their contributions using their inclusion probabilities. Both methods let us adjust the sampling budget to balance memory access against approximation accuracy. We implement these methods on GPUs and measure their effects on model accuracy and inference speed. These methods also guide our investigation of hardware designed for sparse, additive computation.

Scalable architectures

As p-computers grow, communication between devices can become slower than local updates. We build programmable p-computers that place interaction weights near the circuits that use them. Our recent system connects one million p-bits across 18 FPGAs. We study how communication between devices affects computation, and develop methods that divide large problems into smaller, interacting clusters.

A transformer trained on equilibrium samples proposes configurations within a parallel-tempering search.
A transformer trained on equilibrium samples proposes configurations within a parallel-tempering search.

Optimization and scientific computing

Generative models can also help probabilistic computers explore difficult optimization problems. We study learned proposals that move between distant configurations, alongside parallel tempering and variational optimization methods. For quantum many-body physics, we use p-computers to accelerate sampling in neural quantum states. These applications also help us understand where probabilistic methods are useful and where limitations such as the sign problem remain.

Figure: Delacour et al., Probabilistic algorithms for Ising machines from optimization to generative AI, Fig. 4.

Selected Recent Papers

Foundational Publications