Thao Nguyen, Krishna Kumar Singh, Jing Shi, Trung Bui, Yong Jae Lee, Yuheng Li
Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into
powerful tools with millions of users. However, they remain generic models and
lack personalized knowledge of specific user concepts. Previous work has
explored personalization for text generation, yet it remains unclear how ...
2025-04-29
CV
AI
Thao Nguyen, Krishna Kumar Singh, Jing Shi, Trung Bui, Yong Jae Lee, Yuheng Li
Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into
powerful tools with millions of users. However, they remain generic models and
lack personalized knowledge of specific user concepts. Previous work has
explored personalization for text generation, yet it remains unclear how these
methods can be adapted to new modalities, such as image generation. In this
paper, we introduce Yo'Chameleon, the first attempt to study personalization
for large multimodal models. Given 3-5 images of a particular concept,
Yo'Chameleon leverages soft-prompt tuning to embed subject-specific information
to (i) answer questions about the subject and (ii) recreate pixel-level details
to produce images of the subject in new contexts. Yo'Chameleon is trained with
(i) a self-prompting optimization mechanism to balance performance across
multiple modalities, and (ii) a ``soft-positive" image generation approach to
enhance image quality in a few-shot setting.
Dilip Arumugam, Thomas L. Griffiths
A burgeoning area within reinforcement learning (RL) is the design of
sequential decision-making agents centered around large language models (LLMs).
While autonomous decision-making agents powered by modern LLMs could facilitate
numerous real-world applications, such successes demand agents that ar...
2025-04-29
LG
AI
Dilip Arumugam, Thomas L. Griffiths
A burgeoning area within reinforcement learning (RL) is the design of
sequential decision-making agents centered around large language models (LLMs).
While autonomous decision-making agents powered by modern LLMs could facilitate
numerous real-world applications, such successes demand agents that are capable
of data-efficient RL. One key obstacle to achieving data efficiency in RL is
exploration, a challenge that we demonstrate many recent proposals for LLM
agent designs struggle to contend with. Meanwhile, classic algorithms from the
RL literature known to gracefully address exploration require technical
machinery that can be challenging to operationalize in purely natural language
settings. In this work, rather than relying on finetuning or in-context
learning to coax LLMs into implicitly imitating a RL algorithm, we illustrate
how LLMs can be used to explicitly implement an existing RL algorithm
(Posterior Sampling for Reinforcement Learning) whose capacity for
statistically-efficient exploration is already well-studied. We offer empirical
results demonstrating how our LLM-based implementation of a known,
data-efficient RL algorithm can be considerably more effective in natural
language tasks that demand prudent exploration.
Atul Sharma, Kavindu Herath, Saurabh Bagchi, Chaoyue Liu, Somali Chaterji
We introduce the Hubs and Spokes Learning (HSL) framework, a novel paradigm
for collaborative machine learning that combines the strengths of Federated
Learning (FL) and Decentralized Learning (P2PL). HSL employs a two-tier
communication structure that avoids the single point of failure inherent in ...
2025-04-29
LG
AI
DC
Atul Sharma, Kavindu Herath, Saurabh Bagchi, Chaoyue Liu, Somali Chaterji
We introduce the Hubs and Spokes Learning (HSL) framework, a novel paradigm
for collaborative machine learning that combines the strengths of Federated
Learning (FL) and Decentralized Learning (P2PL). HSL employs a two-tier
communication structure that avoids the single point of failure inherent in FL
and outperforms the state-of-the-art P2PL framework, Epidemic Learning Local
(ELL). At equal communication budgets (total edges), HSL achieves higher
performance than ELL, while at significantly lower communication budgets, it
can match ELL's performance. For instance, with only 400 edges, HSL reaches the
same test accuracy that ELL achieves with 1000 edges for 100 peers (spokes) on
CIFAR-10, demonstrating its suitability for resource-constrained systems. HSL
also achieves stronger consensus among nodes after mixing, resulting in
improved performance with fewer training rounds. We substantiate these claims
through rigorous theoretical analyses and extensive experimental results,
showcasing HSL's practicality for large-scale collaborative learning.
Evan Li, Tushin Mallick, Evan Rose, William Robertson, Alina Oprea, Cristina Nita-Rotaru
LLM-integrated app systems extend the utility of Large Language Models (LLMs)
with third-party apps that are invoked by a system LLM using interleaved
planning and execution phases to answer user queries. These systems introduce
new attack vectors where malicious apps can cause integrity violation o...
2025-04-29
CR
LG
Evan Li, Tushin Mallick, Evan Rose, William Robertson, Alina Oprea, Cristina Nita-Rotaru
LLM-integrated app systems extend the utility of Large Language Models (LLMs)
with third-party apps that are invoked by a system LLM using interleaved
planning and execution phases to answer user queries. These systems introduce
new attack vectors where malicious apps can cause integrity violation of
planning or execution, availability breakdown, or privacy compromise during
execution.
In this work, we identify new attacks impacting the integrity of planning, as
well as the integrity and availability of execution in LLM-integrated apps, and
demonstrate them against IsolateGPT, a recent solution designed to mitigate
attacks from malicious apps. We propose Abstract-Concrete-Execute (ACE), a new
secure architecture for LLM-integrated app systems that provides security
guarantees for system planning and execution. Specifically, ACE decouples
planning into two phases by first creating an abstract execution plan using
only trusted information, and then mapping the abstract plan to a concrete plan
using installed system apps. We verify that the plans generated by our system
satisfy user-specified secure information flow constraints via static analysis
on the structured plan output. During execution, ACE enforces data and
capability barriers between apps, and ensures that the execution is conducted
according to the trusted abstract plan. We show experimentally that our system
is secure against attacks from the INJECAGENT benchmark, a standard benchmark
for control flow integrity in the face of indirect prompt injection attacks,
and our newly introduced attacks. Our architecture represents a significant
advancement towards hardening LLM-based systems containing system facilities of
varying levels of trustworthiness.
Giuseppe De Giacomo, Gianmarco Parretti, Shufang Zhu
We study a variant of LTLf synthesis that synthesizes adaptive strategies for
achieving a multi-tier goal, consisting of multiple increasingly challenging
LTLf objectives in nondeterministic planning domains. Adaptive strategies are
strategies that at any point of their execution (i) enforce the sat...
2025-04-29
AI
Giuseppe De Giacomo, Gianmarco Parretti, Shufang Zhu
We study a variant of LTLf synthesis that synthesizes adaptive strategies for
achieving a multi-tier goal, consisting of multiple increasingly challenging
LTLf objectives in nondeterministic planning domains. Adaptive strategies are
strategies that at any point of their execution (i) enforce the satisfaction of
as many objectives as possible in the multi-tier goal, and (ii) exploit
possible cooperation from the environment to satisfy as many as possible of the
remaining ones. This happens dynamically: if the environment cooperates (ii)
and an objective becomes enforceable (i), then our strategies will enforce it.
We provide a game-theoretic technique to compute adaptive strategies that is
sound and complete. Notably, our technique is polynomial, in fact quadratic, in
the number of objectives. In other words, it handles multi-tier goals with only
a minor overhead compared to standard LTLf synthesis.
Tyler Chen, Archan Ray, Akshay Seshadri, Dylan Herman, Bao Bach, Pranav Deshpande, Abhishek Som, Niraj Kumar, Marco Pistoia
The $k$-means algorithm (Lloyd's algorithm) is a widely used method for
clustering unlabeled data. A key bottleneck of the $k$-means algorithm is that
each iteration requires time linear in the number of data points, which can be
expensive in big data applications. This was improved in recent works ...
2025-04-29
quant-ph
DS
LG
Tyler Chen, Archan Ray, Akshay Seshadri, Dylan Herman, Bao Bach, Pranav Deshpande, Abhishek Som, Niraj Kumar, Marco Pistoia
The $k$-means algorithm (Lloyd's algorithm) is a widely used method for
clustering unlabeled data. A key bottleneck of the $k$-means algorithm is that
each iteration requires time linear in the number of data points, which can be
expensive in big data applications. This was improved in recent works proposing
quantum and quantum-inspired classical algorithms to approximate the $k$-means
algorithm locally, in time depending only logarithmically on the number of data
points (along with data dependent parameters) [$q$-means: A quantum algorithm
for unsupervised machine learning; Kerenidis, Landman, Luongo, and Prakash,
NeurIPS 2019; Do you know what $q$-means?, Doriguello, Luongo, Tang]. In this
work, we describe a simple randomized mini-batch $k$-means algorithm and a
quantum algorithm inspired by the classical algorithm. We prove worse-case
guarantees that significantly improve upon the bounds for previous algorithms.
Our improvements are due to a careful use of uniform sampling, which preserves
certain symmetries of the $k$-means problem that are not preserved in previous
algorithms that use data norm-based sampling.
Neil F. Johnson, Frank Yingjie Huo
Trust in AI is undermined by the fact that there is no science that predicts
-- or that can explain to the public -- when an LLM's output (e.g. ChatGPT) is
likely to tip mid-response to become wrong, misleading, irrelevant or
dangerous. With deaths and trauma already being blamed on LLMs, this
uncer...
2025-04-29
AI
CY
AO
comp-ph
soc-ph
Neil F. Johnson, Frank Yingjie Huo
Trust in AI is undermined by the fact that there is no science that predicts
-- or that can explain to the public -- when an LLM's output (e.g. ChatGPT) is
likely to tip mid-response to become wrong, misleading, irrelevant or
dangerous. With deaths and trauma already being blamed on LLMs, this
uncertainty is even pushing people to treat their 'pet' LLM more politely to
'dissuade' it (or its future Artificial General Intelligence offspring) from
suddenly turning on them. Here we address this acute need by deriving from
first principles an exact formula for when a Jekyll-and-Hyde tipping point
occurs at LLMs' most basic level. Requiring only secondary school mathematics,
it shows the cause to be the AI's attention spreading so thin it suddenly
snaps. This exact formula provides quantitative predictions for how the
tipping-point can be delayed or prevented by changing the prompt and the AI's
training. Tailored generalizations will provide policymakers and the public
with a firm platform for discussing any of AI's broader uses and risks, e.g. as
a personal counselor, medical advisor, decision-maker for when to use force in
a conflict situation. It also meets the need for clear and transparent answers
to questions like ''should I be polite to my LLM?''
Elias Nyholm, Oscar Carlsson, Maurice Weiler, Daniel Persson
This paper presents a novel framework for non-linear equivariant neural
network layers on homogeneous spaces. The seminal work of Cohen et al. on
equivariant $G$-CNNs on homogeneous spaces characterized the representation
theory of such layers in the linear setting, finding that they are given by
co...
2025-04-29
LG
RT
ML
Elias Nyholm, Oscar Carlsson, Maurice Weiler, Daniel Persson
This paper presents a novel framework for non-linear equivariant neural
network layers on homogeneous spaces. The seminal work of Cohen et al. on
equivariant $G$-CNNs on homogeneous spaces characterized the representation
theory of such layers in the linear setting, finding that they are given by
convolutions with kernels satisfying so-called steerability constraints.
Motivated by the empirical success of non-linear layers, such as self-attention
or input dependent kernels, we set out to generalize these insights to the
non-linear setting. We derive generalized steerability constraints that any
such layer needs to satisfy and prove the universality of our construction. The
insights gained into the symmetry-constrained functional dependence of
equivariant operators on feature maps and group elements informs the design of
future equivariant neural network layers. We demonstrate how several common
equivariant network architectures - $G$-CNNs, implicit steerable kernel
networks, conventional and relative position embedded attention based
transformers, and LieTransformers - may be derived from our framework.
Mete Erdogan, Sebnem Demirtas
Accurate and early diagnosis of pneumonia through X-ray imaging is essential
for effective treatment and improved patient outcomes. Recent advancements in
machine learning have enabled automated diagnostic tools that assist
radiologists in making more reliable and efficient decisions. In this work, ...
2025-04-29
CV
AI
LG
Mete Erdogan, Sebnem Demirtas
Accurate and early diagnosis of pneumonia through X-ray imaging is essential
for effective treatment and improved patient outcomes. Recent advancements in
machine learning have enabled automated diagnostic tools that assist
radiologists in making more reliable and efficient decisions. In this work, we
propose a Singular Value Decomposition-based Least Squares (SVD-LS) framework
for multi-class pneumonia classification, leveraging powerful feature
representations from state-of-the-art self-supervised and transfer learning
models. Rather than relying on computationally expensive gradient based
fine-tuning, we employ a closed-form, non-iterative classification approach
that ensures efficiency without compromising accuracy. Experimental results
demonstrate that SVD-LS achieves competitive performance while offering
significantly reduced computational costs, making it a viable alternative for
real-time medical imaging applications.
Yiting Zhang, Shichen Li, Elena Shrestha
Mechanical search (MS) in cluttered environments remains a significant
challenge for autonomous manipulators, requiring long-horizon planning and
robust state estimation under occlusions and partial observability. In this
work, we introduce XPG-RL, a reinforcement learning framework that enables
age...
2025-04-29
RO
LG
Yiting Zhang, Shichen Li, Elena Shrestha
Mechanical search (MS) in cluttered environments remains a significant
challenge for autonomous manipulators, requiring long-horizon planning and
robust state estimation under occlusions and partial observability. In this
work, we introduce XPG-RL, a reinforcement learning framework that enables
agents to efficiently perform MS tasks through explainable, priority-guided
decision-making based on raw sensory inputs. XPG-RL integrates a task-driven
action prioritization mechanism with a learned context-aware switching strategy
that dynamically selects from a discrete set of action primitives such as
target grasping, occlusion removal, and viewpoint adjustment. Within this
strategy, a policy is optimized to output adaptive threshold values that govern
the discrete selection among action primitives. The perception module fuses
RGB-D inputs with semantic and geometric features to produce a structured scene
representation for downstream decision-making. Extensive experiments in both
simulation and real-world settings demonstrate that XPG-RL consistently
outperforms baseline methods in task success rates and motion efficiency,
achieving up to 4.5$\times$ higher efficiency in long-horizon tasks. These
results underscore the benefits of integrating domain knowledge with learnable
decision-making policies for robust and efficient robotic manipulation.
Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji
We introduce softpick, a rectified, not sum-to-one, drop-in replacement for
softmax in transformer attention mechanisms that eliminates attention sink and
massive activations. Our experiments with 340M parameter models demonstrate
that softpick maintains performance parity with softmax on standard b...
2025-04-29
LG
Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji
We introduce softpick, a rectified, not sum-to-one, drop-in replacement for
softmax in transformer attention mechanisms that eliminates attention sink and
massive activations. Our experiments with 340M parameter models demonstrate
that softpick maintains performance parity with softmax on standard benchmarks
while achieving 0% sink rate. The softpick transformer produces hidden states
with significantly lower kurtosis (340 vs 33,510) and creates sparse attention
maps (46.97% sparsity). Models using softpick consistently outperform softmax
when quantized, with particularly pronounced advantages at lower bit
precisions. Our analysis and discussion shows how softpick has the potential to
open new possibilities for quantization, low-precision training, sparsity
optimization, pruning, and interpretability. Our code is available at
https://github.com/zaydzuhri/softpick-attention.
Zikui Cai, Shayan Shabihi, Bang An, Zora Che, Brian R. Bartoldson, Bhavya Kailkhura, Tom Goldstein, Furong Huang
We introduce AegisLLM, a cooperative multi-agent defense against adversarial
attacks and information leakage. In AegisLLM, a structured workflow of
autonomous agents - orchestrator, deflector, responder, and evaluator -
collaborate to ensure safe and compliant LLM outputs, while self-improving over
...
2025-04-29
LG
Zikui Cai, Shayan Shabihi, Bang An, Zora Che, Brian R. Bartoldson, Bhavya Kailkhura, Tom Goldstein, Furong Huang
We introduce AegisLLM, a cooperative multi-agent defense against adversarial
attacks and information leakage. In AegisLLM, a structured workflow of
autonomous agents - orchestrator, deflector, responder, and evaluator -
collaborate to ensure safe and compliant LLM outputs, while self-improving over
time through prompt optimization. We show that scaling agentic reasoning system
at test-time - both by incorporating additional agent roles and by leveraging
automated prompt optimization (such as DSPy)- substantially enhances robustness
without compromising model utility. This test-time defense enables real-time
adaptability to evolving attacks, without requiring model retraining.
Comprehensive evaluations across key threat scenarios, including unlearning and
jailbreaking, demonstrate the effectiveness of AegisLLM. On the WMDP unlearning
benchmark, AegisLLM achieves near-perfect unlearning with only 20 training
examples and fewer than 300 LM calls. For jailbreaking benchmarks, we achieve
51% improvement compared to the base model on StrongReject, with false refusal
rates of only 7.9% on PHTest compared to 18-55% for comparable methods. Our
results highlight the advantages of adaptive, agentic reasoning over static
defenses, establishing AegisLLM as a strong runtime alternative to traditional
approaches based on model modifications. Code is available at
https://github.com/zikuicai/aegisllm
Shangyu Li, Juyong Jiang, Tiancheng Zhao, Jiasi Shen
We introduce OSVBench, a new benchmark for evaluating Large Language Models
(LLMs) in generating complete specification code pertaining to operating system
kernel verification tasks. The benchmark first defines the specification
generation problem into a program synthesis problem within a confined s...
2025-04-29
CL
AI
OS
PL
SE
Shangyu Li, Juyong Jiang, Tiancheng Zhao, Jiasi Shen
We introduce OSVBench, a new benchmark for evaluating Large Language Models
(LLMs) in generating complete specification code pertaining to operating system
kernel verification tasks. The benchmark first defines the specification
generation problem into a program synthesis problem within a confined scope of
syntax and semantics by providing LLMs with the programming model. The LLMs are
required to understand the provided verification assumption and the potential
syntax and semantics space to search for, then generate the complete
specification for the potentially buggy operating system code implementation
under the guidance of the high-level functional description of the operating
system. This benchmark is built upon a real-world operating system kernel,
Hyperkernel, and consists of 245 complex specification generation tasks in
total, each is a long context task of about 20k-30k tokens. Our comprehensive
evaluation of 12 LLMs exhibits the limited performance of the current LLMs on
the specification generation tasks for operating system verification.
Significant disparities in their performance on the benchmark highlight
differences in their ability to handle long-context code generation tasks. The
evaluation toolkit and benchmark are available at
https://github.com/lishangyu-hkust/OSVBench.
Tyler McDonald, Ali Emami
As Large Language Models (LLMs) continue to be leveraged for daily tasks,
prompt engineering remains an active field of contribution within computational
linguistics, particularly in domains requiring specialized knowledge such as
arithmetic reasoning. While these LLMs are optimized for a variety of...
2025-04-29
CL
AI
Tyler McDonald, Ali Emami
As Large Language Models (LLMs) continue to be leveraged for daily tasks,
prompt engineering remains an active field of contribution within computational
linguistics, particularly in domains requiring specialized knowledge such as
arithmetic reasoning. While these LLMs are optimized for a variety of tasks,
their exhaustive employment may become computationally or financially
cumbersome for small teams. Additionally, complete reliance on proprietary,
closed-source models often limits customization and adaptability, posing
significant challenges in research and application scalability. Instead, by
leveraging open-source models at or below 7 billion parameters, we can optimize
our resource usage while still observing remarkable gains over standard
prompting approaches. To cultivate this notion, we introduce Trace-of-Thought
Prompting, a simple, zero-shot prompt engineering method that instructs LLMs to
create observable subproblems using critical problem-solving, specifically
designed to enhance arithmetic reasoning capabilities. When applied to
open-source models in tandem with GPT-4, we observe that Trace-of-Thought not
only allows novel insight into the problem-solving process but also introduces
performance gains as large as 125% on language models at or below 7 billion
parameters. This approach underscores the potential of open-source initiatives
in democratizing AI research and improving the accessibility of high-quality
computational linguistics applications.
Kleanthis Avramidis, Woojae Jeong, Aditya Kommineni, Sudarsana R. Kadiri, Marcus Ma, Colin McDaniel, Myzelle Hughes, Thomas McGee, Elsi Kaiser, Dani Byrd, Assal Habibi, B. Rael Cahn, Idan A. Blank, Kristina Lerman, Takfarinas Medani, Richard M. Leahy, Shrikanth Narayanan
Identifying physiological and behavioral markers for mental health conditions
is a longstanding challenge in psychiatry. Depression and suicidal ideation, in
particular, lack objective biomarkers, with screening and diagnosis primarily
relying on self-reports and clinical interviews. Here, we invest...
2025-04-29
LG
SP
Kleanthis Avramidis, Woojae Jeong, Aditya Kommineni, Sudarsana R. Kadiri, Marcus Ma, Colin McDaniel, Myzelle Hughes, Thomas McGee, Elsi Kaiser, Dani Byrd, Assal Habibi, B. Rael Cahn, Idan A. Blank, Kristina Lerman, Takfarinas Medani, Richard M. Leahy, Shrikanth Narayanan
Identifying physiological and behavioral markers for mental health conditions
is a longstanding challenge in psychiatry. Depression and suicidal ideation, in
particular, lack objective biomarkers, with screening and diagnosis primarily
relying on self-reports and clinical interviews. Here, we investigate eye
tracking as a potential marker modality for screening purposes. Eye movements
are directly modulated by neuronal networks and have been associated with
attentional and mood-related patterns; however, their predictive value for
depression and suicidality remains unclear. We recorded eye-tracking sequences
from 126 young adults as they read and responded to affective sentences, and
subsequently developed a deep learning framework to predict their clinical
status. The proposed model included separate branches for trials of positive
and negative sentiment, and used 2D time-series representations to account for
both intra-trial and inter-trial variations. We were able to identify
depression and suicidal ideation with an area under the receiver operating
curve (AUC) of 0.793 (95% CI: 0.765-0.819) against healthy controls, and
suicidality specifically with 0.826 AUC (95% CI: 0.797-0.852). The model also
exhibited moderate, yet significant, accuracy in differentiating depressed from
suicidal participants, with 0.609 AUC (95% CI 0.571-0.646). Discriminative
patterns emerge more strongly when assessing the data relative to response
generation than relative to the onset time of the final word of the sentences.
The most pronounced effects were observed for negative-sentiment sentences,
that are congruent to depressed and suicidal participants. Our findings
highlight eye tracking as an objective tool for mental health assessment and
underscore the modulatory impact of emotional stimuli on cognitive processes
affecting oculomotor control.
Christopher Watson, Rajeev Alur, Divya Gopinath, Ravi Mangal, Corina S. Pasareanu
Recent advances in deep learning have enabled the development of autonomous
systems that use deep neural networks for perception. Formal verification of
these systems is challenging due to the size and complexity of the perception
DNNs as well as hard-to-quantify, changing environment conditions. To...
2025-04-29
LG
RO
Christopher Watson, Rajeev Alur, Divya Gopinath, Ravi Mangal, Corina S. Pasareanu
Recent advances in deep learning have enabled the development of autonomous
systems that use deep neural networks for perception. Formal verification of
these systems is challenging due to the size and complexity of the perception
DNNs as well as hard-to-quantify, changing environment conditions. To address
these challenges, we propose a probabilistic verification framework for
autonomous systems based on the following key concepts: (1) Scenario-based
Modeling: We decompose the task (e.g., car navigation) into a composition of
scenarios, each representing a different environment condition. (2)
Probabilistic Abstractions: For each scenario, we build a compact abstraction
of perception based on the DNN's performance on an offline dataset that
represents the scenario's environment condition. (3) Symbolic Reasoning and
Acceleration: The abstractions enable efficient compositional verification of
the autonomous system via symbolic reasoning and a novel acceleration proof
rule that bounds the error probability of the system under arbitrary variations
of environment conditions. We illustrate our approach on two case studies: an
experimental autonomous system that guides airplanes on taxiways using
high-dimensional perception DNNs and a simulation model of an F1Tenth
autonomous car using LiDAR observations.
Maximilian Stupp, P. S. Koutsourelakis
Coarse-grained (CG) models offer an effective route to reducing the
complexity of molecular simulations, yet conventional approaches depend heavily
on long all-atom molecular dynamics (MD) trajectories to adequately sample
configurational space. This data-driven dependence limits their accuracy and
...
2025-04-29
chem-ph
LG
comp-ph
Maximilian Stupp, P. S. Koutsourelakis
Coarse-grained (CG) models offer an effective route to reducing the
complexity of molecular simulations, yet conventional approaches depend heavily
on long all-atom molecular dynamics (MD) trajectories to adequately sample
configurational space. This data-driven dependence limits their accuracy and
generalizability, as unvisited configurations remain excluded from the
resulting CG model. We introduce a data-free generative framework for
coarse-graining that directly targets the all-atom Boltzmann distribution. Our
model defines a structured latent space comprising slow collective variables,
which are statistically associated with multimodal marginal densities capturing
metastable states, and fast variables, which represent the remaining degrees of
freedom with simple, unimodal conditional distributions. A potentially
learnable, bijective map from the full latent space to the all-atom
configuration space enables automatic and accurate reconstruction of molecular
structures. The model is trained using an energy-based objective that minimizes
the reverse Kullback-Leibler divergence, relying solely on the interatomic
potential rather than sampled trajectories. A tempering scheme is used to
stabilize training and promote exploration of diverse configurations. Once
trained, the model can generate unbiased, one-shot equilibrium all-atom
samples. We validate the method on two synthetic systems-a double-well
potential and a Gaussian mixture-as well as on the benchmark alanine dipeptide.
The model captures all relevant modes of the Boltzmann distribution, accurately
reconstructs atomic configurations, and learns physically meaningful
coarse-grained representations, all without any simulation data.
Zhengfu He, Junxuan Wang, Rui Lin, Xuyang Ge, Wentao Shu, Qiong Tang, Junping Zhang, Xipeng Qiu
We propose Low-Rank Sparse Attention (Lorsa), a sparse replacement model of
Transformer attention layers to disentangle original Multi Head Self Attention
(MHSA) into individually comprehensible components. Lorsa is designed to
address the challenge of attention superposition to understand
attention...
2025-04-29
LG
CL
Zhengfu He, Junxuan Wang, Rui Lin, Xuyang Ge, Wentao Shu, Qiong Tang, Junping Zhang, Xipeng Qiu
We propose Low-Rank Sparse Attention (Lorsa), a sparse replacement model of
Transformer attention layers to disentangle original Multi Head Self Attention
(MHSA) into individually comprehensible components. Lorsa is designed to
address the challenge of attention superposition to understand
attention-mediated interaction between features in different token positions.
We show that Lorsa heads find cleaner and finer-grained versions of previously
discovered MHSA behaviors like induction heads, successor heads and attention
sink behavior (i.e., heavily attending to the first token). Lorsa and Sparse
Autoencoder (SAE) are both sparse dictionary learning methods applied to
different Transformer components, and lead to consistent findings in many ways.
For instance, we discover a comprehensive family of arithmetic-specific Lorsa
heads, each corresponding to an atomic operation in Llama-3.1-8B. Automated
interpretability analysis indicates that Lorsa achieves parity with SAE in
interpretability while Lorsa exhibits superior circuit discovery properties,
especially for features computed collectively by multiple MHSA heads. We also
conduct extensive experiments on architectural design ablation, Lorsa scaling
law and error analysis.
Taisuke Kobayashi
Continual learning is the one of the most essential abilities for autonomous
agents, which can incrementally learn daily-life skills. For this ultimate
goal, a simple but powerful method, dark experience replay (DER), has been
proposed recently. DER mitigates catastrophic forgetting, in which the sk...
2025-04-29
LG
Taisuke Kobayashi
Continual learning is the one of the most essential abilities for autonomous
agents, which can incrementally learn daily-life skills. For this ultimate
goal, a simple but powerful method, dark experience replay (DER), has been
proposed recently. DER mitigates catastrophic forgetting, in which the skills
acquired in the past are unintentionally forgotten, by stochastically storing
the streaming data in a reservoir sampling (RS) buffer and by relearning them
or retaining the past outputs for them. However, since DER considers multiple
objectives, it will not function properly without appropriate weighting of
them. In addition, the ability to retain past outputs inhibits learning if the
past outputs are incorrect due to distribution shift or other effects. This is
due to a tradeoff between memory consolidation and plasticity. The tradeoff is
hidden even in the RS buffer, which gradually stops storing new data for new
skills in it as data is continuously passed to it. To alleviate the tradeoff
and achieve better balance, this paper proposes improvement strategies to each
of DER and RS. Specifically, DER is improved with automatic adaptation of
weights, block of replaying erroneous data, and correction of past outputs. RS
is also improved with generalization of acceptance probability, stratification
of plural buffers, and intentional omission of unnecessary data. These
improvements are verified through multiple benchmarks including regression,
classification, and reinforcement learning problems. As a result, the proposed
methods achieve steady improvements in learning performance by balancing the
memory consolidation and plasticity.
Ziqing Fan, Cheng Liang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, Weidi Xie
Recent advances in reasoning-enhanced large language models (LLMs) and
multimodal LLMs (MLLMs) have significantly improved performance in complex
tasks, yet medical AI models often overlook the structured reasoning processes
inherent in clinical practice. In this work, we present ChestX-Reasoner, a
...
2025-04-29
AI
CL
CV
Ziqing Fan, Cheng Liang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, Weidi Xie
Recent advances in reasoning-enhanced large language models (LLMs) and
multimodal LLMs (MLLMs) have significantly improved performance in complex
tasks, yet medical AI models often overlook the structured reasoning processes
inherent in clinical practice. In this work, we present ChestX-Reasoner, a
radiology diagnosis MLLM designed to leverage process supervision mined
directly from clinical reports, reflecting the step-by-step reasoning followed
by radiologists. We construct a large dataset by extracting and refining
reasoning chains from routine radiology reports. Our two-stage training
framework combines supervised fine-tuning and reinforcement learning guided by
process rewards to better align model reasoning with clinical standards. We
introduce RadRBench-CXR, a comprehensive benchmark featuring 59K visual
question answering samples with 301K clinically validated reasoning steps, and
propose RadRScore, a metric evaluating reasoning factuality, completeness, and
effectiveness. ChestX-Reasoner outperforms existing medical and general-domain
MLLMs in both diagnostic accuracy and reasoning ability, achieving 16%, 5.9%,
and 18% improvements in reasoning ability compared to the best medical MLLM,
the best general MLLM, and its base model, respectively, as well as 3.3%, 24%,
and 27% improvements in outcome accuracy. All resources are open-sourced to
facilitate further research in medical reasoning MLLMs.