Q-DEG: Red-Teaming Vision-Language-Action Models

Overview

Motivation and approach

Vision-Language-Action (VLA) models perform well on robotic pick-and-place tasks when the environment is within or near their training distribution. However, they can fail when the environment is perturbed in ways such as object positions, background, lighting, and camera angle. Different VLAs have different vulnerabilities, so hand-designed benchmarks may miss failure modes that are specific to a model. Red-teaming methods probe a target model for vulnerabilities, but existing approaches have yet to use interpretable simulator parameters as the search space.

By running Quality-Diversity (QD) search on environment parameterizations, Q-DEG generates a collection of diverse adversarial environments targeting a VLA model. Environments in the collection cause the VLA to fail while inducing diverse pre-action layer activations from the VLA. The latter represents the VLA's experience in an environment, so diversifying these activations allows Q-DEG to capture a range of failure modes.

After QD search, Q-DEG patches discovered failure modes by fine-tuning the VLA to imitate its own successful trajectories collected in these environments. This improves its robustness to environmental changes without requiring additional data collection.

Method

Q-DEG pipeline

Generate environments.
CMA-ES proposes environment parameters such as object locations, background, lighting, and camera pose. A MIQP repair step ensures the environments' validity.
Evaluate the VLA.
Target VLA is rolled out multiple times in each proposed environment to estimate the failure rate and the VLA's "experience" encoded through policy embeddings (VLA's pre-action layer activations).
Update the archive.
Each environment is added to the QD archive based on its policy embeddings, and rewarded with its failure rate improvement over predecessors with similar embeddings. CMA-ES learns from these rewards to propose diverse adversarial environments.
Fine-tune through self-imitation.
Target VLA is fine-tuned to imitate its own successful trajectories on archived environments that are feasibly adversarial (on which the VLA neither succeeds nor fails all the time).

Results

Simulation experiments

We evaluate Q-DEG in two stages: first as a red-teaming method for discovering diverse, feasibly adversarial environments based on LIBERO-Spatial tasks, then as a source of self-imitation data for robustness fine-tuning. The evaluation is performed across three target VLA models: π₀-FAST, OpenVLA-OFT, and π_0.5. We compare Q-DEG against two baselines using the same environment parameterization and VLA rollout budget.

Rand(omization) Samples from fixed Gaussian distributions centered on each task's default parameters; it does not adapt or perform QD search.

CMA-ES Adapts the environment parameter distributions toward high VLA failure rates, but disregards embedding diversity and does not perform QD search.

Red-teaming

Environment count and diversity

Q-DEG matches or exceeds baselines in the number of feasibly adversarial environments found, but its environments are more diverse across all three VLAs. We measure diversity by obtaining the DINOv2 embedding for each environment's initial overview, and then computing the average pairwise embedding distance across environments found by each method.

Q-DEG Rand CMA-ES

Environment countNum. of feasibly adversarial environments

Loading chart…

Environment diversityAvg. pairwise DINOv2 embedding distance

Loading chart…

Lines show the mean over three trials. Shaded regions show 95% confidence intervals.

Robustness

VLA success rates on default and adversarial evaluation sets

Across all three target VLAs, the checkpoints fine-tuned on Q-DEG data achieve the highest average success rates across the four evaluation sets. They consistently outperform the corresponding base checkpoints on adversarial evaluation sets and retain high success rates on environments generated by the baseline methods. In contrast, the baseline-fine-tuned checkpoints transfer poorly across evaluation sets, suggesting that the limited diversity of their fine-tuning data restricts generalization.

Rows denote the default LIBERO-Spatial tasks and adversarial evaluation sets generated by each method; columns denote the base VLA and checkpoints fine-tuned on successful trajectories collected during each method's rollouts.

π0-FAST success rates, in percent
Evaluation set	Base	Rand	CMA-ES	Q-DEG
Default	96.80	68.60	82.60	89.60
Rand	29.15	39.40	37.79	48.73
CMA-ES	27.90	38.46	24.81	29.82
Q-DEG	54.83	51.28	36.89	56.13
Average	52.17	49.44	45.52	56.07

OpenVLA-OFT success rates, in percent
Evaluation set	Base	Rand	CMA-ES	Q-DEG
Default	96.20	94.80	86.70	93.80
Rand	56.25	82.29	68.75	78.13
CMA-ES	45.75	52.52	48.27	75.79
Q-DEG	47.19	50.77	55.61	69.13
Average	61.35	70.10	64.83	79.21

π0.5 success rates, in percent
Evaluation set	Base	Rand	CMA-ES	Q-DEG
Default	98.40	85.60	93.60	95.60
Rand	77.38	77.98	83.33	88.09
CMA-ES	54.98	49.39	69.17	58.00
Q-DEG	38.03	61.82	70.45	81.81
Average	67.20	68.70	79.14	80.88

Environment visualizations

Feasibly adversarial environments found by each method

To visualize a target VLA's vulnerabilities, Q-DEG organizes generated adversarial environments into a heatmap that maps the VLA's experience of each environment, represented by policy embeddings, to its failure rate. We create comparable heatmaps for the baseline methods. Intermediate colors indicate feasibly adversarial environments, where the VLA sometimes succeeds and sometimes fails. Across three VLAs, the feasibly adversarial environments found by Q-DEG generally show better coverage over (PCA-reduced) embedding space. Randomization found few such environments, whereas CMA-ES focused on clusters of similar embeddings.

The heatmap positions environments using the first two principal components of the policy embeddings, so nearby cells represent environments that are similar from the VLA's perspective. Hover over a marked cell to compare its behavior before and after robustness fine-tuning.

results

Sim-to-real evaluations

We examine whether the robustness gains observed in simulation transfer to the real world. For two custom tasks, we red-team π_0.5 with Q-DEG and the same baselines, fine-tune separate checkpoints on each method's successful rollouts, and evaluate them on a physical robot over 20 trials per task.

The checkpoint fine-tuned with Q-DEG performs best among the policies evaluated, outperforming the base π_0.5 model and both baselines' checkpoints. These results are preliminary, but provide an initial indication that the robustness gained through Q-DEG in simulation can facilitate sim-to-real transfer.

The overview images show the final gripper position in each trial: green denotes success and red denotes failure.

BibTeX

Coming soon