arXiv subjects

Siddharth Aphale

Publications and source records attributed to Siddharth Aphale.

4 recordsLinked to original sources

arXiv2026

Rethinking Self-Repair in Code-as-Policy Agents: What Drives Retry Success?

Code-as-policy agents often use execution feedback to retry failed programs. Additional sampling and task guidance can also improve success, leaving the contribution of failure-specific feedback unclear. We study this question in simulated robot manipulation. A fixed lookup selects feedback templates using failure labels and the API variant. A frozen code model generates a fresh program after each scene reset. On the same 252 initial failures, Qwen2.5-Coder-7B recovers 85.3% within five retries, compared with 68.3% for resampling with the same retry cap. The advantage persists under fixed total-token budgets. Feedback from other episodes and fixed task guidance yield similar recovery point estimates, with no detected benefit from matching the coarse failure label. In a separate study, a fixed guided prompt used from the first attempt achieves an observed overall success rate similar to that of template-based feedback. Template-based feedback also lowers recovery on a nut-insertion task requiring a rim grasp, consistent with an incompatible center-grasp instruction. Evaluations of robot self-repair should include resampling, mismatched feedback, and concise, task-compatible static guidance as baselines.

cs.RO↗

arXiv2026

Goal Retrieval versus Action-Value Ordering: Contrastive Critics under Best-of-$K$ Selection

Goal-conditioned action generators can produce many plausible actions. Best-of-$K$ selection then uses a critic to rank those actions for the current state and goal. Standard contrastive critics are trained and evaluated on a different ranking: distinguishing future goals from unrelated negatives for a fixed state--action pair. The unnormalized contrastive score is monotone in behavior-policy value at the population optimum. Cosine-normalized scores lack this guarantee, and finite trained networks may fail to recover the correct ordering. We audit the resulting gap using shared state--goal queries and frozen candidate pools, so critics differ only in their action ordering. Raw and cosine critics reach retrieval AUCs of $0.96$--$1.00$ across eight OGBench tasks. Their progress ordering on navigation tasks is much weaker and sometimes inverted. In the exact-value controlled problem, all three contrastive selectors have higher mean regret than random selection. Their regret can increase with $K$. TD-Q stays close to the pool oracle. PointMaze is the only task with a reliable learned value reference. TD-Q gives substantially better fixed-query ordering there, though the differences in executed return are smaller and vary by task. Matched controls suggest that graded value supervision accounts for most of the improvement. Changing score bounds or model capacity alone has much less effect. High retrieval performance does not establish whether a critic ranks actions correctly for best-of-$K$ selection. That requires a separate action-ordering evaluation before deployment.

cs.LG↗

arXiv2026

Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learning

Sparse-reward reinforcement learning often fails because rollouts from the unassisted evaluation start rarely reach later task stages. Reverse curricula address this by starting some training rollouts from easier intermediate states, called scaffolds. The main choices are which intermediate starts to use (scaffold access) and how to distribute training across them (scaffold allocation). Most prior curricula pace removal on one shared schedule, which can fail when task instances, or contexts, learn at different rates. We introduce an online, learner-agnostic reset controller that gives every context its own curriculum. We use binary rollout success to decide whether to move a context's start forward or back. A stalled frontier also gets an occasional harder-start trial. A counting construction shows that synchronized global pacing can be insufficient when contexts need conflicting amounts of assisted practice. Across six navigation and manipulation settings, scaffold access improves learning and enables success in three where unassisted training fails within the reported budget. None of the global schedules we tested solved both groups in the constructed conflict. The per-context controller did. Group-level pacing works if the labels capture how contexts learn. Without those labels, per-context pacing still handles differences within a group. These results suggest pacing assistance separately for contexts that learn at different rates.

cs.LG↗

arXiv2026

SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR

The standard heuristic of selecting the SFT checkpoint with the highest pass@1 for GRPO can fail when SFT compresses the rollout distribution. For binary rewards, the expected within group advantage variance is $p(1{-}p)(g{-}1)/g$; when early GRPO drives $p$ below $p^*(g)$, most groups have identical rewards and provide no group relative signal. We study SFT depth ladders for Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B. We test Qwen2.5-Coder-3B across five depths and three seeds, and DeepSeek-Coder-6.7B across four matched depths and three seeds. On Qwen, pre RL pass@1 rises with SFT depth, but peak GRPO pass@10 falls from $0.806$ to $0.481$ (3 seed mean, $n{=}20$); pre RL entropy is positively associated with the GRPO outcome ($ρ{=}{+}0.69$). On DeepSeek, pass@1 remains far above $p^*(8){=}0.083$, and GRPO outcomes compress rather than invert. A two stage diagnostic, combining pre RL entropy triage with an early GRPO entropy monitor, flags high risk checkpoints and can stop failing runs early. Simple KL to reference regularisation and label smoothing variants do not rescue the collapsed Qwen checkpoint in our setting, suggesting the failure is not a trivial GRPO hyperparameter artefact.

cs.LG↗