Journal version of “Predict to Minimize Swap Regret for All Payoff-Bounded Tasks” (FOCS 24). Reject and resubmit at Operations Research.
A calibration error should calculate the contribution of miscalibration to downstream decision payoff. Optimizing a decision-theoretic error metric identifies better algorithms, while optimizing heuristic errors does not.
Large language models can be treated as perfect or imperfect oracles. We can design mechanisms with provable strategic robustness guarantees using oracle calls.
Improving model self-coherence leads to provable improvement in loss.
A strong but poorly calibrated model (e.g., a post-trained LLM) can be improved by a weaker, better-calibrated model (e.g., a pre-trained LLM) without ground-truth labels.
Truthfulness, the game-theoretic principle, is important to the evaluation of AI predictors. Prior work has shown that the ranking among AI predictors is not robust to the estimation method of a calibration error. We show that truthfulness improves this robustness of predictor rankings.
The evaluation of evaluation methods: we can evaluate evaluation methods by the alignment with human preferences; we can also evaluate their robustness to strategic manipulations.
In the sequential prediction setting, calibration errors cannot be perfectly truthful. However, simple reductions can make calibration errors approximately truthful in this setting.
AI systems should know the uncertainty in their outputs.
In the batch setting, there is no inherent conflict between calibration error and game-theoretic principles: simple modifications can make calibration errors truthful.
Predictions can look approximately calibrated under smooth errors yet still mislead decision-makers. We can fix them.
A calibration error should calculate the contribution of miscalibration to downstream decision payoff. Optimizing a decision-theoretic error metric identifies better algorithms, while optimizing heuristic errors does not.
Design AI systems with worst-case guarantees, just as we design algorithms.
Final-answer fine-tuning can be recast as a tractable two-player game, avoiding an explicit search over hidden reasoning paths.
A next-token predictor predicts a sequence of random states. The predictor can be robustified to support decisions in adversarial environments without losing in-distribution performance.
Information elicitation from strategic data sources. The theoretical foundation of my research.
The optimal scoring rule that maximizes revenue can be learned online, even when costs and information structures are initially unknown.
When a strategic data source exerts separate effort for each task, the optimal scoring rule can be unboundedly better than scoring each task separately.
We study optimal scoring-rule design across several mechanism-design paradigms: exact optimality for one-dimensional information, prior-independent optimality, simple approximations for multidimensional information, sample-complexity bounds, and computational results.
Understand how humans use computational systems, and how we can improve them.
AI explanation should not just present information that are critical to AI outputs, but present information that human does not know.
By comparing human decision-makers with a rational decision-maker in the same environment, we can understand appropriate reliance on AI and why humans make suboptimal decisions.
We design a differentially private user interface and understand human performance by benchmarking it against a rational decision-maker.
The rational behavior provides theoretical upperbounds and lowerbounds for performances in an experiment. When an experiment presents uncertainty information, human performance can be better understood by comparing it with this rational agent.
Market design and learning theory.
A switchback experiment posts random prices across a sequence of time horizons to test buyer response to price changes. When buyers strategically wait for discounts, inference from the classical two-price switchback experiment suffers a bias for that waiting behavior, but adding just one more price can fix it.
Our publication system is evolving in response to strategic publication behavior. We study the signals revealed by this system.
We give an efficient online algorithm that minimizes swap regret for convex losses.
A switchback experiment posts random prices across a sequence of time horizons to test buyer response to price changes. When buyers strategically wait for discounts, inference from the classical two-price switchback experiment suffers a bias for that waiting behavior, but adding just one more price can fix it.
In the batch setting, there is no inherent conflict between calibration error and game-theoretic principles: simple modifications can make calibration errors truthful.
We give an efficient online algorithm that minimizes swap regret for convex losses.
Final-answer fine-tuning can be recast as a tractable two-player game, avoiding an explicit search over hidden reasoning paths.
A next-token predictor predicts a sequence of random states. The predictor can be robustified to support decisions in adversarial environments without losing in-distribution performance.
AI explanation should not just present information that are critical to AI outputs, but present information that human does not know.
Our publication system is evolving in response to strategic publication behavior. We study the signals revealed by this system.
Predictions can look approximately calibrated under smooth errors yet still mislead decision-makers. We can fix them.
A calibration error should calculate the contribution of miscalibration to downstream decision payoff. Optimizing a decision-theoretic error metric identifies better algorithms, while optimizing heuristic errors does not.
By comparing human decision-makers with a rational decision-maker in the same environment, we can understand appropriate reliance on AI and why humans make suboptimal decisions.
We design a differentially private user interface and understand human performance by benchmarking it against a rational decision-maker.
The rational behavior provides theoretical upperbounds and lowerbounds for performances in an experiment. When an experiment presents uncertainty information, human performance can be better understood by comparing it with this rational agent.
The optimal scoring rule that maximizes revenue can be learned online, even when costs and information structures are initially unknown.
When a strategic data source exerts separate effort for each task, the optimal scoring rule can be unboundedly better than scoring each task separately.
We study optimal scoring-rule design across several mechanism-design paradigms: exact optimality for one-dimensional information, prior-independent optimality, simple approximations for multidimensional information, sample-complexity bounds, and computational results.