Towards a theory of inference-time alignment with unknown rewards

TL;DR - This paper casts inference-time alignment with unknown rewards as a PAC-style weak-to-strong learning problem. It introduces “alignment dimension,” a combinatorial measure that exactly characterizes whether a reward class is alignment-learnable.

• Learns entirely from data without assuming access to an accurate reward estimate.
• Allows each prompt to have multiple good responses.
• Proves learnability holds if and only if the reward class has finite alignment dimension.
• Uses the one-inclusion graph algorithm to conduct tournaments between incomparable label sets.