Learning best response strategies for agents in ad exchanges

Ad exchanges are widely used in platforms for online display advertising. Autonomous agents operating in these exchanges must learn policies for interacting profitably with a diverse, continually changing, but unknown market. We consider this problem from the perspective of a publisher, strategically interacting with an advertiser through a posted price mechanism. The learning problem for this agent is made difficult by the fact that information is censored. We address this problem using an approach that conceptualises this interaction in terms of a Stochastic Bayesian Game and arrives at optimal actions by best responding with respect to probabilistic beliefs maintained over a candidate set of opponent behaviour profiles, while, also, addressing the case of stochastic opponents. We evaluate the proposed method using simulations wherein we show that we achieve substantially better competitive ratio and lower variance of return than baselines.