Compulsion and the mechanisms of model-based reinforcement learnin

Decisions and neural correlates of decision variables both indicate that humans and animals make decisions taking into account task structure. Such deliberative, "model-based" choice is thought to be important for overcoming habits and various sorts of compulsions, but there is still little evidence about the algorithmic or neural mechanisms that support it. I discuss recent studies attempting to address these questions. First, although it is widely envisioned that such model-based choices are supported by prospective computations at decision time, there are also indications that such behaviors may instead be produced by various sorts of precomputations. I present fMRI data from a sequential decision task in which states are tagged with decodable stimulus categories, which demonstrate a correspondence between predictive neural activity and other behavioral and neural signatures of model-based and model-free learning. This supports the widespread supposition that these behaviors are indeed supported by prospection. Second, I present some early and ongoing studies examining to what extent decisions are informed by representations of individual episodes, vs. statistics aggregated over multiple experiences as learned by typical algorithms, both model-based and model-free. Memory for episodes could support distinct computational approaches to the decision problem, including monte carlo and kernel methods, and also might support some apparently model-based behaviors. Finally, I discuss recent efforts to test the hypothesis that disorders of compulsion are related to deficient model-based learning. Although several patient studies appear to support this idea, there seems to be a lack of specificity in this effect, as patients with apparently non-compulsive disorders show similar deficits. We suspect that the issue relates to a much more general issue in psychiatry: the comorbidity and the poor specificity of psychiatric diagnoses. I present results from a large-scale online study of variation in psychiatric symptoms in a healthy population, which suggests a way to cope with these issues.