According to the study from Mathmatica:
Preliminary results from a Chicago program containing performance-based compensation for teachers show no evidence that it has boosted student achievement on math and reading tests, compared with a group of similar, nonparticipating schools, an analysis released today concludes.
The program also failed to improve teacher retention rates. Over the two years of the study, achievement gains from the first year evaporated in the second year. Researchers and commentators speculate at least three factors account for the disappointing outcomes:
1. Reform takes time. It is unreasonable to expect stable achievement gains after only two years.
2. Schools did not adhere to a standardized implementation of the plan. Since not everybody did it the same way, the averages may be washing out positive effects in some schools.
3. The basic design of the plan needs tweaking. Perhaps higher or “more meaningfully differentiated” payouts.
It is impossible to generalize from the mere handful of studies on the efficacy of merit-based or performance-based incentives. However, many researchers have studied extrinsic versus intrinsic rewards.* The meta-conclusion is that extrinsic rewards actually undermine achievement. What works is intrinsic, not extrinsic rewards.
You don't hear hardly anyone but teachers mention what may be the most salient factor. Merit pay did not lead to achievement gains as measured by higher test scores because teachers simply do not have control over the myriad of variables that affect student achievement.
*Results of Google search on “research extrinsic rewards”