| Summary: | Impact evaluation is concerned with generating evidence of a causal effect of a policy or a treatment. The central question in any impact evaluation is what would have happened to those receiving the intervention had they not received it. Randomized controlled Trials (RCT) are best at answering this question because balance is achieved through randomization. I note that the balance achieved through randomization is in terms of distribution and not only in some moments of the covariates being compared. When RCTs are not feasible the alternative approach is an observational study or a quasi-experiment. Unlike RCT balance checks in observational studies are often limited to comparing the first or first and second moments of the distributions being compared. This distinction between balance in distribution and balance in a few moments (of observables) forms the basis of the questions this thesis attempts to answer. It is argued that various balance measures used to assess balance in the literature capture different aspects of balance. Measures that utilize only the first two moments may be ignoring information that affect bias and robustness of treatment effect estimates. Additionally, it is argued that measures that compare distributions may also differ in their performance depending on the way they quantify the difference between distributions. In this thesis I introduce the entropic distance metric as a measure of balance. Unlike the mean balancing approach, the proposed measure assesses balance by comparing all moments of the distribution of covariates across treatment status. My result suggests that the proposed entropy measure quantifies balance better than the standardized difference in means and the Kolmogorov-Smirnov (KS) statistic. It is shown that when the summarized distribution of covariates in the treatment and the control group are close in terms of the entropic distance, the treatment effect will be les biased and more robust across econometric techniques. After an introduction, the first substantive chapter introduces the theory of the entropic measure as a way to better assess balance. The chapter provides a simple discrete example where the entropic distance measure detects differences (imbalance) in covariate distribution that other measures of balance used in the literature will not pick up. Since the kind of imbalance in this example has consequences for the bias and robustness of treatment effect estimates, I argue that the proposed measure has an important role to play. The second substantive chapter applies the entropic distance measure to the well-studied National Supported Work Demonstration (NSW) Programme. It shows that the entropic distance measure performs well in identifying non-experimental control groups that yield better estimates of the average treatment effect when result from an experiment is taken as the unbiased treatment effect. The last substantive chapter considers the case of the South African Child Support Grant (CSG). In this chapter, the treatment effect of CSG on height-for-age z score is re-estimated using Genetic matching (GenMatch). GenMatch optimized balance by reweighting covariates. This method compares balance in different (weighted) samples iteratively until balance can no longer be improved. Therefore it is important to use a balance measure that can adequately differentiate between different levels of balance under this method. My result show that treatment effect estimates depend on the balance measure used to optimize balance. The result also suggests that using the entropy measure to assess balance leads to stronger effect estimates in terms of the size of the effect.
|
|---|
