Because both validity and reliability indices are a function of the scores on a given administration of a measure, their values can often vary across samples. It is a common mistake to say that a test is reliable when in fact it is not the test that is reliable but the scores on the test that are reliable. In 1998, Vacha-Haase proposed a fixed-effects meta-analytic method for evaluating reliability that is similar to validity generalization studies called reliability generalization (RG). This study was conducted to evaluate alternative analysis strategies for the meta-analysis method of reliability generalization when the reliability estimates are not statistically independent. Five approaches for handling the violation of independence were implemented: ignoring the violation and treating each observation as independent, calculating one mean or median from each study, randomly selecting only one observation per study, or using a mixed effects model. This Monte Carlo study included five factors in the method. These factors were (a) the coefficient alpha, (b) sample size in the primary studies, (c) number of primary studies in the RG study, (d) number of reliability estimates from each, and (e) the degree of violation of independence where the strength of the dependence is related to the number of reliability indices (i.e. coefficient alpha) derived from a simulated set of examines and the magnitude of the correlation between the journal studies (with intra-class correlation ICC = 0, .01, .30, and .90). These factors were used to simulate samples under known and controlled population conditions. In general, the results suggested that the type of treatment does not have a noticeable impact on the accuracy of the reliability results but that researchers should be cautious when the intra-class correlation is relatively large. In addition, the simulations in this study resulted in very poor confidence band coverage. This research suggested that RG meta-analysis methods are appropriate for describing the overall average reliability of a measure or construct but the RG researcher should be careful in regards to the construction of confidence intervals.
|Advisor:||Kromrey, Jeffrey D.|
|School:||University of South Florida|
|School Location:||United States -- Florida|
|Source:||DAI-A 68/04, Dissertation Abstracts International|
|Keywords:||Independent observations, Reliability generalization, Statistical assumptions, Violation of independence|
Copyright in each Dissertation and Thesis is retained by the author. All Rights Reserved
The supplemental file or files you are about to download were provided to ProQuest by the author as part of a
dissertation or thesis. The supplemental files are provided "AS IS" without warranty. ProQuest is not responsible for the
content, format or impact on the supplemental file(s) on our system. in some cases, the file type may be unknown or
may be a .exe file. We recommend caution as you open such files.
Copyright of the original materials contained in the supplemental file is retained by the author and your access to the
supplemental files is subject to the ProQuest Terms and Conditions of use.
Depending on the size of the file(s) you are downloading, the system may take some time to download them. Please be