Examining the testing processes, as well as the scores, is needed for a complete understanding of validity and fairness of computer-based assessments. Examinees’ rapid-guessing and insufficient familiarity with computers have been found to be major issues that weaken the validity arguments of scores. This study has three goals: (a) improving methods to set and evaluate threshold values for detecting rapid-guessing, (b) comparing the response behaviors of examinees with high and low computer familiarity performed, and © understanding the potential relationships between computer familiarity, response processes, and scores.
Data were drawn from the Programme for International Student Assessment (PISA) 2012. Specifically, the analysis of included Australian examinees’ responses to ten released computer-based mathematics items, log data associated with these questions, and seven questions in the information and communications technology (ICT) questionnaire on using computers in schools for mathematical tasks.
Response time based methods suggested by Kong, Wise, and Bhola (2007) were used for setting thresholds for identifying rapid-guessing. Enhanced methods were constructed by incorporating response time and number of response behaviors. Both response time based methods and enhanced methods were evaluated using Wise and Kong’s (2005) criteria as well as number and type of response behaviors. To explore computer familiarity, response behaviors of examinees with low-computer familiarity and high-computer familiarity were compared, and correlations were calculated between computer-familiarity, response behaviors, and response times.
Findings indicated that enhanced methods performed superior than the response time based methods on some of the evaluation criteria; examinees with low computer familiarity displayed some response behaviors more frequently than the examinees with high computer familiarity; and no association between computer familiarity, scores, number of response behaviors, and response time were found. This study concluded that using response times and response behaviors were useful for improving identifying rapid-guessing and exploring the potential role of low computer familiarity on examinees’ testing processes and scores.
|Advisor:||Colvin, Kimberly F.|
|Commitee:||Feyzi Behnagh, Reza, Ford, Michael T.|
|School:||State University of New York at Albany|
|School Location:||United States -- New York|
|Source:||DAI-A 78/09(E), Dissertation Abstracts International|
|Subjects:||Educational tests & measurements, Education, Educational psychology|
|Keywords:||Computer familiarity, Computer-based tests, Rapid-guessing, Response behavior, Response time, Validity|
Copyright in each Dissertation and Thesis is retained by the author. All Rights Reserved
The supplemental file or files you are about to download were provided to ProQuest by the author as part of a
dissertation or thesis. The supplemental files are provided "AS IS" without warranty. ProQuest is not responsible for the
content, format or impact on the supplemental file(s) on our system. in some cases, the file type may be unknown or
may be a .exe file. We recommend caution as you open such files.
Copyright of the original materials contained in the supplemental file is retained by the author and your access to the
supplemental files is subject to the ProQuest Terms and Conditions of use.
Depending on the size of the file(s) you are downloading, the system may take some time to download them. Please be