continued the ability of the group being tested varies
the test administrators vary
the pool of graders varies
the pool of standard setters (final graders) varies
the final standards are contingent upon the approval of SED commissioners
A huge factor in successful performance on any test is motivation. The third through eighth grade test scores do not (cannot) count on the students’ grades. Students know that. So put yourself in their position. You are told you are going to have a state test that will not affect your grade and you are not able to study for the test anyway. What, therefore, is your motivation for doing well? What does this do to the validity of the test? The test is skewed from the start.
As part of the attempt to create valid tests, field tests (trial versions) are administered to selected schools, after the actual test is administered. Those students who didn’t care much about the actual test are going to be even less enthusiastic about this one — yet their performance is accepted by New York State SED as valid for test creation purposes. In fact, in the past, results from these trial tests have been plugged into the equations used to score the actual test.
Thus, the SED not only fails to provide a consistently valid evaluation tool, it sets up a constantly moving target that becomes increasingly difficult to hit.