Grading the test: How glitchy exams and poor funding left Montana with few assessment options

Grading the test: How glitchy exams and poor funding left Montana with few assessment options

  • 0
Reading Test (IR copy)

Butte's East Middle School eighth-graders take a reading test in this January 2015 file photo. The students are tested three times a year on vocabulary, comprehension and silent fluency. Teachers continually glean data as students learn in order to better adjust Montana's Common Core standards.

After glitches bogged down standardized testing in many Montana schools last spring, it was easy to point fingers.

Smarter Balanced, the consortium of states Montana worked with to design the tests, rolled out a different platform than used during pilot testing. Measured Progress, the vendor for the test, didn’t adapt well, and in turn blamed the company that built the platform for not delivering its work on time.

But the bottom line is that in a year when a new wave of standardized tests swept across the nation, Montana had few options.

“If Montana was forced to leave the consortium and was forced to go at it on its own, I honestly don’t know what you guys could get,” said Scott Marion, a former Wyoming education official who is now executive director for the New Hampshire nonprofit Center for Assessment. “Unless the state was willing to pay a considerable amount of money.”

That doesn’t appear likely. Montana offers no additional funding for state standardized testing other than what federal officials provide, which has been about $3.6 million per year in the last few years. Some districts offer their own standardized tests, but those are not consistently comparable to other schools.

State officials and local leaders have expressed confidence that last spring’s problems were a one-time issue and that testing will go smoothly in the future. There’s no apparent reason it shouldn’t. Montana has worked with Measured Progress for several years.

“We had one unusual year,” Superintendent of Public Instruction Denise Juneau said. “Prior to that, we had great testing experiences.”

Test score improvement requirements that were mandated by No Child Left Behind were dumped in new education legislation passed earlier in December, but statewide testing is still required.

Higher standards

NCLB, while still on the books through this year, was already fading in relevance when Montana adopted new Common Core standards in 2011. It was a significant change for the state — previous standards only defined where students should be at grades 4 and 8 and at graduation. Now each grade has benchmarks, which are largely recognized as more rigorous.

Many states adopted Common Core standards with encouragement from federal education officials, although the feds were prohibited from mandating their adoption. Montana officials have been adamant that the state changed its standards on its own terms.

New standards set the stage for new tests, and the market shifted toward computer-administered assessments that could offer deeper insights than paper-and-pencil, fill-in-the-bubble versions. But creating the new tests isn’t cheap — just one question could cost thousands of dollars.

That’s part of the reason that states banded together into a pair of consortia — Smarter Balanced and the Partnership for Assessment of Readiness for College and Careers, or PARCC. When Montana’s bill was totaled for the creation of new tests, it ran about $27 per student, which was “amazingly cheap for a state like Montana,” Marion said.

Smarter Balanced currently has 15 governing member states and three affiliate member states.

By comparison, Wyoming, another state with a small population, decided to go it alone and contracted a company to create tests specific to Wyoming. It cost about $76 per student.

“That’s why we joined a consortium,” Juneau said. “To get a better bang for our buck.”

Pilot testing

During the 2013-14 school year, state officials decided to scrap Montana’s Criterion-referenced Test, the previous statewide measuring stick, in favor of a pilot run of the new assessment with 70,000 students. By most accounts, things went swimmingly.

There was one unforeseen issue. Smarter Balanced contracted with American Institutes for Research to create an open-platform for the test, which would be accessible to other vendors in turn.

But AIR used the proprietary version of the platform for the pilot test. Schools weren’t using the exact same product they would for the real deal in 2014-15.

AIR itself is a vendor for several states — essentially a competitor with Measured Progress, though both groups are nonprofits. The relationship between the two would get messy as problems popped up. Measured Progress said that AIR didn’t deliver on its end of the deal on time, leading to a domino-effect of delays and glitches.

Montana signed up Measured Progress as its vendor in October 2014 on a five-year contract worth about $1.3 million per year. It was the only bid that Montana received that the state could afford to pay, state officials said. Nevada and North Dakota were the only other states to sign with Measured Progress.

In South Dakota, they took the consortium approach a step farther by partnering with five other states to sign a vendor. After AIR won that bidding process, each state signed its own contract with the company.

“The states are not set up to do that kind of collaboration,” Marion said. “You wouldn’t believe how hard it is to pull that off.”

South Dakota education officials didn’t respond to requests for more information about how the six-state process worked.

Problems begin

There were troubles with tests last spring before they even started. On March 12, Measured Progress notified OPI that tests wouldn’t be ready for the beginning on the testing window on March 18. On March 22, state officials told schools that the rescheduled start date of March 25 would be pushed back even further.

Many larger schools had staggered schedules set up for students in order to not overwhelm their computer networks. They scrambled to adjust schedules as officials announced testing would begin March 30 and run through June 12.

Things didn’t go well once testing started as glitches popped up sporadically across the state. Bozeman Public Schools were especially hard hit.

“There was either freezing or students were locked out and couldn’t get back on,” Bozeman Superintendent Rob Watson said.

Juneau announced that schools could opt out of testing despite federal requirements, raising the unlikely possibility that Montana could be penalized for not meeting participation benchmarks. The state was later granted a waiver, giving it a pass.

Some districts, like Billings, were able to test most of their students. But Bozeman only tested about one-third of eligible students and didn’t test high school juniors at all. That grade level takes the ACT test, and the district decided that, given the issues, it would do for this year.

“It was affecting our instructional time,” Watson said. The district decided to only test certain groups of students, to hopefully create a representative sample. Bozeman administers other standardized tests on a district level, and hopes to use Smarter Balanced scores to gauge how well a different test measures up to the new state standards.

Juneau also said that as tests were being scored, some data had to be thrown out because of other glitches even if students completed the test.


Nevada and North Dakota experienced many of the same problems as Montana. All three states announced that they’d consider options for seeking recourse, with Nevada and North Dakota threatening legal action. North Dakota eventually went after Smarter Balanced for more than $300,000.

Nevada, whose contract with Measured Progress expired after last school year, eventually signed a new vendor to a $51 million dollar contract. Montana officials have said that even if their contract was up, the state couldn’t afford a similar offer.

Montana withheld monthly payments of $118,000 from July to November, saying it wouldn’t pay until Measured Progress delivered results. The state also negotiated a Smarter Balanced membership fee reduction of $375,596, about half the total costs.

Smarter Balanced also worked with the Center for Assessment, which has advised the state on standardized testing in the past, to create a validity study to determine the impact of glitches on the results. It’s taken longer than expected to finish.

“Getting the data on the interruptions is the most challenging feat,” Marion said. But overall, “we tend not to find huge effects.”

The state released results on Monday, which showed that about 71 percent of students recorded scores on the test. Proficiency levels dropped compared to previous standardized tests in the state, but the Smarter Balanced tests are widely recognized as more rigorous and other states have seen similar drops. Juneau said that comparing the tests, which are tailored to different standards and delivered in different ways, is like comparing apples and oranges.

Common Core opposition

Common Core has become a loaded phrase around the nation. Several states that initially adopted the new standards have reversed course and dropped them.

In the last legislative session, a bill was introduced that would have required the state to use a test that was designed only for Montana standards, effectively killing the option of using a consortium. The bill died in committee.

If Montana were to ditch the Smarter Balanced tests after a short period, it would effectively create a gap in achievement data. Smarter Balanced isn’t comparable to previous tests, and another new test likely wouldn’t be comparable to either.

“They shouldn’t be political footballs,” Marion said. “I just think it’s really shortsighted.”

Common Core opponents often consider Smarter Balanced tests to be part of a larger overreach into local control. State Sen. Roger Webb submitted a bill request for the 2015 session to bar any funding for Common Core, saying in 2014, "I'm going to defund it. They can do what they want. I'm going to shut the money off."

At the time, Juneau noted that there was nothing to defund, as the Legislature didn’t have any money allocated to Common Core development.

Ditto for testing.

Going forward

Since receiving results, OPI has paid Measured Progress for October and November, but it still is withholding about $130,000 until the company delivers “everything required in the contract.”

Montana intends to see through its contract with Measured Progress, and education officials have expressed confidence in the new tests for this spring, even in some of the schools that had the most problems.

“We don’t have any concerns looking ahead to next year,” Watson said.

Measured Progress is negotiating with AIR to use the proprietary platform that was used during the well-run pilot tests.

The Every Student Succeeds Act, NCLB’s successor, doesn’t requiring the same kind of testing accountability.

“I’m not sure if the assessment will change,” in response to ESSA, Juneau said. The way it ties into school accountability, “may shift a little bit for us.”

Some states have left consortia, arguing that they can save money with their own tests. Those tests won't offer the same level of feedback, Marion said.

“It’s like everything else in life," he said. "You get what you pay for.”

Most people agree that no matter how the state moves forward, consistency is huge.

“We have a saying in education measurement,” Marion said. “If you want to measure change, don’t change the measure.”


Be the first to know

* I understand and agree that registration on or use of this site constitutes agreement to its user agreement and privacy policy.

Related to this story

Get up-to-the-minute news sent straight to your device.


News Alerts

Breaking News