Major Questions Surround I-PIRP Assessment Process

By F. Joseph Wertz

This past year, New York State extended the pilot of the Internet Points Insurance Reduction Program (IPIRP) despite major questions regarding the studies supporting its effectiveness. There has been no conclusive proof that the I-PIRP actually improves driver safety and reduces the risk of accidents, yet the I-PIRP pilot program has been extended another five years.

The DMV has an obvious financial incentive to approve the program, yet had a hand in producing the report that recommended its adoption. The I-PIRP effectiveness study side-stepped regulatory requirements laid out in New York State law, and also suffers from serious methodological problems. Most importantly, the DMV has failed to directly address a simple question— does the I-PIRP make drivers safer?

This is worthy of public attention—if the effectiveness of a program that has a direct impact on public safety is unclear, innocent people will suffer. Accidents caused by dangerous drivers claim thousands of innocent lives every year and cost the public billions of dollars in damage and traffic delays. On top of that, if drivers get insurance benefits from ineffective courses, the general public will wind up footing the bill via higher insurance premiums.

The New York State Legislature should take action to repeal the I-PIRP until it can be adequately assessed. Unlike the U.S. Criminal Justice System, there isn’t a presumption of “innocent until proven guilty” when it comes to public policy. New programs need to be rigorously studied and critically assessed, and that hasn’t happened in the case of the I-PIRP. In 2009, the DMV began a five-year pilot of the I-PIRP course as an alternative to taking the standard in-classroom PIRP. The pilot was scheduled to last five years, and required the Commissioner of the DMV to prepare a report evaluating the effectiveness of the course. After this report, the I-PIRP was then to be either permanently adopted or rejected. The content of this report, titled the New York State Effectiveness Study of the I-PIRP Pilot, published in December 2013, raises more questions than answers about the viability of the I-PIRP and the DMV’s process of assessment in general.

Financial Incentives for the DMV to Adopt the I-PIRP A DMV document titled I-PIRP Pilot Results and Recommendations, published in February 2014, lists increased revenue as a benefit of the I-PIRP: “New York State receives $8 for each I-PIRP/ADM course completion compared to $2 for each PIRP course completion.” Not only is there a clear financial incentive for the DMV to permanently adopt the I-PIRP, but it’s openly stated as a main benefit. This raises the obvious question—is it appropriate for the DMV to be listing increased revenue as a benefit when public safety is in question?

Another important question that follows from this piece of information: Is it appropriate for the DMV to be involved in assessing a program that, if adopted, would quadruple their revenue compared to classroom PIRP delivery? It’s reasonable to think that the financial incentive might bias the DMV’s assessment of the more important question: Does the I-PIRP increase driver safety and reduce accidents?

According to the 2013 effectiveness study, “the DMV contracted with the University at Albany’s Institute for Traffic Safety Management and Research (ITSMR) to conduct the effectiveness study of the IPIRP. This report includes the results from both components of the study, a survey of participant satisfaction and a comparison of the Classroom and Internet delivery methods to effectively deliver the PIRP course.” The University of Albany hasn’t admitted to producing anything other than the data collection and analysis, yet the report contains a summary of findings and conclusions based on the data.

Now the question is, who authored the summary and conclusions contained in the report? Assuming the University of Albany sent the data to the DMV, as they were contracted to do, it must have been the DMV. Given the financial incentive stated above, this opens up the possibility of biased conclusions. Furthermore, one can wonder if placing the University of Albany’s name prominently on the cover of the study was an attempt to mislead the public into thinking the entire report was produced by an independent third party.

Effectiveness Study Doesn’t Fulfill NYS Statute Requirements Aside from the financial and authorship concerns, the report itself doesn’t even examine the correct question. According to DMV regulations, one of the goals of the I PIRP effectiveness study is to measure the impact and improvement of the safety course on the driving habits of those who take it.

The New York State DMV Commissioner’s Regulations part 141 (CR-141) states that the method for demonstrating effectiveness of the I-PIRP should be done “by measuring a significant impact in accident and conviction reduction in the 18 months after completing the course.” The problem is that neither the Effectiveness Study of the I-PIRP Pilot, nor the I-PIRP Pilot Results and Recommendations satisfy this requirement.

According to the I-PIRP Pilot Results and Recommendations report, the Effectiveness Study of the I-PIRP Pilot, conducted in part by University of Albany’s ITSMR “includes two major components: a survey of participant satisfaction with the I-PIRP/ADM course, and a comparison of the classroom and Internet delivery methods to effectively deliver the PIRP course.”

The data presented in this report focuses on comparing the Internet course to the classroom course, and concludes that the study data has shown “that the IPIRP/ ADM courses are as effective as the PIRP classroom course.” According to the Effectiveness Study of the I-PIRP Pilot, “the hypothesis of this component of the study is that the Internet method of delivering the PIRP course is comparable to the Classroom PIRP delivery method in regard to its impact on driver behavior after course completion.”

This is clearly not what CR-141 outlines. CR-141 asks if the I-PIRP has an impact on driver safety. The DMV did not answer this question. The DMV answered whether or not the I-PRIP delivery method is as good as the classroom method (as we will see later, even this conclusion is questionable). The hypothesis is clearly not what the statute outlines is needed to determine the pilot’s effectiveness. The conclusion side-steps the issue, which is asking whether or not the DMV has measured a significant impact in accident and conviction reduction in the 18 months after completing the course. The DMV’s non-answer to this question is a disturbing indication that the pure effectiveness of the I-PIRP hasn’t been proven.

Flawed Methodology of the Effectiveness Study But even the conclusion of the Effectiveness Study of the I-PIRP Pilot, which answers the wrong question in the first place, is questionable. Any findings based on the data collected and analyzed by University of Albany’s ITSMR are very vulnerable to doubt and shouldn’t be stated as fact. This is due to a methodological flaw in the way the study was conducted.

The major flaw in the 2013 study is that the participants were not randomly assigned to take either the I-PIRP or the classroom PIRP — the DMV simply used those who had already made the selection themselves. This will result in skewed data due to what statisticians call “selection bias,” which damages the integrity of the data collected.

The reason you can’t rely on this data is that there may be pre-existing differences between people who choose to take the IPIRP and people who choose to do the classroom PIRP. These differences could very possibly affect changes in driving behavior and post-course outcome. Since we don’t know if there are any pre-existing differences between the groups, we don’t know how the data might be biased.

A more rigorous study would have taken a group of people, and randomly assigned each to either the I-PIRP or the classroom PIRP, eliminating these so-called “selection effects.” By randomly assigning participants, there is no possibility that the groups are biased, because the individual’s choice is removed from the equation. Instead, the DMV effectiveness study relied on whatever process led each individual to decide between the I-PIRP and the classroom PIRP to determine the population for each program.

At the very least, one would expect the DMV to acknowledge the possibility of selection bias and selection effects. It is ethical to alert readers to potential flaws in the integrity of the data they’re using to assess the I-PIRP. It is very troubling that the DMV doesn’t delve into a thorough discussion of what the pre-existing differences between I-PIRP users and classroom PIRP participants might be. Neither do they acknowledge that the potential selection effects mean that any conclusions should be viewed very cautiously.

New York Needs to Act Aside from the specific facts of this case, New York State residents should think carefully about the nature of online education in general. While many institutions are moving towards online education, it is far from clear that it offers equal knowledge retention and learning to in-classroom education.

There is no opportunity to interact with an instructor, and we all know the infinite amount of distractions the internet puts at our fingertips. Sure, I-PIRP participants watch videos, read text and answer quiz questions. But this is nothing more than a glorified textbook. If you told your history professor that you read the textbook, would he give you credit for the course based on that alone?

Online education is certainly convenient, nobody is arguing that point. But what are we to sacrifice in the name of convenience? Major universities have hesitated to fully embrace online education. While some have been experimenting, the vast majority of college students are required to earn their degrees in a classroom. When public safety is on the line, shouldn’t New York State have equal or more rigid standards?

As a program that has direct impact on public safety, the New York state legislature has a duty to provide more oversight for the I-PIRP. Organizations such as the DMV are only effective if they can be properly controlled by legislative authorities. In this case, evidence suggests that a conflict of interest has slanted the assessment process. It’s time for the State Legislature to step in and fix this issue by repealing the I-PIRP pilot until it can be studied properly. There have been no concrete statements or plans from the DMV or the Commissioner about plans for a more extensive study of the effectiveness, so it will be up to the public to supply pressure on their State Legislators and the DMV itself.