NPTE - Mutuality & Mutual Preference

Hi all,

First of all, I am happy that there are conversations about helping the NPTE and the community’s wants and needs about how we use judge preferences now and in the future.

Secondly, I would like to remind folks - the NPTE Board officially does its business on Net-Benefits. I am probably one of the few that lacks access to the book of face, so I don’t have the ability to contribute to these conversations. Though, I am happy they are occurring. In the future, though, I have a request - I would like someone to post the link to the post or discussion on Net-benefits in a new thread to alert the board and myself that a response is sought.

Next, and most important, I am certainly on board with having us look at this pref data; it was something that was on my list of things to consider when I started as president, as I know it’s something that is very important to many members of our community. I have been in the process of compiling the last several years of data anonymized from past tab staff so that steps can be taken to evaluate these important questions about the implications of judge preferences. We are looking at the NPTE the last 6 years (may go back further, too), but there are some problems with obtaining the data - anonymizing the data is very time consuming and a process we’re looking into. Also, some of the STA files may have been eliminated or may be incomplete. Like I mentioned, we’re seeking the data we can find, if possible.

Now here are the next steps. I understand there are some who want copies of the data right, but this is my plan of action, as no data will be publicly released.

When thinking about how to analyze the data, there are some issues and things to consider regarding its use, which is why I have sought input from our IRB folks at my institution. One of the primary issues on the table is that we have not gathered consent for use of the data. Likely we will need to obtain legal consent from any competitor or coach to conduct the research. There is also a need to obtain the demographic data from all of former judges and competitors for which decisions in determining switches from various pref systems is needed. I have sought the help of one of my colleagues with a PhD in statistics to help me with this endeavor. As data becomes available, I will be happy to post it to Net-Benefits. As this is being cross posted, those stats gurus who want to contribute suggestions, possible research questions, ways to deal with a possible incomplete data set, feel free to post them there.

As many of you academics know, research takes time, and most importantly, I want to make sure it’s done right. I will seek help in expediting things as much as possible, so be on the look out for posts on NB etc. This year, prefs will remain the same until the research can be done and discussions can be had. We will certainly be seeking consent from attendees this upcoming year to ensure we can analyze the full data set so we can continue to make the tournament better in the future.

Starkey

Has also been posted to facebook by Zach Schneider for me!

Here’s the original thread for anyone who missed it (should be public)

I think there are a few important points of clarification that should be made regarding your post.

First, who is “we” in your posts? You note that “We are looking at the NPTE the last 6 years,” which implies that you are not the only one with the data nor are you the only person working with the data.

Second, some of the things in your discussion seem contradictory. For example, there was no consent provided when prefs were submitted for them to be distributed to anyone for research. This has been used as a justification to not release the data. If that is true, it is also true that no one provided permission for you to use the data. For example, when I debated (or coached in the last 6 years) we provided our prefs to Joe Gantt. There was never any indication that the prefs would be maintained beyond the use of that tournament (which is an important note in and of itself). In addition, there was never any notice that the data would then be given to the future, non-democratically elected president. You are therefore working with data that no one ever provided consent to be maintained or transferred to other members of the NPTE board or future board. If it is true that IRB would require consent of participants to use the data, then no one, including members of the NPTE board, should be opening or working with the data prior to obtaining that consent.

Finally, it seems that this is something you have been working on for awhile. I think that notice should have been provided to the community that this was an endeavor you were thinking about undertaking since there are obvious concerns about data.

There are a whole host of statistical tips that I could provide (I’ve previously put together proposals to do the type of research it appears you are now undertaking. I even think I explained all of my approaches and concerns to you previously). But without knowledge about what type of data there will be, it is difficult to provide any specifics at this point.

I agree these are worthwhile endeavors (hence me trying to do this research years ago), but I also think there should be more transparency (e.g., who has the data, who has opened the data, who is using the data, who has IRB approval, etc) and equal applicability of ethics concerns (e.g., if consent is needed to distribute, then no one should have the data other than Joe Gantt).

Cross-posting on facebook.

I have received a few questions from different folks for clarification from my original post, so here is some more information that should ameliorate any of those concerns.

Like I already mentioned, I’m still in the process of collecting and finding the data. I haven’t yet received a single full STA xls, so no one is working with it yet. There have been multiple presidents and multiple tab directors over the last several years, so the data is in several places already.

I’ve also spoken to a lawyer about this, and the NPTE technically owns the data, which means that as President of this non-profit, I am the gate-keeper of said data. Since this is the case, I am going to do the most ethical thing I can regarding how we move forward. It would be strange to think that tab directors that do not have an official affiliation with the NPTE anymore would be able to “own” the data. In the future, there will be an official policy to have tab directors pass along the final STA xls to the President in a password-protected folder and all other copies will be destroyed to ensure there are no other issues. Also related, there will be no releasing of any data to anyone. The NPTE Board is unanimous in this decision.

Also like I mentioned, I contacted my IRB office (with whom I have a meeting scheduled this week) to see if and how we ought to proceed to get the answers that the community seeks. I will not be bringing on any one affiliated with debate on the study to help protect the privacy of all of the judges and students.

If IRB says this is something that is possible, then steps will be taken to make sure the study is done right, and we’ll move forward in the most ethical way possible, such as with obtaining consent before any research happens, which I’m sure the IRB office will help us make sure happens ethically and effectively. If we move forward, I can inform folks who has IRB approval and what we will be looking at with the data. If they say it isn’t possible or consent cannot be obtained, then the data will be destroyed.

I will post again when I have more information after my meeting with my IRB office. Thanks!

Hi all,

Quick update - met with my IRB team on Monday. It looks like we’re going to go ahead and seek IRB approval. They aren’t sure if they’ll approve the study or not since we’d have to obtain retroactive consent from every judge and every competitor, but I was told it’s worth the try!

I’m planning to start working on the proposal this week and next with the hopes of the committee having their review period finished by January some time.

Thanks for everyone’s patience and support as we move forward!

Hi all,

Another update - I was informed by a director that there will not be consent given for any of their students or judges if retroactive consent is approved by IRB.

As a result, if there are individuals who do not consent to the data being used in a study, it cannot happen, thus, this study cannot occur.

Moving forward, participants at the NPTE this year will have the option of opting-in to data collection for a study to be done with this year’s data only. I’m rewriting the original IRB request to change the scope of the study due to these recent developments.

As I have more information as I move through the IRB process, I’ll certainly keep folks updated.

Starkey

For those of us not interested in the whole data set-- just clarity on what the system rewards and generally the trends of how preference/pairing happens, could we get some more explanation on that? Certainly that can happen without releasing a full set of data, right?

<3

-KD

Hey Kyle,

I’m waiting on the most recent version of STA (waiting on former tab directors to get it to me) to send to Gary Larson who will then give us more information on how the algorithm works.

Once I get that done, I’ll certainly post what Gary sends me here!

Starkey

Sounds good.