Program Chairs Comments on Ethics Review Process
Slides (PDF): Takeaways from Ethics Review by Program Co-Chairs
The process followed for the NAACL 2021 ethics review was outlined by the Ethics Chairs in an earlier post. Comments from the ethics reviewers and track chairs were reviewed by the Program Chairs when making final acceptance decisions on the papers. As a result, 7 papers were accepted conditionally, which means that there were certain conditions imposed on the paper in the form of questions or requirements to be fulfilled for the paper to be accepted to the conference. For the majority of conditionally accepted papers, these included clarifications of the process the studies underwent for human subjects research, institutional review board protocols established for the studies, and discussion of whether and how the data collection procedures were coordinated with and complied with the rules outlined in terms of service for the websites and organizations that owned the data. As a result, 6 conditionally accepted papers were accepted into the conference, and one paper was rejected.
Our takeaway from this process is as follows: While we think the ethics review process is highly valuable, we found it very difficult to execute well in its current form. We believe that ethics review should be done as much as possible prior to paper submission, as described below. A conference program committee is not the best group to ensure that the reported work complies with ethical standards.
The reasons are two-fold. First, making last-minute ethical decisions about specific studies so late in the game, when the study is completed and written up, is less than ideal. A much better way to do it would be when the study is designed and considered, and this is normally precisely when the investigators set up human subjects research protocols with their institutional review boards (IRBs). Setting up a study protocol approved by an IRB may take up to several months and often involves back-and-forth, with clarifications from the authors, suggestions from the IRB to reframe the research or correct the protocol appropriately. This is simply impossible (and also too late) to do in the context of ethics review in the timeframe of a conference. Further, IRBs are typically composed of diverse individuals with complementary expertise from different fields, and as such, already are designed for such a review process. This is why we believe the burden of ethics review should be shifted back to this existing and well thought-out mechanism of ensuring that the research is conducted ethically.
No doubt, IRBs have to be educated about some of the subtleties of AI work and its implications for society at large. This should be done by all of us who work in the NLP field and understand these issues.
As far as compliance of the authors with the terms of service and other considerations related to data copyright and use ethics, we felt that we lacked the expertise and bandwidth to address such questions properly, and as a result, we believe that this might be much better handled by legal departments in the authors’ institutions.
To sum up, because of the difficulties involved in making ethical judgments, we felt that many of the questions we were trying to address during the ethics review process were better left to the properly composed multi-disciplinary IRBs and, as far as terms of service compliance is concerned, institutional legal departments. If an author reports that a human subjects study protocol and compliance with service terms is approved by their institution, that should provide sufficient guarantees for the ethics review process done by the conference.
This, of course, raises the question of what to do for unaffiliated authors or in cases when an author’s institution does not have an IRB and/or a legal department that can help address such issues. This is a difficult question to resolve, and perhaps one of the solutions might be to establish a standing review board associated with the ACL which can review such cases at author’s request.