May 10, 2018
1. Rationale of this “Community Proposal”
The main findings of the R2 community feedback were that, of the respondents:
~74% believe in-person PC meetings are important
~50% feel that PC members are overloaded
~50% would like to see ISCA tied to a journal
~50% believe integrity and fairness are paramount
This proposal uses this feedback to suggest a cross-conference change to the reviewing process to decrease the load of the reviewers, while increasing the number of good papers that are accepted per year to top architecture conferences.
The hypothesis of this proposal is that a very large fraction of papers being reviewed are resubmissions, many with minor changes. A paper gets different reviewers every time,
who provide different feedback. This situation frustrates authors and reviewers.
This proposal’s goal is to help get these papers accepted (or rejected) by providing a robust Revision Process. The load of the reviewers will decrease because they will only review the incremental changes since the last review, and the total number of papers resubmitted will go down.
A revision-based model gives authors the opportunity to address reviewer comments and be re-evaluated by the same reviewers. Moreover, it opens a path to journal publication.
2. Detailed Proposal
The proposal is to coordinate the review process for the ISCA, HPCA, and MICRO conferences. Papers submitted to a conference may be (1) accepted (possibly subject to some revision as already done in our community), (2) invited for a larger revision, or (3) rejected. A revised paper will be re-submitted with highlighted changes plus a summary of changes, the old paper, and the old reviews.
In the following, we outline an implementation, where a revised paper can be re-submitted to the deadlines of one of the next two conferences in the calendar year, and will be reviewed by the same set of reviewers.
Each conference has a Program Committee (PC) and an External Review Committee (ERC). Some PC members of one conference serve as ERC for the next two conferences, to re-review revised papers. PC members review new submissions. ERC members review revised submissions. The PC always meets in-person. The ERC meets via a conference call. The ERC reviewers need to agree on whether the authors have adequately addressed the review comments for the revised papers. The ERC reviewers make a recommendation to the PC to accept or reject a revised paper, but the final decision needs to be endorsed by the current PC. The PC Chair oversees the re-review of the revised papers.
The PC Chair may choose to invite additional ERC members to broaden the pool of expertise in the ERC to assist with the review process of new submissions. The PC Chair may choose to delegate the oversight of the re-review of a revised paper to an expert PC member. The PC member may be asked to present the revised paper at the in-person PC meeting so that the entire PC is still engaged in the selection of the program.
This proposal satisfies the key findings of the R2 community feedback listed above. First, it decreases reviewing load, as PC members only need to review new submissions, and ERC members review incremental changes. Second, the PC reviewers always meet in-person to make the final decision on a paper. Third, the quality and fairness of the review process improves because the authors have the opportunity to revise their paper and be re-evaluated by
the same set of reviewers.
To prevent the blurring of the identity of individual conferences, each conference still retains full authority to accept/reject revised papers. Also, a given conference may encourage and/or discourage certain topics. This allows for technical diversity to thrive in our growing field.
Other implementations are possible. They may involve using only a subset of the same reviewers, and/or re-submitting only to the same conference, to appear one year later.
3. Key Assets of this Proposal
1) It is inclusive of our entire community, helping all our conferences and our overall growing community of authors thrive. It allows for technical diversity to thrive.
2) It reduces the reviewer load, improving reviewer quality and reducing reviewer fatigue.
3) It retains in-person PC meetings, which enhance the integrity of the reviewing process.
4) It supports a revision-based review model, hopefully by the same set of reviewers, improving review fairness.
5) It paves the way for journal publication of all papers accepted at any of the three architecture conferences.
4. Further Action Items
This proposal welcomes feedback from the broad computer architecture community. Such feedback will determine how this proposal eventually shapes up.
This proposal requires cooperation between the multiple conferences. It is highly expected that the different sponsoring organizations will closely cooperate for the sake of ensuring a thriving and growing computer architecture community.
The IEEE CS and ACM Publications Boards will be approached for the creation of a jointly sponsored ACM/IEEE journal to publish papers accepted at any of the three conferences. This will increase the visibility of our community.
– Antonio Gonzalez, Daniel A. Jimenez, Hsien-Hsin Sean Lee, John Kim, Josep Torrellas, Lixin Zhang, Onur Mutlu, Per Stenstrom and Vijay Janapa Reddi
In addition, please leave your comments (optionally anonymous) for everyone else to see.