The TCCA Executive Committee (EC) did not approve the changes to the ISCA review system that the R2 Committee was proposing to implement without first getting community feedback. This blog post summarizes the concerns that were raised by the TCCA EC Members who voted against the Revised R2 proposal. Such concerns were shared with the R2 Committee.

This blog also outlines an alternative approach we call the Community Proposal to improve the ISCA review system that we hope will be combined with the R2 efforts after gathering extensive community feedback. The TCCA EC voted to post the Community Proposal in the TCCA Website to gather community feedback but, to be considerate to the R2 Committee, only after the R2 Committee presents at ISCA 2018.

Concerns Raised by the TCCA EC Members who voted against the Final (or Revised) R2 Proposal

We sincerely appreciated the work of the R2 Committee. However, after  extensive thought, we did not approve the Revised R2 proposal for the following reasons:

1.The proposal entails such a big change over current practices that it should not be instituted from above by about 20 people; it should be made known to, discussed by, and vetted by the community first.

The core of the R2 proposal (multiple submission deadlines for ISCA) is a major change. It changes the direction of ISCA reviewing and has major ramifications (see below). A change of this size should not be instituted from above by about 20 people to an unwitting community. Instead, it should be discussed and vetted by the overall community first.

2. By proposing an “October-to-January, Multiple-deadline ISCA Submission Period”, the proposal would reduce submissions to the other top conferences.

Given the perceived higher prestige of ISCA, the outcome of this 4-month “ISCA Period with Multiple ISCA Deadlines” would likely be to shift paper submissions away from other top conferences such as HPCA and MICRO, in favor of ISCA submissions. It is likely that a two-tier conference ranking structure would emerge: ISCA first, then the rest. We believe this is a negative outcome for our growing community. To nurture our growing community, we need to: 1) preserve multiple high-prestige conferences and 2) retain the diversity that multiple such conferences provide. The impact that the proposal would have on other conferences is another compelling reason why the whole community would need to vet the R2 proposal first.

After an Initial R2 proposal that had year-round ISCA submission deadlines was proposed, a majority of TCCA EC members agreed to let the R2 committee explore a revision to the R2 proposal that (quoting the request) “schedule(d) the paper submission deadlines such as to prevent/minimize ISCA hurting the number of submissions to HPCA (and MICRO).” We believe that the Revised R2 proposal with the “4-month ISCA Period” still fails to safeguard the other conferences.

3. The proposal does not fully address the main feedback received by the R2 survey “to reduce the reviewer’s load,” but rather increases the load of the ISCA reviewer.

The main feedback received by the R2 survey is the need to reduce the reviewer’s load. We believe that the way to do so is to set up a robust revision system, so that the ubiquitous repeatedly-resubmitted papers are finally accepted (or rejected). The R2 proposal, instead, introduces multiple deadlines for ISCA (ranging from early October to late January). The main outcome of this “October-to-January ISCA Period” would be to increase the fraction of the yearly paper submissions to the top conferences that go to ISCA. Hence, rather than seeing a load reduction, ISCA reviewers would see their load increase.

4. While the proposal is presented as a two-year trial, there is no discussion of how to evaluate whether the experiment is a success and should be implemented in the steady state.

There is no clear statement that describes how to decide whether the trial has improved the reviewing process or, instead, has had undesirable effects on reviewers’ load and on other conferences, and should stop. Spelling out the criteria that would be used to decide if an experiment should stop or continue is an important requirement of any proposal of this kind. Such criteria should include feedback from the community.

A Community Proposal to Improve Our Reviewing System

Our Community Proposal is based on the feedback of the R2 Survey , one of whose main highlights is that about 50% of respondents feel that PC members are overloaded. Our hypothesis is that a very large fraction of papers being reviewed are resubmissions. Many such papers are good. However, a paper gets different reviewers every time, who provide different feedback. Hence, some authors tend to resubmit with only minor changes. This situation frustrates authors and reviewers.

The proposal’s goal is to help get many of these papers accepted (and some rejected) by providing a robust Revision Process. The load of the reviewers will decrease because reviewers will only review the incremental changes since the last review, and the total number of re-submitted papers will eventually decline.

A revision-based model gives authors the opportunity to address reviewer comments and be re-evaluated by the same reviewers. Moreover, it opens a path to journal publication.

This proposal uses the R2 feedback to coordinate the review process for the ISCA, HPCA, and MICRO conferences. Papers submitted to a conference may be: accepted (possibly with revision), invited for a “Large Revision”, or rejected. It is envisioned that many papers that would otherwise be rejected will receive a “Large Revision”. Such papers:

  • Can be resubmitted to the deadlines of one of the next two conferences in the calendar year

  • If resubmitted, they should include the old reviews and highlight the changes

  • If resubmitted, they are reviewed by (mostly) the same set of reviewers

The Community Proposal document presents one possible implementation. Its highlights are:

  • The Program Committee (PC) focuses on reviewing the new papers

  • Many members in the External Reviewing Committee (ERC) focus on reviewing resubmissions

  • There is some overlap between the PC of a conference and the ERC of the next conference(s)

  • Final decisions on publishing revised papers at a given conference need to be endorsed by that conference’s PC

This general approach has several advantages:

  • It is inclusive of our entire community, helping all our diverse conferences thrive

  • It reduces the reviewer load, as the PC focuses on new papers, many in the ERC focus on incremental paper changes, and the number of resubmissions decrease

  • It improves review fairness, as it encourages an honest effort to accept papers, and keeps reviewers accountable

  • It paves the way for journal publication of all papers accepted at any of the three architecture conferences. This will increase the visibility of our community

The Community Proposal will be posted in the TCCA Website with the list of authors after ISCA 2018, and open for a multi-month period of extensive community feedback collection. It is our hope that this initiative can help improve the R2 effort.

– Josep Torrellas, Antonio Gonzalez, Daniel A. Jimenez, John Kim, Hsien-Hsin Sean Lee, Onur Mutlu, Vijay Janapa Reddi, Per Stenstrom and Lixin Zhang