|
Size: 3390
Comment:
|
Size: 4423
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 12: | Line 12: |
| Each review ''must'' provide the following information: | Each review ''should'' provide the following information: |
| Line 15: | Line 15: |
| <p>1. A short summary of the main contribution(s) of the submission in the words of the reviewer</p> | <!-- <div style="display: inline-block; border: 2px solid #f0f0f0; background-color:#f9f9f9; text-align: left; vertical-align: middle; padding:10px 20px"> --> <p style="color: darkblue"> 1. A short summary of the main contribution(s) of the submission in the words of the reviewer<br/> 2. An itemized list of the strength and weaknesses of the submission<br/> 2.1 The strengths should be numbered (S1), (S2), ...<br/> 2.2 The weaknesses should be numbered (W1), (W2), ...</p> |
| Line 17: | Line 22: |
2. An itemized list of the strength and weaknesses of the submission<<BR>> 2.1 The strengths should be numbered (S1), (S2), ...<<BR>> 2.2 The weaknesses should be numbered (W1), (W2), ...<<BR>> |
|
| Line 24: | Line 25: |
| 3. More detailed explanations of the strengths and weaknesses<<BR>> 4. Comments to the authors for improving the paper<<BR>> |
{{{#!html <p style="color: darkblue"> 3. More detailed explanations of the strengths and weaknesses<br/> 4. Comments to the authors for improving the paper</p> }}} You can change the text of your review in the [[http://ad-wiki.informatik.uni-freiburg.de/research/ESA2018Experiment/DiscussionPhase|discussion phase]]. However, the discussion phase (and the whole reviewing process) will not work, if your initial review is not substantial. |
| Line 29: | Line 35: |
| Each review should provide one of the following scores. | Each review should provide one of the following scores. Just like the text of your reviews, these scores are important for the [[http://ad-wiki.informatik.uni-freiburg.de/research/ESA2018Experiment/DiscussionPhase|discussion phase]]. You can change your scores during the discussion phase, but it will greatly help the efficiency and quality of the process, if you hit the "right" score for a paper already in your initial review. |
| Line 31: | Line 37: |
| || '''Score''' || '''Verdict''' || '''Behavior during discussion''' || || +2 (accept) || Good fit and no major weaknesses || I would champion this paper and fight against rejection || || +1 (weak accept) || Significant weaknesses, but nothing fatal || I would support this paper, but not fight against rejection || || 0 (borderline) || Hovering between +1 and −1 || Not sure yet about the severity of the weaknesses / the threshold for ESA || || −1 (weak reject) || Significant weaknesses, but nothing fatal || I would not support this paper, but not fight against acceptance || || −2 (reject) || Bad fit or major weaknesses || I would oppose this paper and fight against acceptance || |
{{{#!html <table style="color: darkblue"> <tr><th>Score</th><th>Verdict</th><th>Behavior during discussion</th></tr> <tr><td>+2 (accept)</td><td>Good fit and no major weaknesses</td><td>I would champion this paper and fight against rejection</td></tr> <tr><td>+1 (weak accept)</td><td>Significant weaknesses, but nothing fatal</td><td>I would support this paper, but not fight against rejection</td></tr> <tr><td> 0 (borderline)</td><td>Hovering between +1 and −1</td><td>Not sure yet about the severity of the weaknesses / the threshold for ESA</td></tr> <tr><td>−1 (weak reject)</td><td>Significant weaknesses, but nothing fatal</td><td>I would not support this paper, but not fight against acceptance</td></tr> <tr><td>−2 (reject)</td><td>Bad fit or major weaknesses</td><td>I would oppose this paper and fight against acceptance</td></tr></table> }}} |
| Line 38: | Line 47: |
| Remark 1: Some conference also have +3 (strong accept) and −3 (strong reject). Experience shows that they are of little use for deciding on the set of accepted papers for a moderate number of submissions, as in ESA Track B (around 50). | Remark 1: Some conferences also have +3 (strong accept) and −3 (strong reject). Experience shows that they are of little use for deciding on the set of accepted papers for a moderate number of submissions, as in ESA Track B (around 50). |
This page is about the reviews for the ESA 2018 Track B PCs. We expect around 50 submissions. Each submission should receive 3 reviews (more reviews are possible, but this is the exception). Each of the two PCs has exactly 12 members. This is an expected load of around 12 submissions per PC member.
Contents
Sub-reviewers
We recommend that you review the papers yourself, but you may ask sub-reviewers for some of the submissions if you prefer to do so. In any case, you should familiarize yourself with the paper and the review, so that you can have a competent discussion with the other PC members. The discussion phase is an essential part of the reviewing process.
Review Text
Each review should provide the following information:
1. A short summary of the main contribution(s) of the submission in the words of the reviewer
2. An itemized list of the strength and weaknesses of the submission
2.1 The strengths should be numbered (S1), (S2), ...
2.2 The weaknesses should be numbered (W1), (W2), ...
Each review also can provide the following information (the authors will thank you):
3. More detailed explanations of the strengths and weaknesses
4. Comments to the authors for improving the paper
You can change the text of your review in the discussion phase. However, the discussion phase (and the whole reviewing process) will not work, if your initial review is not substantial.
Review Score
Each review should provide one of the following scores. Just like the text of your reviews, these scores are important for the discussion phase. You can change your scores during the discussion phase, but it will greatly help the efficiency and quality of the process, if you hit the "right" score for a paper already in your initial review.
| Score | Verdict | Behavior during discussion |
|---|---|---|
| +2 (accept) | Good fit and no major weaknesses | I would champion this paper and fight against rejection |
| +1 (weak accept) | Significant weaknesses, but nothing fatal | I would support this paper, but not fight against rejection |
| 0 (borderline) | Hovering between +1 and −1 | Not sure yet about the severity of the weaknesses / the threshold for ESA |
| −1 (weak reject) | Significant weaknesses, but nothing fatal | I would not support this paper, but not fight against acceptance |
| −2 (reject) | Bad fit or major weaknesses | I would oppose this paper and fight against acceptance |
Remark 1: Some conferences also have +3 (strong accept) and −3 (strong reject). Experience shows that they are of little use for deciding on the set of accepted papers for a moderate number of submissions, as in ESA Track B (around 50).
Remark 2: Some conferences disallow the borderline score of 0, to enforce a clear opininion on the reviewer. In the discussion phase, we indeed ask revievers to commit to one of the other scores. But for the first review, we think it makes sense to allow this score, because it reflects one of the typical sentiments about a paper at this stage of the reviewing process, as expressed by the hovering between +1 and −1 in the table above.
Remark 2: Reviewers might not be fully aware yet of their behavior during the discussion phase for various reasons (for example: not sure about some aspects of the paper, not sure about the nature of the threshold for ESA Track B, general inexperience in reviewing). This can make choosing the right score difficult. But this is exactly one of the tasks of the discussion phase (described below): to bring the final scores (and reviews) closer to what they were supposed to reflect.