{"id":3117,"date":"2020-12-21T15:28:13","date_gmt":"2020-12-21T20:28:13","guid":{"rendered":"https:\/\/qomex2021.itec.aau.at\/?page_id=3117"},"modified":"2021-02-16T10:05:20","modified_gmt":"2021-02-16T15:05:20","slug":"reviewer-instructions","status":"publish","type":"page","link":"https:\/\/qomex2021.itec.aau.at\/reviewer-instructions\/","title":{"rendered":"Reviewer Instructions"},"content":{"rendered":"[vc_row][vc_column css=&#8221;.vc_custom_1608583285669{margin-bottom: 20px !important;padding-bottom: 20px !important;}&#8221;][vc_column_text]\n<h1><b>Guidelines for the Review Process<\/b><\/h1>\n<h2><span style=\"font-weight: 400\">Overall Process<\/span><\/h2>\n<p><span style=\"font-weight: 400\">Each submitted paper will be reviewed by at least three experts in the field. Submissions will be judged on the following criteria.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Suitability: Does the contribution fit to the scope of QoMEX and in particular QoE, UX, immersive experiences, as well as to the list of topics of interest?<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Originality: What new ideas, approaches and\/or results are introduced?<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Significance\/contribution: Is the paper significant for QoE and UX research? What is the benefit that others can gain from the contribution?<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Validity of the work: How confidently can researchers and practitioners use the results? Is the paper technically\/methodologically sound? E.g. measurement setup, design of studies, conduction of simulation studies, statistical analysis<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Related work: Is state-of-the-art relevant to the described work sufficiently addressed? Does the paper refer to peer-reviewed publications?<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Thoroughness: Is the research carried out in a thorough way? Are the discussions and conclusions detailed enough?<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Replicability: Are the results and conclusions clearly described and reproducible? Are public data sets or open source code available?<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Presentation clarity: Is the structure of the text and the presentation of the results clear?<\/span><\/li>\n<\/ul>\n<p><b>Submitted papers of the general and special session tracks will undergo a double-blind review process. <\/b><span style=\"font-weight: 400\">Only the dataset &amp; tools track will follow a single-blind review process as authors need to include a link to the dataset or tool which will be evaluated by the reviewers.<\/span><\/p>\n<p><span style=\"font-weight: 400\">Each paper is assigned a <\/span><b>Direct Responsible Individual (DRI)<\/b><span style=\"font-weight: 400\"> acting who is responsible for a certain paper. The DRI is selected by the TPC chairs and may be a TPC chair, special session organizer, or selected TPC member. The responsibility of the DRI is as follows.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Initiate discussions in EDAS after the review submission deadline about diverging reviews. Reviewers can argue for or against the acceptance of a paper.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">If necessary, provide a meta review summarizing the reviews and the discussions among reviewers.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">If accepted papers need to be improved, shepherding of the papers is initiated and led by the DRI.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">For <\/span><b>short and full papers<\/b><span style=\"font-weight: 400\">, there will be a <\/span><b>paper bidding <\/b><span style=\"font-weight: 400\">phase in order to ensure a good match between an individual reviewer\u2019s expertise\/interests and her\/his review assignments. The online TPC meeting for full papers will take place on March 23, 2021 and is organized by the TPC Chairs. Details will be provided later. All TPC Members are invited to participate in the TPC meeting.<\/span><\/p>\n<p><span style=\"font-weight: 400\">For <\/span><b>short papers<\/b><span style=\"font-weight: 400\">, an online TPC meeting will be hosted on May 11, 2021. All TPC Members are invited to participate in that TPC meeting.<\/span><\/p>\n<p><span style=\"font-weight: 400\">The TPC members are invited by the TPC chairs to cover all topics as mentioned in the Call for Papers. In particular, the domains of QoE and UX are reflected in the technical program committee.<\/span><\/p>\n<h2><span style=\"font-weight: 400\">For TPC Members and Reviewers<\/span><\/h2>\n<p><span style=\"font-weight: 400\">Important information related to the paper assignment and review process:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">TPC members pick topics (areas) of interest, designating them as \u201cof interest\u201d, \u201cneutral\u201d, \u201cno interest\u201d. These topics are used to limit the number of papers TPC members can \u201cclaim\u201d in the paper bidding phase for possible review, as only papers having at least one topic \u201cof interest\u201d and no topics \u201cnot of interest\u201d are included.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Reviews are assigned to the reviewers, manually, taking into account topic preferences and conflicts of interest.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Submissions will be judged on suitability, originality, significance\/contribution, validity of the work, related work, thoroughness, replicability, presentation clarity.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">In the TPC meeting and during the discussion phase, a review with a high score but with a short unsubstantiated review will be discounted.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">High quality reviews are of utmost importance to QoMEX 2021 and critical for the advancement of the field. For this reason, please keep the following review guidelines in mind:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Avoid being overly negative: try to be critical yet constructive in your reviews, as any research paper has its advantages and drawbacks.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Please write a review with the quality that you would expect for your own papers. Do not submit very short, unsubstantiated reviews.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Keep a fair and open-minded approach, favoring selection of breakthrough papers that can open new areas and new research directions.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Be particularly open-minded when reviewing Special Session papers as they might address different scientific communities.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">If any of the papers are reviewed with the help of graduate students, we expect a TPC member to provide sufficient guidance and oversight, and personally submit the reviews to EDAS.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Use the \u201ccomments to the TPC chairs\u201d field to share confidential remarks or concerns you may have regarding a submission.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Finally, we recognize the efforts and time that TPC members and reviewers contribute to reviewing assigned papers. A best reviewer recognition award will be handed out during the conference.<\/span><\/li>\n<\/ul>\n<h2><span style=\"font-weight: 400\">Full papers<\/span><\/h2>\n<p><span style=\"font-weight: 400\">Submitted full papers of the general and special session tracks will undergo a double-blind review process. Submitted full papers must not exceed six (6) pages, including figures, tables and references. Please see the <\/span><a href=\"https:\/\/qomex2021.itec.aau.at\/papersubmission\/\"><span style=\"font-weight: 400\">paper submission instructions<\/span><\/a><span style=\"font-weight: 400\">.<\/span><\/p>\n<p><span style=\"font-weight: 400\">Reviewers should consider the following in their evaluation: suitability, originality, significance\/contribution, validity of the work, related work, thoroughness, replicability, presentation clarity. Please note that we also invite to submit position papers as full papers which describe a novel or bold idea; in that case experimental results may not be existing, but the paper should be nevertheless scientifically valid and convincing.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Full paper Submission: 16.2.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Paper Bidding: 17.2. \u2013\u00a0 21.2.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Review phase: 23.2. \u2013 15.3.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Reviewer discussion phase: until 21.3.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">TPC Meeting: 23.3.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Author notification: 28.3.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Camera-ready: 11.4.2021<\/span><\/li>\n<\/ul>\n<h2><span style=\"font-weight: 400\">Short papers<\/span><\/h2>\n<p><span style=\"font-weight: 400\">Submitted short papers of the general and special session tracks will undergo a double-blind review process. Submitted papers short must not exceed three (3) pages, including figures, tables and references. Short paper submissions must describe original and unpublished work. Please see the <\/span><a href=\"https:\/\/qomex2021.itec.aau.at\/papersubmission\/\"><span style=\"font-weight: 400\">paper submission instructions<\/span><\/a><span style=\"font-weight: 400\">.<\/span><\/p>\n<p><span style=\"font-weight: 400\">Reviewers should consider the following in their evaluation: suitability, originality, significance\/contribution, validity of the work, related work, thoroughness, replicability, presentation clarity. While a short paper is not a shortened long paper, the characteristics of short papers may include one or more of the following:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Small, but focused contribution<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Work-in-progress or late-breaking results<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">A position paper or contribution that is pushing\/challenging existing boundaries<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">A counter-example to an existing technique that helps to understand its limitations<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Extensions or evaluations of existing methods or explorations of new methods<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">An interesting application nugget<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">The timeline for the short papers is as follows.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Short paper submission: 13.4.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Paper Bidding: 14.4. \u2013 18.4.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Review phase: 20.4. \u2013 4.5.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Reviewer discussion phase: until 9.5.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Short paper TPC meeting:\u00a0 11.5.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Short paper notification: 13.5.2021<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Short paper camera-ready: 20.5.2021<\/span><\/li>\n<\/ul>\n<h2><span style=\"font-weight: 400\">Dataset &amp; Tools track<\/span><\/h2>\n<p><span style=\"font-weight: 400\">Full and short paper submissions to the dataset &amp; tools track are single-blind. The authors need to provide a link to the dataset or tool which will be elaborated by the reviewers as part of the submission. The data and tool must be made available to the reviewers upon submission and should eventually be published upon acceptance.<\/span><\/p>\n<p><span style=\"font-weight: 400\">Reviewers should consider the specific features of papers submitted to the dataset &amp; tools track and evaluate the submissions as follows.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The dataset or tool must be available online (e.g. on Github).<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Dataset papers must contain all relevant information and describe its content and origin, such that readers can fully understand the experiment environment and execution. Additional information may be provided online in the description of the dataset.<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">its content and origin;<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">the methodology and tools used to obtain it;<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">the schema, structure, or layout to store it;<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">usage scenarios or open analysis challenges for it;<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">relationships to other existing datasets;<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">its current limitations.<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Tool papers must present either a new tool, a new tool component, or novel extensions to an existing tool. Tool papers should provide a short description of the theoretical foundations, after which emphasis should be on the design and implementation concerns (incl. software architecture and core data structures), closed by the description of experience with realistic case studies.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Dataset and Tool papers must clearly describe the novelty and significance and how this dataset\/tool advances QoE or UX research.<\/span><\/li>\n<\/ul>\n<p><b>References<\/b><\/p>\n<p><span style=\"font-weight: 400\">Papers submitted to QoMEX 2021 should refer to peer-reviewed publications. References to non-peer-reviewed publications (including public repositories such as arXiv, Preprints, software, and personal communications) should only be made if there is no peer-reviewed publication available, should be kept to a minimum, and should appear as footnotes in the text (i.e., not in the references list). Reviewers should comment on the suitability of references.<\/span>[\/vc_column_text][\/vc_column][\/vc_row]\n","protected":false},"excerpt":{"rendered":"<p>[vc_row][vc_column css=&#8221;.vc_custom_1608583285669{margin-bottom: 20px !important;padding-bottom: 20px !important;}&#8221;][vc_column_text] Guidelines for the Review Process Overall Process Each submitted paper will be reviewed by at least three experts in the field. Submissions will be judged on the following criteria. Suitability: Does the contribution fit to the scope of QoMEX and in particular QoE, UX, immersive experiences, as well as<\/p>\n","protected":false},"author":33,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"acf":[],"_links":{"self":[{"href":"https:\/\/qomex2021.itec.aau.at\/wp-json\/wp\/v2\/pages\/3117"}],"collection":[{"href":"https:\/\/qomex2021.itec.aau.at\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/qomex2021.itec.aau.at\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/qomex2021.itec.aau.at\/wp-json\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/qomex2021.itec.aau.at\/wp-json\/wp\/v2\/comments?post=3117"}],"version-history":[{"count":6,"href":"https:\/\/qomex2021.itec.aau.at\/wp-json\/wp\/v2\/pages\/3117\/revisions"}],"predecessor-version":[{"id":3310,"href":"https:\/\/qomex2021.itec.aau.at\/wp-json\/wp\/v2\/pages\/3117\/revisions\/3310"}],"wp:attachment":[{"href":"https:\/\/qomex2021.itec.aau.at\/wp-json\/wp\/v2\/media?parent=3117"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}