Call For Artifacts
Last Update : [5 October, 2023]
All authors of accepted ACM CCS 2023 papers are encouraged to submit artifacts for Artifact Evaluation (AE). Each submitted artifact will be reviewed by the Artifact Evaluation Committee (AEC). Please carefully check the Artifact Evaluation Information below, this is the first time an artifact evaluation is organized at ACM CCS. Should you have any questions or concerns, you can reach the AEC chairs at ccs2023-artefactevaluation@cispa.de.
Evaluation Submission
Information
Quick Overview:
24 October, 2023
Submission Deadline for Artifact Evaluation
1 - 17 November, 2023
Author Discussion Period
22 November, 2023
Artifacts Decisions Announced
This year, ACM CCS is introducing an optional evaluation of research artifacts that support a paper accepted at the conference. A research artifact can take a variety of forms, including software, hardware, evaluation data and documentation, raw measurement data, raw survey results, mechanized proofs, models, test suites, benchmarks, etc. In some cases, the quality of these artifacts is as important as that of the document itself. To emphasize the importance of such artifacts, the benefits to the authors and the community as a whole, and promote the availability and reproducibility of experimental results, ACM CCS now introduces an optional AE process. This process closely follows the artifact evaluation process organized at USENIX Security and similar conferences. The Artifact Evaluation Committee (AEC) will review each submitted artifact and grant badges that report on the results of the AE process.
Process
To maintain a wall of separation between paper review and the artifacts, authors will be given the option to submit their artifacts only after their papers have been accepted for publication at ACM CCS. Now that the reviewing process for ACM CCS is finalized, the AE will take place and authors of accepted papers can submit an artifact for review. By the artifact submission deadline, authors can submit their artifacts, Artifact Appendix, and other supporting information via the submission form, which will be available starting October 20.
Authors define the contents of their artifact submission. For example, software, hardware, data sets, survey results, test suites, mechanized (but not paper) proofs, access to special hardware, and so on. Authors choose which badges their artifact should be evaluated towards, we will use the same process as USENIX Security. In general, good artifacts are expected to be: consistent with the paper, as complete as possible, well documented, and easy to (re)use. The AEC will read the paper and then judge if the artifact meets the criteria for each of the requested badges.
Each artifact submission will be reviewed by at least two AEC members. The review is single-blind and strictly confidential. All AEC members are instructed to treat the artifact confidential during or after completing evaluation, nor retain any part of it after evaluation. Thus, you are free to include models, data files, proprietary binaries, exploits under embargo, etc., in your artifact. Since we anticipate small glitches with installation and use, reviewers may communicate with authors for the period November 1-17 to help resolve glitches while preserving reviewer anonymity. Please make sure that at least one of the authors is reachable to answer questions in a timely manner.
The AEC will then complete its evaluation and notify the authors of the outcome several days before ACM CCS takes place.
Acknowledgements
The AE process at ACM CSS'23 is closely following and inspired by multiple other conferences, such as USENIX Security, OSDI, EuroSys, and several other systems conferences. See artifact-eval.org for the origins of the AE process.
Artifact Evaluation Committee
Recruiting Now
We are in the process of recruiting the AEC. Please (self-)nominate in case you are interested.
