Project Lead: Young-Jun Lee (Contact: [email protected])

If you did not select this track, you do not need to complete it. However, if you would like to participate even though you did not indicate your intention to do so on the Google Form, please let us know via the Slack channel.

👋 Welcome SciVeri-Bench Task Proposal Track!

This page provides step-by-step guidance on how to submit your proposed task and interact with the reviewer and manager. It also explains the overall workflow of our scientist-in-the-loop harness. If you are unfamiliar with the “scientist-in-the-loop” harness, please read the overview here before participating in this track.

Prerequisite

• Task proposers and reviewers should have an OpenReview profile. We will invite you as Task Proposer or Task Reviewer in our venue through your institution email. If you did not receive this email, please let us know in Slack.
• Task proposer should nominate a reviewer on their proposed task though https://forms.gle/WHYN6Dvdhjr5buco7

Eligibility for Task Proposers and Task Reviewers

• Task proposers should have published 1–2 papers in Nature-family journals or at top-tier conferences within the past two years.
• Nominated task reviewers should have published 1–2 papers in Nature-family journals or at top-tier conferences. AI conferences such as NeurIPS, ICLR, ICML, and CVPR are acceptable. However, please note that in Phase 2, tasks will be evaluated by external reviewers who have published in Nature-family journals.
• Nominated task reviewers should have completed 1–2 or more peer reviews for Nature-family journals or top-tier conferences.

Task Submission Deadline

• There is no fixed submission deadline. The goal is to develop high-quality, open-ended, and novel tasks, along with strong verification materials, through active and ongoing interaction between task proposers and task reviewers.

🚨 Exception: Contributors to SciVeri-Bench v0.1 must complete their tasks within 2–3 weeks. I will notify the relevant contributors on Slack.

🍀 How to Contribute Your Task

Our task collection procedure is based on our proposed “scientist-in-the-loop” harness. Please follow the step-by-step instructions below.

Step 0: Propose an Open-Ended and Novel Task

When proposing a scientific task, ensure that it is open-ended and novel by considering the following minimum requirements:

• The task must be open-ended, with outcomes that cannot be adequately evaluated using a single scalar score
• There must be no reference solution available before the experiment is conducted (i.e., because the task is open-ended, you do not know in advance what will constitute a good or poor discovery)