Once a participant has completed your study, it is up to you as the researcher to review their data and decide if that participant should be approved and paid, rejected and penalised, or returned.

At Prolific we have very specific criteria for the acceptance and rejection of participant submissions.

This article will tell you how to make the decision on which participants to approve and which to reject, how to action these decisions within Prolific, and how to adjust previously made decisions on submissions.

How do I decide who to approve, reject, or ask to return?

You have a maximum of 21 days to review submissions. If you haven't reviewed submissions after 21 days, our system will automatically approve all submissions that are awaiting review.

After you have checked your participants’ submissions, you should pay them as soon as possible. Ideally, you would reward them within 24-48 hours after they have completed your study.

Although we ask that you action submissions as soon as possible, please do not rush this process. Ensure that you are happy with your decision and have read our guidelines before taking action.

Who should I approve?

Our rule of thumb

Any participant who has completed your study and has provided you with data should be approved and paid unless they meet any of the rejection criteria listed below.

Who should I reject?

On Prolific, you have the option to reject participant submissions. However, please think carefully about whether the rejection is justified, and try to keep rejections to a minimum. Participants are penalized for rejections, and if they receive too many, they will be removed from the platform entirely. Please review all submissions holistically and remember that participants have spent time and effort taking your study.

Valid reasons for rejection ✅

  • The participant completed your study exceptionally fast
    • By 'exceptionally fast' we mean participants who are statistical outliers (3 standard deviations below the mean)
    • This is the only criteria under which speed of submission can be used to reject.
    • Participants can't be rejected for not spending enough time on a specific page within your study.
  • The participant skipped crucial questions, i.e. questions that are critical to answering your specific research question.
    • These questions must not have been shown as optional for the participants
    • Please note that demographic questions do not fall under 'crucial questions' because participants have the right to withhold such sensitive information (according to guidelines by the British Psychological Society)
  • The participant failed one (for studies under 5 minutes) or multiple (for studies 5 minutes or longer) fair attention checks
  • The participant objectively demonstrated clear low-effort throughout the experiment, even when the required level of engagement was clearly specified
    • For example, the participant was asked to write a paragraph but only put a few words or gibberish

Invalid reasons for rejection ❌

  • A participant was screened out within your study (e.g. for failing to meet demographic criteria)
    • While we do allow researchers to validate their prescreeners applied on Prolific by asking the prescreening question(s) again in your survey (worded exactly as appears on Prolific), any participant that fails this validation should be asked to return their submission rather than being rejected
    • We do not allow within-study screening for criteria not applied in your Prolific prescreening
  • A participant completed your study 'too fast' (except where they are a statistical outlier - see our valid reasons for rejection on the left) or 'too slowly'
    • Participants' completion times vary around an average. They have variable reading speeds and variable knowledge of keyboard shortcuts and other browser tools
    • Please use more reliable measures to check for attention rather than completion time
  • The participant failed an internal measure (e.g. the participant's accuracy on a certain measure was not above an arbitrary threshold)
    • You are of course more than welcome to exclude these participants from your subsequent analysis; however, you'll still need to pay for their responses even if you can't use their data
    • If you feel that your specific case merits a rejection, then please get in touch with our Researcher Support Team using the button at the bottom of this page
  • There was a technical error in my study that caused the data not to be saved but wasn't the participant's fault
    • Unfortunately, technical errors sometimes occur, but if it was not the participant's fault, then this is not a valid reason for rejection
    • Please get in touch using the button at the bottom of this article if this happens and we'll help as best as we can

Please also remember that:

  • If you are concerned that a rejected submission might have been made by a bot, then please let us know immediately
  • Prolific reserves the right to overturn invalid rejections in certain circumstances
  • Prolific reserves the right to overturn invalid rejections in certain circumstances

Who should I ask to return their submission?

You may request that a participant returns their submission if:

  • They only completed part of your study (and didn't finish it)
  • They encountered technical problems (in which case, please consider awarding partial payments)
  • They withdrew their consent (in which case please also delete any data that you have from that participant in your external software)
  • You feel they should be given the opportunity to return their submission for another reason rather than be penalized with a rejection. Remember that rejections should only be used in cases where a participant has been clearly negligent.

Participants are able to voluntarily return submissions and will often do so upon request if it's demonstrated that a mistake was made. Researchers cannot return submissions themselves.

You can ask a participant to return their awaiting review submission by clicking the blue and white arrow next to their submission:

After you select a reason for requesting the return, a message will automatically be sent to the participant asking them to return their submission.

Please note that this button will only be activated if your study status is ‘active’ or ‘awaiting review’. It’s not possible to use this button on ‘paused’ studies. Instead, you can message the participants directly and ask them to return their submission.

The participant can return submissions from their 'Submissions page', by selecting the red circular arrow to 'Return + cancel reward':

Please allow the participant at least 7 days to respond to your request. If the participant does not respond to your request within 7 days then please contact the Researcher Support Team using the button at the bottom of this article to request the returns.

How do I make an approval or a rejection from the submissions page?

Approvals

1. Manual approval

You can manually approve each submission in your list by clicking the green check mark on the right-hand side of the page:

Alternatively, you can select the check box to the left of the participant's ID and then click 'Approve selected' at the bottom of the list. This can be done for multiple participants at the same time.

2. Approve in bulk

You can approve submissions from specific participant IDs by using the 'Approve in bulk' feature. To do this select 'Bulk actions' > 'Approve in bulk':

On the resulting pop-up, simply enter the IDs of the participants you want to approve separated by a comma, and then click 'Approve':

Rejections

To reject a participants submission simply click the red cross on the right-hand side of the page:

You will then be asked to explain why the rejection is being made:

We have provided a number of common rejection reasons in the dropdown menu for you to choose from, or you are welcome to submit your own reason by selecting 'Other' and providing detail in the message. Once you select a reason a preformatted message will appear that will be sent to the participant.

Please note that there is no option to bulk reject submissions, each rejection must be made one at a time.

FAQs

I've reached my rejection limit - what should I do?

All studies on Prolific have a limit on the number of submissions researchers can reject by default. This is because a rejection negatively impacts a participant's Prolific Score, which could prevent them from taking part in further studies on our platform.

We typically find that the automatic rejection limit is sufficient, but if you reach the maximum number of rejections for your study and require more, you will need to get in touch with the Support Team using the button at the bottom of this article to request a review of this limit.

It is important to remember that we are a platform that serves both researchers and participants, so we will need to make sure that your rejection reasons are fair and in line with the above guidelines. Therefore, when you submit your request, we will ask for the following information:

  • Name of study (including internal name, if relevant)
  • Study questions, the related concerning responses, and the relevant participant IDs (any format is fine)
  • Any additional information that might be useful to assist our review (e.g. context about your study). If you are submitting a rejection limit increase request due to failed attention check items, please include a screenshot of the attention check(s) in question.

Please note, we won't be able to increase your rejection limit until we have this information. By providing this, you are helping us to make sure everything is fair for everyone on Prolific!

I've mistakenly rejected a submission instead of approving it or asking the participant to return it. What should I do?

1. 'Unreject' the submission'

2. The submission will then show as 'awaiting review.'

3. Either approve the submission, or send the participant a message to let them know they can 'return' it (when a submission is returned, that spot opens back up in your study so that another participant can take part and you don't miss out on getting all the data you need)

4. If you don't hear back from participants after 7 days, our Support team can assist you with returning submissions.

Please note:

  • You will only be able to reverse a rejection if you have enough funds in your account to approve the submission. If necessary, you can top-up with additional funds, then request a refund.
  • If you 'unreject' a submission after 21 days, the submission status would return to 'awaiting review'. It would then be automatically approved within 24 hours, or you can approve it manually before then.
  • If you require the participant to return their submission after 21 days, please contact us using the button below for assistance. This is to avoid the submission being automatically approved before the participant has had a chance to return it.

Need further help?
Click here to contact us