Contents:
1. Overview
2. Step 1: Setting up your study on Prolific to record Prolific IDs via URL parameters
3. Step 2 - Setting up Qualtrics to run the authenticity checks
4. Step 3 - What you should see when you start collecting data
Helpful resources:
1. What are authenticity checks?
2. How to create an API token for authenticity checks
Overview
You can use both LLM and bot authenticity checks with Qualtrics studies.
LLM authenticity checks assess behavior during free-text responses and are only relevant for studies with open-ended written questions.
Bot authenticity checks assess whether a study has been completed by automated bots, agents, or scripts, rather than a human participant
To get started:
Set up the study on Prolific to include URL parameters
Select the type of checks you’d like to run, and generate the script
Add a Qualtrics-specific custom script to Header. This will track user interactions and send data through to our authenticity check model.
Detailed instructions for each step are provided below.
✋ Note: Before you proceed, JavaScript use in Qualtrics survey headers and questions is not available on the free account plan
Step 1: Setting up your study on Prolific to record Prolific IDs via URL parameters
1. Go to Qualtrics, add a block to your survey named ‘Prolific ID’, and create a 'Text entry' question. In the question text, include the following message:
2. Go to 'Survey Flow' and add an 'Embedded Data' element at the beginning of your study. Into the embedded data field, type 'PROLIFIC_PID' and make sure it says 'Value will be set from panel or URL'. Do the same for ‘STUDY_ID’ and ‘SESSION_ID’. For authenticity checks, it’s important that SESSION_ID is included. It allows us to link the authenticity check to the submission in your study.
3. Now go to the survey question that records the participant's Prolific ID. Right-click on the text-entry field and select 'Default Choices'.
4. When the window opens, click the blue drop-down arrow on the right. Then select 'Embedded data field'. Here, either enter 'PROLIFIC_PID', or click the drop-down arrow to the right of the text entry box and select 'PROLIFIC_PID', then click 'Insert'.
5. You should see a string called '${e://Field/PROLIFIC_PID}' populate the field. Now click 'Save', and you're done!
6. Finally, to record the Prolific ID, use the URL parameters option in Prolific. In the Recording Prolific IDs section on Prolific, enter your Qualtrics study URL in the 'What is the URL of your study?' box first, then select 'URL parameters' - this will automatically add the necessary GET parameters to your URL.
Your 'Study Link' box should now look something like this:
Step 2 - Setting up Qualtrics to run the authenticity checks
1. In the Authenticity checks section on Prolific, check the relevant box for the checks you want to use. You can use both LLM and bot checks together, or just one of them.
2. Click "Copy"
3. Copy and paste the generated script into the Qualtrics Header. To find this:
On Qualtrics, click "Survey" on the top toolbar
Click "Look and Feel" from the left side options
Click the "General" option
Scroll down to "Header" and paste your script using one of these methods:
Option 1: Simply click in the input field and paste the code script
Option 2: Click the "Edit" button to open a modal window, activate "Code Mode", then paste the script in the text area
Click save
4. Press the “Publish” button in Qualtrics
5. Go back to Prolific and click "Check script"
If the script is successfully detected, you will see a green "Connected" tick next to the Check button
If the script is not checked or not found, you will see a red error message next to the Check button. If you receive the "Script not found" error, double-check that you've correctly added the script to the Header of your Qualtrics survey and try again.
The script must show as "Connected" for you to publish the study
Step 3 - What you should see when you start collecting data
As your study goes live, you should see new columns called “Auth check: LLM” and "Auth check: Bots" appear on the submissions page.
As submissions come in, results appear in the authenticity check columns with the following indicators:
High (green): All responses have high authenticity
Mixed (orange):
For bot checks: this means some responses or behaviours were flagged. This can sometimes occur for legitimate reasons (for example, the use of accessibility tools). We recommend reviewing the submission and, where appropriate, contacting the participant to understand what may have caused this result
For LLM checks: this means some questions (e.g. 2 out of 4 responses) seemed inauthentic
Low (red): Low authenticity patterns detected consistently
For more detail on interpreting results, see What are authenticity checks.