Survey Under Siege:

Exploring AI, Bot, and Human Interference in Online Surveys

As online surveys become more common in higher education research, so do new threats to data integrity.

AI-generated answers, bot activity, and human interference can quietly distort research outcomes. This whitepaper from the National Disability Center examines how fraudulent participation affected the launch of the National Report on Disabled College Student Experiences. It offers a transparent, step-by-step breakdown of the Center’s multi-layered response strategy and practical guidance for researchers working in digital spaces.

In spring 2024, the College Accessibility Measure (CAM) survey collected more than 2,200 online responses, but researchers flagged many as potentially fraudulent. Using tools like reCAPTCHA scoring, metadata tracking, and open-text response analysis, the Center’s research team developed a methodical screening and data-cleaning process to safeguard results.

The report explores these findings and offers a framework to help other research teams design stronger surveys.

What’s Inside:

The number “70%” is displayed in large, bold, navy blue type with subtle shadowing. Behind the “7,” a circular green element is filled with diagonal stripes in alternating olive and dark green tones. The design combines bold typography with a visual texture to emphasize a key statistic.
Two side-by-side line graphs show the distribution of reCAPTCHA scores. Both have white backgrounds and olive green wave-like density curves. The X-axes are labeled “ReCAPTCHA Score” ranging from 0.0 to 1.0, and the Y-axes are labeled “Density.” The left graph shows a more irregular pattern with multiple peaks, while the right graph has a smoother curve rising toward the higher end of the score range.
A centered white triangle with rounded edges contains a navy blue exclamation point on a light green background. The triangle is outlined by a bold olive green circle with a thick white inner ring. The graphic is flat and minimalistic, signaling a warning or alert.

Key Findings

Top Recommendations for Researchers

Online research is a powerful tool, but it comes with new responsibilities. Findings in this whitepaper point to important steps researchers can take to protect the quality of their data and reduce the risk of survey fraud. These recommendations offer practical guidance for designing, managing, and reviewing online surveys in a way that prioritizes trust and transparency.

Identify

Detect back-end indicators of suspected bot behavior in online surveys.

Evaluate

Assess the effectiveness of Google’s “reCAPTCHA,” a non-human screening extension to online programs which can be implemented to mitigate bot intrusions during survey data collection.

Analyze

Review the data cleaning process and the proportion of suspicious responses flagged using each identification strategy.

Discuss

Talk through the decision-making process for managing survey data and determining next steps for future surveys.

Losing most of a dataset to bots is the kind of lesson you only need once. This attack nearly hijacked our research story, but instead we used it to create recommendations for preserving community-engaged research while getting the most out of online data collection innovations.

Ryan A. Mata, PhD