AI Meetups are series of seminars organized by REAL AI at UCSB, "a consortium of researchers building Artificial Scientific Intelligence (ASI)." The topic of the meetup described below stems from work undertaken during Summer 2024 by ACTION AI Interns at UCSB. Dr. Yigitcan Kaya mentored the students during the research experience.
When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Plugins
Yigitcan Kaya
SecLab
UC Santa Barbara
May 30 @ 2:30 pm PT
Abstract:
Prompt injections have emerged as a critical vulnerability against large language models (LLM), particularly in advanced LLM systems, such as computer-use agents, which prior research has studied in depth. However, we observe a gap between research and practice: while simpler LLM applications (such as customer service chatbots) are ubiquitous on the web, their exposure to such vulnerabilities remains largely unexplored. These applications often depend on third-party chatbot plugins, which provide website builders (typically non-experts) with intuitive tools for customizing chatbot behavior and accessing commercial LLM APIs.
To bridge this gap, we present the first large-scale study of 17 third-party chatbot plugins used by over 10,000 public websites, revealing previously unknown prompt injection risks in practice. First, 8 of these plugins (used by 8,000 websites) fail to enforce the integrity of the conversation history transmitted in network requests between the website visitor and the chatbot. This oversight enables direct prompt injection, where adversaries can forge conversation histories (including fake system messages) to manipulate chatbots, significantly boosting their ability to elicit unintended behavior (e.g., code generation). Second, all plugins offer tools, such as web-scraping, to augment the chatbot's context with website-specific content. However, these tools do not distinguish the website's trusted content (e.g., product descriptions) from its unverified, third-party content (e.g., customer reviews), introducing a risk of indirect prompt injection. In practice, we found that 12% of e-commerce websites expose their chatbots to third-party content. We evaluate both vulnerabilities through real-world measurements and controlled experiments, analyzing contributing factors such as system prompt design. Our findings reveal a gap between research settings in LLM security and the deployment of LLMs by non-expert users in an expanding web ecosystem.
Biography:
Yigitcan Kaya is an Intelligence Community Postdoctoral Fellow at UC Santa Barbara, working with Giovanni Vigna and Chris Kruegel. He received his PhD in Computer Science from the University of Maryland College Park, advised by Tudor Dumitras. His research interests span the areas of machine learning for security and adversarial machine learning, with an emphasis on bridging between these two fields. In the past, he identified a common pathology of deep neural networks and coined the term neural network overthinking, developed realistic threat models against ML systems, such as inconspicuous poisoning attacks, and studied the feasibility of practical mechanisms to make ML models more private. He has published his work at top ML venues, such as ICML and ICLR, which garnered press interest from popular outlets such as VentureBeat and MIT Tech Review. During his PhD, he interned twice as an applied scientist at AWS and contributed to two patent applications on ML robustness. Nowadays, he is applying his expertise to make ML models more secure and robust in critical real-world applications, such as malware detection or customer service chatbots. He usually just goes by Can, pronounced like, "John."