Incidents tagged LLM Security

Prompt injection is a critical vulnerability in Large Language Models (LLMs) that manipulates their natural language processing to override system instructions and safety filters using crafted malicious prompts. This exploit can lead to significant impacts such as data exfiltration, including the disclosure of sensitive files like `/etc/passwd`, and enables the AI system to perform unauthorized actions.

The article details critical security challenges associated with cloud-hosted Large Language Models (LLMs), including prompt injection, adversarial exploits, model jailbreaks, sensitive data leakage, and misconfigurations. These vulnerabilities, stemming from lack of visibility and ungoverned AI behavior, necessitate proactive discovery and risk management to safeguard AI workloads.

Researchers discovered a reflected Cross-Site Scripting (XSS) vulnerability in Yellow.ai's chatbot, which could be tricked into generating malicious HTML/JavaScript code. This flaw enabled attackers to steal support agent session cookies, potentially leading to account hijacking and data exfiltration from customer support platforms.