Your AI app is a new attack surface. Here is the open source kit to secure it
The model cannot tell your instructions from instructions hidden in the content it reads. That is the whole problem, and the whole opportunity.
For twenty years, application security rested on one line: code is the instruction a system runs, and data is the untrusted content it only reads. Firewalls and input validation police that line.
Large language models erase it. To an LLM, the prompt you wrote and the text it reads from a web page, an email or a PDF are the same thing: language, to be interpreted and acted on. That single property is why AI security is not just old application security with a chatbot on top. It is a new discipline. The good news is that the open source community has already built a mature toolkit for it, and you do not need a vendor licence to start.
Here is the lifecycle I use.
Map the risk. The OWASP LLM Top 10 gives an engineer and a CISO the same words for the same problem. Top of that list, for good reason, is prompt injection: the SQL injection of the AI era. The dangerous form is indirect, where the payload hides inside content the model will later read. When that model drives a retrieval pipeline or an agent with real tools, an instruction hidden in a document becomes an instruction your system obeys.
Attack it first. Because an LLM’s output is not deterministic, you cannot prove it safe by reading the code. You attack it. garak runs a broad library of probes, a scanner for models. promptfoo runs like a test suite and fails the build on regressions. PyRIT is for deliberate, multi-turn campaigns against a high value system.
Defend at runtime. Guardrails wrap the model in production: they scan inputs, filter outputs, and gate the actions an agent can take. LLM Guard for input and output scanning, Llama Guard as a safety classifier, NeMo Guardrails for behavioural rails, Rebuff for prompt injection detection. No single one is a silver bullet, and that is the point. You layer them.
Evaluate and govern. A one off test is a snapshot, not assurance. Models drift and prompts change. Giskard scans for hallucination, injection and bias and gives you a report. promptfoo in CI turns a safety regression into a failing check on the commit that caused it.
Where to start, without trying to do everything at once: inventory every LLM powered feature and rate each against the OWASP Top 10, point garak or promptfoo at your highest risk feature so the findings write your business case, then add one guardrail and wire the checks into CI so safety cannot regress silently.
The threats are new. The discipline is familiar: know your risks, test against them, control what you can, and prove it continuously.
I wrote the full version, with every tool and where it fits, here: AI security field guide: securing the LLM stack with open source.
Saleem Yousaf is a Cloud and Cyber Security Architect. He is the Founder of BreachForge, a breach and attack simulation platform, and Director at Cyber Spartans Ltd. He works across AWS and Azure, securing UK government, critical national infrastructure, and global enterprise. Website: saleemyousaf.co.uk. Consultancy: cyberspartans.co.uk.
