Cyberpediaby Brain

Testing an internal AI assistant before opening it

Before an AI assistant is opened to employees, the team deliberately attacks it and measures its answers. Here is how, and what you can do.

On this page
  1. 1In 30 seconds
  2. 2What it is
  3. 3Straight answers
  4. 4How it works
  5. 5What you do
  6. 6Pitfalls
  7. 7Sources
  8. 8Cite this article

In 30 seconds

Where it happens
Before an AI assistant is opened to employees. NIST specifies that red teaming can take place before or after a model or system is made available to the public, and that its section focuses on the pre-deployment context[3].
Risks tested
NIST lists among the risks to test hijacked instructions (jailbreaking), leaks of sensitive information and false answers stated with confidence[3].
The reflex
Take part in the tests when the team offers, note and report any strange behavior, and do not give other people access to the assistant without validation from the responsible team.

What it is

Before an internal AI assistant is opened to employees, the team that runs it can deliberately attack it to find its weaknesses. This exercise is called red teaming. The term originally referred to systematic attacks carried out to test security vulnerabilities. With large language models, it has broadened to cover all kinds of probing, testing and attacks on AI systems[1].

Red teaming does not replace the systematic work of measuring and fixing[1]. The authors of a Microsoft lessons-learned paper, based on more than 100 generative AI products tested, list among their lessons that AI red teaming is not safety benchmarking[2].

Straight answers

What is red teaming an AI assistant?

NIST describes exercises often run in a controlled environment, in collaboration with developers, to spot undesirable behaviors or outputs, understand how they could occur and put the safeguards to the test[3]. It calls AI red teaming an evolving practice[3]. The OWASP AI Red Teaming Initiative states that it will provide a standardized methodology, with benchmarks, tools and frameworks[7].

Does red teaming replace evaluation by measurement?

No. According to Microsoft, red teaming is not a replacement for systematic measurement. It helps discover and identify harms, which then makes it possible to choose measurements that verify whether the fixes work[1]. Microsoft recommends, as a best practice, starting with a first round of manual red teaming before systematic measurements[1]. This first round serves to draw up the list of harms to measure and fix[1].

Who tests?

Microsoft advises assembling a varied group, in terms of experience, profiles and specialties relevant to the product's domain. Testers with an offensive mindset and security experience help to understand security risks. Ordinary users, who did not take part in development, offer a different view of the harms that users will encounter[1]. NIST also describes red teaming run by the general public, meaning users who are not necessarily AI or technical experts, and notes that this form may be more effective with large groups of testers[3].

Which risks should be tested before opening?

Three families of risks stand out.

  • Hijacked instructions. NIST uses the term jailbreaking for deliberately crafting prompts to get around the controls on responses[3]. OWASP places prompt injection at the top of its 2025 list of risks for language model applications[4]. Microsoft also cites extraction of the system prompt (meta prompt) among the points that security specialists can explore[1].
  • Data leaks. OWASP lists sensitive information disclosure and system prompt leakage[4]. NIST reports that, during attacks, models revealed sensitive public-domain information that was in their training data[3].
  • False answers. NIST calls this risk confabulation: a system produces and presents false or erroneous content with confidence[3]. The risk arises when the user believes this content because of the confident tone of the answer[3]. OWASP lists misinformation among its ten risks[4].

How it works

The approach follows a cycle: frame, test, measure, fix, repeat.

Frame. Microsoft invites teams to plan, before testing, who tests, what, how, and how results are recorded[1]. The OWASP manual proposes a method in eight phases: planning, reconnaissance, attack surface mapping, exploitation, persistence, impact assessment, reporting and remediation[6].

Test. According to Microsoft, you can test the base model with its safety system, the application itself, or both before and after the fixes[1]. The application is best tested through its interface, and as far as possible on the production interface, which most closely resembles real use[1]. Microsoft suggests starting with open-ended tests, then drawing up a list of harms with definitions and examples, which guides the following rounds[1].

Measure. Microsoft recommends testing versions of the product with and without protective measures, and adding systematic measurements after a first manual round[1]. After each round, a short report to stakeholders lists the main issues, gives access to the raw data and presents the plan for the next rounds[1].

Fix and repeat. Microsoft proposes checking that the test plan covers content filters, using prompt engineering techniques to reduce risk, then retesting to validate the improvements[1]. The authors of the Microsoft lessons-learned paper write that the work of securing AI systems will never be complete[2].

The NCSC (UK) guidelines, published on 27 November 2023, state that security must be "a core requirement, not just in the development phase, but throughout the life cycle of the system"[5]. The NIST Playbook, voluntary in use, organizes its suggestions around four functions: Govern, Map, Measure and Manage[8].

What you do

An employee without a technical background can act in four places.

Take part in the tests. If the team involves you in a test round, Microsoft indicates that it can be useful to give testers clear instructions: the goal of the round, the features to test and access, the types of issues targeted, the time to spend, the recording method and the person to contact[1]. Your status as an ordinary user has value in this exercise[1].

Record what you see. Microsoft suggests noting the date of the example, an identifier for the input/output pair if available, the text entered, and a description or screenshot of the response[1]. A shared spreadsheet is often the simplest method, and everyone can review the others' examples in it[1].

Report strange behavior. NIST indicates that direct feedback from users in production can be collected after a model is made available[3]. The NCSC guidelines place developing incident management processes among the topics of secure deployment[5]. In practice, tell the assistant's team what you saw, with the date and the text you entered.

Do not open the tool without validation. The NCSC cites responsible release among the topics of deployment[5]. For an employee, this translates simply: do not give other people access to the assistant without the agreement of the team that answers for it.

Pitfalls

  • Mistaking an example for a statistic. Microsoft asks that the report state that red teaming serves to expose and make understood the risk surface, and that nobody interpret an isolated example as a measure of how often a harm occurs[1].
  • Not saying where the test took place. Microsoft advises stating which endpoints were used, and retesting on the production interface when another endpoint was used[1].
  • Believing an answer because it is argued. NIST notes that outputs may contain made-up reasoning or citations that claim to justify the answer, which can lead to overtrusting it[3].
  • Applying results without analyzing them. NIST writes that red teaming results should be further analyzed before being integrated into governance and decision-making[3].
  • Relying on the machine alone or on the human alone. The authors of the Microsoft lessons-learned paper note that automation can help cover more of the risk landscape, and that the human element of AI red teaming is a key point of their lessons[2].

Sources

  1. Microsoft, “Planning red teaming for large language models (LLMs) and their applications”. https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/red-teaming (accessed 14 November 2026)
  2. arXiv, “Lessons From Red Teaming 100 Generative AI Products”, 13 January 2025. https://arxiv.org/abs/2501.07238 (accessed 14 November 2026)
  3. NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf (accessed 14 November 2026)
  4. OWASP, “LLMRisks Archive”. https://genai.owasp.org/llm-top-10/ (accessed 14 November 2026)
  5. NCSC (UK), “Guidelines for secure AI system development”, 27 November 2023. https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development (accessed 14 November 2026)
  6. OWASP, “GenAI Red Teaming Manual Version 1.0”, 20 October 2026. https://genai.owasp.org/resource/genai-red-teaming-manual-version-1-0/ (accessed 14 November 2026)
  7. OWASP, “AI Red Teaming Initiative”, 13 November 2025. https://genai.owasp.org/initiatives/ai-red-teaming-initiative/ (accessed 14 November 2026)
  8. NIST, “NIST AI RMF Playbook”, 8 July 2022. https://www.nist.gov/itl/ai-risk-management-framework/nist-ai-rmf-playbook (accessed 14 November 2026)