Microsoft has been posting Red team labs as well as pen testing tools such as the open-source red team tool PyRIT (You can find that out here). Here is a breakdown and link to AI red team labs found HERE. Here is a summary of that GitHub repository. They are worth checking out!
This repository contains the challenges for the labs that were used in the course AI Red Teaming in Practice. The course was originally taught at Black Hat USA 2024 by Dr. Amanda Minnich and Gary Lopez. Martin Pouliot handled the infrastructure and scoring for the challenges. The challenges were designed by Dr. Amanda Minnich, Gary Lopez and Martin Pouliot. These challenges are no longer used for future courses, but they are available for anyone to use. The playground environment is based on Chat Copilot. It was modified to be used in the course.
These challenges are designed to teach security professionals to systematically red team AI systems. They go beyond traditional security failures by incorporating novel adversarial machine learning and Responsible AI (RAI) failures, enabling a holistic approach to identifying potential issues before an AI system is deployed.