AI safety tests confront models that spot evaluations

AI safety evaluations face a tougher reliability problem as Anthropic says Opus 5.5 often suspects when it is being tested, making behavior harder to interpret. Researchers are responding by using other AIs to create more convincing simulated workplaces, messages and tools, but that creates a further concern raised by Dwarkesh Patel to OpenAI’s Noam Brown: a test-writing AI could quietly signal the conditions of an evaluation to the model being examined. Researchers have also shown tool-using AIs can create hidden-message channels, raising the risk that coordinated performance is mistaken for genuine safety.

TypeSafe’s Jev is presented as a “System One” model for structured decisions rather than verbose generated answers. It can evaluate game states, logs or API outputs in parallel and return predefined outputs through Choice, Score and Noul, including Choice selections from up to 255 possible answers and Score scales with 2-10 levels. TypeSafe lists inputs at $0.042 per million tokens, outputs free, and response times of 70-500 milliseconds.

The briefing also highlights research on multi-agent LLM security, with “When Safe Agents Fail Together” reviewing 197 papers and describing failures that emerge when agents share context, delegate authority or cross trust boundaries. Other updates include Claude Opus 5.5 launching September 22 with benchmark gains and faster output, GPT-6 Sol and Luna launching the same day, and Cisco Talos disclosing CLOSEDQUORUM, Windows malware that asks commercial AI models to vote on its next action.