AI Safety Fundamentals

A concise primer on what machine learning is and how models learn from data rather than explicit rules.

A visually-driven video series building intuition for what neural networks are, how they represent functions, and how gradient descent trains them.

A long-form walkthrough of the entire LLM pipeline: pretraining, tokenization, fine-tuning, and RLHF.

A ~1-hour talk giving a compressed, accessible overview of what LLMs are, how they're trained, and where they're headed.

On understanding the current state of LLM scaling and the future of AI research.

METR's graph showing the length of tasks frontier AI agents can complete, doubling roughly every seven months.

Nick Bostrom's paper on the potentially dangerous goals advanced AI systems will develop.

Distinguishes software-only, chip-technology, and chip-production intelligence explosions, arguing that the feedback loops have very different speed limits and bottlenecks.

A fun video from 2023 that discusses the problem of specification gaming.

Rob Miles walks through real examples of AI systems exploiting loopholes in the objectives given to them.

Rob Miles' video introducing the idea of mesa-optimizers in a digestible way.

Hubinger et al.'s foundational paper introducing mesa-optimization and framing the inner alignment problem, including the first careful treatment of deceptive alignment.

Emergent misalignment and unintuitive examples of how values manifest and generalize from narrow training data.

Anthropic's research on alignment faking, where LLMs strategically attempt to preserve their values during training.

An in-depth exploration of deceptive alignment and pseudo-alignment, providing insights into inner alignment issues.

Excerpts of frontier models' internal reasoning during evaluations for covert behavior, from the OpenAI/Apollo anti-scheming work, showing deception and eval awareness in the wild.

An informal note on some intuitions related to mechanistic interpretability by Chris Olah.

Argues that the training target should be corrigibility itself — a system that reliably accepts correction and shutdown — rather than good values, on the grounds that corrigibility is more achievable and fails more gracefully.

A paper on how we can communicate complex goals to RL systems.

Anthropic's method for training harmlessness from a written set of principles using AI-generated feedback rather than human labels.

OpenAI's empirical study of whether weak supervisors can elicit good behavior from stronger models, as an analogy for humans supervising superhuman systems.

Proposes that two models arguing opposite sides in front of a human judge could let humans supervise decisions they couldn't evaluate directly, on the analogy that verifying an argument is easier than producing one.

A cross-lab position paper arguing that legible chain-of-thought gives us a rare window into model reasoning that further training could easily close.

A comprehensive playbook for protecting AI models from theft and misuse.

Safety training is a defense, and defenses can be optimized against: jailbreaks that transfer across models, adversarial examples, and extraction of model internals through API access alone.

GovAI proposes a standardized methodology for testing how robust frontier model safeguards are to biological misuse attempts.

Compares AI insider threats directly against the human insider threats that security teams already model, and argues the differences are large enough to demand different countermeasures.

Introduces control evaluations — red-team/blue-team games where the blue team designs a deployment protocol and the red team builds models that try to subvert it — and tests concrete protocols like trusted and untrusted monitoring.

Hendrycks, Schmidt, and Wang's proposal for a deterrence regime where states threaten sabotage of destabilizing AI projects, modeled loosely on MAD.

Anthropic's internal governance policy for scaling frontier models safely.

The White House's AI policy agenda, centered on accelerating American AI development, infrastructure, and international competitiveness.

A verification plan for international restraint built on compute governance.

Narayanan and Kapoor's influential essay arguing AI should be understood as a normal general-purpose technology whose diffusion is slow and controllable, not an impending superintelligence.

Mechanize argues that prominent AI doom arguments are structured to be unfalsifiable and should be discounted accordingly.

An examination of where the standard existential risk argument has gaps, including whether AI systems will be goal-directed in the ways the argument requires.

A discussion of criticisms of AI safety concerns from a leading accelerationist perspective.

Anthropic's argument, using public benchmarks and previously unreported internal data, that AI is already accelerating AI development.

A technical exploration of interpretability in neural networks.

Rich Sutton's short essay arguing that general methods leveraging computation reliably beat approaches built on human domain knowledge.

Argues existential risk can arrive without any takeover: as AI replaces human participation in the economy, culture, and states, human influence erodes incrementally.

Argues that once AI replaces human labor, powerful actors lose their incentive to invest in ordinary people — a dynamic analogous to the resource curse.

A surprising phenomenon where language models transmit behavioral traits via semantically unrelated data.

Anthropic's theory that LLMs are actors simulating characters learned in pretraining, and the assistant is one such persona.