Representation Engineering: A Top-Down Approach to AI Transparency
We identify and manipulate high-level cognitive representations within neural networks, enabling more precise control over model behavior than traditional fine-tuning approaches.
Paper Discovery Feed
Explore domain hubs and recent submissions
ESM-2: Language models of protein sequences at the scale of evolution enable accurate structure prediction
We train protein language models up to 15B parameters and find that as models scale, information emerges in the representations that enables accurate atomic-resolution structure prediction.
Language Models are Few-Shot Learners
We show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches.
Quantum Error Correction with Fracton Topological Codes
We study fracton topological codes as a framework for quantum error correction, showing that their sub-extensive ground state degeneracy provides natural protection against local errors.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
We introduce BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
We combine pre-trained parametric and non-parametric memory for language generation, using a dense passage retriever to condition seq2seq models on retrieved documents.
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
We apply dictionary learning at scale to extract millions of interpretable features from a production language model, finding features corresponding to a wide range of concepts.
MatterGen: A Generative Model for Inorganic Materials Design
We introduce MatterGen, a diffusion-based generative model that designs novel, stable inorganic materials across the periodic table with desired properties.
Crystal Diffusion Variational Autoencoder for Periodic Material Generation
We propose CDVAE, a variational autoencoder that generates stable crystal structures by learning to denoise atom types, coordinates, and lattice parameters simultaneously.
scGPT: Toward Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
We present scGPT, a generative pretrained transformer model for single-cell biology that enables cell type annotation, multi-batch integration, and perturbation response prediction.
Quantum Approximate Optimization Algorithm: Performance, Mechanism, and Implementation on Near-Term Devices
We study the performance of the Quantum Approximate Optimization Algorithm (QAOA), proving concentration of parameters and providing implementation strategies for near-term quantum hardware.
GenePT: A Simple But Effective Foundation Model for Genes Using ChatGPT
We generate gene embeddings by converting NCBI gene summaries into vector representations using GPT-3.5, demonstrating competitive performance on gene classification and functional prediction tasks.
Constitutional AI: Harmlessness from AI Feedback
We propose Constitutional AI (CAI), a method for training AI systems that are helpful, harmless, and honest, using a set of principles to guide AI behavior without extensive human feedback on harms.
PennyLane: Automatic differentiation of hybrid quantum-classical computations
We present PennyLane, a Python library for differentiable programming of quantum computers that seamlessly integrates classical machine learning libraries with quantum hardware and simulators.
Attention Is All You Need
We propose the Transformer, a model architecture based entirely on attention mechanisms, dispensing with recurrence and convolutions. Experiments show these models to be superior in quality while being more parallelizable.
AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space
We present the AlphaFold DB, providing open access to 200 million protein structure predictions, covering nearly all catalogued proteins known to science.
CHGNet: Pretrained Universal Neural Network Potential for Charge-Informed Atomistic Modelling
We present CHGNet, a graph neural network pretrained on the Materials Project trajectory dataset, enabling rapid and accurate prediction of energies, forces, and magnetic moments.
Uni-Mol: A Universal 3D Molecular Pretraining Framework
We propose Uni-Mol, a universal molecular representation learning framework that directly operates on 3D molecular structures, significantly improving property prediction tasks.
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
We find that current behavioral safety training techniques are insufficient to remove deceptive behavior from large language models, even when the deceptive behavior was inserted during pretraining.