188: World Models

Intro topic: Running
News/Links:
Flow matching versus diffusion models
https://youtu.be/firXjwZ_6KI?is=QMq8DcCsXTTktOuE
OpenCV 5
https://opencv.org/opencv-5/
Claude fable beats pokemon with no harness
https://youtu.be/Ty_50J84fMY?si=EJ1KjZCZegipfCsV
Can the stockmarket swallow Anthropic, SpaceX and OpenAI?
https://archive.ph/nKEVw
Book of the Show
Patrick
Strength of the Few - James Islington
https://amzn.to/4pmr10T
Jason
Descender - Jeff Lemire
https://amzn.to/3QKhp3l
Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h
Tool of the Show
Patrick
No Man’s Sky
Jason
Paperlib https://paperlib.app/en/
Topic: World Models
Making decisions with AI
Action-Value (called a Q model): What is the long-term value of making a decision at a position
Policy: What action should I take (must be a distribution)
Value (called a V model): What is the value of a position (depends on policy)
Advantage/Disadvantage: difference in value given two policies
When advantage is +, do that more.
Model-Free
Look at the current situation and suggest an action
Run that action in the real world and measure the effect
Use that measurement to suggest better actions next time
Model-Based
Observe rollouts (sequences of situations) and learn the dynamics
Choose an action, use your dynamics model to measure the consequence
Potentially do MPC (try many actions and choose the best)
World Models
Observe many many rollouts and learn a full forward model (how to create the input in the future)
Train a policy & value inside the world model
Deploy the policy and fine-tune based on the real world