It looks like when you have template caching on, a page gets cached after it's visited. Then subsequent visits are served from cache.
When something triggers the cache to empty, such as an entry being edited, the "user visiting the page for it to cache" process starts again.
But my non-cached page speed is around 3 seconds, and I'd rather avoid anybody hitting this. Is there any way to load pages into cache en masse? If not, maybe a server script that gets all pages via cURL might work?