This is a minor update before a big one.

People have been asking me for a long time to add shuffle to likes, playlists, and the main page. I've tried to build it a few times already, and trust me, this problem is a lot harder than it looks. A true shuffle needs either a lot of CPU, a lot of RAM, or a lot of disk space.

Either way, a lot of something.

The simplest and most direct approach is to sort all posts randomly at request time and return the result. But on a large database that's a brutally expensive operation. Pagination is a problem too. You need to keep the sort order stable so that page two is guaranteed to have no posts from page one. Which means you also need to store the seed of that shuffle somewhere.

This time I managed to solve it without putting any serious load on the server. The only catch is a 20k limit for likes, playlists, and bookmarks. The main page has no limits at all.

How it works

If you don't care about the technical details, just scroll past.

Instead of reshuffling posts on every request, I sort them by hash. A post's position in the random order is computed as hash(post id + salt). The salt is just one number, shared across the whole site, so the order is fixed and known in advance. For the main page there's a regular database index built on top of that order, and grabbing a random page is as cheap as any other, no need to re-sort millions of posts on the fly. The posts actually got shuffled only once, and the randomness you see on every refresh is an illusion. To make it work, and to make the posts look random every time, you just start from a new random point in that sequence and go around it in a circle. Once a day a background job generates a new salt and swaps the index, and that's when the posts get reshuffled for real.

All the traversal state lives in a cursor that goes back and forth between the browser and the server, so the server stores nothing and spends no memory on sessions. You get proper pagination out of this for free. Page two is guaranteed to have no posts from page one, a full walk shows every post exactly once, and refreshing the page just starts a new shuffle. Likes, bookmarks, and playlists don't need an index, the hash is computed on the fly over your selection. But that means the cost of a page grows with the size of the feed, and that's where the 20k post limit comes from. It doesn't affect the main page, that one is served by the index.

Reverse sorting

By default everything is sorted from newest to oldest. For likes that means from "recently added" to "added long ago". People have been asking me for reverse sorting for a long time.

Here it is. Enjoy.

Bigger blacklist limit

When I look at the content on the site, I get that not everything works for everyone. And as it turns out, a blacklist of three tags is just too small. So I'm raising the limit to 10 tags. That should be enough for most of you.

If you don't have these changes yet, they'll show up during the day.

The thing I've been teasing for a while

Just wait a little longer, almost everything is ready. I'm ALREADY writing the announcement. I need to get everything in place first. I think it's very close now, probably next week.