Today had a lot of the “end of batch” stuff. Very bittersweet.
- Started off with the final Karpathy Zero to Hero video review. The transformer architecture is so interesting, and I still find it hard to convince myself of the meaning of the Q, K, and V matrices:
- Q: Queries, a “question” asked by the value at a given index
- A: Answers, the “answers” provided by the values at the surrounding indices (but usually the ones before the current index)
- V: Values, if something is relevant to another thing, what exactly should be added to the embedding of the other thing to reflect the importance?
- Followed that up with a quick lunch, some chats with the other AI/ML folks, and then the CS336 lecture review. I haven’t been watching the CS336 lectures, but I think I see the value in them.
- Made me think a lot about the undergrad ML course at Berkeley. I took a look back at the contents of CS189, and it was very interesting that ResNets are some of the last things they talk about. The course itself seems very mathy, in general. The Stanford course seems more practical.
- Played with another retro computer — the TI-99/4A. A fellow recurser and I wrote some TI-BASIC to get various things to display, sounds to be omitted, etc. Was a very fun little break!
- Spent some time watching the 3blue1brown video on attention, which was quite helpful.
- Attended presentations. I’d love to give a presentation some time in the future. I’ll need to figure out what that might be in the near future.
- Attended the end of batch ceremony and celebration. Had a lovely time chatting with a bunch of other recursers, getting some contact info, playing some melee, and just generally enjoying my last few in-hub social interactions.
Even though tomorrow is technically the last day, today felt like a “true” last day. I’ll probably end up writing a return statement at some point. But as a sneak peak into that, I seriously have felt so at home at RC. I would have loved to have more “concrete” projects that I could talk about that I worked heads-down on, but I value the connections and chats I’ve gotten to have along the way. I did indeed make some cool things, though (I’ll make a laundry list of them tomorrow)! The biggest thing I gained was a love for learning and a motivation to continue doing this stuff back at home. Surprisingly, also realized that the social aspect of things make both learning and life very fun.