Bitter Lesson

[giscus[bot]]

Bitter Lesson

src: blog Summary In AI research, leveraging computation > incorporating domain knowledge. Given finite resources, it always…

https://deepmind.vercel.app/bitter-lesson/

[JohnReid]

Not sure I buy all the arguments on this page. Massive computation is important but certainly not the only component of success in the AI game tasks listed above.

[dfeng]

My first comment – this is exciting! Also, another meta-comment: I think some kind of discussion forum where we can go back and forth offline would be a great addition to our reading group. Happy for my notes to be a starting point :)

I think the question here is, where is our time better spent: on, as you pointed out on Thursday, the underappreciated engineering effort required to have models scale to ever increasing sizes; or, to spend more time crafting ever more complicated architectures? Even in the scaling world, there needs to be sufficient investment in the architecture for the benefits of scaling to bear fruit. But at some point the opportunity costs flip, perhaps?