i jus wanted to get dis outta my system >v< …
i dun like those boring linear model structures… they work… bt they dun look fun, nor intuitive. they jus produce output… which is boring!
pls, if some researcher with lotsa gpus sees this, maybsies try this kinda architecture… u dont evn have to credit me, just try it out n see where it goes ~ ~ ~
yeaaaa you’re right… i was referring specifically to LLMs, but yes, recurrent models are essentially everywhere else.
i am just surprised we don’t have many LLMs with recurrent blocks in them, like this model here did for example. i really hope we go that direction soon…
Afaik all LLMs have very derp recurrance, as that’s what provides their context window size.
The more recurrant params they have, the more context window they can store.