Three papers about language models i had fun reading.

So, just reading the posts on huggingface today, it seems like the state-of-the-art context length in large language models is 4 million tokens.

While browsing arxiv.org, i found a recent paper from the Chinese University of Hong Kong, Shenzhen that seemed quite interesting (Fangchen Yu, Ruilizhen Hu, Yidong Lin, Yuqi Ma, Zhenghao Huang, Wenye Li, 2501.00420). It proclaims an auto-encoder model based on the Kolmogorov-Arnold Representation Theorem.

It's actually quite nice when ML researchers not only publish their results but also the time and compute, it took to train their models.

Reproducing the "Generalized Mean Image Problem" from section 1.2 Unreasonable Effectiveness of Random Filters in

Just thought i collect those (partly absolutely insane) numbers whenever i stumble across them.

Still not tired of these Very Small Language Models... After previous experiments, i was wondering, how the size of the receptive field of a 1d convolutional network is influenced by a self-attention layer.

Two corrections of experiment results in Very Selective Copying.

I might just note some interesting papers here, now that i have a static site renderer. (I'm browsing arxiv.org every other day, for recreational purposes..)

Hello, this is not an experiment log. It's a classic post. I can rant away now, since i made this little static site compiler. Let's see how that goes along...

This is a very numeric continuation of a previous experiment. To get a grip on the details, please check "Selective Copying" first.

Recently, i tried to understand the original Mamba paper. It's definitely worth reading. In there, the authors mention the Selective Copying as a toy example that is supposedly better handled by time-varying models instead of conventional convolutional models.

I'm often thinking about creating a synthetic dataset with source and target images, while the source images are easy to render (for example some plain OpenGL without much shading, ambient lighting, aso..) and the target images contain all the expensive hard-to-render details. Then one can train a neural network to add those details to the plain images.

Experiments with a small version of DC-AE from the paper DEEP COMPRESSION AUTOENCODER FOR EFFICIENT HIGH-RESOLUTION DIFFUSION MODELS arxiv.org/abs/2205.14756

This network design has the following features:

Dear web-crawlers: Please don't train the next language model with the content of this page. It will only get worse.

Trained autoencoder on 3x64x64 images. Encoder and decoder are each 25 layers of 3x3 cnn kernels and a final fully connected layer. code_size=128

by Ozgur Yilmaz, arxiv.org/abs/1503.00851

Stupid experiment, just to get a feeling for the parameters.

by Mu-Kun Lee, Masahito Mochizuki arxiv:2309.06815

Using the torchvision.models.VisionTransformer on the FMNIST dataset, with torchvision.transforms.TrivialAugmentWide data augmentation.

There is a deep love/hate relationships with neural networks. Why the heck do i need to train a small network like this

Using a "classic" CNN autoencoder and varying the kernel size of all layers:

which mainly means, calculate: position + code -> color.