Mohammed Hamdy's picture

Open to Collab

Mohammed Hamdy

mmhamdy

hugging-science

·

https://surfingmanifolds.substack.com/

AI & ML interests

AI4Sci | NLP | Reinforcement Learning

Recent Activity

repliedto their post about 5 hours ago

What if you could train a model on just 10 images instead of 60,000 and still get close to the same performance? Traditional machine learning requires thousands, even millions, of data points to achieve high accuracy. But what if we could "distill" the entire dataset into just a few synthetic samples? This is what Dataset Distillation offers. Unlike traditional knowledge distillation, we keep the model fixed and distill the knowledge contained in a massive training set into a tiny set of synthetic distilled images. The goal is to train a model on this ultra-small set and achieve performance that almost matches what the same model would get when trained on the massive original dataset. For example, training on only 10 distilled MNIST images (this is equivalent to a single image per class) yields 94% accuracy, compared to 99% when training on the full 60,000 images. Interestingly, these distilled images look significantly different (as you can see in the image below) from natural images because they are optimized for model training rather than for matching the correct data distribution. But that's not all. Most importantly, this same method opens the door to a potent form of data poisoning. Because distilled images are specifically optimized for rapid learning, an attacker can create a tiny set of adversarial distilled images to cause a well-trained model to forget or misclassify a specific category. What I find fascinating about dataset distillation is this: it mimics human-like learning by letting a model grasp a concept from a single example, but it does so using alien synthetic images that mean absolutely nothing to a human eye! What about you? What are your thoughts on it?

posted an update about 5 hours ago

What if you could train a model on just 10 images instead of 60,000 and still get close to the same performance? Traditional machine learning requires thousands, even millions, of data points to achieve high accuracy. But what if we could "distill" the entire dataset into just a few synthetic samples? This is what Dataset Distillation offers. Unlike traditional knowledge distillation, we keep the model fixed and distill the knowledge contained in a massive training set into a tiny set of synthetic distilled images. The goal is to train a model on this ultra-small set and achieve performance that almost matches what the same model would get when trained on the massive original dataset. For example, training on only 10 distilled MNIST images (this is equivalent to a single image per class) yields 94% accuracy, compared to 99% when training on the full 60,000 images. Interestingly, these distilled images look significantly different (as you can see in the image below) from natural images because they are optimized for model training rather than for matching the correct data distribution. But that's not all. Most importantly, this same method opens the door to a potent form of data poisoning. Because distilled images are specifically optimized for rapid learning, an attacker can create a tiny set of adversarial distilled images to cause a well-trained model to forget or misclassify a specific category. What I find fascinating about dataset distillation is this: it mimics human-like learning by letting a model grasp a concept from a single example, but it does so using alien synthetic images that mean absolutely nothing to a human eye! What about you? What are your thoughts on it?

repliedto their post 5 days ago

It was supposed to be a failed experiment. Instead, it led to the discovery of one of the most intriguing phenomena in neural networks, simply because a researcher forgot to turn it off and left it running....for a week! In 2022, researchers at OpenAI were studying how neural networks generalize from their training data. For this task, they were training small transformer models to perform modular arithmetic. The thing is, neural networks are weird. When a model has an abundance of parameters (like neural nets), it can easily overfit. It essentially memorizes its training data, scoring a perfect 100% accuracy when tested on it, but remains completely clueless when faced with any new instances not present in the training set (close to 0 accuracy). It is like memorizing 1 + 2 = 3 without understanding the concept of addition, so if 2 + 3 wasn't in the training set, the model fails miserably! Usually, when a model overfits like this, people just cut their losses, turn off the experiment, and move on with their lives. But sometimes they forget. And that is exactly what happened to our researchers at OpenAI. A week later, they checked back in, and a miracle had happened! They discovered Grokking (And no, this has nothing to do with xAI's Grok , the term was originally coined by sci-fi author Robert Heinlein to mean understanding something so deeply that it becomes part of you). Grokking is when a neural network suddenly and abruptly learns to generalize long after it has overfitted. Just take a look at the graph in the image below! Spooky, right! I told you neural nets are weird!

View all activity

Organizations

mmhamdy 's collections 4