Dear Deep Learning Engineer, if all you know currently is how to use Supervised Learning, then this page will potentially unleash the "next level" in your Deep Learning journey...
Here's why:
Get a dataset, label it, train on it, hope to generalize. If you can't, add data until you can. This is what I did for years.
This is what most of us engineers still do: we download public datasets and try to train our models on them. When there's a lack of labels, we spend WEEKS trying to get more labels to improve the model.
But while we're doing this, the "field" has advanced tremendously and is now capable of doing better. Public datasets don't work everywhere. A self-driving car trained on USA roads might not drive as well in Italy, or India.
Companies know this and now have incredibly smarter solutions...
They understand the patterns contained within the data rather than learning how to reproduce a specific output.
And when you look at the recent breakthrough happening lately, whether it's Chat-GPT, or Tesla FSD12, or Nvidia's latest models, or Meta's research... do you notice how they claim to have all been trained on "the entire internet"?
What does it mean? How do you do this? The method has a named, it's called...
And this relatively "new" idea is in fact what's powering the latest Deep Learning breakthrough. From Chat-GPT to FSD12, to BERT, DINO, and others.
For example, would you believe that this model has been trained without a single label?
DINOv2 got released in 2023, and was trained from 142 Million Images without any supervision... This means it has not naively "learned" how to find the right label... it understood the data.
You might think that "Transformers" were the main reason of the Deep Learning boom, but the reality is, they're merely a "vehicle". The "fuel" happened through self-supervised learning.
And this position is what I'm most interested in. Being an engineer working on a specific project is fine, but it's very different from being seen as THE expert who's able to keep up to date with the field. It's very different from having this wide Deep Learning understanding.
And this is what you want to do:
Whenever a new Breakthrough AI model comes out, you want to be able to "get" how it works and explain it to colleagues.
When you are given a weakly labeled dataset, you want to be capable of suggesting better solutions than to hire human labellers and lose weeks in the process.
When you're collecting a dataset, you want to be able to precisely indicate what will have to be labeled, and what won't be.
These are the kind of "intels" that tell colleagues and superiors that you're not just a resource, but someone they can go to and ask questions... someone that can lead them when a Deep Learning doubt raises.
Expertise in Deep Learning now includes Self-Supervised Learning, especially if working with new models, but it's still quite hard to understand how this works. In the blogging and communication area, we're still at the research state. There is a lack of introduction material, courses, projects, and everything that will make you competent in using SSL at work.
Introducing...
A quick guide to understand Self-Supervised Learning and learn how to train Deep Learning models at imperial scale...
What is Self-Supervised Learning, and an intro to the "jargon" used in Research Papers
How Knowledge Transfer works, and the difference between Self-Supervised Learning and Transfer Learning
The "Pretext" my friend Max used to avoid meeting his girlfriend's parents, and where it led him (nothing to do with SSL, or... maybe just a little)
The 3 Types of Self-Supervised Learning Tasks, and which one companies use the most
Examples of Transformative Tasks used in Self-Supervised Training (you may not realize how many data augmentation techniques such as random cropping, colorization, rotation, can be effective in learning..)
A Deep Dive in Contrastive Learning, and how it helps us learn from millions of images without any label
Introduction to SimCLR, BYOL, DINOv2, and other Self-Distillation Networks (we'll also take a look at Negative Sampling, a technique quite used recently)
How to pick a "Pretext Task" (and a list of best practices I noticed Engineers should apply when using Self-Supervised Learning in production)
How to use Generative Networks to create "pseudo-labels" (and why it's not the best method to train neural networks)
A cutting-edge deep dive into Bird Eye View Segmentation Networks trained using Self-Supervised Learning for Path Planning, Object Tracking, and Trajectory prediction
STOP!
Let's take a break a second.
In this last bullet point, we'll do a cutting-edge deep dive, and we will do a block-by-block analysis of several state of the art models, such as the Bird Eye View segmentation model, producing results like this:
This model has been trained without a single label. We were able to send an image and output a Semantic Bird Eye View map using contrastive learning, but also techniques such as Implicit Learning, Explicit Representation Learning, but also Depth Networks, Feature Lifting, and more...
In this codex, we'll do a Block by Block analysis of how to achieve this. We'll also see another network doing 3D Reconstruction using Self-Supervised Learning.
This model is even cooler, and the animation you see above has been made by Think Autonomous Edgeneer Dheeraj Madda who trained a Self-Supervised Learning 3D Reconstruction Model using Depth Estimators and Self-Supervised Learning.
Now, at this point, you'll be very familiar with Self-Supervised Learning, you'll have:
Understood the core Self-Supervised Learning Techniques
Be familiarized with techniques used by companies like Meta to train their models
Have done a deep dive into each of the 3 categories of Self-Supervised Learning algorithms
Done a block-by-block review of 2 Self-Driving Car models trained using SSL
And I believe that if your goal is to be familiar with Self-Supervised Learning and have the knowledge needed to continue on your own, this will be great.
If you want to combine your learning with practice, we added an extra chapter in which you'll train your first models using SSL. We created a self-driving car notebook where you'll be able to implement a contrastive learning algorithm to train algorithms to "clusterize" scenes without a single label!
This will be an epic way to finish your journey in the Self-Supervised Learning world, and this will also be a mini "visual" project you can add to your portfolio...
"Really loved your codex (I read it twice over and wasn't satisfied so I printed it out to read again). The maginot line example was really good, but I must admit the diarrhea example was kind of unforgettable š.
Personally, I am already familiar and have worked with SSL before (for multi-task learning infact with lightly), so I found the pretext tasks section and the later half of your codex to be super fascinating and useful (the SkyEye + the Sfm example).
Right now, I'm working with some of my friends on a multi-task autonomous driving dataset for Indian roads (purely based on the data I collected), and your courses have been helping me a lot with shaping the future path for this paper. Can't wait to share it with you when I'm done!"
"In a small company, we easily gather thousands of relevant photos daily, but labeling this data is expensive and requires specialized knowledge from the wood industry. Being able to use unlabelled data effectively has made a significant impact.
The explanations in the course were crystal clear, allowing me to implement methods from scratch, even with basic pretext tasks. This was particularly valuable as I had some knowledge of self-supervised learning but hadn't used it before.
The course led to improved results in image segmentation, a better understanding of how CNNs work, and the pretext tasks were fun to create.
I would 100% recommend this product due to the clarity of the explanations and the short time it took me to grasp and implement the concepts. This course is a must-have for anyone working in the industry.
I hope to find time to take the image segmentation course and the Segformer DLC, aiming for even better results without needing to label more data."
A look "inside" the Self-Supervised Learning Codex (PDF format):
Before you join, a few notes:
The choice has deliberately been done to build a product without videos. The goal here is to have you being able to go and grab a concept as you wish. Meaning using Ctrl-F anytime you want to revisit a concept.
SSL is very advanced and hard to understand. Most algorithms we'll study are from 2023. My goal is NOT to do an exhaustive review of all the techniques that have ever existed, and is NOT to bury you under maths. If you join this product, it's because you want to "get the gist" of what SSL is and understand how to use it in your job (not write an SSL thesis).
The codex is perfect for engineers who would like to find more "solutions" in their day job. Not just "add more data", or "label more data", or "to go faster, reduce your image size by 2!". But real, creative, smart tricks that Deep Learning Engineers use in their day job that actually work.
This codex has been optimized for consumption. I want you to finish it. Not have it 23% completed for months. The course is deliberately concise so you can read it during your lunch break (it will probably take a bit longer, but you get the point).
However, it gives lots of room to continue exploring on your own, and even has a project so you can practice and build your first SSL algorithms.
If you're good with this and ready to level up your Deep Learning skills, then below is the link to join:
*In both plans, 2-time payments are available at no extra cost.
ā Self-Supervised Learning Codex
ā Self-Supervised Learning Project
ā
79ā¬
ā Currently not available