Login

375€ or 2 x 189€

MASTER 4D PERCEPTION: Ultimate 3D Tracking Skills for Perception Engineers

Learn 3D Object Detection, 3D Tracking, and Build Advanced 4D Perception Projects

This course is now closed and will open Early 2026. Make sure to be on the waitlist to get notified and receive goodies while you wait.

Join Waitlist

Autonomous Shuttle Engineer Launches the First Ever 4D Perception Course made for Engineers looking to build cutting-edge skills that STAND OUT!

Dear Perception Engineer:

If you would like to learn how to build the 4D Perception algorithms that the top self-driving car companies like Zoox, Cruise, or even Waymo are working on, then this page will show you how...

Here's a story:

One day of 2017, I have been invited to an interview with an autonomous tech company.

The company was specialized in submarines, robots, aircrafts, military equipment, and somehow — I made it to the technical interview.

And of course, I brought many of my projects!
I needed to "show" them what I could do, I needed to be visual, I needed to drive home incredible results!
So I showed them a lane line detection algorithm I've learned through an online course.

"How did you build this?" he asked

Unexperienced, I started to mumble, and ended up sharing some details, including how I've used OpenCV functions such as Canny Edge Detection or the Sobel Operator.

... the recruiter stayed silent...

Any question?

Suddenly, he starts to have a blast. "Hahaha! the SOBEL?! Nobody's been doing this for over 20 years! Where did you learn this?!"

I felt ashamed. How could I not know this? And the recruiter wasn't a bad one; he was fascinating. But I simply wasn't "connected" enough to the industry, and what the companies were doing at the moment.

Could that ever happen to you?

What if it did? Are your projects advanced and relevant enough?

When you start looking around, you'll see a bunch of weird differences between the projects shared by engineers online, and what companies are actually looking for. Things like:
  • Sharing your OpenCV skills to companies employing dozens of Deep Learning researchers
  • Sharing a 2D Object Tracking algorithm to a company whose expertise is 3D Perception
  • Sharing traditional point clouds processing algorithms to companies used to 3D Deep Learning
It evolves so fast... it's easy to learn something and not realize you're completely off the mark.

In a couple of years, we've moved from YOLO, basic Sensor Fusion algorithms, and A* based planners...
... To FMCW LiDARs, 3D Object Detection and Tracking, Reinforcement Learning, and even now Transformer based architectures.

But if you want to work in the Perception field, you should start paying attention to what Perception Teams do.

What do they do?

Recorded at Zoox

Look at what top self-driving car companies like Waymo, Zoox, or even Cruise are working on and sharing online:

Recorded at Zoox, CES 2023, Las Vegas

Can you see it? It's all 3D! And it's even tracking objects and predicting their future trajectories!

Look no further. This is what their perception teams work on: 3D Object Detection & Tracking.

And I'd argue most of the related fields, such as robotics, drones, and others, are on the warfare of this too.

This if the field of 4D!
It's fascinating, but it's also one of the hardest.
One of the things I've noticed in the past few months is how little the 3D Object Detection and Tracking field is documented. There is a clear LACK of overview, easy-to-understand code, and courses.

Many engineers have no simple way to learn how to build these skills, other than reading dozens of research papers and getting bogged down for hours in GitHub repos. Reading blog posts after blog posts, many of them contradicting themselves, and leading nowhere.
And this can take months...

Until now:

Introducing...

MASTER 4D PERCEPTION: Ultimate 3D Tracking Skills for Perception Engineers

In this course, we're going to learn how to build radically advanced skills in the 4th dimension, so let's see the program:

MODULE 1

3D Object Detection

We'll begin the course right with 3D Object Detection. Many engineers know how 2D Object Detection works, but almost nobody knows how 3D Object Detection works. 

Whether it's on cameras or LiDARs, this is a largely unexplored field, and it's time to pioneer it.

What you'll learn:

  • How to build your own Self-Driving Car Perception Visualizer (it's 100% customizable — and originally comes from a library made for scientific analysis, but some engineers have accidentally discovered how to adapt it to the self-driving car scenario, and we'll use that too)

  • 11 Examples of 3D Object Tracking companies (we'll see startups in various industries such as Robotics, Sports, Hollywood, or even Retail)

  • A comprehensive Mindmap Overview of the field of 3D Object Detection (to my knowledge, this is the only overview existing for this field)

  • How Monocular 3D Object Detection Works — and the 6 Main approaches used to detect 3D Bounding Boxes (at least one of them will see a massive amount of research papers coming in the next few years)

  • How to modify the Faster-RCNN algorithm, used for 2D Object Detection, to generate 3D Bounding Boxes (more on that later, but the Faster-RCNN algorithm can be used in many other use cases)

  • 5 Techniques used to find 3D Objects with Deep Learning on Point Clouds (among these approaches, one of them is a point-based technique called "Point-RCNN")

  • Why you should never use or finetune a model trained on self-driving car data to detect a flying aircraft (and what to do instead)

  • The fastest known way to run a 3D Object Detector on any environment (we'll actually do it on Colab, which is known to be impossible for 3D Visualization)

  • An in-depth exploration of the Point-RCNN algorithm (this algorithm is one of the top algorithms used in LiDAR Object Detection)

Wait a minute.

Ever heard of the Point-RCNN algorithm?
This is an algorithm used on Point Cloud Object Detection with Deep Learning. The term "R-CNN" implies it belongs to the family of Faster-RCNN algorithms. Because most of the 3D field is full of 2-Stage algorithms from this family, we'll explore algorithms such as Point-RCNN (for LiDARs) or even Stereo-RCNN (for stereo cameras).

And for that, I'll use a methodology I used in my other tracking course, that got incredibly good reviews (in fact, the review I got the most of the former obstacle tracking course is that people loved the object detection module)

We'll do exactly the same with 3D, showing you step-by-step, how a 3D Object Detector works:

But this module is far from over, we'll also cover a lot of the implementation:

  • Where to find pretrained weights for the most common object detectors on Tensorflow and PyTorch. (some repos don't work — some repos work only for some libraries — we tested them all, and we'll share the ones we chose)

  • The "dark mode" secret to getting more attention in your LinkedIn posts when sharing point clouds results (yes — dark mode works best, and point clouds work better than images!)

  • The difference between ROI Pooling and ROI Align, and which one to use when detecting 3D Bounding Boxes on cameras and LiDARs.

  • The technical "know-how" to project 3D Bounding Boxes from one sensor to another — or from one dataset format to another (like KITTI and Waymo Datasets) — I know this point to have the potential to completely kill the fun in any 3D Object Detection project, so I'm giving the solution here.

  • 🟢 3D Object Detection Project — How to engineer your own Point Cloud Object Detector with Deep Learning

Want to see the project you'll build?
This is cool, because you'll learn to load a point cloud, run several different object detectors,  visualize the results, and even compare with multiple labels. 

Your project will look like this:

Honestly, I could have stopped there.

3D Object Detection is a fascinating enough topic. And building a 3D Object Detection course would have been a great idea already.

But I didn't want to. I wanted you to get exactly what the self-driving car companies do: 4D. So I built 2 more modules...

MODULE 2

3D Association & Fusion

Now we're going to the heart of the course: 3D Association & Sensor Fusion. You will learn how to merge 3D Bounding Boxes from frame to frame, or from sensor to sensor!

In this module, we'll try to match bounding boxes from 2 consecutive frames. We'll see cost functions related to the 3D field...

So here's a preview of what's included in Module 2:

  • 6 Techniques to fuse 3D Objects from frame to frame (3 of them will be very simple, such as calculating the distance between centroids, and 3 will involve heavy calculations such as comparing Point Clouds, or analyzing the motions vectors of objects through time)

  • How to display Bounding Boxes from 2 different time frames in the same visualizer (and the magic command that makes it possible to visualize endless 3D scenes in Google Colab)

  • Why you should never use 2D Object Detection Association Techniques, such as the IOU or Hungarian Algorithm, in a 3D Tracker.

  • To prove the previous point, and incidentally, I'm giving away: The implementation of the 3D IOU function

  • A Dead giveaway technique, inspired by the IOU, that could radically improve your 3D Fusion Results when fusing objects of different shapes.

  • The implications of Visual Fusion (LiDAR-Camera) on 3D Bounding Box Matching, and why comparing point clouds to pixels isn't a good idea.

  • Structural Loss: 5 Techniques to compare to sets of point clouds, and which one is the most commonly used by robotics companies and car manufacturers

In this little section, I'm going to show how to grasp the point clouds inside 3D bounding boxes, and compare them.

Notice how a bounding box moves fast in the 3D space, but how the point clouds keep the same shape:
Therefore, it implies we'll see:
  • A fast and elegant piece of code to grab the point clouds inside your bounding boxes, discovered by Chat-GPT (from there, we'll also learn the different ways to get these points, from the nested loop to the KD-Tree/Octree)

  • The exact combination of cost metrics to pick to build a State-Of-The-Art 3D Object Tracker running at over 200 FPS.
In this last point, I'll show you a 3D Object Tracker that has been able to get over 200 FPS, and beat any other tracking algorithm.

This matters, because in 3D, and in particular with point clouds, we have a LOT MORE things to process. If you're comparing point clouds inside bounding boxes for examples, you may end up with an algorithm running at just a few FPS. And if all you do is a basic euclidean distance, your algorithm may be too simple to succeed.

In this module, we'll see a winning combination that makes algorithms not only incredibly fast... but also accurate to the point of winning State-Of-The-Art Leaderboards!

But it's not all we do, we'll also see advanced techniques in Module 3:

MODULE 3

3D Object Tracking

In the final module, you will build your 4D Perception project. How to engineer a system that detects objects in 3D, assign IDs to them, and track them all along a sequence?

What you'll learn:

  • How to "sucker-punch" the competition when sharing a portfolio project online (we'll see a little "trick" that — doesn't make such a big difference — but still makes recruiters interested in your project rather than others)

  • The Mandalorian's approach to 3D Object Tracking, and why they'd pick Kalman Filters over Deep Learning any day of the week (likewise, we'll see the reason why most companies implement this way too)

  • The only type of Kalman Filter you should ever use when implementing a 3D Tracker (and why without doubt, companies don't use any other)

  • The industries you can work on as a 2D vs 3D Object Tracking Engineer, and what you should know before applying to a Perception role.

  • How to engineer "re-identification" after an object has been occluded, and why engineers sometimes prefer to leave a trajectory "dead" rather than re-identifying it

  • The obvious reason why Chat-GPT won't replace Search Engines, and how to develop an intuition about a technology's future. (it's not directly related to 3D Tracking, but part of a discussion about the use of older vs newer algorithms in tracking)

  • A quickie "idiot's guide" to Kalman Filters (although I recommend to already be familiar with Kalman Filters when taking this course, this little chapter aims to help you get up to speed in a few minutes)

  • The 11 core variables a 4D Perception algorithm must "feed" to a Motion Planning Module.

  • The broadcasting technique developers when they have lots of calculations to do (this improves performance, speed, code look, and will overall show you're more professional when coding)

  • How to convert vehicle physics and motion models to code and matrices, and the errors you should avoid at initialization.

  • Expanded Kalman Filters: The secret, almost automatic way to ensure you'll get rid of association errors on a tracking project.

  • How to use Kalman Filters with Autonomous UAV (Unmanned Aerial Vehicles) — if you're tracking a flying drone or car, VTOL and other differences with cars implies huge differences with the 3D field, we'll tackle them so you can use your algorithms in any possible situation)

  • 🟢 Project: Implement your own 4D Perception Algorithm for 3D Vehicle Tracking 

  • and many, many, more...
About the project...

Here's what you'll learn to do!

4D Perception & LiDAR Object Tracking on a Highway scenario

This project is advanced, and as I said, it has been built to be similar to what real self-driving car companies do.

It's also 100% customizable, and can be adapted to any self-driving car Dataset (see Waymo below) with almost no code modification.

Want to see how I teach Tracking?

Here are some reviews from my 2 other tracking courses (on Obstacle Tracking & Kalman Filters), and then we'll explore some on 4D Perception:

"Really liked it"

Mohammad Anas Khan, CSE Upcoming SWE Intern @Google

"I recently completed the updated version of your obstacle tracking course. Really liked it.

I liked that you gave a quick walkthrough of all the major 2D OD networks. I also liked the part where we used additional metrics and not just IOU for our hungarian matching.

Lastly, what better than seeing all of this being put into action on a live video?"

Review on my Kalman Filter Course (shared at the launch of 4D Perception)

"Exactly what I was looking for"

Ivo Germann, Development engineer R&D, Project leader bei Zumbach AG

"This is a great course and it is exactly what I was looking for! I've done many Kalman courses in the past but the example with the bicycle really opened my eyes.

To be honest I played with the code quite a few hours and could develop a really good understanding how a Kalman filter works. 

Thanks a lot!"

4D Perception Review:

"I got the thing which I am missing as advanced stuff in my resume to shortlist myself."

MLV Prasad, Data Science and analytics - Mentor @codingInvaders

Before purchasing the course, I was doubting because of the course fee wrt Indian purchasing power parity (it's my fathers 1 monthly salary.. to give you context and also the income per person in  India).

But I got the thing which I am missing as advanced stuff in my resume to shortlist myself. I really liked the project Implementation and simplicity of course page. Now I have an advanced project to keep in my resume, and your emails made me keep a high standard for myself. It also made me  keep you as my role model to be proficient in CV space in India in next two 2 years.

I thank you very much, because what I was looking for was found here in structured and guided manner, where I can't find elsewhere for sure.

I would definitely recommend. Thank you very much jeremy..

This engineer completed the course and shared it online, here are the results...

You think 297 likes is a lot of attention?

Well, his post has been re-shared by another channel, and look at the results!

THIS IS THE POWER OF 4D PERCEPTION!

STOP! Before enrolling, you should know these things:

First, this course is advanced, and has some prerequisites — most won't be able to join.

(Mandatory) So here are some prerequisites:
  • Code in Python — We'll use Python 3 as our main language
  • Knowing your high-school maths — including matrix multiplications
  • Have experience with 2D Object Detection — I recommend to, at least, having ran an object detector and been through a paper once in your life.
(Optional) You should also be familiar with the following:
  • The Kalman Filter, which probably belongs to the Top 3 most important algorithms in autonomous tech, and probably Top 1 in Object Tracking. 
  • ID Assignment (totally optional but it would prove you know what 2D Object Tracking is)
  • 2D association cost metrics such as the Mahalanobis distance or the Intersection Over Union (IOU).
The second half is optional, but highly recommended. This course has been built for an intermediate/advanced level, and therefore I won't reexplain every beginner concept.

And even though I included lots of reminders, in particular in Kalman Filters and other association techniques, enrolling without the prerequisites is not advised.

In fact, I recommend engineers who have never experimented with 2D Object Detection or Tracking to simply not enroll. I have courses to master these techniques too, and I recommend you go through them first.

Plus, if you have never done any kind of 2D Tracking... Why starting with 3D? 

About the duration and format:

This course isn't video only. Its always a hybrid of videos, text, stories, keynotes, papers, mindmaps, images, memes, code, notebook, interviews, and tutorials, ... I'm personally not a fan of video-only courses, especially when they're looooong. So I do the opposite and add lots of text, images, etc.. so your visual memory gets activated too.

🚀 For projects, the main format is notebooks and videos, all done on Colab with nothing to setup (yes! even for the 3D Visualizer)

About the duration: It probably depends on your prerequisites. If you validate the prerequisites, you'll be able to go fast. Still, 3D Object Detection and Tracking is HARD and in particular, the 3D Object Detection module will be very complete.
⏳ Plan somewhere between 10 and 15 hours (or more) to go through it.

You may or may not be "there" yet. How can you know?

While this course could totally change the career of many engineers, helping you totally stand out and be seen as a master of the Perception field; it can be totally optional for others.

It's important to understand which side you're on before taking it.

I recommend not to join if:
🔴 You don't validate the prerequisites and don't feel excited by the content shown above (obvious, but worth mentioning)
🔴 You're interested in Computer Vision and 2D only (this course won't focus on images much, it will be LiDAR and 3D oriented)
🔴  You don't plan to work on Real Physical Robots or 3D Computer Vision systems
🔴 You're judging a course by its duration, and expect 300+ hours of content — sorry, wrong website. If anything, I'm trying to help you finish as fast as possible, not to lose you in endless amount of content.

On the other hand, this course has been designed for you if:
🟢 You have some good prerequisites, and feel ready to experience the cutting-edge of Perception
🟢 You're excited by the idea of 3D Object Detection with Deep learning, and with sensors such as LiDARs and Cameras
🟢 You feel like something is "missing" to really be considered advanced and are ready to learn more about 3D and 4D systems used in the industry
🟢 One of your goal is to be considered an expert in Perception and Tracking systems, such as robots, drones, augmented reality, 3D Computer Vision, or 3D Medical Imaging.
🟢 You want to stand out, build a competitive edge, and this by learning skills most engineers don't know

If you identify in the second half, I'll give more precision about enrollment below.

One more question:

"What is the difference between this course, and your other tracking courses?"

The main difference is that this course focuses more on 3D Object Detection, Visualization, and Tracking than anything else. It's also the most advanced, which implies the skills acquired are more rare.
If you'd like to compare the two together, let's run a side-by-side comparison of 2D vs 3D Object Tracking, on the same sequence:

2D MOT

3D MOT

☄️

This 4D Perception course is unique, but it also has an incredible advantage...

Do you remember in Harry Potter 6 when Harry finds the book of the half-blood prince? Suddenly, he gets incredible grades at school, and is even the only student that wins the Felix Felicis potion. He becomes his teacher's favourite student...

He stands out!

This 4D Perception course is like the half-blood prince's book — with it, you're going to learn algorithms most of the others don't know, such as 3D Object Detection, 3D Fusion, and 3D Tracking, and you'll build your own 3D Visualizer, that will help you stand out.

But the biggest advantage? The visualizer!

There's a good underlying reason why most self-driving car companies share their work using 3D Visualizers and not just images. It's because Point Clouds and 3D Data is always more eye-catching to the human eye. This has been proved by hundreds of engineers and content creators on LinkedIn, even demonstrated in some posts.

And with this Visualizer, you will be able to build custom projects for this course... but also for all your other Perception courses! In fact, this visualizer is adaptable to other projects, such as aerial navigation and ground robotics.

But how much can you really stand out sharing a 4D Perception Project?
4D implies you know both 2D and 3D, it implies you've been paying attention to what the field is doing, and more importantly...

It gives you that credibility that can make the difference in an interview. It shows recruiter you're part of their world. 

On that note, below is the button to enroll:

375€ or 2 x 189€

MASTER 4D PERCEPTION: Ultimate 3D Tracking Skills for Perception Engineers

Learn 3D Object Detection, 3D Tracking, and build advanced 4D Perception Projects.

This course is now closed and will open Early 2026. Make sure to be on the waitlist to get notified and receive goodies while you wait.

Join Waitlist