Meta FAIR Computer Vision Interview: Object Detection, Image Segmentation, and Video Understanding

Background

Let me start with my background: automation undergrad, switched to computer vision for my master's, then spent 2 years as a CV algorithm engineer at an autonomous driving company, mainly working on object detection and semantic segmentation. Meta FAIR has always been one of the pinnacles of CV research — I'd read most of their papers — so when I saw the opening, I applied without hesitation.

I applied for the Computer Vision Researcher position at Meta FAIR, based in Menlo Park. The whole interview process took about three weeks — three technical rounds, each with considerable depth. Honestly, FAIR's interview style is quite different from business unit interviews — it's more research-oriented, digging deep into your understanding of method fundamentals. Let me walk through the details.

Interview Process Review

Round 1: CV Fundamentals + CNN/Transformer

My first interviewer was a young PhD, likely recently graduated. Started with self-introduction, then moved to CV fundamentals.

First question: Why do ResNet's residual connections work? I gave a comprehensive answer: from the gradient propagation perspective, residual connections provide a "highway" for gradients, mitigating vanishing gradients; from the optimization perspective, residual mapping is easier to learn than identity mapping; from the ensemble perspective, ResNet can be viewed as an implicit ensemble of paths at different depths. The interviewer asked what happens if you remove residual connections — I said deep networks would experience degradation, with training error actually higher than shallower networks.

Then Transformer applications in CV: How does Vision Transformer's patch embedding work? Why does it work? I explained that ViT cuts images into fixed-size patches (e.g., 16x16), linearly projects them into token sequences, adds positional encoding, and feeds them into a standard Transformer. It works because large-scale pre-training data compensates for the lack of inductive bias, but it underperforms CNNs on small datasets. The interviewer asked how DeiT solves the small data problem — I mentioned knowledge distillation and stronger data augmentation.

A classic question: What's the difference between anchor-based and anchor-free object detection? Pros and cons? I said anchor-based methods (Faster R-CNN, YOLOv5) require preset anchors, are sensitive to hyperparameters but more stable; anchor-free methods (FCOS, CenterNet) directly predict points or centers, are cleaner but potentially less stable during training. The interviewer asked about ATSS's adaptive anchor selection — I explained the statistics-based adaptive strategy in detail.

A practical question: If you need to train an object detection model with only 1000 annotated images, what would you do? I mentioned several strategies: pre-trained model fine-tuning (COCO pre-training), data augmentation (Mosaic, MixUp, CopyPaste), semi-supervised learning (pseudo-label expansion), and few-shot learning methods. The interviewer was interested in CopyPaste augmentation and asked for implementation details.

Round 1 lasted about 50 minutes. The interviewer said "solid fundamentals" and told me to wait for Round 2.

Round 2: Object Detection + YOLO/DETR

Round 2's interviewer was clearly more senior, with questions leaning toward frontier research and depth of thinking.

Started with the YOLO series: From YOLOv1 to YOLOv8, what do you think are the most important improvements? I highlighted key milestones: YOLOv2's anchor mechanism, YOLOv3's multi-scale detection, YOLOv4's CSPNet and Mosaic augmentation, YOLOv5's auto hyperparameters, YOLOX's anchor-free design and decoupled head, YOLOv8's distributed focal loss. The interviewer asked why CSPNet speeds things up — I said it reduces computation through cross-stage partial connections while maintaining feature reuse.

Then the DETR series: Why does DETR converge slowly? How does Deformable DETR solve this? I explained that DETR converges slowly because global attention pays too-uniform attention to each position in early training, making it hard to focus on key regions. Deformable DETR uses deformable attention to only attend to a small number of sampling points near reference points, reducing attention from global O(n²) to O(nk), both accelerating convergence and reducing computation. The interviewer asked about DAB-DETR's improvements — I said it explicitly introduces anchor position information as queries, further accelerating convergence.

An open-ended question: What do you think is the future direction of object detection? I mentioned three: end-to-end detection (continued DETR evolution), open-vocabulary detection (detecting arbitrary categories beyond training set limits), and 3D/video detection. The interviewer was very interested in open-vocabulary detection, and we discussed OWL-ViT and Grounding DINO approaches.

A practical question: What are good solutions for small object detection? I mentioned multi-scale feature fusion (FPN, BiFPN, PANet), high-resolution input, slice-assisted inference (SAHI), and specialized small object data augmentation. The interviewer asked about FPN's feature fusion approach — I detailed the top-down upsampling fusion and lateral connections.

Round 2 lasted about 60 minutes — a thrilling conversation.

Round 3: Image Segmentation + Video Understanding + Project Deep Dive

Round 3 was with a senior researcher at FAIR — definitely more pressure. This round focused on segmentation, video understanding, and projects.

Started with image segmentation: What are the differences between semantic segmentation, instance segmentation, and panoptic segmentation? I said semantic segmentation is pixel-level classification without distinguishing same-class instances; instance segmentation distinguishes each instance but doesn't handle background; panoptic segmentation unifies both, classifying pixels while distinguishing instances. The interviewer asked how Mask2Former unifies the three segmentation tasks — I said it uses a unified mask classification paradigm, handling semantics and instances through different queries.

Then video understanding: What's the core difference between video understanding and image understanding? I said the core difference is temporal modeling — videos have temporal dependencies requiring modeling of inter-frame motion and changes. Method-wise, early work used 3D CNNs (C3D, I3D), later temporal attention (TimeSformer, ViViT), and now dual-stream architectures with temporal modules are mainstream. The interviewer asked about VideoMAE's self-supervised pre-training — I said it randomly masks out a large proportion of tubes (spatiotemporal patches) and then reconstructs them, forcing the model to learn spatiotemporal representations.

Deep project dive: How do you handle occlusion and truncation in autonomous driving object detection? I mentioned several strategies: data-level occlusion augmentation simulation, feature-level contextual reasoning to complement occluded parts, and post-processing using soft NMS variants to avoid false deletions. The interviewer asked what to do when severe occlusion makes objects completely invisible — I said temporal information can help: the current frame might not show it but adjacent frames might, using tracking algorithms for association.

A research direction question: What important unsolved problems remain in CV? I mentioned several: 3D understanding (reasoning from 2D to 3D), long video understanding (beyond short video clips), physical world understanding (understanding objects' physical properties and interactions), and generalization of CV foundation models. The interviewer was very interested in physical world understanding, saying it's also a direction they're exploring.

Round 3 lasted about 55 minutes. The interviewer said "welcome aboard" at the end and told me to wait for HR.

Key Questions Summary

1. Why do ResNet's residual connections work?

2. How does ViT's patch embedding work? Why does it work?

3. Differences between anchor-based and anchor-free detection?

4. Strategies for training object detection with small datasets?

5. Most important improvements in the YOLO series?

6. Why does DETR converge slowly? How does Deformable DETR solve this?

7. Future directions for object detection?

8. Solutions for small object detection?

9. Differences between semantic, instance, and panoptic segmentation?

10. Core difference between video and image understanding?

11. VideoMAE's self-supervised pre-training method?

12. Handling occlusion and truncation in autonomous driving?

13. Important unsolved problems in CV?

Insights and Advice

1. Know not just what, but why: FAIR's interviews don't test how many methods you've memorized — they test how deeply you understand them. "Why does ResNet work?" and "Why does DETR converge slowly?" — these "whys" matter more than the "whats."

2. Follow method evolution: From YOLOv1 to v8, from DETR to DAB-DETR — interviewers like seeing if you can connect methods and understand the logic behind improvements.

3. Practical experience matters: Small data training, small object detection, occlusion handling — these practical questions are hard to answer well without project experience.

4. Frontier awareness: Open-vocabulary detection, 3D understanding — interviewers value whether you follow field development trends.

5. Cross-domain knowledge: CV-NLP intersection (like CLIP), CV-3D intersection — cross-domain knowledge is a bonus.

FAQ

Q: How do FAIR interviews differ from Meta business unit interviews?
A: FAIR is more research-oriented, digging into method principles; business units are more engineering-focused, concerned with deployment and performance optimization.

Q: Do I need top conference papers?
A: Not a hard requirement, but papers are definitely a plus. Research thinking and deep understanding matter more.

Q: Is CV still worth getting into?
A: Yes. Competition is fierce, but CV application scenarios (autonomous driving, robotics, AR/VR) continue to expand.

Q: Will there be coding in the interview?
A: Yes — Round 1 has algorithm coding questions, Round 2 might ask you to write key model code.

Q: What's the work atmosphere at FAIR?
A: Academic-leaning, high freedom, encourages publishing. But there's still output pressure — it's not a pure ivory tower.