Scene Understanding and Threat Recognition System

• Built an end-to-end pipeline that ingests multi-object tracking outputs and leverages Video-LLMs for complex-scene understanding and per-object threat scoring.
• Led a 3-person project; authored and executed a one-year roadmap (versioning, scope per release, task allocation) to deliver the project.
• Benchmarked multimodal models from 3B–78B (Qwen2.5-VL, LLaVA-OV, etc.) on multiple datasets, comparing accuracy and runtime/latency to select deployment candidates.

Multi-Object (Multi-Sensor) Tracking

• Developed a tracking solution that increased precision from 72% (naive off-the-shelf YOLO+DeepSORT pipeline) to 93% using a customized pipeline.
• Integrated an ensemble of video classification models (MViT, TimeSFormer, etc.) focused on reducing false alarm rates.
• Implemented a three-sensor tracking system (RGB, SWIR, MWIR) to ensure robust performance across different imaging modalities.

Visualization of simple basic off-the-shelf multi-object tracking in RGB (Vis) sensor without the full pipeline (due to IP and confidentiality).

Zero-Shot Keypoint Tracking

• Explored zero-shot tracking for keypoints in infrared videos characterized by extreme noise conditions.
• Utilized simulative data generated with UE5 specifically for testing under high interference scenarios.
• Rigorously evaluated SOTA models to determine limitations and ensure their suitability under adverse conditions.

Visualization of tracking keypoints on a passenger airplane from standard RGB (no noise) video (due to IP and confidentiality).

Video Restoration and Super Resolution

• Investigated and compared multiple SOTA models, including a self-designed architecture and a self-trained model.
• Applied knowledge distillation from a FeMaSR teacher model to optimize the chosen architecture.
• Achieved performance improvements of 5–20% over classical methods, as measured by metrics such as MTF, SSIM, and ESF.

Demonstration of video restoration and turbulence mitigation. This output is from the DATUM model (Zhang et al., CVPR 2024), as my own work's data is confidential.

Video Classification for False Target Filtering

• Designed a comprehensive data management infrastructure for experimental model training and evaluation.
• Developed tools for dynamic data integration, version management, attribute-based filtering, and complex augmentations (e.g., smart copy-paste and 3D rotations relative to the camera plane).
• Conducted hundreds of training experiments across both lightweight and heavy SOTA architectures, rigorously evaluating each component (data version, augmentations, architecture, transfer learning, etc.).
• Increased overall accuracy by 22%.