Piyush Bagad

Piyush Bagad

I am a DPhil student at the VGG Lab (Oxford), advised by Prof. Andrew Zisserman at the University of Oxford. I am interesting in multimodal video understanding and generation with a focus on temporal sensitivity.

Previously, I finished my Master’s in AI at the University of Amsterdam. Along with the coursework, I did research on video understanding at the VISLab advised by Prof. dr. Cees Snoek and Dr. Makarand Tapaswi.

In the past, I’ve been a Research Fellow at Wadhwani AI where I primarily worked on estimating infant anthropometry from a monocular video. I did my BS in Math at IIT Kanpur where I worked with Dr. Piyush Rai and Dr. Swaprava Nath.

Research

The central theme of my research is to better understand, model and utilize the temporal dynamics in video data beyond what can be inferred from a single frame. I have studied temporal structure in both low-level physical processes (recovering form-from-motion, inferring physical properties from pouring sounds, and estimating hand-object contact forces) and high-level action semantics (learning time-sensitive video representations and improving retrieval for temporal actions).

Recognition by Imagination: A Generative Approach to Form-from-Motion

Under review

Piyush Bagad, Makarand Tapaswi, Andrew Zisserman

Inspired by Gunnar Johansson’s experiments (Johansson, 1973), we train a video-to-video generation model to map sparse keypoints to dense RGB. The same model generalizes to other psychophysics tests such as the Hidden Bird Illusion.

Forces at your fingertips: Estimating Hand-Object Contact Forces using Newtonian Physics

Under review

Piyush Bagad, Taein Kwon

Forces at your fingertips: Estimating Hand-Object Contact Forces using Newtonian Physics — research illustration

Given 3D hand and object meshes, we use Newton’s laws (free-body diagrams from high school physics) to optimize the distribution of forces on the hand in equilibrium. This is a work-in-progress project with the ultimate goal of estimating hand-object contact forces from a single image.

TARA: Text Adapted Retrieval Alignment for Nuanced Video Retrieval

Piyush Bagad, Andrew Zisserman

We post-train a multimodal LLM (MLLM) for text-to-video retrieval on nuanced, temporally sensitive queries (e.g., folding vs unfolding paper) with only using text data. We show the embeddings from this model are more time-sensitive that prior work.

Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening

Piyush Bagad, Andrew Zisserman

We discover that per-frame DINOv2 embeddings for a video lie on a smooth time-aware manifold. We exploit this to train an adapter called Linearized Feature Trajectories (LiFT) inspired by perceptual straightening hypothesis. LiFT produces time-sensitive, compact video embeddings with a simple self-supervised objective.

The Sound of Water: Inferring Physical Properties from Pouring Liquids

ICASSP 2025 and TPAMI 2025

Piyush Bagad, Makarand Tapaswi, Cees G. M. Snoek, Andrew Zisserman

We discover that the simple sounds of pouring water contain rich structure which connects to the physical properties of the glass-liquid system. We use this to train a model which predicts physical properties (e.g., size, radius of glass) merely from the sound of pouring.

Test of Time: Instilling Video-Language Models with a Sense of Time

Piyush Bagad, Makarand Tapaswi, Cees G. M. Snoek

Test of Time: Instilling Video-Language Models with a Sense of Time — research illustration

At the time, we discover that standard video-text models did not accurately encode simple temporal relations (e.g., before vs after). We fine-tune a video-text model in a CLIP like manner with carefully designed hard-negatives to instill this sense of time.

How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?

ECCV, 2022 (Tel Aviv, Israel)

Fida Mohammad Thoker, Hazel Doughty, Piyush Bagad, Cees Snoek

How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning? — research illustration

We carefully studied various contemporary video benchmarks and models and found that standard benchmarks like Kinetics400/UCF101 are not representative of generalization. Based on our insights, we proposed a new benchmark, SEVERE, which systematically evaluates generalization along four axes of variation.

C-3PO: Towards Rotation Equivariant Feature Detection and Description

Piyush Bagad*, Floor Eijkelboom*, Mark Fokkema*, Danilo de Goede*, Paul Hilders*, Miltiadis Kofinas

C-3PO: Towards Rotation Equivariant Feature Detection and Description — research illustration

When an image rotates, its keypoints should rotate with it and remain easy to match. We build this rotation equivariance directly into a feature detector and descriptor using steerable convolutions, improving matching over its non-equivariant counterparts for most rotations.

Reproducibility Study of "Counterfactual Generative Networks"

Piyush Bagad*, Paul Hilders*, Jesse Maas*, Danilo de Goede*

Reproducibility Study of "Counterfactual Generative Networks" — research illustration

We reproduce a generative model that independently controls object shape, texture and background to create counterfactual images (e.g., the same object against a different background). We largely confirm that training classifiers on these images improves generalization beyond spurious cues, while identifying reproducibility issues and cases where the model fails to separate these factors.

Invited to present at Journal Showcase Poster Session @ NeurIPS 2022.

Quantifying CLIP’s ability to Perform Cross-Modal Grounding Using Attention-Model Explainability

Best poster award in the Interpretability course at UvA

Piyush Bagad*, Paul Hilders*, Danilo de Goede*

Quantifying CLIP’s ability to Perform Cross-Modal Grounding Using Attention-Model Explainability — research illustration

We use transformer attribution maps to test whether CLIP connects words in a caption to the relevant regions of an image. Comparing these maps with annotated object regions provides evidence that CLIP learns some fine-grained grounding, beyond simply matching whole images and captions.

Industry

I have had experience of working in the industry on a variety of projects. At Adobe Research (India), I developed a method to generate synthetic tabular data with a controllable privacy-accuracy trade-off. At Wadhwani AI (India), I worked on several projects applying AI to important problems in heathcare including newborn anthropometry and sound-based disease diagnosis.

Data-Sharing Economy: Value-Addition from Data meets Privacy

WSDM Demos, 2021 (Virtual event, Israel)

Piyush Bagad, Subrata Mitra, Sunny Dhamnani, Atanu R Sinha, Raunak Gautam, Haresh Khanna

Data-Sharing Economy: Value-Addition from Data meets Privacy — research illustration

Work done during Internship at Adobe Research (India) in 2018. We built a Generative Adversarial Network (GAN)-based framework to generate synthetic tabular data with a controllable privacy-accuracy trade-off.

Cough Against COVID: Evidence of COVID-19 Signature in Cough Sounds

ArXiv, 2021 (Mumbai, India)

Piyush Bagad, Aman Dalmia, Jigar Doshi, Arsha Nagrani, Parag Bhamare, Amrita Mahale, Saurabh Rane, Neeraj Agarwal, Rahul Panicker

Cough Against COVID: Evidence of COVID-19 Signature in Cough Sounds — research illustration

Media credits: Wadhwani AI

We built a cough sound classification model that can detect COVID-19 symptoms from cough sounds. We collected a large-scale dataset of cough sounds from COVID-19 patients and healthy individuals to train the model. We fine-tuned CNN-based audio models that are robust to confounders like recording location, etc.

Newborn Anthropometry: Estimating infant weight from a monocular video

Wadhwani AI, 2019–2021 (Mumbai, India)

Piyush Bagad, Aman Dalmia, Makarand Tapaswi, Sansiddh Jain, Aditya A Sarma, Namrata Deka, Jigar Doshi, Parag Bhamare, Amrita Mahale, Rahul Panicker (Not in order)

Newborn Anthropometry: Estimating infant weight from a monocular video — research illustration

Media credits: Wadhwani AI

We built a model that can estimate the weight of a newborn baby from a single video. As of September 2026, the deployed Shishu Maapan application has supported 26K Home-Based Newborn Care (HBNC) visits, registered 13K newborns, and is used by 471 ASHA workers.

The subsequent NurtureNet paper received the Best Paper Award at the CVPM workshop at CVPR 2024.

Synthetic Data Generation for Visual Weighing Machine

Platform: Blender 2.8

Piyush Bagad, Aman Dalmia, Siddharth Singh, Vishal Gupta, Aditya AS, Namrata Deka

Synthetic Data Generation for Visual Weighing Machine — research illustration

Copyrights: Wadhwani AI

We used the 3D deformable model to generate a large-scale synthetic dataset to train models to estimate the weight of a newborn baby from a single video.

Deformable 3D Model for Newborn Babies

Underlying Model: SMIL (Nicholas Hesse et. al)

Piyush Bagad, Aman Dalmia, Rahul Kumar, Arsh Verma, Rahul Panicker

Deformable 3D Model for Newborn Babies — research illustration

Copyrights: Wadhwani AI

We built a deformable 3D model for Indian newborn babies based on the SMIL (European) model. We collected a large-scale 4D scan dataset, registered the dataset to SMIL model, and then trained a new model for Indian context.

Miscellaneous

News

Mar 5, 2023
A shorted version of our paper got accepted to ME-FoMo Workshop ICLR 2023!
Feb 27, 2023
Our paper on time in video-language models got accepted to CVPR 2023!
Jul 1, 2022
Our paper got accepted to ECCV 2022!
Apr 10, 2022
Our paper recieved the Best Paper Award at MLRC 2021!
Jul 16, 2021
Today is my last day at Wadhwani AI after completing incredible 2 years.

I am always available on my personal email.