The Sound of Water: Inferring Physical Properties from Pouring Liquids

Piyush Bagad1,   Makarand Tapaswi2,   Cees G. M. Snoek3,   Andrew Zisserman1
1University of Oxford   2IIIT, Hyderabad   3University of Amsterdam
📋 arXiv Code 🤗 Data 🤗 Models Demo

drawing

The key nugget: As water is poured, the fundamental frequency (in green) that we hear changes predictably over time as a function of physical properties (e.g., container dimensions). We exploit this insight to (i) detect pitch in the audio signal, and (ii) infer physical properties of the pouring action.

🔭 Quick Preview

  • The physics of liquid pouring. We show that inferring the pitch of sound of pouring lets us infer physical properties of the container-liquid system.
  • Pitch detection without manual supervision. We train wav2vec2 model to detect pitch using supervision from (i) simulated data, (ii) co-supervision from visual stream.
  • What does this achieve? We are able to estimate properties such as container height, radius, length of air column precisely. We also generalise strongly to other datasets and samples from YouTube.

⚛️ The Physics of Pouring

Have you ever wondered by we can still pouring water in a glass without spilling even in the dark? It turns out that the sound of pouring water is intricately related to the height of water at a given time.

drawing

🎼 Pitch Detection in Pouring Sounds

Okay, so how do we detect pitch in the sound of pouring water? We draw inspiration from music literature and adapt `wav2vec2` model to detect pitch in the audio signal.

drawing

But we do not have supervision. How should we train it? We use (i) simulated data, and (ii) co-supervision from visual stream to train the model.

drawing

🗳 Estimating Physical Properties and Beyond

We achieve strong results on physical property estimation while learning meaningful representations. We also show strong generalisation to other datasets and samples from YouTube.

drawing

Kindly refer to our paper for more details.

📁 Dataset: Sound Of Water 50

A key contribution of our work is collection of a clean, large datasets of videos of pouring water. We collect a dataset of 805 clean videos that show the action of pouring water in a container. Our dataset spans over 50 unique containers made of 5 different materials, 4 different shapes and with hot and cold water. Some example containers are shown below.

drawing

🙏 Acknowledgements

📜 Citation


      @article{sound_of_water_bagad,
        title={The Sound of Water: Inferring Physical Properties from Pouring Liquids},
        author={Bagad, Piyush and Tapaswi, Makarand and Snoek, Cees G. M. and Zisserman, Andrew},
        journal={arXiv},
        year={2024}
      }

📙 Related Work

Please also consider looking at the following related papers that we were inspired by: