b'David Harwath'

research

∙ 09/19/2023

AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models

Audio-visual representation learning aims to develop systems with human-...

0 Yuan Tseng, et al. ∙

research

∙ 06/27/2023

Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos

To realize human-robot collaboration, robots need to execute actions for...

0 Chiori Hori, et al. ∙

research

∙ 06/14/2023

When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants

We present the first unified study of the efficiency of self-attention-b...

0 Anuj Diwan, et al. ∙

research

∙ 05/24/2023

Unit-based Speech-to-Speech Translation Without Parallel Data

We propose an unsupervised speech-to-speech translation (S2ST) system th...

0 Anuj Diwan, et al. ∙

research

∙ 05/21/2023

Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages

Recent models such as XLS-R and Whisper have made multilingual speech te...

0 Andrew Rouditchenko, et al. ∙

research

∙ 05/19/2023

Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Mode

In this paper, we show that representations capturing syllabic units eme...

0 Puyuan Peng, et al. ∙

research

∙ 05/18/2023

Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization

We investigate the emergent abilities of the recently proposed web-scale...

0 Puyuan Peng, et al. ∙

research

∙ 12/03/2022

Unsupervised Fine-Tuning Data Selection for ASR Using Self-Supervised Speech Models

Self-supervised learning (SSL) has been able to leverage unlabeled data ...

0 Reem Gody, et al. ∙

research

∙ 12/02/2022

Continual Learning for On-Device Speech Recognition using Disentangled Conformers

Automatic speech recognition research focuses on training and evaluating...

0 Anuj Diwan, et al. ∙

research

∙ 11/02/2022

Phoneme Segmentation Using Self-Supervised Speech Models

We apply transfer learning to the task of phoneme segmentation and demon...

0 Luke Strgar, et al. ∙

research

∙ 11/02/2022

M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval

This work investigates the use of large-scale, pre-trained models (CLIP ...

0 Layne Berry, et al. ∙

research

∙ 11/01/2022

Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality

Recent visuolinguistic pre-trained models show promising progress on var...

0 Anuj Diwan, et al. ∙

research

∙ 10/07/2022

C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval

Multilingual text-video retrieval methods have improved significantly in...

0 Andrew Rouditchenko, et al. ∙

research

∙ 10/03/2022

SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model

Data-driven speech processing models usually perform well with a large a...

0 Yi-Jen Shih, et al. ∙

research

∙ 03/30/2022

MAE-AST: Masked Autoencoding Audio Spectrogram Transformer

In this paper, we propose a simple yet powerful improvement over the rec...

22 Alan Baade, et al. ∙

research

∙ 03/28/2022

Word Discovery in Visually Grounded, Self-Supervised Speech Models

We present a method for visually-grounded spoken term discovery. After t...

5 Puyuan Peng, et al. ∙

research

∙ 03/21/2022

Automated detection of foreground speech with wearable sensing in everyday home environments: A transfer learning approach

Acoustic sensing has proved effective as a foundation for numerous appli...

0 Dawei Liang, et al. ∙

research

∙ 02/07/2022

Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling

In this paper, we describe our submissions to the ZeroSpeech 2021 Challe...

0 Puyuan Peng, et al. ∙

research

∙ 12/08/2021

Everything at Once – Multi-modal Fusion Transformer for Video Retrieval

Multi-modal learning from video data has seen increased attention recent...

0 Nina Shvetsova, et al. ∙

research

∙ 12/01/2021

Routing with Self-Attention for Multimodal Capsule Networks

The task of multimodal learning has seen a growing interest recently as ...

0 Kevin Duarte, et al. ∙

research

∙ 11/08/2021

Cascaded Multilingual Audio-Visual Learning from Videos

In this paper, we explore self-supervised audio-visual models that learn...

0 Andrew Rouditchenko, et al. ∙

research

∙ 09/16/2021

Fast-Slow Transformer for Visually Grounding Speech

We present Fast-Slow Transformer for Visually Grounding Speech, or FaST-...

0 Puyuan Peng, et al. ∙

research

∙ 06/14/2021

Learning Audio-Visual Dereverberation

Reverberation from audio reflecting off surfaces and objects in the envi...

0 Changan Chen, et al. ∙

research

∙ 05/10/2021

Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions

When people observe events, they are able to abstract key information an...

4 Mathew Monfort, et al. ∙

research

∙ 04/26/2021

Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos

Multimodal self-supervised learning is getting more and more attention a...

0 Brian Chen, et al. ∙

research

∙ 12/31/2020

Text-Free Image-to-Speech Synthesis Using Learned Segmental Units

In this paper we present the first model for directly synthesizing fluen...

12 Wei-Ning Hsu, et al. ∙

research

∙ 06/16/2020

AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

Current methods for learning visually grounded language from videos ofte...

14 Andrew Rouditchenko, et al. ∙

research

∙ 11/21/2019

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech

In this paper, we present a method for learning discrete linguistic unit...

0 David Harwath, et al. ∙

research

∙ 07/09/2019

Transfer Learning from Audio-Visual Grounding to Speech Recognition

Transfer learning aims to reduce the amount of data required to excel at...

0 Wei-Ning Hsu, et al. ∙

research

∙ 02/21/2019

Towards Visually Grounded Sub-Word Speech Unit Discovery

In this paper, we investigate the manner in which interpretable sub-word...

0 David Harwath, et al. ∙

research

∙ 04/09/2018

Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech

In this paper, we explore the learning of neural network embeddings for ...

0 David Harwath, et al. ∙

research

∙ 04/04/2018

Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input

In this paper, we explore neural network models that learn to associate ...

2 David Harwath, et al. ∙

research

∙ 12/11/2017

Learning Modality-Invariant Representations for Speech and Images

In this paper, we explore the unsupervised learning of a semantic embedd...

0 Kenneth Leidal, et al. ∙

research

∙ 01/25/2017

Learning Word-Like Units from Joint Audio-Visual Analysis

Given a collection of images and spoken audio captions, we present a met...

0 David Harwath, et al. ∙

research

∙ 11/11/2015

Deep Multimodal Semantic Embeddings for Speech and Images

In this paper, we present a model which takes as input a corpus of image...

0 David Harwath, et al. ∙

David Harwath

Featured Co-authors