An Integrated Approach for Video Captioning and Applications

by   Soheyla Amirian, et al.

Physical computing infrastructure, data gathering, and algorithms have recently had significant advances to extract information from images and videos. The growth has been especially outstanding in image captioning and video captioning. However, most of the advancements in video captioning still take place in short videos. In this research, we caption longer videos only by using the keyframes, which are a small subset of the total video frames. Instead of processing thousands of frames, only a few frames are processed depending on the number of keyframes. There is a trade-off between the computation of many frames and the speed of the captioning process. The approach in this research is to allow the user to specify the trade-off between execution time and accuracy. In addition, we argue that linking images, videos, and natural language offers many practical benefits and immediate practical applications. From the modeling perspective, instead of designing and staging explicit algorithms to process videos and generate captions in complex processing pipelines, our contribution lies in designing hybrid deep learning architectures to apply in long videos by captioning video keyframes. We consider the technology and the methodology that we have developed as steps toward the applications discussed in this research.


page 2

page 6

page 9

page 11


End-to-End Video Captioning with Multitask Reinforcement Learning

Although end-to-end (E2E) learning has led to promising performance on a...

Deep Learning for Video Classification and Captioning

Accelerated by the tremendous increase in Internet bandwidth and storage...

Automatic Generation of Descriptive Titles for Video Clips Using Deep Learning

Over the last decade, the use of Deep Learning in many applications prod...

Dense-Captioning Events in Videos

Most natural videos contain numerous events. For example, in a video of ...

Text with Knowledge Graph Augmented Transformer for Video Captioning

Video captioning aims to describe the content of videos using natural la...

The Use of Video Captioning for Fostering Physical Activity

Video Captioning is considered to be one of the most challenging problem...

Oracle performance for visual captioning

The task of associating images and videos with a natural language descri...

Please sign up or login with your details

Forgot password? Click here to reset