Research Summary
Participants: Smita Vemulapalli (Lead) and Monson H. Hayes
To meet the educational needs of people at different locations, with different study and work schedules, most schools and universities offer distance learning options. The most popular media for distance learning is video for various reasons, especially while teaching complex mathematical and scientific concepts. This research addresses the challenges involved in recognizing mathematical content written on the whiteboard in a classroom video.
We have designed and implemented a framework for a system that uses both the audio and video components of the classroom video to recognize the mathematical content on the whiteboard of a classroom video. Text extraction and segmented have been implemented on real classroom videos using features such as edges, average intensity and frame difference, followed by connected component labeling. We currently simulate a math character recognizer by using manually recognized math content with the addition of errors and uncertainty. We deploy an audio-video synchronization strategy to find the most likely audio segment match and then resolve the ambiguity in the text recognizer output by using a combination of the video (text recognizer) and audio (Nexidia word spotting) confidences and other features. This combination approach has shown that the purely text-based recognition accuracy may be significantly improved by making use of the audio information.
On successful completion of the audio-video based character recognition stage, we will move onto the structure analysis stage which makes the problem much more challenging than regular text recognition. This analysis of the structure of equations would make use of the logical and spatial relationships between recognized symbols and once again, we intend to use audio information to resolve ambiguity.
System Overview
Papers
Conference Papers
- S. Vemulapalli and M.H. Hayes, “Using Audio-Based Disambiguation for Improving Handwritten Mathematical Content Recognition in Classroom Videos,” Ninth IAPR International Workshop on Document Analysis Systems, Boston, MA., June 2010.
- S. Vemulapalli and M.H. Hayes, “Ambiguity Detection Methods for Improving Handwritten Mathematical Character Recognition Accuracy in Classroom Videos," 17th International Conf. of Digital Signal Processing (DSP2011), Corfu, Greece, July 2011.
- S. Vemulapalli and M.H. Hayes, “Towards Audio-Video Based Handwritten Mathematical Content Recognition in Classroom Videos,” Proc. 2011 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, Victoria, Canada, August 2011.
- S. Vemulapalli and M.H. Hayes, “Synchronization and Combination Techniques for Audio-videoBased Handwritten Mathematical Content recognition in Classroom Videos,” Proc. 11th International Conference on Intelligent Systems Design and Applications, Cordoba, Spain, November 2011.
