1.     Emotional-Social Intelligence Toolkit (current)

2.     Computerized Interventions to Promote Verbal Expression in Individuals with Autism Spectrum Disorder (current)

3.     Dialogue Act Classification using Prosody, Discourse and their Fusion

4.     Robust Recognition of Emotion from Speech

5.     Multimodal Communications

6.     Captchas from 3d models to 2d images

7.     Automated Analysis of Microstructures

8.     VisionPSU

Emotional-Social Intelligence Toolkit (current)

Many people on the autism spectrum understand the semantics involved in social interaction; however, embodied information such as facial expressions, gestures, and voice often prove elusive. First-hand accounts from people with autism highlight the challenges inherent in processing these complex and unpredictable social cues. These challenges can be debilitating, complicating social interaction and making integration with society difficult. While many intervention methods have been developed to provide help, the majority fail to include rich, real-world social interactions in their methodology.

 Our goal is to develop a technology-based intervention that helps individuals on the autism spectrum capture, analyze, systemize, and reflect on social-emotional signals communicated by facial and head movements in natural, everyday social interactions. Our approach utilizes an ultra-mobile computer customized with a video camera and pattern analysis algorithms that can automatically identify facial expressions using facial feature tracking. In an effort to make our system robust to real-world conditions and usable by individuals with cognitive, motor, and sensory impairments, we have engaged in a number of user-centered design sessions with people on the autism spectrum and their caregivers.

An experimental evaluation of our redesigned system is forthcoming to determine if just-in-time, in-situ assistance can help facilitate learning of facial expressions and underlying emotions for persons on the autism spectrum.

Feel free to visit the project website.

Computerized Interventions to Promote Verbal Expression in Individuals with Autism Spectrum Disorder (current)

PI and Co-PIs: Rosalind Picard, Rana Kaliouby, Matthew Goodwin, Graduate Research Assistant: M. Ehsan Hoque, UROP: Joseph Lane

Approximately one third to one half of individuals on the autism spectrum have significant difficulty using speech and language as an effective means of communication. Their difficulties in speech production and processing can result in interpersonal interactions being overwhelming, confusing, and stressful. Also, these difficulties often get misinterpreted as a general disinterest in social interaction.

The goal of this project is to introduce novel technology that automatically identifies areas of speech production difficulties such as loudness and speech rate, and provides an interactive interface for individuals to make an attempt to rectify it. This study is motivated by our hypothesis that real time visualizations of speech properties, which often act as social mirrors, can influence social communication. Our objective is not to replace the traditional speech therapist with this tool. Instead, we propose this tool being an easily customizable and freely available supplementary therapy to help individuals with speech difficulties.

Publications:

1. M. E. Hoque, Analysis of Speech Properties of Neurotypicals and Individuals Diagnosed with Autism and Down Syndrome, 10th ACM conference on Computers and Accessibility (ASSETS), Halifax, Nova Scotia, October, 2008. [Finalist of the Student Research Competition] (PDF: 40 KB) (Poster: 518 KB)

Dialogue Act Classification using Prosody, Discourse and their Fusion

This work investigates the automatic dialogue acts classification in multimodal communication using prosody, discourse features, and their fusion. From an experiment investigating multimodal communication, eight hours of natural audio data was collected. Prosodic and discourse features, which were believed to be strong correlates of dialogue acts, were extracted and the best features were selected using a combination of feature selection algorithms. A variety of classifiers, including traditional and ensemble, were designed and evaluated on a dialogue act classification to compare their performance. The results show that the ensemble feature selection based classification performs consistently across all the models with high validation scores. The final results demonstrated 55% accuracy on classifying 14 dialogues based on prosody, 75% accuracy with discourse and 74% with their fusion, for the best classifier. The unexpected reduction of performance due to fusion is possibly due to the lack of proper normalization of data coming from two different sources and is subject to further exploration.  

Details: to be added

Publications:

1. M. E.  Hoque, M. S.  Sorower, M. Yeasin, M. M. Louwerse,  What Speech Tells us about Discourse: The Role of Prosodic and Discourse Features in Dialogue Act Classification, IEEE International Joint Conference on Neural Networks (IJCNN), Orlando, Florida, 2007. (PDF: 138 KB)

2. M. E. Hoque, What Speech Tells us about Discourse: The Role of Prosodic and Discourse Features in Dialogue Act Classification, MS Thesis, May 2007. (PDF: 552 KB)

Robust Recognition of Emotion from Speech

This project investigates robust recognition of selected emotions from salient spoken words. The prosodic and acoustic features were used to extract the intonation patterns and correlates of emotion from speech samples in order to develop and evaluate models of emotion. The computed features are projected using a combination of linear projection techniques for compact and clustered representation of features. The projected features are used to build models of emotions using a set of classifiers organized in hierarchical fashion. The performances of the models were obtained using number of classifiers from WEKA tools. Results showed that the lexical information computed from both the prosodic and acoustic features at word level yielded robust classification of emotions.  

Details: Here

Publications:

1) M. E. Hoque, M. Yeasin, M. M. Louwerse. Robust Recognition of Emotion from Speech, 6th International Conference on Intelligent Virtual Agents, Marina Del Rey, CA, August 2006. [Nominated for Best Paper Award] (PDF: 94 KB)

2) M. E. Hoque, M. Yeasin, M. M. Louwerse. Robust Recognition of Emotion in e-Learning Environment.18th Annual Student Research Forum, Memphis, TN April, 2006. [Best Poster Award] (PPT: 415 KB)

Multimodal Communications

In this project we investigate multimodal communications between humans, monitoring linguistic and non-linguistic modalities like prosody, dialogue structure, gesture, body posture, facial expressions and eye gaze. These findings are then used to develop an agent that can take the role of one of the humans. To constrain the domain in which modalities are used we use the Map Task scenario, as we think in this setting, conversation could be well encouraged (Anderson et al., 1991). In this scenario two participants coordinate the route on a map. One dialogue participant (Instruction Giver) gives the directions to the other participant who follows these directions (Instruction Follower). There has are differences between the Instruction Giver’s and Instruction Follower’s map to elicit discourse. The research results will benefit a large variety of fields, including cognitive science, computational linguistics, artificial intelligence, and computer science. In addition, the integration of the modalities into a working model will advance the development and use of intelligent conversational system.

Details: Here

Publications:

1) M. M. Louwerse, N. Benesh, M. E. Hoque, P. Jeuniaux, G. Lewis, J. Wu, M. Zirnstein, Multimodal Communication in Face-to-Face Conversations, the 29th meeting of Cognitive Science Society, Nashville, TN, 2007. (PDF: 608 KB)

2) M. Louwerse, P. Jeuniaux, M. Hoque, J. Wu, G. Lewis. Multimodal Communication in Computer-Mediated Map Task Scenarios The 28th Annual Conference of the Cognitive Science Society, Vancouver, Canada, July 2006. (PDF: 304 KB)

3) P. Jeuniaux, M. Louwerse, J. Wu, M. Hoque, (2005). Embodied Conversational Agents: Multimodal Communication and Embodiment. The Garachico workshop, Symbols, Embodiment and Meaning : A Debate. December 15-19, Tenerife, Spain.

Captchas from 3d models to 2d images

Networked computers are susceptible to cyber attacks by software robots and spiders posing as humans to gain access to and possibly abuse systems intended for human use only. Prevention of such attacks has motivated the study of a new security protocol called Captchas: Completely automated public tests to tell computers and humans apart.

In this project, a different image based Captcha was developed and prototyped to address the mislabeling and other shortcomings of traditional schemes, which utilize huge public image databases.

Details: Here (yet to be added)

Publications:

1) M. Hoque, D. Russomanno, M. Yeasin. 2D Captchas from 3D Models IEEE SoutheastCon 2006 Memphis, TN, April 2006. (PDF: 715 KB)

Automated Analysis of Microstructures

Automated Analysis of Microstructures - Aluminum and tungsten carbide - for improved tool life.         

Details: Here

Publications:

1) Mohammed E. Hoque, Ralph M. Ford, and John T. Roth. Automated Image Analysis of Microstructure Changes in Metal Alloys SPIE Conference on Industrial Applications of Machine Vision XVIII San Jose, CA, January 2005. (PDF)

2)1. Joseph S. Andrawes, Jarred C. Heigel, John T. Roth, Mohammed E. Hoque, Ralph M. Ford Determining the Viability of Electrically Treating 6061 T6511 Aluminum for Use in Manufacturing Processes, Transactions of the North American Manufacturing Research Institute of the Society of Manufacturing Engineers, NAMRI/SME, May 2005. (PDF)