{"id":7448,"date":"2017-01-26T10:27:00","date_gmt":"2018-05-30T14:35:00","guid":{"rendered":"https:\/\/mediax.stanford.edu\/hmi-bailenson\/"},"modified":"2018-11-15T08:48:15","modified_gmt":"2018-11-15T16:48:15","slug":"hmi-bailenson","status":"publish","type":"page","link":"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/","title":{"rendered":"Detection of Comprehension & Emotion from Real-time Video Capture of Facial Expressions"},"content":{"rendered":"

From The Theme<\/em>
\nHUMAN MACHINE INTERACTION AND SENSING<\/a><\/strong><\/p>\n

WHAT IF?<\/strong>
\nWhat if we could automatically detect how well someone is comprehending information presented on a computer screen, based on the emotions they display?<\/p>\n

\"\"<\/p>\n

WHAT WE SET OUT TO DO<\/strong>
\nWe set out to assess the effects of emotion on comprehension, using educational systems already in place at Stanford\u2019s Virtual Human Interaction Laboratory. The four part strategy included: 1) tracking a learner\u2019s face during a comprehension task; 2) capturing facial expressions and categorizing the learner\u2019s emotions; 3) investigating how emotions effect comprehension; and finally; 4) measuring the effects of changing the learning system in real time, based on learner emotions.<\/p>\n

WHAT WE FOUND
\n<\/strong>We used machine learning algorithms to develop automated, real-time models that analyzed subjects\u2019 emotional states based on facial expressions and physiological measurements. We created two types of evaluations – models specific to the subject, as well as general models. Our approach predicted emotion type (amusement versus sadness) and intensity level, with immediate comparisons against trained coders\u2019 assessments.<\/p>\n

Results demonstrated good fits for the models overall, with better performance for emotion categories than for emotion intensity. Amusement ratings were more accurate than sadness ratings. The full model using physiological measures as well as facial tracking performed better than facial tracking alone, and our person-specific models performed better than general models.<\/p>\n

LEARN MORE<\/strong>
\nKizilcec, R.F., Bailenson, J.N., & Gomez, C.J. (2015). \u201cThe Instructor\u2019s Face in Video Instruction: Evidence from Two Large-Scale Field Studies<\/a>.\u201d Journal of Educational Psychology<\/em>. 107(3), 724-739.<\/p>\n

Jabon, M.E., Ahn, S.J., & Bailenson, J.N. (2011). \u201cAutomatically Analyzing Facial-Feature Movements to Identify Human Errors.\u201d IEEE Journal of Intelligent Systems, 26<\/em> (2), 54-63.<\/p>\n

Jabon, M.E., Bailenson J.N., Pontikakis, E.D., Takayama, L., & Nass, C. (2011). Facial Expression Analysis for Predicting Unsafe Driving Behavior. IEEE Pervasive Computing, 10 (4), 84-95.<\/p>\n

Ahn, S.J., Bailenson, J.N., Fox. J, & Jabon, M.E. (2010). \u201cUsing Automated Facial Expression Analysis for Emotion and Behavior Prediction.\u201d In Doeveling, K., von Scheve, C., & Konjin, E. A. (Eds.), Handbook of Emotions and Mass Media<\/em> (349-369). London\/New York: Routledge.<\/p>\n

Bailenson, J.N., Yee, N., Blascovich, J., Beall, A.C., Lundblad, N., & Jin, M. (2008). \u201cThe Use of Immersive Virtual Reality in the Learning Sciences: Digital Transformations of Teachers, Students, and Social Context<\/a>.\u201d The Journal of the Learning Sciences, 17<\/em>, 102-141.<\/p>\n

Bailenson, J.N., Pontikakis, E. D., Mauss, I.B., Gross, J.J., Jabon, M.E., Hutcherson, C.A., Nass, C., & John, O. (2008). \u201cReal- Time Classification of Evoked Emotions using Facial Feature Tracking and Physiological Responses<\/a>.\u201d International Journal of Human Machine Studies, 66<\/em>, 303-317.<\/p>\n

Bailenson, J.N., Yee, N., Blascovich J., Guadagno R.E. (2008) \u201cTransformed Social Interaction in Mediated Interpersonal Communication\u201d In Konijn, E., Tanis, M., Utz, S. & Linden, A. (Eds.), Mediated Interpersonal Communication (pp. 77-99). Lawrence Erlbaum Associates.<\/p>\n

PEOPLE BEHIND THE PROJECT<\/strong>
\n\"\"Jeremy Bailenson<\/a><\/strong> is founding director of Stanford University\u2019s Virtual Human Interaction Lab, the Thomas More Storke Professor in the Department of Communication at Stanford, and a Senior Fellow at the Woods Institute for the Environment. He designs and studies virtual reality systems that allow physically remote individuals to meet in virtual space, and explores the manner in which these systems change the nature of verbal and nonverbal interaction. In particular, he explores how virtual reality can change the way people think about education, environmental.<\/p>\n","protected":false},"excerpt":{"rendered":"

From The Theme HUMAN MACHINE INTERACTION AND SENSING WHAT IF? What if we could automatically detect how well someone is comprehending information presented on a computer screen, based on the emotions they display? WHAT WE SET OUT TO DO We set out to assess the effects of emotion on comprehension, using educational systems already in […]<\/p>\n","protected":false},"author":3,"featured_media":0,"parent":363,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"footnotes":"","_links_to":"","_links_to_target":""},"tags":[18,17,16,19,20,21],"class_list":["post-7448","page","type-page","status-publish","hentry","tag-emotion","tag-hci","tag-hmi","tag-learning","tag-sensors","tag-virtual-reality"],"acf":[],"yoast_head":"\nDetection of Comprehension & Emotion from Real-time Video Capture of Facial Expressions - mediaX at Stanford University<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Detection of Comprehension & Emotion from Real-time Video Capture of Facial Expressions - mediaX at Stanford University\" \/>\n<meta property=\"og:description\" content=\"From The Theme HUMAN MACHINE INTERACTION AND SENSING WHAT IF? What if we could automatically detect how well someone is comprehending information presented on a computer screen, based on the emotions they display? WHAT WE SET OUT TO DO We set out to assess the effects of emotion on comprehension, using educational systems already in […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/\" \/>\n<meta property=\"og:site_name\" content=\"mediaX at Stanford University\" \/>\n<meta property=\"article:modified_time\" content=\"2018-11-15T16:48:15+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mediax.stanford.edu\/wp-content\/uploads\/2017\/01\/Facial_Expressions.jpg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mediax.stanford.edu\\\/research-projects\\\/hmi-bailenson\\\/\",\"url\":\"https:\\\/\\\/mediax.stanford.edu\\\/research-projects\\\/hmi-bailenson\\\/\",\"name\":\"Detection of Comprehension & Emotion from Real-time Video Capture of Facial Expressions - mediaX at Stanford University\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mediax.stanford.edu\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mediax.stanford.edu\\\/research-projects\\\/hmi-bailenson\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mediax.stanford.edu\\\/research-projects\\\/hmi-bailenson\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mediax.stanford.edu\\\/wp-content\\\/uploads\\\/2017\\\/01\\\/Facial_Expressions.jpg\",\"datePublished\":\"2018-05-30T14:35:00+00:00\",\"dateModified\":\"2018-11-15T16:48:15+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mediax.stanford.edu\\\/research-projects\\\/hmi-bailenson\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mediax.stanford.edu\\\/research-projects\\\/hmi-bailenson\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mediax.stanford.edu\\\/research-projects\\\/hmi-bailenson\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mediax.stanford.edu\\\/wp-content\\\/uploads\\\/2017\\\/01\\\/Facial_Expressions.jpg\",\"contentUrl\":\"https:\\\/\\\/mediax.stanford.edu\\\/wp-content\\\/uploads\\\/2017\\\/01\\\/Facial_Expressions.jpg\",\"width\":995,\"height\":421},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mediax.stanford.edu\\\/research-projects\\\/hmi-bailenson\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mediax.stanford.edu\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Research Projects\",\"item\":\"https:\\\/\\\/mediax.stanford.edu\\\/research-projects\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Detection of Comprehension & Emotion from Real-time Video Capture of Facial Expressions\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mediax.stanford.edu\\\/#website\",\"url\":\"https:\\\/\\\/mediax.stanford.edu\\\/\",\"name\":\"mediaX at Stanford University\",\"description\":\"Industry Affiliate Program of Stanford\u2019s H-STAR Institute\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mediax.stanford.edu\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Detection of Comprehension & Emotion from Real-time Video Capture of Facial Expressions - mediaX at Stanford University","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/","og_locale":"en_US","og_type":"article","og_title":"Detection of Comprehension & Emotion from Real-time Video Capture of Facial Expressions - mediaX at Stanford University","og_description":"From The Theme HUMAN MACHINE INTERACTION AND SENSING WHAT IF? What if we could automatically detect how well someone is comprehending information presented on a computer screen, based on the emotions they display? WHAT WE SET OUT TO DO We set out to assess the effects of emotion on comprehension, using educational systems already in […]","og_url":"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/","og_site_name":"mediaX at Stanford University","article_modified_time":"2018-11-15T16:48:15+00:00","og_image":[{"url":"https:\/\/mediax.stanford.edu\/wp-content\/uploads\/2017\/01\/Facial_Expressions.jpg","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/","url":"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/","name":"Detection of Comprehension & Emotion from Real-time Video Capture of Facial Expressions - mediaX at Stanford University","isPartOf":{"@id":"https:\/\/mediax.stanford.edu\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/#primaryimage"},"image":{"@id":"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/#primaryimage"},"thumbnailUrl":"https:\/\/mediax.stanford.edu\/wp-content\/uploads\/2017\/01\/Facial_Expressions.jpg","datePublished":"2018-05-30T14:35:00+00:00","dateModified":"2018-11-15T16:48:15+00:00","breadcrumb":{"@id":"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/#primaryimage","url":"https:\/\/mediax.stanford.edu\/wp-content\/uploads\/2017\/01\/Facial_Expressions.jpg","contentUrl":"https:\/\/mediax.stanford.edu\/wp-content\/uploads\/2017\/01\/Facial_Expressions.jpg","width":995,"height":421},{"@type":"BreadcrumbList","@id":"https:\/\/mediax.stanford.edu\/research-projects\/hmi-bailenson\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mediax.stanford.edu\/"},{"@type":"ListItem","position":2,"name":"Research Projects","item":"https:\/\/mediax.stanford.edu\/research-projects\/"},{"@type":"ListItem","position":3,"name":"Detection of Comprehension & Emotion from Real-time Video Capture of Facial Expressions"}]},{"@type":"WebSite","@id":"https:\/\/mediax.stanford.edu\/#website","url":"https:\/\/mediax.stanford.edu\/","name":"mediaX at Stanford University","description":"Industry Affiliate Program of Stanford\u2019s H-STAR Institute","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mediax.stanford.edu\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mediax.stanford.edu\/wp-json\/wp\/v2\/pages\/7448","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mediax.stanford.edu\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mediax.stanford.edu\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mediax.stanford.edu\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/mediax.stanford.edu\/wp-json\/wp\/v2\/comments?post=7448"}],"version-history":[{"count":5,"href":"https:\/\/mediax.stanford.edu\/wp-json\/wp\/v2\/pages\/7448\/revisions"}],"predecessor-version":[{"id":10987,"href":"https:\/\/mediax.stanford.edu\/wp-json\/wp\/v2\/pages\/7448\/revisions\/10987"}],"up":[{"embeddable":true,"href":"https:\/\/mediax.stanford.edu\/wp-json\/wp\/v2\/pages\/363"}],"wp:attachment":[{"href":"https:\/\/mediax.stanford.edu\/wp-json\/wp\/v2\/media?parent=7448"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mediax.stanford.edu\/wp-json\/wp\/v2\/tags?post=7448"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}