Multi-Modal and Multi-Sensorial Interaction

About Multi-Modal and Milti-Sensorial Interaction

Research into multi-modal and multi-sensorial interaction investigates how the involvement of additional senses can affect the user when interacting with digital content. Virtual Experiences can be enhanced by implementing haptic, olfactory, and even gustatory interactions between the multimedia content and the users for a more immersive experience. Our research investigates the ways in which additional senses can be involved to enhance the multimedia content through mulsemedia - multiple sensorial media - where users experience sensations such as odors, tastes, and the sensation of wind blowing against their face.

Research Papers

• Narkhede, P., Walambe, R., Mandaokar, S., Chandel, P., Kotecha, L. and Ghinea, G. (2021) 'Gas Detection and Identification Using Multimodal Artificial Intelligence Based Sensor Fusion', Applied System Innovation. 4, 3
• Mesfin, G., Saleme, EB., Ademoye, O., Kani-Zabihi, E., Santos CAS. and Ghinea, G. (2021) 'Less is (Just as Good as) More - an Investigation of Odor Intensity and Hedonic Valence in Mulsemedia QoE using Heart Rate and Eye Tracking', IEEE Transactions on Multimedia, 23, 1095-1105
• Saleme, EB., Santos, CAS. and Ghinea, G. (2019) 'Coping with the Challenges of Delivering Multiple Sensorial Media'. IEEE MultiMedia, 26 (2). pp. 66 - 75. ISSN: 1070-986X
• de Amorim, MN., Saleme, EB., Assis Neto, FRD., Santos, CAS. and Ghinea, G. (2019) 'Crowdsourcing authoring of sensory effects on videos'. Multimedia Tools and Applications. ISSN: 1380-7501
• Mesfin, G., Hussain, N., Covaci, A. and Ghinea, G. (2018) 'Inverse and transitivity of cross-modal correspondence in mulsemedia'.2018 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). San Diego, CA. USA. 29 - 27 July. IEEE.
• Hussain, N., Mesfin, G., Covaci, A. and Ghinea, G. (2018) 'Towards augmenting multimedia QoE with wearable devices: Perspectives from an empirical study'. 2018 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). San Diego, CA, USA. 29 - 27 July. IEEE.
• Bi, T., Silva, F., Ghinea, G. and Muntean, GM. (2018) 'Improving Learning Experience by Employing DASH-Based Mulsemedia Delivery'.2018 IEEE Games, Entertainment, Media Conference (GEM). Galway, Ireland. 30 - 17 August. IEEE. pp. 414 - 417.
• Covaci, A., Kani-Zabihi, E., Mesfin, G., Andres, F., Hussain, N. and Ghinea, G. (2018) 'A study on the quality of experience of crossmodal mulsemedia'.10th International Conference on Management of Digital EcoSystems. Tokyo, Japan. 25 - 28 September. ACM Press. pp. 176 - 182.
• Saleme, EB., Santos, CAS., Falbo, RA., Ghinea, G. and Andres, F. (2018) 'Towards a reference ontology on mulsemedia systems'.0th International Conference on Management of Digital EcoSystems. Tokyo, Japan. 25 - 28 September. ACM Press. pp. 23 - 30.
• Covaci, A., Zhou, L., Tal, I., Muntean, G-M. and Ghinea, G. (2018) 'Is Multimedia Multisensorial? - A Review of Mulsemedia Systems'. ACM Computing Surveys, 51 (5). pp. 1 - 35. ISSN: 0360-0300
• Saleme, EB., Santos, CAS. and Ghinea, G. (2018) 'Coping with the Challenges of Delivering Multiple Sensorial Media'. IEEE MultiMedia. ISSN: 1070-986X
• Bi, T., Chen, S., Pichon, A., Ghinea, G., Zou, L. and Muntean, GM. (2018) 'A DASH-based mulsemedia adaptive delivery solution'.10th International Workshop on Immersive Mixed and Virtual Environment Systems. Amsterdam, Netherlands. 12 - 15 June. ACM Press. pp. 1 - 6.
• Covaci, A., Ghinea, G., Lin, CH., Huang, SH. and Shih, JL. (2018) 'Multisensory games-based learning - lessons learnt from olfactory enhancement of a digital board game'. Multimedia Tools and Applications, 77 (16). pp. 21245 - 21263. ISSN: 1380-7501
• Comsa, LS., Trestian, R. and Ghinea, G. (2018) '360° Mulsemedia Experience over Next Generation Wireless Networks-A Reinforcement Learning Approach'.2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX). Cagliari, Italy. 11 - 1 June. IEEE. pp. 1 - 6. ISSN: 2472-7814
• Saleme, EB., Santos, CAS. and Ghinea, G. (2018) 'Improving response time interval in networked event-based mulsemedia systems'.9th ACM Multimedia Systems Conference. Amsterdam, Netherlands. 12 - 15 June. ACM Press. pp. 216 - 224.
• Murray, N., Ademoye, OA., Ghinea, G. and Muntean, GM. (2017) 'A tutorial for olfaction-based multisensorial media application design and evaluation'. ACM Computing Surveys, 50 (5). pp. 1 - 30. ISSN: 0360-0300
• Maiero, J., Kruijff, E., Hinkenjann, A. and Ghinea, G. (2017) 'ForceTab: Visuo-haptic interaction with a force-sensitive actuated tablet'.2017 IEEE International Conference on Multimedia and Expo (ICME). IEEE. pp. 169 - 174. ISSN: 1945-7871
• Murray, N., Ademoye, OA., Ghinea, G., Qiao, Y., Muntean, GM. and Lee, B. (2017) 'Olfactory-enhanced multimedia video clips datasets'.2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX). IEEE.
• Zou, L., Tal, I., Covaci, A., Ibarrola, E., Ghinea, G. and Muntean, GM. (2017) 'Can multisensorial media improve learner experience?'.the 8th ACM. ACM Press. pp. 315 - 320.
• Ademoye, OA., Murray, N., Muntean, GM. and Ghinea, G. (2016) 'Audio masking effect on inter-component skews in olfaction-enhanced multimedia presentations'. ACM Transactions on Multimedia Computing, Communications and Applications, 12 (4). pp. 1 - 14. ISSN: 1551-6857
• Maiero, J., Kruijff, E., Hinkenjann, A. and Ghinea, G. (2015) 'PICOZOOM: A context sensitive multimodal zooming interface'.2015 IEEE International Conference on Multimedia and Expo (ICME). Turin. 3 - 3 July. IEEE. ISSN: 1945-7871
• Yuan, Z., Bi, T., Muntean, GM. and Ghinea, G. (2015) 'Perceived synchronization of mulsemedia services'. IEEE Transactions on Multimedia, 17 (7). pp. 957 - 966. ISSN: 1520-9210
• Yuan, Z., Ghinea, G. and Muntean, G-M. (2014) 'Beyond multimedia adaptation: Quality of experience-aware multi-sensorial media delivery'. IEEE Transactions on Multimedia, 17 (1). pp. 104 - 117. ISSN: 1941-0077
• Ghinea, G., Timmerer, C., Lin, W. and Gulliver, SR. (2014) 'Mulsemedia: State of the art, perspectives, and challenges'. ACM Transactions on Multimedia Computing, Communications and Applications, 11 (1s). pp. 1 - 23. ISSN: 1551-6857
• Ghinea, G., Timmerer, C., Lin, W. and Gulliver, S. (2014) 'Introduction to special issue on multiple sensorial (MulSeMedia) multimodal media: Advances and applications'. ACM Transactions on Multimedia Computing, Communications and Applications, 11 (1s). pp. 1 - 2. ISSN: 1551-6857
• Yuan, Z., Chen, S., Ghinea, G. and Muntean, G-M. (2014) 'User quality of experience of mulsemedia applications'. ACM Transactions on Multimedia Computing, Communications and Applications, 11 (1s). pp. 1 - 19. ISSN: 1551-6857
• Yuan, Z., Ghinea, G. and Muntean, G-M. (2014) 'Quality of experience study for multiple sensorial media delivery'.2014 International Wireless Communications and Mobile Computing Conference (IWCMC). Institute of Electrical and Electronics Engineers Inc.. pp. 1142 - 1146.
• Ademoye, OA. and Ghinea, G. (2013) 'Information recall task impact in olfaction-enhanced multimedia'. ACM Transactions on Multimedia Computing, Communications and Applications, 9 (3). pp. 1 - 16. ISSN: 1551-6857
• Ghinea, G., Andres, F. and Gulliver, S. (2011) 'Multiple sensorial media advances and applications: New developments in MulSeMedia'. Multiple Sensorial Media Advances and Applications: New Developments in MulSeMedia. pp. 1 - 320.