Technical Program


Monday, June 14

Monday, June 14 8:00 – 8:30 EDT (UTC-4)

open: Opening ceremony

Chairs: Tiago Falk, Amy Reibman

Monday, June 14 8:30 – 9:30 EDT (UTC-4)

S1: QoE analytics and datasets

Chair: Lea Skorin-Kapov

Towards High Resolution Video Quality Assessment in the Crowd
Rakesh Rao Ramachandra Rao, Steve Göring and Alexander Raake
Validation of dynamic subjective quality assessment methodology for holographic coding solutions
Ayyoub Ahar, Manuela Pereira, Tobias Birnbaum, Antonio M. G. Pinheiro and Peter Schelkens
Groovability: Using Groove as a Novel Measure for Audio QoE with the Example of Smartphones
Dominik Keller, Markus Vaalgamaa, Erkki Paajanen, Rakesh Rao Ramachandra Rao, Steve Göring and Alexander Raake
Study on user quitting in the Puffer live TV video streaming service
Pierre Lebreton and Kazuhisa Yamagishi

Monday, June 14 10:00 – 11:30 EDT (UTC-4)

S2: Crowdsourcing

Chair: Matthias Hirth

A Crowdsourcing-based QoE Evaluation of an Immersive VR Autonomous Driving Experience
Guilherme D Gomes, Ronan Flynn and Niall Murray
Speech Quality Assessment in Crowdsourcing: Comparison Category Rating Method
Babak Naderi, Sebastian Möller and Ross Cutler
On Inter-Rater Reliability for Crowdsourced QoE
Tobias Hoßfeld, Michael Seufert and Babak Naderi
Influence of Language Differences in Crowdsourcing Speech Quality Assessment Studies
Rafael Zequeira Jiménez, Babak Naderi and Sebastian Möller
Removing the Bias in Speech Quality Scores Collected in Noisy Crowdsourcing Environments
Rafael Zequeira Jiménez, Gabriel Mittag and Sebastian Möller
QoE Models in the Wild: Comparing Video QoE Models Using a Crowdsourced Data Set
Anika Seufert, Florian Wamser, David Yarish, Hunter Macdonald and Tobias Hoßfeld

Monday, June 14 12:00 – 13:00 EDT (UTC-4)

kn1: Keynote #1

The QoE Journey of a Streaming Company
Dr. Anne Aaron

Chair: Niall Murray

Dr. Anne Aaron is Director of Encoding Technologies at Netflix. The team is responsible for generating media assets that deliver a compelling A/V experience to millions of Netflix members worldwide, no matter where, how and what they watch. In her previous role at Netflix as lead of Video Algorithms, Anne helped drive innovation in the space (per-title encoding, video quality assessment and perceptual metrics, shot-based encoding, HDR, next-generation codecs) and grow a team of experts in cloud-based video coding and processing. Prior to Netflix, Anne had technical lead roles at Cisco, working on the software deployed with millions of Flip Video cameras, Dyyno, an early stage startup which developed a real-time peer-to-peer video distribution system, and Modulus Video, a broadcast video encoder company. During her Ph.D. studies at Stanford University, she was a member of the Image, Video and Multimedia Systems Laboratory, led by Prof. Bernd Girod. Her research was one of the pioneering work in the sub-field of Distributed Video Coding.

Monday, June 14 13:30 – 15:00 EDT (UTC-4)

e1: Meet the QoMEX community: fireside chat

Chairs: Tanja Kojic, Atidel Lahoulou, Sebastian Möller, Steven Schmidt

Are you new to QoE research and to QoMEX? Did you always want to ask leading experts about how they got where there are today, what challenges they face in their daily life, or how you might learn from them on how to best plan your future? If you answered yes to any of these questions, then this event is for you! This 1-hour “fireside chat” event is aimed at students and young researchers new to QoE and new to the QoMEX community. In the first half of the event, four experts – Dr. Lina J. Karam, Dr. Markus Vaalgamaa, Dr. Anne Aaron, and Dr. Martín Varela – will share their thoughts on emerging trends in QoE (both in academia and industry) in a moderated panel-like format. In the second half, young researchers will be given the opportunity to continue the interaction but at a smaller (moderated) table discussion. All are invited to attend! More details can be found here!

Tuesday, June 15

Tuesday, June 15 8:00 – 8:30 EDT (UTC-4)

nt1: Networking

Chairs: Tiago Falk, Liviu Ivanescu

Tuesday, June 15 8:30 – 9:30 EDT (UTC-4)

S3: Short papers session 1: New tools and databases

Chair: Oliver Hohlfeld

Image Super-Resolution Quality Assessment: Structural Fidelity Versus Statistical Naturalness
Wei Zhou, Zhou Wang and Zhibo Chen
Impact of Spatial and Temporal Information on Video Quality and Compressibility
Werner Robitza, Rakesh Rao Ramachandra Rao, Steve Göring and Alexander Raake
A Real Haze Video Database for Haze Level Evaluation
Ying Chu, Guoxing Luo and Fan Chen
Cognitive Load Estimation Based on Pupillometry in Virtual Reality with Uncontrolled Scene Lighting
Marie Eckert, Emanuël Habets and Olli Rummukainen
Visualizing navigation difficulties in video game experiences
Hippolyte Dubois, Patrick Le Callet and Antoine Coutrot
EVENT-CLASS: Dataset of events in the classroom
Marta Orduna, Jesús Gutiérrez, Carlos Manzano, David Ruiz, Julián Cabrera, César Díaz, Pablo Pérez and Narciso García

Tuesday, June 15 10:00 – 11:30 EDT (UTC-4)

S4: Machine learning and QoE

Chair: Mylene Farias

Full-Reference and No-Reference Objective Evaluation of Deep Neural Network Speech
Stephen D Voran
Towards Blind Audio Quality Assessment using a Convolutional-Recurrent Neural Network
Xiguang Zheng and Chen Zhang
Bias-Aware Loss for Training Image and Speech Quality Prediction Models from Multiple Datasets
Gabriel Mittag, Saman Zadtootaghaj, Thilo Michael, Babak Naderi and Sebastian Möller
More for Less: Non-Intrusive Speech Quality Assessment with Limited Annotations
Alessandro Ragano, Emmanouil Benetos and Andrew Hines
Performance Evaluation of Objective Image Quality Metrics on Conventional and Learning-Based Compression Artifacts
Michela Testolina, Evgeniy Upenik, João Ascenso, Fernando Pereira and Touradj Ebrahimi
MARMnet: On the Application of Artificial Intelligence to Video Quality Control for Broadcasting Companies
Riccardo Musmeci and Adriano Manfrè

Tuesday, June 15 12:00 – 13:00 EDT (UTC-4)

kn2: Keynote #2

Towards Truly Immersive Holographic-Type Experiences
Professor Filip De Turck

Chair: Lea Skorin-Kapov

Dr. Filip De Turck is a full professor at the department of Information Technology (Intec) of Ghent University and imec, Belgium with expertise in communication software, network resource management, adaptive multimedia service delivery and next generation video delivery platforms. In this research area, he is involved in and successfully completed many research projects with industry and academia, serves as Chair of the IEEE Technical Committee on Network Operations and Management (CNOM), former chair of the Future Internet Cluster of the European Commission, is on the TPC of many international network and service management conferences and workshops and serves in the Editorial Board of several network and service management journals. Prof. Filip De Turck regularly organizes international workshops on the above mentioned topics, serves as Editor-in-Chief of IEEE Transactions on Network and Service Management (TNSM), and steering committee member of the IEEE/IFIP Network Operations and Management Symposium (NOMS), the IFIP/IEEE Integrated Network Management (IM) Symposium, the IEEE Conference on Network Softwarization (IEEE NetSoft) and the IFIP/IEEE Conference on Network and Service Management (CNSM). He also served as chair of the IEEE SDN Initiative Conference Committee, which coordinates initiative IEEE events and conferences on Softwarized Networks. Prof. Filip De Turck was named a Fellow of the IEEE (Institute of Electrical and Electronics Engineers) for outstanding technical contributions to network resource management and adaptive service delivery. He is currently (co-)author of over 750 publications.

Tuesday, June 15 13:30 – 15:00 EDT (UTC-4)

e2: Social (mixology class and jazz show)

Chairs: Tiago Falk, Liviu Ivanescu

Join us for a one-of-a-kind virtual mixology class from Canada’s own Paul Saucier, where you will learn to make one (or all) of three famous cocktails (non-alcoholic ones will also be available). Next, with your cocktail(s) in hands, sit back and enjoy a jazz concert from one of Canada’s most respected and influential jazz duos.Paul Saucier is a world-renowned mixologist, tending in cocktail bars in NYC and Europe and now in his home town of Ottawa, Ontario where he became the first Lead Mixologist and Head Bartender at the prestigious Fairmont Chateau Laurier. He continues to consult for top restaurants and bars, creating their cocktail lists and training bar staff across North America and has appeared in numerous publications and TV shows.

Pianist David Restivo and Fawn Fritzen both stand among the most celebrated performers in Canada’s jazz family. As musical and life partners, Fawn and David have a special and heart-warming chemistry in their duo shows. David is a JUNO award-winning pianist and composer, well-known for his work with Rob McConnell and The Boss Brass, the Mike Murley Quintet, songwriter Marc Jordan, and a host of international jazz greats. In January, he released his latest album, Arancina, on Chronograph Records. Fawn embodies Canada’s diversity with her mixed cultural roots of Chinese, French, and German. She sings in her four ancestral languages while blending jazz, roots, and gospel with candid storytelling. Fawn has is the winner of the 2014 Julian Award of Excellence for Emerging Jazz Artists and last summer released her third album, How to Say Sorry and Other Lessons, also on Chronograph Records.

Cocktail ingredient list

Rum Old Fashion

2oz dark rum
.75oz Banana Oleo
1 strawberry
2 dashes Angostura Bitters
1 dash Angostura Cacao Bitters (optional)

Banana Olea – cut one banana peel into 1/2″ pieces, put in a lock tight jar with equal weight of sugar, seal the top and let it rest overnight at room temperature. After 12-18 hours, pour the liquid through a fine mesh strainer, discard the banana peel and keep the oleo refrigerated.

Beach Bum

1.5oz of tequila blanco
1oz fresh pineapple juice
1oz fresh lemon juice
3/4oz fresh orange juice
3/4oz Vanilla syrup
2 dashes Angostura Bitters

Vanilla Syrup – mix 2 cups of water with 2 cups of sugar in a pot over low heat to dissolve sugar. Add 1 whole vanilla bean, 2 cinnamon sticks, 1/2 teaspoon of nutmeg – and keep on simmer for 10 minutes. Rest until cool. Pour into a bottle through a strainer and keep refrigerated.

Rooftop Breakfast (non-alcoholic)

1/2 strawberry/mint syrup
1 cucumber
1 lime
1 sprig fresh dill
1 can Perrier Watermelon

Strawberry/ mint syrup – mix 2 cups of water with 2 cups of sugar in a pot over low heat to dissolve sugar. Add 2 cups of hulled and sliced strawberries, simmer for 10 minutes. Remove from heat. Add in 1/2 cup of mint leaves, cover and let rest for 10 minutes. Pour into a bottle through a strainer and keep refrigerated.

More details:

Wednesday, June 16

Wednesday, June 16 8:00 – 8:30 EDT (UTC-4)

nt2: Networking

Chairs: Tiago Falk, Liviu Ivanescu

Wednesday, June 16 8:30 – 9:30 EDT (UTC-4)

S5: Short papers session 2: Audiovisual Quality of Experience

Chair: Atidel Lahoulou

Emotional Experience Analysis in Response to HDR and SDR content
Majid Riaz, Muhammad Majid and Junaid Mir
Measuring Speech Quality of System Input while Observing only System Output
Stephen D Voran
User Behaviour Analysis of Volumetric Video in Augmented Reality
Emin Zerman, Radhika Kulkarni and Aljosa Smolic
Parametric Audiovisual Quality Measurement Based on Interpretable Fuzzy Logic
Fatima Boudjerida, Atidel Lahoulou, Tiago Falk and Zahid Akhtar
Factors influencing QoE of video consultations
Even Brattbak Øie, Kamil Koniuch, Natalia Cieplińska and Katrien De Moor
Evaluating the translation of speech to virtually-performed sign language on AR glasses
Lan Thao Nguyen, Florian Schicktanz, Aeneas Stankowski and Eleftherios Avramidis

Wednesday, June 16 10:00 – 11:30 EDT (UTC-4)

S6: Interactive Applications and Games

Chairs: Florian Metzger, Steven Schmidt

Relationship Status: It’s Complicated. Using APM as a QoE-qualifying Tempo Metric
Florian Metzger, Norman Stulier, Kathrin Borchert and Matthias Hirth
The Effects of Network Latency on Competitive First-Person Shooter Game Players
Shengmei Liu, Mark Claypool, Atsuo Kuwahara, Jamie Sherman and James Scovell
Modeling and Understanding the Quality of Experience of Online Mobile Gaming Services
Steven Schmidt, Saman Zadtootaghaj, Saeed Shafiee Sabet and Sebastian Möller
The Effect of VR Gaming on Discomfort, Cybersickness, and Reaction Time
Sara Vlahovic, Mirko Suznjevic, Nina Pavlin-Bernardic and Lea Skorin-Kapov
Towards the Influence of Audio Quality on Gaming Quality of Experience
Steven Schmidt, Saman Zadtootaghaj, Shijie Wang and Sebastian Möller
QoE Assessment of FPV Drone Control in a Cloud Gaming Based Simulation
Matko Šilić, Mirko Suznjevic and Lea Skorin-Kapov

Wednesday, June 16 12:00 – 13:00 EDT (UTC-4)

kn3: Keynote #3

The neuroscience behind building a good auditory display
Professor Barbara Shinn-Cunningham

Chair: Andrew Hines

Dr. Barbara Shinn-Cunningham trained as an electrical engineer (Brown University, Sc.B.; Massachusetts Institute of Technology, M.S. and Ph.D.). She is Director and Professor of the Neuroscience Institute at Carnegie Mellon University, where she holds the Cowan Chair in Auditory Neuroscience. Before joining CMU, she spent 21 years on the faculty of Boston University in Biomedical Engineering. She is a committed mentor, and has won mentorship awards from the Society for Neuroscience as well as the Acoustical Society of America (ASA). Her research has been recognized with the Helmholtz-Rayleigh Interdisciplinary Silver Medal in Psychological and Physiological Acoustics, Speech Acoustics, and Architectural Acoustics from ASA, a Sloan Foundation Fellowship, and the Vannevar Bush Fellowship. She is a Fellow of ASA and the American Institute for Medical and Biological Engineers and is a lifetime National Associate of the National Research Council. In her spare time, she plays oboe and English horn. She is also a competitive saber fencer, and was part of the silver-medal winning USA Women’s Veteran Saber Team at the 2019 World Veteran Fencing Championships in Cairo, before the world shut down.

Wednesday, June 16 13:30 – 15:00 EDT (UTC-4)

e3: Expert panel: The impact of lack-of-diversity and AI bias in QoE research

Chairs: Katrien De Moor, Mylene Farias

Thursday, June 17

Thursday, June 17 8:00 – 8:30 EDT (UTC-4)

nt3: Networking

Chairs: Tiago Falk, Liviu Ivanescu

Thursday, June 17 8:30 – 9:30 EDT (UTC-4)

S7: QoE beyond visual and audio perception

Chair: Jan-Niklas Voigt-Antons

How does it feel? – Investigating Soft-Touch Surface Experience Dimensions
Georg Regal, Sebastian Egger-Lampl, Till Bieg, Thomas Ules, Manfred Tscheligi and Dieter P. Gruber
Beyond Shopping: The Motivations and Experience of Live Stream Shopping Viewers
Xiaoxing Liu and Sung Kim
The WEIRDness of QoE research: The diversity of QoMEX authorship considering locality and gender
Julia Himmelsbach, Katrien De Moor, Andreas Sackl, Raimund Schatz and Manfred Tscheligi
A Questionnaire to Assess Virtual Conference Participation Experience
Oliver Hohlfeld, Dennis Guse and Katrien De Moor

Thursday, June 17 10:00 – 11:30 EDT (UTC-4)

S8: Immersive QoE

Chair: Jesús Gutiérrez

A Multi-View Stereoscopic Video Database With Green Screen (MTF) For Video Transition Quality-of-Experience Assessment
Nour Hobloss, Lu Zhang and Marco Cagnazzo
Remapping of Hidden Area Mesh Pixels for Codec Speed-up in Remote VR
Viktor Kelkkanen, Markus Fiedler and David Lindero
Focus Guided Light Field Saliency Estimation
Ailbhe Gill, Emin Zerman, Martin Alain, Mikael Le Pendu and Aljosa Smolic
Evaluating Hand-tracking Interaction for Performing Motor-tasks in VR Learning Environments
Asim Hameed, Andrew Perkis and Sebastian Möller
A Full- and No-Reference Metrics Accuracy Analysis for Volumetric Media Streaming
Sam Van Damme, Maria Torres Vega and Filip De Turck
Legibility and readability in Augmented Reality
Julia Falk, Siri Eksvärd, Bo Schenkman, Börje Andrén and Kjell Brunnström

Thursday, June 17 12:00 – 13:00 EDT (UTC-4)

kn4: Keynote #4

Spatial audio for virtual and augmented reality devices
Dr. Ivan Tashev

Chair: Amy Reibman

Dr. Ivan Tashev received his Master’s degree (1984) in Electronic Engineering and PhD in Computer Science (1990) from the Technical University of Sofia, Bulgaria. He was assistant professor in this university when joined Microsoft in 1998. Currently Ivan Tashev leads the Audio and Acoustics Group in Microsoft Research Labs. His research interests include audio signal processing, machine learning, multichannel transducers, bio-signal processing. Dr. Tashev published two books, two book chapters, 100+ scientific papers, listed as inventor in 50 US patents. He is affiliate professor in University of Washington in Seattle and honorary professor at Technical University of Sofia, Bulgaria. Dr. Tashev transferred algorithms to RoundTable device, Windows, Microsoft Auto platform, and served as the audio architect of Kinect for Xbox and of HoloLens. More details about him can be found in his web page.

Thursday, June 17 13:00 – 14:00 EDT (UTC-4)

end: Closing ceremony and awards

Chairs: Tiago Falk, Amy Reibman

Thursday, June 17 14:00 – 15:30 EDT (UTC-4)

nt3: Networking

Chairs: Tiago Falk, Liviu Ivanescu

© QoMEX 2021