A Multisensory Interactive Memory Game

My role: I am responsible for literature review, experiment design, questionnaire draft, and video editing, half presentation
Keywords: Literature review; Prototyping; Multisensory design
                
       
Tool: Processing; Aduino uno; Puresound

Team: Muhan Liu(HCI student)
          Sara(HCI student)

The problem

The cognitive decline between elderly people: Especially in Memory capability 

Based-on Theories 

1. Brain plasticity-Human brain is adaptive 

2. Multimodal perception-Multimodal stimuli with congruent features helps            complete task more accurate and faster

3. Cooperation helps to improve memory-Social interactions in games

4. Haptic simulation usage in games- Reminding

 

Formation of idea
 



 
By combining visual, auditory, and haptic elements, a superadditive effect might be created where the responses to the multisensory stimuli are greater than the combined response to each separate modality. Using such stimuli in the context of a cooperative memory game, we aim to contribute to the enhancement of elderly’s memory functioning.

Hypothesis

1)Multimodal stimuli (audio-visual) with congruent features between modalities have a greater positive effect than unimodal stimulus (visual only) on information processing

2)The use of multimodal stimuli (audio-visual) with congruent features between modalities will have a greater positive effect than unimodal stimulus (visual only) on memory performance

3)Cooperation plays a positive role in memory training

4)Haptic stimulation may help maintain the focus and attention of the user

Architecture design
Components image.png

1.The processing on the computer provides visual interface

2. The network connect between both computers

3. The puredata connect with Processing collect information sent from SBC

4. A soundcard and microcontroller are connected to SBC

Evaluation
Hypothesis 1: 
 

•Condition 1 (Visual only): Two participants in turn use the controller to select two virtual cards on the computer screen by moving the controller to match the two chosen cards images. We track the amount of time elapsed per turn while the game is played.
 

•Condition 2 (Audio-visual): Repeat the same step as condition 1. While the user is flipping the card,they will hear a congruent sound related to the image on the card. We track the amount of time elapsed per turn while the game is played.

Hypothesis 2: 
 

•Condition 1 (Visual only): Participants will be asked image content.

•Condition 2 (Audio-visual): Participants will be asked image content.

Hypothesis 3: 
 

•Condition 1 (Single player): One participant plays the single player version of the game with audio-visual feedback. This means that no turns

are taken but the participant can continue their search for pairs continuously. An oral question about the content of images on the chosen card will be asked along each turn.

•Condition 2 (Multiplayer): Two participants play the game with audio-visual feedback as presented in condition 2 for hypothesis 2. An oral question about the content of images on the chosen card will be asked along each turn.

Hypothesis 4: 
 

•Condition 1 (Haptic stimulation): Two participants play the game with audio-visual feedback and haptic stimulation (i.e. vibration). At the end of the game, they will be asked about their subjective feelings of focus and attention through a questionnaire.

•Condition 2 (No haptic stimulation): Two participants play the game with audio-visual feedback without haptic stimulation (i.e. no vibration). At the end of the game, they will be asked about their subjective feelings of focus and attention through a questionnaire.

Can be found here
Questionnaire 
Can be found here
Full report