Projects

Virtual, Mixed and Augmented Reality

Conversational Agents

Multisensory Environments

Smart-Objects

Multisensory Environments

LUDOMI

Ludomi is the project awarded by Polisocial Awards 2017 in which we developed Magika. Magika is the innovative technological solution that allows to transform any room into a “smart” space in which lights, immersive projections on walls and floors, music, sounds, aromas, and physical materials are digitally controllable, programmable, and interactive. The result is a “Magic Room” where children are exposed to multisensory stimuli and can interact with physical materials and multimedia contents through movements, gestures, and manipulation of objects. Magika allows to perform playful and educational activities (individual or collective) that stimulate all senses and are engaging. Through a tablet, teachers can control and configure each activity and personalize the experience according to the specific needs of children. Magika can be controlled and configured through a tablet. Thanks to an easy and intuitive control panel, the teacher can monitor, select, and improve the activities of the room. It takes just a few steps to choose activities and to keep an eye on what’s happening. Through the tablet you can start or end the activity any time, thus teachers can manage children inside the room better. Compared to traditional multi-sensory rooms like Snoezelen, the playful-educational experiences in the Magic Room have greater potential for children with cognitive disabilities because they are more customizable, stimulating and engaging.

Virtual, Mixed and Augmented Reality

Wildcard

The purpose of this work is to collaborate with medical associations to help children with NDD to provide for themselves, reducing their dependencies on third parties. The personal and social autonomy is the desired end state to be achieved, thus enabling integration in the real world. To reach this goal the project leverages immersive virtual realities using a special viewer. This feature allows patients to stay focused on the exercise that must be performed. Such therapeutic sessions trigger a learning process aiming at a seamless understanding of the sourrounding world. This study leads to the construction of Wildcard: a framework for creation, management and storytelling of avtivities. Theese stories help children in facing everydays life’s challenges. The system has been designed in cooperation with the non-profit therapy center “L’abilità”, resulting in an intelligent and immersive platform for children with autism. This enables a cognitive maturation through a playful entertainment that becomes a life lesson.

TWB

TWB is a EIT-funded project that addresses the therapeutic needs of persons with neuro-functional impairments (NFI), e.g. attention deficits or mental decline. We offer innovative services bridging wearable VR and MR apps & bio-sensing devices integrated in a customizable platform for patient care, therapy management and data analysis.
TWB services would enable novel treatments for adults with NFI to mitigate mental decline and for children with NFI to develop cognitive-behavioral skills, improving life quality for these subjects and reducing social costs.

Social MatchUP

Social MatchUp is a multiplayer Virtual Reality mobile game for children with NDD.
Shared virtual reality environments (SVREs) allow NDD children to interact in the same virtual environment, but without the possible inconvenience caused by having a real person in front of them. The application has been designed in collaboration with psychologist and it’s the result of a partecipatory design process with people with NDD.
The goal of the game is to enhance social and communication skills through a platform in which, in pairs, players will have to join the same images by staring at them; in order to agree on the image to look at they only have a way to communicate: by speaking

Conversational  Agents

In Brevis

The aim of the project was to develop a system exploiting the power of NLP techniques in order support therapists in the context of group therapy session with patient suffering from neurodevelopmental disorders – NDD. In this kind of session patients speak freely expressing their thoughts and feelings. The main problems highlighted were that it is extremely hard to gather insights on the thoughts of every single patient during group therapy and that often they stop coming to therapy because they feel uncomfortable or uninterested. The developed system takes these problems into account and tries to offer a solution. An iOS application is used by the therapist to manage and record a session as well to extract the content of the conversation using NLP techniques. The extracted topics and some general info about the session are saved into a database. A web application allows patients to interact in a more private way with the content of the session and to express some feedback related to the perceived importance of each topic as well as an emotional feedback about the session as a whole. The therapist on the other hand can visualize in various way the data extracted from the interaction of the patients to develop a better understanding of how they felt

ISI

This project merges the naturalness and power of conversational technology to the immediacy of visual communication. ISI is a conversational application able to talk both orally and in writing using natural language. The system is designed for children and stimulates them in their physical self-description by giving them some tips during the talk and by generating in real time personalized avatars based on the way the user describes herself/himself. The main goals of the application are to enhance communication and linguistic skills of children in their mother tongue and in second languages and to support children in their self-knowledge and self-acceptance track. This project merges the naturalness and power of conversational technology to the immediacy of visual communication. ISI is a conversational application able to talk both orally and in writing using natural language. The system is designed for children and stimulates them in their physical self-description by giving them some tips during the talk and by generating in real time personalized avatars based on the way the user describes herself/himself. The main goals of the application are to enhance communication and linguistic skills of children in their mother tongue and in second languages and to support children in their self-knowledge and self-acceptance track.
The name ISI stands for Io Sono Io, that is the Italian version of I am me. This name takes inspiration from the German book Das kleine Ich bin ich by Mira Lobe and Susi Weigel that supports children to answer to the question “Who am I?”. Furthermore, in Italian, ISI is pronounced in the same way as the English word easy and this perfectly fits with the ideas behind my system:
  • it is simple to use and to access (it is thought as a web application)
  • it facilitates the communication and trains a progressive self-knowledge

BEEH

Puffy, a robotic companion that we have designed in cooperation with NDD specialists. Puffy is meant to support educational and therapeutic interventions for children with NDD, particularly those with perceptual and sensory processing impairments and deficits in the cognitive, social and behavioral spheres. The design of the physical and interaction characteristics of Puffy is informed by:
  1. general design guidelines reported in the current literature on socially assistive robots for autistic children;
  2. lessons learned from our own previous experience on robots for children with NDD;
  3. feedbacks and suggestions on the progressive prototypes of Puffy offered by a team of 15 therapists (psychologists, neuro-psychiatrists, and special educators) from two different rehabilitation centers who have long-term, everyday experience of NDD subjects (children and adults) and have been collaborating with our research in the last 5 years.

Emoty

This research project aims at exploiting the advances in conversational technology to support people with Neurodevelopmental Disorder (NDD). NDD is a group of conditions that are characterized by severe deficits in the cognitive, emotional, and motor areas and produce severe impairments in communication and social functioning. In particular, we investigate the use of spoken Conversational Agents to mitigate the impairments of people with NDD related to the difficulty of recognizing and expressing emotions – a problem clinically referred to as Alexithymia.
The project is carried out in close collaboration with psychologists, neurological doctors, and caregivers who actively participated in eliciting the key requirements, evaluating iterative prototypes, and performing an empirical evaluation. The main result of this work is the web application called Emoty. 
Emoty is a voice-based Italian speaking Dialog System able to converse with the users in ordinary natural language and to entertain them with small talks and educational games.
Emoty does not act as a virtual assistant for daily life support but aims at helping people with NDD to develop better emotional control and self-awareness, which would lead them to enhance their communication capabilities and consequently to improve their quality of life. Emoty exploits conversational technologies, Machine Learning and Deep Learning techniques for emotion recognition from the text and the voice based on the processing of user’s audio pitch.

Voxana

Voice-based CAs and chatbots are progressively getting more and more embedded into people’s home: 46% of United States adults uses them in their daily routines. The most famous are Apple’s Siri, Amazon’s Alexa (over 8 million users in January 2017), Google Assistant and IBM conversational services. Facebook alone has 300000 active chatbots powering 8 billion interactions between consumers and businesses every month. On the wave of these facts, Voxana originates from the international collaboration between academic world and industry and sees Politecnico di Milano, TIM, Alfstore, and GroupeSEB as partners, and it is funded by EIT Digital. It is an analytics solution for voice-controlled connected products. VoxAna intends to help marketing people to convert voice interactions into meaningful insights. It makes voice of customers real, by analyzing the voice channel to detect user’s expectations and turn tone into comprehensive insights. More in detail, as consumer brands keep looking to fully understand the user’s context, VoxAna is designed to deliver more personalized and contextualized experiences using Text (Speech to Text) and Sentiment and Emotional (both from content and voice tone) analysis. VoxAna offers a dashboard solution with metrics and insights for voice-controlled product or service to conduct personalized market research.

Smart-Objects

Puffy

Puffy, a robotic companion that we have designed in cooperation with NDD specialists. Puffy is meant to support educational and therapeutic interventions for children with NDD, particularly those with perceptual and sensory processing impairments and deficits in the cognitive, social and behavioral spheres. The design of the physical and interaction characteristics of Puffy is informed by:
  1. general design guidelines reported in the current literature on socially assistive robots for autistic children;
  2. lessons learned from our own previous experience on robots for children with NDD;
  3. feedbacks and suggestions on the progressive prototypes of Puffy offered by a team of 15 therapists (psychologists, neuro-psychiatrists, and special educators) from two different rehabilitation centers who have long-term, everyday experience of NDD subjects (children and adults) and have been collaborating with our research in the last 5 years.

PLET

PLET has been designed and developed by Politecnico di Milano to:
  1. “smartify” physical toys and transform them into “e-toys” (i.e., digitally controlled interactive objects)
  2. connect e-toys with the PLET service management platform, therefore bridging the system embedded in e-toys with the applications for the toy manufacturer.

This technology is the digital core of next generation Smart Toys, being responsible of managing the e-toy interactive behaviour and transmitting (via Wi-Fi) the interaction data to the back end of the PLET service platform in a configurable, robust, and optimised way (e.g., pre-processing some raw data received from e-toy embedded devices such as RFIDs). This technology consists of an embedded platform that includes various hardware components and a software middleware. The embedded platform integrates different technologies, involving commercial sensors and actuators, orchestrated by a powerful and yet low-cost microcontroller, managed by software components that have been designed ad hoc to be as efficient and performant as possible and to maximize the potential of the hardware capabilities embedded in the e-toy.

The middleware is “hardware agnostic”, in that it provides a communication interface e-toy→PLET services platform] that is independent from the specific sensors/actuators embedded in the toy and can be customized to the specific hardware technology adopted by a specific manufacturer. To mask the hardware details completely, the middleware offers high level APIs that can be invoked using HTTP(s) and JSON calls.
The document also includes the process performed to achieve the above results, including the following activities:
  • selection of the hardware components – commercial sensors and actuators – to be embedded in e-toys;
  • integration of the chosen hardware components, in terms of both electronics and software control;
  • design of the PCB (printed circuit board), to be produced by an external manufacturer;
  • middleware API specification and implementation,

Each of the above activities comprised intense technical testing of the progressive results.

Reflex

Reflex is a mirrored camera mobile training application for persons with Neuro-Developmental Disorder (NDD). The game, offered through a cross-platform application for smart-phones and tablets, bridges the digital and the physical worlds by tracking, via a bottom-looking mirror positioned on the device camera, physical items placed on the table. This interaction paradigm defined as phygital, its co-designed features and the first pilot study reveal an unexplored potential for learning. 

We chose to adopt a hybrid approach defined as “phygital” as it digitally virtualizes a physical environment of Tangible Objects (TOs) manipulable by the user. This approach uses a platform which augments a handheld computing device, such as a phone or tablet and utilizes novel computer vision algorithms to sense user interaction with the tangible objects. This technology yields numerous advantages including, but not limited to providing a low-cost alternative for developing a nearly limitless range of applications that blend both physical and digital mediums by reusing existing hardware (e.g., device camera); leveraging novel lightweight detection and recognition algorithms; having low implementation costs; being cross-compatible with existing computing device hardware; operating in real-time to provide for a rich, virtual experience; processing numerous interactions with one or more TOs simultaneously without overwhelming the computing device; recognizing TOs with substantially perfect recall and precision (e.g., 99% and 99.5%, respectively); being capable of adapting to lighting changes and wear and tear of TOs; providing a collaborative tangible experience between users; being natural and intuitive to use; requiring few or no constraints on the types of TOs that can be processed (no specialized markers or symbols are required to be included on the TOs in order for the platform to recognize them