Lifestyle Understanding through the Analysis of Egocentric Photo-streams

Abstract

Describing people’s lives has become a hot topic in several disciplines. Lifelogging appeared in the 1960s as the process of recording and tracking personal activity data generated by the daily behaviour of a person. The development of new wearable technologies allows to auto- matically record data from our daily living. Wearable devices are light-ware and affordable, which shows potential for the increase of their use by our society. Egocentric images are recorded by wearable cameras and show a first-person view of the life of the camera wearer. These collected images show an objective view of the daily life of a person and thus are a rich source of information about her or his habits. However, there is lack of tools for the analysis of collections of egocentric photo-sequences and thus room for progress. This thesis investigates the development of automatic tools for the analysis of egocentric images with the ultimate goal of getting understanding of the lifestyle of the camera wearer. This work addresses five main topics in the field of egocentric vision: 1. Temporal photo-sequences segmentation: We introduce an automatic model for the defi- nition of temporal boundaries for the division of egocentric photo-sequences into mo- ments, which are sequences of images describing the same environment. The model is based on global and semantic features and achieves a 66% F-score over the EDUB-Seg dataset.

2. Routine discovery: We propose an automatic tool for the discovery of routine-related days and the visualization of patterns of behaviour, based on the use of topic modelling over semantic concepts extracted from the photo-sequences. The introduction of the EgoRoutine dataset composed of a total of 104 days is part of this work. The model is able to classify days into routine and non-routine related with an accuracy of 80%.

3. Food-related scenes recognition: We introduce a hierarchical classifier for the recognition of visually highly similar food-related images into 15 different classes that describe daily activities related to food consumption, acquisition, and preparation. We intro- duce the EgoFoodScenes dataset, which our model is able to classify into the 15 cate- gories with an accuracy of 68%.

4. Sentiment retrieval: We explore the sentiment associated with images by classifying them into Positive, Neutral, and Negative. Our model is based on the analysis of global features and obtained semantic concepts with associated sentiment. We obtain an ac- curacy of 75%. Results show that positive images relate to outdoor environments or with social interactions, neutral to work-related environments, and negative to non- informative or visually not clear images .

5. Social pattern characterization: We propose a model that characterizes the social be- haviour of the camera wearer based on the occurrence of people that the camera wearer meets throughout her/his data collection. The proposed social parameters allow the definition of a radar chart that shows its potential for the comparison of social patterns among individuals. The introduced and made publicly available egocentric datasets and the obtained results in the different performed experiments indicate that behaviour can be identified and studied. We conclude that the developed automatic algorithms for the analysis of egocentric images allow a better understanding of the lifestyle of the camera wearer. Applications based on the analysis of this data can lead to the improvement of the quality of life of people and therefore, are worth to continue exploring.