Aim: Developing tools to model and recognize situations using all available (device and human) data sources.
We have developed a theoretical approach to model situations, a computational framework to integrate and analyze relevant data, and a practical system to test out the ideas in real-world scenarios.
The developed EventShop system can integrate diverse data streams to situation recognition and personalized alerts.
A) Situation Modeling
Recognizing personalized situations from the large amount of publically available data (e.g. twitter, cameras, pollution, weather info) has applications
ranging from healthcare, to business analysis, to political decision making. We computationally define the notion of situations
and describe an approach for collecting and combining spatio-temporal-thematic (STT) data into actionable situations. We provide a conceptual framework
for domain experts to model situations in terms of generic STT building blocks and demonstrate how these building blocks can express situations of
different types, and be physically implemented in diverse applications.
Figure above shows a high level situation description gets recursively partitioned into explicitly observable or computable components.
Relevant publications:
1. V.K. Singh, M. Gao, R. Jain, Situation Recogintion: An evolving problem for Dynamic Big Multimedia Data , ACM Multimedia Conference(ACM MM-12).
B) Social Pixels: Data-representation Approach for Situation Recognition
Abstract:
Huge amounts of social multimedia is being created daily by a combination of globally distributed disparate sensors, including human-sensors (e.g. tweets) and video cameras. Taken together, these represent information about multiple aspects of the evolving world. Understanding the various events, patterns and situations emerging in such data has applications in multiple domains. We develop abstractions and tools to decipher various spatio-temporal phenomena which manifest themselves across such social media data. We describe an approach for aggregating social interest of users about any particular theme from any particular location into `social pixels’. Aggregating such pixels spatio-temporally allows creation of social versions of images and videos, which then become amenable to various media processing techniques (like segmentation, convolution) to derive semantic situation information. We define a declarative set of operators upon such data to allow for users to formulate queries to visualize, characterize, and analyze such data. Results of applying these operations over an evolving corpus of millions of Twitter and Flickr posts, to answer situation-based queries in multiple application domains are promising.
Example — iPhone related tweets peak on the date of launch of a new iPhone model.
Note: Green, Yellow, and Red represent number of people talking about the term ‘iphone’ from that location. Clearly our footprint in the social media can be used to detect real world events and situations.
Relevant publications:
1. V.K. Singh, M. Gao, R. Jain, Social Pixels: Genesis and Evaluation , ACM Multimedia Conference(ACM MM-10).
C) Situation Recognition Toolkit: EventShop
EventShop is our initial effort towards developing a framework over which applications for Social Life Networks will be developed.
These applications make use of heterogeneous streams of data in order to detect situations. EventShop has abilities to process streaming data
from heterogeneous data sources, detect various situations, as well as send out appropriate alerts to relevant users based on their current situations.
Users can configure the system to detect different situations at different granularities, and take corresponding actions. In addition, our simple and
user friendly GUI allows nontechnical users to experiment with different data streams, and integrate them for diverse applications.
EventShop draws inspiration from ImageShop. ImageShop provides several image processing operators which a user can interactively play around with.
In the same way, EventShop lets a user to formulate and process a query on spatio-temporal data by using several spatio-temporal operators in the
Operators Panel. The user can register data streams in the Data Source Panel, and then select one or more data streams to process. There is a panel
to view the intermediate queries. Once the user is satisfied with the query formulation, it can be executed. Result of the query would be displayed
in the Results Panel. The following video demonstrates how EventShop can be used to find regions within a hurricane’s path are not adequate covered by relief shelters.
https://www.youtube.com/watch?v=Y0pr3x66rvU
You can play with the SandBox version.
Relevant Publications:
1. M. Gao, V.K. Singh, R. Jain, EventShop: From Heterogeneous Web Streams to Personalized Situation Detection and Control, Web Science Conference, 2012.
2. V.K. Singh, M. Gao, R. Jain, EventShop: Recognizing situations in web data streams, S. Pongpaichet, World Wide Web conference: Workshop on Web Observatories, 2013.
D) Toward Social Life Networks
By using the enormous reach of mobile phones equipped with myriads of sensors, combined with Internet of Things, and current powerful computing infrastructure,
the next generation of social networks can be designed not only to connect people with other people, but to connect people with other people and essential life resources.
This work builds towards a bigger ambitious umbrella of ‘Social Life Networks’.
More details are here.
Relevant Publications:
1). R. Jain, V.K. Singh, and M. Gao, Social Life Networks for the Middle of the Pyramid. WWW Workshop on Social Media Engagement, 2011.