Identifying Real World Concepts in Social Media
The benefits of mining social media are rapidly increasing with social media's increasing popularity and the increasing amount of social media data available. Mining social media presents new difficulties, however, because users' posts are informal and often do not even claim to be authoritative. Users often describe concepts using lay phrases that cannot be mapped to an existing knowledge base, users often change the word choices and constructions in these phrases rather than treating them as a precise vocabulary, and users often post anecdotes or reactions to news stories that must be separated from users' descriptions of their experiences. Many types of social media mining require that these difficulties be addressed. Users' vocabularies must be mapped onto a knowledge base before a system can determine how concepts expressed by one user relate to concepts expressed by another, and many types of mining are intended to be performed only on posts describing users' first person experiences.In this dissertation I propose and evaluate methods for overcoming these difficulties. I propose a synonym discovery method for discovering equivalent lay phrases and mapping between lay phrases and expert terms in a knowledge base; I show this method performs substantially better than comparable previously proposed synonym discovery methods (i.e., methods that do not require query logs or parallel corpora). I propose several concept extraction methods that use a thesaurus to identify variations of known lay phrases, as are often created when users speak informally. These methods outperform existing concept extraction methods as measured by both F1 and Precision. I then describe an architecture for identifying real world trends using the structured output of the synonym discovery and concept extraction methods, and describe methods for quantifying the influence of the news cycle on social media activity and reducing the impact of incomplete information by learning relationships between user mentions of symptoms, conditions, and drugs. Finally, I investigate the effect of Twitter's sampling on trend detection performance, find that Twitter's publicly available 1% sample is not always representative of the whole, and suggest heuristics for determining when Twitter's 1% sample may be used.
MetadataShow full item record
Showing items related by title, author, creator and subject.