IEEE ICSC 2016 · Tenth International Conference on Semantic Computing
A Scalable Approach for Sentiment Analysis of Turkish Tweets and Linking Tweets to News
Summary
At a Glance
A framework that collects Turkish news and related tweets, links each tweet to a news item and classifies its sentiment, so that the public reaction to every news item can be counted, all on a Hadoop cluster.
Naive Bayes, Complement Naive Bayes and logistic regression from Apache Mahout.
Zemberek handles stemming and misspelled words in an agglutinative language.
Two MapReduce jobs process new tweets every ten minutes and merge the results.
1 · Introduction
What Does Twitter Think about the News?
- Decisions depend on "what other people think"; sentiment analysis finds out whether a text is positive or negative.
- Two families: supervised machine learning (train on labelled data) and semantic orientation (unsupervised, lexicon-like).
- Tweets arrive continuously, so the classifier must scale: the work is distributed with Hadoop (HDFS + MapReduce) and Mahout.
- Most earlier work is on English; this study builds a Turkish dataset and applies the same methods to Turkish.
Agglutinative: one root takes many suffixes, e.g. gel-mi-yor-um "I am not coming". Words with the same root can mean different things.
Short texts, misspellings, missing Turkish characters, URLs, user names and "RT" without sentiment.
2 · Related Work · Interactive
Related Work
Click a column header to sort.
| Study | Approach | Data | Finding |
|---|---|---|---|
| Pang et al. | NB, maximum entropy, SVM | movie reviews | machine learning beats human-made baselines; unigrams beat bigrams |
| Pak and Paroubek | emoticon-labelled training data, n-grams | Twitter (English) | bigrams work best; URLs, stop words and special words removed |
| Khuc et al. | lexicon builder + Mahout classifier on MapReduce | real-time distributed system, 73.7% accuracy | |
| Pandarachalil et al. | own unigram NB on Hadoop MapReduce and HDFS | Amazon movie reviews | 82% accuracy |
| Abel et al. | bag-of-words matching | tweets and news | links tweets to news articles to give them context |
| This study | NB, Complement NB, logistic regression on Hadoop + Mahout; Zemberek | 6,000 Turkish tweets | 79.38% accuracy; tweet-to-news linking |
3 · Method · Fig. 1
From News Feed to Sentiment per News Item
New tweets are collected in ten-minute batches and analysed after each batch.
JDOM and Rome parse the RSS XML; Twitter4J streams tweets.
For each news item: how many supporters and opposers it has on Twitter.
3.3 · Tables 1–2
Turkish NLP with Zemberek
Stemming (Table 1)
| Word | Root | Type |
|---|---|---|
| gelmiyorum | gel | verb |
| gitmedim | git | verb |
| çok | çok | adjective |
| erken | erken | adverb |
| hakaret | hakaret | noun |
| özgürlük | özgür | noun |
Word similarity (Table 2)
| Word 1 | Word 2 | Score |
|---|---|---|
| Yaşasın | yaşasın | 0.904 |
| kötüyüm | kotuyüm | 0.768 |
| akın | yakın | 0.933 |
| akın | sakın | 0.933 |
| hakaret | hakret | 0.966 |
| özgürlük | özgürlüğü | 0.935 |
Note: akın/yakın pass the 0.90 threshold although they are different words.
3.3 · Algorithm 2 · Interactive
Mapping Lab: Which News Is This Tweet About?
| News item (made up) | Score | Matching word pairs |
|---|
Illustrative: made-up news and tweets; edit the tweet freely. Every word pair above the threshold adds one point, the top-scoring item wins. Zemberek's stemming and similarity are approximated by edit distance on ASCII-folded words plus a shared-stem rule.
3.4 · Figs. 2–3
Two MapReduce Jobs
Input: (news item, tweet) pairs from the last ten minutes.
Map: classify each tweet with the model trained in Mahout beforehand.
Reduce: per news item, count positive, neutral and negative tweets (popularity, supporters, opposers) and write the result to HDFS.
Input: the new batch result and all earlier results since system start.
Map: key each record by a hash id of its news item.
Reduce: add up the counts per key into one consolidated result.
4.1 · Dataset · Table 3
A New Labelled Turkish Tweet Dataset
- 6,000 tweets from early 2015, each longer than 50 characters.
- Average length: 9.43 words, 75.3 characters.
- 80% training, 20% test; metrics: accuracy, precision, F1.
Every tweet labelled positive, negative or neutral and checked by 10 graduate students.
Half of the tweets are negative, which motivates Complement Naive Bayes.
Published on GitHub: sercankulcu/twitterdata.
4.2 · Classifiers · Interactive
Naive Bayes vs. Complement Naive Bayes
- Naive Bayes: sum the evidence of each word, plus the class prior, which favours the larger class.
- Complement NB (Rennie et al.) learns from the complement classes and drops the prior; with two classes that missing prior is the visible difference. On maç and yeni karar the two disagree.
Illustrative: 9 made-up training tweets (6 negative, 3 positive), Laplace smoothing; unknown words are ignored. Blue: towards positive, orange: towards negative.
4.2 · Tables 4, 6, 7, 8 · Interactive
Experiments Explorer
4.2 · Table 5 · Interactive
How Much Data Is Enough?
4.3–4.4 · Tables 9–10
Mapping Accuracy and Scalability
Tweet-to-news mapping
218 tweets for 29 news items, checked by hand. Plain bag-of-words matching is the weak link of the pipeline; lexicon-based strategies were the next step.
Execution time, 100 MB
Six nodes finish in 637 s instead of 2,137 s. The speed-up is real but well below 6×: job start-up and data shuffling do not shrink with more nodes.
5 · Conclusion and Future Work
Conclusion
- A scalable framework links Turkish tweets to news items and rates each item by its positive, neutral and negative tweets.
- On 6,000 manually labelled Turkish tweets, the Bayes classifiers (Complement NB 79.38%, NB 78.61%) clearly beat logistic regression (63.12%).
- Unigrams are the better feature model, and two-class problems are easier than three-class ones.
- Mapping by bag of words reaches 40.3%; better mapping, e.g. lexicon-based, is future work.
5 · Conclusion
Check Questions
- Why does Turkish need stemming before tweets and news can be compared word by word?
- What problem of Naive Bayes does Complement Naive Bayes address, and why does it matter for this dataset?
- Why can bigrams and trigrams do worse than unigrams on 6,000 tweets?
- Why is the second MapReduce job needed?
- Six nodes give a 3.4× speed-up, not 6×. Why?
Show answers
- Turkish is agglutinative: the same root appears with many suffixes, so surface forms rarely match exactly.
- Bias towards the class with more training examples; half of the tweets are negative.
- More, sparser features on a small dataset make overfitting easier.
- To merge each ten-minute batch result with all earlier results into one consolidated count per news item.
- Fixed overheads such as job start-up, data distribution and the shuffle between map and reduce do not shrink with more nodes.
References
Selected References
- Pang, B., Lee, L., Vaithyanathan, S. (2002). Thumbs up? Sentiment classification using machine learning techniques. Proc. EMNLP, 79–86.
- Rennie, J. D. et al. (2003). Tackling the poor assumptions of naive Bayes text classifiers. ICML, 616–623.
- Pak, A., Paroubek, P. (2010). Twitter as a corpus for sentiment analysis and opinion mining. LREC, 1320–1326.
- Khuc, V. N. et al. (2012). Towards building large-scale distributed systems for Twitter sentiment analysis. ACM SAC, 459–464.
- Abel, F. et al. (2011). Semantic enrichment of Twitter posts for user profile construction on the social web. ESWC, 375–389.
- Can, F. et al. (2008). Information retrieval on Turkish texts. JASIST, 59(3), 407–421.
See the full paper for the complete reference list. Charts on these slides are drawn from the paper's tables.