针对微博的免登录分布式网络爬虫的研究

Research on Distributed Web Crawler without login for Microblog

摘要:

微博作为优质的数据源，其中的数据非常适合做舆情分析等。新浪官方提供的API限制数据采集速度，而利用模拟登录的网络爬虫采集数据又相对复杂且会降低效率。针对这些问题，设计了一个免登录的微博网络爬虫。通过实验表明，该爬虫可以更快的对微博数据进行完整稳定的采集。随着对数据需求量越来越大，单机网络爬虫已经不足以满足要求，将Hadoop分布式计算平台与免登录爬虫相结合，设计了一个基于MapReduce的分布式网络爬虫系统，利用多台计算机组成的集群，实现短时间内免登录抓取海量微博数据。通过实验证明，该爬虫系统可以每天稳定抓取近千万条微博。

Abstract:

Weibo is a good source of data,and the data is very suitable for public opinion analysis.The API provided by Sina officially limits the speed of data collection,and the network crawler using simulated login is relatively complicated and reduces efficiency.For these problems,a crawler without login for Weibo is designed.Experiments show that the crawler can perform complete and stable collection of Weibo data more quickly.With the increasing demand for data,the single network crawler can’t meet the requirements.The Hadoop distributed computing platform is combined with the crawler without login to design a distributed network crawler system based on MapReduce.Using a cluster of multiple computers, you can capture massive amounts of Weibo data in a short period of time.Through experiments,the crawler system can stably capture nearly 10 million micro blog per day.