Master
pyspider00
data_db(存储抓取数据)
- phantomjs
- scheduler
- processor
- –phantomjs-proxy=’localhost:25555′ fetcher
- result_worker
- webui
Slave_01
pyspider01
taskdb,projectdb,resultdb(存储主要任务和脚本数据,比较重要)
- phantomjs
- processor
- –phantomjs-proxy=”localhost:25555″ fetcher
Slave_02
pyspider02
- phantomjs
- processor
- –phantomjs-proxy=”localhost:25555″ fetcher