Spider·爬取小说

导入资源包

import requests
import bs4

获取链接

url = 'https://www.17k.com/top/refactor/top100/18_popularityListScore/18_popularityListScore_finishBook_top_100_pc.html?TabIndex=1&typeIndex=0'

伪装用户

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'}

发送网络请求

req=requests.get(url=url,headers=headers)

将网络请求的响应由二进制转换为可读模式

req.encoding='utf-8'

将爬取到的内容进行装饰

soup=bs4.BeautifulSoup(req.text,"html.parser")	

将小说的序号，分类，名称，作者分别爬取出来

div=soup.find_all('div',attrs={'class':'BOX'})[1]
tr_list=div.find_all('tr')
try:
    for item in tr_list[1:]:
        id=item.find('td').text
        sign=item.find('a').text
        book=item.find('a',attrs={'class':'red'}).text
        author=item.find_all('a')[3].text
        print(id,sign,book,author)


except Exception:
    print('异常')

原文地址：http://www.cnblogs.com/JK8395/p/16831091.html