Python 30 天 – 第 23 天 – 网页抓取（python爬虫网站抓取）

# loop though all links    
for idx, link in enumerate(links_list):    
    # fetch the title of the post    
    post_title = link.get_text()    
    # fetch the link of the post    
    post_href = link.get('href')    
    # fetch the point text using the index of the link    
    # convert the point to integer    
    post_points = int(points_list[idx].get_text().replace(' points', ''))    
    # append to popular posts as a dictionary object if points is atleast 100    
    if post_points >= 100:    
        popular_posts.append(    
        {'title': post_title, 'link': post_href, 'points': post_points})  

上述脚本仅从 Hacker News 的第一页获取热门帖子。但是，根据所需的目标，我们需要从前五页或可能输入的任何页数中获取列表。因此可以相应地修改脚本。

import requests    
from bs4 import BeautifulSoup    
import pprint    
import time    
    
BASE_URL = 'https://news.ycombinator.com'    
# response = requests.get(BASE_URL)    
    
def get_lists_and_points(soup):    
    # extract all the links using the class selector    
    links_list = soup.select('.storylink')    
    
    # extract all the points using the class selector    
    points_list = soup.select('.score')    
    
    return (links_list, points_list)    
    
def parse_response(response):    
    # extract the text content of the web page    
    response_text = response.text    
    # parse HTML    
    soup = BeautifulSoup(response_text, 'html.parser')    
    return soup    
    
def get_paginated_data(pages):    
    total_links_list = []    
    total_points_list = []    
    for page in range(pages):    
        URL = BASE_URL + f'?p={page+1}'    
        response = requests.get(URL)    
        soup = parse_response(response)    
        links_list, points_list = get_lists_and_points(soup)    
        for link in links_list:    
            total_links_list.append(link)    
        for point in points_list:    
            total_points_list.append(point)    
        # add 30 seconds delay as per hacker news robots.txt rules    
        time.sleep(30)    
    return (total_links_list, total_points_list)    
    
def generate_popular_posts(links_list, points_list):    
    # create an empty popular posts list    
    popular_posts = []    
    
    # loop though all links    
    for idx, link in enumerate(links_list):    
        # fetch the title of the post    
        post_title = link.get_text()    
        # fetch the link of the post    
        post_href = link.get('href')    
        # fetch the point text using the index of the link    
        # convert the point to integer    
        # if points data is not available, assign it a default of 0    
        try:    
            post_points = int(    
                points_list[idx].get_text().replace(' points', ''))    
        except:    
            points_list = 0    
        # append to popular posts as a dictionary object if points is atleast 100    
        if post_points >= 100:    
            popular_posts.append(    
                {'title': post_title, 'link': post_href, 'points': post_points})    
    return popular_posts    
    
def sort_posts_by_points(posts):    
    return sorted(posts, key=lambda x: x['points'], reverse=True)    
    
def main():    
    total_links_list, total_points_list = get_paginated_data(5)    
    popular_posts = generate_popular_posts(total_links_list, total_points_list)    
    sorted_posts = sort_posts_by_points(popular_posts)    
    # print posts sorted by highest to lowest    
    pprint.pprint(sorted_posts)    
    
if(__name__ == '__main__'):    
    main()    

现在使用这个脚本，我们甚至不需要访问 Hacker News 和搜索热门新闻。我们可以从控制台运行此脚本并获取最新消息。随意根据您的需要调整脚本并对其进行试验或尝试从您自己喜欢的网站上抓取数据。

我们可以用上面的数据做很多事情，比如

流行的爬虫库

Beautiful Soup 在从网站上抓取数据时有其限制。它使用起来非常简单，但是为了从在客户端呈现的复杂网站（Angular、基于 React 的网站）中抓取数据，当网站加载时 HTML 标记将不可用。要从此类网站获取数据，可以使用更高级的库。以下是一些流行的 Python 库和框架。

网页抓取是一个广阔的领域。使用 Beautiful Soup，我们可能只是触及了表面。在这个领域有很多可能性，我将在探索更多关于使用 Python 的数据分析的同时进行探索。希望我已经能够涵盖进一步探索所需的基本概念。

明天我将讨论使用 Python 进行 Web 开发的概念。