本教程详细介绍了如何使用python的`collections.counter`模块高效、准确地统计两段文本或一个关键词列表与文本之间共同词汇的出现频率。通过实际代码示例，我们将学习文本预处理、词频统计以及如何查找并汇总所有共同词汇或特定关键词的出现次数，从而解决简单集合交集无法提供词频信息的局限性。
在文本分析任务中，我们经常需要找出两段文本或一个预设关键词列表与一段文本之间共同的词汇，并且更重要的是，需要统计这些共同词汇在各自文本中出现的总次数。传统的集合操作虽然能快速找出共同元素，但无法提供每个元素出现的频率信息。本文将通过collections.Counter这一强大的工具，详细讲解如何高效且准确地实现这一目标。
1. 准备工作：文本获取与初步清洗
在进行词频统计之前，我们需要获取文本内容并对其进行初步清洗，例如转换为小写、移除标点符号等，以确保统计的准确性。这里我们以从网页获取文本为例。
import requests
import re
from collections import Counter
from bs4 import BeautifulSoup
# 示例：获取两个维基百科页面的内容
url_a = 'https://en.wikipedia.org/wiki/Leonhard_Euler'
url_b = 'https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss'
# 使用requests和BeautifulSoup获取并解析网页文本
try:
txt_a = BeautifulSoup(requests.get(url_a).content, 'html.parser').get_text()
txt_b = BeautifulSoup(requests.get(url_b).content, 'html.parser').get_text()
except requests.exceptions.RequestException as e:
print(f"获取网页内容失败: {e}")
# 提供备用文本或退出
txt_a = "Leonhard Euler was a pioneering Swiss mathematician and physicist. He made important discoveries in fields as diverse as infinitesimal calculus and graph theory. He also introduced much of the modern mathematical terminology and notation, particularly for mathematical analysis, such as the notion of a mathematical function. He is also renowned for his work in mechanics, fluid dynamics, optics, astronomy, and music theory. Euler was one of the most eminent mathematicians of the 18th century and is considered one of the greatest mathematicians of all time. He is also widely considered to be the most prolific mathematician ever, with his collected works filling 60–80 quarto volumes. He spent most of his adult life in St. Petersburg, Russia, and in Berlin, Prussia. A statement attributed to Pierre-Simon Laplace expresses Euler's influence on mathematics: Read Euler, read Euler, he is the master of us all."
txt_b = "Carl Friedrich Gauss was a German mathematician and physicist who made significant contributions to many fields in mathematics and science. Gauss contributed to number theory, algebra, statistics, analysis, differential geometry, geodesy, geophysics, mechanics, electrostatics, astronomy, matrix theory, and optics. Sometimes referred to as the Princeps mathematicorum (Latin for 'the foremost of mathematicians') and 'the greatest mathematician since antiquity', Gauss had an exceptional influence in many fields of mathematics and science and is ranked as one of history's most influential mathematicians. He was a child prodigy. Gauss completed his magnum opus, Disquisitiones Arithmeticae, in 1798 at the age of 21, though it was not published until 1801. This work was fundamental in consolidating number theory as a discipline and has shaped the field to the present day."
# 初步清洗：转换为小写并按非字母数字字符分割
# re.split(r'\W+', text.lower()) 会将所有非字母数字字符作为分隔符，并返回一个词汇列表
words_a = re.split(r'\W+', txt_a.lower())
words_b = re.split(r'\W+', txt_b.lower())
# 移除可能存在的空字符串（例如，文本开头或结尾的非字母数字字符）
words_a = [word for word in words_a if word]
words_b = [word for word in words_b if word]
print(f"文本A词汇数量: {len(words_a)}")
print(f"文本B词汇数量: {len(words_b)}")2. 使用 collections.Counter 进行词频统计
collections.Counter是一个字典的子类，用于存储可哈希对象的计数。它非常适合统计列表中元素的出现频率。
# 使用Counter统计每个文本的词频
cnt_a = Counter(words_a)
cnt_b = Counter(words_b)
print("\n文本A中'euler'的出现次数:", cnt_a['euler'])
print("文本B中'gauss'的出现次数:", cnt_b['gauss'])
print("文本A中'the'的出现次数:", cnt_a['the'])
print("文本B中'the'的出现次数:", cnt_b['the'])3. 统计所有共同词汇及其频率
要找出两段文本中所有共同的词汇及其在各自文本中的出现频率，我们可以利用Counter对象可以像集合一样进行交集操作的特性。cnt_a & cnt_b会返回一个新的Counter对象，其中包含两个Counter对象中都存在的元素，并且其计数是两个原始Counter中对应计数的最小值。然而，为了获取每个词在各自文本中的实际计数，我们需要遍历交集后的键，并从原始Counter中检索。
立即学习“Python免费学习笔记（深入）”；
# 找出所有共同词汇（键的交集）
common_words_keys = cnt_a.keys() & cnt_b.keys()
# 构建一个字典，存储共同词汇及其在两文本中的频率
# 按照总频率降序排列
all_common_word_counts = {}
for word in common_words_keys:
all_common_word_counts[word] = (cnt_a[word], cnt_b[word])
# 打印出现频率最高的共同词汇
sorted_common_words = sorted(
all_common_word_counts.items(),
key=lambda kv: sum(kv[1]), # 按两文本总出现次数排序
reverse=True
)
print("\n出现频率最高的共同词汇及其在两文本中的次数 (前10名):")
for word, counts in sorted_common_words[:10]:
print(f"'{word}': (文本A: {counts[0]}, 文本B: {counts[1]})")
# 示例输出：
# 'the': (文本A: 596, 文本B: 991)
# 'of': (文本A: 502, 文本B: 874)
# 'in': (文本A: 226, 文本B: 576)
# ...4. 统计指定关键词的出现频率
如果我们的目标是统计一个预设的关键词列表（wordset）中每个词汇在特定文本中出现的频率，可以使用dict.get()方法安全地从Counter对象中获取计数，即使关键词不存在也不会报错。
# 定义一个关键词列表
wordset = {'mathematical', 'equation', 'theorem', 'university', 'integral', 'basel', 'physics'}
print("\n指定关键词在文本A中的出现次数:")
for word in wordset:
# 使用.get()方法，如果词不存在则返回0
count = cnt_a.get(word, 0)
if count > 0:
print(f"'{word}': {count}")
print("\n指定关键词在文本B中的出现次数:")
for word in wordset:
count = cnt_b.get(word, 0)
if count > 0:
print(f"'{word}': {count}")
# 也可以直接生成一个字典：
keyword_counts_a = {k: cnt_a.get(k, 0) for k in wordset}
keyword_counts_b = {k: cnt_b.get(k, 0) for k in wordset}
print("\n文本A中关键词统计结果:", keyword_counts_a)
print("文本B中关键词统计结果:", keyword_counts_b)注意事项
- 文本预处理的重要性： 词频统计的结果质量高度依赖于文本预处理的彻底性。除了小写转换和标点移除，可能还需要进行词形还原（lemmatization）或词干提取（stemming）来处理单词的不同形式（如"running"、"ran"、"runs"都归为"run"），以及移除停用词（stop words，如"the", "is", "a"）以聚焦于更具信息量的词汇。
- 性能考量： collections.Counter在C语言层面实现，性能非常高。对于非常大的文本，它的效率远高于手动编写的计数循环。
- 内存使用： Counter会将所有唯一词汇及其计数存储在内存中。对于极大规模的文本数据，如果唯一词汇数量庞大，可能需要考虑分块处理或使用更内存高效的数据结构（如基于数据库的计数）。
- 灵活的词汇定义： re.split(r'\W+', ...) 是一种简单的词汇分割方式。根据具体需求，可能需要更复杂的正则表达式来处理连字符词、数字、特定领域术语等。
总结
collections.Counter是Python标准库中一个非常实用的工具，它极大地简化了词频统计和集合交集操作中词汇计数的复杂性。通过结合文本预处理技术，我们可以准确、高效地分析文本数据，无论是找出所有共同词汇的频率，还是统计特定关键词在文本中的出现次数。掌握这一技术，将为您的文本分析项目提供坚实的基础。
