config.yaml

我来为您介绍一个精简版的OpenClaw（网络爬虫框架）设计：

核心架构

OpenClaw-Lite
├── 调度器 (Scheduler)
├── 下载器 (Downloader)
├── 解析器 (Parser)
├── 数据处理器 (Pipeline)
└── 配置文件 (Config)

调度器 (Scheduler)

class Scheduler:
    def __init__(self):
        self.queue = deque()  # 待爬取URL队列
        self.visited = set()  # 已访问URL集合
    def add_url(self, url):
        if url not in self.visited:
            self.queue.append(url)
    def get_url(self):
        return self.queue.popleft() if self.queue else None

下载器 (Downloader)

class Downloader:
    def __init__(self, max_retries=3):
        self.session = requests.Session()
        self.max_retries = max_retries
    def fetch(self, url, headers=None):
        for i in range(self.max_retries):
            try:
                response = self.session.get(url, headers=headers, timeout=10)
                response.encoding = 'utf-8'
                return response.text
            except Exception as e:
                print(f"下载失败: {url}, 重试 {i+1}/{self.max_retries}")
        return None

解析器 (Parser)

class Parser:
    @staticmethod
    def parse_html(html, rules):
        """根据规则解析HTML"""
        soup = BeautifulSoup(html, 'html.parser')
        results = {}
        for key, selector in rules.items():
            if selector.get('type') == 'xpath':
                # XPath解析
                pass
            else:
                # CSS选择器
                elements = soup.select(selector['selector'])
                if selector.get('attr'):
                    results[key] = [el.get(selector['attr']) for el in elements]
                else:
                    results[key] = [el.text.strip() for el in elements]
        return results

核心爬虫类

class OpenClawLite:
    def __init__(self, config):
        self.config = config
        self.scheduler = Scheduler()
        self.downloader = Downloader()
        self.parser = Parser()
        self.data_pipeline = []
    def run(self):
        # 1. 添加初始URL
        for url in self.config.start_urls:
            self.scheduler.add_url(url)
        # 2. 开始爬取
        while True:
            url = self.scheduler.get_url()
            if not url:
                break
            # 下载
            html = self.downloader.fetch(url, self.config.headers)
            # 解析
            data = self.parser.parse_html(html, self.config.rules)
            # 数据处理
            for processor in self.data_pipeline:
                processor.process(data)
            # 提取新URL
            new_urls = self.extract_urls(html)
            for new_url in new_urls:
                self.scheduler.add_url(new_url)
            time.sleep(self.config.delay)

配置示例

  - "https://example.com/page1"
  - "https://example.com/page2"
headers:
  User-Agent: "OpenClaw-Lite/1.0"
rules:
    selector: "h1.title"
  links:
    selector: "a[href]"
    attr: "href"
delay: 1.0  # 爬取间隔
max_depth: 3  # 最大深度

扩展功能（可选）

1 并发版本

from concurrent.futures import ThreadPoolExecutor
class ConcurrentClaw(OpenClawLite):
    def run(self, workers=4):
        with ThreadPoolExecutor(max_workers=workers) as executor:
            while True:
                urls = [self.scheduler.get_url() for _ in range(workers)]
                urls = [url for url in urls if url]
                if not urls:
                    break
                futures = [executor.submit(self.crawl, url) for url in urls]
                for future in futures:
                    future.result()

2 数据导出

class ExportPipeline:
    def __init__(self, output_format='csv'):
        self.output_format = output_format
    def process(self, data):
        if self.output_format == 'csv':
            self.to_csv(data)
        elif self.output_format == 'json':
            self.to_json(data)
    def to_csv(self, data):
        with open('output.csv', 'a', encoding='utf-8') as f:
            writer = csv.writer(f)
            writer.writerow(data.values())

使用示例

# 创建爬虫
claw = OpenClawLite(config)
# 添加数据处理管道
claw.data_pipeline.append(ExportPipeline('json'))
# 运行爬虫
claw.run()

特点

1. 轻量：约200行核心代码
2. 易扩展：模块化设计
3. 实用：支持基本的爬取-解析-存储流程
4. 可配置：YAML配置文件

这个精简版保留了爬虫的核心功能,适合学习和中小规模项目使用，如需完整功能，可以基于此框架逐步扩展。

标签： 配置 YAML