PHP前端开发

# 使用 ThreadPoolExecutor 增强你的 Python 任务

百变鹏仔 5天前 #Python
文章标签 ThreadPoolExecutor

当涉及到在 python 中同时运行多个任务时,concurrent.futures 模块是一个强大而简单的工具。在本文中,我们将探讨如何使用 threadpoolexecutor 并行执行任务,并结合实际示例。

为什么使用threadpoolexecutor?

在python中,线程非常适合i/o操作占主导地位的任务,例如网络调用或文件读/写操作。使用 threadpoolexecutor,您可以:


示例:并行运行任务

让我们看一个简单的例子来理解这个概念。

守则

from concurrent.futures import threadpoolexecutorimport time# function simulating a taskdef task(n):    print(f"task {n} started")    time.sleep(2)  # simulates a long-running task    print(f"task {n} finished")    return f"result of task {n}"# using threadpoolexecutordef execute_tasks():    tasks = [1, 2, 3, 4, 5]  # list of tasks    results = []    # create a thread pool with 3 simultaneous threads    with threadpoolexecutor(max_workers=3) as executor:        # execute tasks in parallel        results = executor.map(task, tasks)    return list(results)if __name__ == "__main__":    results = execute_tasks()    print("all results:", results)

预期输出

当您运行此代码时,您将看到类似这样的内容(以某种并行顺序):

task 1 startedtask 2 startedtask 3 startedtask 1 finishedtask 4 startedtask 2 finishedtask 5 startedtask 3 finishedtask 4 finishedtask 5 finishedall results: ['result of task 1', 'result of task 2', 'result of task 3', 'result of task 4', 'result of task 5']

任务 1、2 和 3 同时启动,因为 max_workers=3。其他任务(4 和 5)等待线程可用。

立即学习“Python免费学习笔记(深入)”;


何时使用它?

典型用例:


最佳实践

  1. 限制线程数:

  2. 处理异常:

  3. 使用 processpoolexecutor 执行 cpu 密集型任务:


高级示例:并行获取 url

这是一个真实的示例:并行获取多个 url。

import requestsfrom concurrent.futures import ThreadPoolExecutor# Function to fetch a URLdef fetch_url(url):    try:        response = requests.get(url)        return f"URL: {url}, Status: {response.status_code}"    except Exception as e:        return f"URL: {url}, Error: {e}"# List of URLs to fetchurls = [    "https://example.com",    "https://httpbin.org/get",    "https://jsonplaceholder.typicode.com/posts",    "https://invalid-url.com"]def fetch_all_urls(urls):    with ThreadPoolExecutor(max_workers=4) as executor:        results = executor.map(fetch_url, urls)    return list(results)if __name__ == "__main__":    results = fetch_all_urls(urls)    for result in results:        print(result)

结论

threadpoolexecutor 简化了 python 中的线程管理,是加速 i/o 密集型任务的理想选择。只需几行代码,您就可以并行操作并节省宝贵的时间。