Python进阶:并发编程之Futures
2019-07-24 09:14:43来源:博客园 阅读 ()
区分并发和并行
并发(Concurrency).
并行(Parallelism)
并发编程之 Futures
单线程与多线程性能比较
import requests import time def download_one(url): resp = requests.get(url) print('Read {} from {}'.format(len(resp.content), url)) def download_all(sites): for site in sites: download_one(site) def main(): sites = [ 'https://en.wikipedia.org/wiki/Portal:Arts', 'https://en.wikipedia.org/wiki/Portal:History', 'https://en.wikipedia.org/wiki/Portal:Society', 'https://en.wikipedia.org/wiki/Portal:Biography', 'https://en.wikipedia.org/wiki/Portal:Mathematics', 'https://en.wikipedia.org/wiki/Portal:Technology', 'https://en.wikipedia.org/wiki/Portal:Geography', 'https://en.wikipedia.org/wiki/Portal:Science', 'https://en.wikipedia.org/wiki/Computer_science', 'https://en.wikipedia.org/wiki/Python_(programming_language)', 'https://en.wikipedia.org/wiki/Java_(programming_language)', 'https://en.wikipedia.org/wiki/PHP', 'https://en.wikipedia.org/wiki/Node.js', 'https://en.wikipedia.org/wiki/The_C_Programming_Language', 'https://en.wikipedia.org/wiki/Go_(programming_language)' ] start_time = time.perf_counter() download_all(sites) end_time = time.perf_counter() print('Download {} sites in {} seconds'.format(len(sites), end_time - start_time)) if __name__ == '__main__': main() # 输出 Read 129196 from https://en.wikipedia.org/wiki/Portal:Arts Read 183867 from https://en.wikipedia.org/wiki/Portal:History Read 224161 from https://en.wikipedia.org/wiki/Portal:Society Read 114387 from https://en.wikipedia.org/wiki/Portal:Biography Read 152871 from https://en.wikipedia.org/wiki/Portal:Mathematics Read 156339 from https://en.wikipedia.org/wiki/Portal:Technology Read 162872 from https://en.wikipedia.org/wiki/Portal:Geography Read 91504 from https://en.wikipedia.org/wiki/Portal:Science Read 323262 from https://en.wikipedia.org/wiki/Computer_science Read 391073 from https://en.wikipedia.org/wiki/Python_(programming_language) Read 319710 from https://en.wikipedia.org/wiki/Java_(programming_language) Read 470754 from https://en.wikipedia.org/wiki/PHP Read 180774 from https://en.wikipedia.org/wiki/Node.js Read 56799 from https://en.wikipedia.org/wiki/The_C_Programming_Language Read 325451 from https://en.wikipedia.org/wiki/Go_(programming_language) Download 15 sites in 67.349395015 seconds
import concurrent.futures import requests import threading import time def download_one(url): try: resp = requests.get(url) print('Read {} from {}'.format(len(resp.content), url)) except Exception as ex: print(ex) def download_all(sites): with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: results = executor.map(download_one, sites) # with concurrent.futures.ProcessPoolExecutor() as executor: # results = executor.map(download_one,sites) def main(): sites = [ 'https://en.wikipedia.org/wiki/Portal:Arts', 'https://en.wikipedia.org/wiki/Portal:History', 'https://en.wikipedia.org/wiki/Portal:Society', 'https://en.wikipedia.org/wiki/Portal:Biography', 'https://en.wikipedia.org/wiki/Portal:Mathematics', 'https://en.wikipedia.org/wiki/Portal:Technology', 'https://en.wikipedia.org/wiki/Portal:Geography', 'https://en.wikipedia.org/wiki/Portal:Science', 'https://en.wikipedia.org/wiki/Computer_science', 'https://en.wikipedia.org/wiki/Python_(programming_language)', 'https://en.wikipedia.org/wiki/Java_(programming_language)', 'https://en.wikipedia.org/wiki/PHP', 'https://en.wikipedia.org/wiki/Node.js', 'https://en.wikipedia.org/wiki/The_C_Programming_Language', 'https://en.wikipedia.org/wiki/Go_(programming_language)' ] start_time = time.perf_counter() download_all(sites) end_time = time.perf_counter() print('Download {} sites in {} seconds'.format(len(sites), end_time - start_time)) if __name__ == '__main__': main() # 输出 Read 114387 from https://en.wikipedia.org/wiki/Portal:Biography Read 129196 from https://en.wikipedia.org/wiki/Portal:Arts Read 183867 from https://en.wikipedia.org/wiki/Portal:History Read 152871 from https://en.wikipedia.org/wiki/Portal:Mathematics Read 224161 from https://en.wikipedia.org/wiki/Portal:Society Read 156339 from https://en.wikipedia.org/wiki/Portal:Technology Read 91504 from https://en.wikipedia.org/wiki/Portal:Science Read 391073 from https://en.wikipedia.org/wiki/Python_(programming_language) Read 162872 from https://en.wikipedia.org/wiki/Portal:Geography Read 323262 from https://en.wikipedia.org/wiki/Computer_science Read 56799 from https://en.wikipedia.org/wiki/The_C_Programming_Language Read 319710 from https://en.wikipedia.org/wiki/Java_(programming_language) Read 325451 from https://en.wikipedia.org/wiki/Go_(programming_language) Read 180774 from https://en.wikipedia.org/wiki/Node.js Read 470754 from https://en.wikipedia.org/wiki/PHP Download 15 sites in 10.022916933 seconds
with futures.ThreadPoolExecutor(workers) as executor => with futures.ProcessPoolExecutor() as executor:
对于这种IO场景,用并行的方式并不会比并发的方式效率高.
到底什么是 Futures ?
import concurrent.futures import requests import time def download_one(url): resp = requests.get(url) print('Read {} from {}'.format(len(resp.content), url)) return f'download {len(resp.content)} ok' # def over(arg): # print(arg) # print('over') def download_all(sites): #future列表中每个future完成的顺序,和它在列表中的顺序并不一定完全一致。 #到底哪个先完成、哪个后完成,取决于系统的调度和每个future的执行时间 with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: to_do = [] for site in sites: #executor.submit返回future实例 future = executor.submit(download_one, site) to_do.append(future) #future.add_done_callback(over) #在futures完成后打印结果 for future in concurrent.futures.as_completed(to_do): print(future.result()) def main(): sites = [ 'https://en.wikipedia.org/wiki/Portal:Arts', 'https://en.wikipedia.org/wiki/Portal:History', 'https://en.wikipedia.org/wiki/Portal:Society', 'https://en.wikipedia.org/wiki/Portal:Biography', 'https://en.wikipedia.org/wiki/Portal:Mathematics', 'https://en.wikipedia.org/wiki/Portal:Technology', 'https://en.wikipedia.org/wiki/Portal:Geography', 'https://en.wikipedia.org/wiki/Portal:Science', 'https://en.wikipedia.org/wiki/Computer_science', 'https://en.wikipedia.org/wiki/Python_(programming_language)', 'https://en.wikipedia.org/wiki/Java_(programming_language)', 'https://en.wikipedia.org/wiki/PHP', 'https://en.wikipedia.org/wiki/Node.js', 'https://en.wikipedia.org/wiki/The_C_Programming_Language', 'https://en.wikipedia.org/wiki/Go_(programming_language)' ] start_time = time.perf_counter() download_all(sites) end_time = time.perf_counter() print('Download {} sites in {} seconds'.format(len(sites), end_time - start_time)) if __name__ == '__main__': main() # 输出 Read 129886 from https://en.wikipedia.org/wiki/Portal:Arts Read 107634 from https://en.wikipedia.org/wiki/Portal:Biography Read 224118 from https://en.wikipedia.org/wiki/Portal:Society Read 158984 from https://en.wikipedia.org/wiki/Portal:Mathematics Read 184343 from https://en.wikipedia.org/wiki/Portal:History Read 157949 from https://en.wikipedia.org/wiki/Portal:Technology Read 167923 from https://en.wikipedia.org/wiki/Portal:Geography Read 94228 from https://en.wikipedia.org/wiki/Portal:Science Read 391905 from https://en.wikipedia.org/wiki/Python_(programming_language) Read 321352 from https://en.wikipedia.org/wiki/Computer_science Read 180298 from https://en.wikipedia.org/wiki/Node.js Read 321417 from https://en.wikipedia.org/wiki/Java_(programming_language) Read 468421 from https://en.wikipedia.org/wiki/PHP Read 56765 from https://en.wikipedia.org/wiki/The_C_Programming_Language Read 324039 from https://en.wikipedia.org/wiki/Go_(programming_language) Download 15 sites in 0.21698231499976828 seconds
future列表中每个future完成的顺序,和它在列表中的顺序并不一定完全一致。到底哪个先完成、哪个后完成,取决于系统的调度和每个future的执行时间。
并发通常用于 I/O 操作频繁的场景,而并行则适用于 CPU heavy 的场景。
参考
极客时间《Python核心技术与实战》专栏
原文链接:https://www.cnblogs.com/xiaoguanqiu/p/11136665.html
如有疑问请与原作者联系
标签:
版权申明:本站文章部分自网络,如有侵权,请联系:west999com@outlook.com
特别注意:本站所有转载文章言论不代表本站观点,本站所提供的摄影照片,插画,设计作品,如需使用,请与原作者联系,版权归原作者所有
上一篇:NumPy基础操作(2)
下一篇:pymysql 的简单使用
- python3基础之“术语表(2)” 2019-08-13
- python3 之 字符串编码小结(Unicode、utf-8、gbk、gb2312等 2019-08-13
- Python3安装impala 2019-08-13
- 小白如何入门 Python 爬虫? 2019-08-13
- python_字符串方法 2019-08-13
IDC资讯: 主机资讯 注册资讯 托管资讯 vps资讯 网站建设
网站运营: 建站经验 策划盈利 搜索优化 网站推广 免费资源
网络编程: Asp.Net编程 Asp编程 Php编程 Xml编程 Access Mssql Mysql 其它
服务器技术: Web服务器 Ftp服务器 Mail服务器 Dns服务器 安全防护
软件技巧: 其它软件 Word Excel Powerpoint Ghost Vista QQ空间 QQ FlashGet 迅雷
网页制作: FrontPages Dreamweaver Javascript css photoshop fireworks Flash