用HTML5构建一个流程图绘制工具
使用Python进行并发编程

Python 并行分布式框架之 PP

posted @ 2015年3月20日 05:44 in 软件开发 with tags Python Distributed , 2467 阅读
分享到: 更多

PP (Parallel Python)是基于Python的一个轻量级的,提供在SMP(多处理器或者多核系统)或者集群环境中并行执行Python代码的机制。

最简单和最常见的并行方式是使用多线程,然而如果应用程序使用Python提供的线程库, 它实际上并不能并行的运行Python的字节码(Byte-Code)。这是因为Pyton解释器使用GIL(全局解释器锁),这样的机制是的在同一时间,即使是多核的机器,也只能运行一个字节码指令。PP试图克服这样的限制,提供一种更简单的方式来编写并行应用。PP采用了多进程和进程间通信来处理并发,并隐藏所有的实现细节,使得其容易使用。

功能介绍

  • 并行运行Python代码(废话)

  • 易于理解的基于任务(Job)的并行技术

  • 自动检测优化配置

  • 动态处理器分配

  • 负载均衡

  • 容错

  • 自动发现和动态分配计算资源

  • 基于SHA的网络连接认证

  • 跨平台(Windows,Linux, Unix, Mac OS)和架构(X86,X86-64)支持

  • 开源 (BSD)

安装

wget  
tar -xvf pp-1.6.4.tar.gz
cd pp-1.6.4
sudo python setup.py install

架构和设计

Server

PP server 包含并管理多个worker并行的执行客户端发送的任务

Client

客户端负责发送任务(Python Function)到服务器

Cluster

多个PP Server可以构成一个PP Cluster,在CLuster模式客户端提交任务到Cluster,cluster 找到合适的Server来运行任务。

使用方式

SMP

在SMP的模式下使用PP非常简单

import pp
# create a job serer
job_server = pp.Server()

# submit some jobs with python functions
f1 = job_server.submit(func1, args1, depfuncs1, modules1)
f2 = job_server.submit(func1, args2, depfuncs1, modules1)
f3 = job_server.submit(func2, args3, depfuncs2, modules2)

# Get result from each jobs
r1 = f1()
r2 = f2()
r3 = f3()

Cluster

在Cluster模式下,需要在不同的节点运行ppserver

node-1> ./ppserver.py
node-2> ./ppserver.py
node-3> ./ppserver.py

客户端代码和SMP模式类似

import pp

# create cluster
ppservers=("node-1", "node-2", "node-3")

# create a job serer
pp.Server(ppservers=ppservers) 

# submit some jobs with python functions
f1 = job_server.submit(func1, args1, depfuncs1, modules1)
f2 = job_server.submit(func1, args2, depfuncs1, modules1)
f3 = job_server.submit(func2, args3, depfuncs2, modules2)

# Get result from each jobs
r1 = f1()
r2 = f2()
r3 = f3()


总结

PP是一个轻量级的并行框架,代码不多,安装使用起来也比较简单,并行方式是多进程,但是缺乏对任务的封装,也缺少调度的功能。适合于构造简单的并行分布式系统。


分享到: 更多
Avatar_small
deep cleaning servic 说:
2019年10月12日 01:08

Currently best maintaining services on abu dhabi, now we have professional maintaining staff individuals company, we normally give alot more importance and even preference to customers, now we have thousands in happy people in abu dhabi, we always think of yourself as best maintaining services small business in abu dhabi and right across UAE.

Avatar_small
part time maids in d 说:
2021年9月27日 21:15

You ought to avoid costs your services over a per hour or so basis, as this might just become pricing the labor : and achieving this will not necessarily cover the skills and also efficiency as soon as you reach an increased level. A tiny research about existing opposition can produce the costs information you should let an individual set the proper price to your cleaning rates. If you would like to work professionally, you need to put an amount on the labor and also practice simply by cleaning your house to determine enough time required to execute certain washing jobs.

Avatar_small
Junior Dakil Result 说:
2022年9月02日 03:18

Dinajpur also one of the best education in the country and the Dinajpur Division also successfully completed the Grade 8 terminal examination tests along with other education boards or divisions of the country, and there are a huge number of students are participated from Dinajpur Board also, Junior Dakil Result Dinajpur Board and all are waiting to check their JSC Result 2022 with full or total marksheet. Both of Junior School Certificate & Junior Dakhil Certificate students are waiting to get official result date to check their total GPA Grade point with subject wise marksheet, the Dinajpur Division also completed those STD-8 final exams in the month of November as per date sheet issued by Bangladesh Secondary and Higher Secondary Education Board and the result is also announced as per the schedule.

Avatar_small
Alyssa 说:
2023年1月06日 03:44

Python Parallel (PP) is a parallel and distributed framework for writing Python programs. It implements message passing and functional programming in a uniform way that is easy to real estate Santa Paula use and efficient. PP is built on top of the MPI standard, and thus can be used with any MPI implementation.

Avatar_small
cleaning company dub 说:
2023年8月27日 21:54

Acquire references from your maid program. Call each and every reference and have if they've got had virtually any complaints of course, if they advise the program. Inquire concerning if they are still while using the service. Or even, ask exactly why. Look into perhaps the company has any type of guarantee with respect to quality. Do the particular maids have got supervisors of course, if so exactly what are they in charge of? Request information from your company on if they provide virtually any assurances, and also what their particular current issue process will be.


登录 *


loading captcha image...
(输入验证码)
or Ctrl+Enter