[NTLUG:Discuss] Distributed processing

[cbbrowne@hex.net]

On Wed, 18 Jul 2001 21:34:28 CDT, the world broke into rejoicing as
Greg Edwards <greg at nas-inet.com>  said:
> Each host would be assigned a set of functions that it would support and
> not a set of applications.  This does not fit the load balance or
> clustering model very well.  Some hosts could act as database servers,
> some in a Beowulf cluster, some as game tree processors, some as search
> engines (these are just examples).  Not every host would support every
> request that could be generated.  This makes for a poor fit for a strict
> message queue solution as you don't want to send the queue to hosts that
> are non players in the logic thread.

I would object _slightly_ on the message queue comment; it would be quite
reasonable to have a set of queues named suitably for better sharing...
I'm not sure what your exact objection is here, but would speculate it's
of two things:

a) Perhaps you're indicating that some machines shouldn't participate
at all with the "logic processing" part of the process.

That should be simply resolvable by only having the "logic processing"
work processes on the machines that _are_ to be players in the logic
thread.

Suppose the work queue is named "Web-Page-Rendering."

There will be a set of machines that are suited for doing the work;
they'll each have a work process that watches that work queue, pulling
items off the "Web-Page-Rendering" queue as they get done with a
previous request.

The other hosts that aren't suitable won't have a work process pulling
items off that queue.

b) Perhaps you're indicating that once the work starts on one machine,
it should be passed on to further work processes on the same machine.
(I don't think so, though.)

This could be accomplished by having further work queues local to the
host.  There might be "WPR-Phase 1 - Host 1", "WPR-Phase 2 - Host 1",
"WPR-Phase 3 - Host 1", on the first host, and then a similar set of
queues on other hosts.

Work gets forwarded to local queues for what needs to be local, and
can go back to the global queues as that work gets completed.

> To expand (scale) the farm would be a simple matter of adding
> another system to the network and activating the desired functions
> on that host.  As a particular application sees heavier traffic more
> hosts that support functions needed by that application can take
> more of the load while applications that are currently supporting
> fewer users can reduce the number of active hosts.  Dropping a host
> out of the farm (for maint) would simply be a matter of blocking new
> requests until all active requests have completed and then shutting
> down the server.

Sounds fairly suited for the message queueing approach, albeit not as
an automatic result of the messaging scheme, but rather as an aspect
of the design of the work process.
--
(reverse (concatenate 'string "gro.mca@" "enworbbc"))
http://vip.hyperusa.com/~cbbrowne/sap.html
"The Linux  philosophy is laugh in  the face of  danger.  Oops.  Wrong
One.  'Do it yourself.'  That's it."  -- Linus Torvalds