The short answer is you want a book or article on 'concurrent
programming'. The main programming abstraction for doing this is
generally 'threads'. Which are supported in different ways in different
environments (languages and OSs). Another way this is sometimes done
especially in the UNIX environment is with "fork and exec" of processes,
at the OS level, rather than 'threads'. Concurrent programming brings
it's own challenges and conventional design patterns/abstractions to
solve them; I'd definitely reccommend looking for some reading on this
topic, but don't have anything in particular to reccommend, I'm afraid.
And indeed, I need to address exactly the problems you mentioned in both
my Umlaut application and in federated search. The solution is indeed
concurrent programming of some kind. In Umlaut, it's a pain since Rails
isn't particularly happy about concurrent programming.
Eric Lease Morgan wrote:
> How do I write a computer program that spawns many processes but
> returns one result?
> I suppose the classic example of my query is the federated search. Get
> user input. Send it to many remote indexes. Wait. Combine results.
> Return. In this scenario when one of the remote indexes is slow things
> grind to a halt.
> I have a more modern example. Suppose I want to take advantage of many
> Web Services. One might be spell checker. Another might be a
> thesaurus. Another might be an index. Another might be a user lookup
> function. Given this environment, where each Web Service will return
> different sets of streams, how do I query each of them simultaneously
> and then aggregate the result? I don't want to so this sequentially. I
> want to fork them all at once and wait for their return before a
> specific time out. In Perl I can use the system command to fork a
> process, but I must wait for it to return. There is another Perl
> command allowing me to fork a process and keep going but I don't
> remember what it is. Neither one of these solutions seem feasible. Is
> the idea of threading in Java suppose to be able to address this
> Eric Lease Morgan
> University Libraries of Notre Dame
> (574) 631-8604
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
rochkind (at) jhu.edu