Using concurrent to write a load balancer

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Using concurrent to write a load balancer

syg6
Hello all.

I am trying to write a load balancer using the new Java 1.5 concurrent classes. I am currently trudging through all of the examples, trying to see what I might use. I wanted to bounce my idea off the group.

I will have a more-or-less fixed pool of a small number of threads (it might grow or shrink every now an then but not often). Each thread will have the same Task associated with it, a task that sends out a 'ping' (using java.net's isReachable, which is another can of worms, but anyway ...)

So every time a request comes in I'd like to use a CyclicBarrier (I think) to launch all of the threads and when the first thread answers (with the ip of the machine pinged), cancel all of the other Tasks, reset and wait for the next request.

I *think* the best way to do this is with a CyclicBarrier, because I need a reusable way to launch all of my threads/tasks at once. CountDownLatch isn't reusable and it I were to use a ExecutorCompletionService I'd have to call the submit() method for each Task, one after the other, which means the first Task would probably almost always be the first to respond.

The only possible problem I see with CyclicBarrier is, what happens when, say, a request comes in, I launch my 5 (for example) threads, one responds but before the other 4 can respond I get another request? Then I'll have to wait until they all respond to re-launch. Ideally I'd like to re-launch immediately with the 1 thread I have available, but I don't know if this is possible ...

Does this sound doable? Anyone have any code that resembles this?

Many thanks,
Bob

ps. I posted on this last week on the java.sun.com forums, but decided to move over to Nabble ... Here's the link:

Reply | Threaded
Open this post in threaded view
|

Re: Using concurrent to write a load balancer

Kasper Nielsen-2
syg6 wrote:

> Hello all.
>
> I am trying to write a load balancer using the new Java 1.5 concurrent
> classes. I am currently trudging through all of the examples, trying to see
> what I might use. I wanted to bounce my idea off the group.
>
> I will have a more-or-less fixed pool of a small number of threads (it might
> grow or shrink every now an then but not often). Each thread will have the
> same Task associated with it, a task that sends out a 'ping' (using
> java.net's isReachable, which is another can of worms, but anyway ...)
>
> So every time a request comes in I'd like to use a CyclicBarrier (I think)
> to launch all of the threads and when the first thread answers (with the ip
> of the machine pinged), cancel all of the other Tasks, reset and wait for
> the next request.
>
> I *think* the best way to do this is with a CyclicBarrier, because I need a
> reusable way to launch all of my threads/tasks at once. CountDownLatch isn't
> reusable and it I were to use a ExecutorCompletionService I'd have to call
> the submit() method for each Task, one after the other, which means the
> first Task would probably almost always be the first to respond.
>
Hi Bob,

I think you might want to have a look at
the invokeAny method of ExecutorService
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ExecutorService.html#invokeAny(java.util.Collection)

Should be much simpler then using a CyclicBarrier.

- Kasper
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Using concurrent to write a load balancer

David Holmes-3
In reply to this post by syg6
Hello again Bob :)

Welcome to concurrency-interest.

> The only possible problem I see with CyclicBarrier is, what happens when,
> say, a request comes in, I launch my 5 (for example) threads, one responds
> but before the other 4 can respond I get another request? Then
> I'll have to wait until they all respond to re-launch. Ideally I'd like to
re-launch
> immediately with the 1 thread I have available, but I don't know
> if this is possible ...

This seems to be the crux of your problem - you can't interrupt the PING so
if requests come in faster than your ping timeout then your threads are
going to be delayed. In that sense a CountDownLatch is more suitable than a
barrier - despite having to keep recreating it - because you want to be able
to "open" it immediately. But even then having your threads tied up on a
previous ping isn't going to help you with the next one.

Presuming your machines reply to this "ping" based on actual load then you
might be able to use the order of responses to maintain a list of "next host
to use", and you only need issue an actual ping when the list is empty
again. So the basic operation would be like this:
    // "ping" task
    while (!stopped) {
        wait-for-request
        do_ping();
        queue.add(host id);
    }

    // request processing thread
    while (!stopped) {
        Request req = getNextRequesr();
        HostID host;
        if (queue.isEmpty())
           signal-request-waiting
        host = queue.take();
        process(req, host);
    }

Hmmm looking at it now I think perhaps CyclicBarrier will work - assuming
the model I outlined suites what you want to do.

Cheers,
David Holmes

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Using concurrent to write a load balancer

syg6
In reply to this post by Kasper Nielsen-2
Hi Kasper, thanks for the response.

Yea, I was meaning to look at invokeAny(). My only problem with ExecutiveService is, I still need a mechanism that allows me to launch every thread / task in my pool at the same time, instead of looping through each thread, one at a time, and calling its start() method, and then having it run.

I thought using CyclicBarrier would be the answer but I'm having issues with that as well -- A program that uses CyclicBarrier does the following:

1. Create the barrier, say with 5 threads.
2. Loops through all of the threads calling each thread's start() method.
3. When all of the threads are started the barrier is 'breached' and they all execute.

Fine. But what I want to do is start all the threads and have the barrier wait for some *other* event, in my case, an incoming request. And upon receiving it, launch all of the threads, grab the result from the 1st one, cancel the rest, and go back to waiting.

It would seem I can't do this with CyclicBarrier, because it doesn't wait for external events but rather other threads to be started. One they are all started it triggers. It would seem a CountDownLatch would make more sense but they are not reusable!

In summary, I'd love to use the ExecutiveService.invokeAny() but I still need a way to continually (with each request) launch all of my threads at the same time.

Can I do this with ExecutiveService?

Thanks again,
Bob


----- Original Message ----
From: Kasper Nielsen <[hidden email]>
To: syg6 <[hidden email]>
Cc: [hidden email]
Sent: Monday, September 18, 2006 1:23:44 PM
Subject: Re: [concurrency-interest] Using concurrent to write a load balancer

syg6 wrote:

> Hello all.
>
> I am trying to write a load balancer using the new Java 1.5 concurrent
> classes. I am currently trudging through all of the examples, trying to see
> what I might use. I wanted to bounce my idea off the group.
>
> I will have a more-or-less fixed pool of a small number of threads (it might
> grow or shrink every now an then but not often). Each thread will have the
> same Task associated with it, a task that sends out a 'ping' (using
> java.net's isReachable, which is another can of worms, but anyway ...)
>
> So every time a request comes in I'd like to use a CyclicBarrier (I think)
> to launch all of the threads and when the first thread answers (with the ip
> of the machine pinged), cancel all of the other Tasks, reset and wait for
> the next request.
>
> I *think* the best way to do this is with a CyclicBarrier, because I need a
> reusable way to launch all of my threads/tasks at once. CountDownLatch isn't
> reusable and it I were to use a ExecutorCompletionService I'd have to call
> the submit() method for each Task, one after the other, which means the
> first Task would probably almost always be the first to respond.
>
Hi Bob,

I think you might want to have a look at
the invokeAny method of ExecutorService
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ExecutorService.html#invokeAny(java.util.Collection)

Should be much simpler then using a CyclicBarrier.

- Kasper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Using concurrent to write a load balancer

Stefan Skoglund
In reply to this post by syg6
[concurrency-interest] Using concurrent to write a load balancer
Hello Bob
 
CyclicBarrier has all or nothing guarantees. The idea is to let exactly, in this case 5, threads fall through. It will never let more or less than 5 threads fall through at once. This is a tricky thing to implement, and the concurrent guys has done it for you. Still, it is a rather tricky to use.
It seems that this isn't the thing you are looking for though. It sounds like you need a powerful thread pool. Take a look at the ExecutorService, also in concurrent.
 
/Stefan


Från: [hidden email] genom syg6
Skickat: må 2006-09-18 12:40
Till: [hidden email]
Ämne: [concurrency-interest] Using concurrent to write a load balancer


Hello all.

I am trying to write a load balancer using the new Java 1.5 concurrent
classes. I am currently trudging through all of the examples, trying to see
what I might use. I wanted to bounce my idea off the group.

I will have a more-or-less fixed pool of a small number of threads (it might
grow or shrink every now an then but not often). Each thread will have the
same Task associated with it, a task that sends out a 'ping' (using
java.net's isReachable, which is another can of worms, but anyway ...)

So every time a request comes in I'd like to use a CyclicBarrier (I think)
to launch all of the threads and when the first thread answers (with the ip
of the machine pinged), cancel all of the other Tasks, reset and wait for
the next request.

I *think* the best way to do this is with a CyclicBarrier, because I need a
reusable way to launch all of my threads/tasks at once. CountDownLatch isn't
reusable and it I were to use a ExecutorCompletionService I'd have to call
the submit() method for each Task, one after the other, which means the
first Task would probably almost always be the first to respond.

The only possible problem I see with CyclicBarrier is, what happens when,
say, a request comes in, I launch my 5 (for example) threads, one responds
but before the other 4 can respond I get another request? Then I'll have to
wait until they all respond to re-launch. Ideally I'd like to re-launch
immediately with the 1 thread I have available, but I don't know if this is
possible ...

Does this sound doable? Anyone have any code that resembles this?

Many thanks,
Bob

ps. I posted on this last week on the java.sun.com forums, but decided to
move over to Nabble ... Here's the
http://forum.java.sun.com/thread.jspa?threadID=768022&tstart=0 link :


--
View this message in context: http://www.nabble.com/Using-concurrent-to-write-a-load-balancer-tf2290433.html#a6361357
Sent from the JSR166 Concurrency mailing list archive at Nabble.com.

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Using concurrent to write a load balancer

Kasper Nielsen-2
In reply to this post by syg6
Robert Bowen wrote:
> Hi Kasper, thanks for the response.
>
> Yea, I was meaning to look at invokeAny(). My only problem with ExecutiveService is, I still need a mechanism that allows me to launch every thread / task in my pool at the same time, instead of looping through each thread, one at a time, and calling its start() method, and then having it run.

If you use ThreadPoolExecutor you can just prestart all the core threads
and set the keep alive time to a high value.


-Kasper
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Using concurrent to write a load balancer

syg6
In reply to this post by Kasper Nielsen-2
Well I have something up and running using ExecutorService. Basically it's like this:

class Ping implements Callable<String> {

    public String call() throws Exception {
       InetAddress iNetAddress = InetAddress.getByName(address);
       if (iNetAddress.isReachable(3000)) return address;
       else return null;
    }
}

public class ThreadPoolExecutorTest {

    public static void main(String[] args) {
        ThreadPoolExecutor tpe = new ThreadPoolExecutor(...);
        tpe.prestartAllCoreThreads();

        ArrayList<Callable> tasks = new ArrayList<Callable>();
        Ping ping = new Ping("192.168.0.27");
        tasks.add(ping);
        ping = new Ping("192.168.0.28");
        tasks.add(ping);

        while (true) {
            String result = (String)tpe.invokeAny(tasks);
            Thread.sleep(1000);
        }
    }
}

It's just a demo but it seems to work very well, many thanks for all of the tips! Working with threads is confusing to start with in Java and with this new API it's *much* easier, but even so there's a bunch of different classes that have similar functionality, it's a question of navigating through all the docs and (of course) speaking to people more clever than yourself on the forums.

My only other question is how to cancel the other tasks that finish after the first one, so they are available again for the next request? I know I can cancel tasks using Future.cancel() but if I use a FutureTask instead of a Callable, then I can't use invokeAny() ...

Anyway, I'm further along than I was with your help!

Bob


----- Original Message ----
From: Kasper Nielsen <[hidden email]>
To: syg6 <[hidden email]>
Cc: [hidden email]
Sent: Monday, September 18, 2006 1:23:44 PM
Subject: Re: [concurrency-interest] Using concurrent to write a load balancer

syg6 wrote:

> Hello all.
>
> I am trying to write a load balancer using the new Java 1.5 concurrent
> classes. I am currently trudging through all of the examples, trying to see
> what I might use. I wanted to bounce my idea off the group.
>
> I will have a more-or-less fixed pool of a small number of threads (it might
> grow or shrink every now an then but not often). Each thread will have the
> same Task associated with it, a task that sends out a 'ping' (using
> java.net's isReachable, which is another can of worms, but anyway ...)
>
> So every time a request comes in I'd like to use a CyclicBarrier (I think)
> to launch all of the threads and when the first thread answers (with the ip
> of the machine pinged), cancel all of the other Tasks, reset and wait for
> the next request.
>
> I *think* the best way to do this is with a CyclicBarrier, because I need a
> reusable way to launch all of my threads/tasks at once. CountDownLatch isn't
> reusable and it I were to use a ExecutorCompletionService I'd have to call
> the submit() method for each Task, one after the other, which means the
> first Task would probably almost always be the first to respond.
>
Hi Bob,

I think you might want to have a look at
the invokeAny method of ExecutorService
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ExecutorService.html#invokeAny(java.util.Collection)

Should be much simpler then using a CyclicBarrier.

- Kasper
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest




_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Using concurrent to write a load balancer

Kasper Nielsen-2
Robert Bowen wrote:
> My only other question is how to cancel the other tasks that finish after the first one, so they are available again for the next request? I know I can cancel tasks using Future.cancel() but if I use a FutureTask instead of a Callable, then I can't use invokeAny() ...
>
invokeAny automatically interrupts all threads that have not finished
before returning the result. However, for some reason isReachable() does
not respond to interrupt signals. So not much you can do about that.

If it turns out to be a problem you would probably need to add some more
threads to the pool (don't worry to much about the overhead) or use
java.lang.Runtime to execute an external ping application

You might also want to checkout this previous thread
http://altair.cs.oswego.edu/pipermail/concurrency-interest/2006-May/002574.html

- Kasper
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Using concurrent to write a load balancer

David Holmes-3
In reply to this post by syg6
Bob,
 
This approach might be viable for you but it doesn't deal with two things you initially indicated were an issue:
 
a) there will be a skew toward the first Ping always being the first to respond because of the time taken to add all the Ping tasks to the executor - especially in light of how invokeAny works. Nowhere near as bad as starting new threads but still might be an issue. Of course there is no instantaneous release mechanism - even using a CyclicBarrier requires the released threads to serialize through the internal lock, one at a time.
 
b) No cancellation. But as I've said before the ping isn't cancellable - so you really have no choice here but to live with that and just use more threads.
 
Cheers,
David
-----Original Message-----
From: [hidden email] [mailto:[hidden email]]On Behalf Of Robert Bowen
Sent: Tuesday, 19 September 2006 1:56 AM
To: Kasper Nielsen
Cc: [hidden email]
Subject: Re: [concurrency-interest] Using concurrent to write a load balancer

Well I have something up and running using ExecutorService. Basically it's like this:

class Ping implements Callable<String> {

    public String call() throws Exception {
       InetAddress iNetAddress = InetAddress.getByName(address);
       if (iNetAddress.isReachable(3000)) return address;
       else return null;
    }
}

public class ThreadPoolExecutorTest {

    public static void main(String[] args) {
        ThreadPoolExecutor tpe = new ThreadPoolExecutor(...);
        tpe.prestartAllCoreThreads();

        ArrayList<Callable> tasks = new ArrayList<Callable>();
        Ping ping = new Ping("192.168.0.27");
        tasks.add(ping);
        ping = new Ping("192.168.0.28");
        tasks.add(ping);

        while (true) {
            String result = (String)tpe.invokeAny(tasks);
            Thread.sleep(1000);
        }
    }
}

It's just a demo but it seems to work very well, many thanks for all of the tips! Working with threads is confusing to start with in Java and with this new API it's *much* easier, but even so there's a bunch of different classes that have similar functionality, it's a question of navigating through all the docs and (of course) speaking to people more clever than yourself on the forums.

My only other question is how to cancel the other tasks that finish after the first one, so they are available again for the next request? I know I can cancel tasks using Future.cancel() but if I use a FutureTask instead of a Callable, then I can't use invokeAny() ...

Anyway, I'm further along than I was with your help!

Bob


----- Original Message ----
From: Kasper Nielsen <[hidden email]>
To: syg6 <[hidden email]>
Cc: [hidden email]
Sent: Monday, September 18, 2006 1:23:44 PM
Subject: Re: [concurrency-interest] Using concurrent to write a load balancer

syg6 wrote:

> Hello all.
>
> I am trying to write a load balancer using the new Java 1.5 concurrent
> classes. I am currently trudging through all of the examples, trying to see
> what I might use. I wanted to bounce my idea off the group.
>
> I will have a more-or-less fixed pool of a small number of threads (it might
> grow or shrink every now an then but not often). Each thread will have the
> same Task associated with it, a task that sends out a 'ping' (using
> java.net's isReachable, which is another can of worms, but anyway ...)
>
> So every time a request comes in I'd like to use a CyclicBarrier (I think)
> to launch all of the threads and when the first thread answers (with the ip
> of the machine pinged), cancel all of the other Tasks, reset and wait for
> the next request.
>
> I *think* the best way to do this is with a CyclicBarrier, because I need a
> reusable way to launch all of my threads/tasks at once. CountDownLatch isn't
> reusable and it I were to use a ExecutorCompletionService I'd have to call
> the submit() method for each Task, one after the other, which means the
> first Task would probably almost always be the first to respond.
>
Hi Bob,

I think you might want to have a look at
the invokeAny method of ExecutorService
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ExecutorService.html#invokeAny(java.util.Collection)

Should be much simpler then using a CyclicBarrier.

- Kasper
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest




_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest