A thread pool for servers?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

A thread pool for servers?

Martin Buchholz-3

Colleague Douglas Dickinson at Google pointed out that none of the convenience executors in Executors is truly resource bounded.

I've never run a "production Java server", but trying to bound resource usage so that your server doesn't die with OOME or gc thrashing seems important.  So your thread pool needs to be bounded in the number of threads *and* in the size of the queue, and be prepared to handle task rejection sensibly.

So it makes sense to me to add another convenience method to Executors that will act like newFixedThreadPool but will additionally have a bounded queue.  It's not completely obvious whether ArrayBlockingQueue or LinkedBlockingQueue is the better choice for the bounded queue, but I suspect for "serious servers" it's the former, because:
- modern server environments tend to like pre-allocating all their resources at startup
- there's little GC overhead to the "dead weight" of the ArrayBlockingQueue's backing array (a single object!)
- LinkedBlockingQueue will allocate many more small objects, and servers don't like that
- with today's economics, "memory is cheap"

(and of course, we can try to write an actual better bounded thread pool not based on ThreadPoolExecutor, but that's actually hard)

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: A thread pool for servers?

David Holmes-6

Hi Martin,

 

All those requirements suggest direct use of a thread pool constructor not a “convenience” method. You get to set the size of the pool, the type and size of queue and the rejection policy.

 

David

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Martin Buchholz
Sent: Saturday, September 12, 2015 7:51 AM
To: concurrency-interest; Douglas Dickinson
Subject: [concurrency-interest] A thread pool for servers?

 

 

Colleague Douglas Dickinson at Google pointed out that none of the convenience executors in Executors is truly resource bounded.

 

I've never run a "production Java server", but trying to bound resource usage so that your server doesn't die with OOME or gc thrashing seems important.  So your thread pool needs to be bounded in the number of threads *and* in the size of the queue, and be prepared to handle task rejection sensibly.

 

So it makes sense to me to add another convenience method to Executors that will act like newFixedThreadPool but will additionally have a bounded queue.  It's not completely obvious whether ArrayBlockingQueue or LinkedBlockingQueue is the better choice for the bounded queue, but I suspect for "serious servers" it's the former, because:

- modern server environments tend to like pre-allocating all their resources at startup

- there's little GC overhead to the "dead weight" of the ArrayBlockingQueue's backing array (a single object!)

- LinkedBlockingQueue will allocate many more small objects, and servers don't like that

- with today's economics, "memory is cheap"

 

(and of course, we can try to write an actual better bounded thread pool not based on ThreadPoolExecutor, but that's actually hard)


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: A thread pool for servers?

Mike Axiak
At HubSpot, we've found the standard Executors helper methods to be a somewhat onerous API. We provide a bounded thread pool as part of a fluent builder that makes it clearer what you're wanting.

Rather than adding a new "newFixedThreadPool" or having people create a "new ThreadPoolExecutor" and possibly subsequently calling "allowCoreThreadTimeOut", would a JEP for a builder be reasonable?

Best,
Mike

On Fri, Sep 11, 2015 at 7:46 PM, David Holmes <[hidden email]> wrote:

Hi Martin,

 

All those requirements suggest direct use of a thread pool constructor not a “convenience” method. You get to set the size of the pool, the type and size of queue and the rejection policy.

 

David

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Martin Buchholz
Sent: Saturday, September 12, 2015 7:51 AM
To: concurrency-interest; Douglas Dickinson
Subject: [concurrency-interest] A thread pool for servers?

 

 

Colleague Douglas Dickinson at Google pointed out that none of the convenience executors in Executors is truly resource bounded.

 

I've never run a "production Java server", but trying to bound resource usage so that your server doesn't die with OOME or gc thrashing seems important.  So your thread pool needs to be bounded in the number of threads *and* in the size of the queue, and be prepared to handle task rejection sensibly.

 

So it makes sense to me to add another convenience method to Executors that will act like newFixedThreadPool but will additionally have a bounded queue.  It's not completely obvious whether ArrayBlockingQueue or LinkedBlockingQueue is the better choice for the bounded queue, but I suspect for "serious servers" it's the former, because:

- modern server environments tend to like pre-allocating all their resources at startup

- there's little GC overhead to the "dead weight" of the ArrayBlockingQueue's backing array (a single object!)

- LinkedBlockingQueue will allocate many more small objects, and servers don't like that

- with today's economics, "memory is cheap"

 

(and of course, we can try to write an actual better bounded thread pool not based on ThreadPoolExecutor, but that's actually hard)


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest



_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: A thread pool for servers?

Luke Sandberg
I have found the "be prepared to handle task rejection sensibly" advice to be significantly easier said than done.  For example, few of Guava's ListenableFuture utilities are 'hardened' against RejectedExecutionException,  It was only about a year ago that Futures.transform would give you a 'dead' future if these executor threw an REE.  I have also seen a number of production deadlocks/stuck threads due to improper REE handling.  At least in the applications I work on, the number of places that submit tasks to thread pools is large so ensuring consistent reasonable handling is very hard.

The strategy my teams has taken in some servers is to use fixed size thread pools (ThreadPoolExecutor, though we are experimenting with FJP) with unbounded queues and higher level throttling support.  In theory errant tasks could still cause OOMEs by producing tasks too fast, but in practice I have never observed this, while I have been involved in debugging production issues due to REE.  

On Fri, Sep 11, 2015 at 5:15 PM Mike Axiak <[hidden email]> wrote:
At HubSpot, we've found the standard Executors helper methods to be a somewhat onerous API. We provide a bounded thread pool as part of a fluent builder that makes it clearer what you're wanting.

Rather than adding a new "newFixedThreadPool" or having people create a "new ThreadPoolExecutor" and possibly subsequently calling "allowCoreThreadTimeOut", would a JEP for a builder be reasonable?

Best,
Mike


On Fri, Sep 11, 2015 at 7:46 PM, David Holmes <[hidden email]> wrote:

Hi Martin,

 

All those requirements suggest direct use of a thread pool constructor not a “convenience” method. You get to set the size of the pool, the type and size of queue and the rejection policy.

 

David

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Martin Buchholz
Sent: Saturday, September 12, 2015 7:51 AM
To: concurrency-interest; Douglas Dickinson
Subject: [concurrency-interest] A thread pool for servers?

 

 

Colleague Douglas Dickinson at Google pointed out that none of the convenience executors in Executors is truly resource bounded.

 

I've never run a "production Java server", but trying to bound resource usage so that your server doesn't die with OOME or gc thrashing seems important.  So your thread pool needs to be bounded in the number of threads *and* in the size of the queue, and be prepared to handle task rejection sensibly.

 

So it makes sense to me to add another convenience method to Executors that will act like newFixedThreadPool but will additionally have a bounded queue.  It's not completely obvious whether ArrayBlockingQueue or LinkedBlockingQueue is the better choice for the bounded queue, but I suspect for "serious servers" it's the former, because:

- modern server environments tend to like pre-allocating all their resources at startup

- there's little GC overhead to the "dead weight" of the ArrayBlockingQueue's backing array (a single object!)

- LinkedBlockingQueue will allocate many more small objects, and servers don't like that

- with today's economics, "memory is cheap"

 

(and of course, we can try to write an actual better bounded thread pool not based on ThreadPoolExecutor, but that's actually hard)


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: A thread pool for servers?

Henri Tremblay
I indeed have encountered this problem frequently.When trying to do basic producer -> consumers system without having to think too much about it.

I was really surprised that nothing allowing the producer to stop producing exists. So I usually end-up doing something like this:

BlockingQueue<Runnable> queue = new BlockingQueue<>(10_000);
ThreadPoolExecutor pool = new ThreadPoolExecutor(
        parallelism, parallelism,
        0L, TimeUnit.MILLISECONDS,
        queue);
pool.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
It does the job but I would love to have something readily available in Executors.

On 11 September 2015 at 21:33, Luke Sandberg <[hidden email]> wrote:
I have found the "be prepared to handle task rejection sensibly" advice to be significantly easier said than done.  For example, few of Guava's ListenableFuture utilities are 'hardened' against RejectedExecutionException,  It was only about a year ago that Futures.transform would give you a 'dead' future if these executor threw an REE.  I have also seen a number of production deadlocks/stuck threads due to improper REE handling.  At least in the applications I work on, the number of places that submit tasks to thread pools is large so ensuring consistent reasonable handling is very hard.

The strategy my teams has taken in some servers is to use fixed size thread pools (ThreadPoolExecutor, though we are experimenting with FJP) with unbounded queues and higher level throttling support.  In theory errant tasks could still cause OOMEs by producing tasks too fast, but in practice I have never observed this, while I have been involved in debugging production issues due to REE.  

On Fri, Sep 11, 2015 at 5:15 PM Mike Axiak <[hidden email]> wrote:
At HubSpot, we've found the standard Executors helper methods to be a somewhat onerous API. We provide a bounded thread pool as part of a fluent builder that makes it clearer what you're wanting.

Rather than adding a new "newFixedThreadPool" or having people create a "new ThreadPoolExecutor" and possibly subsequently calling "allowCoreThreadTimeOut", would a JEP for a builder be reasonable?

Best,
Mike


On Fri, Sep 11, 2015 at 7:46 PM, David Holmes <[hidden email]> wrote:

Hi Martin,

 

All those requirements suggest direct use of a thread pool constructor not a “convenience” method. You get to set the size of the pool, the type and size of queue and the rejection policy.

 

David

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Martin Buchholz
Sent: Saturday, September 12, 2015 7:51 AM
To: concurrency-interest; Douglas Dickinson
Subject: [concurrency-interest] A thread pool for servers?

 

 

Colleague Douglas Dickinson at Google pointed out that none of the convenience executors in Executors is truly resource bounded.

 

I've never run a "production Java server", but trying to bound resource usage so that your server doesn't die with OOME or gc thrashing seems important.  So your thread pool needs to be bounded in the number of threads *and* in the size of the queue, and be prepared to handle task rejection sensibly.

 

So it makes sense to me to add another convenience method to Executors that will act like newFixedThreadPool but will additionally have a bounded queue.  It's not completely obvious whether ArrayBlockingQueue or LinkedBlockingQueue is the better choice for the bounded queue, but I suspect for "serious servers" it's the former, because:

- modern server environments tend to like pre-allocating all their resources at startup

- there's little GC overhead to the "dead weight" of the ArrayBlockingQueue's backing array (a single object!)

- LinkedBlockingQueue will allocate many more small objects, and servers don't like that

- with today's economics, "memory is cheap"

 

(and of course, we can try to write an actual better bounded thread pool not based on ThreadPoolExecutor, but that's actually hard)


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest



_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: A thread pool for servers?

Doug Lea
In reply to this post by Martin Buchholz-3
On 09/11/2015 05:51 PM, Martin Buchholz wrote:
>
> Colleague Douglas Dickinson at Google pointed out that none of the convenience
> executors in Executors is truly resource bounded.

It's great that people are advocating use of the resource-control
features of TPE. Back in pre-j.u.c days when TPE was introduced,
nearly everyone complained about how many arguments were needed in
the constructor, which led to us introducing Executors convenience
methods. But in these cases "convenience" means "we are letting you
get away with default choices that you should probably someday revisit",
and the TPE javadoc includes discussion about how to make better choices.
The need to contemplate bounding, rejection, and flow control
has been increasing as containerized, resource-managed runtime
environments become more common.

But, like David, I'm not sure that we can add more Executors methods
that don't amount to just replicating the TPE constructors.
Maybe a builder, but that seems like overkill given the j.u.c audience.

Some of these issues also affect ForkJoinPool, which we addressed by
adding a configurable spare-thread limit and tolerance for bounded
thread factories. It would also be possible to add a per-thread local
queue-size limit, although the cases where it might come into play
are uncommon and preventable using FJ status methods.

People facing these issues might also consider using the new
SubmissionPublisher class, that pushes resource and flow control
up one level: Producers need to pick buffer sizes, consumers
explicitly perform requests, and you must choose per-item
whether to use block vs drop vs timeout vs handler modes
(methods submit, offer, and timed-offer, with optional drop
handlers). These can be made to work well no matter what kind
Executor you use, because the SubmissionPublisher deals with
most of them instead of the Executor.

> (and of course, we can try to write an actual better bounded thread pool not
> based on ThreadPoolExecutor, but that's actually hard)

The main use case that j.u.c. lacks explicit coverage for
is when there is known to be a single task submitter thread.
FJP's multi-laned queues add some unnecessary data structure overhead
and TPE's insistence on a queue implementation meeting the full
BlockingQueue API adds some unnecessary sync overhead.
In most usages, either choice is fine, but we might consider
addressing this.

-Doug



_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: A thread pool for servers?

Viktor Klang
Doug,

yes, pre prospect of making thread pools *pull* in work rather than get *pushed* work would make for a very interesting side-effect in terms of flow control / back pressure without having to block any threads in the process.

On Sat, Sep 12, 2015 at 2:18 PM, Doug Lea <[hidden email]> wrote:
On 09/11/2015 05:51 PM, Martin Buchholz wrote:

Colleague Douglas Dickinson at Google pointed out that none of the convenience
executors in Executors is truly resource bounded.

It's great that people are advocating use of the resource-control
features of TPE. Back in pre-j.u.c days when TPE was introduced,
nearly everyone complained about how many arguments were needed in
the constructor, which led to us introducing Executors convenience
methods. But in these cases "convenience" means "we are letting you
get away with default choices that you should probably someday revisit",
and the TPE javadoc includes discussion about how to make better choices.
The need to contemplate bounding, rejection, and flow control
has been increasing as containerized, resource-managed runtime
environments become more common.

But, like David, I'm not sure that we can add more Executors methods
that don't amount to just replicating the TPE constructors.
Maybe a builder, but that seems like overkill given the j.u.c audience.

Some of these issues also affect ForkJoinPool, which we addressed by
adding a configurable spare-thread limit and tolerance for bounded
thread factories. It would also be possible to add a per-thread local
queue-size limit, although the cases where it might come into play
are uncommon and preventable using FJ status methods.

People facing these issues might also consider using the new
SubmissionPublisher class, that pushes resource and flow control
up one level: Producers need to pick buffer sizes, consumers
explicitly perform requests, and you must choose per-item
whether to use block vs drop vs timeout vs handler modes
(methods submit, offer, and timed-offer, with optional drop
handlers). These can be made to work well no matter what kind
Executor you use, because the SubmissionPublisher deals with
most of them instead of the Executor.

(and of course, we can try to write an actual better bounded thread pool not
based on ThreadPoolExecutor, but that's actually hard)

The main use case that j.u.c. lacks explicit coverage for
is when there is known to be a single task submitter thread.
FJP's multi-laned queues add some unnecessary data structure overhead
and TPE's insistence on a queue implementation meeting the full
BlockingQueue API adds some unnecessary sync overhead.
In most usages, either choice is fine, but we might consider
addressing this.

-Doug




_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest



--
Cheers,

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: A thread pool for servers?

David M. Lloyd-3
Where would you pull from, if not a queue?

On 09/12/2015 09:06 AM, Viktor Klang wrote:

> Doug,
>
> yes, pre prospect of making thread pools *pull* in work rather than get
> *pushed* work would make for a very interesting side-effect in terms of
> flow control / back pressure without having to block any threads in the
> process.
>
> On Sat, Sep 12, 2015 at 2:18 PM, Doug Lea <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On 09/11/2015 05:51 PM, Martin Buchholz wrote:
>
>
>         Colleague Douglas Dickinson at Google pointed out that none of
>         the convenience
>         executors in Executors is truly resource bounded.
>
>
>     It's great that people are advocating use of the resource-control
>     features of TPE. Back in pre-j.u.c days when TPE was introduced,
>     nearly everyone complained about how many arguments were needed in
>     the constructor, which led to us introducing Executors convenience
>     methods. But in these cases "convenience" means "we are letting you
>     get away with default choices that you should probably someday revisit",
>     and the TPE javadoc includes discussion about how to make better
>     choices.
>     The need to contemplate bounding, rejection, and flow control
>     has been increasing as containerized, resource-managed runtime
>     environments become more common.
>
>     But, like David, I'm not sure that we can add more Executors methods
>     that don't amount to just replicating the TPE constructors.
>     Maybe a builder, but that seems like overkill given the j.u.c audience.
>
>     Some of these issues also affect ForkJoinPool, which we addressed by
>     adding a configurable spare-thread limit and tolerance for bounded
>     thread factories. It would also be possible to add a per-thread local
>     queue-size limit, although the cases where it might come into play
>     are uncommon and preventable using FJ status methods.
>
>     People facing these issues might also consider using the new
>     SubmissionPublisher class, that pushes resource and flow control
>     up one level: Producers need to pick buffer sizes, consumers
>     explicitly perform requests, and you must choose per-item
>     whether to use block vs drop vs timeout vs handler modes
>     (methods submit, offer, and timed-offer, with optional drop
>     handlers). These can be made to work well no matter what kind
>     Executor you use, because the SubmissionPublisher deals with
>     most of them instead of the Executor.
>
>         (and of course, we can try to write an actual better bounded
>         thread pool not
>         based on ThreadPoolExecutor, but that's actually hard)
>
>
>     The main use case that j.u.c. lacks explicit coverage for
>     is when there is known to be a single task submitter thread.
>     FJP's multi-laned queues add some unnecessary data structure overhead
>     and TPE's insistence on a queue implementation meeting the full
>     BlockingQueue API adds some unnecessary sync overhead.
>     In most usages, either choice is fine, but we might consider
>     addressing this.
>
>     -Doug
>
>
>
>
>     _______________________________________________
>     Concurrency-interest mailing list
>     [hidden email]
>     <mailto:[hidden email]>
>     http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>
>
>
>
> --
> Cheers,
> √
>
>
> _______________________________________________
> Concurrency-interest mailing list
> [hidden email]
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>

--
- DML
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: A thread pool for servers?

Viktor Klang
A Publisher of Runnable?

On Mon, Sep 14, 2015 at 2:17 PM, David M. Lloyd <[hidden email]> wrote:
Where would you pull from, if not a queue?

On 09/12/2015 09:06 AM, Viktor Klang wrote:
Doug,

yes, pre prospect of making thread pools *pull* in work rather than get
*pushed* work would make for a very interesting side-effect in terms of
flow control / back pressure without having to block any threads in the
process.

On Sat, Sep 12, 2015 at 2:18 PM, Doug Lea <[hidden email]
<mailto:[hidden email]>> wrote:

    On 09/11/2015 05:51 PM, Martin Buchholz wrote:


        Colleague Douglas Dickinson at Google pointed out that none of
        the convenience
        executors in Executors is truly resource bounded.


    It's great that people are advocating use of the resource-control
    features of TPE. Back in pre-j.u.c days when TPE was introduced,
    nearly everyone complained about how many arguments were needed in
    the constructor, which led to us introducing Executors convenience
    methods. But in these cases "convenience" means "we are letting you
    get away with default choices that you should probably someday revisit",
    and the TPE javadoc includes discussion about how to make better
    choices.
    The need to contemplate bounding, rejection, and flow control
    has been increasing as containerized, resource-managed runtime
    environments become more common.

    But, like David, I'm not sure that we can add more Executors methods
    that don't amount to just replicating the TPE constructors.
    Maybe a builder, but that seems like overkill given the j.u.c audience.

    Some of these issues also affect ForkJoinPool, which we addressed by
    adding a configurable spare-thread limit and tolerance for bounded
    thread factories. It would also be possible to add a per-thread local
    queue-size limit, although the cases where it might come into play
    are uncommon and preventable using FJ status methods.

    People facing these issues might also consider using the new
    SubmissionPublisher class, that pushes resource and flow control
    up one level: Producers need to pick buffer sizes, consumers
    explicitly perform requests, and you must choose per-item
    whether to use block vs drop vs timeout vs handler modes
    (methods submit, offer, and timed-offer, with optional drop
    handlers). These can be made to work well no matter what kind
    Executor you use, because the SubmissionPublisher deals with
    most of them instead of the Executor.

        (and of course, we can try to write an actual better bounded
        thread pool not
        based on ThreadPoolExecutor, but that's actually hard)


    The main use case that j.u.c. lacks explicit coverage for
    is when there is known to be a single task submitter thread.
    FJP's multi-laned queues add some unnecessary data structure overhead
    and TPE's insistence on a queue implementation meeting the full
    BlockingQueue API adds some unnecessary sync overhead.
    In most usages, either choice is fine, but we might consider
    addressing this.

    -Doug




    _______________________________________________
    Concurrency-interest mailing list
    [hidden email]
    <mailto:[hidden email]>
    http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
Cheers,



_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest


--
- DML

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest



--
Cheers,

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest