Exception handling & Executors

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Exception handling & Executors

John Adams-5
Message
Hello,
 
First off, thanks for maintaining this list, it has proven invaluable to me as I learn to use the concurrency utilities in Java 5.0.
 
The reason I'm posting is that I've had a lot of problems using the Executor framework to deal with exceptions from errant tasks.  I've complained about this to Sun, whose response was essentially that exceptions belong with the task, so they should be try / caught at that level. 
 
I don't agree.  I don't, for example, believe that it is the responsibility of the task to handle OutOfMemoryError's, or in fact most of the classes that occur within the Error class of throwables.   Many of those exceptions imply that the tasks should have some knowledge about how to respond to events are within the domain of the platform the tasks are executing on.  Now you could reasonably argue --> What is there to do anyways in that case ?   All I want to do is a System.exit(), since we have a watchdog process that does "the right thing" when the VM goes down.  In theory that would have been easy for me to do with our custom threadpools pre-1.5, however to get it working with the concurrent utilities has been a painful experience. 
 
I thought it may be helpful to note what we tried in what order, so that maybe we could get some support beyond the custom Future task support going in later VM releases.  So, in order ....
 
1. Override the ThreadGroup before creating the Executor.    We did this initially expecting things to be easy, however we were quickly thwarted by the DefaultThreadFactory that does this ...
 
       DefaultThreadFactory() {
            SecurityManager s = System.getSecurityManager();
            group = (s != null)? s.getThreadGroup() :
                                 Thread.currentThread().getThreadGroup();
         }
 
I can see some security reasons for this, however I was not expecting it --> I have stages that I wanted to log & name according to threadgroups.  However, that's not insurmountable ... I created my own threadfactory which just took the currentThread().getThreadGroup() ... no luck.   I also used the setExceptionHandler() method on Thread & ThreadGroup, but to no avail.  I left this code in since I wanted this code to be independent of our security manager choices, and explicitly setting the exception handler seemed like a reasonable thing to do.  (and maybe relevant to ppl on this list) 
 
2.  So then we went to look at the Future and realized what was happening, the future task was catching throwable. (Yes, I know all about custom futures, they don't help me now)  The only way we could rationally seem to do this is to override afterExecute() in the executor and do the following ....
 
try {
                // note I'd rather cast than use temporary objects 
                if ( !((FutureTask) task).isCancelled()){  // I don't care about CancellationExceptions
                    ((FutureTask) task).get();
                }
             } catch (ExecutionException ee) {
                 // ie if (ee == OutOfMemoryError) {System.exit()}
            }
}
 
3. Ok, so now for every task we're checking if it is cancelled, casting 2x, and getting a result we almost never want, all of which have a performance impact.   I tried throwing an OutOfMemoryError .... didn't trip the exit().  More than a little aggravated, I looked at the exception, and realized that ExecutionException wraps Throwable, which means I need to inspect the cause.  So I changed that bit of code to
 
try {
               if ( !((FutureTask) task).isCancelled()){ // I still don't care about CancellationExceptions
                    ((FutureTask) task).get();
                }
             } catch (ExecutionException ee) {
                 //  ie if (ee.getCause() == OutOfMemoryError) {System.exit()}
            }
}
 
4.  The fourth problem I had was a bit subtle, and didn't come up until later.   We have subsequent tasks that queue up on other executors.  This was resulting in a chain of ExecutionExceptions being set up, which meant we had to recursively inspect all exceptions in the chain to make sure there was no out of memory error.   Think SEDA, and you'll be able to picture why we have multiple stages and chained execution exceptions.  It also makes us more vulnerable to StackOverflow errors.
 
Once you pull that all together, you can catch the OutOfMemoryError cleanly, I think.  The moral of this story may be to not use the Executors if you want to be a framework and don't trust your tasks, but I don't think it has to be this hard.  There are several places in the Java, and patterns that I can think of that have a setExceptionHandler() pattern going on, wouldn't it be reasonable to add similar functionality instead of forcing us to go through these hoops to find an Error ?  
 
Again, all of this can be avoided if you trust your tasks to do a try / catch (Throwable t) , but as part of a framework I don't exactly have that luxury.  Thanks again for all the information, I hope this is useful to others on this list.
 
Regards,
John Adams
 

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Exception handling & Executors

Joe Bowbeer
A few ideas:

1. If you execute Runnable instead of FutureTask then any
RuntimeExceptions and Errors thrown can escape the Executor.

2. You can hook the ThreadPoolExecutor.afterExecute method to handle
the RuntimeExceptions thrown by your task.  (Note: the Throwable
signature is a little misleading.)

3. You can use the Thread.set[Default]UncaughtExceptionHandler to
handle uncaught exceptions and errors of all types.  (To associate the
exception handling with your executor, you can provide your own thread
factory.)

4. If you want to use FutureTask then you can hook its done() method
and deal with the exception there.


A comment:

I don't know how you expect to do anything with an
OutOfMemoryException, except perhaps log it.  I don't know of any way
to reliably recover from these.


On 10/25/05, John Adams <[hidden email]> wrote:

>
> Hello,
>
> First off, thanks for maintaining this list, it has proven invaluable to me
> as I learn to use the concurrency utilities in Java 5.0.
>
> The reason I'm posting is that I've had a lot of problems using the Executor
> framework to deal with exceptions from errant tasks.  I've complained about
> this to Sun, whose response was essentially that exceptions belong with the
> task, so they should be try / caught at that level.
>
> I don't agree.  I don't, for example, believe that it is the responsibility
> of the task to handle OutOfMemoryError's, or in fact most of the classes
> that occur within the Error class of throwables.   Many of those exceptions
> imply that the tasks should have some knowledge about how to respond to
> events are within the domain of the platform the tasks are executing on.
> Now you could reasonably argue --> What is there to do anyways in that case
> ?   All I want to do is a System.exit(), since we have a watchdog process
> that does "the right thing" when the VM goes down.  In theory that would
> have been easy for me to do with our custom threadpools pre-1.5, however to
> get it working with the concurrent utilities has been a painful experience.
>
> I thought it may be helpful to note what we tried in what order, so that
> maybe we could get some support beyond the custom Future task support going
> in later VM releases.  So, in order ....
>
> 1. Override the ThreadGroup before creating the Executor.    We did this
> initially expecting things to be easy, however we were quickly thwarted by
> the DefaultThreadFactory that does this ...
>
>        DefaultThreadFactory() {
>             SecurityManager s = System.getSecurityManager();
>             group = (s != null)? s.getThreadGroup() :
>
> Thread.currentThread().getThreadGroup();
>          }
>
> I can see some security reasons for this, however I was not expecting it -->
> I have stages that I wanted to log & name according to threadgroups.
> However, that's not insurmountable ... I created my own threadfactory which
> just took the currentThread().getThreadGroup() ... no luck.
>   I also used the setExceptionHandler() method on Thread & ThreadGroup, but
> to no avail.  I left this code in since I wanted this code to be independent
> of our security manager choices, and explicitly setting the exception
> handler seemed like a reasonable thing to do.  (and maybe relevant to ppl on
> this list)
>
> 2.  So then we went to look at the Future and realized what was happening,
> the future task was catching throwable. (Yes, I know all about custom
> futures, they don't help me now)  The only way we could rationally seem to
> do this is to override afterExecute() in the executor and do the following
> ....
>
> try {
>                 // note I'd rather cast than use temporary objects
>                 if ( !((FutureTask) task).isCancelled()){  // I don't care
> about CancellationExceptions
>                     ((FutureTask) task).get();
>                 }
>              } catch (ExecutionException ee) {
>                  // ie if (ee == OutOfMemoryError) {System.exit()}
>             }
> }
>
> 3. Ok, so now for every task we're checking if it is cancelled, casting 2x,
> and getting a result we almost never want, all of which have a performance
> impact.   I tried throwing an OutOfMemoryError .... didn't trip the exit().
> More than a little aggravated, I looked at the exception, and realized that
> ExecutionException wraps Throwable, which means I need to inspect the cause.
>  So I changed that bit of code to
>
>
> try {
>                if ( !((FutureTask) task).isCancelled()){ // I still don't
> care about CancellationExceptions
>                     ((FutureTask) task).get();
>                 }
>              } catch (ExecutionException ee) {
>                  //  ie if (ee.getCause() == OutOfMemoryError)
> {System.exit()}
>             }
> }
>
> 4.  The fourth problem I had was a bit subtle, and didn't come up until
> later.   We have subsequent tasks that queue up on other executors.  This
> was resulting in a chain of ExecutionExceptions being set up, which meant we
> had to recursively inspect all exceptions in the chain to make sure there
> was no out of memory error.   Think SEDA, and you'll be able to picture why
> we have multiple stages and chained execution exceptions.  It also makes us
> more vulnerable to StackOverflow errors.
>
> Once you pull that all together, you can catch the OutOfMemoryError cleanly,
> I think.  The moral of this story may be to not use the Executors if you
> want to be a framework and don't trust your tasks, but I don't think it has
> to be this hard.  There are several places in the Java, and patterns that I
> can think of that have a setExceptionHandler() pattern going on, wouldn't it
> be reasonable to add similar functionality instead of forcing us to go
> through these hoops to find an Error ?
>
> Again, all of this can be avoided if you trust your tasks to do a try /
> catch (Throwable t) , but as part of a framework I don't exactly have that
> luxury.  Thanks again for all the information, I hope this is useful to
> others on this list.
>
> Regards,
> John Adams
>

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Exception handling & Executors

Jan Nielsen-2
Joe,

With respect to your comment below about not being able to do anything
with an out-of-memory condition, could you not reliably release, for
example, cached data in an OutOfMemoryError handler:

  Map m = new HashMap();
  Object o = new PreAllocatedMemoryInBytes( 16 );
  try
  {
    // blah, blah, blah...allocating all the memory to 'm' values...
  }
  catch( OutOfMemoryError e )
  {
      o = null;
      m.clear();
      // only log after we have enough memory
      e.printStackTrace();
  }

The m.clear() method may require some allocation to work, so it would be
susceptible to failure without having that much available memory,
specifically:

    public void clear() {
        modCount++;
        Entry[] tab = table;
        for (int i = 0; i < tab.length; i++)
            tab[i] = null;
        size = 0;
    }

In the single-threaded case, pre-allocating sufficient memory to perform
this operation should address this, should it not?

Thanks,

-Jan

Joe Bowbeer wrote:

>A few ideas:
>
>1. If you execute Runnable instead of FutureTask then any
>RuntimeExceptions and Errors thrown can escape the Executor.
>
>2. You can hook the ThreadPoolExecutor.afterExecute method to handle
>the RuntimeExceptions thrown by your task.  (Note: the Throwable
>signature is a little misleading.)
>
>3. You can use the Thread.set[Default]UncaughtExceptionHandler to
>handle uncaught exceptions and errors of all types.  (To associate the
>exception handling with your executor, you can provide your own thread
>factory.)
>
>4. If you want to use FutureTask then you can hook its done() method
>and deal with the exception there.
>
>
>A comment:
>
>I don't know how you expect to do anything with an
>OutOfMemoryException, except perhaps log it.  I don't know of any way
>to reliably recover from these.
>
>
>On 10/25/05, John Adams <[hidden email]> wrote:
>  
>
>>Hello,
>>
>>First off, thanks for maintaining this list, it has proven invaluable to me
>>as I learn to use the concurrency utilities in Java 5.0.
>>
>>The reason I'm posting is that I've had a lot of problems using the Executor
>>framework to deal with exceptions from errant tasks.  I've complained about
>>this to Sun, whose response was essentially that exceptions belong with the
>>task, so they should be try / caught at that level.
>>
>>I don't agree.  I don't, for example, believe that it is the responsibility
>>of the task to handle OutOfMemoryError's, or in fact most of the classes
>>that occur within the Error class of throwables.   Many of those exceptions
>>imply that the tasks should have some knowledge about how to respond to
>>events are within the domain of the platform the tasks are executing on.
>>Now you could reasonably argue --> What is there to do anyways in that case
>>?   All I want to do is a System.exit(), since we have a watchdog process
>>that does "the right thing" when the VM goes down.  In theory that would
>>have been easy for me to do with our custom threadpools pre-1.5, however to
>>get it working with the concurrent utilities has been a painful experience.
>>
>>I thought it may be helpful to note what we tried in what order, so that
>>maybe we could get some support beyond the custom Future task support going
>>in later VM releases.  So, in order ....
>>
>>1. Override the ThreadGroup before creating the Executor.    We did this
>>initially expecting things to be easy, however we were quickly thwarted by
>>the DefaultThreadFactory that does this ...
>>
>>       DefaultThreadFactory() {
>>            SecurityManager s = System.getSecurityManager();
>>            group = (s != null)? s.getThreadGroup() :
>>
>>Thread.currentThread().getThreadGroup();
>>         }
>>
>>I can see some security reasons for this, however I was not expecting it -->
>>I have stages that I wanted to log & name according to threadgroups.
>>However, that's not insurmountable ... I created my own threadfactory which
>>just took the currentThread().getThreadGroup() ... no luck.
>>  I also used the setExceptionHandler() method on Thread & ThreadGroup, but
>>to no avail.  I left this code in since I wanted this code to be independent
>>of our security manager choices, and explicitly setting the exception
>>handler seemed like a reasonable thing to do.  (and maybe relevant to ppl on
>>this list)
>>
>>2.  So then we went to look at the Future and realized what was happening,
>>the future task was catching throwable. (Yes, I know all about custom
>>futures, they don't help me now)  The only way we could rationally seem to
>>do this is to override afterExecute() in the executor and do the following
>>....
>>
>>try {
>>                // note I'd rather cast than use temporary objects
>>                if ( !((FutureTask) task).isCancelled()){  // I don't care
>>about CancellationExceptions
>>                    ((FutureTask) task).get();
>>                }
>>             } catch (ExecutionException ee) {
>>                 // ie if (ee == OutOfMemoryError) {System.exit()}
>>            }
>>}
>>
>>3. Ok, so now for every task we're checking if it is cancelled, casting 2x,
>>and getting a result we almost never want, all of which have a performance
>>impact.   I tried throwing an OutOfMemoryError .... didn't trip the exit().
>>More than a little aggravated, I looked at the exception, and realized that
>>ExecutionException wraps Throwable, which means I need to inspect the cause.
>> So I changed that bit of code to
>>
>>
>>try {
>>               if ( !((FutureTask) task).isCancelled()){ // I still don't
>>care about CancellationExceptions
>>                    ((FutureTask) task).get();
>>                }
>>             } catch (ExecutionException ee) {
>>                 //  ie if (ee.getCause() == OutOfMemoryError)
>>{System.exit()}
>>            }
>>}
>>
>>4.  The fourth problem I had was a bit subtle, and didn't come up until
>>later.   We have subsequent tasks that queue up on other executors.  This
>>was resulting in a chain of ExecutionExceptions being set up, which meant we
>>had to recursively inspect all exceptions in the chain to make sure there
>>was no out of memory error.   Think SEDA, and you'll be able to picture why
>>we have multiple stages and chained execution exceptions.  It also makes us
>>more vulnerable to StackOverflow errors.
>>
>>Once you pull that all together, you can catch the OutOfMemoryError cleanly,
>>I think.  The moral of this story may be to not use the Executors if you
>>want to be a framework and don't trust your tasks, but I don't think it has
>>to be this hard.  There are several places in the Java, and patterns that I
>>can think of that have a setExceptionHandler() pattern going on, wouldn't it
>>be reasonable to add similar functionality instead of forcing us to go
>>through these hoops to find an Error ?
>>
>>Again, all of this can be avoided if you trust your tasks to do a try /
>>catch (Throwable t) , but as part of a framework I don't exactly have that
>>luxury.  Thanks again for all the information, I hope this is useful to
>>others on this list.
>>
>>Regards,
>>John Adams
>>
>>    
>>
>
>_______________________________________________
>Concurrency-interest mailing list
>[hidden email]
>http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
>
>  
>


--
Jan Nielsen
System Architect
Luminis Solutions
SunGard SCT

[hidden email]
http://www.sungardsct.com

+1 801 257 4155 (voice)
+1 801 485 6606 (facsimile)

90 South 400 West
Salt Lake City, UT 84101

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Exception handling & Executors

Joe Bowbeer
On 10/25/05, Jan Nielsen <[hidden email]> wrote:
>
> With respect to your comment below about not being able to do anything
> with an out-of-memory condition, could you not reliably release, for
> example, cached data in an OutOfMemoryError handler:
>

In my experience, recovery from OOME is dicey at best and therefore
the only robust approach is to design and implement the app in such as
way that OOME never happens.

(Note: real-time Java tells a different story.)

I haven't investigated carefully why OOME recovery is dicey, but since
nothing is supposed to catch it I assume that once it is thrown the
system is in an unrecoverable state: no cleanup has been performed
along the way, connections and streams are broken, third-party
libraries are left in some indeterminate state, etc.

If your app could control where the OOME occurs then it would have a
better chance to catch it and recover -- by releasing buffers, for
example.  But I don't have a lot of faith that this can be done
reliably (or efficiently).


On 10/25/05, Jan Nielsen <[hidden email]> wrote:

> Joe,
>
> With respect to your comment below about not being able to do anything
> with an out-of-memory condition, could you not reliably release, for
> example, cached data in an OutOfMemoryError handler:
>
>   Map m = new HashMap();
>   Object o = new PreAllocatedMemoryInBytes( 16 );
>   try
>   {
>     // blah, blah, blah...allocating all the memory to 'm' values...
>   }
>   catch( OutOfMemoryError e )
>   {
>       o = null;
>       m.clear();
>       // only log after we have enough memory
>       e.printStackTrace();
>   }
>
> The m.clear() method may require some allocation to work, so it would be
> susceptible to failure without having that much available memory,
> specifically:
>
>     public void clear() {
>         modCount++;
>         Entry[] tab = table;
>         for (int i = 0; i < tab.length; i++)
>             tab[i] = null;
>         size = 0;
>     }
>
> In the single-threaded case, pre-allocating sufficient memory to perform
> this operation should address this, should it not?
>
> Thanks,
>
> -Jan
>
> Joe Bowbeer wrote:
> >
> >I don't know how you expect to do anything with an
> >OutOfMemoryException, except perhaps log it.  I don't know of any way
> >to reliably recover from these.
> >

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

RE: Exception handling & Executors

John Adams-5
In reply to this post by John Adams-5
Hi Joe,

Thank you for the reply.  Regarding your comment about OutOfMemory, as
was buried in the original post, I want to log the exception and then do
a System.exit().  We have a watchdog process that restarts the server
and starts again.  If we have a gradual leak going on, it's better in
the field for us to raise an alarm and restart the server than it is to
thrash with out of memory.  We're trying to build a reliable system, and
we have to deal with the possibility that this could happen in the
field, despite all of our best leak detection efforts.  

To respond ...

1. We use runnables in our implementation already.  If you pass a
runnable into a task, this code in AbstractExecutorService gets hit ...

 public <T> Future<T> submit(Runnable task, T result) {
        if (task == null) throw new NullPointerException();
        FutureTask<T> ftask = new FutureTask<T>(task, result);
        execute(ftask);
        return ftask;
    }

Note the wrapping of the FutureTask, and the source of my complaint.
The inner task uses Sync, an inner class of FutureTask, which does this
..

 void innerRun() {
            if (!compareAndSetState(0, RUNNING))
                return;
            try {
                runner = Thread.currentThread();
                innerSet(callable.call());
            } catch(Throwable ex) {
                innerSetException(ex);
            }
        }

2. I do hook into the afterExecute() which resolves my problem, albeit
with some ugly code.   I'm really just saying that this solution is
complex and hoping someone in the future will implement a
setDefaultExceptionHandler() equivalent for the Executor framework.  I
also thought others might have this problem, so it would be useful to
provide this code to others with similar requirements.

3. The Thread.setDefaultExceptionHandler doesn't resolve the problem,
the FutureTask/Sync catches throwable.

4. The done() idea is interesting, although I'm guessing that's part of
how the custom future task will deal with the problem in later releases.
I'm not sure how I can use that to my advantage in 1.5.

Incidentally, someone here did suggest re-implementing the whole
threadpool (ExecutorService) ourselves to "work around" the issue.  I'll
consider that notion if the casts and the isCancelled() call show up in
profiling, but for now I'd like to stick with the standard libraries if
possible.

Regards,
John.

-----Original Message-----
From: Joe Bowbeer [mailto:[hidden email]]
Sent: Tuesday, October 25, 2005 1:24 PM
To: Adams, John [CAR:3P41:EXCH]
Cc: [hidden email]
Subject: Re: [concurrency-interest] Exception handling & Executors


A few ideas:

1. If you execute Runnable instead of FutureTask then any
RuntimeExceptions and Errors thrown can escape the Executor.

2. You can hook the ThreadPoolExecutor.afterExecute method to handle the
RuntimeExceptions thrown by your task.  (Note: the Throwable signature
is a little misleading.)

3. You can use the Thread.set[Default]UncaughtExceptionHandler to handle
uncaught exceptions and errors of all types.  (To associate the
exception handling with your executor, you can provide your own thread
factory.)

4. If you want to use FutureTask then you can hook its done() method and
deal with the exception there.


A comment:

I don't know how you expect to do anything with an OutOfMemoryException,
except perhaps log it.  I don't know of any way to reliably recover from
these.


On 10/25/05, John Adams <[hidden email]> wrote:

>
> Hello,
>
> First off, thanks for maintaining this list, it has proven invaluable
> to me as I learn to use the concurrency utilities in Java 5.0.
>
> The reason I'm posting is that I've had a lot of problems using the
> Executor framework to deal with exceptions from errant tasks.  I've
> complained about this to Sun, whose response was essentially that
> exceptions belong with the task, so they should be try / caught at
> that level.
>
> I don't agree.  I don't, for example, believe that it is the
> responsibility of the task to handle OutOfMemoryError's, or in fact
most of the classes
> that occur within the Error class of throwables.   Many of those
exceptions
> imply that the tasks should have some knowledge about how to respond
> to events are within the domain of the platform the tasks are
> executing on. Now you could reasonably argue --> What is there to do
anyways in that case
> ?   All I want to do is a System.exit(), since we have a watchdog
process
> that does "the right thing" when the VM goes down.  In theory that
> would have been easy for me to do with our custom threadpools pre-1.5,

> however to get it working with the concurrent utilities has been a
> painful experience.
>
> I thought it may be helpful to note what we tried in what order, so
> that maybe we could get some support beyond the custom Future task
> support going in later VM releases.  So, in order ....
>
> 1. Override the ThreadGroup before creating the Executor.    We did
this

> initially expecting things to be easy, however we were quickly
> thwarted by the DefaultThreadFactory that does this ...
>
>        DefaultThreadFactory() {
>             SecurityManager s = System.getSecurityManager();
>             group = (s != null)? s.getThreadGroup() :
>
> Thread.currentThread().getThreadGroup();
>          }
>
> I can see some security reasons for this, however I was not expecting
> it --> I have stages that I wanted to log & name according to
> threadgroups. However, that's not insurmountable ... I created my own
> threadfactory which just took the currentThread().getThreadGroup() ...
no luck.
>   I also used the setExceptionHandler() method on Thread &
> ThreadGroup, but to no avail.  I left this code in since I wanted this

> code to be independent of our security manager choices, and explicitly

> setting the exception handler seemed like a reasonable thing to do.  
> (and maybe relevant to ppl on this list)
>
> 2.  So then we went to look at the Future and realized what was
> happening, the future task was catching throwable. (Yes, I know all
> about custom futures, they don't help me now)  The only way we could
> rationally seem to do this is to override afterExecute() in the
> executor and do the following ....
>
> try {
>                 // note I'd rather cast than use temporary objects
>                 if ( !((FutureTask) task).isCancelled()){  // I don't
> care about CancellationExceptions
>                     ((FutureTask) task).get();
>                 }
>              } catch (ExecutionException ee) {
>                  // ie if (ee == OutOfMemoryError) {System.exit()}
>             }
> }
>
> 3. Ok, so now for every task we're checking if it is cancelled,
> casting 2x, and getting a result we almost never want, all of which
have a performance
> impact.   I tried throwing an OutOfMemoryError .... didn't trip the
exit().
> More than a little aggravated, I looked at the exception, and realized

> that ExecutionException wraps Throwable, which means I need to inspect

> the cause.  So I changed that bit of code to
>
>
> try {
>                if ( !((FutureTask) task).isCancelled()){ // I still
> don't care about CancellationExceptions
>                     ((FutureTask) task).get();
>                 }
>              } catch (ExecutionException ee) {
>                  //  ie if (ee.getCause() == OutOfMemoryError)
> {System.exit()}
>             }
> }
>
> 4.  The fourth problem I had was a bit subtle, and didn't come up
until
> later.   We have subsequent tasks that queue up on other executors.
This
> was resulting in a chain of ExecutionExceptions being set up, which
> meant we had to recursively inspect all exceptions in the chain to
make sure there
> was no out of memory error.   Think SEDA, and you'll be able to
picture why
> we have multiple stages and chained execution exceptions.  It also
> makes us more vulnerable to StackOverflow errors.
>
> Once you pull that all together, you can catch the OutOfMemoryError
> cleanly, I think.  The moral of this story may be to not use the
> Executors if you want to be a framework and don't trust your tasks,
> but I don't think it has to be this hard.  There are several places in

> the Java, and patterns that I can think of that have a
> setExceptionHandler() pattern going on, wouldn't it be reasonable to
> add similar functionality instead of forcing us to go through these
> hoops to find an Error ?
>
> Again, all of this can be avoided if you trust your tasks to do a try
> / catch (Throwable t) , but as part of a framework I don't exactly
> have that luxury.  Thanks again for all the information, I hope this
> is useful to others on this list.
>
> Regards,
> John Adams
>


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Exception handling & Executors

Joe Bowbeer
I meant:

executor.execute(runnable)

This gives you the most control over the actual task that is
executing.  For example, your task could be a custom FutureTask with a
done() method specialized for handling OOME.

To isolate the impact, you could create a "task factory" that took a
runnable and returned a FutureTask customized for OOME handling.

Or (and this is probably the best solution for you) you could create a
custom ExecutorService that did this for you, by extending
AbstractExecutorService.

This sort of extension is made even easier in 1.6 Mustang with the
addition of the newTaskFor method:

    protected <T> RunnableFuture<T> newTaskFor(Runnable runnable, T value) {
        return new FutureTask<T>(runnable, value);
    }


On 10/25/05, John Adams <[hidden email]> wrote:

> Hi Joe,
>
> Thank you for the reply.  Regarding your comment about OutOfMemory, as
> was buried in the original post, I want to log the exception and then do
> a System.exit().  We have a watchdog process that restarts the server
> and starts again.  If we have a gradual leak going on, it's better in
> the field for us to raise an alarm and restart the server than it is to
> thrash with out of memory.  We're trying to build a reliable system, and
> we have to deal with the possibility that this could happen in the
> field, despite all of our best leak detection efforts.
>
> To respond ...
>
> 1. We use runnables in our implementation already.  If you pass a
> runnable into a task, this code in AbstractExecutorService gets hit ...
>
>  public <T> Future<T> submit(Runnable task, T result) {
>         if (task == null) throw new NullPointerException();
>         FutureTask<T> ftask = new FutureTask<T>(task, result);
>         execute(ftask);
>         return ftask;
>     }
>
> Note the wrapping of the FutureTask, and the source of my complaint.
> The inner task uses Sync, an inner class of FutureTask, which does this
> ..
>
>  void innerRun() {
>             if (!compareAndSetState(0, RUNNING))
>                 return;
>             try {
>                 runner = Thread.currentThread();
>                 innerSet(callable.call());
>             } catch(Throwable ex) {
>                 innerSetException(ex);
>             }
>         }
>
> 2. I do hook into the afterExecute() which resolves my problem, albeit
> with some ugly code.   I'm really just saying that this solution is
> complex and hoping someone in the future will implement a
> setDefaultExceptionHandler() equivalent for the Executor framework.  I
> also thought others might have this problem, so it would be useful to
> provide this code to others with similar requirements.
>
> 3. The Thread.setDefaultExceptionHandler doesn't resolve the problem,
> the FutureTask/Sync catches throwable.
>
> 4. The done() idea is interesting, although I'm guessing that's part of
> how the custom future task will deal with the problem in later releases.
> I'm not sure how I can use that to my advantage in 1.5.
>
> Incidentally, someone here did suggest re-implementing the whole
> threadpool (ExecutorService) ourselves to "work around" the issue.  I'll
> consider that notion if the casts and the isCancelled() call show up in
> profiling, but for now I'd like to stick with the standard libraries if
> possible.
>
> Regards,
> John.
>
> -----Original Message-----
> From: Joe Bowbeer [mailto:[hidden email]]
> Sent: Tuesday, October 25, 2005 1:24 PM
> To: Adams, John [CAR:3P41:EXCH]
> Cc: [hidden email]
> Subject: Re: [concurrency-interest] Exception handling & Executors
>
>
> A few ideas:
>
> 1. If you execute Runnable instead of FutureTask then any
> RuntimeExceptions and Errors thrown can escape the Executor.
>
> 2. You can hook the ThreadPoolExecutor.afterExecute method to handle the
> RuntimeExceptions thrown by your task.  (Note: the Throwable signature
> is a little misleading.)
>
> 3. You can use the Thread.set[Default]UncaughtExceptionHandler to handle
> uncaught exceptions and errors of all types.  (To associate the
> exception handling with your executor, you can provide your own thread
> factory.)
>
> 4. If you want to use FutureTask then you can hook its done() method and
> deal with the exception there.
>
>
> A comment:
>
> I don't know how you expect to do anything with an OutOfMemoryException,
> except perhaps log it.  I don't know of any way to reliably recover from
> these.
>
>
> On 10/25/05, John Adams <[hidden email]> wrote:
> >
> > Hello,
> >
> > First off, thanks for maintaining this list, it has proven invaluable
> > to me as I learn to use the concurrency utilities in Java 5.0.
> >
> > The reason I'm posting is that I've had a lot of problems using the
> > Executor framework to deal with exceptions from errant tasks.  I've
> > complained about this to Sun, whose response was essentially that
> > exceptions belong with the task, so they should be try / caught at
> > that level.
> >
> > I don't agree.  I don't, for example, believe that it is the
> > responsibility of the task to handle OutOfMemoryError's, or in fact
> most of the classes
> > that occur within the Error class of throwables.   Many of those
> exceptions
> > imply that the tasks should have some knowledge about how to respond
> > to events are within the domain of the platform the tasks are
> > executing on. Now you could reasonably argue --> What is there to do
> anyways in that case
> > ?   All I want to do is a System.exit(), since we have a watchdog
> process
> > that does "the right thing" when the VM goes down.  In theory that
> > would have been easy for me to do with our custom threadpools pre-1.5,
>
> > however to get it working with the concurrent utilities has been a
> > painful experience.
> >
> > I thought it may be helpful to note what we tried in what order, so
> > that maybe we could get some support beyond the custom Future task
> > support going in later VM releases.  So, in order ....
> >
> > 1. Override the ThreadGroup before creating the Executor.    We did
> this
> > initially expecting things to be easy, however we were quickly
> > thwarted by the DefaultThreadFactory that does this ...
> >
> >        DefaultThreadFactory() {
> >             SecurityManager s = System.getSecurityManager();
> >             group = (s != null)? s.getThreadGroup() :
> >
> > Thread.currentThread().getThreadGroup();
> >          }
> >
> > I can see some security reasons for this, however I was not expecting
> > it --> I have stages that I wanted to log & name according to
> > threadgroups. However, that's not insurmountable ... I created my own
> > threadfactory which just took the currentThread().getThreadGroup() ...
> no luck.
> >   I also used the setExceptionHandler() method on Thread &
> > ThreadGroup, but to no avail.  I left this code in since I wanted this
>
> > code to be independent of our security manager choices, and explicitly
>
> > setting the exception handler seemed like a reasonable thing to do.
> > (and maybe relevant to ppl on this list)
> >
> > 2.  So then we went to look at the Future and realized what was
> > happening, the future task was catching throwable. (Yes, I know all
> > about custom futures, they don't help me now)  The only way we could
> > rationally seem to do this is to override afterExecute() in the
> > executor and do the following ....
> >
> > try {
> >                 // note I'd rather cast than use temporary objects
> >                 if ( !((FutureTask) task).isCancelled()){  // I don't
> > care about CancellationExceptions
> >                     ((FutureTask) task).get();
> >                 }
> >              } catch (ExecutionException ee) {
> >                  // ie if (ee == OutOfMemoryError) {System.exit()}
> >             }
> > }
> >
> > 3. Ok, so now for every task we're checking if it is cancelled,
> > casting 2x, and getting a result we almost never want, all of which
> have a performance
> > impact.   I tried throwing an OutOfMemoryError .... didn't trip the
> exit().
> > More than a little aggravated, I looked at the exception, and realized
>
> > that ExecutionException wraps Throwable, which means I need to inspect
>
> > the cause.  So I changed that bit of code to
> >
> >
> > try {
> >                if ( !((FutureTask) task).isCancelled()){ // I still
> > don't care about CancellationExceptions
> >                     ((FutureTask) task).get();
> >                 }
> >              } catch (ExecutionException ee) {
> >                  //  ie if (ee.getCause() == OutOfMemoryError)
> > {System.exit()}
> >             }
> > }
> >
> > 4.  The fourth problem I had was a bit subtle, and didn't come up
> until
> > later.   We have subsequent tasks that queue up on other executors.
> This
> > was resulting in a chain of ExecutionExceptions being set up, which
> > meant we had to recursively inspect all exceptions in the chain to
> make sure there
> > was no out of memory error.   Think SEDA, and you'll be able to
> picture why
> > we have multiple stages and chained execution exceptions.  It also
> > makes us more vulnerable to StackOverflow errors.
> >
> > Once you pull that all together, you can catch the OutOfMemoryError
> > cleanly, I think.  The moral of this story may be to not use the
> > Executors if you want to be a framework and don't trust your tasks,
> > but I don't think it has to be this hard.  There are several places in
>
> > the Java, and patterns that I can think of that have a
> > setExceptionHandler() pattern going on, wouldn't it be reasonable to
> > add similar functionality instead of forcing us to go through these
> > hoops to find an Error ?
> >
> > Again, all of this can be avoided if you trust your tasks to do a try
> > / catch (Throwable t) , but as part of a framework I don't exactly
> > have that luxury.  Thanks again for all the information, I hope this
> > is useful to others on this list.
> >
> > Regards,
> > John Adams
> >
>
>

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Exception handling & Executors

tpeierls
Joe Bowbeer wrote:

> I meant:
>
> executor.execute(runnable)
>
> This gives you the most control over the actual task that is
> executing.  For example, your task could be a custom FutureTask with a
> done() method specialized for handling OOME.
>
> To isolate the impact, you could create a "task factory" that took a
> runnable and returned a FutureTask customized for OOME handling.
>
> Or (and this is probably the best solution for you) you could create a
> custom ExecutorService that did this for you, by extending
> AbstractExecutorService.

Specifically, with something like this:

   public OomeHandler {
       void handleOutOfMemoryError(OutOfMemoryError e);
   }

   public class OomeHandlingExecutorService extends AbstractExecutorService {
       private final ExecutorService exec;
       private final OomeHandler handler;
       public OomeHandlingExecutorService(ExecutorService exec, OomeHandler handler) {
           this.exec = exec;
       }
       public void execute(Runnable runnable) {
           exec.execute(runnable);
       }
       public <V> FutureTask<V> submit(Callable<V> callable) {
           if (callable == null) throw new NullPointerException();
           FutureTask<T> ftask = new OomeHandlingFutureTask<V>(callable);
           execute(ftask);
           return ftask;
       }
       public <V> FutureTask<V> submit(Runnable runnable, V result) {
           if (runnable == null) throw new NullPointerException();
           FutureTask<T> ftask = new OomeHandlingFutureTask<V>(runnable);
           execute(ftask);
           return ftask;
       }
       public FutureTask<?> submit(Runnable runnable) {
           if (runnable == null) throw new NullPointerException();
           FutureTask<Object> ftask = new OomeHandlingFutureTask<Object>(runnable, null);
           execute(ftask);
           return ftask;
       }

       // invokeAny/All and lifecycle methods all delegate to exec

       private static class OomeHandlingFutureTask<V> extends FutureTask {
           private final OomeHandler handler;
           OomeHandlingFutureTask(Callable<V> callable, OomeHandler handler) {
               super(callable);
               this.handler = handler;
           }
           OomeHandlingFutureTask(Runnable runnable, V result) {
               super(runnable, result);
               this.handler = handler;
           }
           protected void done() {
               try {
                   get();
               } catch (CancellationException e) {
                   // ignore
               } catch (ExecutionException e) {
                   Throwable t = e.getCause();
                   if (handler != null && t instanceof OutOfMemoryError)
                       handler.handleOutOfMemoryError((OutOfMemoryError) t);
               }
           }
       }
   }

Delegating the invokeAll and invokeAny methods above means that invoked tasks don't
get OOME handling. This can be addressed with copy-and-paste, but as Joe says:

> This sort of extension is made even easier in 1.6 Mustang with the
> addition of the newTaskFor method:
>
>     protected <T> RunnableFuture<T> newTaskFor(Runnable runnable, T value) {
>         return new FutureTask<T>(runnable, value);
>     }

For example, you could subclass TPE so that it returns OomeHandlingFutureTask, which is much more
compact that the code above.

--tim

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Exception handling & Executors

Dawid Kurzyniec
Tim Peierls wrote:

> Joe Bowbeer wrote:
>
>> I meant:
>>
>> executor.execute(runnable)
>>
>> This gives you the most control over the actual task that is
>> executing.  For example, your task could be a custom FutureTask with a
>> done() method specialized for handling OOME.
>>
>> To isolate the impact, you could create a "task factory" that took a
>> runnable and returned a FutureTask customized for OOME handling.
>>
>> Or (and this is probably the best solution for you) you could create a
>> custom ExecutorService that did this for you, by extending
>> AbstractExecutorService.
>
>
> Specifically, with something like this:
>
...

>       public void execute(Runnable runnable) {
>           exec.execute(runnable);
>       }

...

This seems to require "And", not "Or": in the above, you still cannot
pass a FutureTask to execute(), or it will swallow the throwable.
Instead, you need to use OomeHandlingFutureTask, or just a plain
runnables that do not catch Throwable.

Regards,
Dawid


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Exception handling & Executors

Dawid Kurzyniec
In reply to this post by Joe Bowbeer
Joe Bowbeer wrote:

>On 10/25/05, Jan Nielsen <[hidden email]> wrote:
>  
>
>>With respect to your comment below about not being able to do anything
>>with an out-of-memory condition, could you not reliably release, for
>>example, cached data in an OutOfMemoryError handler:
>>
>>    
>>
>In my experience, recovery from OOME is dicey at best and therefore
>the only robust approach is to design and implement the app in such as
>way that OOME never happens.
>
>(Note: real-time Java tells a different story.)
>
>I haven't investigated carefully why OOME recovery is dicey, but since
>nothing is supposed to catch it I assume that once it is thrown the
>system is in an unrecoverable state: no cleanup has been performed
>along the way, connections and streams are broken, third-party
>libraries are left in some indeterminate state, etc.
>
>  
>
I think that you have just said precisely why it is dicey (at least it
convinced me): OOME can be thrown by any memory allocation attempt,
including that performed by the library code that you don't control. If
the allocation was attempted inside a synchronized block, the error
causes the abrupt completion, and may leave shared data structures in an
inconsistent state, since no library is actually written to guard
against OOME. It is a bit similar to dangers of Thread.stop(), although
just a little bit less bad, since OOME is not asynchronous.

>If your app could control where the OOME occurs then it would have a
>better chance to catch it and recover -- by releasing buffers, for
>example.  But I don't have a lot of faith that this can be done
>reliably (or efficiently).
>
>  
>
If something can be safely released to recover from OOME, it is better
to make that something soft -referenceable, in which case it will be
automatically released, *preventing* OOME, when you're running low on
memory. Soft-referenced objects are released only if the system cannot
otherwise satisfy an allocation request.

The problem with OOME is that you never know which thread will
experience it first if you are running low on memory.

Regards,
Dawid


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest