BACKPORT - Deadlocks and FixedThreadPool executors...

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

BACKPORT - Deadlocks and FixedThreadPool executors...

Mark Derricutt
Hi all,

I've been using the backported concurrency library and hitting some locked threads which only seem to occur when using a fixed thread pool.

In the first instance I have a ScheduledExecutorServer (Executors.newSingleThreadScheduledExecutor ) which runs a process every 30 seconds, this process queues up jobs into fixed thread pool (Executors.newFixedThreadPool) if there not currently being processed.

Currently theres 10 jobs being submitted, to a fixed thread pool with a size of 5.  Everything seems to run fine for about 10 hours or so, I see jobs being processed by 5 threads, and I see the remaining jobs sitting in a blocking queue and being pulled out as threads come available.

Slowly thou, I see each of the threads blocking inside a socket read/wait from the JavaMail system, eventually all 5 threads block and I start pulling my hair out.

Initially I traced things down to a known problem with calling log4j from within synchronized methods but the problems still occured.  I switched my over to using a cached thread pool and I no longer see any threads getting locked, but having a larger number of threads concerns me (not immediately, but the number of processes being scheduled is likely to grow exponentially as time grows).

Has anyone here seen similar problems?  Or suggestions on improving on what I'm doing?

Mark

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: BACKPORT - Deadlocks and FixedThreadPool executors...

Gregg Wonderly-2


Mark Derricutt wrote:
> Slowly thou, I see each of the threads blocking inside a socket read/wait
> from the JavaMail system, eventually all 5 threads block and I start pulling
> my hair out.

Do you have a thread dump to share that would show where the threads are getting
stuck at?  You can send a SIGQUIT to the JVM on unix like operating systems by
typing CTRL-\ if you are running the JVM interactively.  If not, you just need
to PID of the process and then you can type "kill -3 <pid>" where <pid> is the
process number.  In windows, use CTRL-BREAK to do the same thing to a JVM
process that is running interactively in a DOS window.  Also, in JDK1.5 and
later you can use JMX to see where there the threads are.  JMX won't show you
the locks that are held to help you see circular waiting scenarios though.

Gregg Wonderly
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: BACKPORT - Deadlocks and FixedThreadPoolexecutors...

Dawid Kurzyniec
In reply to this post by Mark Derricutt
Mark Derricutt wrote:

> Hi all,
>
> I've been using the backported concurrency library and hitting some
> locked threads which only seem to occur when using a fixed thread pool.
>
> In the first instance I have a ScheduledExecutorServer
> (Executors.newSingleThreadScheduledExecutor ) which runs a process
> every 30 seconds, this process queues up jobs into fixed thread pool
> (Executors.newFixedThreadPool) if there not currently being processed.
>
> Currently theres 10 jobs being submitted, to a fixed thread pool with
> a size of 5.  Everything seems to run fine for about 10 hours or so, I
> see jobs being processed by 5 threads, and I see the remaining jobs
> sitting in a blocking queue and being pulled out as threads come
> available.
>
> Slowly thou, I see each of the threads blocking inside a socket
> read/wait from the JavaMail system, eventually all 5 threads block and
> I start pulling my hair out.
>
> Initially I traced things down to a known problem with calling log4j
> from within synchronized methods but the problems still occured.  I
> switched my over to using a cached thread pool and I no longer see any
> threads getting locked, but having a larger number of threads concerns
> me (not immediately, but the number of processes being scheduled is
> likely to grow exponentially as time grows).
>
> Has anyone here seen similar problems?  Or suggestions on improving on
> what I'm doing?

 From your description, I suspect that you're trying to implement some
sort of e-mail monitoring over several accounts. My suspicions are that
the reason for the behavior you're seeing are transient network
failures/delays. A task tries to check e-mail on a server, and the call
usually completes quickly, but occassionally it blocks for a longer
time, maybe because the server is busy or because the network is slow.
If it happens for all worker threads at the same time, you have a
temporary "clog". With a cached pool, you may not notice it, since the
throughput does not suffer, and the thing eventually "unclogs" so the
symptoms are harder to see. My recommendation, if it is possible to do,
would be to put a time constraint on a task. (I.e. if the mailbox cannot
be polled in 30 sec., just give up and bail out).

Try to measure time-to-completion of your tasks over a large time
interval to find the maximum value, to find out if that's the reason
behind the problem.

Other general possibilities (if I didn't guess your application scenario
correctly):

1. Your tasks have some hidden dependencies; e.g. the I/O call made by
task 1 cannot complete until task 6 is executed, but task 6 is on queue
and waits for an available thread -> deadlock. General remedy is to use
unbounded pool, or to increase max pool size enough to guarantee that
deadlocks cannot happen, if it can be guaranteed at all.

2. The code you're invoking from worker threads is not thread-safe;
after some time, one worker thread messes up a data structure used by
another thread, which causes that other one to behave badly - in this
case, reading from a stream that is not written to.

3. The code you're invoking from worker threads uses thread local
variables in some nasty way, so that the behavior of a task is partially
dependent on earlier tasks performed by the same worker thread, and you
have undeterminism.


Regards,
Dawid


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: BACKPORT - Deadlocks and FixedThreadPool executors...

Andrew Lentvorski
In reply to this post by Mark Derricutt
Mark Derricutt wrote:

> Slowly thou, I see each of the threads blocking inside a socket read/wait
> from the JavaMail system, eventually all 5 threads block and I start pulling
> my hair out.

Email systems often use file system locks to avoid mail store
corruption.  You can create circular lock dependencies through these as
well.  You might want to see what locks exist in the mail store.

The easiest solution is probably a timeout on your tasks.  Any thread
blocked longer than some amount of minutes should get smacked.  This
should work fairly easily since your problems are occurring on the order
of hours.

You may have to make the task interruptible before putting it onto the
queue.  When you interrupt the task, the JVM will close your socket, though.

-a
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest