Backport: ConcurrentLinkedQueue/BlockingLinkedQueue Memory Leak?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Backport: ConcurrentLinkedQueue/BlockingLinkedQueue Memory Leak?

Brian O'Connell

   I am seeing some odd behavior in a project I am working on.  I have a
system that contains 5 work queues all of which have objects put on the
queue with add() and removed with a poll() until null loop.
I found that after sometime, perhaps in the area of 10 minutes I run out
of memory.  I had this occur with both the ConcurrentLinkedQueue and the
BlockingLinkedQueue implementations.  If I change to an
ArrayBlockingQueue I never see this behavior (and I also never throw an
IllegalStateException so I am not putting too many objects on the queue).
   In debugging this I had a thread print out the return from size() on
all 5 queues every second.  None of them ever grew larger than 9000
(another thread was inserting 9000 entries per second on one of the
queues.)  However, when I ran out of memory I analyzed the heapdump and
had over 2 million ConcurrentLinkedQueue$Node or
BlockingLinkedQueue$Node objects in memory.
   I am using version 2.0 from August.  I was wondering if this is a
reported bug?  I imagine if not, then there must be a bug somewhere else
in my program but I have yet to locate it.  I found this entry by Doug
Lea
http://altair.cs.oswego.edu/pipermail/concurrency-interest/2005-January/001319.html 
describing some leak condition, but I can't tell if it is the same
because I never poll() with a timeout.
   I see version 2.1 is available but in the release history I saw no
mention of the issue I am seeing so I am leery of blindly upgrading and
possibly masking this issue until it occurs at a much worse time than
stress testing.

Thanks,
Brian

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Backport: ConcurrentLinkedQueue/BlockingLinkedQueueMemory Leak?

Dawid Kurzyniec
Brian O'Connell wrote:

>
>   I am seeing some odd behavior in a project I am working on.  I have
> a system that contains 5 work queues all of which have objects put on
> the queue with add() and removed with a poll() until null loop.
> I found that after sometime, perhaps in the area of 10 minutes I run
> out of memory.  I had this occur with both the ConcurrentLinkedQueue
> and the BlockingLinkedQueue implementations.  If I change to an
> ArrayBlockingQueue I never see this behavior (and I also never throw
> an IllegalStateException so I am not putting too many objects on the
> queue).
>   In debugging this I had a thread print out the return from size() on
> all 5 queues every second.  None of them ever grew larger than 9000
> (another thread was inserting 9000 entries per second on one of the
> queues.)  However, when I ran out of memory I analyzed the heapdump
> and had over 2 million ConcurrentLinkedQueue$Node or
> BlockingLinkedQueue$Node objects in memory.
>   I am using version 2.0 from August.  I was wondering if this is a
> reported bug?  I imagine if not, then there must be a bug somewhere
> else in my program but I have yet to locate it.  (...)

Brian,

First: are you using j.u.c. on 5.0, or backport-util-concurrent on 1.4?
What platform / JVM?


ConcurrentLinkedQueue and BlockingLinkedQueue implementations are
actually quite different; I would be surprised if they both contained
bugs manifesting themselves in identical ways. So I would be inclined to
suspect that, after all, your code occassionally "go crazy" on filling
up the queue; e.g. maybe one of the consumers crashes?... Perhaps you
just wasn't lucky enough to reproduce this behavior while debugging?...

Why don't you use bounded queues anyway - it is just safer, unless
producers and consumers are synced in some way so you can be
_absolutely_ sure that producers' rate won't exceed consumers'. So try
using a bounded LinkedBlockingQueue and see if the problem persists. I
would also try to run the program under a memory profiler, which can
show you black on white where the leak comes from.

Regards,
Dawid Kurzyniec

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest