Why might java.lang.Object.notifyAll take 15 seconds? Java server app performance defect

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Why might java.lang.Object.notifyAll take 15 seconds? Java server app performance defect

JSR166 Concurrency mailing list
In a large customer production deployment, we're using JProfiler 11.x to measure an application's performance.  JProfiler is reporting over a 5 minute period <10 times a thread is getting stuck for 15 seconds performing a java.lang.Object#notifyAll.

Why might java.lang.Object.notifyAll take so long to complete?  The notifyAll is being performed on a java.util.collection.LinkedList.  This list contains a queue of inbound TCP messages.

Environment:
Windows Server 2012 SP2 64-bit
Java 8u171 64-bit
4 CPU sockets, 2 CPU cores in each socket

Brian Brooks
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Why might java.lang.Object.notifyAll take 15 seconds? Java server app performance defect

JSR166 Concurrency mailing list
Thundering herd? How many threads are trying to pull messages out from
the linked list? If one message is added to the list, then notifyAll
wakes up all the threads, but one at a time. Each notify might cause the
thread to immediately force a context switch, causing it to take longer
to regain control and finish iterating over the list of waiting threads.

If this is a custom work queue design, then perhaps one of the built-in
JDK thread pool classes would work better?

On 2019-04-22 08:01 AM, Brian Brooks (US) via Concurrency-interest wrote:

> In a large customer production deployment, we're using JProfiler 11.x to measure an application's performance.  JProfiler is reporting over a 5 minute period <10 times a thread is getting stuck for 15 seconds performing a java.lang.Object#notifyAll.
>
> Why might java.lang.Object.notifyAll take so long to complete?  The notifyAll is being performed on a java.util.collection.LinkedList.  This list contains a queue of inbound TCP messages.
>
> Environment:
> Windows Server 2012 SP2 64-bit
> Java 8u171 64-bit
> 4 CPU sockets, 2 CPU cores in each socket
>
> Brian Brooks
> _______________________________________________
> Concurrency-interest mailing list
> [hidden email]
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Why might java.lang.Object.notifyAll take 15 seconds? Java server app performance defect

JSR166 Concurrency mailing list
In reply to this post by JSR166 Concurrency mailing list
My first guess would be that this is a false report.  The threads might be blocked on getting the lock and with all the JITting and TTSP going on the profiler might be getting the place in the code wrong.  

On Mon, 22 Apr 2019 at 18:07, Brian Brooks (US) via Concurrency-interest <[hidden email]> wrote:
In a large customer production deployment, we're using JProfiler 11.x to measure an application's performance.  JProfiler is reporting over a 5 minute period <10 times a thread is getting stuck for 15 seconds performing a java.lang.Object#notifyAll.

Why might java.lang.Object.notifyAll take so long to complete?  The notifyAll is being performed on a java.util.collection.LinkedList.  This list contains a queue of inbound TCP messages.

Environment:
Windows Server 2012 SP2 64-bit
Java 8u171 64-bit
4 CPU sockets, 2 CPU cores in each socket

Brian Brooks
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
--
Dr Heinz M. Kabutz (PhD CompSci)
Author of "The Java(tm) Specialists' Newsletter"
Sun/Oracle Java Champion
JavaOne Rockstar Speaker
http://www.javaspecialists.eu
Tel: +30 69 75 595 262
Skype: kabutz

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Why might java.lang.Object.notifyAll take 15 seconds? Java server app performance defect

JSR166 Concurrency mailing list
+100 for this guess...
Enable debug non safepoints, GC stopped time logs + using a proper profiler would help (eg async-profiler with -e wall option)

Il lun 22 apr 2019, 17:36 Dr Heinz M. Kabutz via Concurrency-interest <[hidden email]> ha scritto:
My first guess would be that this is a false report.  The threads might be blocked on getting the lock and with all the JITting and TTSP going on the profiler might be getting the place in the code wrong.  

On Mon, 22 Apr 2019 at 18:07, Brian Brooks (US) via Concurrency-interest <[hidden email]> wrote:
In a large customer production deployment, we're using JProfiler 11.x to measure an application's performance.  JProfiler is reporting over a 5 minute period <10 times a thread is getting stuck for 15 seconds performing a java.lang.Object#notifyAll.

Why might java.lang.Object.notifyAll take so long to complete?  The notifyAll is being performed on a java.util.collection.LinkedList.  This list contains a queue of inbound TCP messages.

Environment:
Windows Server 2012 SP2 64-bit
Java 8u171 64-bit
4 CPU sockets, 2 CPU cores in each socket

Brian Brooks
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
--
Dr Heinz M. Kabutz (PhD CompSci)
Author of "The Java(tm) Specialists' Newsletter"
Sun/Oracle Java Champion
JavaOne Rockstar Speaker
http://www.javaspecialists.eu
Tel: +30 69 75 595 262
Skype: kabutz
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Why might java.lang.Object.notifyAll take 15 seconds? Java server app performance defect

JSR166 Concurrency mailing list
In reply to this post by JSR166 Concurrency mailing list
There's no thundering-herd with Object.notifyAll, it is simply a transfer from the wait-set queue to the monitor-queue (which is still owned by the thread doing the notifyAll). All the notified threads remain blocked until the monitor is released, which will release them one at a time.

David

> -----Original Message-----
> From: Concurrency-interest <[hidden email]> On Behalf Of Brian S O'Neill via Concurrency-interest
> Sent: Tuesday, April 23, 2019 1:32 AM
> To: Brian Brooks (US) <[hidden email]>; [hidden email]
> Subject: Re: [concurrency-interest] Why might java.lang.Object.notifyAll take 15 seconds? Java server app performance defect
>
> Thundering herd? How many threads are trying to pull messages out from the linked list? If one message is added to the list, then
> notifyAll wakes up all the threads, but one at a time. Each notify might cause the thread to immediately force a context switch, causing
> it to take longer to regain control and finish iterating over the list of waiting threads.
>
> If this is a custom work queue design, then perhaps one of the built-in JDK thread pool classes would work better?
>
> On 2019-04-22 08:01 AM, Brian Brooks (US) via Concurrency-interest wrote:
> > In a large customer production deployment, we're using JProfiler 11.x to measure an application's performance.  JProfiler is
> reporting over a 5 minute period <10 times a thread is getting stuck for 15 seconds performing a java.lang.Object#notifyAll.
> >
> > Why might java.lang.Object.notifyAll take so long to complete?  The notifyAll is being performed on a java.util.collection.LinkedList.
> This list contains a queue of inbound TCP messages.
> >
> > Environment:
> > Windows Server 2012 SP2 64-bit
> > Java 8u171 64-bit
> > 4 CPU sockets, 2 CPU cores in each socket
> >
> > Brian Brooks
> > _______________________________________________
> > Concurrency-interest mailing list
> > [hidden email]
> > http://cs.oswego.edu/mailman/listinfo/concurrency-interest
> >
> _______________________________________________
> Concurrency-interest mailing list
> [hidden email]
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest