Re: ClassLoader deadlock

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: ClassLoader deadlock

Peter Levart


On 02/06/2016 01:17 PM, Peter Firmstone wrote:
The security manager is non blocking, unfortunately java system classes aren't.

It doesn't seem so. At least, I can't find where main thread locks the 0x040ebee8 monitor:

"Thread-1":
  waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
  which is held by "main"


"main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor entry [0x0185e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
        - waiting to lock <0x03f624b8> (a java.lang.Object)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
        at org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
        at org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
        at org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
        at org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
        at org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
        at org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
        at org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
        at org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
        at org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
        at org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
        at java.lang.System.checkIO(System.java:253)
        at java.lang.System.setErr(System.java:199)
        at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)


...no mention of 0x040ebee8 in main thread...?!

Peter


The policy provider in use thread confines PermissionCollection instances, which never leave the cpu cache, they are discarded immediately after use.

The problem here is that two different threads are attempting to load the same class at the same time.

I'll have to make sure that all classes are loaded before the security manager becomes the system security manager, either that or eliminate the AccessControlContext check permission cache.

While the standard java security manager and policy provider consume around 10% cpu, this consumes less than 1%.

On 6/02/2016 9:57 PM, David Holmes wrote:
On 6/02/2016 9:39 PM, Peter Firmstone wrote:
Hmm, thought the new parallel lock stategy in ClassLoader wasn't
deadlock prone?  Guess I was wrong.

The deadlock is introduced by a custom security manager. SM's play a critical role in many parts of the core libraries so if they utilize synchronization then they can easily introduce deadlock.

Observation:  On an unrelated occassion, I had a URLClassLoader
synchronization hotspot (well not standard URLClassLoader, but a high
performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
instead of URL DNS lookup), I solved that problem by thread confining
class loading.  I've never experienced deadlock using thread
confinement, initially I was worried that there would be
interdependencies, however I didn't experience a problem, as the single
thread would always load the classes it needed.

I'm wondering if multiple locks for ClassLoader's might be the wrong
strategy, I found thread confinement was very performant.

Are you suggesting that a class-load request is actually transmitted to a single class-loading thread, while the initial thread blocks until the loading is complete? That would still seem prone to the same SM related deadlock scenario. And it would not generally be particularly performant.

I thought that too, until I tried it.

Would you like to see some performance figures?

Cheers,

Peter.


David

Regards,

Peter.

2016-02-06 21:06:07
Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
runnable [0x00000000]
    java.lang.Thread.State: RUNNABLE

"C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
nid=0x111c waiting on condition [0x00000000]
    java.lang.Thread.State: RUNNABLE

"Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00 nid=0x13d8
waiting on condition [0x00000000]
    java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400 nid=0x10b8
runnable [0x00000000]
    java.lang.Thread.State: RUNNABLE

"Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
monitor entry [0x03ded000]
    java.lang.Thread.State: BLOCKED (on object monitor)
         at
org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)

         at
org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
         at
org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)

         at
org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
         at
org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)

         at
java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
         at
java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)

         at
java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
         at
java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)

         at
java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)

         at
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
         at
java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)

         at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
         - locked <0x03f624b8> (a java.lang.Object)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
         at
org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)

         at
org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)

         at
org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)

         - locked <0x03ef8b30> (a
org.cliffc.high_scale_lib.NonBlockingHashMap)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
         at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j

ava:180)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294

)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

         at java.lang.Thread.run(Thread.java:744)

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
Object.wait() [0x03cff000]
    java.lang.Thread.State: WAITING (on object monitor)
         at java.lang.Object.wait(Native Method)
         - waiting on <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
         - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
nid=0x1398 in Object.wait() [0x144cf000]
    java.lang.Thread.State: WAITING (on object monitor)
         at java.lang.Object.wait(Native Method)
         - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:502)
         at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
         - locked <0x03e05200> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor
entry [0x0185e000]
    java.lang.Thread.State: BLOCKED (on object monitor)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
         - waiting to lock <0x03f624b8> (a java.lang.Object)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
         at
org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)

         at
org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
         at
org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)

         at
org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
         at
org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)

         at java.lang.System.checkIO(System.java:253)
         at java.lang.System.setErr(System.java:199)
         at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)

"VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable

"VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8 waiting on
condition

JNI global references: 19


Found one Java-level deadlock:
=============================
"Thread-1":
   waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
   which is held by "main"
"main":
   waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
java.lang.Object),
   which is held by "Thread-1"

Java stack information for the threads listed above:
===================================================
"Thread-1":
         at
org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)

         at
org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
         at
org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)

         at
org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
         at
org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)

         at
java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
         at
java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)

         at
java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
         at
java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)

         at
java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)

         at
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
         at
java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)

         at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
         - locked <0x03f624b8> (a java.lang.Object)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
         at
org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)

         at
org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)

         at
org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)

         - locked <0x03ef8b30> (a
org.cliffc.high_scale_lib.NonBlockingHashMap)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
         at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j

ava:180)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294

)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

         at java.lang.Thread.run(Thread.java:744)
"main":
         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
         - waiting to lock <0x03f624b8> (a java.lang.Object)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
         at
org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)

         at
org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
         at
org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)

         at
org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
         at
org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)

         at java.lang.System.checkIO(System.java:253)
         at java.lang.System.setErr(System.java:199)
         at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)

Found 1 deadlock.



_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: ClassLoader deadlock

Peter
The 0x040ebee8 monitor is most likely being held by native code.

Regards,

Peter Firmstone.


On 7/02/2016 4:27 AM, Peter Levart wrote:

>
>
> On 02/06/2016 01:17 PM, Peter Firmstone wrote:
>> The security manager is non blocking, unfortunately java system
>> classes aren't.
>
> It doesn't seem so. At least, I can't find where main thread locks the
> 0x040ebee8 monitor:
>
> "Thread-1":
>   waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>   which is held by "main"
>
>
> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for
> monitor entry [0x0185e000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>         - waiting to lock <0x03f624b8> (a java.lang.Object)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         at
> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>         at
> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>         at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>         at
> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>         at
> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>         at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>         at
> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>         at
> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>         at
> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>         at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>         at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>         at java.lang.System.checkIO(System.java:253)
>         at java.lang.System.setErr(System.java:199)
>         at
> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>
>
> ...no mention of 0x040ebee8 in main thread...?!
>
> Peter
>
>>
>> The policy provider in use thread confines PermissionCollection
>> instances, which never leave the cpu cache, they are discarded
>> immediately after use.
>>
>> The problem here is that two different threads are attempting to load
>> the same class at the same time.
>>
>> I'll have to make sure that all classes are loaded before the
>> security manager becomes the system security manager, either that or
>> eliminate the AccessControlContext check permission cache.
>>
>> While the standard java security manager and policy provider consume
>> around 10% cpu, this consumes less than 1%.
>>
>> On 6/02/2016 9:57 PM, David Holmes wrote:
>>> On 6/02/2016 9:39 PM, Peter Firmstone wrote:
>>>> Hmm, thought the new parallel lock stategy in ClassLoader wasn't
>>>> deadlock prone?  Guess I was wrong.
>>>
>>> The deadlock is introduced by a custom security manager. SM's play a
>>> critical role in many parts of the core libraries so if they utilize
>>> synchronization then they can easily introduce deadlock.
>>>
>>>> Observation:  On an unrelated occassion, I had a URLClassLoader
>>>> synchronization hotspot (well not standard URLClassLoader, but a high
>>>> performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
>>>> instead of URL DNS lookup), I solved that problem by thread confining
>>>> class loading.  I've never experienced deadlock using thread
>>>> confinement, initially I was worried that there would be
>>>> interdependencies, however I didn't experience a problem, as the
>>>> single
>>>> thread would always load the classes it needed.
>>>>
>>>> I'm wondering if multiple locks for ClassLoader's might be the wrong
>>>> strategy, I found thread confinement was very performant.
>>>
>>> Are you suggesting that a class-load request is actually transmitted
>>> to a single class-loading thread, while the initial thread blocks
>>> until the loading is complete? That would still seem prone to the
>>> same SM related deadlock scenario. And it would not generally be
>>> particularly performant.
>>
>> I thought that too, until I tried it.
>>
>> Would you like to see some performance figures?
>>
>> Cheers,
>>
>> Peter.
>>
>>>
>>> David
>>>
>>>> Regards,
>>>>
>>>> Peter.
>>>>
>>>> 2016-02-06 21:06:07
>>>> Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):
>>>>
>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
>>>> runnable [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
>>>> nid=0x111c waiting on condition [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00 nid=0x13d8
>>>> waiting on condition [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400
>>>> nid=0x10b8
>>>> runnable [0x00000000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>
>>>> "Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
>>>> monitor entry [0x03ded000]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>
>>>>
>>>>          at
>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>>>
>>>>
>>>>          at
>>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>          - locked <0x03f624b8> (a java.lang.Object)
>>>>          at
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>>>>
>>>>
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>>>>
>>>>
>>>>          - locked <0x03ef8b30> (a
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>          at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>
>>>>          at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>>>>
>>>>
>>>> ava:180)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>>>>
>>>>
>>>> )
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>
>>>>
>>>>          at java.lang.Thread.run(Thread.java:744)
>>>>
>>>> "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
>>>> Object.wait() [0x03cff000]
>>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          - waiting on <0x03e056d8> (a
>>>> java.lang.ref.ReferenceQueue$Lock)
>>>>          at
>>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
>>>>          - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>          at
>>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
>>>>          at
>>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>
>>>> "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
>>>> nid=0x1398 in Object.wait() [0x144cf000]
>>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>>          at java.lang.Object.wait(Native Method)
>>>>          - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>          at java.lang.Object.wait(Object.java:502)
>>>>          at
>>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>          - locked <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>
>>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for
>>>> monitor
>>>> entry [0x0185e000]
>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>          at
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>
>>>>
>>>>          at java.lang.System.checkIO(System.java:253)
>>>>          at java.lang.System.setErr(System.java:199)
>>>>          at
>>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>
>>>> "VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable
>>>>
>>>> "VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8
>>>> waiting on
>>>> condition
>>>>
>>>> JNI global references: 19
>>>>
>>>>
>>>> Found one Java-level deadlock:
>>>> =============================
>>>> "Thread-1":
>>>>    waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>>>    which is held by "main"
>>>> "main":
>>>>    waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
>>>> java.lang.Object),
>>>>    which is held by "Thread-1"
>>>>
>>>> Java stack information for the threads listed above:
>>>> ===================================================
>>>> "Thread-1":
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>
>>>>
>>>>          at
>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>>>
>>>>          at
>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>>>
>>>>
>>>>          at
>>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>          - locked <0x03f624b8> (a java.lang.Object)
>>>>          at
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>>>>
>>>>
>>>>          at
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>>>>
>>>>
>>>>          - locked <0x03ef8b30> (a
>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>          at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>
>>>>          at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>>>>
>>>>
>>>> ava:180)
>>>>          at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>>>>
>>>>
>>>> )
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>
>>>>
>>>>          at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>
>>>>
>>>>          at java.lang.Thread.run(Thread.java:744)
>>>> "main":
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>          at
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>          at
>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>
>>>>          at
>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>
>>>>
>>>>          at
>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>
>>>>
>>>>          at java.lang.System.checkIO(System.java:253)
>>>>          at java.lang.System.setErr(System.java:199)
>>>>          at
>>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>
>>>> Found 1 deadlock.
>>
>
>
> _______________________________________________
> Concurrency-interest mailing list
> [hidden email]
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: ClassLoader deadlock

Peter Levart


On 02/06/2016 10:32 PM, Peter wrote:
The 0x040ebee8 monitor is most likely being held by native code.

Regards,

Peter Firmstone.

Ok, but where? The deadlock report says it is held by main thread. Somewhere between the following two java frames?

"main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor entry [0x0185e000]
...
...
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
--here--
        at org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)


As I understand this is where JVM resolves some class that for the 1st time while ReferenceCollection.iterator is being executed. So before calling-back into ClassLoader.loadClass, it acquires a monitor lock on some int[] object?


Anyway. Looking at the following part of Thread-1 stacktrace:

        at org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
        at java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
        at java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
        at java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
        at java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
        at java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
        at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
        at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)

...I can see you are using some early release of JDK 8 where ThreadLocalRandom initialization involved obtaining a hardware address of some interface. This in turn checks for security manager. Since JDK 8u45 this code has been removed and only current time is used to derive initial seed, so you might not see this deadlock any more with JDK 8u45 and later.

Regards, Peter



On 7/02/2016 4:27 AM, Peter Levart wrote:


On 02/06/2016 01:17 PM, Peter Firmstone wrote:
The security manager is non blocking, unfortunately java system classes aren't.

It doesn't seem so. At least, I can't find where main thread locks the 0x040ebee8 monitor:

"Thread-1":
  waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
  which is held by "main"


"main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor entry [0x0185e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
        - waiting to lock <0x03f624b8> (a java.lang.Object)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
        at org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
        at org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
        at org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
        at org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
        at org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
        at org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
        at org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
        at org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
        at org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
        at org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
        at java.lang.System.checkIO(System.java:253)
        at java.lang.System.setErr(System.java:199)
        at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)


...no mention of 0x040ebee8 in main thread...?!

Peter


The policy provider in use thread confines PermissionCollection instances, which never leave the cpu cache, they are discarded immediately after use.

The problem here is that two different threads are attempting to load the same class at the same time.

I'll have to make sure that all classes are loaded before the security manager becomes the system security manager, either that or eliminate the AccessControlContext check permission cache.

While the standard java security manager and policy provider consume around 10% cpu, this consumes less than 1%.

On 6/02/2016 9:57 PM, David Holmes wrote:
On 6/02/2016 9:39 PM, Peter Firmstone wrote:
Hmm, thought the new parallel lock stategy in ClassLoader wasn't
deadlock prone?  Guess I was wrong.

The deadlock is introduced by a custom security manager. SM's play a critical role in many parts of the core libraries so if they utilize synchronization then they can easily introduce deadlock.

Observation:  On an unrelated occassion, I had a URLClassLoader
synchronization hotspot (well not standard URLClassLoader, but a high
performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
instead of URL DNS lookup), I solved that problem by thread confining
class loading.  I've never experienced deadlock using thread
confinement, initially I was worried that there would be
interdependencies, however I didn't experience a problem, as the single
thread would always load the classes it needed.

I'm wondering if multiple locks for ClassLoader's might be the wrong
strategy, I found thread confinement was very performant.

Are you suggesting that a class-load request is actually transmitted to a single class-loading thread, while the initial thread blocks until the loading is complete? That would still seem prone to the same SM related deadlock scenario. And it would not generally be particularly performant.

I thought that too, until I tried it.

Would you like to see some performance figures?

Cheers,

Peter.


David

Regards,

Peter.

2016-02-06 21:06:07
Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
runnable [0x00000000]
    java.lang.Thread.State: RUNNABLE

"C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
nid=0x111c waiting on condition [0x00000000]
    java.lang.Thread.State: RUNNABLE

"Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00 nid=0x13d8
waiting on condition [0x00000000]
    java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400 nid=0x10b8
runnable [0x00000000]
    java.lang.Thread.State: RUNNABLE

"Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
monitor entry [0x03ded000]
    java.lang.Thread.State: BLOCKED (on object monitor)
         at
org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)

         at
org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
         at
org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)

         at
org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
         at
org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)

         at
java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
         at
java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)

         at
java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
         at
java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)

         at
java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)

         at
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
         at
java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)

         at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
         - locked <0x03f624b8> (a java.lang.Object)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
         at
org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)

         at
org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)

         at
org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)

         - locked <0x03ef8b30> (a
org.cliffc.high_scale_lib.NonBlockingHashMap)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
         at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j

ava:180)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294

)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

         at java.lang.Thread.run(Thread.java:744)

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
Object.wait() [0x03cff000]
    java.lang.Thread.State: WAITING (on object monitor)
         at java.lang.Object.wait(Native Method)
         - waiting on <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
         - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
nid=0x1398 in Object.wait() [0x144cf000]
    java.lang.Thread.State: WAITING (on object monitor)
         at java.lang.Object.wait(Native Method)
         - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:502)
         at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
         - locked <0x03e05200> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor
entry [0x0185e000]
    java.lang.Thread.State: BLOCKED (on object monitor)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
         - waiting to lock <0x03f624b8> (a java.lang.Object)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
         at
org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)

         at
org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
         at
org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)

         at
org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
         at
org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)

         at java.lang.System.checkIO(System.java:253)
         at java.lang.System.setErr(System.java:199)
         at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)

"VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable

"VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8 waiting on
condition

JNI global references: 19


Found one Java-level deadlock:
=============================
"Thread-1":
   waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
   which is held by "main"
"main":
   waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
java.lang.Object),
   which is held by "Thread-1"

Java stack information for the threads listed above:
===================================================
"Thread-1":
         at
org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)

         at
org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
         at
org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)

         at
org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
         at
org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)

         at
java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
         at
java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)

         at
java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
         at
java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)

         at
java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)

         at
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
         at
java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)

         at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
         - locked <0x03f624b8> (a java.lang.Object)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
         at
org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)

         at
org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)

         at
org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)

         - locked <0x03ef8b30> (a
org.cliffc.high_scale_lib.NonBlockingHashMap)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
         at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j

ava:180)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294

)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

         at java.lang.Thread.run(Thread.java:744)
"main":
         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
         - waiting to lock <0x03f624b8> (a java.lang.Object)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
         at
org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)

         at
org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
         at
org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
         at
org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)

         at
org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)

         at
org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
         at
org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)

         at
org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)

         at java.lang.System.checkIO(System.java:253)
         at java.lang.System.setErr(System.java:199)
         at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)

Found 1 deadlock.



_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest



_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: ClassLoader deadlock

Peter
Thanks Peter & David,

That's good to know.

I've altered the SecurityManager to perform a permission check prior to
becoming the security manager, this ensures the cache has been created,
to avoid any recursive deadlock prone calls.

The class that both threads are attempting to load and init is called
org.apache.river.concurrent.ReferenceIterator.

The code can be found here:

https://github.com/pfirmstone/river-internet/tree/Input-validation-for-Serialization

Cheers,

Peter.


On 7/02/2016 6:37 PM, Peter Levart wrote:

>
>
> On 02/06/2016 10:32 PM, Peter wrote:
>> The 0x040ebee8 monitor is most likely being held by native code.
>>
>> Regards,
>>
>> Peter Firmstone.
>
> Ok, but where? The deadlock report says it is held by main thread.
> Somewhere between the following two java frames?
>
> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for
> monitor entry [0x0185e000]
> ...
> ...
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> --here--
>         at
> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>
>
> As I understand this is where JVM resolves some class that for the 1st
> time while ReferenceCollection.iterator is being executed. So before
> calling-back into ClassLoader.loadClass, it acquires a monitor lock on
> some int[] object?
>
>
> Anyway. Looking at the following part of Thread-1 stacktrace:
>
>         at
> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>         at
> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>         at
> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>         at
> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>         at
> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>         at
> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>         at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>         at
> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>
> ...I can see you are using some early release of JDK 8 where
> ThreadLocalRandom initialization involved obtaining a hardware address
> of some interface. This in turn checks for security manager. Since JDK
> 8u45 this code has been removed and only current time is used to
> derive initial seed, so you might not see this deadlock any more with
> JDK 8u45 and later.
>
> Regards, Peter
>
>>
>>
>> On 7/02/2016 4:27 AM, Peter Levart wrote:
>>>
>>>
>>> On 02/06/2016 01:17 PM, Peter Firmstone wrote:
>>>> The security manager is non blocking, unfortunately java system
>>>> classes aren't.
>>>
>>> It doesn't seem so. At least, I can't find where main thread locks
>>> the 0x040ebee8 monitor:
>>>
>>> "Thread-1":
>>>   waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>>   which is held by "main"
>>>
>>>
>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for
>>> monitor entry [0x0185e000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>         - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>         at
>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>         at
>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>         at
>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>         at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>         at
>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>         at
>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>         at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>         at
>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>         at
>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>         at
>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>         at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>         at
>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>         at java.lang.System.checkIO(System.java:253)
>>>         at java.lang.System.setErr(System.java:199)
>>>         at
>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>
>>>
>>> ...no mention of 0x040ebee8 in main thread...?!
>>>
>>> Peter
>>>
>>>>
>>>> The policy provider in use thread confines PermissionCollection
>>>> instances, which never leave the cpu cache, they are discarded
>>>> immediately after use.
>>>>
>>>> The problem here is that two different threads are attempting to
>>>> load the same class at the same time.
>>>>
>>>> I'll have to make sure that all classes are loaded before the
>>>> security manager becomes the system security manager, either that
>>>> or eliminate the AccessControlContext check permission cache.
>>>>
>>>> While the standard java security manager and policy provider
>>>> consume around 10% cpu, this consumes less than 1%.
>>>>
>>>> On 6/02/2016 9:57 PM, David Holmes wrote:
>>>>> On 6/02/2016 9:39 PM, Peter Firmstone wrote:
>>>>>> Hmm, thought the new parallel lock stategy in ClassLoader wasn't
>>>>>> deadlock prone?  Guess I was wrong.
>>>>>
>>>>> The deadlock is introduced by a custom security manager. SM's play
>>>>> a critical role in many parts of the core libraries so if they
>>>>> utilize synchronization then they can easily introduce deadlock.
>>>>>
>>>>>> Observation:  On an unrelated occassion, I had a URLClassLoader
>>>>>> synchronization hotspot (well not standard URLClassLoader, but a
>>>>>> high
>>>>>> performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
>>>>>> instead of URL DNS lookup), I solved that problem by thread
>>>>>> confining
>>>>>> class loading.  I've never experienced deadlock using thread
>>>>>> confinement, initially I was worried that there would be
>>>>>> interdependencies, however I didn't experience a problem, as the
>>>>>> single
>>>>>> thread would always load the classes it needed.
>>>>>>
>>>>>> I'm wondering if multiple locks for ClassLoader's might be the wrong
>>>>>> strategy, I found thread confinement was very performant.
>>>>>
>>>>> Are you suggesting that a class-load request is actually
>>>>> transmitted to a single class-loading thread, while the initial
>>>>> thread blocks until the loading is complete? That would still seem
>>>>> prone to the same SM related deadlock scenario. And it would not
>>>>> generally be particularly performant.
>>>>
>>>> I thought that too, until I tried it.
>>>>
>>>> Would you like to see some performance figures?
>>>>
>>>> Cheers,
>>>>
>>>> Peter.
>>>>
>>>>>
>>>>> David
>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Peter.
>>>>>>
>>>>>> 2016-02-06 21:06:07
>>>>>> Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):
>>>>>>
>>>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
>>>>>> runnable [0x00000000]
>>>>>>     java.lang.Thread.State: RUNNABLE
>>>>>>
>>>>>> "C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
>>>>>> nid=0x111c waiting on condition [0x00000000]
>>>>>>     java.lang.Thread.State: RUNNABLE
>>>>>>
>>>>>> "Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00
>>>>>> nid=0x13d8
>>>>>> waiting on condition [0x00000000]
>>>>>>     java.lang.Thread.State: RUNNABLE
>>>>>>
>>>>>> "Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400
>>>>>> nid=0x10b8
>>>>>> runnable [0x00000000]
>>>>>>     java.lang.Thread.State: RUNNABLE
>>>>>>
>>>>>> "Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
>>>>>> monitor entry [0x03ded000]
>>>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>>>          - locked <0x03f624b8> (a java.lang.Object)
>>>>>>          at
>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>          at
>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>>>>>>
>>>>>>
>>>>>>          - locked <0x03ef8b30> (a
>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>>>          at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>>>          at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>>>>>>
>>>>>>
>>>>>> ava:180)
>>>>>>          at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>>>>>>
>>>>>>
>>>>>> )
>>>>>>          at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>
>>>>>>
>>>>>>          at java.lang.Thread.run(Thread.java:744)
>>>>>>
>>>>>> "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
>>>>>> Object.wait() [0x03cff000]
>>>>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>>>>          at java.lang.Object.wait(Native Method)
>>>>>>          - waiting on <0x03e056d8> (a
>>>>>> java.lang.ref.ReferenceQueue$Lock)
>>>>>>          at
>>>>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
>>>>>>          - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>>>          at
>>>>>> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
>>>>>>          at
>>>>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>>
>>>>>> "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
>>>>>> nid=0x1398 in Object.wait() [0x144cf000]
>>>>>>     java.lang.Thread.State: WAITING (on object monitor)
>>>>>>          at java.lang.Object.wait(Native Method)
>>>>>>          - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>>>          at java.lang.Object.wait(Object.java:502)
>>>>>>          at
>>>>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>>>          - locked <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>>>
>>>>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for
>>>>>> monitor
>>>>>> entry [0x0185e000]
>>>>>>     java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>>>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>>>          at
>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>>>
>>>>>>
>>>>>>          at java.lang.System.checkIO(System.java:253)
>>>>>>          at java.lang.System.setErr(System.java:199)
>>>>>>          at
>>>>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>>>
>>>>>> "VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable
>>>>>>
>>>>>> "VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8
>>>>>> waiting on
>>>>>> condition
>>>>>>
>>>>>> JNI global references: 19
>>>>>>
>>>>>>
>>>>>> Found one Java-level deadlock:
>>>>>> =============================
>>>>>> "Thread-1":
>>>>>>    waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>>>>>    which is held by "main"
>>>>>> "main":
>>>>>>    waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
>>>>>> java.lang.Object),
>>>>>>    which is held by "Thread-1"
>>>>>>
>>>>>> Java stack information for the threads listed above:
>>>>>> ===================================================
>>>>>> "Thread-1":
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>>>          - locked <0x03f624b8> (a java.lang.Object)
>>>>>>          at
>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>          at
>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>>>>>>
>>>>>>
>>>>>>          - locked <0x03ef8b30> (a
>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>>>          at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>>>          at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>>>>>>
>>>>>>
>>>>>> ava:180)
>>>>>>          at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>>>>>>
>>>>>>
>>>>>> )
>>>>>>          at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>
>>>>>>
>>>>>>          at java.lang.Thread.run(Thread.java:744)
>>>>>> "main":
>>>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>>>          - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>>>          at
>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>>>
>>>>>>
>>>>>>          at
>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>>>
>>>>>>
>>>>>>          at java.lang.System.checkIO(System.java:253)
>>>>>>          at java.lang.System.setErr(System.java:199)
>>>>>>          at
>>>>>> org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>>>
>>>>>> Found 1 deadlock.
>>>>
>>>
>>>
>>> _______________________________________________
>>> Concurrency-interest mailing list
>>> [hidden email]
>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>
>

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: ClassLoader deadlock

Mandy Chung
Out of curiousity, what class loader is using to define org.apache.river.api.security.CombinerSecurityManager?

Mandy

> On Feb 7, 2016, at 1:54 AM, Peter <[hidden email]> wrote:
>
> Thanks Peter & David,
>
> That's good to know.
>
> I've altered the SecurityManager to perform a permission check prior to becoming the security manager, this ensures the cache has been created, to avoid any recursive deadlock prone calls.
>
> The class that both threads are attempting to load and init is called org.apache.river.concurrent.ReferenceIterator.
>
> The code can be found here:
>
> https://github.com/pfirmstone/river-internet/tree/Input-validation-for-Serialization
>
> Cheers,
>
> Peter.
>
>
> On 7/02/2016 6:37 PM, Peter Levart wrote:
>>
>>
>> On 02/06/2016 10:32 PM, Peter wrote:
>>> The 0x040ebee8 monitor is most likely being held by native code.
>>>
>>> Regards,
>>>
>>> Peter Firmstone.
>>
>> Ok, but where? The deadlock report says it is held by main thread. Somewhere between the following two java frames?
>>
>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor entry [0x0185e000]
>> ...
>> ...
>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> --here--
>>        at org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>
>>
>> As I understand this is where JVM resolves some class that for the 1st time while ReferenceCollection.iterator is being executed. So before calling-back into ClassLoader.loadClass, it acquires a monitor lock on some int[] object?
>>
>>
>> Anyway. Looking at the following part of Thread-1 stacktrace:
>>
>>        at org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>        at java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>        at java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>        at java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>        at java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>        at java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>        at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>        at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>
>> ...I can see you are using some early release of JDK 8 where ThreadLocalRandom initialization involved obtaining a hardware address of some interface. This in turn checks for security manager. Since JDK 8u45 this code has been removed and only current time is used to derive initial seed, so you might not see this deadlock any more with JDK 8u45 and later.
>>
>> Regards, Peter
>>
>>>
>>>
>>> On 7/02/2016 4:27 AM, Peter Levart wrote:
>>>>
>>>>
>>>> On 02/06/2016 01:17 PM, Peter Firmstone wrote:
>>>>> The security manager is non blocking, unfortunately java system classes aren't.
>>>>
>>>> It doesn't seem so. At least, I can't find where main thread locks the 0x040ebee8 monitor:
>>>>
>>>> "Thread-1":
>>>>  waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>>>  which is held by "main"
>>>>
>>>>
>>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor entry [0x0185e000]
>>>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>        - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>        at org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>        at org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>        at org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>        at org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>        at org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>        at org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>        at org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>        at org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>        at org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>        at org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>        at org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>        at java.lang.System.checkIO(System.java:253)
>>>>        at java.lang.System.setErr(System.java:199)
>>>>        at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>
>>>>
>>>> ...no mention of 0x040ebee8 in main thread...?!
>>>>
>>>> Peter
>>>>
>>>>>
>>>>> The policy provider in use thread confines PermissionCollection instances, which never leave the cpu cache, they are discarded immediately after use.
>>>>>
>>>>> The problem here is that two different threads are attempting to load the same class at the same time.
>>>>>
>>>>> I'll have to make sure that all classes are loaded before the security manager becomes the system security manager, either that or eliminate the AccessControlContext check permission cache.
>>>>>
>>>>> While the standard java security manager and policy provider consume around 10% cpu, this consumes less than 1%.
>>>>>
>>>>> On 6/02/2016 9:57 PM, David Holmes wrote:
>>>>>> On 6/02/2016 9:39 PM, Peter Firmstone wrote:
>>>>>>> Hmm, thought the new parallel lock stategy in ClassLoader wasn't
>>>>>>> deadlock prone?  Guess I was wrong.
>>>>>>
>>>>>> The deadlock is introduced by a custom security manager. SM's play a critical role in many parts of the core libraries so if they utilize synchronization then they can easily introduce deadlock.
>>>>>>
>>>>>>> Observation:  On an unrelated occassion, I had a URLClassLoader
>>>>>>> synchronization hotspot (well not standard URLClassLoader, but a high
>>>>>>> performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
>>>>>>> instead of URL DNS lookup), I solved that problem by thread confining
>>>>>>> class loading.  I've never experienced deadlock using thread
>>>>>>> confinement, initially I was worried that there would be
>>>>>>> interdependencies, however I didn't experience a problem, as the single
>>>>>>> thread would always load the classes it needed.
>>>>>>>
>>>>>>> I'm wondering if multiple locks for ClassLoader's might be the wrong
>>>>>>> strategy, I found thread confinement was very performant.
>>>>>>
>>>>>> Are you suggesting that a class-load request is actually transmitted to a single class-loading thread, while the initial thread blocks until the loading is complete? That would still seem prone to the same SM related deadlock scenario. And it would not generally be particularly performant.
>>>>>
>>>>> I thought that too, until I tried it.
>>>>>
>>>>> Would you like to see some performance figures?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Peter.
>>>>>
>>>>>>
>>>>>> David
>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Peter.
>>>>>>>
>>>>>>> 2016-02-06 21:06:07
>>>>>>> Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):
>>>>>>>
>>>>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
>>>>>>> runnable [0x00000000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>
>>>>>>> "C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
>>>>>>> nid=0x111c waiting on condition [0x00000000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>
>>>>>>> "Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00 nid=0x13d8
>>>>>>> waiting on condition [0x00000000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>
>>>>>>> "Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400 nid=0x10b8
>>>>>>> runnable [0x00000000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>
>>>>>>> "Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
>>>>>>> monitor entry [0x03ded000]
>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>>>>
>>>>>>>         at
>>>>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>>>>>>
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>>>>>>
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>>>>>>
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>>>>>>
>>>>>>>         at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>>>>         - locked <0x03f624b8> (a java.lang.Object)
>>>>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>         at
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>>>>>>>
>>>>>>>         at
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>>>>>>>
>>>>>>>         - locked <0x03ef8b30> (a
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>>>>         at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>         at
>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>>>>         at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>>>>>>>
>>>>>>> ava:180)
>>>>>>>         at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>>>>>>>
>>>>>>> )
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>>>
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>
>>>>>>>         at java.lang.Thread.run(Thread.java:744)
>>>>>>>
>>>>>>> "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
>>>>>>> Object.wait() [0x03cff000]
>>>>>>>    java.lang.Thread.State: WAITING (on object monitor)
>>>>>>>         at java.lang.Object.wait(Native Method)
>>>>>>>         - waiting on <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>>>>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
>>>>>>>         - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>>>>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
>>>>>>>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>>>
>>>>>>> "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
>>>>>>> nid=0x1398 in Object.wait() [0x144cf000]
>>>>>>>    java.lang.Thread.State: WAITING (on object monitor)
>>>>>>>         at java.lang.Object.wait(Native Method)
>>>>>>>         - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>>>>         at java.lang.Object.wait(Object.java:502)
>>>>>>>         at
>>>>>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>>>>         - locked <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>>>>
>>>>>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor
>>>>>>> entry [0x0185e000]
>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>>>>         - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>>>>
>>>>>>>         at java.lang.System.checkIO(System.java:253)
>>>>>>>         at java.lang.System.setErr(System.java:199)
>>>>>>>         at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>>>>
>>>>>>> "VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable
>>>>>>>
>>>>>>> "VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8 waiting on
>>>>>>> condition
>>>>>>>
>>>>>>> JNI global references: 19
>>>>>>>
>>>>>>>
>>>>>>> Found one Java-level deadlock:
>>>>>>> =============================
>>>>>>> "Thread-1":
>>>>>>>   waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>>>>>>   which is held by "main"
>>>>>>> "main":
>>>>>>>   waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
>>>>>>> java.lang.Object),
>>>>>>>   which is held by "Thread-1"
>>>>>>>
>>>>>>> Java stack information for the threads listed above:
>>>>>>> ===================================================
>>>>>>> "Thread-1":
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>>>>
>>>>>>>         at
>>>>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>>>>>>
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>>>>>>
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>>>>>>
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>>>>>>>
>>>>>>>         at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>>>>         - locked <0x03f624b8> (a java.lang.Object)
>>>>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>         at
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167)
>>>>>>>
>>>>>>>         at
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166)
>>>>>>>
>>>>>>>         - locked <0x03ef8b30> (a
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>>>>         at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>         at
>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>>>>         at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j
>>>>>>>
>>>>>>> ava:180)
>>>>>>>         at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294
>>>>>>>
>>>>>>> )
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>>>
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>
>>>>>>>         at java.lang.Thread.run(Thread.java:744)
>>>>>>> "main":
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>>>>         - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>>>>
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>>>>
>>>>>>>         at java.lang.System.checkIO(System.java:253)
>>>>>>>         at java.lang.System.setErr(System.java:199)
>>>>>>>         at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>>>>
>>>>>>> Found 1 deadlock.
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Concurrency-interest mailing list
>>>> [hidden email]
>>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>
>>
>
> _______________________________________________
> Concurrency-interest mailing list
> [hidden email]
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: ClassLoader deadlock

Peter
In reply to this post by Peter Levart
The application ClassLoader.

It's loaded using the -Djava.security.manager argument.

The policy is loaded by the extension class loader for historical reasons, with the extension directory argument.

Regards,

Peter.



Sent from my Samsung device.
 
---- Original message ----
From: Mandy Chung <[hidden email]>
Sent: 09/02/2016 08:14:39 am
To: Peter <[hidden email]>
Cc: Mandy Chung <[hidden email]>; Peter Levart <[hidden email]>; core-libs-dev <[hidden email]>; David Holmes <[hidden email]>; [hidden email]
Subject: Re: [concurrency-interest] ClassLoader deadlock

Out of curiousity, what class loader is using to define org.apache.river.api.security.CombinerSecurityManager?

Mandy

> On Feb 7, 2016, at 1:54 AM, Peter <[hidden email]> wrote:

> Thanks Peter & David,

> That's good to know.

> I've altered the SecurityManager to perform a permission check prior to becoming the security manager, this ensures the cache has been created, to avoid any recursive deadlock prone calls.

> The class that both threads are attempting to load and init is called org.apache.river.concurrent.ReferenceIterator.

> The code can be found here:

> https://github.com/pfirmstone/river-internet/tree/Input-validation-for-Serialization

> Cheers,

> Peter.


> On 7/02/2016 6:37 PM, Peter Levart wrote:
>> 
>> 
>> On 02/06/2016 10:32 PM, Peter wrote:
>>> The 0x040ebee8 monitor is most likely being held by native code.
>>> 
>>> Regards,
>>> 
>>> Peter Firmstone.
>> 
>> Ok, but where? The deadlock report says it is held by main thread. Somewhere between the following two java frames?
>> 
>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor entry [0x0185e000]
>> ...
>> ...
>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> --here--
>>        at org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>> 
>> 
>> As I understand this is where JVM resolves some class that for the 1st time while ReferenceCollection.iterator is being executed. So before calling-back into ClassLoader.loadClass, it acquires a monitor lock on some int[] object?
>> 
>> 
>> Anyway. Looking at the following part of Thread-1 stacktrace:
>> 
>>        at org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>        at java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447)
>>        at java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158)
>>        at java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137)
>>        at java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526)
>>        at java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266)
>>        at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
>>        at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
>> 
>> ...I can see you are using some early release of JDK 8 where ThreadLocalRandom initialization involved obtaining a hardware address of some interface. This in turn checks for security manager. Since JDK 8u45 this code has been removed and only current time is used to derive initial seed, so you might not see this deadlock any more with JDK 8u45 and later.
>> 
>> Regards, Peter
>> 
>>> 
>>> 
>>> On 7/02/2016 4:27 AM, Peter Levart wrote:
>>>> 
>>>> 
>>>> On 02/06/2016 01:17 PM, Peter Firmstone wrote:
>>>>> The security manager is non blocking, unfortunately java system classes aren't.
>>>> 
>>>> It doesn't seem so. At least, I can't find where main thread locks the 0x040ebee8 monitor:
>>>> 
>>>> "Thread-1":
>>>>  waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>>>  which is held by "main"
>>>> 
>>>> 
>>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor entry [0x0185e000]
>>>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>        - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>        at org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124)
>>>>        at org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65)
>>>>        at org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44)
>>>>        at org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57)
>>>>        at org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64)
>>>>        at org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128)
>>>>        at org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44)
>>>>        at org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244)
>>>>        at org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68)
>>>>        at org.apache.river.api.securityCombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261)
>>>>        at org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202)
>>>>        at java.lang.System.checkIO(System.java:253)
>>>>        at java.lang.System.setErr(System.java:199)
>>>>        at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>> 
>>>> 
>>>> ...no mention of 0x040ebee8 in main thread...?!
>>>> 
>>>> Peter
>>>> 
>>>>> 
>>>>> The policy provider in use thread confines PermissionCollection instances, which never leave the cpu cache, they are discarded immediately after use.
>>>>> 
>>>>> The problem here is that two different threads are attempting to load the same class at the same time.
>>>>> 
>>>>> I'll have to make sure that all classes are loaded before the security manager becomes the system security manager, either that or eliminate the AccessControlContext check permission cache
>>>>> 
>>>>> While the standard java security manager and policy provider consume around 10% cpu, this consumes less than 1%.
>>>>> 
>>>>> On 6/02/2016 9:57 PM, David Holmes wrote:
>>>>>> On 6/02/2016 9:39 PM, Peter Firmstone wrote:
>>>>>>> Hmm, thought the new parallel lock stategy in ClassLoader wasn't
>>>>>>> deadlock prone?  Guess I was wrong.
>>>>>> 
>>>>>> The deadlock is introduced by a custom security manager. SM's play a critical role in many parts of the core libraries so if they utilize synchronization then they can easily introduce deadlock
>>>>>> 
>>>>>>> Observation:  On an unrelated occassion, I had a URLClassLoader
>>>>>>> synchronization hotspot (well not standard URLClassLoader, but a high
>>>>>>> performance RFC3986 URL ClassLoader, that use normalized RFC3986 URI
>>>>>>> instead of URL DNS lookup), I solved that problem by thread confining
>>>>>>> class loading.  I've never experienced deadlock using thread
>>>>>>> confinement, initially I was worried that there would be
>>>>>>> interdependencies, however I didn't experience a problem, as the single
>>>>>>> thread would always load the classes it needed.
>>>>>>> 
>>>>>>> I'm wondering if multiple locks for ClassLoader's might be the wrong
>>>>>>> strategy, I found thread confinement was very performant.
>>>>>> 
>>>>>> Are you suggesting that a class-load request is actually transmitted to a single class-loading thread, while the initial thread blocks until the loading is complete? That would still seem prone to the same SM related deadlock scenario. And it would not generally be particularly performant.
>>>>> 
>>>>> I thought that too, until I tried it.
>>>>> 
>>>>> Would you like to see some performance figures?
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> Peter.
>>>>> 
>>>>>> 
>>>>>> David
>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Peter.
>>>>>>> 
>>>>>>> 2016-02-06 21:06:07
>>>>>>> Full thread dump Java HotSpot(TM) Client VM (25.0-b70 mixed mode):
>>>>>>> 
>>>>>>> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x14387400 nid=0xc54
>>>>>>> runnable [0x00000000]
>>>>>>>    javalang.Thread.State: RUNNABLE
>>>>>>> 
>>>>>>> "C1 CompilerThread0" #8 daemon prio=9 os_prio=2 tid=0x14332c00
>>>>>>> nid=0x111c waiting on condition [0x00000000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> 
>>>>>>> "Attach Listener" #7 daemon prio=5 os_prio=2 tid=0x14331c00 nid=0x13d8
>>>>>>> waiting on condition [0x00000000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> 
>>>>>>> "Signal Dispatcher" #6 daemon prio=9 os_prio=2 tid=0x14331400 nid=0x10b8
>>>>>>> runnable [0x00000000]
>>>>>>>    java.lang.ThreadState: RUNNABLE
>>>>>>> 
>>>>>>> "Thread-1" #5 prio=10 os_prio=2 tid=0x14318800 nid=0x4a8 waiting for
>>>>>>> monitor entry [0x03ded000]
>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrentReferenceSet.hashCode(ReferenceSet.java:65) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference<init>(StrongReference.java:44) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>>>>> 
>>>>>>>         at
>>>>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447) 
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158) 
>>>>>>> 
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137) 
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526) 
>>>>>>> 
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266) 
>>>>>>> 
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070) 
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535) 
>>>>>>> 
>>>>>>>         at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>>>>         - locked <0x03f624b8> (a java.lang.Object)
>>>>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>         at
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166) 
>>>>>>> 
>>>>>>>         - locked <0x03ef8b30> (a
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>>>>         at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
>>>>>>>         at
>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>>>>         at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j 
>>>>>>> 
>>>>>>> ava:180)
>>>>>>>         at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294 
>>>>>>> 
>>>>>>> )
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
>>>>>>> 
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
>>>>>>> 
>>>>>>>         at java.lang.Thread.run(Thread.java:744)
>>>>>>> 
>>>>>>> "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x14277800 nid=0x15cc in
>>>>>>> Object.wait() [0x03cff000]
>>>>>>>    java.lang.Thread.State: WAITING (on object monitor)
>>>>>>>         at java.lang.Object.wait(Native Method)
>>>>>>>         - waiting on <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>>>>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
>>>>>>>         - locked <0x03e056d8> (a java.lang.ref.ReferenceQueue$Lock)
>>>>>>>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
>>>>>>>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>>>>>> 
>>>>>>> "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x14271c00
>>>>>>> nid=0x1398 in Object.wait() [0x144cf000]
>>>>>>>    java.lang.Thread.State: WAITING (on object monitor)
>>>>>>>         at javalang.Object.wait(Native Method)
>>>>>>>         - waiting on <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>>>>         at java.lang.Object.wait(Object.java:502)
>>>>>>>         at
>>>>>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>>>>>>>         - locked <0x03e05200> (a java.lang.ref.Reference$Lock)
>>>>>>> 
>>>>>>> "main" #1 prio=5 os_prio=0 tid=0x017cf400 nid=0x1284 waiting for monitor
>>>>>>> entry [0x0185e000]
>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
>>>>>>>         - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrentReferenceSet.hashCode(ReferenceSet.java:65) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference<init>(StrongReference.java:44) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.riverconcurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>>>>> 
>>>>>>>         at java.lang.System.checkIO(System.java:253)
>>>>>>>         at java.lang.System.setErr(System.java:199)
>>>>>>>         at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>>>> 
>>>>>>> "VM Thread" os_prio=2 tid=0x1426e400 nid=0x16a8 runnable
>>>>>>> 
>>>>>>> "VM Periodic Task Thread" os_prio=2 tid=0x14388400 nid=0x17a8 waiting on
>>>>>>> condition
>>>>>>> 
>>>>>>> JNI global references: 19
>>>>>>> 
>>>>>>> 
>>>>>>> Found one Java-level deadlock:
>>>>>>> =============================
>>>>>>> "Thread-1":
>>>>>>>   waiting to lock monitor 0x142766ac (object 0x040ebee8, a [I),
>>>>>>>   which is held by "main"
>>>>>>> "main":
>>>>>>>   waiting to lock monitor 0x14274a3c (object 0x03f624b8, a
>>>>>>> java.lang.Object),
>>>>>>>   which is held by "Thread-1"
>>>>>>> 
>>>>>>> Java stack information for the threads listed above:
>>>>>>> ===================================================
>>>>>>> "Thread-1":
>>>>>>>         at
>>>>>>> orgapache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrentReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>>>>> 
>>>>>>>         at
>>>>>>> java.net.NetworkInterface.getHardwareAddress(NetworkInterface.java:447) 
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadLocalRandom.initialSeed(ThreadLocalRandom.java:158) 
>>>>>>> 
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadLocalRandom.<clinit>(ThreadLocalRandom.java:137) 
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.fullAddCount(ConcurrentHashMap.java:2526) 
>>>>>>> 
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2266) 
>>>>>>> 
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070) 
>>>>>>>         at
>>>>>>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535) 
>>>>>>> 
>>>>>>>         at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>>>>>>>         - locked <0x03f624b8> (a java.lang.Object)
>>>>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>         at
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$SnapshotK.<init>(NonBlockingHashMap.java:1167) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap$2.iterator(NonBlockingHashMap.java:1200) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor$EnqueGarbageTask.run(ReferenceProcessor.java:166) 
>>>>>>> 
>>>>>>>         - locked <0x03ef8b30> (a
>>>>>>> org.cliffc.high_scale_lib.NonBlockingHashMap)
>>>>>>>         at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
>>>>>>>         at
>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>>>>         at
>>>>>>> java.utilconcurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.j 
>>>>>>> 
>>>>>>> ava:180)
>>>>>>>         at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294 
>>>>>>> 
>>>>>>> )
>>>>>>>         at
>>>>>>> java.utilconcurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
>>>>>>> 
>>>>>>>         at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
>>>>>>> 
>>>>>>>         at java.lang.Thread.run(Thread.java:744)
>>>>>>> "main":
>>>>>>>         at java.lang.ClassLoaderloadClass(ClassLoader.java:406)
>>>>>>>         - waiting to lock <0x03f624b8> (a java.lang.Object)
>>>>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceCollection.iterator(ReferenceCollection.java:124) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceSet.hashCode(ReferenceSet.java:65) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:44) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.StrongReference.<init>(StrongReference.java:57) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceFactory.create(ReferenceFactory.java:64) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:128) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceProcessor.referenced(ReferenceProcessor.java:44) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceMap.wrapVal(ReferenceMap.java:244) 
>>>>>>>         at
>>>>>>> org.apache.river.concurrent.ReferenceConcurrentMap.putIfAbsent(ReferenceConcurrentMap.java:68) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:261) 
>>>>>>> 
>>>>>>>         at
>>>>>>> org.apache.river.api.security.CombinerSecurityManager.checkPermission(CombinerSecurityManager.java:202) 
>>>>>>> 
>>>>>>>         at java.lang.System.checkIO(System.java:253)
>>>>>>>         at java.lang.System.setErr(System.java:199)
>>>>>>>         at org.apache.river.qa.harness.MasterTest.main(MasterTest.java:84)
>>>>>>> 
>>>>>>> Found 1 deadlock.
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Concurrency-interest mailing list
>>>> [hidden email]
>>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>> 
>> 

> _______________________________________________
> Concurrency-interest mailing list
[hidden email]
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest



_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest