multi JVM on a node / JVM in a container

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

multi JVM on a node / JVM in a container

Ken Sipe
I’m looking into the consequence of a JRE sharing a multi-core node but not owning the full resource space.  

GIVEN:  A node with 16 cores.

SCENARIO 1:  4 JVMs

All JREs will see 16 cores and have an internal setup for 16 cores… 16 GC threads for old parallel gc.  The target parallelism level of fork join, etc.  However with 4 JREs running their processes will be scheduled and prioritized such that under idealistic conditions they “realize” 4 cores of time each.   Questions that come to mind.

1. Does this have a negative consequence?  such as the time to safe point?
2. Any other consequences?
3. Any recommendations to reduce any negative impact?  


SCENARIO 2:   the REAL reason for the question… 1 JVM in a container under cgroup CPU-Shares with a weighted value of 4 cores

I’m troubled with the fact that cpu-share (vs cpu-sets) is the common / default option for containers.   The result is a situation where in a heterogeneous datacenter where my container lands (16 core, 32 core, 4 core) setups on the JVM with different levels of efficiencies.   It’s been on my mind for over 1 year old and I would love to discuss it with this group.

SCENARO 3: 1 JVM in a container with CPU-Sets set to 1 core

Now the JRE “sees” only 1 core regardless of node it lands on… and is setup for client mode… awesome! NOT.

Thoughts?

ken
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: multi JVM on a node / JVM in a container

David Holmes-6
Ken Sipe writes:

> I�m looking into the consequence of a JRE sharing a multi-core node but not
> owning the full resource space.
>
> GIVEN:  A node with 16 cores.
>
> SCENARIO 1:  4 JVMs
>
> All JREs will see 16 cores and have an internal setup for 16 cores� 16 GC
> threads for old parallel gc.  The target parallelism level of fork join, etc.
> However with 4 JREs running their processes will be scheduled and prioritized
> such that under idealistic conditions they �realize� 4 cores of time each.
> Questions that come to mind.
>
> 1. Does this have a negative consequence?  such as the time to safe point?
> 2. Any other consequences?
> 3. Any recommendations to reduce any negative impact?

If each JVM would quite fully utilize the entire machine then obviously have four run concurrently will effectively overload the machine.

Ergonomic selection assumes an ideal world and what the VM sees by way of resources is what it can expect to have. In this scenario where you know you are deploying multiple JVMs onto shared resources you should configure each appropriately - number of GC threads, number of compiler threads, and limit the maximum concurrency of the default FJPool.
 
>
> SCENARIO 2:   the REAL reason for the question� 1 JVM in a container under
> cgroup CPU-Shares with a weighted value of 4 cores
>
> I�m troubled with the fact that cpu-share (vs cpu-sets) is the common /
> default option for containers.   The result is a situation where in a
> heterogeneous datacenter where my container lands (16 core, 32 core, 4
> core) setups on the JVM with different levels of efficiencies.   It�s been on
> my mind for over 1 year old and I would love to discuss it with this group.

I don't really see any way to deal with this. You want the JVM to utilize all available processors, even if the "share" on any given processor is limited.
 
> SCENARO 3: 1 JVM in a container with CPU-Sets set to 1 core
>
> Now the JRE �sees� only 1 core regardless of node it lands on� and is setup
> for client mode� awesome! NOT.

Again you need to override the ergonomic selection - either with command-line options or environment variables. But running a server VM one a 1 processor system is not going to be awesome either.
 
> Thoughts?

It is a problem trying to balance overall productivity, with transparent operation when running in resource constrained environments. It is only the latest JDK 9 that will even see cpusets on Linux. And dealing with memory constraints in containers is still an unresolved issue. When the environment lies to the VM about what is available it makes it very hard for the VM to try to adjust.

Cheers,
David

>
> ken
> _______________________________________________
> Concurrency-interest mailing list
> [hidden email]
> http://cs.oswego.edu/mailman/listinfo/concurrency-interest


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: multi JVM on a node / JVM in a container

Ken Sipe
thx david.. this is what I was expecting… checking to see I wasn’t missing something.  more notes below…


On Mar 15, 2016, at 3:47 PM, David Holmes <[hidden email]> wrote:

Ken Sipe writes:
I�m looking into the consequence of a JRE sharing a multi-core node but not
owning the full resource space.

GIVEN:  A node with 16 cores.

SCENARIO 1:  4 JVMs

All JREs will see 16 cores and have an internal setup for 16 cores� 16 GC
threads for old parallel gc.  The target parallelism level of fork join, etc.
However with 4 JREs running their processes will be scheduled and prioritized
such that under idealistic conditions they �realize� 4 cores of time each.
Questions that come to mind.

1. Does this have a negative consequence?  such as the time to safe point?
2. Any other consequences?
3. Any recommendations to reduce any negative impact?

If each JVM would quite fully utilize the entire machine then obviously have four run concurrently will effectively overload the machine.

Ergonomic selection assumes an ideal world and what the VM sees by way of resources is what it can expect to have. In this scenario where you know you are deploying multiple JVMs onto shared resources you should configure each appropriately - number of GC threads, number of compiler threads, and limit the maximum concurrency of the default FJPool.


SCENARIO 2:   the REAL reason for the question� 1 JVM in a container under
cgroup CPU-Shares with a weighted value of 4 cores

I�m troubled with the fact that cpu-share (vs cpu-sets) is the common /
default option for containers.   The result is a situation where in a
heterogeneous datacenter where my container lands (16 core, 32 core, 4
core) setups on the JVM with different levels of efficiencies.   It�s been on
my mind for over 1 year old and I would love to discuss it with this group.

I don't really see any way to deal with this. You want the JVM to utilize all available processors, even if the "share" on any given processor is limited.

SCENARO 3: 1 JVM in a container with CPU-Sets set to 1 core

Now the JRE �sees� only 1 core regardless of node it lands on� and is setup
for client mode� awesome! NOT.

Again you need to override the ergonomic selection - either with command-line options or environment variables. But running a server VM one a 1 processor system is not going to be awesome either.

Thoughts?

It is a problem trying to balance overall productivity, with transparent operation when running in resource constrained environments. It is only the latest JDK 9 that will even see cpusets on Linux. And dealing with memory constraints in containers is still an unresolved issue.

I’m not sure what you mean here… The JRE 7/8 runtime “sees” all cores in shares mode and just assigned number of cores in set mode.  I’m not familiar with the changes in 9.. but very interested.

When the environment lies to the VM about what is available it makes it very hard for the VM to try to adjust.

This is my favorite part of your request…  the JVM expects to “own” the platform.   The world of lies is spiking.   micro-services + containers + JVM are the new norm.  The problem is few understand the consequences.     We can expect that the environment the JVM is running on is not detectable in the “old” ways.   We need a way to provide platform “profiles” to the JVM.  The oracle java team is very against setting every effected flag for the environment (lessons learned from the Sun days).   The only alternative is to provide a mechanism to inform the JVM of the truth.


Cheers,
David


ken
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: multi JVM on a node / JVM in a container

David Holmes-6
Ken Sipe writes:

> thx david.. this is what I was expecting… checking to see I wasn’t missing something.  more notes below…
>
>
> On Mar 15, 2016, at 3:47 PM, David Holmes <mailto:[hidden email]> wrote:
>>
>> Ken Sipe writes:
>>>
>>> I�m looking into the consequence of a JRE sharing a multi-core node but not owning the full resource space.
>>>
>>> GIVEN:  A node with 16 cores.
>>>
>>> SCENARIO 1:  4 JVMs
>>>
>>> All JREs will see 16 cores and have an internal setup for 16 cores� 16 GC threads for old parallel gc.  The target parallelism level of fork join, etc.
>>> However with 4 JREs running their processes will be scheduled and prioritized such that under idealistic conditions they �realize� 4 cores of time >>> each. Questions that come to mind.
>>>
>>> 1. Does this have a negative consequence?  such as the time to safe point?
>>> 2. Any other consequences?
>>> 3. Any recommendations to reduce any negative impact?
>>>
>> If each JVM would quite fully utilize the entire machine then obviously have four run concurrently will effectively overload the machine.
>>
>> Ergonomic selection assumes an ideal world and what the VM sees by way of resources is what it can expect to have. In this scenario where you
>> know you are deploying multiple JVMs onto shared resources you should configure each appropriately - number of GC threads, number of compiler
>> threads, and limit the maximum concurrency of the default FJPool.
>>
>>
>>> SCENARIO 2:   the REAL reason for the question� 1 JVM in a container under cgroup CPU-Shares with a weighted value of 4 cores
>>>
>>> I�m troubled with the fact that cpu-share (vs cpu-sets) is the common /default option for containers.   The result is a situation where in a
>>> heterogeneous datacenter where my container lands (16 core, 32 core, 4 core) setups on the JVM with different levels of efficiencies.  
>>> It�s been on my mind for over 1 year old and I would love to discuss it with this group.
>>>
>> I don't really see any way to deal with this. You want the JVM to utilize all available processors, even if the "share" on any given processor is limited.
>>
>>>
>>> SCENARO 3: 1 JVM in a container with CPU-Sets set to 1 core
>>>
>>> Now the JRE �sees� only 1 core regardless of node it lands on� and is setup for client mode� awesome! NOT.
>>>
>> Again you need to override the ergonomic selection - either with command-line options or environment variables. But running a server VM one a 1 processor system is not going to be awesome either.
>>
>>>  Thoughts?
>>
>> It is a problem trying to balance overall productivity, with transparent operation when running in resource constrained environments. It is only the
>> latest JDK 9 that will even see cpusets on Linux. And dealing with memory constraints in containers is still an unresolved issue.
>
> I’m not sure what you mean here… The JRE 7/8 runtime “sees” all cores in shares mode and just assigned number of cores in set mode.  I’m not
> familiar with the changes in 9.. but very interested.

On Linux, JDK 7, 8 and until recently 9, will report the number of  online processors as reported by sysconf. That doesn't account for cpusets reducing the number of online processors that are actually available for the JVM to use. Now in 9 we use sched_getaffinity, which does account for cpusets.

>> When the environment lies to the VM about what is available it makes it very hard for the VM to try to adjust.
>>
> This is my favorite part of your request…  the JVM expects to “own” the platform.   The world of lies is spiking.   micro-services + containers + JVM are
> the new norm.  The problem is few understand the consequences.     We can expect that the environment the JVM is running on is not detectable in
> the “old” ways.   We need a way to provide platform “profiles” to the JVM.  The oracle java team is very against setting every effected flag for the
> environment (lessons learned from the Sun days).   The only alternative is to provide a mechanism to inform the JVM of the truth.

The interaction between containers and the underlying OS and the impact on API's exposed by the OS has not been clearly thought out in my opinion in many cases. Solaris zones at least look like a separate OS instance and all the API's report the right values. But Solaris also has resource pools, and similar to Linux cgroups, these don't modify (in general) what the OS API's report - so the VM has to become aware of these specialized environments and that is very painful. We resisted making the JVM resource pool aware on Solaris and that turned out to be a win because Zones became dominant. But now we face a similar problem with cgroups on Linux and environments like Docker. We were lucky that cpusets are reflected in the output of sched_getaffinity - a nice simple change. But memory constraints are far more problematic and may not even be queryable in general. If there are no API's to tell the VM the real resource story what is the VM supposed to do? I don't have any answers to that.

Cheers,
David


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest