For those of us who bench on Linux

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

For those of us who bench on Linux

Viktor Klang

TL;DR: problems found and fixed with Linux scheduler affecting multicore systems.

Story & paper: https://blog.acolyer.org/2016/04/26/the-linux-scheduler-a-decade-of-wasted-cores/

Code: https://github.com/jplozi/wastedcores

Would love to hear your thoughts.

--
Cheers,


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: For those of us who bench on Linux

David Holmes-6

Very interesting. In particular that scheduling affinity (to take advantage of a warm cache) can be a performance hit.

 

Thanks for the link.

 

David

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Viktor Klang
Sent: Wednesday, April 27, 2016 5:24 PM
To: concurrency-interest <[hidden email]>
Subject: [concurrency-interest] For those of us who bench on Linux

 

TL;DR: problems found and fixed with Linux scheduler affecting multicore systems.

Story & paper: https://blog.acolyer.org/2016/04/26/the-linux-scheduler-a-decade-of-wasted-cores/

Code: https://github.com/jplozi/wastedcores

Would love to hear your thoughts.

--
Cheers,


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: For those of us who bench on Linux

Gregg Wonderly-3
In reply to this post by Viktor Klang
There are so many pieces of the processor usability visible to software.  The optimizations that used to be simple are now very complex to get right.  It would be great to see the processors managing cores as virtual entities to do power management for us.

Gregg

Sent from my iPhone

On Apr 27, 2016, at 3:50 AM, David Holmes <[hidden email]> wrote:

Very interesting. In particular that scheduling affinity (to take advantage of a warm cache) can be a performance hit.

 

Thanks for the link.

 

David

 

From: [hidden email] [[hidden email]] On Behalf Of Viktor Klang
Sent: Wednesday, April 27, 2016 5:24 PM
To: concurrency-interest <[hidden email]>
Subject: [concurrency-interest] For those of us who bench on Linux

 

TL;DR: problems found and fixed with Linux scheduler affecting multicore systems.

Story & paper: https://blog.acolyer.org/2016/04/26/the-linux-scheduler-a-decade-of-wasted-cores/

Code: https://github.com/jplozi/wastedcores

Would love to hear your thoughts.

--
Cheers,

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: For those of us who bench on Linux

David Dice
In reply to this post by Viktor Klang
The linux scheduler(s) -- O(1) and then CFS -- have been non work conserving for some time, often leading to odd behavior for java applications.   See https://blogs.oracle.com/dave/entry/a_scheduling_and_dispatching_oddity.   

I'll sometimes tell afflicted customers about the following work-around : 

sudo sh -c echo 5000 > /proc/sys/kernel/sched_migration_cost_ns"

The above is palliative, but can be helpful.  Caveat utor.  

Sched_migration_cost_ns represents a trade-off between eagerness to migrate (and a tendency toward being work conserving) vs affinity to a CPU where a thread might have some residual cache affinity.   It also reflects migration costs, as evidenced by the name.  

When a thread has been migrated, the scheduler will prevent the thread from re-migrating for the duration specified via the sched_migration_cost_ns pseudo-file.  IIRC, this holds even if the thread is blocked, or preempted, btw.     Even if there are other idle CPUs on which a ready thread T1 might run,  the scheduler will refrain from migrating T1 during T1's sched_migration_cost_ns interval.  This damps migration rates (avoiding rampant migration) which is normally a good thing, but it also prevents the scheduler from being work conserving.  Lower sched_migration_cost_ns values make the system tend toward being more work conserving, but also increase migration rates, so there's a trade-off.      

For the JVM, I’ve found that I can reduce the “wasted dead time” — time where we have both idle CPUs and runnable threads languishing on ready queues — by reducing the value.  It’s not prefect, but it can make a considerable difference for some benchmarks.   Beware, however, that it's been a few years since I've looked at this issue, so YMMV.    

Solaris, in contrast, preserves residual affinity to the extent possible, but it's always work conserving. Affinity-aware scheduling is a win, but being work conserving  generally needs to have precedence over affinity.  (You can contrive cases where that claim doesn’t hold, but it’s a good rule of thumb).    

Dave


David Holmes wrote:
 
Message: 1
Date: Wed, 27 Apr 2016 18:50:54 +1000
From: "David Holmes" <[hidden email]>
To: "'Viktor Klang'" <[hidden email]>,
        "'concurrency-interest'" <[hidden email]>
Subject: Re: [concurrency-interest] For those of us who bench on Linux
Message-ID: <002901d1a061$e8faafb0$baf00f10$@aapt.net.au>
Content-Type: text/plain; charset="utf-8"

Very interesting. In particular that scheduling affinity (to take advantage of a warm cache) can be a performance hit.



Thanks for the link.



David



From: [hidden email] [mailto:[hidden email]] On Behalf Of Viktor Klang
Sent: Wednesday, April 27, 2016 5:24 PM
To: concurrency-interest <[hidden email]>
Subject: [concurrency-interest] For those of us who bench on Linux



TL;DR: problems found and fixed with Linux scheduler affecting multicore systems.

Story & paper: https://blog.acolyer.org/2016/04/26/the-linux-scheduler-a-decade-of-wasted-cores/

Code: https://github.com/jplozi/wastedcores

Would love to hear your thoughts.

--
Cheers,
?

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest