Concurrent Queue Sizes and Hot Fields

classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
On 09/23/2018 10:16 AM, Francesco Nigro wrote:
> Sorry to be OT but now I'm curious about the original Channel API
> proposal :)

Basically, the methods introduced in Queue, but not extending
Collection. Candidates were based mainly on my pre-j.u.c package
http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/intro.html
; in particular,
http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/Channel.html

-Doug



>
> Il dom 23 set 2018, 14:01 Doug Lea via Concurrency-interest
> <[hidden email]
> <mailto:[hidden email]>> ha scritto:
>
>     On 09/20/2018 11:32 AM, Dr Heinz M. Kabutz via Concurrency-interest
>     wrote:
>     > Most likely too basic for this audience ... :-)
>     >
>     > Abstract: ConcurrentLinkedQueue's size() method is not very useful
>     in a
>     > multi-threaded environment, because it counts the number of
>     elements in
>     > the queue, rather than relying on a "hot field" to store the size. The
>     > result might be completely incorrect, or in strange situations, never
>     > return.
>     >
>     > Full article: https://www.javaspecialists.eu/archive/Issue261.html
>     >
>
>     This was one of the very first API design issues in j.u.c (some of the
>     discussions are probably in old concurrency-interest
>     archives): Should concurrent queues be forced to implement Collection?
>     Because supporting both the size() and remove(x) methods are
>     problematic, the initial versions included unrelated interface
>     "Channel". But people successfully argued for integrating with
>     Collections via Queue interface, even though it forces choice among bad
>     implementation options. For size(), the simplest is to add a sequence
>     field to each node, ensuring that it is one greater than predecessor
>     when CASing in. This leads to fast and accurate size(), at the expense
>     of a 25% increase in space. Another is to add an atomic counter, which
>     adds contention and still presents before-vs-after increment
>     inaccuracies. The third, which we adopted, is to traverse, with all the
>     disadvantages mentioned in the article. As others have mentioned, the
>     main rationale is that people should not be calling size for any kind of
>     synchronization control anyway; for occasional uses in monitoring etc,
>     the slow and inaccurate version is normally OK. We now have a few more
>     options available (LongAdder or variants), but the same arguments for
>     the status quo still seem to apply.
>
>
>     -Doug
>
>
>     _______________________________________________
>     Concurrency-interest mailing list
>     [hidden email]
>     <mailto:[hidden email]>
>     http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
Many thanks, nice one! 

Il dom 23 set 2018, 16:25 Doug Lea <[hidden email]> ha scritto:
On 09/23/2018 10:16 AM, Francesco Nigro wrote:
> Sorry to be OT but now I'm curious about the original Channel API
> proposal :)

Basically, the methods introduced in Queue, but not extending
Collection. Candidates were based mainly on my pre-j.u.c package
http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/intro.html
; in particular,
http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/Channel.html

-Doug



>
> Il dom 23 set 2018, 14:01 Doug Lea via Concurrency-interest
> <[hidden email]
> <mailto:[hidden email]>> ha scritto:
>
>     On 09/20/2018 11:32 AM, Dr Heinz M. Kabutz via Concurrency-interest
>     wrote:
>     > Most likely too basic for this audience ... :-)
>     >
>     > Abstract: ConcurrentLinkedQueue's size() method is not very useful
>     in a
>     > multi-threaded environment, because it counts the number of
>     elements in
>     > the queue, rather than relying on a "hot field" to store the size. The
>     > result might be completely incorrect, or in strange situations, never
>     > return.
>     >
>     > Full article: https://www.javaspecialists.eu/archive/Issue261.html
>     >
>
>     This was one of the very first API design issues in j.u.c (some of the
>     discussions are probably in old concurrency-interest
>     archives): Should concurrent queues be forced to implement Collection?
>     Because supporting both the size() and remove(x) methods are
>     problematic, the initial versions included unrelated interface
>     "Channel". But people successfully argued for integrating with
>     Collections via Queue interface, even though it forces choice among bad
>     implementation options. For size(), the simplest is to add a sequence
>     field to each node, ensuring that it is one greater than predecessor
>     when CASing in. This leads to fast and accurate size(), at the expense
>     of a 25% increase in space. Another is to add an atomic counter, which
>     adds contention and still presents before-vs-after increment
>     inaccuracies. The third, which we adopted, is to traverse, with all the
>     disadvantages mentioned in the article. As others have mentioned, the
>     main rationale is that people should not be calling size for any kind of
>     synchronization control anyway; for occasional uses in monitoring etc,
>     the slow and inaccurate version is normally OK. We now have a few more
>     options available (LongAdder or variants), but the same arguments for
>     the status quo still seem to apply.
>
>
>     -Doug
>
>
>     _______________________________________________
>     Concurrency-interest mailing list
>     [hidden email]
>     <mailto:[hidden email]>
>     http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>


Il 23 set 2018 4:25 PM, "Doug Lea" <[hidden email]> ha scritto:
On 09/23/2018 10:16 AM, Francesco Nigro wrote:
> Sorry to be OT but now I'm curious about the original Channel API
> proposal :)

Basically, the methods introduced in Queue, but not extending
Collection. Candidates were based mainly on my pre-j.u.c package
http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/intro.html
; in particular,
http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/Channel.html

-Doug




>
> Il dom 23 set 2018, 14:01 Doug Lea via Concurrency-interest
> <[hidden email]
> <mailto:[hidden email]>> ha scritto:

>
>     On 09/20/2018 11:32 AM, Dr Heinz M. Kabutz via Concurrency-interest
>     wrote:
>     > Most likely too basic for this audience ... :-)
>     >
>     > Abstract: ConcurrentLinkedQueue's size() method is not very useful
>     in a
>     > multi-threaded environment, because it counts the number of
>     elements in
>     > the queue, rather than relying on a "hot field" to store the size. The
>     > result might be completely incorrect, or in strange situations, never
>     > return.
>     >
>     > Full article: https://www.javaspecialists.eu/archive/Issue261.html
>     >
>
>     This was one of the very first API design issues in j.u.c (some of the
>     discussions are probably in old concurrency-interest
>     archives): Should concurrent queues be forced to implement Collection?
>     Because supporting both the size() and remove(x) methods are
>     problematic, the initial versions included unrelated interface
>     "Channel". But people successfully argued for integrating with
>     Collections via Queue interface, even though it forces choice among bad
>     implementation options. For size(), the simplest is to add a sequence
>     field to each node, ensuring that it is one greater than predecessor
>     when CASing in. This leads to fast and accurate size(), at the expense
>     of a 25% increase in space. Another is to add an atomic counter, which
>     adds contention and still presents before-vs-after increment
>     inaccuracies. The third, which we adopted, is to traverse, with all the
>     disadvantages mentioned in the article. As others have mentioned, the
>     main rationale is that people should not be calling size for any kind of
>     synchronization control anyway; for occasional uses in monitoring etc,
>     the slow and inaccurate version is normally OK. We now have a few more
>     options available (LongAdder or variants), but the same arguments for
>     the status quo still seem to apply.
>
>
>     -Doug
>
>
>     _______________________________________________
>     Concurrency-interest mailing list
>     [hidden email]
>     <mailto:[hidden email]>
>     http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>



_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
In reply to this post by JSR166 Concurrency mailing list
Collection and friends have always been intentionally permissive in their semantics, to permit atypical implementations. That said, the only reason to implement it is because it's useful to programmers to do so. In this case, I suspect the primary use is that it ensures that standard methods have the correct names and signatures. That's not to be sneezed at, and the disadvantages seem small, right? Sequence numbers feel like overkill.

On Sun, Sep 23, 2018 at 10:20 AM Doug Lea via Concurrency-interest <[hidden email]> wrote:
On 09/23/2018 09:18 AM, Alex Otenko wrote:
> I don’t think you can do that reliably. In the end, who’s faster: a
> consumer removing elements, or remove(x) incrementing the sequence
> numbers? There will be subtle races showing up as weird values for
> size().

To first answer:

> I think you need to start with what a “correct result” means.
>
> First the queue is forced to be a Collection, then we find that the
> methods cannot be implemented “correctly” :-) Maybe it isn’t a
> Collection?
>

I initially resisted integrating with Collection on these grounds. But
one argument in favor is that implementations can at least preserve
quiescent consistency: results for one thread meet specs if no other
threads execute. This is the notion underlying all j.u.c methods that
contain "not for use in synchronization control" disclaimers.  Also, all
j.u.c queues, being Collections, are iterable, so already have defined
weak consistency properties on these grounds. So any solution based on
traversal is no worse than what people could do themselves; and some
surely would do so if we did not supply the method.

Given all this, using sequence numbers could provide stronger guarantees
(but with an unattractive time/space tradeoff) in the absence of
internal remove, but no better than now in their presence.

-Doug


>
> Alex
>
>> On 23 Sep 2018, at 13:38, Doug Lea via Concurrency-interest
>> <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>> On 09/23/2018 08:28 AM, Dr Heinz M. Kabutz wrote:
>>
>>> Collections via Queue interface, even though it forces choice
>>> among bad implementation options. For size(), the simplest is to
>>> add a sequence field to each node, ensuring that it is one
>>> greater than predecessor when CASing in. This leads to fast and
>>> accurate size(), at the expense of a 25% increase in space.
>>>
>>>
>>> On 64-bit with compressed oops, each node is 12+4+4 bytes, which
>>> is rounded up to 24 bytes.  Thus in terms of memory usage the
>>> extra int would be free on that architecture.
>>>
>>> That said, I think the case of removal in the middle would then
>>> also give us wrong results.
>>
>> Now "wrong", but internal remove(x) would be even slower: the
>> method would need to retraverse and atomically increment each
>> sequence number of each predecessor.
>>
>> -Doug
>>
>>
>>
>>
>>>
>>> Another is to add an atomic counter, which adds contention and
>>> still presents before-vs-after increment inaccuracies. The third,
>>> which we adopted, is to traverse, with all the disadvantages
>>> mentioned in the article. As others have mentioned, the main
>>> rationale is that people should not be calling size for any kind
>>> of synchronization control anyway; for occasional uses in
>>> monitoring etc, the slow and inaccurate version is normally OK.
>>> We now have a few more options available (LongAdder or variants),
>>> but the same arguments for the status quo still seem to apply.
>>>
>>>
>>> In my newsletter i was mainly pointing out the behavior, rather
>>> than suggesting ways to fix it, because I’ve seen a lot of
>>> puzzled faces from experienced Java programmers over the years.
>>>
>>>
>>>
>>>
>>>
>>> -Doug
>>>
>>>
>>> _______________________________________________
>>> Concurrency-interest mailing list
>>> [hidden email]
>>> <mailto:[hidden email]>
>>> <mailto:[hidden email]>
>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>
>>> -- Dr Heinz M. Kabutz (PhD CompSci) Author of "The Java(tm)
>>> Specialists' Newsletter" Sun/Oracle Java Champion JavaOne
>>> Rockstar Speaker http://www.javaspecialists.eu Tel: +30 69 75 595
>>> 262 Skype: kabutz
>>
>>
>> _______________________________________________
>> Concurrency-interest mailing list
>> [hidden email]
>> <mailto:[hidden email]>
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
No, it is sequence numbers that you set up non-atomically at creation time, and only update atomically in rare and unlikely circumstances, and likely with no overhead, if Dr Kabutz’s working out is right, vs extra atomic operations for every single put or take to keep the counters for the method that has no use in concurrent scenarios (that only have use for contrived use cases).


Alex

On 23 Sep 2018, at 21:53, Joshua Bloch <[hidden email]> wrote:

Collection and friends have always been intentionally permissive in their semantics, to permit atypical implementations. That said, the only reason to implement it is because it's useful to programmers to do so. In this case, I suspect the primary use is that it ensures that standard methods have the correct names and signatures. That's not to be sneezed at, and the disadvantages seem small, right? Sequence numbers feel like overkill.

On Sun, Sep 23, 2018 at 10:20 AM Doug Lea via Concurrency-interest <[hidden email]> wrote:
On 09/23/2018 09:18 AM, Alex Otenko wrote:
> I don’t think you can do that reliably. In the end, who’s faster: a
> consumer removing elements, or remove(x) incrementing the sequence
> numbers? There will be subtle races showing up as weird values for
> size().

To first answer:

> I think you need to start with what a “correct result” means.
>
> First the queue is forced to be a Collection, then we find that the
> methods cannot be implemented “correctly” :-) Maybe it isn’t a
> Collection?
>

I initially resisted integrating with Collection on these grounds. But
one argument in favor is that implementations can at least preserve
quiescent consistency: results for one thread meet specs if no other
threads execute. This is the notion underlying all j.u.c methods that
contain "not for use in synchronization control" disclaimers.  Also, all
j.u.c queues, being Collections, are iterable, so already have defined
weak consistency properties on these grounds. So any solution based on
traversal is no worse than what people could do themselves; and some
surely would do so if we did not supply the method.

Given all this, using sequence numbers could provide stronger guarantees
(but with an unattractive time/space tradeoff) in the absence of
internal remove, but no better than now in their presence.

-Doug


>
> Alex
>
>> On 23 Sep 2018, at 13:38, Doug Lea via Concurrency-interest
>> <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>> On 09/23/2018 08:28 AM, Dr Heinz M. Kabutz wrote:
>>
>>> Collections via Queue interface, even though it forces choice
>>> among bad implementation options. For size(), the simplest is to
>>> add a sequence field to each node, ensuring that it is one
>>> greater than predecessor when CASing in. This leads to fast and
>>> accurate size(), at the expense of a 25% increase in space.
>>>
>>>
>>> On 64-bit with compressed oops, each node is 12+4+4 bytes, which
>>> is rounded up to 24 bytes.  Thus in terms of memory usage the
>>> extra int would be free on that architecture.
>>>
>>> That said, I think the case of removal in the middle would then
>>> also give us wrong results.
>>
>> Now "wrong", but internal remove(x) would be even slower: the
>> method would need to retraverse and atomically increment each
>> sequence number of each predecessor.
>>
>> -Doug
>>
>>
>>
>>
>>>
>>> Another is to add an atomic counter, which adds contention and
>>> still presents before-vs-after increment inaccuracies. The third,
>>> which we adopted, is to traverse, with all the disadvantages
>>> mentioned in the article. As others have mentioned, the main
>>> rationale is that people should not be calling size for any kind
>>> of synchronization control anyway; for occasional uses in
>>> monitoring etc, the slow and inaccurate version is normally OK.
>>> We now have a few more options available (LongAdder or variants),
>>> but the same arguments for the status quo still seem to apply.
>>>
>>>
>>> In my newsletter i was mainly pointing out the behavior, rather
>>> than suggesting ways to fix it, because I’ve seen a lot of
>>> puzzled faces from experienced Java programmers over the years.
>>>
>>>
>>>
>>>
>>>
>>> -Doug
>>>
>>>
>>> _______________________________________________
>>> Concurrency-interest mailing list
>>> [hidden email]
>>> <mailto:[hidden email]>
>>> <mailto:[hidden email]>
>>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>>
>>> -- Dr Heinz M. Kabutz (PhD CompSci) Author of "The Java(tm)
>>> Specialists' Newsletter" Sun/Oracle Java Champion JavaOne
>>> Rockstar Speaker http://www.javaspecialists.eu Tel: +30 69 75 595
>>> 262 Skype: kabutz
>>
>>
>> _______________________________________________
>> Concurrency-interest mailing list
>> [hidden email]
>> <mailto:[hidden email]>
>> http://cs.oswego.edu/mailman/listinfo/concurrency-interest
>


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
The conservative maintainers of the concurrent queues are unlikely to
change the implementation.  The behavior of the queues when quiescent
is correct and useful.  Any fix for size() would be unsatisfying if it
did not also apply to toArray().  It might be good to add scare
javadocs about how traversal may end up visiting many more elements
than the high water mark size of the queue (but that "leaks"
implementation details into the javadoc).
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
And that in rare circumstances it might never return at all, even for a queue of essentially fixed length. 

On Tue, 25 Sep 2018 at 00:16, Martin Buchholz via Concurrency-interest <[hidden email]> wrote:
The conservative maintainers of the concurrent queues are unlikely to
change the implementation.  The behavior of the queues when quiescent
is correct and useful.  Any fix for size() would be unsatisfying if it
did not also apply to toArray().  It might be good to add scare
javadocs about how traversal may end up visiting many more elements
than the high water mark size of the queue (but that "leaks"
implementation details into the javadoc).
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
--
Dr Heinz M. Kabutz (PhD CompSci)
Author of "The Java(tm) Specialists' Newsletter"
Sun/Oracle Java Champion
JavaOne Rockstar Speaker
http://www.javaspecialists.eu
Tel: +30 69 75 595 262
Skype: kabutz

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
On Mon, Sep 24, 2018 at 4:12 PM, Dr Heinz M. Kabutz
<[hidden email]> wrote:
> And that in rare circumstances it might never return at all, even for a
> queue of essentially fixed length.

Think of it like this - never reaching the end of a traversal is a
problem with many (most?) mutable data structures.  Even iterating
single-threaded over LinkedList could take forever if an element is
enqueued and dequeued at every step.  it's a treadmill.
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
Out of curiosity, suppose that Java had access to 2word atomics (like CMPXCHG16B on x64).  It might be possible to align the head pointer with a counter word, and update them both at the same time.  With the tail pointer, a similar counter could also be updated when offer()ing.   That would allow size() to be wait free, though likely still inaccurate.

On Mon, Sep 24, 2018 at 6:43 PM Martin Buchholz via Concurrency-interest <[hidden email]> wrote:
On Mon, Sep 24, 2018 at 4:12 PM, Dr Heinz M. Kabutz
<[hidden email]> wrote:
> And that in rare circumstances it might never return at all, even for a
> queue of essentially fixed length.

Think of it like this - never reaching the end of a traversal is a
problem with many (most?) mutable data structures.  Even iterating
single-threaded over LinkedList could take forever if an element is
enqueued and dequeued at every step.  it's a treadmill.
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
On Mon, Sep 24, 2018 at 9:33 PM, Carl Mastrangelo <[hidden email]> wrote:
> Out of curiosity, suppose that Java had access to 2word atomics (like
> CMPXCHG16B on x64).  It might be possible to align the head pointer with a
> counter word, and update them both at the same time.  With the tail pointer,
> a similar counter could also be updated when offer()ing.   That would allow
> size() to be wait free, though likely still inaccurate.

There's interior remove to deal with.

ConcurrentLinkedQueue does not necessarily update head and tail for
each enqueue and dequeue.

There's toArray to deal with, although it could give up after
encountering size() elements.

I don't know whether double-width CAS would be faster or slower than
two single-width CASes.
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
In reply to this post by JSR166 Concurrency mailing list
In a concurrent setting non-termination could also be a suspension, too. No one said size() is meant to be lock-free. :-)

Once it is not lock-free, you cannot guarantee the progress of any thread, once some chosen single thread is suspended. This requirement is so weak, that it allows to not guarantee the progress of more than one thread at any time.


Alex

On 25 Sep 2018, at 00:12, Dr Heinz M. Kabutz <[hidden email]> wrote:

And that in rare circumstances it might never return at all, even for a queue of essentially fixed length. 

On Tue, 25 Sep 2018 at 00:16, Martin Buchholz via Concurrency-interest <[hidden email]> wrote:
The conservative maintainers of the concurrent queues are unlikely to
change the implementation.  The behavior of the queues when quiescent
is correct and useful.  Any fix for size() would be unsatisfying if it
did not also apply to toArray().  It might be good to add scare
javadocs about how traversal may end up visiting many more elements
than the high water mark size of the queue (but that "leaks"
implementation details into the javadoc).
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
--
Dr Heinz M. Kabutz (PhD CompSci)
Author of "The Java(tm) Specialists' Newsletter"
Sun/Oracle Java Champion
JavaOne Rockstar Speaker
http://www.javaspecialists.eu
Tel: +30 69 75 595 262
Skype: kabutz


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent Queue Sizes and Hot Fields

JSR166 Concurrency mailing list
Maybe a better "other" method to think about is contains(Object).
Here there's no choice but to traverse the collection.  If you are
consistently slow, you will keep falling off the front of the
treadmill and have to get back on.  A plausible implementation
strategy would fall back to a more expensive but more terminating
traversal after the first fall off the treadmill.

(but these concurrent queue traversal methods are already pretty
complicated and hard to test)
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
12