Reactive Streams Utility API

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Reactive Streams Utility API

JSR166 Concurrency mailing list
Hi all,

We (Lightbend) would like to put forward a proposal for a Reactive Streams utility API for building instances of juc.Flow interfaces. The rationale, goals and non goals for this, along with our proposed approach to the API, and an actual (incomplete) API proposal, TCK, with implementation examples both in Akka Streams and RxJava, can be found here:


We are also concurrently discussing this on the core-libs-dev mailing list:


We would love to hear any feedback that people have.

Regards,

--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list
Hi,

I have some observations:

1) Building up the SPI on top of Processors is an overkill. As RxJava and Reactor demonstrate, a processing pipeline is a series (linked list) of Subscribers and there is no need to introduce an unicast subscription interface (aka the Publisher side of a Processor) to establish a chain. The chain of Subscribers has also the benefit of allowing multiple realization of the same cold Publisher as each chain will be independent of the other. A Processor suggest there is an inherent multicasting and coordination to happen, even if the chain is linear.

2) Having an arbitrary processing graph, which may include cycles, looks great on paper but is rarely necessary in practice. Yes, there are cases with feedback loop such as issuing a HTTP request once the previous response has been processed by the chain, but that can often be implemented through a Processor at the front of the chain. In addition, thinking in linear or fork-join like flows is easier than thinking in arbitrary processing graphs.

3) There are some semi-standard naming conventions for operators in RxJava and Reactor (or ReactiveX in general). If the naming in the SPI strays too much away, it adds an additional burden for the user to learn and match operators and concepts between these components.

4) Having the SPI as a mediator may be adding a significant overhead in terms of internal allocations and indirections. These include the potential functional interface conversion in case a particular library allows checked exceptions via its own custom defined functional interfaces. Current RS implementation feature conversions to and from the Flow API as a very thin wrapper stage and the rest of the processing is expressed in the native DSL of the particular library. Given that RS is meant primarily as an interop library, such conversions shouldn't be much of a burden.

5) C# has extension methods and expression trees for basically the same purpose as this proposal. Advocating for either or both may be more enabling not just for RS, but for other uses.

6) About the reference implementation:


Makes no sense, bind them directly together. Of course, a Processor in the middle makes it overcomplicated.


Stages could have default implementations the SPI can reference back if the particular engine doesn't support it. I don't think failing the construction via
UnsupportedStageException is a good idea.


Terminating the CompletableFuture should likely happen after the downstream has been notified so the termination includes the processing inside those handlers.




2018-03-01 4:37 GMT+01:00 James Roper via Concurrency-interest <[hidden email]>:
Hi all,

We (Lightbend) would like to put forward a proposal for a Reactive Streams utility API for building instances of juc.Flow interfaces. The rationale, goals and non goals for this, along with our proposed approach to the API, and an actual (incomplete) API proposal, TCK, with implementation examples both in Akka Streams and RxJava, can be found here:


We are also concurrently discussing this on the core-libs-dev mailing list:


We would love to hear any feedback that people have.

Regards,

--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
Best regards,
David Karnok

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list
Hi Dávid,

Responses inline.

On 1 March 2018 at 20:56, Dávid Karnok via Concurrency-interest <[hidden email]> wrote:
Hi,

I have some observations:

1) Building up the SPI on top of Processors is an overkill. As RxJava and Reactor demonstrate, a processing pipeline is a series (linked list) of Subscribers and there is no need to introduce an unicast subscription interface (aka the Publisher side of a Processor) to establish a chain. The chain of Subscribers has also the benefit of allowing multiple realization of the same cold Publisher as each chain will be independent of the other. A Processor suggest there is an inherent multicasting and coordination to happen, even if the chain is linear.

Could you give an example of building a chain of subscribers without having an initial publisher - eg, some the code that I would write in rx java to do it? Because I looked but couldn't find any way to do that. For example, how would I use rxjava to parse and process responses from the JDK9 HTTP client, which requires passing to the API a Function<HttpResponse, Subscriber<ByteBuffer>> in order to consume the bytes in the response?
 
2) Having an arbitrary processing graph, which may include cycles, looks great on paper but is rarely necessary in practice. Yes, there are cases with feedback loop such as issuing a HTTP request once the previous response has been processed by the chain, but that can often be implemented through a Processor at the front of the chain. In addition, thinking in linear or fork-join like flows is easier than thinking in arbitrary processing graphs.

At this stage it's not likely that we'll offer any arbitrary graph features, I agree, it's not a common enough requirement and adds complexity to the conceptual modelling of an API that provides it.

3) There are some semi-standard naming conventions for operators in RxJava and Reactor (or ReactiveX in general). If the naming in the SPI strays too much away, it adds an additional burden for the user to learn and match operators and concepts between these components.

The naming of the stages in the SPI I've tried to make match the naming of the API methods that sit on top of them in the API, and the naming in the API I've tried to make match the naming in the JDK8 Streams API. I definitely think we should be looking to RxJava, Reactor and Akka Streams for inspiration for naming where the JDK8 Streams API lacks features that we want to add, but I think in order to reduce the burden of learning and matching operators for new users who aren't already selecting a streaming framework (because if they're already using RxJava, then why would they want to use this API? So those familiar with RxJava aren't the target audience), that we should stick as close to the JDK8 streams API as possible.
 
4) Having the SPI as a mediator may be adding a significant overhead in terms of internal allocations and indirections. These include the potential functional interface conversion in case a particular library allows checked exceptions via its own custom defined functional interfaces. Current RS implementation feature conversions to and from the Flow API as a very thin wrapper stage and the rest of the processing is expressed in the native DSL of the particular library. Given that RS is meant primarily as an interop library, such conversions shouldn't be much of a burden.

I don't think conversions to/from functional interface types are avoidable.
 

5) C# has extension methods and expression trees for basically the same purpose as this proposal. Advocating for either or both may be more enabling not just for RS, but for other uses.

6) About the reference implementation:

I wouldn't call this a reference implementation, it's just a proof of concept to show that the SPI can be implemented.


Makes no sense, bind them directly together. Of course, a Processor in the middle makes it overcomplicated.

This goes back to the "how do I create a Subscriber with transforms like map/filter applied to it in rxjava without first having a Publisher". If there's a straight forward existing way to do that, then I agree, this can be removed.


Stages could have default implementations the SPI can reference back if the particular engine doesn't support it. I don't think failing the construction via
UnsupportedStageException is a good idea.

Lets say an implementation provides a tracing feature (such as Reactor), and I've selected that particular implementation because it does that tracing. A default implementation is likely to break that tracing, and cause messages traversing the graph to become dissociated with downstream messages emitted. In that case, I would rather it fail loudly to tell me that this would happen, rather than silently fail.


Terminating the CompletableFuture should likely happen after the downstream has been notified so the termination includes the processing inside those handlers.

Good point, I'll change it.
 




2018-03-01 4:37 GMT+01:00 James Roper via Concurrency-interest <[hidden email]>:
Hi all,

We (Lightbend) would like to put forward a proposal for a Reactive Streams utility API for building instances of juc.Flow interfaces. The rationale, goals and non goals for this, along with our proposed approach to the API, and an actual (incomplete) API proposal, TCK, with implementation examples both in Akka Streams and RxJava, can be found here:


We are also concurrently discussing this on the core-libs-dev mailing list:


We would love to hear any feedback that people have.

Regards,

--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
Best regards,
David Karnok

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list


2018-03-02 7:26 GMT+01:00 James Roper <[hidden email]>:
Hi Dávid,

Responses inline.

On 1 March 2018 at 20:56, Dávid Karnok via Concurrency-interest <[hidden email]> wrote:
Hi,

I have some observations:

1) Building up the SPI on top of Processors is an overkill. As RxJava and Reactor demonstrate, a processing pipeline is a series (linked list) of Subscribers and there is no need to introduce an unicast subscription interface (aka the Publisher side of a Processor) to establish a chain. The chain of Subscribers has also the benefit of allowing multiple realization of the same cold Publisher as each chain will be independent of the other. A Processor suggest there is an inherent multicasting and coordination to happen, even if the chain is linear.

Could you give an example of building a chain of subscribers without having an initial publisher - eg, some the code that I would write in rx java to do it? Because I looked but couldn't find any way to do that.

It always starts with a Publisher, we never have the user create any chain of Subscribers directly. That's why I'm not fond of your SubscriberBuilder.
 
For example, how would I use rxjava to parse and process responses from the JDK9 HTTP client, which requires passing to the API a Function<HttpResponse, Subscriber<ByteBuffer>> in order to consume the bytes in the response?

That looks like a bad design, there was no reason in the past years to have users provide a Subscriber for other than Publisher.subscribe(). If you wish to consume something from a provider, you should get a Publisher back.
  

Makes no sense, bind them directly together. Of course, a Processor in the middle makes it overcomplicated.

This goes back to the "how do I create a Subscriber with transforms like map/filter applied to it in rxjava without first having a Publisher". If there's a straight forward existing way to do that, then I agree, this can be removed.

You don't, you create Publisher transformations. This way, a blueprint is saved on how to create a chain once the end Publisher is subscribed to. This is almost the same as Akka-Stream's Source type.
 





2018-03-01 4:37 GMT+01:00 James Roper via Concurrency-interest <[hidden email]>:
Hi all,

We (Lightbend) would like to put forward a proposal for a Reactive Streams utility API for building instances of juc.Flow interfaces. The rationale, goals and non goals for this, along with our proposed approach to the API, and an actual (incomplete) API proposal, TCK, with implementation examples both in Akka Streams and RxJava, can be found here:


We are also concurrently discussing this on the core-libs-dev mailing list:


We would love to hear any feedback that people have.

Regards,

--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
Best regards,
David Karnok

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper




--
Best regards,
David Karnok

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list
On 2 March 2018 at 19:38, Dávid Karnok <[hidden email]> wrote:


2018-03-02 7:26 GMT+01:00 James Roper <[hidden email]>:
Hi Dávid,

Responses inline.

On 1 March 2018 at 20:56, Dávid Karnok via Concurrency-interest <[hidden email]> wrote:
Hi,

I have some observations:

1) Building up the SPI on top of Processors is an overkill. As RxJava and Reactor demonstrate, a processing pipeline is a series (linked list) of Subscribers and there is no need to introduce an unicast subscription interface (aka the Publisher side of a Processor) to establish a chain. The chain of Subscribers has also the benefit of allowing multiple realization of the same cold Publisher as each chain will be independent of the other. A Processor suggest there is an inherent multicasting and coordination to happen, even if the chain is linear.

Could you give an example of building a chain of subscribers without having an initial publisher - eg, some the code that I would write in rx java to do it? Because I looked but couldn't find any way to do that.

It always starts with a Publisher, we never have the user create any chain of Subscribers directly. That's why I'm not fond of your SubscriberBuilder.

If we don't provide this, then the JDKs own Reactive Streams utilities library will be incompatible with its own Reactive Streams integrations, which makes no sense.

For example, how would I use rxjava to parse and process responses from the JDK9 HTTP client, which requires passing to the API a Function<HttpResponse, Subscriber<ByteBuffer>> in order to consume the bytes in the response?

That looks like a bad design, there was no reason in the past years to have users provide a Subscriber for other than Publisher.subscribe(). If you wish to consume something from a provider, you should get a Publisher back.

There's actually some quite good reasons to design an API like this:

* The response body must be consumed, if it's not, a connection will be leaked. By requiring the user to supply a Subscriber, the API can ensure that the body is always consumed, and so can safely protect users against inadvertent connection leaks. We've been providing streaming APIs for HTTP clients that work in this fashion since well before Reactive Streams was even thought of, and have found that it's a very good way to ensure that users don't accidentally stuff up.
* Requiring users to supply a Subscriber also solves the problem of what to do if the user consumes a Publisher that represents a hot connection twice - by virtue of the fact that they can't, since there's no Publisher.
* For APIs such as messaging APIs, you want a subscription to be restarted (typically after a backoff period) after failures are encountered. If you supply the stream of messages to the user as a Publisher, and require them to connect it to the Subscriber, then that also forces them to monitor the lifecycle of the stream, and implement the reconnect and backoff logic. In contrast, if the user supplies a Subscriber to the framework, the framework itself can then implement that logic for the user. We offer APIs like this and find it's a great way to provide simple, easy to use managed integration with message brokers, and MicroProfile is currently considering a proposal that offers this style of API.

Of course, there's also drawbacks to this approach, for example if you want to connect an API that requires a Subscriber to an API that requires a Publisher, you need some sort of Processor bridge, just to name one. But it's not a clear cut one is better than the other, it's a trade off, and there a good reasons to use either approach. I don't think you can write it off as bad design. As it happens I did raise this as an issue on the net-dev list, that it wouldn't work well with RxJava, but the developers of the HTTP client weren't that interested. (http://mail.openjdk.java.net/pipermail/net-dev/2017-December/011063.html)

At any rate, libraries are always going to want to be able to supply developers with Subscribers to do various things. This API allows the to build those Subscribers without having to implement them from scratch, they can use the map/filter etc functions before feeding the stream elsewhere.
 
  

Makes no sense, bind them directly together. Of course, a Processor in the middle makes it overcomplicated.

This goes back to the "how do I create a Subscriber with transforms like map/filter applied to it in rxjava without first having a Publisher". If there's a straight forward existing way to do that, then I agree, this can be removed.

You don't, you create Publisher transformations. This way, a blueprint is saved on how to create a chain once the end Publisher is subscribed to. This is almost the same as Akka-Stream's Source type.
 





2018-03-01 4:37 GMT+01:00 James Roper via Concurrency-interest <[hidden email]>:
Hi all,

We (Lightbend) would like to put forward a proposal for a Reactive Streams utility API for building instances of juc.Flow interfaces. The rationale, goals and non goals for this, along with our proposed approach to the API, and an actual (incomplete) API proposal, TCK, with implementation examples both in Akka Streams and RxJava, can be found here:


We are also concurrently discussing this on the core-libs-dev mailing list:


We would love to hear any feedback that people have.

Regards,

--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
Best regards,
David Karnok

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper




--
Best regards,
David Karnok



--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list
If we don't provide this, then the JDKs own Reactive Streams utilities library will be incompatible with its own Reactive Streams integrations, which makes no sense.

Perhaps this is one point against doing this proposal inside the JDK. As the Akka-Stream architecture demonstrates, in complex frameworks it is better to expose a Publisher at the very end of a processing chain only so that the framework induced complications don't leak out to the user. In the HTTP client case, this complication comes from the underlying NIO interoperation.

The response body must be consumed, if it's not, a connection will be leaked.

Java 9 Introduced the Cleaner API that allows resolving such leaks of unconsumed components.

For APIs such as messaging APIs, you want a subscription to be restarted 

Retrying and repeating sources is a common practice, but having a pre-build Subscriber chain won't work there and adds the burden to re-establish the whole chain to the end user of the API.

As it happens I did raise this as an issue on the net-dev list, that it wouldn't work well with RxJava, but the developers of the HTTP client weren't that interested.

I believe the success of RS was partly due to it being external and independent of JDK developments. People experienced in the respective technology but inexperienced with reactive programming tend to project habits of design and coding that doesn't work well in general (but only in narrow cases which tricks them to think it actually generalizes). It was also hinted to me that with big corporations, no amount of blogs, discussions or mailing list posts will likely change their minds (see ADBC).


2018-03-02 11:00 GMT+01:00 James Roper <[hidden email]>:
On 2 March 2018 at 19:38, Dávid Karnok <[hidden email]> wrote:


2018-03-02 7:26 GMT+01:00 James Roper <[hidden email]>:
Hi Dávid,

Responses inline.

On 1 March 2018 at 20:56, Dávid Karnok via Concurrency-interest <[hidden email]> wrote:
Hi,

I have some observations:

1) Building up the SPI on top of Processors is an overkill. As RxJava and Reactor demonstrate, a processing pipeline is a series (linked list) of Subscribers and there is no need to introduce an unicast subscription interface (aka the Publisher side of a Processor) to establish a chain. The chain of Subscribers has also the benefit of allowing multiple realization of the same cold Publisher as each chain will be independent of the other. A Processor suggest there is an inherent multicasting and coordination to happen, even if the chain is linear.

Could you give an example of building a chain of subscribers without having an initial publisher - eg, some the code that I would write in rx java to do it? Because I looked but couldn't find any way to do that.

It always starts with a Publisher, we never have the user create any chain of Subscribers directly. That's why I'm not fond of your SubscriberBuilder.

If we don't provide this, then the JDKs own Reactive Streams utilities library will be incompatible with its own Reactive Streams integrations, which makes no sense.

For example, how would I use rxjava to parse and process responses from the JDK9 HTTP client, which requires passing to the API a Function<HttpResponse, Subscriber<ByteBuffer>> in order to consume the bytes in the response?

That looks like a bad design, there was no reason in the past years to have users provide a Subscriber for other than Publisher.subscribe(). If you wish to consume something from a provider, you should get a Publisher back.

There's actually some quite good reasons to design an API like this:

* The response body must be consumed, if it's not, a connection will be leaked. By requiring the user to supply a Subscriber, the API can ensure that the body is always consumed, and so can safely protect users against inadvertent connection leaks. We've been providing streaming APIs for HTTP clients that work in this fashion since well before Reactive Streams was even thought of, and have found that it's a very good way to ensure that users don't accidentally stuff up.
* Requiring users to supply a Subscriber also solves the problem of what to do if the user consumes a Publisher that represents a hot connection twice - by virtue of the fact that they can't, since there's no Publisher.
* For APIs such as messaging APIs, you want a subscription to be restarted (typically after a backoff period) after failures are encountered. If you supply the stream of messages to the user as a Publisher, and require them to connect it to the Subscriber, then that also forces them to monitor the lifecycle of the stream, and implement the reconnect and backoff logic. In contrast, if the user supplies a Subscriber to the framework, the framework itself can then implement that logic for the user. We offer APIs like this and find it's a great way to provide simple, easy to use managed integration with message brokers, and MicroProfile is currently considering a proposal that offers this style of API.

Of course, there's also drawbacks to this approach, for example if you want to connect an API that requires a Subscriber to an API that requires a Publisher, you need some sort of Processor bridge, just to name one. But it's not a clear cut one is better than the other, it's a trade off, and there a good reasons to use either approach. I don't think you can write it off as bad design. As it happens I did raise this as an issue on the net-dev list, that it wouldn't work well with RxJava, but the developers of the HTTP client weren't that interested. (http://mail.openjdk.java.net/pipermail/net-dev/2017-December/011063.html)

At any rate, libraries are always going to want to be able to supply developers with Subscribers to do various things. This API allows the to build those Subscribers without having to implement them from scratch, they can use the map/filter etc functions before feeding the stream elsewhere.
 
  

Makes no sense, bind them directly together. Of course, a Processor in the middle makes it overcomplicated.

This goes back to the "how do I create a Subscriber with transforms like map/filter applied to it in rxjava without first having a Publisher". If there's a straight forward existing way to do that, then I agree, this can be removed.

You don't, you create Publisher transformations. This way, a blueprint is saved on how to create a chain once the end Publisher is subscribed to. This is almost the same as Akka-Stream's Source type.
 





2018-03-01 4:37 GMT+01:00 James Roper via Concurrency-interest <[hidden email]>:
Hi all,

We (Lightbend) would like to put forward a proposal for a Reactive Streams utility API for building instances of juc.Flow interfaces. The rationale, goals and non goals for this, along with our proposed approach to the API, and an actual (incomplete) API proposal, TCK, with implementation examples both in Akka Streams and RxJava, can be found here:


We are also concurrently discussing this on the core-libs-dev mailing list:


We would love to hear any feedback that people have.

Regards,

--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
Best regards,
David Karnok

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper




--
Best regards,
David Karnok



--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper




--
Best regards,
David Karnok

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list


On 2 March 2018 at 21:41, Dávid Karnok <[hidden email]> wrote:
If we don't provide this, then the JDKs own Reactive Streams utilities library will be incompatible with its own Reactive Streams integrations, which makes no sense.

Perhaps this is one point against doing this proposal inside the JDK. As the Akka-Stream architecture demonstrates, in complex frameworks it is better to expose a Publisher at the very end of a processing chain only so that the framework induced complications don't leak out to the user. In the HTTP client case, this complication comes from the underlying NIO interoperation.

I don't think the Akka-Stream architecture demonstrates this at all - Akka Streams works well with any shape of stream.
 
The response body must be consumed, if it's not, a connection will be leaked.

Java 9 Introduced the Cleaner API that allows resolving such leaks of unconsumed components.

Relying on GC to clean up pooled resources is not a good design at all.

For APIs such as messaging APIs, you want a subscription to be restarted 

Retrying and repeating sources is a common practice, but having a pre-build Subscriber chain won't work there and adds the burden to re-establish the whole chain to the end user of the API.

We provide APIs that allow a user to instantiate a new Subscriber per invocation. It works. It's being used in production.

As it happens I did raise this as an issue on the net-dev list, that it wouldn't work well with RxJava, but the developers of the HTTP client weren't that interested.

I believe the success of RS was partly due to it being external and independent of JDK developments. People experienced in the respective technology but inexperienced with reactive programming tend to project habits of design and coding that doesn't work well in general (but only in narrow cases which tricks them to think it actually generalizes). It was also hinted to me that with big corporations, no amount of blogs, discussions or mailing list posts will likely change their minds (see ADBC).

How can you write the choice of this API off as inexperience in reactive programming? I just told you Lightbend has been providing APIs like this for years, Play Framework 2, released in 2012, has always had a model where to consume a request body on the server, you supply a Subscriber (or equivalent interface, Sink, and Iteratee back before Akka Streams), and that it has worked very well for us. If the only reason the HTTP client developers chose that pattern is out of inexperience, then why is Lightbend using it? Are you saying Lightbend is inexperienced with reactive programming? These are tried and trusted patterns that have been proved to work well for reactive systems development. I'm not asking you to like them, but don't call people that choose to use them inexperienced.


2018-03-02 11:00 GMT+01:00 James Roper <[hidden email]>:
On 2 March 2018 at 19:38, Dávid Karnok <[hidden email]> wrote:


2018-03-02 7:26 GMT+01:00 James Roper <[hidden email]>:
Hi Dávid,

Responses inline.

On 1 March 2018 at 20:56, Dávid Karnok via Concurrency-interest <[hidden email]> wrote:
Hi,

I have some observations:

1) Building up the SPI on top of Processors is an overkill. As RxJava and Reactor demonstrate, a processing pipeline is a series (linked list) of Subscribers and there is no need to introduce an unicast subscription interface (aka the Publisher side of a Processor) to establish a chain. The chain of Subscribers has also the benefit of allowing multiple realization of the same cold Publisher as each chain will be independent of the other. A Processor suggest there is an inherent multicasting and coordination to happen, even if the chain is linear.

Could you give an example of building a chain of subscribers without having an initial publisher - eg, some the code that I would write in rx java to do it? Because I looked but couldn't find any way to do that.

It always starts with a Publisher, we never have the user create any chain of Subscribers directly. That's why I'm not fond of your SubscriberBuilder.

If we don't provide this, then the JDKs own Reactive Streams utilities library will be incompatible with its own Reactive Streams integrations, which makes no sense.

For example, how would I use rxjava to parse and process responses from the JDK9 HTTP client, which requires passing to the API a Function<HttpResponse, Subscriber<ByteBuffer>> in order to consume the bytes in the response?

That looks like a bad design, there was no reason in the past years to have users provide a Subscriber for other than Publisher.subscribe(). If you wish to consume something from a provider, you should get a Publisher back.

There's actually some quite good reasons to design an API like this:

* The response body must be consumed, if it's not, a connection will be leaked. By requiring the user to supply a Subscriber, the API can ensure that the body is always consumed, and so can safely protect users against inadvertent connection leaks. We've been providing streaming APIs for HTTP clients that work in this fashion since well before Reactive Streams was even thought of, and have found that it's a very good way to ensure that users don't accidentally stuff up.
* Requiring users to supply a Subscriber also solves the problem of what to do if the user consumes a Publisher that represents a hot connection twice - by virtue of the fact that they can't, since there's no Publisher.
* For APIs such as messaging APIs, you want a subscription to be restarted (typically after a backoff period) after failures are encountered. If you supply the stream of messages to the user as a Publisher, and require them to connect it to the Subscriber, then that also forces them to monitor the lifecycle of the stream, and implement the reconnect and backoff logic. In contrast, if the user supplies a Subscriber to the framework, the framework itself can then implement that logic for the user. We offer APIs like this and find it's a great way to provide simple, easy to use managed integration with message brokers, and MicroProfile is currently considering a proposal that offers this style of API.

Of course, there's also drawbacks to this approach, for example if you want to connect an API that requires a Subscriber to an API that requires a Publisher, you need some sort of Processor bridge, just to name one. But it's not a clear cut one is better than the other, it's a trade off, and there a good reasons to use either approach. I don't think you can write it off as bad design. As it happens I did raise this as an issue on the net-dev list, that it wouldn't work well with RxJava, but the developers of the HTTP client weren't that interested. (http://mail.openjdk.java.net/pipermail/net-dev/2017-December/011063.html)

At any rate, libraries are always going to want to be able to supply developers with Subscribers to do various things. This API allows the to build those Subscribers without having to implement them from scratch, they can use the map/filter etc functions before feeding the stream elsewhere.
 
  

Makes no sense, bind them directly together. Of course, a Processor in the middle makes it overcomplicated.

This goes back to the "how do I create a Subscriber with transforms like map/filter applied to it in rxjava without first having a Publisher". If there's a straight forward existing way to do that, then I agree, this can be removed.

You don't, you create Publisher transformations. This way, a blueprint is saved on how to create a chain once the end Publisher is subscribed to. This is almost the same as Akka-Stream's Source type.
 





2018-03-01 4:37 GMT+01:00 James Roper via Concurrency-interest <[hidden email]>:
Hi all,

We (Lightbend) would like to put forward a proposal for a Reactive Streams utility API for building instances of juc.Flow interfaces. The rationale, goals and non goals for this, along with our proposed approach to the API, and an actual (incomplete) API proposal, TCK, with implementation examples both in Akka Streams and RxJava, can be found here:


We are also concurrently discussing this on the core-libs-dev mailing list:


We would love to hear any feedback that people have.

Regards,

--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
Best regards,
David Karnok

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper




--
Best regards,
David Karnok



--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper




--
Best regards,
David Karnok



--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list
I can't speak for Lightbend, but rememeber that the Reactive Streams specification was finalized in 2015, setting the requirements and expectations of multi-valued asynchronous streams straight. In this aspect, I don't see Iteratees reemerging and competing against RS, even though I'm sure it did get the job done in its time. As did any older, non-reactive technologies, work, yet they are now being replaced by RS-based designs. Of course, what parts of RS is being/going to be used is the question here. I suggest not designing the API around Processor- and Subscriber-builders but instead Publisher transformations.

2018-03-02 12:35 GMT+01:00 James Roper <[hidden email]>:


On 2 March 2018 at 21:41, Dávid Karnok <[hidden email]> wrote:
If we don't provide this, then the JDKs own Reactive Streams utilities library will be incompatible with its own Reactive Streams integrations, which makes no sense.

Perhaps this is one point against doing this proposal inside the JDK. As the Akka-Stream architecture demonstrates, in complex frameworks it is better to expose a Publisher at the very end of a processing chain only so that the framework induced complications don't leak out to the user. In the HTTP client case, this complication comes from the underlying NIO interoperation.

I don't think the Akka-Stream architecture demonstrates this at all - Akka Streams works well with any shape of stream.
 
The response body must be consumed, if it's not, a connection will be leaked.

Java 9 Introduced the Cleaner API that allows resolving such leaks of unconsumed components.

Relying on GC to clean up pooled resources is not a good design at all.

For APIs such as messaging APIs, you want a subscription to be restarted 

Retrying and repeating sources is a common practice, but having a pre-build Subscriber chain won't work there and adds the burden to re-establish the whole chain to the end user of the API.

We provide APIs that allow a user to instantiate a new Subscriber per invocation. It works. It's being used in production.

As it happens I did raise this as an issue on the net-dev list, that it wouldn't work well with RxJava, but the developers of the HTTP client weren't that interested.

I believe the success of RS was partly due to it being external and independent of JDK developments. People experienced in the respective technology but inexperienced with reactive programming tend to project habits of design and coding that doesn't work well in general (but only in narrow cases which tricks them to think it actually generalizes). It was also hinted to me that with big corporations, no amount of blogs, discussions or mailing list posts will likely change their minds (see ADBC).

How can you write the choice of this API off as inexperience in reactive programming? I just told you Lightbend has been providing APIs like this for years, Play Framework 2, released in 2012, has always had a model where to consume a request body on the server, you supply a Subscriber (or equivalent interface, Sink, and Iteratee back before Akka Streams), and that it has worked very well for us. If the only reason the HTTP client developers chose that pattern is out of inexperience, then why is Lightbend using it? Are you saying Lightbend is inexperienced with reactive programming? These are tried and trusted patterns that have been proved to work well for reactive systems development. I'm not asking you to like them, but don't call people that choose to use them inexperienced.


2018-03-02 11:00 GMT+01:00 James Roper <[hidden email]>:
On 2 March 2018 at 19:38, Dávid Karnok <[hidden email]> wrote:


2018-03-02 7:26 GMT+01:00 James Roper <[hidden email]>:
Hi Dávid,

Responses inline.

On 1 March 2018 at 20:56, Dávid Karnok via Concurrency-interest <[hidden email]> wrote:
Hi,

I have some observations:

1) Building up the SPI on top of Processors is an overkill. As RxJava and Reactor demonstrate, a processing pipeline is a series (linked list) of Subscribers and there is no need to introduce an unicast subscription interface (aka the Publisher side of a Processor) to establish a chain. The chain of Subscribers has also the benefit of allowing multiple realization of the same cold Publisher as each chain will be independent of the other. A Processor suggest there is an inherent multicasting and coordination to happen, even if the chain is linear.

Could you give an example of building a chain of subscribers without having an initial publisher - eg, some the code that I would write in rx java to do it? Because I looked but couldn't find any way to do that.

It always starts with a Publisher, we never have the user create any chain of Subscribers directly. That's why I'm not fond of your SubscriberBuilder.

If we don't provide this, then the JDKs own Reactive Streams utilities library will be incompatible with its own Reactive Streams integrations, which makes no sense.

For example, how would I use rxjava to parse and process responses from the JDK9 HTTP client, which requires passing to the API a Function<HttpResponse, Subscriber<ByteBuffer>> in order to consume the bytes in the response?

That looks like a bad design, there was no reason in the past years to have users provide a Subscriber for other than Publisher.subscribe(). If you wish to consume something from a provider, you should get a Publisher back.

There's actually some quite good reasons to design an API like this:

* The response body must be consumed, if it's not, a connection will be leaked. By requiring the user to supply a Subscriber, the API can ensure that the body is always consumed, and so can safely protect users against inadvertent connection leaks. We've been providing streaming APIs for HTTP clients that work in this fashion since well before Reactive Streams was even thought of, and have found that it's a very good way to ensure that users don't accidentally stuff up.
* Requiring users to supply a Subscriber also solves the problem of what to do if the user consumes a Publisher that represents a hot connection twice - by virtue of the fact that they can't, since there's no Publisher.
* For APIs such as messaging APIs, you want a subscription to be restarted (typically after a backoff period) after failures are encountered. If you supply the stream of messages to the user as a Publisher, and require them to connect it to the Subscriber, then that also forces them to monitor the lifecycle of the stream, and implement the reconnect and backoff logic. In contrast, if the user supplies a Subscriber to the framework, the framework itself can then implement that logic for the user. We offer APIs like this and find it's a great way to provide simple, easy to use managed integration with message brokers, and MicroProfile is currently considering a proposal that offers this style of API.

Of course, there's also drawbacks to this approach, for example if you want to connect an API that requires a Subscriber to an API that requires a Publisher, you need some sort of Processor bridge, just to name one. But it's not a clear cut one is better than the other, it's a trade off, and there a good reasons to use either approach. I don't think you can write it off as bad design. As it happens I did raise this as an issue on the net-dev list, that it wouldn't work well with RxJava, but the developers of the HTTP client weren't that interested. (http://mail.openjdk.java.net/pipermail/net-dev/2017-December/011063.html)

At any rate, libraries are always going to want to be able to supply developers with Subscribers to do various things. This API allows the to build those Subscribers without having to implement them from scratch, they can use the map/filter etc functions before feeding the stream elsewhere.
 
  

Makes no sense, bind them directly together. Of course, a Processor in the middle makes it overcomplicated.

This goes back to the "how do I create a Subscriber with transforms like map/filter applied to it in rxjava without first having a Publisher". If there's a straight forward existing way to do that, then I agree, this can be removed.

You don't, you create Publisher transformations. This way, a blueprint is saved on how to create a chain once the end Publisher is subscribed to. This is almost the same as Akka-Stream's Source type.
 





2018-03-01 4:37 GMT+01:00 James Roper via Concurrency-interest <[hidden email]>:
Hi all,

We (Lightbend) would like to put forward a proposal for a Reactive Streams utility API for building instances of juc.Flow interfaces. The rationale, goals and non goals for this, along with our proposed approach to the API, and an actual (incomplete) API proposal, TCK, with implementation examples both in Akka Streams and RxJava, can be found here:


We are also concurrently discussing this on the core-libs-dev mailing list:


We would love to hear any feedback that people have.

Regards,

--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
Best regards,
David Karnok

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper




--
Best regards,
David Karnok



--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper




--
Best regards,
David Karnok



--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper




--
Best regards,
David Karnok

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list
In reply to this post by JSR166 Concurrency mailing list
On 02/28/2018 10:37 PM, James Roper via Concurrency-interest wrote:

> We (Lightbend) would like to put forward a proposal for a Reactive
> Streams utility API for building instances of juc.Flow interfaces. The
> rationale, goals and non goals for this, along with our proposed
> approach to the API, and an actual (incomplete) API proposal, TCK, with
> implementation examples both in Akka Streams and RxJava, can be found here:
>
> https://github.com/lightbend/reactive-streams-utils
>

I initially sat this out hoping that more people using Flow and
reactive-streams would comment. But not many. And I'm reminded by
my last post about my main reservation, that history shows that
fluent APIs always grow to cover all combinations of all capabilities.
Which is an explicit non-goal of your proposal, but seems
inevitable anyway. This is a little scary from the point of view
creating JDK APIs. A much less ambitious first step would be
to create some static utilities (in class Flow) for map(),
collect() and a few others that could be used by layered
third-party frameworks. A few of these were initially present
in jsr166 prerelease versions but we killed them out of even
more cowardice.

To decide among these and other options, we do need some input
from current and prospective users!

-Doug
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list
On 30 March 2018 at 23:19, Doug Lea via Concurrency-interest <[hidden email]> wrote:
On 02/28/2018 10:37 PM, James Roper via Concurrency-interest wrote:

> We (Lightbend) would like to put forward a proposal for a Reactive
> Streams utility API for building instances of juc.Flow interfaces. The
> rationale, goals and non goals for this, along with our proposed
> approach to the API, and an actual (incomplete) API proposal, TCK, with
> implementation examples both in Akka Streams and RxJava, can be found here:
>
> https://github.com/lightbend/reactive-streams-utils
>

I initially sat this out hoping that more people using Flow and
reactive-streams would comment. But not many. And I'm reminded by
my last post about my main reservation, that history shows that
fluent APIs always grow to cover all combinations of all capabilities.
Which is an explicit non-goal of your proposal, but seems
inevitable anyway.

Perhaps relevant here would be to get some feedback from the java.util.stream API developers/maintainers. Have they found it inevitable that that API needs to grow to cover all combinations of all capabilities? Have they found any major issues preventing adoption due to the current capabilities (or lack there of) offered?

I haven't seen anything much out there saying that java.util.stream is incomplete and needs to be extended (aside from people saying that an equivalent is needed for asynchronous streams, ie, this proposal). Assuming that my assessment is correct, why haven't they found that their API hasn't needed to grow to cover all combinations of all capabilities? Does it already cover all combinations of all capabilities? Is there anything about asynchronous streaming that makes it fundamentally different to in memory collection streaming that would lead us to have a different experience when designing such an API?

This is a little scary from the point of view
creating JDK APIs. A much less ambitious first step would be
to create some static utilities (in class Flow) for map(),
collect() and a few others that could be used by layered
third-party frameworks. A few of these were initially present
in jsr166 prerelease versions but we killed them out of even
more cowardice.

To decide among these and other options, we do need some input
from current and prospective users!

-Doug
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest



--
James Roper
Senior Octonaut

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list
In reply to this post by JSR166 Concurrency mailing list


On Fri, Mar 30, 2018 at 2:19 PM, Doug Lea via Concurrency-interest <[hidden email]> wrote:
On 02/28/2018 10:37 PM, James Roper via Concurrency-interest wrote:

> We (Lightbend) would like to put forward a proposal for a Reactive
> Streams utility API for building instances of juc.Flow interfaces. The
> rationale, goals and non goals for this, along with our proposed
> approach to the API, and an actual (incomplete) API proposal, TCK, with
> implementation examples both in Akka Streams and RxJava, can be found here:
>
> https://github.com/lightbend/reactive-streams-utils
>

I initially sat this out hoping that more people using Flow and
reactive-streams would comment.

This is most likely not the right channel to elicit feedback from Flow/RS *end users*—may I suggest that if feedback from end-users is desired that we announce some sort of poll, or otherwise, on a different media such as, but not limited to, Twitter?
 
But not many. And I'm reminded by
my last post about my main reservation, that history shows that
fluent APIs always grow to cover all combinations of all capabilities.
Which is an explicit non-goal of your proposal, but seems
inevitable anyway. This is a little scary from the point of view
creating JDK APIs. A much less ambitious first step would be
to create some static utilities (in class Flow) for map(),
collect() and a few others that could be used by layered
third-party frameworks. A few of these were initially present
in jsr166 prerelease versions but we killed them out of even
more cowardice.

To decide among these and other options, we do need some input
from current and prospective users!

-Doug
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest



--
Cheers,

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list
I thought these RS API questions in the JDK were already past us since we created RS and argued about already. My take is that we already have 3 different flavors of using the RS and many more different implementations:

- As a pure reactive-stream library: rxjava, akka streams, reactor...
  -- Note that rx/reactor and akka differ in design decisions, with the formers using Publisher at all stage with Subscriber lifting, and the later doing "materialization" and Publisher API at the edge.
- As an adaptable API: spring framework, spring boot, jakarta  EE(?), factory based libs...
- As an edge API (drivers...): jdk http client, jetty, tomcat, vert.x, ratpack...

In these 3 categories, the last 2 ones don't really need anything more than RS utils, often opinionated, and the JDK can build on top of those for various low level concerns (IO). The reactive streams "pure" operational libraries are only chosen to experience some transformation pipelining while connecting two or more reactive-streams ready framework, drivers or Publisher-ready JDK apis. Note that I already conclude about using Publisher based API since the implementation burden is mostly located in them. However some could argue about having Subscriber factories which I consider dangerous given the spec rules around its non reusability and state isolation via Subscription. 
So the point here is that the spec shines for its interop protocol, even inspiring other tech such as JS or .Net, but its variety of flavors would discourage any opinionated take from the JDK.

To date we still have updates to the scope, fixes and optimizations to work on (at least in rx and reactor apis/impls), and we have been implementing RS for already 4 years. Some Reactive Streams libs have added exclusive features such as operator fusion and contextual transportation (thread-local equivalent). Breaking the flow with JDK provided operators with their own opinion on the matter would be an interesting discussion to have. Generally speaking, I wouldn't see that full effort or experience translated into the JDK maintenance cycle.  

Finally I am not a fan of the Stream API or design and I could sidetrack quite a bit on it (especially on the state design or the parallelism api), so the argument about it as a prime example doesn't really resonate. We even have rxjava and reactor users only using our APIs in a synchronous fashion because they just prefer them and because they microbench well. Even with the best common denominator of API, let's say a Monad with filter, map and flatMap, we would still compete between larger implementations and the core JDK. It would then cause many wrapping to bridge between them, loosing context and optimizations each of those libraries try to provide to the user with. 

At the end of the day, IMHO, it's better to expose JDK based Publisher apis as factories, or dedicated edge API for IO and other time-bound data processing.

On Tue, Apr 3, 2018 at 12:53 AM, Viktor Klang via Concurrency-interest <[hidden email]> wrote:


On Fri, Mar 30, 2018 at 2:19 PM, Doug Lea via Concurrency-interest <[hidden email]> wrote:
On 02/28/2018 10:37 PM, James Roper via Concurrency-interest wrote:

> We (Lightbend) would like to put forward a proposal for a Reactive
> Streams utility API for building instances of juc.Flow interfaces. The
> rationale, goals and non goals for this, along with our proposed
> approach to the API, and an actual (incomplete) API proposal, TCK, with
> implementation examples both in Akka Streams and RxJava, can be found here:
>
> https://github.com/lightbend/reactive-streams-utils
>

I initially sat this out hoping that more people using Flow and
reactive-streams would comment.

This is most likely not the right channel to elicit feedback from Flow/RS *end users*—may I suggest that if feedback from end-users is desired that we announce some sort of poll, or otherwise, on a different media such as, but not limited to, Twitter?
 
But not many. And I'm reminded by
my last post about my main reservation, that history shows that
fluent APIs always grow to cover all combinations of all capabilities.
Which is an explicit non-goal of your proposal, but seems
inevitable anyway. This is a little scary from the point of view
creating JDK APIs. A much less ambitious first step would be
to create some static utilities (in class Flow) for map(),
collect() and a few others that could be used by layered
third-party frameworks. A few of these were initially present
in jsr166 prerelease versions but we killed them out of even
more cowardice.

To decide among these and other options, we do need some input
from current and prospective users!

-Doug
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest



--
Cheers,

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
Stephane Maldini | Project Reactor Lead, Spring Engineering | San Francisco | Pivotal


_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest
Reply | Threaded
Open this post in threaded view
|

Re: Reactive Streams Utility API

JSR166 Concurrency mailing list
Hi Stephane!

Great to hear from you :)

On Tue, Apr 3, 2018 at 6:10 PM, Stephane Maldini <[hidden email]> wrote:
I thought these RS API questions in the JDK were already past us since we created RS and argued about already. My take is that we already have 3 different flavors of using the RS and many more different implementations:

- As a pure reactive-stream library: rxjava, akka streams, reactor...
  -- Note that rx/reactor and akka differ in design decisions, with the formers using Publisher at all stage with Subscriber lifting, and the later doing "materialization" and Publisher API at the edge.
- As an adaptable API: spring framework, spring boot, jakarta  EE(?), factory based libs...
- As an edge API (drivers...): jdk http client, jetty, tomcat, vert.x, ratpack...

In these 3 categories, the last 2 ones don't really need anything more than RS utils, often opinionated, and the JDK can build on top of those for various low level concerns (IO). The reactive streams "pure" operational libraries are only chosen to experience some transformation pipelining while connecting two or more reactive-streams ready framework, drivers or Publisher-ready JDK apis. Note that I already conclude about using Publisher based API since the implementation burden is mostly located in them. However some could argue about having Subscriber factories which I consider dangerous given the spec rules around its non reusability and state isolation via Subscription. 

Subscriber factories: Supplier<Subscriber<T>>? I think returning a Subscriber from user code is fine. Taking a Subscriber as a parameter is more challenging in the sense that there's no typelevel indication that it is to be used.

No matter what, and I believe we all agree, is that all implementations of the RS interfaces (and especially those who are exposed to third-party code) conform to the spec and pass the TCK.
 
So the point here is that the spec shines for its interop protocol, even inspiring other tech such as JS or .Net, but its variety of flavors would discourage any opinionated take from the JDK.

I think the JDK could greatly benefit from having an RS "utils" implementation (even if it turns out to be completely internal to the JDK) so that more things in the JDK can add streaming support. That said, without being able to use it—as an application developer—when just having access to the JDK might turn out to be a challenge.
 

To date we still have updates to the scope, fixes and optimizations to work on (at least in rx and reactor apis/impls), and we have been implementing RS for already 4 years.

As they say: Software is never done, only discontinued. :)
 
Some Reactive Streams libs have added exclusive features such as operator fusion

AFAIK all "full" implementations have operator fusion. (Reactor, RxJava, Akka Streams)
 
and contextual transportation (thread-local equivalent).

This functionality would be nice to generalize into a standalone "async local" or similar under the java.util.concurrent flag. Less use of ThreadLocals would be desirable in general. :)
 
Breaking the flow with JDK provided operators with their own opinion on the matter would be an interesting discussion to have. Generally speaking, I wouldn't see that full effort or experience translated into the JDK maintenance cycle.  

Hmmm, could you elaborate?
 

Finally I am not a fan of the Stream API or design and I could sidetrack quite a bit on it (especially on the state design or the parallelism api), so the argument about it as a prime example doesn't really resonate.

The question (IMO) is: Are Java developers in general happy with the Stream API?
 
We even have rxjava and reactor users only using our APIs in a synchronous fashion because they just prefer them and because they microbench well. Even with the best common denominator of API, let's say a Monad with filter, map and flatMap, we would still compete between larger implementations and the core JDK.

 
It would then cause many wrapping to bridge between them, loosing context and optimizations each of those libraries try to provide to the user with. 

Sure—but viewing RS/Flow as an integration API, it would be expected to not get cross-connection optimizations magically happening (since that requires more specific knowledge than simply the RS/Flow interfaces), would it not?
 

At the end of the day, IMHO, it's better to expose JDK based Publisher apis as factories,

Isn't Publisher/Processor/Subscriber on a case-by-case basis?
 
or dedicated edge API for IO and other time-bound data processing.

Having a Flow-enabled Java NIO 3000(tm) would be really nice, the question is if we can get there step-by-step?
 

On Tue, Apr 3, 2018 at 12:53 AM, Viktor Klang via Concurrency-interest <[hidden email]> wrote:


On Fri, Mar 30, 2018 at 2:19 PM, Doug Lea via Concurrency-interest <[hidden email]> wrote:
On 02/28/2018 10:37 PM, James Roper via Concurrency-interest wrote:

> We (Lightbend) would like to put forward a proposal for a Reactive
> Streams utility API for building instances of juc.Flow interfaces. The
> rationale, goals and non goals for this, along with our proposed
> approach to the API, and an actual (incomplete) API proposal, TCK, with
> implementation examples both in Akka Streams and RxJava, can be found here:
>
> https://github.com/lightbend/reactive-streams-utils
>

I initially sat this out hoping that more people using Flow and
reactive-streams would comment.

This is most likely not the right channel to elicit feedback from Flow/RS *end users*—may I suggest that if feedback from end-users is desired that we announce some sort of poll, or otherwise, on a different media such as, but not limited to, Twitter?
 
But not many. And I'm reminded by
my last post about my main reservation, that history shows that
fluent APIs always grow to cover all combinations of all capabilities.
Which is an explicit non-goal of your proposal, but seems
inevitable anyway. This is a little scary from the point of view
creating JDK APIs. A much less ambitious first step would be
to create some static utilities (in class Flow) for map(),
collect() and a few others that could be used by layered
third-party frameworks. A few of these were initially present
in jsr166 prerelease versions but we killed them out of even
more cowardice.

To decide among these and other options, we do need some input
from current and prospective users!

-Doug
_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest



--
Cheers,

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest




--
Stephane Maldini | Project Reactor Lead, Spring Engineering | San Francisco | Pivotal




--
Cheers,

_______________________________________________
Concurrency-interest mailing list
[hidden email]
http://cs.oswego.edu/mailman/listinfo/concurrency-interest