Selector implementation in MB 4

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Selector implementation in MB 4

Riyafa Abdul Hameed-2
Dear all,

This is in reference to the mail titled "[MB4] Some design decisions".
The current selector implementation in MB 3.x.x do not work according to the JMS specifications.
Following quoted from the previous mail:
The issue with current selector implementation is that when the selector of any subscriber do not match a given message it is routed to the DLC to avoid the following issue:
When you have a queue, and consumers filtering the queue with a very restrictive selector you may get into a situation where you won't be able to read more data from paging until you consume messages from the queue.

Example: in one consumer you make a selector as 'color="red"' but you only have one color red 1 millions messages after blue, you won't be able to consume red until you consume blue ones.

The solution implemented was to use multiple cursors with sliding buffer. This solution causes a performance bottleneck, but ensures that the selector functionality works without issues.
Implementation details:
  • A cursor is given to each subscription.
  • The cursor is moved through the queue for each subscription till a message for the selector is found or till the cursor comes to the end of the buffer.
  • Then a minimum cursor that is the smallest cursor from all the bound subscriptions for a queue is selected. All messages before this cursor is removed from the buffer.
  • If a new subscription comes the minimum cursor would become 0 and all the previous messages would be returned to the buffer.
  • The new subscription will then move from the beginning.

There are several concerns that can be raised in this implementation. But there are concerns in all the ways that we could think of in implementing selectors.

The initial plan was to get something working and then proceed to address the issues like performance bottlenecks, starvation of subscribers etc. The current implementation is a working solution. Please find the PR[1].

The performance reduction when using selectors is anticipated[2, 3]. Hence my suggestion is that if someone wants to use selectors it must be configurable and the performance reduction should be acceptable to him. Of course more improvements to the current implementation would be explored and made in the future.

Or we could separate the selector solution from queues that have subscription with no selectors which is a feasible solution and we will be attempting that in the near future.

[1] https://github.com/wso2/andes/pull/905

[2] https://stackoverflow.com/questions/13446072/how-do-jms-selectors-scale-with-queue-depth

[3] https://stackoverflow.com/a/588194/3599535

Thanks.

Riyafa



--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Selector implementation in MB 4

Pamod Sylvester
Hi Riyafa,

Could you elaborate where the bottlenecks are occurring ? 

Thanks,
Pamod 

On Tue, Jun 20, 2017 at 12:04 PM, Riyafa Abdul Hameed <[hidden email]> wrote:
Dear all,

This is in reference to the mail titled "[MB4] Some design decisions".
The current selector implementation in MB 3.x.x do not work according to the JMS specifications.
Following quoted from the previous mail:
The issue with current selector implementation is that when the selector of any subscriber do not match a given message it is routed to the DLC to avoid the following issue:
When you have a queue, and consumers filtering the queue with a very restrictive selector you may get into a situation where you won't be able to read more data from paging until you consume messages from the queue.

Example: in one consumer you make a selector as 'color="red"' but you only have one color red 1 millions messages after blue, you won't be able to consume red until you consume blue ones.

The solution implemented was to use multiple cursors with sliding buffer. This solution causes a performance bottleneck, but ensures that the selector functionality works without issues.
Implementation details:
  • A cursor is given to each subscription.
  • The cursor is moved through the queue for each subscription till a message for the selector is found or till the cursor comes to the end of the buffer.
  • Then a minimum cursor that is the smallest cursor from all the bound subscriptions for a queue is selected. All messages before this cursor is removed from the buffer.
  • If a new subscription comes the minimum cursor would become 0 and all the previous messages would be returned to the buffer.
  • The new subscription will then move from the beginning.

There are several concerns that can be raised in this implementation. But there are concerns in all the ways that we could think of in implementing selectors.

The initial plan was to get something working and then proceed to address the issues like performance bottlenecks, starvation of subscribers etc. The current implementation is a working solution. Please find the PR[1].

The performance reduction when using selectors is anticipated[2, 3]. Hence my suggestion is that if someone wants to use selectors it must be configurable and the performance reduction should be acceptable to him. Of course more improvements to the current implementation would be explored and made in the future.

Or we could separate the selector solution from queues that have subscription with no selectors which is a feasible solution and we will be attempting that in the near future.

[1] https://github.com/wso2/andes/pull/905

[2] https://stackoverflow.com/questions/13446072/how-do-jms-selectors-scale-with-queue-depth

[3] https://stackoverflow.com/a/588194/3599535

Thanks.

Riyafa



--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: +94 77 7779495

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture