[BMB] Full In-memory operating mode for message broker

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[BMB] Full In-memory operating mode for message broker

Asanka Abeyweera
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: +94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [BMB] Full In-memory operating mode for message broker

Pamod Sylvester
Hi Asanka,

If we're considering to handle durable subscriptions in-memory it would be good if we can inform the client on it. 

IMHO Semantically it also could mislead the applications. Otherwise isn't it an option for us to restrict connection creation as durable ? (if the broker is started in, "in-memory" mode)

Thanks,
Pamod

On Fri, Feb 23, 2018 at 2:14 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648





--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: +94 77 7779495

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [BMB] Full In-memory operating mode for message broker

Asanka Abeyweera
I think since this is something the user has to do explicitly (by changing the configuration), we are ok.

On Fri, Feb 23, 2018 at 3:32 PM, Pamod Sylvester <[hidden email]> wrote:
Hi Asanka,

If we're considering to handle durable subscriptions in-memory it would be good if we can inform the client on it. 

IMHO Semantically it also could mislead the applications. Otherwise isn't it an option for us to restrict connection creation as durable ? (if the broker is started in, "in-memory" mode)

Thanks,
Pamod

On Fri, Feb 23, 2018 at 2:14 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648





--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:+94%2077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: +94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [BMB] Full In-memory operating mode for message broker

Hasitha Hiranya
In reply to this post by Pamod Sylvester
Hi Asanka,

I think we should not restrict. 
What we are doing is, configuring MB to work with an "in-memory store". 
It should be a configuration at broker side (applied to all queues/topics).
When coding, better keep all logics same attached to an in-memory impl of the  store. 

Thoughts? 

Thanks

On Fri, Feb 23, 2018 at 3:32 PM, Pamod Sylvester <[hidden email]> wrote:
Hi Asanka,

If we're considering to handle durable subscriptions in-memory it would be good if we can inform the client on it. 

IMHO Semantically it also could mislead the applications. Otherwise isn't it an option for us to restrict connection creation as durable ? (if the broker is started in, "in-memory" mode)

Thanks,
Pamod

On Fri, Feb 23, 2018 at 2:14 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648





--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: +94 77 7779495



--
Hasitha Abeykoon
Associate Technical LeadWSO2, Inc.; http://wso2.com


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [BMB] Full In-memory operating mode for message broker

Sanjeewa Malalgoda
+1 This will be perfect implementation for throttling events like messages( due to high efficiency, small message size, no durable requirements etc).
Do you have any time line estimation for this feature? I believe from user point of view there will be no any changes other than remove database configuration.

Thanks,
sanjeewa.

On Fri, Feb 23, 2018 at 4:31 PM, Hasitha Hiranya <[hidden email]> wrote:
Hi Asanka,

I think we should not restrict. 
What we are doing is, configuring MB to work with an "in-memory store". 
It should be a configuration at broker side (applied to all queues/topics).
When coding, better keep all logics same attached to an in-memory impl of the  store. 

Thoughts? 

Thanks

On Fri, Feb 23, 2018 at 3:32 PM, Pamod Sylvester <[hidden email]> wrote:
Hi Asanka,

If we're considering to handle durable subscriptions in-memory it would be good if we can inform the client on it. 

IMHO Semantically it also could mislead the applications. Otherwise isn't it an option for us to restrict connection creation as durable ? (if the broker is started in, "in-memory" mode)

Thanks,
Pamod

On Fri, Feb 23, 2018 at 2:14 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648





--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
Hasitha Abeykoon
Associate Technical LeadWSO2, Inc.; http://wso2.com


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : +94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [BMB] Full In-memory operating mode for message broker

Asanka Abeyweera
We are planning to complete the feature in this week's milestone. Configuration wise, you will have to set the in-memory mode to "true" in the config. We will explain this in the doc.

On a separate note, When you use non durable queues (for example with non durable topic subscribers) with BMB it will operate in memory (without persisting to database) irrespective of the configured mode. 


On Tue, Feb 27, 2018 at 11:40 AM, Sanjeewa Malalgoda <[hidden email]> wrote:
+1 This will be perfect implementation for throttling events like messages( due to high efficiency, small message size, no durable requirements etc).
Do you have any time line estimation for this feature? I believe from user point of view there will be no any changes other than remove database configuration.

Thanks,
sanjeewa.

On Fri, Feb 23, 2018 at 4:31 PM, Hasitha Hiranya <[hidden email]> wrote:
Hi Asanka,

I think we should not restrict. 
What we are doing is, configuring MB to work with an "in-memory store". 
It should be a configuration at broker side (applied to all queues/topics).
When coding, better keep all logics same attached to an in-memory impl of the  store. 

Thoughts? 

Thanks

On Fri, Feb 23, 2018 at 3:32 PM, Pamod Sylvester <[hidden email]> wrote:
Hi Asanka,

If we're considering to handle durable subscriptions in-memory it would be good if we can inform the client on it. 

IMHO Semantically it also could mislead the applications. Otherwise isn't it an option for us to restrict connection creation as durable ? (if the broker is started in, "in-memory" mode)

Thanks,
Pamod

On Fri, Feb 23, 2018 at 2:14 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648





--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
Hasitha Abeykoon
Associate Technical LeadWSO2, Inc.; http://wso2.com


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : <a href="tel:+94%2071%20306%208779" value="+94713068779" target="_blank">+94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: +94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [BMB] Full In-memory operating mode for message broker

Shazni Nazeer
Hi Asanka,

How would the in-memory storage be handled between high availability scenario where we have two MB nodes used? Do we have any distributed caching of some sort? 

On Tue, Feb 27, 2018 at 12:30 AM, Asanka Abeyweera <[hidden email]> wrote:
We are planning to complete the feature in this week's milestone. Configuration wise, you will have to set the in-memory mode to "true" in the config. We will explain this in the doc.

On a separate note, When you use non durable queues (for example with non durable topic subscribers) with BMB it will operate in memory (without persisting to database) irrespective of the configured mode. 


On Tue, Feb 27, 2018 at 11:40 AM, Sanjeewa Malalgoda <[hidden email]> wrote:
+1 This will be perfect implementation for throttling events like messages( due to high efficiency, small message size, no durable requirements etc).
Do you have any time line estimation for this feature? I believe from user point of view there will be no any changes other than remove database configuration.

Thanks,
sanjeewa.

On Fri, Feb 23, 2018 at 4:31 PM, Hasitha Hiranya <[hidden email]> wrote:
Hi Asanka,

I think we should not restrict. 
What we are doing is, configuring MB to work with an "in-memory store". 
It should be a configuration at broker side (applied to all queues/topics).
When coding, better keep all logics same attached to an in-memory impl of the  store. 

Thoughts? 

Thanks

On Fri, Feb 23, 2018 at 3:32 PM, Pamod Sylvester <[hidden email]> wrote:
Hi Asanka,

If we're considering to handle durable subscriptions in-memory it would be good if we can inform the client on it. 

IMHO Semantically it also could mislead the applications. Otherwise isn't it an option for us to restrict connection creation as durable ? (if the broker is started in, "in-memory" mode)

Thanks,
Pamod

On Fri, Feb 23, 2018 at 2:14 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648





--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
Hasitha Abeykoon
Associate Technical LeadWSO2, Inc.; http://wso2.com


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : <a href="tel:+94%2071%20306%208779" value="+94713068779" target="_blank">+94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [BMB] Full In-memory operating mode for message broker

Asanka Abeyweera
HA is not supported in in-memory mode. This is a non-reliable broker mode (unless your memory/system is reliable ;)). If you need to have HA for your use case you should not use in-memory mode.

On Wed, Feb 28, 2018 at 10:19 AM, Shazni Nazeer <[hidden email]> wrote:
Hi Asanka,

How would the in-memory storage be handled between high availability scenario where we have two MB nodes used? Do we have any distributed caching of some sort? 

On Tue, Feb 27, 2018 at 12:30 AM, Asanka Abeyweera <[hidden email]> wrote:
We are planning to complete the feature in this week's milestone. Configuration wise, you will have to set the in-memory mode to "true" in the config. We will explain this in the doc.

On a separate note, When you use non durable queues (for example with non durable topic subscribers) with BMB it will operate in memory (without persisting to database) irrespective of the configured mode. 


On Tue, Feb 27, 2018 at 11:40 AM, Sanjeewa Malalgoda <[hidden email]> wrote:
+1 This will be perfect implementation for throttling events like messages( due to high efficiency, small message size, no durable requirements etc).
Do you have any time line estimation for this feature? I believe from user point of view there will be no any changes other than remove database configuration.

Thanks,
sanjeewa.

On Fri, Feb 23, 2018 at 4:31 PM, Hasitha Hiranya <[hidden email]> wrote:
Hi Asanka,

I think we should not restrict. 
What we are doing is, configuring MB to work with an "in-memory store". 
It should be a configuration at broker side (applied to all queues/topics).
When coding, better keep all logics same attached to an in-memory impl of the  store. 

Thoughts? 

Thanks

On Fri, Feb 23, 2018 at 3:32 PM, Pamod Sylvester <[hidden email]> wrote:
Hi Asanka,

If we're considering to handle durable subscriptions in-memory it would be good if we can inform the client on it. 

IMHO Semantically it also could mislead the applications. Otherwise isn't it an option for us to restrict connection creation as durable ? (if the broker is started in, "in-memory" mode)

Thanks,
Pamod

On Fri, Feb 23, 2018 at 2:14 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648





--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
Hasitha Abeykoon
Associate Technical LeadWSO2, Inc.; http://wso2.com


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : <a href="tel:+94%2071%20306%208779" value="+94713068779" target="_blank">+94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: +94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [BMB] Full In-memory operating mode for message broker

Shazni Nazeer
yeah that makes sense. No memory system is reliable ☺. But certainly good to have for some scenarios like the throttling events mentioned above.

On Tue, Feb 27, 2018 at 11:24 PM, Asanka Abeyweera <[hidden email]> wrote:
HA is not supported in in-memory mode. This is a non-reliable broker mode (unless your memory/system is reliable ;)). If you need to have HA for your use case you should not use in-memory mode.

On Wed, Feb 28, 2018 at 10:19 AM, Shazni Nazeer <[hidden email]> wrote:
Hi Asanka,

How would the in-memory storage be handled between high availability scenario where we have two MB nodes used? Do we have any distributed caching of some sort? 

On Tue, Feb 27, 2018 at 12:30 AM, Asanka Abeyweera <[hidden email]> wrote:
We are planning to complete the feature in this week's milestone. Configuration wise, you will have to set the in-memory mode to "true" in the config. We will explain this in the doc.

On a separate note, When you use non durable queues (for example with non durable topic subscribers) with BMB it will operate in memory (without persisting to database) irrespective of the configured mode. 


On Tue, Feb 27, 2018 at 11:40 AM, Sanjeewa Malalgoda <[hidden email]> wrote:
+1 This will be perfect implementation for throttling events like messages( due to high efficiency, small message size, no durable requirements etc).
Do you have any time line estimation for this feature? I believe from user point of view there will be no any changes other than remove database configuration.

Thanks,
sanjeewa.

On Fri, Feb 23, 2018 at 4:31 PM, Hasitha Hiranya <[hidden email]> wrote:
Hi Asanka,

I think we should not restrict. 
What we are doing is, configuring MB to work with an "in-memory store". 
It should be a configuration at broker side (applied to all queues/topics).
When coding, better keep all logics same attached to an in-memory impl of the  store. 

Thoughts? 

Thanks

On Fri, Feb 23, 2018 at 3:32 PM, Pamod Sylvester <[hidden email]> wrote:
Hi Asanka,

If we're considering to handle durable subscriptions in-memory it would be good if we can inform the client on it. 

IMHO Semantically it also could mislead the applications. Otherwise isn't it an option for us to restrict connection creation as durable ? (if the broker is started in, "in-memory" mode)

Thanks,
Pamod

On Fri, Feb 23, 2018 at 2:14 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648





--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
Hasitha Abeykoon
Associate Technical LeadWSO2, Inc.; http://wso2.com


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : <a href="tel:+94%2071%20306%208779" value="+94713068779" target="_blank">+94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [BMB] Full In-memory operating mode for message broker

Chanaka Fernando
Hi Asanka,

Could you please list down the advantages of this in-memory mode over database backed (default) mode? We used to have in-memory JMS store in the WSO2 ESB/EI world but it was never used in a real production system (in fact we recommended not to use that in production). More or less, it was used for demonstration purposes. I'm a bit unclear on why we need an in-memory mode without any database backup. 

In this mode, if the node goes down, is there a way to write the status periodically to a database and revert to the last active state once the server restarted?  

Thanks,
Chanaka

On Wed, Feb 28, 2018 at 11:05 AM, Shazni Nazeer <[hidden email]> wrote:
yeah that makes sense. No memory system is reliable ☺. But certainly good to have for some scenarios like the throttling events mentioned above.

On Tue, Feb 27, 2018 at 11:24 PM, Asanka Abeyweera <[hidden email]> wrote:
HA is not supported in in-memory mode. This is a non-reliable broker mode (unless your memory/system is reliable ;)). If you need to have HA for your use case you should not use in-memory mode.

On Wed, Feb 28, 2018 at 10:19 AM, Shazni Nazeer <[hidden email]> wrote:
Hi Asanka,

How would the in-memory storage be handled between high availability scenario where we have two MB nodes used? Do we have any distributed caching of some sort? 

On Tue, Feb 27, 2018 at 12:30 AM, Asanka Abeyweera <[hidden email]> wrote:
We are planning to complete the feature in this week's milestone. Configuration wise, you will have to set the in-memory mode to "true" in the config. We will explain this in the doc.

On a separate note, When you use non durable queues (for example with non durable topic subscribers) with BMB it will operate in memory (without persisting to database) irrespective of the configured mode. 


On Tue, Feb 27, 2018 at 11:40 AM, Sanjeewa Malalgoda <[hidden email]> wrote:
+1 This will be perfect implementation for throttling events like messages( due to high efficiency, small message size, no durable requirements etc).
Do you have any time line estimation for this feature? I believe from user point of view there will be no any changes other than remove database configuration.

Thanks,
sanjeewa.

On Fri, Feb 23, 2018 at 4:31 PM, Hasitha Hiranya <[hidden email]> wrote:
Hi Asanka,

I think we should not restrict. 
What we are doing is, configuring MB to work with an "in-memory store". 
It should be a configuration at broker side (applied to all queues/topics).
When coding, better keep all logics same attached to an in-memory impl of the  store. 

Thoughts? 

Thanks

On Fri, Feb 23, 2018 at 3:32 PM, Pamod Sylvester <[hidden email]> wrote:
Hi Asanka,

If we're considering to handle durable subscriptions in-memory it would be good if we can inform the client on it. 

IMHO Semantically it also could mislead the applications. Otherwise isn't it an option for us to restrict connection creation as durable ? (if the broker is started in, "in-memory" mode)

Thanks,
Pamod

On Fri, Feb 23, 2018 at 2:14 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648





--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
Hasitha Abeykoon
Associate Technical LeadWSO2, Inc.; http://wso2.com


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : <a href="tel:+94%2071%20306%208779" value="+94713068779" target="_blank">+94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Thank you and Best Regards,
Chanaka Fernando
Senior Technical Lead
m: +94 773337238





_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [BMB] Full In-memory operating mode for message broker

Asanka Abeyweera
Hi Chanaka,

Yes, We would lose all data if the node goes down.

This mode is mainly targeted for the embedded broker use cases. There can be scenarios where we only have publishers and non-durable topic subscribers. For such scenarios, it does not matter if we have configured a database or not ,since the underline queue is non-durable.This is different from the message store use case where we expect messages to be persisted. Previously even for non-durable use cases, we had to configure a database. By implementing this feature we are trying to avoid that.

Another use case is dev and unit testing as you have mentioned.


On Mon, Mar 5, 2018 at 2:07 PM, Chanaka Fernando <[hidden email]> wrote:
Hi Asanka,

Could you please list down the advantages of this in-memory mode over database backed (default) mode? We used to have in-memory JMS store in the WSO2 ESB/EI world but it was never used in a real production system (in fact we recommended not to use that in production). More or less, it was used for demonstration purposes. I'm a bit unclear on why we need an in-memory mode without any database backup. 

In this mode, if the node goes down, is there a way to write the status periodically to a database and revert to the last active state once the server restarted?  

Thanks,
Chanaka

On Wed, Feb 28, 2018 at 11:05 AM, Shazni Nazeer <[hidden email]> wrote:
yeah that makes sense. No memory system is reliable ☺. But certainly good to have for some scenarios like the throttling events mentioned above.

On Tue, Feb 27, 2018 at 11:24 PM, Asanka Abeyweera <[hidden email]> wrote:
HA is not supported in in-memory mode. This is a non-reliable broker mode (unless your memory/system is reliable ;)). If you need to have HA for your use case you should not use in-memory mode.

On Wed, Feb 28, 2018 at 10:19 AM, Shazni Nazeer <[hidden email]> wrote:
Hi Asanka,

How would the in-memory storage be handled between high availability scenario where we have two MB nodes used? Do we have any distributed caching of some sort? 

On Tue, Feb 27, 2018 at 12:30 AM, Asanka Abeyweera <[hidden email]> wrote:
We are planning to complete the feature in this week's milestone. Configuration wise, you will have to set the in-memory mode to "true" in the config. We will explain this in the doc.

On a separate note, When you use non durable queues (for example with non durable topic subscribers) with BMB it will operate in memory (without persisting to database) irrespective of the configured mode. 


On Tue, Feb 27, 2018 at 11:40 AM, Sanjeewa Malalgoda <[hidden email]> wrote:
+1 This will be perfect implementation for throttling events like messages( due to high efficiency, small message size, no durable requirements etc).
Do you have any time line estimation for this feature? I believe from user point of view there will be no any changes other than remove database configuration.

Thanks,
sanjeewa.

On Fri, Feb 23, 2018 at 4:31 PM, Hasitha Hiranya <[hidden email]> wrote:
Hi Asanka,

I think we should not restrict. 
What we are doing is, configuring MB to work with an "in-memory store". 
It should be a configuration at broker side (applied to all queues/topics).
When coding, better keep all logics same attached to an in-memory impl of the  store. 

Thoughts? 

Thanks

On Fri, Feb 23, 2018 at 3:32 PM, Pamod Sylvester <[hidden email]> wrote:
Hi Asanka,

If we're considering to handle durable subscriptions in-memory it would be good if we can inform the client on it. 

IMHO Semantically it also could mislead the applications. Otherwise isn't it an option for us to restrict connection creation as durable ? (if the broker is started in, "in-memory" mode)

Thanks,
Pamod

On Fri, Feb 23, 2018 at 2:14 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi all,

I am working on the $subject. In the previous versions of message broker, we configured H2 database in the in-memory mode to run the broker in in-memory mode. But we are thinking of having a mode where we would skip the database layer completely when in-memory mode is enabled in the broker. This means we will be doing all of the following tasks in memory.
  • Persistent message storing
  • Durable queue information storing
  • Durable exchange information storing
  • Durable binding information storing

Alternatively, we could reject all durable calls when we put the broker in in-memory mode. But If we do that we will be limiting some of the existing applications from using the message broker in in-memory mode. Therefore I think It is better not to that.

WDYT?

--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648





--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: <a href="tel:077%20777%209495" value="+94777779495" target="_blank">+94 77 7779495



--
Hasitha Abeykoon
Associate Technical LeadWSO2, Inc.; http://wso2.com


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

Sanjeewa Malalgoda
WSO2 Inc.
Mobile : <a href="tel:+94%2071%20306%208779" value="+94713068779" target="_blank">+94713068779

blog :http://sanjeewamalalgoda.blogspot.com/



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Thank you and Best Regards,
Chanaka Fernando
Senior Technical Lead
m: <a href="tel:+94%2077%20333%207238" value="+94773337238" target="_blank">+94 773337238





_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: +94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture