[MB4] HA Support

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[MB4] HA Support

Maryam Ziyad
Hi All,

We are currently working on introducing $subject [1]. Please find below a high level description of the approach.

An extension point (HaStrategy) will be introduced, allowing straightforward introduction of different implementations of identification of the active node, where the only requirements would be that these approaches extend the common class and invoke particular methods when the node state changes. 

The broker-core and broker-transport (broker-amqp) modules would introduce listeners to receive notifications of node states changes (active/passive), and change behaviour accordingly.




Configuration

The HA related configuration would be specified in the broker.yaml file including whether HA is enabled and the HA strategy to use.
haConfig:
enabled: true
strategy: org.wso2.broker.coordination.rdbms.RdbmsHaStrategy

The basic/initial HA strategy implementation will be the RdbmsHaStrategy based on the RDBMS based coordinator election approach previously introduced for MB 3.2.0. [2, 3]. ​If HA enabled is set to true but no strategy is specified, the RdbmsHaStrategy will be used. 


RDBMS Coordinator Election based HA Strategy (RdbmsHaStrategy)

The RDBMS based coordinator election algorithm would be extended to provide HA support, by specifying the node elected as coordinator to always be the active node, while the other node(s) will be considered passive. The RDBMS coordinator election based approach, which would also be the default HA strategy, would require the nodes in the HA group to share the same database. All MB nodes pointing to this shared database will be considered as MBs belonging to the same group, and at any given point only one of the nodes will be considered active.

Feedback on the approach would be highly appreciated.

[2] Mail: "[Architecture] RDBMS based coordinator election algorithm for MB"

Thank you,
Maryam
--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [MB4] HA Support

Asanka Abeyweera
Hi Maryam,

Shall we rename the "haConfig" to "ha-clustering"? I'm not sure if we should use camel case in the yaml config.

On Tue, Dec 19, 2017 at 4:42 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

We are currently working on introducing $subject [1]. Please find below a high level description of the approach.

An extension point (HaStrategy) will be introduced, allowing straightforward introduction of different implementations of identification of the active node, where the only requirements would be that these approaches extend the common class and invoke particular methods when the node state changes. 

The broker-core and broker-transport (broker-amqp) modules would introduce listeners to receive notifications of node states changes (active/passive), and change behaviour accordingly.




Configuration

The HA related configuration would be specified in the broker.yaml file including whether HA is enabled and the HA strategy to use.
haConfig:
enabled: true
strategy: org.wso2.broker.coordination.rdbms.RdbmsHaStrategy

The basic/initial HA strategy implementation will be the RdbmsHaStrategy based on the RDBMS based coordinator election approach previously introduced for MB 3.2.0. [2, 3]. ​If HA enabled is set to true but no strategy is specified, the RdbmsHaStrategy will be used. 


RDBMS Coordinator Election based HA Strategy (RdbmsHaStrategy)

The RDBMS based coordinator election algorithm would be extended to provide HA support, by specifying the node elected as coordinator to always be the active node, while the other node(s) will be considered passive. The RDBMS coordinator election based approach, which would also be the default HA strategy, would require the nodes in the HA group to share the same database. All MB nodes pointing to this shared database will be considered as MBs belonging to the same group, and at any given point only one of the nodes will be considered active.

Feedback on the approach would be highly appreciated.

[2] Mail: "[Architecture] RDBMS based coordinator election algorithm for MB"

Thank you,
Maryam
--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: +94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [MB4] HA Support

Maryam Ziyad
Hi Asanka,

Renamed "haConfig" to "failover" based on the offline discussion.

Thank you,
Maryam

On Tue, Dec 19, 2017 at 7:05 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Shall we rename the "haConfig" to "ha-clustering"? I'm not sure if we should use camel case in the yaml config.

On Tue, Dec 19, 2017 at 4:42 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

We are currently working on introducing $subject [1]. Please find below a high level description of the approach.

An extension point (HaStrategy) will be introduced, allowing straightforward introduction of different implementations of identification of the active node, where the only requirements would be that these approaches extend the common class and invoke particular methods when the node state changes. 

The broker-core and broker-transport (broker-amqp) modules would introduce listeners to receive notifications of node states changes (active/passive), and change behaviour accordingly.




Configuration

The HA related configuration would be specified in the broker.yaml file including whether HA is enabled and the HA strategy to use.
haConfig:
enabled: true
strategy: org.wso2.broker.coordination.rdbms.RdbmsHaStrategy

The basic/initial HA strategy implementation will be the RdbmsHaStrategy based on the RDBMS based coordinator election approach previously introduced for MB 3.2.0. [2, 3]. ​If HA enabled is set to true but no strategy is specified, the RdbmsHaStrategy will be used. 


RDBMS Coordinator Election based HA Strategy (RdbmsHaStrategy)

The RDBMS based coordinator election algorithm would be extended to provide HA support, by specifying the node elected as coordinator to always be the active node, while the other node(s) will be considered passive. The RDBMS coordinator election based approach, which would also be the default HA strategy, would require the nodes in the HA group to share the same database. All MB nodes pointing to this shared database will be considered as MBs belonging to the same group, and at any given point only one of the nodes will be considered active.

Feedback on the approach would be highly appreciated.

[2] Mail: "[Architecture] RDBMS based coordinator election algorithm for MB"

Thank you,
Maryam
--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [MB4] HA Support

Maryam Ziyad
Hi All,

As mentioned, an extensible point "org.wso2.broker.coordination.HaStrategy" has been introduced, which can be implemented to provide variations upon which HA support will be based. 

Any new/custom implementation would have to notify the listeners listening on node state changes (when a node state changes from active to passive or passive to active). 

Currently the modules listening on state changes are as follows:
  • broker-core
  • broker-transport
  • broker-rest-runner
All nodes will start up in "passive" mode, and only change state to "active" on notification by the HA strategy. This requires that the listeners are registered prior to starting the HA strategy (identification of the active node).


Default RDBMS Coordinator Election based HA Strategy 

With the default HA strategy (RdbmsHaStrategy - implemented based on the RDBMS based coordinator election approach [1]), the possible coordination node states are mapped to active/passive as follows:
  • COORDINATOR - Active
  • CANDIDATE - Passive
  • ELECTION - Passive
Thus notification of the HA listeners happens when:
  • election is triggered and the node was previously the coordinator node (active → passive)
  • election resulted in the node becoming the coordinator node (passive → active)

Feedback/suggestions would be highly appreciated.

Thank you,
Maryam

On Fri, Jan 12, 2018 at 6:41 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Asanka,

Renamed "haConfig" to "failover" based on the offline discussion.

Thank you,
Maryam

On Tue, Dec 19, 2017 at 7:05 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Shall we rename the "haConfig" to "ha-clustering"? I'm not sure if we should use camel case in the yaml config.

On Tue, Dec 19, 2017 at 4:42 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

We are currently working on introducing $subject [1]. Please find below a high level description of the approach.

An extension point (HaStrategy) will be introduced, allowing straightforward introduction of different implementations of identification of the active node, where the only requirements would be that these approaches extend the common class and invoke particular methods when the node state changes. 

The broker-core and broker-transport (broker-amqp) modules would introduce listeners to receive notifications of node states changes (active/passive), and change behaviour accordingly.




Configuration

The HA related configuration would be specified in the broker.yaml file including whether HA is enabled and the HA strategy to use.
haConfig:
enabled: true
strategy: org.wso2.broker.coordination.rdbms.RdbmsHaStrategy

The basic/initial HA strategy implementation will be the RdbmsHaStrategy based on the RDBMS based coordinator election approach previously introduced for MB 3.2.0. [2, 3]. ​If HA enabled is set to true but no strategy is specified, the RdbmsHaStrategy will be used. 


RDBMS Coordinator Election based HA Strategy (RdbmsHaStrategy)

The RDBMS based coordinator election algorithm would be extended to provide HA support, by specifying the node elected as coordinator to always be the active node, while the other node(s) will be considered passive. The RDBMS coordinator election based approach, which would also be the default HA strategy, would require the nodes in the HA group to share the same database. All MB nodes pointing to this shared database will be considered as MBs belonging to the same group, and at any given point only one of the nodes will be considered active.

Feedback on the approach would be highly appreciated.

[2] Mail: "[Architecture] RDBMS based coordinator election algorithm for MB"

Thank you,
Maryam
--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature



--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [MB4] HA Support

Asanka Abeyweera
Hi Maryam,

Are we keeping the passive node in sync with the active node or are we reloading context and message data when a passive node becomes active?

On Fri, Jan 12, 2018 at 6:46 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

As mentioned, an extensible point "org.wso2.broker.coordination.HaStrategy" has been introduced, which can be implemented to provide variations upon which HA support will be based. 

Any new/custom implementation would have to notify the listeners listening on node state changes (when a node state changes from active to passive or passive to active). 

Currently the modules listening on state changes are as follows:
  • broker-core
  • broker-transport
  • broker-rest-runner
All nodes will start up in "passive" mode, and only change state to "active" on notification by the HA strategy. This requires that the listeners are registered prior to starting the HA strategy (identification of the active node).


Default RDBMS Coordinator Election based HA Strategy 

With the default HA strategy (RdbmsHaStrategy - implemented based on the RDBMS based coordinator election approach [1]), the possible coordination node states are mapped to active/passive as follows:
  • COORDINATOR - Active
  • CANDIDATE - Passive
  • ELECTION - Passive
Thus notification of the HA listeners happens when:
  • election is triggered and the node was previously the coordinator node (active → passive)
  • election resulted in the node becoming the coordinator node (passive → active)

Feedback/suggestions would be highly appreciated.

Thank you,
Maryam

On Fri, Jan 12, 2018 at 6:41 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Asanka,

Renamed "haConfig" to "failover" based on the offline discussion.

Thank you,
Maryam

On Tue, Dec 19, 2017 at 7:05 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Shall we rename the "haConfig" to "ha-clustering"? I'm not sure if we should use camel case in the yaml config.

On Tue, Dec 19, 2017 at 4:42 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

We are currently working on introducing $subject [1]. Please find below a high level description of the approach.

An extension point (HaStrategy) will be introduced, allowing straightforward introduction of different implementations of identification of the active node, where the only requirements would be that these approaches extend the common class and invoke particular methods when the node state changes. 

The broker-core and broker-transport (broker-amqp) modules would introduce listeners to receive notifications of node states changes (active/passive), and change behaviour accordingly.




Configuration

The HA related configuration would be specified in the broker.yaml file including whether HA is enabled and the HA strategy to use.
haConfig:
enabled: true
strategy: org.wso2.broker.coordination.rdbms.RdbmsHaStrategy

The basic/initial HA strategy implementation will be the RdbmsHaStrategy based on the RDBMS based coordinator election approach previously introduced for MB 3.2.0. [2, 3]. ​If HA enabled is set to true but no strategy is specified, the RdbmsHaStrategy will be used. 


RDBMS Coordinator Election based HA Strategy (RdbmsHaStrategy)

The RDBMS based coordinator election algorithm would be extended to provide HA support, by specifying the node elected as coordinator to always be the active node, while the other node(s) will be considered passive. The RDBMS coordinator election based approach, which would also be the default HA strategy, would require the nodes in the HA group to share the same database. All MB nodes pointing to this shared database will be considered as MBs belonging to the same group, and at any given point only one of the nodes will be considered active.

Feedback on the approach would be highly appreciated.

[2] Mail: "[Architecture] RDBMS based coordinator election algorithm for MB"

Thank you,
Maryam
--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature



--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: +94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [MB4] HA Support

Maryam Ziyad
Hi Asanka,

The required queue/binding/exchange related data will be loaded once the previously passive node is notified of becoming the active node.

This is considering the possibility of inconsistencies in a continuous sync, which could also require reloading on becoming the active node. 

Thank you,
Maryam

On Mon, Jan 15, 2018 at 8:14 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Are we keeping the passive node in sync with the active node or are we reloading context and message data when a passive node becomes active?

On Fri, Jan 12, 2018 at 6:46 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

As mentioned, an extensible point "org.wso2.broker.coordination.HaStrategy" has been introduced, which can be implemented to provide variations upon which HA support will be based. 

Any new/custom implementation would have to notify the listeners listening on node state changes (when a node state changes from active to passive or passive to active). 

Currently the modules listening on state changes are as follows:
  • broker-core
  • broker-transport
  • broker-rest-runner
All nodes will start up in "passive" mode, and only change state to "active" on notification by the HA strategy. This requires that the listeners are registered prior to starting the HA strategy (identification of the active node).


Default RDBMS Coordinator Election based HA Strategy 

With the default HA strategy (RdbmsHaStrategy - implemented based on the RDBMS based coordinator election approach [1]), the possible coordination node states are mapped to active/passive as follows:
  • COORDINATOR - Active
  • CANDIDATE - Passive
  • ELECTION - Passive
Thus notification of the HA listeners happens when:
  • election is triggered and the node was previously the coordinator node (active → passive)
  • election resulted in the node becoming the coordinator node (passive → active)

Feedback/suggestions would be highly appreciated.

Thank you,
Maryam

On Fri, Jan 12, 2018 at 6:41 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Asanka,

Renamed "haConfig" to "failover" based on the offline discussion.

Thank you,
Maryam

On Tue, Dec 19, 2017 at 7:05 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Shall we rename the "haConfig" to "ha-clustering"? I'm not sure if we should use camel case in the yaml config.

On Tue, Dec 19, 2017 at 4:42 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

We are currently working on introducing $subject [1]. Please find below a high level description of the approach.

An extension point (HaStrategy) will be introduced, allowing straightforward introduction of different implementations of identification of the active node, where the only requirements would be that these approaches extend the common class and invoke particular methods when the node state changes. 

The broker-core and broker-transport (broker-amqp) modules would introduce listeners to receive notifications of node states changes (active/passive), and change behaviour accordingly.




Configuration

The HA related configuration would be specified in the broker.yaml file including whether HA is enabled and the HA strategy to use.
haConfig:
enabled: true
strategy: org.wso2.broker.coordination.rdbms.RdbmsHaStrategy

The basic/initial HA strategy implementation will be the RdbmsHaStrategy based on the RDBMS based coordinator election approach previously introduced for MB 3.2.0. [2, 3]. ​If HA enabled is set to true but no strategy is specified, the RdbmsHaStrategy will be used. 


RDBMS Coordinator Election based HA Strategy (RdbmsHaStrategy)

The RDBMS based coordinator election algorithm would be extended to provide HA support, by specifying the node elected as coordinator to always be the active node, while the other node(s) will be considered passive. The RDBMS coordinator election based approach, which would also be the default HA strategy, would require the nodes in the HA group to share the same database. All MB nodes pointing to this shared database will be considered as MBs belonging to the same group, and at any given point only one of the nodes will be considered active.

Feedback on the approach would be highly appreciated.

[2] Mail: "[Architecture] RDBMS based coordinator election algorithm for MB"

Thank you,
Maryam
--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature



--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [MB4] HA Support

Waruna Jayaweera
Hi Maryam,

We also store the permissions related data in both memory and persistent storage but permissions will be retrieved from memory. In that case if new permissions added to active node, passive node will have inconsistency with permissions. 
I hope we can use same above approach for permissions as well.

Thanks,
Waruna


  


On Fri, Jan 19, 2018 at 6:01 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Asanka,

The required queue/binding/exchange related data will be loaded once the previously passive node is notified of becoming the active node.

This is considering the possibility of inconsistencies in a continuous sync, which could also require reloading on becoming the active node. 

Thank you,
Maryam

On Mon, Jan 15, 2018 at 8:14 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Are we keeping the passive node in sync with the active node or are we reloading context and message data when a passive node becomes active?

On Fri, Jan 12, 2018 at 6:46 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

As mentioned, an extensible point "org.wso2.broker.coordination.HaStrategy" has been introduced, which can be implemented to provide variations upon which HA support will be based. 

Any new/custom implementation would have to notify the listeners listening on node state changes (when a node state changes from active to passive or passive to active). 

Currently the modules listening on state changes are as follows:
  • broker-core
  • broker-transport
  • broker-rest-runner
All nodes will start up in "passive" mode, and only change state to "active" on notification by the HA strategy. This requires that the listeners are registered prior to starting the HA strategy (identification of the active node).


Default RDBMS Coordinator Election based HA Strategy 

With the default HA strategy (RdbmsHaStrategy - implemented based on the RDBMS based coordinator election approach [1]), the possible coordination node states are mapped to active/passive as follows:
  • COORDINATOR - Active
  • CANDIDATE - Passive
  • ELECTION - Passive
Thus notification of the HA listeners happens when:
  • election is triggered and the node was previously the coordinator node (active → passive)
  • election resulted in the node becoming the coordinator node (passive → active)

Feedback/suggestions would be highly appreciated.

Thank you,
Maryam

On Fri, Jan 12, 2018 at 6:41 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Asanka,

Renamed "haConfig" to "failover" based on the offline discussion.

Thank you,
Maryam

On Tue, Dec 19, 2017 at 7:05 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Shall we rename the "haConfig" to "ha-clustering"? I'm not sure if we should use camel case in the yaml config.

On Tue, Dec 19, 2017 at 4:42 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

We are currently working on introducing $subject [1]. Please find below a high level description of the approach.

An extension point (HaStrategy) will be introduced, allowing straightforward introduction of different implementations of identification of the active node, where the only requirements would be that these approaches extend the common class and invoke particular methods when the node state changes. 

The broker-core and broker-transport (broker-amqp) modules would introduce listeners to receive notifications of node states changes (active/passive), and change behaviour accordingly.




Configuration

The HA related configuration would be specified in the broker.yaml file including whether HA is enabled and the HA strategy to use.
haConfig:
enabled: true
strategy: org.wso2.broker.coordination.rdbms.RdbmsHaStrategy

The basic/initial HA strategy implementation will be the RdbmsHaStrategy based on the RDBMS based coordinator election approach previously introduced for MB 3.2.0. [2, 3]. ​If HA enabled is set to true but no strategy is specified, the RdbmsHaStrategy will be used. 


RDBMS Coordinator Election based HA Strategy (RdbmsHaStrategy)

The RDBMS based coordinator election algorithm would be extended to provide HA support, by specifying the node elected as coordinator to always be the active node, while the other node(s) will be considered passive. The RDBMS coordinator election based approach, which would also be the default HA strategy, would require the nodes in the HA group to share the same database. All MB nodes pointing to this shared database will be considered as MBs belonging to the same group, and at any given point only one of the nodes will be considered active.

Feedback on the approach would be highly appreciated.

[2] Mail: "[Architecture] RDBMS based coordinator election algorithm for MB"

Thank you,
Maryam
--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature



--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Regards,

Waruna Lakshitha Jayaweera
Senior Software Engineer
WSO2 Inc; http://wso2.com
phone: +94713255198


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [MB4] HA Support

Gihan Anuruddha
Hi All,

Without reloading everything when node becomes active from passive state, is there any possibility of doing lazy loading specially in permission area? This will speed up the server readiness to serve live traffic.

Regards,
Gihan

On Tue, Jan 23, 2018 at 6:57 PM, Waruna Jayaweera <[hidden email]> wrote:
Hi Maryam,

We also store the permissions related data in both memory and persistent storage but permissions will be retrieved from memory. In that case if new permissions added to active node, passive node will have inconsistency with permissions. 
I hope we can use same above approach for permissions as well.

Thanks,
Waruna


  


On Fri, Jan 19, 2018 at 6:01 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Asanka,

The required queue/binding/exchange related data will be loaded once the previously passive node is notified of becoming the active node.

This is considering the possibility of inconsistencies in a continuous sync, which could also require reloading on becoming the active node. 

Thank you,
Maryam

On Mon, Jan 15, 2018 at 8:14 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Are we keeping the passive node in sync with the active node or are we reloading context and message data when a passive node becomes active?

On Fri, Jan 12, 2018 at 6:46 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

As mentioned, an extensible point "org.wso2.broker.coordination.HaStrategy" has been introduced, which can be implemented to provide variations upon which HA support will be based. 

Any new/custom implementation would have to notify the listeners listening on node state changes (when a node state changes from active to passive or passive to active). 

Currently the modules listening on state changes are as follows:
  • broker-core
  • broker-transport
  • broker-rest-runner
All nodes will start up in "passive" mode, and only change state to "active" on notification by the HA strategy. This requires that the listeners are registered prior to starting the HA strategy (identification of the active node).


Default RDBMS Coordinator Election based HA Strategy 

With the default HA strategy (RdbmsHaStrategy - implemented based on the RDBMS based coordinator election approach [1]), the possible coordination node states are mapped to active/passive as follows:
  • COORDINATOR - Active
  • CANDIDATE - Passive
  • ELECTION - Passive
Thus notification of the HA listeners happens when:
  • election is triggered and the node was previously the coordinator node (active → passive)
  • election resulted in the node becoming the coordinator node (passive → active)

Feedback/suggestions would be highly appreciated.

Thank you,
Maryam

On Fri, Jan 12, 2018 at 6:41 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Asanka,

Renamed "haConfig" to "failover" based on the offline discussion.

Thank you,
Maryam

On Tue, Dec 19, 2017 at 7:05 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Shall we rename the "haConfig" to "ha-clustering"? I'm not sure if we should use camel case in the yaml config.

On Tue, Dec 19, 2017 at 4:42 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

We are currently working on introducing $subject [1]. Please find below a high level description of the approach.

An extension point (HaStrategy) will be introduced, allowing straightforward introduction of different implementations of identification of the active node, where the only requirements would be that these approaches extend the common class and invoke particular methods when the node state changes. 

The broker-core and broker-transport (broker-amqp) modules would introduce listeners to receive notifications of node states changes (active/passive), and change behaviour accordingly.




Configuration

The HA related configuration would be specified in the broker.yaml file including whether HA is enabled and the HA strategy to use.
haConfig:
enabled: true
strategy: org.wso2.broker.coordination.rdbms.RdbmsHaStrategy

The basic/initial HA strategy implementation will be the RdbmsHaStrategy based on the RDBMS based coordinator election approach previously introduced for MB 3.2.0. [2, 3]. ​If HA enabled is set to true but no strategy is specified, the RdbmsHaStrategy will be used. 


RDBMS Coordinator Election based HA Strategy (RdbmsHaStrategy)

The RDBMS based coordinator election algorithm would be extended to provide HA support, by specifying the node elected as coordinator to always be the active node, while the other node(s) will be considered passive. The RDBMS coordinator election based approach, which would also be the default HA strategy, would require the nodes in the HA group to share the same database. All MB nodes pointing to this shared database will be considered as MBs belonging to the same group, and at any given point only one of the nodes will be considered active.

Feedback on the approach would be highly appreciated.

[2] Mail: "[Architecture] RDBMS based coordinator election algorithm for MB"

Thank you,
Maryam
--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature



--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Regards,

Waruna Lakshitha Jayaweera
Senior Software Engineer
WSO2 Inc; http://wso2.com
phone: <a href="tel:071%20325%205198" value="+94713255198" target="_blank">+94713255198


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
W.G. Gihan Anuruddha
Associate Technical Lead | WSO2, Inc.

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [MB4] HA Support

Waruna Jayaweera
Hi Gihan,
Thanks for suggestion. Yes. Lazy loading is the other option we can use for sync in memory data. One drawback would be if given resource permission is not exist in both memory and database, lazy loader will try to read the database for every request call. I will check both approaches.

Thanks,
Waruna


On Wed, Jan 24, 2018 at 6:02 PM, Gihan Anuruddha <[hidden email]> wrote:
Hi All,

Without reloading everything when node becomes active from passive state, is there any possibility of doing lazy loading specially in permission area? This will speed up the server readiness to serve live traffic.

Regards,
Gihan

On Tue, Jan 23, 2018 at 6:57 PM, Waruna Jayaweera <[hidden email]> wrote:
Hi Maryam,

We also store the permissions related data in both memory and persistent storage but permissions will be retrieved from memory. In that case if new permissions added to active node, passive node will have inconsistency with permissions. 
I hope we can use same above approach for permissions as well.

Thanks,
Waruna


  


On Fri, Jan 19, 2018 at 6:01 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Asanka,

The required queue/binding/exchange related data will be loaded once the previously passive node is notified of becoming the active node.

This is considering the possibility of inconsistencies in a continuous sync, which could also require reloading on becoming the active node. 

Thank you,
Maryam

On Mon, Jan 15, 2018 at 8:14 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Are we keeping the passive node in sync with the active node or are we reloading context and message data when a passive node becomes active?

On Fri, Jan 12, 2018 at 6:46 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

As mentioned, an extensible point "org.wso2.broker.coordination.HaStrategy" has been introduced, which can be implemented to provide variations upon which HA support will be based. 

Any new/custom implementation would have to notify the listeners listening on node state changes (when a node state changes from active to passive or passive to active). 

Currently the modules listening on state changes are as follows:
  • broker-core
  • broker-transport
  • broker-rest-runner
All nodes will start up in "passive" mode, and only change state to "active" on notification by the HA strategy. This requires that the listeners are registered prior to starting the HA strategy (identification of the active node).


Default RDBMS Coordinator Election based HA Strategy 

With the default HA strategy (RdbmsHaStrategy - implemented based on the RDBMS based coordinator election approach [1]), the possible coordination node states are mapped to active/passive as follows:
  • COORDINATOR - Active
  • CANDIDATE - Passive
  • ELECTION - Passive
Thus notification of the HA listeners happens when:
  • election is triggered and the node was previously the coordinator node (active → passive)
  • election resulted in the node becoming the coordinator node (passive → active)

Feedback/suggestions would be highly appreciated.

Thank you,
Maryam

On Fri, Jan 12, 2018 at 6:41 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Asanka,

Renamed "haConfig" to "failover" based on the offline discussion.

Thank you,
Maryam

On Tue, Dec 19, 2017 at 7:05 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi Maryam,

Shall we rename the "haConfig" to "ha-clustering"? I'm not sure if we should use camel case in the yaml config.

On Tue, Dec 19, 2017 at 4:42 PM, Maryam Ziyad <[hidden email]> wrote:
Hi All,

We are currently working on introducing $subject [1]. Please find below a high level description of the approach.

An extension point (HaStrategy) will be introduced, allowing straightforward introduction of different implementations of identification of the active node, where the only requirements would be that these approaches extend the common class and invoke particular methods when the node state changes. 

The broker-core and broker-transport (broker-amqp) modules would introduce listeners to receive notifications of node states changes (active/passive), and change behaviour accordingly.




Configuration

The HA related configuration would be specified in the broker.yaml file including whether HA is enabled and the HA strategy to use.
haConfig:
enabled: true
strategy: org.wso2.broker.coordination.rdbms.RdbmsHaStrategy

The basic/initial HA strategy implementation will be the RdbmsHaStrategy based on the RDBMS based coordinator election approach previously introduced for MB 3.2.0. [2, 3]. ​If HA enabled is set to true but no strategy is specified, the RdbmsHaStrategy will be used. 


RDBMS Coordinator Election based HA Strategy (RdbmsHaStrategy)

The RDBMS based coordinator election algorithm would be extended to provide HA support, by specifying the node elected as coordinator to always be the active node, while the other node(s) will be considered passive. The RDBMS coordinator election based approach, which would also be the default HA strategy, would require the nodes in the HA group to share the same database. All MB nodes pointing to this shared database will be considered as MBs belonging to the same group, and at any given point only one of the nodes will be considered active.

Feedback on the approach would be highly appreciated.

[2] Mail: "[Architecture] RDBMS based coordinator election algorithm for MB"

Thank you,
Maryam
--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature



--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Associate Technical Lead
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Regards,

Waruna Lakshitha Jayaweera
Senior Software Engineer
WSO2 Inc; http://wso2.com
phone: <a href="tel:071%20325%205198" value="+94713255198" target="_blank">+94713255198


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
W.G. Gihan Anuruddha
Associate Technical Lead | WSO2, Inc.



--
Regards,

Waruna Lakshitha Jayaweera
Senior Software Engineer
WSO2 Inc; http://wso2.com
phone: +94713255198


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture