Defining event store specific parameters via stream definition.

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Defining event store specific parameters via stream definition.

Kasun Weranga-2
Hi all,

In the current stream definition implementation there is no way to specify event store specific parameters via stream definition. Main use-case for this requirement is to support built-in secondary indexes for events stored in the Cassandra column family. This allows users to define indexing columns for Cassandra column family in the stream definition. So we can define those indexes when we first create the column family using stream definition. 

We can achieve this by using following format in the stream definition. 

 
                   'name':'org.wso2.bam.phone.retail.store.kpi', 
                   'version':'1.0.0', 
                   'nickName': 'Phone_Retail_Shop', 
                   'description': 'Phone Sales',
  'eventStoreSpecificParameters' :[ {'indexes':'payload_user,payload_brand'} ],     
                   'metaData':[ 
                           {'name':'clientType','type':'STRING'} 
                   ], 
                   'payloadData':[ 
                           {'name':'brand','type':'STRING'}, 
                           {'name':'quantity','type':'INT'}, 
                           {'name':'total','type':'INT'}, 
                           {'name':'user','type':'STRING'} 
                   ] 
                 }


In the eventStoreSpecificParameters we can define the parameters which are specific for that particular event store, so relevant event persistence implementation can use these parameters to achieve the required functionalities. Also we can make this as and optional parameter, so everyone don't need to define this. 

Since we are planing to support different event stores in future, I hope this will be useful feature for other event stores too. 

Suggestions and comments are welcome. 

Thanks,
KasunW. 

--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : +94 772314602
blog 
http://kasunweranga.blogspot.com/
_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Defining event store specific parameters via stream definition.

Senaka Fernando
Hi Kasun,

A good way to help an API to expand is to support a standard format and include a standard extension point such that implementation-specifics could be included in there. While what you've done is nearly similar, so if possible please pay attention into this aspect too.

Thanks,
Senaka.

On Thu, Mar 7, 2013 at 3:38 PM, Kasun Weranga <[hidden email]> wrote:
Hi all,

In the current stream definition implementation there is no way to specify event store specific parameters via stream definition. Main use-case for this requirement is to support built-in secondary indexes for events stored in the Cassandra column family. This allows users to define indexing columns for Cassandra column family in the stream definition. So we can define those indexes when we first create the column family using stream definition. 

We can achieve this by using following format in the stream definition. 

 
                   'name':'org.wso2.bam.phone.retail.store.kpi', 
                   'version':'1.0.0', 
                   'nickName': 'Phone_Retail_Shop', 
                   'description': 'Phone Sales',
  'eventStoreSpecificParameters' :[ {'indexes':'payload_user,payload_brand'} ],     
                   'metaData':[ 
                           {'name':'clientType','type':'STRING'} 
                   ], 
                   'payloadData':[ 
                           {'name':'brand','type':'STRING'}, 
                           {'name':'quantity','type':'INT'}, 
                           {'name':'total','type':'INT'}, 
                           {'name':'user','type':'STRING'} 
                   ] 
                 }


In the eventStoreSpecificParameters we can define the parameters which are specific for that particular event store, so relevant event persistence implementation can use these parameters to achieve the required functionalities. Also we can make this as and optional parameter, so everyone don't need to define this. 

Since we are planing to support different event stores in future, I hope this will be useful feature for other event stores too. 

Suggestions and comments are welcome. 

Thanks,
KasunW. 

--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--


Senaka Fernando

Member - Integration Technologies Management Committee;
Technical Lead; WSO2 Inc.; http://wso2.com
Member; Apache Software Foundation; http://apache.org

E-mail: senaka AT wso2.com
P: +1 408 754 7388; ext: 51736; M: +94 77 322 1818
Linked-In: http://linkedin.com/in/senakafernando

Lean . Enterprise . Middleware

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Defining event store specific parameters via stream definition.

Srinath Perera-3
Was this done? how was this done?


On Fri, Mar 8, 2013 at 5:17 AM, Senaka Fernando <[hidden email]> wrote:
Hi Kasun,

A good way to help an API to expand is to support a standard format and include a standard extension point such that implementation-specifics could be included in there. While what you've done is nearly similar, so if possible please pay attention into this aspect too.

Thanks,
Senaka.

On Thu, Mar 7, 2013 at 3:38 PM, Kasun Weranga <[hidden email]> wrote:
Hi all,

In the current stream definition implementation there is no way to specify event store specific parameters via stream definition. Main use-case for this requirement is to support built-in secondary indexes for events stored in the Cassandra column family. This allows users to define indexing columns for Cassandra column family in the stream definition. So we can define those indexes when we first create the column family using stream definition. 

We can achieve this by using following format in the stream definition. 

 
                   'name':'org.wso2.bam.phone.retail.store.kpi', 
                   'version':'1.0.0', 
                   'nickName': 'Phone_Retail_Shop', 
                   'description': 'Phone Sales',
  'eventStoreSpecificParameters' :[ {'indexes':'payload_user,payload_brand'} ],     
                   'metaData':[ 
                           {'name':'clientType','type':'STRING'} 
                   ], 
                   'payloadData':[ 
                           {'name':'brand','type':'STRING'}, 
                           {'name':'quantity','type':'INT'}, 
                           {'name':'total','type':'INT'}, 
                           {'name':'user','type':'STRING'} 
                   ] 
                 }


In the eventStoreSpecificParameters we can define the parameters which are specific for that particular event store, so relevant event persistence implementation can use these parameters to achieve the required functionalities. Also we can make this as and optional parameter, so everyone don't need to define this. 

Since we are planing to support different event stores in future, I hope this will be useful feature for other event stores too. 

Suggestions and comments are welcome. 

Thanks,
KasunW. 

--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--


Senaka Fernando

Member - Integration Technologies Management Committee;
Technical Lead; WSO2 Inc.; http://wso2.com
Member; Apache Software Foundation; http://apache.org

E-mail: senaka AT wso2.com
P: <a href="tel:%2B1%20408%20754%207388" value="+14087547388" target="_blank">+1 408 754 7388; ext: 51736; M: <a href="tel:%2B94%2077%20322%201818" value="+94773221818" target="_blank">+94 77 322 1818
Linked-In: http://linkedin.com/in/senakafernando

Lean . Enterprise . Middleware



--
============================
Srinath Perera, Ph.D.
  Senior Software Architect, WSO2 Inc.
  Visiting Faculty, University of Moratuwa
  Member, Apache Software Foundation
  Research Scientist, Lanka Software Foundation
  Blog: http://srinathsview.blogspot.com/
  Photos: http://www.flickr.com/photos/hemapani/
 Phone: 0772360902

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Defining event store specific parameters via stream definition.

Kasun Weranga-2
We are planing to do this in BAM 2.5.0. Still we didn't implement it. 


On Mon, Apr 29, 2013 at 7:17 PM, Srinath Perera <[hidden email]> wrote:
Was this done? how was this done?


On Fri, Mar 8, 2013 at 5:17 AM, Senaka Fernando <[hidden email]> wrote:
Hi Kasun,

A good way to help an API to expand is to support a standard format and include a standard extension point such that implementation-specifics could be included in there. While what you've done is nearly similar, so if possible please pay attention into this aspect too.

Thanks,
Senaka.

On Thu, Mar 7, 2013 at 3:38 PM, Kasun Weranga <[hidden email]> wrote:
Hi all,

In the current stream definition implementation there is no way to specify event store specific parameters via stream definition. Main use-case for this requirement is to support built-in secondary indexes for events stored in the Cassandra column family. This allows users to define indexing columns for Cassandra column family in the stream definition. So we can define those indexes when we first create the column family using stream definition. 

We can achieve this by using following format in the stream definition. 

 
                   'name':'org.wso2.bam.phone.retail.store.kpi', 
                   'version':'1.0.0', 
                   'nickName': 'Phone_Retail_Shop', 
                   'description': 'Phone Sales',
  'eventStoreSpecificParameters' :[ {'indexes':'payload_user,payload_brand'} ],     
                   'metaData':[ 
                           {'name':'clientType','type':'STRING'} 
                   ], 
                   'payloadData':[ 
                           {'name':'brand','type':'STRING'}, 
                           {'name':'quantity','type':'INT'}, 
                           {'name':'total','type':'INT'}, 
                           {'name':'user','type':'STRING'} 
                   ] 
                 }


In the eventStoreSpecificParameters we can define the parameters which are specific for that particular event store, so relevant event persistence implementation can use these parameters to achieve the required functionalities. Also we can make this as and optional parameter, so everyone don't need to define this. 

Since we are planing to support different event stores in future, I hope this will be useful feature for other event stores too. 

Suggestions and comments are welcome. 

Thanks,
KasunW. 

--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--


Senaka Fernando

Member - Integration Technologies Management Committee;
Technical Lead; WSO2 Inc.; http://wso2.com
Member; Apache Software Foundation; http://apache.org

E-mail: senaka AT wso2.com
P: <a href="tel:%2B1%20408%20754%207388" value="+14087547388" target="_blank">+1 408 754 7388; ext: 51736; M: <a href="tel:%2B94%2077%20322%201818" value="+94773221818" target="_blank">+94 77 322 1818
Linked-In: http://linkedin.com/in/senakafernando

Lean . Enterprise . Middleware



--
============================
Srinath Perera, Ph.D.
  Senior Software Architect, WSO2 Inc.
  Visiting Faculty, University of Moratuwa
  Member, Apache Software Foundation
  Research Scientist, Lanka Software Foundation
  Blog: http://srinathsview.blogspot.com/
  Photos: http://www.flickr.com/photos/hemapani/
 Phone: <a href="tel:0772360902" value="+61772360902" target="_blank">0772360902

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : +94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Defining event store specific parameters via stream definition.

Srinath Perera-3
Should this be part of the language? May be we should do this via a separate service call instead if putting it inside event definitions. 


On Mon, Apr 29, 2013 at 7:26 PM, Kasun Weranga <[hidden email]> wrote:
We are planing to do this in BAM 2.5.0. Still we didn't implement it. 


On Mon, Apr 29, 2013 at 7:17 PM, Srinath Perera <[hidden email]> wrote:
Was this done? how was this done?


On Fri, Mar 8, 2013 at 5:17 AM, Senaka Fernando <[hidden email]> wrote:
Hi Kasun,

A good way to help an API to expand is to support a standard format and include a standard extension point such that implementation-specifics could be included in there. While what you've done is nearly similar, so if possible please pay attention into this aspect too.

Thanks,
Senaka.

On Thu, Mar 7, 2013 at 3:38 PM, Kasun Weranga <[hidden email]> wrote:
Hi all,

In the current stream definition implementation there is no way to specify event store specific parameters via stream definition. Main use-case for this requirement is to support built-in secondary indexes for events stored in the Cassandra column family. This allows users to define indexing columns for Cassandra column family in the stream definition. So we can define those indexes when we first create the column family using stream definition. 

We can achieve this by using following format in the stream definition. 

 
                   'name':'org.wso2.bam.phone.retail.store.kpi', 
                   'version':'1.0.0', 
                   'nickName': 'Phone_Retail_Shop', 
                   'description': 'Phone Sales',
  'eventStoreSpecificParameters' :[ {'indexes':'payload_user,payload_brand'} ],     
                   'metaData':[ 
                           {'name':'clientType','type':'STRING'} 
                   ], 
                   'payloadData':[ 
                           {'name':'brand','type':'STRING'}, 
                           {'name':'quantity','type':'INT'}, 
                           {'name':'total','type':'INT'}, 
                           {'name':'user','type':'STRING'} 
                   ] 
                 }


In the eventStoreSpecificParameters we can define the parameters which are specific for that particular event store, so relevant event persistence implementation can use these parameters to achieve the required functionalities. Also we can make this as and optional parameter, so everyone don't need to define this. 

Since we are planing to support different event stores in future, I hope this will be useful feature for other event stores too. 

Suggestions and comments are welcome. 

Thanks,
KasunW. 

--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--


Senaka Fernando

Member - Integration Technologies Management Committee;
Technical Lead; WSO2 Inc.; http://wso2.com
Member; Apache Software Foundation; http://apache.org

E-mail: senaka AT wso2.com
P: <a href="tel:%2B1%20408%20754%207388" value="+14087547388" target="_blank">+1 408 754 7388; ext: 51736; M: <a href="tel:%2B94%2077%20322%201818" value="+94773221818" target="_blank">+94 77 322 1818
Linked-In: http://linkedin.com/in/senakafernando

Lean . Enterprise . Middleware



--
============================
Srinath Perera, Ph.D.
  Senior Software Architect, WSO2 Inc.
  Visiting Faculty, University of Moratuwa
  Member, Apache Software Foundation
  Research Scientist, Lanka Software Foundation
  Blog: http://srinathsview.blogspot.com/
  Photos: http://www.flickr.com/photos/hemapani/
 Phone: <a href="tel:0772360902" value="+61772360902" target="_blank">0772360902

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/



--
============================
Srinath Perera, Ph.D.
   http://www.cs.indiana.edu/~hperera/
   http://srinathsview.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Defining event store specific parameters via stream definition.

Kasun Weranga-2
Since we create CF using the stream definition, If we send these information with stream definition we can specify indexing columns at the column family creation time. Also IMO it is easy for users to define the indexing columns via stream definition. 


On Tue, Apr 30, 2013 at 9:08 AM, Srinath Perera <[hidden email]> wrote:
Should this be part of the language? May be we should do this via a separate service call instead if putting it inside event definitions. 


On Mon, Apr 29, 2013 at 7:26 PM, Kasun Weranga <[hidden email]> wrote:
We are planing to do this in BAM 2.5.0. Still we didn't implement it. 


On Mon, Apr 29, 2013 at 7:17 PM, Srinath Perera <[hidden email]> wrote:
Was this done? how was this done?


On Fri, Mar 8, 2013 at 5:17 AM, Senaka Fernando <[hidden email]> wrote:
Hi Kasun,

A good way to help an API to expand is to support a standard format and include a standard extension point such that implementation-specifics could be included in there. While what you've done is nearly similar, so if possible please pay attention into this aspect too.

Thanks,
Senaka.

On Thu, Mar 7, 2013 at 3:38 PM, Kasun Weranga <[hidden email]> wrote:
Hi all,

In the current stream definition implementation there is no way to specify event store specific parameters via stream definition. Main use-case for this requirement is to support built-in secondary indexes for events stored in the Cassandra column family. This allows users to define indexing columns for Cassandra column family in the stream definition. So we can define those indexes when we first create the column family using stream definition. 

We can achieve this by using following format in the stream definition. 

 
                   'name':'org.wso2.bam.phone.retail.store.kpi', 
                   'version':'1.0.0', 
                   'nickName': 'Phone_Retail_Shop', 
                   'description': 'Phone Sales',
  'eventStoreSpecificParameters' :[ {'indexes':'payload_user,payload_brand'} ],     
                   'metaData':[ 
                           {'name':'clientType','type':'STRING'} 
                   ], 
                   'payloadData':[ 
                           {'name':'brand','type':'STRING'}, 
                           {'name':'quantity','type':'INT'}, 
                           {'name':'total','type':'INT'}, 
                           {'name':'user','type':'STRING'} 
                   ] 
                 }


In the eventStoreSpecificParameters we can define the parameters which are specific for that particular event store, so relevant event persistence implementation can use these parameters to achieve the required functionalities. Also we can make this as and optional parameter, so everyone don't need to define this. 

Since we are planing to support different event stores in future, I hope this will be useful feature for other event stores too. 

Suggestions and comments are welcome. 

Thanks,
KasunW. 

--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--


Senaka Fernando

Member - Integration Technologies Management Committee;
Technical Lead; WSO2 Inc.; http://wso2.com
Member; Apache Software Foundation; http://apache.org

E-mail: senaka AT wso2.com
P: <a href="tel:%2B1%20408%20754%207388" value="+14087547388" target="_blank">+1 408 754 7388; ext: 51736; M: <a href="tel:%2B94%2077%20322%201818" value="+94773221818" target="_blank">+94 77 322 1818
Linked-In: http://linkedin.com/in/senakafernando

Lean . Enterprise . Middleware



--
============================
Srinath Perera, Ph.D.
  Senior Software Architect, WSO2 Inc.
  Visiting Faculty, University of Moratuwa
  Member, Apache Software Foundation
  Research Scientist, Lanka Software Foundation
  Blog: http://srinathsview.blogspot.com/
  Photos: http://www.flickr.com/photos/hemapani/
 Phone: <a href="tel:0772360902" value="+61772360902" target="_blank">0772360902

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/



--
============================
Srinath Perera, Ph.D.
   http://www.cs.indiana.edu/~hperera/
   http://srinathsview.blogspot.com/



--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : +94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Defining event store specific parameters via stream definition.

Tharindu Mathew-3
If we store event definitions in mongoDB or HBase, will this work? I believe that is our future todos, so we shouldn't get bound to Cassandra specific ways of doing it...

On Wed, May 1, 2013 at 3:19 PM, Kasun Weranga <[hidden email]> wrote:
Since we create CF using the stream definition, If we send these information with stream definition we can specify indexing columns at the column family creation time. Also IMO it is easy for users to define the indexing columns via stream definition. 


On Tue, Apr 30, 2013 at 9:08 AM, Srinath Perera <[hidden email]> wrote:
Should this be part of the language? May be we should do this via a separate service call instead if putting it inside event definitions. 


On Mon, Apr 29, 2013 at 7:26 PM, Kasun Weranga <[hidden email]> wrote:
We are planing to do this in BAM 2.5.0. Still we didn't implement it. 


On Mon, Apr 29, 2013 at 7:17 PM, Srinath Perera <[hidden email]> wrote:
Was this done? how was this done?


On Fri, Mar 8, 2013 at 5:17 AM, Senaka Fernando <[hidden email]> wrote:
Hi Kasun,

A good way to help an API to expand is to support a standard format and include a standard extension point such that implementation-specifics could be included in there. While what you've done is nearly similar, so if possible please pay attention into this aspect too.

Thanks,
Senaka.

On Thu, Mar 7, 2013 at 3:38 PM, Kasun Weranga <[hidden email]> wrote:
Hi all,

In the current stream definition implementation there is no way to specify event store specific parameters via stream definition. Main use-case for this requirement is to support built-in secondary indexes for events stored in the Cassandra column family. This allows users to define indexing columns for Cassandra column family in the stream definition. So we can define those indexes when we first create the column family using stream definition. 

We can achieve this by using following format in the stream definition. 

 
                   'name':'org.wso2.bam.phone.retail.store.kpi', 
                   'version':'1.0.0', 
                   'nickName': 'Phone_Retail_Shop', 
                   'description': 'Phone Sales',
  'eventStoreSpecificParameters' :[ {'indexes':'payload_user,payload_brand'} ],     
                   'metaData':[ 
                           {'name':'clientType','type':'STRING'} 
                   ], 
                   'payloadData':[ 
                           {'name':'brand','type':'STRING'}, 
                           {'name':'quantity','type':'INT'}, 
                           {'name':'total','type':'INT'}, 
                           {'name':'user','type':'STRING'} 
                   ] 
                 }


In the eventStoreSpecificParameters we can define the parameters which are specific for that particular event store, so relevant event persistence implementation can use these parameters to achieve the required functionalities. Also we can make this as and optional parameter, so everyone don't need to define this. 

Since we are planing to support different event stores in future, I hope this will be useful feature for other event stores too. 

Suggestions and comments are welcome. 

Thanks,
KasunW. 

--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--


Senaka Fernando

Member - Integration Technologies Management Committee;
Technical Lead; WSO2 Inc.; http://wso2.com
Member; Apache Software Foundation; http://apache.org

E-mail: senaka AT wso2.com
P: <a href="tel:%2B1%20408%20754%207388" value="+14087547388" target="_blank">+1 408 754 7388; ext: 51736; M: <a href="tel:%2B94%2077%20322%201818" value="+94773221818" target="_blank">+94 77 322 1818
Linked-In: http://linkedin.com/in/senakafernando

Lean . Enterprise . Middleware



--
============================
Srinath Perera, Ph.D.
  Senior Software Architect, WSO2 Inc.
  Visiting Faculty, University of Moratuwa
  Member, Apache Software Foundation
  Research Scientist, Lanka Software Foundation
  Blog: http://srinathsview.blogspot.com/
  Photos: http://www.flickr.com/photos/hemapani/
 Phone: <a href="tel:0772360902" value="+61772360902" target="_blank">0772360902

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/



--
============================
Srinath Perera, Ph.D.
   http://www.cs.indiana.edu/~hperera/
   http://srinathsview.blogspot.com/



--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Regards,

Tharindu Mathew

Associate Technical Lead, WSO2 BAM
Member - Data Mgmt. Committee

M: +94777759908

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Defining event store specific parameters via stream definition.

Kasun Weranga-2



On Thu, May 2, 2013 at 11:50 AM, Tharindu Mathew <[hidden email]> wrote:
If we store event definitions in mongoDB or HBase, will this work? I believe that is our future todos, so we shouldn't get bound to Cassandra specific ways of doing it...

We can use eventStoreSpecificParameters section to send the event store specific things and relevant data-bridge persistence layer should do the relevant task based on the event store and the parameters. 



On Wed, May 1, 2013 at 3:19 PM, Kasun Weranga <[hidden email]> wrote:
Since we create CF using the stream definition, If we send these information with stream definition we can specify indexing columns at the column family creation time. Also IMO it is easy for users to define the indexing columns via stream definition. 


On Tue, Apr 30, 2013 at 9:08 AM, Srinath Perera <[hidden email]> wrote:
Should this be part of the language? May be we should do this via a separate service call instead if putting it inside event definitions. 


On Mon, Apr 29, 2013 at 7:26 PM, Kasun Weranga <[hidden email]> wrote:
We are planing to do this in BAM 2.5.0. Still we didn't implement it. 


On Mon, Apr 29, 2013 at 7:17 PM, Srinath Perera <[hidden email]> wrote:
Was this done? how was this done?


On Fri, Mar 8, 2013 at 5:17 AM, Senaka Fernando <[hidden email]> wrote:
Hi Kasun,

A good way to help an API to expand is to support a standard format and include a standard extension point such that implementation-specifics could be included in there. While what you've done is nearly similar, so if possible please pay attention into this aspect too.

Thanks,
Senaka.

On Thu, Mar 7, 2013 at 3:38 PM, Kasun Weranga <[hidden email]> wrote:
Hi all,

In the current stream definition implementation there is no way to specify event store specific parameters via stream definition. Main use-case for this requirement is to support built-in secondary indexes for events stored in the Cassandra column family. This allows users to define indexing columns for Cassandra column family in the stream definition. So we can define those indexes when we first create the column family using stream definition. 

We can achieve this by using following format in the stream definition. 

 
                   'name':'org.wso2.bam.phone.retail.store.kpi', 
                   'version':'1.0.0', 
                   'nickName': 'Phone_Retail_Shop', 
                   'description': 'Phone Sales',
  'eventStoreSpecificParameters' :[ {'indexes':'payload_user,payload_brand'} ],     
                   'metaData':[ 
                           {'name':'clientType','type':'STRING'} 
                   ], 
                   'payloadData':[ 
                           {'name':'brand','type':'STRING'}, 
                           {'name':'quantity','type':'INT'}, 
                           {'name':'total','type':'INT'}, 
                           {'name':'user','type':'STRING'} 
                   ] 
                 }


In the eventStoreSpecificParameters we can define the parameters which are specific for that particular event store, so relevant event persistence implementation can use these parameters to achieve the required functionalities. Also we can make this as and optional parameter, so everyone don't need to define this. 

Since we are planing to support different event stores in future, I hope this will be useful feature for other event stores too. 

Suggestions and comments are welcome. 

Thanks,
KasunW. 

--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--


Senaka Fernando

Member - Integration Technologies Management Committee;
Technical Lead; WSO2 Inc.; http://wso2.com
Member; Apache Software Foundation; http://apache.org

E-mail: senaka AT wso2.com
P: <a href="tel:%2B1%20408%20754%207388" value="+14087547388" target="_blank">+1 408 754 7388; ext: 51736; M: <a href="tel:%2B94%2077%20322%201818" value="+94773221818" target="_blank">+94 77 322 1818
Linked-In: http://linkedin.com/in/senakafernando

Lean . Enterprise . Middleware



--
============================
Srinath Perera, Ph.D.
  Senior Software Architect, WSO2 Inc.
  Visiting Faculty, University of Moratuwa
  Member, Apache Software Foundation
  Research Scientist, Lanka Software Foundation
  Blog: http://srinathsview.blogspot.com/
  Photos: http://www.flickr.com/photos/hemapani/
 Phone: <a href="tel:0772360902" value="+61772360902" target="_blank">0772360902

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/



--
============================
Srinath Perera, Ph.D.
   http://www.cs.indiana.edu/~hperera/
   http://srinathsview.blogspot.com/



--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : <a href="tel:%2B94%20772314602" value="+94772314602" target="_blank">+94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Regards,

Tharindu Mathew

Associate Technical Lead, WSO2 BAM
Member - Data Mgmt. Committee

M: <a href="tel:%2B94777759908" value="+94777759908" target="_blank">+94777759908

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Kasun Weranga
Member, Management Committee - Data Technologies
Software Engineer
WSO2, Inc.
lean.enterprise.middleware.
mobile : +94 772314602
blog 
http://kasunweranga.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Defining event store specific parameters via stream definition.

Dipesh Chheda
Kasun Weranga-2 wrote
On Thu, May 2, 2013 at 11:50 AM, Tharindu Mathew <[hidden email]> wrote:

> If we store event definitions in mongoDB or HBase, will this work? I
> believe that is our future todos, so we shouldn't get bound to Cassandra
> specific ways of doing it...


We can use eventStoreSpecificParameters section to send the event store
specific things and relevant data-bridge persistence layer should do the
relevant task based on the event store and the parameters.
+1.

For MongoDB, eventStoreSpecificParameters section would have to support several different options of indexing available in MangoDB:
http://docs.mongodb.org/manual/core/indexes/
 
For HBase, it seems (Secondary) Indexes are not supported out of the box.
http://hbase.apache.org/book/secondary.indexes.html

      *From here on the comments are about index-support in general, not about 'eventStoreSpecificParameters'. Apologies, if they are out of place on this thread.

So HBase indexes seem equivalent to 'Custom Indexes' in Cassandra; required to support 'filtering/sorting' on columns with high-cardinality e.g. time-range. Since Big data ought to be time-bound, 'time-range' becomes de facto requirement for filtering and sorting.
So with Cassandra, we are required to write Hive scripts (and purge task) to maintain (Create and Purge) these Custom Indexes.

I believe, BAM should also support such 'custom indexes' out of the box as it would be a great help to the end user and huge value proposition in general. This is going to be my next feature request :)

Thanks,
Dipesh

Kasun Weranga-2 wrote
...
...
...

>>>>>>> We can achieve this by using following format in the stream
>>>>>>> definition.
>>>>>>>
>>>>>>>   {
>>>>>>>                    'name':'org.wso2.bam.phone.retail.store.kpi',
>>>>>>>                    'version':'1.0.0',
>>>>>>>                    'nickName': 'Phone_Retail_Shop',
>>>>>>>                    'description': 'Phone Sales',
>>>>>>>    'eventStoreSpecificParameters' :[
>>>>>>> {'indexes':'payload_user,payload_brand'} ],
>>>>>>>                    'metaData':[
>>>>>>>                            {'name':'clientType','type':'STRING'}
>>>>>>>                    ],
>>>>>>>                    'payloadData':[
>>>>>>>                            {'name':'brand','type':'STRING'},
>>>>>>>                            {'name':'quantity','type':'INT'},
>>>>>>>                            {'name':'total','type':'INT'},
>>>>>>>                            {'name':'user','type':'STRING'}
>>>>>>>                    ]
>>>>>>>                  }
>>>>>>>
>>>>>>>
>>>>>>> In the eventStoreSpecificParameters we can define the parameters
>>>>>>> which are specific for that particular event store, so relevant event
>>>>>>> persistence implementation can use these parameters to achieve the required
>>>>>>> functionalities. Also we can make this as and optional parameter, so
>>>>>>> everyone don't need to define this.
>>>>>>>
>>>>>>> Since we are planing to support different event stores in future, I
>>>>>>> hope this will be useful feature for other event stores too.