File Based Database for Message Broker Store

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

File Based Database for Message Broker Store

Wishmitha Mendis
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : +94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Indika Sampath
Hi Wishmitha,

No SQL database is not an option as we have prior experience with Cassandra. Basically, the important part is to find out performance and fault tolerance capability in the database. Message broker read/write/delete records all the time and database should be able to handle high concurrency. Some metadata and message content write as byte data. Also, message order matters when delivery. Hence database should capable to sort records rather than bringing that logic into broker code. As you mentioned transaction is another factor we are looking in the database. Shall we start POC with Redis and see how it goes?

Cheers!

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Indika Sampath
Associate Technical Lead
WSO2 Inc. 
http://wso2.com

Phone: +94 716 424 744
Blog: http://indikasampath.blogspot.com/


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Malaka Gangananda
In reply to this post by Wishmitha Mendis
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : +94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Malaka Gangananda
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : +94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

File Based Persistence.pdf (103K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Sasikala Kottegoda
Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[hidden email]> wrote:
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Sasikala Kottegoda
Senior Software Engineer
WSO2 Inc., http://wso2.com/
lean. enterprise. middleware

Mobile: +94 774835928

https://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Riyafa Abdul Hameed-2
Dear Wishmitha,

For all the databases mentioned please provide references to the relevant websites and where you got the information from. You have mentioned for some which is great and you should do so for others as well. Also when referencing don't have the weblink inline with the statement because it distracts the viewer as the links might be long. You can number the references and refer them below (You don't need to follow any standards, simply giving the link would suffice).

Cheers,
Riyafa

On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[hidden email]> wrote:
Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[hidden email]> wrote:
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Sasikala Kottegoda
Senior Software Engineer
WSO2 Inc., http://wso2.com/
lean. enterprise. middleware

Mobile: <a href="tel:+94%2077%20483%205928" value="+94774835928" target="_blank">+94 774835928

https://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Asanka Abeyweera
Hi Wishmitha,

If we use AOF wth fsync at every query, will there be a data loss with Redis? AOF might not have very good performance compared to RDB. But as Sasikal mentioned we cannot allow message losses in the broker.

On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[hidden email]> wrote:
Dear Wishmitha,

For all the databases mentioned please provide references to the relevant websites and where you got the information from. You have mentioned for some which is great and you should do so for others as well. Also when referencing don't have the weblink inline with the statement because it distracts the viewer as the links might be long. You can number the references and refer them below (You don't need to follow any standards, simply giving the link would suffice).

Cheers,
Riyafa

On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[hidden email]> wrote:
Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[hidden email]> wrote:
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Sasikala Kottegoda
Senior Software Engineer
WSO2 Inc., http://wso2.com/
lean. enterprise. middleware

Mobile: <a href="tel:+94%2077%20483%205928" value="+94774835928" target="_blank">+94 774835928

https://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: +94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Manuri
Hi,

From the given description, it seems persistence is supported by writing to hard disk in configurable intervals. Unless we have a very low interval (which might be against any expected performance improvement brought by the database being in-memory) it cannot be assured that there won't be data loss. Also even a low interval would not fully guarantee that. 
I had a quick look at [1] and it says "You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually call the SAVE or BGSAVE commands.". So, if server crashes we could lose 1 - M changes. We can only reduce it but cannot avoid it completely. :)
So I'm not sure if we could use Redis.


Thanks,
Manuri

On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Wishmitha,

If we use AOF wth fsync at every query, will there be a data loss with Redis? AOF might not have very good performance compared to RDB. But as Sasikal mentioned we cannot allow message losses in the broker.

On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[hidden email]> wrote:
Dear Wishmitha,

For all the databases mentioned please provide references to the relevant websites and where you got the information from. You have mentioned for some which is great and you should do so for others as well. Also when referencing don't have the weblink inline with the statement because it distracts the viewer as the links might be long. You can number the references and refer them below (You don't need to follow any standards, simply giving the link would suffice).

Cheers,
Riyafa

On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[hidden email]> wrote:
Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[hidden email]> wrote:
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Sasikala Kottegoda
Senior Software Engineer
WSO2 Inc., http://wso2.com/
lean. enterprise. middleware

Mobile: <a href="tel:+94%2077%20483%205928" value="+94774835928" target="_blank">+94 774835928

https://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Manuri Amaya Perera
Software Engineer
WSO2 Inc.




_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Asanka Abeyweera
Hi Manuri,

Rather than syncing periodically (1-second or less), they say they can sync at every query with AOF [1]. I don't' know the performance impact of that approach. Need to test and see. I am not proposing to use Redis. Just pointing the information. 


On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera <[hidden email]> wrote:
Hi,

From the given description, it seems persistence is supported by writing to hard disk in configurable intervals. Unless we have a very low interval (which might be against any expected performance improvement brought by the database being in-memory) it cannot be assured that there won't be data loss. Also even a low interval would not fully guarantee that. 
I had a quick look at [1] and it says "You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually call the SAVE or BGSAVE commands.". So, if server crashes we could lose 1 - M changes. We can only reduce it but cannot avoid it completely. :)
So I'm not sure if we could use Redis.


Thanks,
Manuri

On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Wishmitha,

If we use AOF wth fsync at every query, will there be a data loss with Redis? AOF might not have very good performance compared to RDB. But as Sasikal mentioned we cannot allow message losses in the broker.

On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[hidden email]> wrote:
Dear Wishmitha,

For all the databases mentioned please provide references to the relevant websites and where you got the information from. You have mentioned for some which is great and you should do so for others as well. Also when referencing don't have the weblink inline with the statement because it distracts the viewer as the links might be long. You can number the references and refer them below (You don't need to follow any standards, simply giving the link would suffice).

Cheers,
Riyafa

On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[hidden email]> wrote:
Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[hidden email]> wrote:
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Sasikala Kottegoda
Senior Software Engineer
WSO2 Inc., http://wso2.com/
lean. enterprise. middleware

Mobile: <a href="tel:+94%2077%20483%205928" value="+94774835928" target="_blank">+94 774835928

https://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Manuri Amaya Perera
Software Engineer
WSO2 Inc.




_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: +94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Maryam Ziyad
Hi Wishmitha, All,

It would be interesting to also look at how AOF persistence in Redis and the append only journal used by Apache ActiveMQ Artemis [1] compare.

Regarding AOF, fsync and durability, this blog post pointed at by the official Redis site, even though a bit old, could be useful. It mentions "Data committed to the disk using the fsync(2) system call (or equivalent) that gives us, virtually, data safety against complete system failure like a power outage." as a safety level [2]. Also, out of the three options made available for fsync (appendfsync no, appendfsync everysec and appendfsync always) they mention that "appendfsync always" (with every query) is the best in terms of durability, but is obviously slower compared to the other modes, and recommend "appendfsync everysec" as a good balance between durability and speed [3]. 

Since these configurations seem straightforward and Redis has other useful advantages as mentioned, doing a POC with Redis and testing seems like a suitable approach.

On Fri, Jul 14, 2017 at 11:38 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Manuri,

Rather than syncing periodically (1-second or less), they say they can sync at every query with AOF [1]. I don't' know the performance impact of that approach. Need to test and see. I am not proposing to use Redis. Just pointing the information. 


On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera <[hidden email]> wrote:
Hi,

From the given description, it seems persistence is supported by writing to hard disk in configurable intervals. Unless we have a very low interval (which might be against any expected performance improvement brought by the database being in-memory) it cannot be assured that there won't be data loss. Also even a low interval would not fully guarantee that. 
I had a quick look at [1] and it says "You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually call the SAVE or BGSAVE commands.". So, if server crashes we could lose 1 - M changes. We can only reduce it but cannot avoid it completely. :)
So I'm not sure if we could use Redis.


Thanks,
Manuri

On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Wishmitha,

If we use AOF wth fsync at every query, will there be a data loss with Redis? AOF might not have very good performance compared to RDB. But as Sasikal mentioned we cannot allow message losses in the broker.

On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[hidden email]> wrote:
Dear Wishmitha,

For all the databases mentioned please provide references to the relevant websites and where you got the information from. You have mentioned for some which is great and you should do so for others as well. Also when referencing don't have the weblink inline with the statement because it distracts the viewer as the links might be long. You can number the references and refer them below (You don't need to follow any standards, simply giving the link would suffice).

Cheers,
Riyafa

On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[hidden email]> wrote:
Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[hidden email]> wrote:
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Sasikala Kottegoda
Senior Software Engineer
WSO2 Inc., http://wso2.com/
lean. enterprise. middleware

Mobile: <a href="tel:+94%2077%20483%205928" value="+94774835928" target="_blank">+94 774835928

https://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Manuri Amaya Perera
Software Engineer
WSO2 Inc.




_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Asanka Abeyweera
Hi Wishmitha,

Since Redis is not an RDBMS it is interesting to know how the transactions are handled. When implementing distributed transactions feature we required row level locking instead of table level locking. We need to check if Redis also supports row level locking kind of feature. Further, in the article [1], it says  Redis does not support rollbacks. Need to investigate more and see if this is a blocker for MB.


On Fri, Jul 14, 2017 at 3:21 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Wishmitha, All,

It would be interesting to also look at how AOF persistence in Redis and the append only journal used by Apache ActiveMQ Artemis [1] compare.

Regarding AOF, fsync and durability, this blog post pointed at by the official Redis site, even though a bit old, could be useful. It mentions "Data committed to the disk using the fsync(2) system call (or equivalent) that gives us, virtually, data safety against complete system failure like a power outage." as a safety level [2]. Also, out of the three options made available for fsync (appendfsync no, appendfsync everysec and appendfsync always) they mention that "appendfsync always" (with every query) is the best in terms of durability, but is obviously slower compared to the other modes, and recommend "appendfsync everysec" as a good balance between durability and speed [3]. 

Since these configurations seem straightforward and Redis has other useful advantages as mentioned, doing a POC with Redis and testing seems like a suitable approach.

On Fri, Jul 14, 2017 at 11:38 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Manuri,

Rather than syncing periodically (1-second or less), they say they can sync at every query with AOF [1]. I don't' know the performance impact of that approach. Need to test and see. I am not proposing to use Redis. Just pointing the information. 


On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera <[hidden email]> wrote:
Hi,

From the given description, it seems persistence is supported by writing to hard disk in configurable intervals. Unless we have a very low interval (which might be against any expected performance improvement brought by the database being in-memory) it cannot be assured that there won't be data loss. Also even a low interval would not fully guarantee that. 
I had a quick look at [1] and it says "You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually call the SAVE or BGSAVE commands.". So, if server crashes we could lose 1 - M changes. We can only reduce it but cannot avoid it completely. :)
So I'm not sure if we could use Redis.


Thanks,
Manuri

On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Wishmitha,

If we use AOF wth fsync at every query, will there be a data loss with Redis? AOF might not have very good performance compared to RDB. But as Sasikal mentioned we cannot allow message losses in the broker.

On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[hidden email]> wrote:
Dear Wishmitha,

For all the databases mentioned please provide references to the relevant websites and where you got the information from. You have mentioned for some which is great and you should do so for others as well. Also when referencing don't have the weblink inline with the statement because it distracts the viewer as the links might be long. You can number the references and refer them below (You don't need to follow any standards, simply giving the link would suffice).

Cheers,
Riyafa

On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[hidden email]> wrote:
Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[hidden email]> wrote:
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Sasikala Kottegoda
Senior Software Engineer
WSO2 Inc., http://wso2.com/
lean. enterprise. middleware

Mobile: <a href="tel:+94%2077%20483%205928" value="+94774835928" target="_blank">+94 774835928

https://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Manuri Amaya Perera
Software Engineer
WSO2 Inc.




_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: +94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Pamod Sylvester
For file based journals/key-value stores transactions should be handled explicitly. 

Let's also evaluate how other brokering technologies i.e artemis, kafka uses file based systems as well. That will be a good comparison to narrow down the list. Also we could evaluate how MariaDB has used levelDB as a store as well, since MariaDB uses LevelDB as it's storage engine [1]. 

Artemis supports features like Tx,Dx 


Thanks,
Pamod

On Fri, Jul 14, 2017 at 4:25 PM, Asanka Abeyweera <[hidden email]> wrote:
Hi Wishmitha,

Since Redis is not an RDBMS it is interesting to know how the transactions are handled. When implementing distributed transactions feature we required row level locking instead of table level locking. We need to check if Redis also supports row level locking kind of feature. Further, in the article [1], it says  Redis does not support rollbacks. Need to investigate more and see if this is a blocker for MB.


On Fri, Jul 14, 2017 at 3:21 PM, Maryam Ziyad <[hidden email]> wrote:
Hi Wishmitha, All,

It would be interesting to also look at how AOF persistence in Redis and the append only journal used by Apache ActiveMQ Artemis [1] compare.

Regarding AOF, fsync and durability, this blog post pointed at by the official Redis site, even though a bit old, could be useful. It mentions "Data committed to the disk using the fsync(2) system call (or equivalent) that gives us, virtually, data safety against complete system failure like a power outage." as a safety level [2]. Also, out of the three options made available for fsync (appendfsync no, appendfsync everysec and appendfsync always) they mention that "appendfsync always" (with every query) is the best in terms of durability, but is obviously slower compared to the other modes, and recommend "appendfsync everysec" as a good balance between durability and speed [3]. 

Since these configurations seem straightforward and Redis has other useful advantages as mentioned, doing a POC with Redis and testing seems like a suitable approach.

On Fri, Jul 14, 2017 at 11:38 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Manuri,

Rather than syncing periodically (1-second or less), they say they can sync at every query with AOF [1]. I don't' know the performance impact of that approach. Need to test and see. I am not proposing to use Redis. Just pointing the information. 


On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera <[hidden email]> wrote:
Hi,

From the given description, it seems persistence is supported by writing to hard disk in configurable intervals. Unless we have a very low interval (which might be against any expected performance improvement brought by the database being in-memory) it cannot be assured that there won't be data loss. Also even a low interval would not fully guarantee that. 
I had a quick look at [1] and it says "You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually call the SAVE or BGSAVE commands.". So, if server crashes we could lose 1 - M changes. We can only reduce it but cannot avoid it completely. :)
So I'm not sure if we could use Redis.


Thanks,
Manuri

On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Wishmitha,

If we use AOF wth fsync at every query, will there be a data loss with Redis? AOF might not have very good performance compared to RDB. But as Sasikal mentioned we cannot allow message losses in the broker.

On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[hidden email]> wrote:
Dear Wishmitha,

For all the databases mentioned please provide references to the relevant websites and where you got the information from. You have mentioned for some which is great and you should do so for others as well. Also when referencing don't have the weblink inline with the statement because it distracts the viewer as the links might be long. You can number the references and refer them below (You don't need to follow any standards, simply giving the link would suffice).

Cheers,
Riyafa

On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[hidden email]> wrote:
Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[hidden email]> wrote:
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Sasikala Kottegoda
Senior Software Engineer
WSO2 Inc., http://wso2.com/
lean. enterprise. middleware

Mobile: <a href="tel:+94%2077%20483%205928" value="+94774835928" target="_blank">+94 774835928

https://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Manuri Amaya Perera
Software Engineer
WSO2 Inc.




_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Maryam Ziyad Mohamed
Software Engineer | WSO2
http://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Pamod Sylvester 
WSO2 Inc.; http://wso2.com
cell: +94 77 7779495

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Wishmitha Mendis
In reply to this post by Indika Sampath
Hi Indika,

I have started a basic implementation to test the concurrency handling, message order and transaction support of Redis. Basic message producers and consumers were implemented and messages are stored in a basic Redis queue. GitHub : https://github.com/Wishmitha/RedisTesting

Best Regards,

On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath <[hidden email]> wrote:
Hi Wishmitha,

No SQL database is not an option as we have prior experience with Cassandra. Basically, the important part is to find out performance and fault tolerance capability in the database. Message broker read/write/delete records all the time and database should be able to handle high concurrency. Some metadata and message content write as byte data. Also, message order matters when delivery. Hence database should capable to sort records rather than bringing that logic into broker code. As you mentioned transaction is another factor we are looking in the database. Shall we start POC with Redis and see how it goes?

Cheers!

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Indika Sampath
Associate Technical Lead
WSO2 Inc. 
http://wso2.com

Phone: <a href="tel:+94%2071%20642%204744" value="+94716424744" target="_blank">+94 716 424 744
Blog: http://indikasampath.blogspot.com/


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : +94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Kevin Ratnasekera-2
Hi Wishmitha,

There are many java clients available for redis, there are several very good clients, see comparison [1]. Any reason using jedis over others?

[1] https://redis.io/clients#java

Regards
Kevin



On Fri, Jul 14, 2017 at 11:02 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi Indika,

I have started a basic implementation to test the concurrency handling, message order and transaction support of Redis. Basic message producers and consumers were implemented and messages are stored in a basic Redis queue. GitHub : https://github.com/Wishmitha/RedisTesting

Best Regards,

On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath <[hidden email]> wrote:
Hi Wishmitha,

No SQL database is not an option as we have prior experience with Cassandra. Basically, the important part is to find out performance and fault tolerance capability in the database. Message broker read/write/delete records all the time and database should be able to handle high concurrency. Some metadata and message content write as byte data. Also, message order matters when delivery. Hence database should capable to sort records rather than bringing that logic into broker code. As you mentioned transaction is another factor we are looking in the database. Shall we start POC with Redis and see how it goes?

Cheers!

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Indika Sampath
Associate Technical Lead
WSO2 Inc. 
http://wso2.com

Phone: <a href="tel:+94%2071%20642%204744" value="+94716424744" target="_blank">+94 716 424 744
Blog: http://indikasampath.blogspot.com/


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Kevin Ratnaskera
Software Engineer
WSO2 Inc. - http://wso2.com
lean . enterprise . middleware
Mobile - +94774696950

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Wishmitha Mendis
In reply to this post by Manuri
Hi  Maryam,

Even the recommended "appendfsync everysec" mode has a potential data loss, the only option would be trying out fsync at every query. So it would be useful to implement a POC as you suggested based on "appendfsync always" and test the performances whether it can deliver the desired results.

Best Regards,

On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera <[hidden email]> wrote:
Hi,

From the given description, it seems persistence is supported by writing to hard disk in configurable intervals. Unless we have a very low interval (which might be against any expected performance improvement brought by the database being in-memory) it cannot be assured that there won't be data loss. Also even a low interval would not fully guarantee that. 
I had a quick look at [1] and it says "You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually call the SAVE or BGSAVE commands.". So, if server crashes we could lose 1 - M changes. We can only reduce it but cannot avoid it completely. :)
So I'm not sure if we could use Redis.


Thanks,
Manuri

On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Wishmitha,

If we use AOF wth fsync at every query, will there be a data loss with Redis? AOF might not have very good performance compared to RDB. But as Sasikal mentioned we cannot allow message losses in the broker.

On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[hidden email]> wrote:
Dear Wishmitha,

For all the databases mentioned please provide references to the relevant websites and where you got the information from. You have mentioned for some which is great and you should do so for others as well. Also when referencing don't have the weblink inline with the statement because it distracts the viewer as the links might be long. You can number the references and refer them below (You don't need to follow any standards, simply giving the link would suffice).

Cheers,
Riyafa

On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[hidden email]> wrote:
Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[hidden email]> wrote:
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Sasikala Kottegoda
Senior Software Engineer
WSO2 Inc., http://wso2.com/
lean. enterprise. middleware

Mobile: <a href="tel:+94%2077%20483%205928" value="+94774835928" target="_blank">+94 774835928

https://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Manuri Amaya Perera
Software Engineer
WSO2 Inc.




_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : +94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Manuri
In reply to this post by Asanka Abeyweera


On Fri, Jul 14, 2017 at 11:38 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Manuri,

Rather than syncing periodically (1-second or less), they say they can sync at every query with AOF [1]. I don't' know the performance impact of that approach. Need to test and see. I am not proposing to use Redis. Just pointing the information. 
Right.. Thanks for the information. 
​I missed what had been said about AOF before replying. 

On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera <[hidden email]> wrote:
Hi,

From the given description, it seems persistence is supported by writing to hard disk in configurable intervals. Unless we have a very low interval (which might be against any expected performance improvement brought by the database being in-memory) it cannot be assured that there won't be data loss. Also even a low interval would not fully guarantee that. 
I had a quick look at [1] and it says "You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually call the SAVE or BGSAVE commands.". So, if server crashes we could lose 1 - M changes. We can only reduce it but cannot avoid it completely. :)
So I'm not sure if we could use Redis.


Thanks,
Manuri

On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[hidden email]> wrote:
Hi Wishmitha,

If we use AOF wth fsync at every query, will there be a data loss with Redis? AOF might not have very good performance compared to RDB. But as Sasikal mentioned we cannot allow message losses in the broker.

On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[hidden email]> wrote:
Dear Wishmitha,

For all the databases mentioned please provide references to the relevant websites and where you got the information from. You have mentioned for some which is great and you should do so for others as well. Also when referencing don't have the weblink inline with the statement because it distracts the viewer as the links might be long. You can number the references and refer them below (You don't need to follow any standards, simply giving the link would suffice).

Cheers,
Riyafa

On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[hidden email]> wrote:
Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[hidden email]> wrote:
Hi All,

I have attached the pdf of above mentioned document.

Thanks.

On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[hidden email]> wrote:
Hi Wishmitha,

I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.


Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340



--
Malaka. 
-- 
Malaka Gangananda - Software Engineer | WSO2
Mobile : <a href="tel:+94%2071%20356%204340" value="+94713564340" target="_blank">+94713564340

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Sasikala Kottegoda
Senior Software Engineer
WSO2 Inc., http://wso2.com/
lean. enterprise. middleware

Mobile: <a href="tel:+94%2077%20483%205928" value="+94774835928" target="_blank">+94 774835928

https://wso2.com/signature

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Riyafa Abdul Hameed
Software Engineer, WSO2 Lanka (Pvt) Ltd

Email: [hidden email]
Website: https://riyafa.wordpress.com/

  

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Manuri Amaya Perera
Software Engineer
WSO2 Inc.




_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Asanka Abeyweera
Senior Software Engineer
WSO2 Inc.

Phone: <a href="tel:+94%2071%20222%208648" value="+94712228648" target="_blank">+94 712228648



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Manuri Amaya Perera
Software Engineer
WSO2 Inc.




_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Wishmitha Mendis
In reply to this post by Kevin Ratnasekera-2
Hi Kevin,

I just tried jedis for the initial implementation, no actual reason reason to be honest. I will try other java-clients you mentioned, and figure out what will fit the best for the project scenario.

Best Regards,

On Fri, Jul 14, 2017 at 11:17 PM, Kevin Ratnasekera <[hidden email]> wrote:
Hi Wishmitha,

There are many java clients available for redis, there are several very good clients, see comparison [1]. Any reason using jedis over others?

[1] https://redis.io/clients#java

Regards
Kevin



On Fri, Jul 14, 2017 at 11:02 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi Indika,

I have started a basic implementation to test the concurrency handling, message order and transaction support of Redis. Basic message producers and consumers were implemented and messages are stored in a basic Redis queue. GitHub : https://github.com/Wishmitha/RedisTesting

Best Regards,

On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath <[hidden email]> wrote:
Hi Wishmitha,

No SQL database is not an option as we have prior experience with Cassandra. Basically, the important part is to find out performance and fault tolerance capability in the database. Message broker read/write/delete records all the time and database should be able to handle high concurrency. Some metadata and message content write as byte data. Also, message order matters when delivery. Hence database should capable to sort records rather than bringing that logic into broker code. As you mentioned transaction is another factor we are looking in the database. Shall we start POC with Redis and see how it goes?

Cheers!

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Indika Sampath
Associate Technical Lead
WSO2 Inc. 
http://wso2.com

Phone: <a href="tel:+94%2071%20642%204744" value="+94716424744" target="_blank">+94 716 424 744
Blog: http://indikasampath.blogspot.com/


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Kevin Ratnaskera
Software Engineer
WSO2 Inc. - http://wso2.com
lean . enterprise . middleware
Mobile - <a href="tel:+94%2077%20469%206950" value="+94774696950" target="_blank">+94774696950

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : +94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Based Database for Message Broker Store

Manoj Gunawardena
Hi All,

Are we use, Redis distributed setup?
In a MB containerized Cluster (DOCKER), how can we ensure, the data remain when containers kill and start?
MB is supporting kubernates cluster?

Thanks

On Sun, Jul 16, 2017 at 9:41 AM, Wishmitha Mendis <[hidden email]> wrote:
Hi Kevin,

I just tried jedis for the initial implementation, no actual reason reason to be honest. I will try other java-clients you mentioned, and figure out what will fit the best for the project scenario.

Best Regards,

On Fri, Jul 14, 2017 at 11:17 PM, Kevin Ratnasekera <[hidden email]> wrote:
Hi Wishmitha,

There are many java clients available for redis, there are several very good clients, see comparison [1]. Any reason using jedis over others?

[1] https://redis.io/clients#java

Regards
Kevin



On Fri, Jul 14, 2017 at 11:02 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi Indika,

I have started a basic implementation to test the concurrency handling, message order and transaction support of Redis. Basic message producers and consumers were implemented and messages are stored in a basic Redis queue. GitHub : https://github.com/Wishmitha/RedisTesting

Best Regards,

On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath <[hidden email]> wrote:
Hi Wishmitha,

No SQL database is not an option as we have prior experience with Cassandra. Basically, the important part is to find out performance and fault tolerance capability in the database. Message broker read/write/delete records all the time and database should be able to handle high concurrency. Some metadata and message content write as byte data. Also, message order matters when delivery. Hence database should capable to sort records rather than bringing that logic into broker code. As you mentioned transaction is another factor we are looking in the database. Shall we start POC with Redis and see how it goes?

Cheers!

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[hidden email]> wrote:
Hi all,

I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

  • A simple key value storage

  • Used in ActiveMQ (replaced with KahaDB)

  1. Fast due to sorted keys

  2. Simple transaction support with batch( ) operation

  3. Sublevel feature (provides more organization and transactions facilitation)

  4. LiveStream feature (able to query things which are yet being receiving)

  5. Support triggers, groupby (mapreduce) and join (fanout) etc.

  1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.)

  2. No object relational mapping.

  3. Poor performance when memory is not enough.   

MongoDB


  • Document oriented

  • Can define JSON schemas


  1. Direct object mapping (good replacement for RDBMS)

  2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue)

  3. Capped collections (automatically remove old docs to make space for new docs.)

  1. No transaction support for multiple documents.

  2. Low performance compared to other NoSQL databases.

Caassandra


  • Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.)

  • Node - Cluster configuration (all nodes are identical)

  1. High availability (No single point of failure. In case of node failure other identical nodes are available.)

  2. No data loss due to replication (identical nodes)

  3. Easy to model (column store is very similar to RDBMS tables.)

  4. Similar query language to SQL (CQL)

  5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.)

  6. Transaction implementation

  1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)

Redis


  1. Fast read,write performances

  2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence)

  3. Can implement message queue structures. (http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/)

  4. Use to implement message stores. (https://redis.io/topics/pubsub)

  5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions)

  1. Loss of data (in case of power outage etc.)

  2. Depends on CPU performances when data amount is too high (https://redis.io/topics/persistence)

RocksDB


  • A simple key value storage (upgrade of LevelDB)

  • In memory database

  1. Fast read,write performances (faster than LevelDB)

  2. Can implement queue structures (https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB)

  3. Support concurrency

  4. Highly configurable (Pluggable Architecture)

  5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.)

  1. Loss of data (in case of power outage etc.)

  2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. 

And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases.

Best Regards

--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Indika Sampath
Associate Technical Lead
WSO2 Inc. 
http://wso2.com

Phone: <a href="tel:+94%2071%20642%204744" value="+94716424744" target="_blank">+94 716 424 744
Blog: http://indikasampath.blogspot.com/


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : <a href="tel:+94%2077%20757%207706" value="+94777577706" target="_blank">+94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Kevin Ratnaskera
Software Engineer
WSO2 Inc. - http://wso2.com
lean . enterprise . middleware
Mobile - <a href="tel:+94%2077%20469%206950" value="+94774696950" target="_blank">+94774696950

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Wishmitha Mendis
Intern - Software Engineering
WSO2

Mobile : +94 777577706



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Manoj Gunawardena
Tech Lead
WSO2, Inc.: http://wso2.com
lean.enterprise.middleware
Mobile : +94 77 2291643

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture