[APIM 2.x.y] Can I plug a Custom File System Provider ?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[APIM 2.x.y] Can I plug a Custom File System Provider ?

Youcef HILEM
Hi,

Having no answer
(http://wso2-oxygen-tank.10903.n7.nabble.com/APIM-2-x-y-Can-I-plug-a-Custom-File-System-Provider-td157891.html)
then I relay to DEV.



APIM 2.x.y] Can I plug a Custom File System Provider ?
Aug 06, 2018; 4:41pm — by  Youcef HILEMonline Youcef HILEM
Hi,

We have a multi-datacenter traget deployment and we want synchronize a
sharred folder between the two DC.
We don't/can't use rsync.

But we can use a solution like
https://github.com/Upplication/Amazon-S3-FileSystem-NIO2

My question : can we plug a Custom File System Provider in APIM ?

Regards
Youcef HILEM



--
Sent from: http://wso2-oxygen-tank.10903.n7.nabble.com/WSO2-Development-f3.html
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [APIM 2.x.y] Can I plug a Custom File System Provider ?

Youcef HILEM
Hi,

I complete my question by specifying my need.

As described here
(http://sanjeewamalalgoda.blogspot.com/2015/06/deploy-wso2-api-manager-across-multiple.html),
we want to push API published in Master Datacenter to a shaared server and
let Read Only nodes pick API config from there (we may not be able to use
deployment synchronizer here).

How to deposit and read this artifacts from the shared server?

Thanks
Youcef



--
Sent from: http://wso2-oxygen-tank.10903.n7.nabble.com/WSO2-Development-f3.html
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [APIM 2.x.y] Can I plug a Custom File System Provider ?

Nuwan Dias
The Deployment Synchronizer has an interface which could be implemented. We could write an S3 implementation for the Deployment Synchronizer.

Can someone point to the Dep-Sync interface and one of the implementations please? We had a Registry Based Dep-Sync implementation and a SVN based dep sync implementation.

On Thu, Aug 16, 2018 at 10:55 AM Youcef HILEM <[hidden email]> wrote:
Hi,

I complete my question by specifying my need.

As described here
(http://sanjeewamalalgoda.blogspot.com/2015/06/deploy-wso2-api-manager-across-multiple.html),
we want to push API published in Master Datacenter to a shaared server and
let Read Only nodes pick API config from there (we may not be able to use
deployment synchronizer here).

How to deposit and read this artifacts from the shared server?

Thanks
Youcef



--
Sent from: http://wso2-oxygen-tank.10903.n7.nabble.com/WSO2-Development-f3.html
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [APIM 2.x.y] Can I plug a Custom File System Provider ?

Kasun Gajasinghe



Important methods are #init, #commit, #checkout, #getRepositoryType.


On Fri, Aug 24, 2018 at 11:36 AM Nuwan Dias <[hidden email]> wrote:
The Deployment Synchronizer has an interface which could be implemented. We could write an S3 implementation for the Deployment Synchronizer.

Can someone point to the Dep-Sync interface and one of the implementations please? We had a Registry Based Dep-Sync implementation and a SVN based dep sync implementation.

On Thu, Aug 16, 2018 at 10:55 AM Youcef HILEM <[hidden email]> wrote:
Hi,

I complete my question by specifying my need.

As described here
(http://sanjeewamalalgoda.blogspot.com/2015/06/deploy-wso2-api-manager-across-multiple.html),
we want to push API published in Master Datacenter to a shaared server and
let Read Only nodes pick API config from there (we may not be able to use
deployment synchronizer here).

How to deposit and read this artifacts from the shared server?

Thanks
Youcef



--
Sent from: http://wso2-oxygen-tank.10903.n7.nabble.com/WSO2-Development-f3.html
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729


--
Kasun Gajasinghe
Technical Lead, WSO2 Inc.
email: kasung AT spamfree wso2.com
linked-in: http://lk.linkedin.com/in/gajasinghe
blog: http://kasunbg.org
phone: +1 650-745-4499, 77 678 0813 
 

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Kasun Gajasinghe
Software Engineer; WSO2 Inc.
http://kasunbg.org
Reply | Threaded
Open this post in threaded view
|

Re: [APIM 2.x.y] Can I plug a Custom File System Provider ?

Youcef HILEM
Thank you very much.
We will therfore implement two solutions:
- One based on S3 with a Cassandra back-end
(https://github.com/exoscale/pithos)
- Another based on kafka with compacted topics
(http://cloudurable.com/blog/kafka-architecture-log-compaction/index.html)
and Kafka mirroring (MirrorMaker)
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330)

The implementation is less complex with kafka. Moreover the updates will be
in "real time".

Regards
Youcef HILEM



--
Sent from: http://wso2-oxygen-tank.10903.n7.nabble.com/WSO2-Development-f3.html
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [APIM 2.x.y] Can I plug a Custom File System Provider ?

Kasun Gajasinghe
Hi Youcef,


On Fri, Aug 24, 2018 at 5:11 PM Youcef HILEM <[hidden email]> wrote:
Thank you very much.
We will therfore implement two solutions:
- One based on S3 with a Cassandra back-end
(https://github.com/exoscale/pithos)

Interesting. I'm curious why not use AWS S3 for storage as well? Data confidentiality concerns?

Does this also mean the ArtifactoryRepository is written independently of data storage? For people who are already on AWS, using AWS S3 object storage would make more sense.
 
- Another based on kafka with compacted topics
(http://cloudurable.com/blog/kafka-architecture-log-compaction/index.html)
and Kafka mirroring (MirrorMaker)
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330)

The implementation is less complex with kafka. Moreover the updates will be
in "real time".


The deployment task runs every 15seconds, and that's when the manager node will check for local changes, and check-in/commit the changes to the external repo. If there were indeed local file changes, carbon sends a depsync cluster message to rest of the nodes asking to update themselves. The cluster message is sent almost real-time.

Btw, with S3 sync, you can just commit and retrieve the file diffs. This is important because you don't want to update the timestamps of unchanged files. That'll lead to carbon deploying all the depl artifacts again and again. 

How do you plan to send just file diffs with Kafka? 

 
Regards
Youcef HILEM



--
Sent from: http://wso2-oxygen-tank.10903.n7.nabble.com/WSO2-Development-f3.html
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev


--
Kasun Gajasinghe
Technical Lead, WSO2 Inc.
email: kasung AT spamfree wso2.com
linked-in: http://lk.linkedin.com/in/gajasinghe
blog: http://kasunbg.org
phone: +1 650-745-4499, 77 678 0813 
 

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Kasun Gajasinghe
Software Engineer; WSO2 Inc.
http://kasunbg.org
Reply | Threaded
Open this post in threaded view
|

Re: [APIM 2.x.y] Can I plug a Custom File System Provider ?

Youcef HILEM
This post was updated on .
Hi Kasun,

With S3 API we can have different storages :
- https://github.com/gaul/s3proxy
- https://github.com/gaul/s3proxy/wiki/Storage-backend-examples 

With kafka I do not see for the moment how the manager node will check for
local changes.
But, I plan to send each changed (new, update, delete) file in a kafka
message :
- key : file path
- kafka "null messages" for deleting in a compacted topic :
https://stackoverflow.com/questions/38894156/kafka-deleting-messages-from-topics-with-retention-compact


Thanks
Youcef



--
Sent from: http://wso2-oxygen-tank.10903.n7.nabble.com/WSO2-Development-f3.html
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [APIM 2.x.y] Can I plug a Custom File System Provider ?

Youcef HILEM
Hi,

Which solution for s3 sync do you recommend between:

A java lib for s3 sync :


- https://jets3t.s3.amazonaws.com/applications/synchronize.html
- gradle task sample : https://github.com/literalice/gradle-aws-s3-sync
- mvn project : https://github.com/iterate-ch/jets3t

A python tool:


- https://s3tools.org/s3cmd-sync
- compatible with s3proxy : http://s3tools.org/s3cmd

Thanks
Youcef



--
Sent from: http://wso2-oxygen-tank.10903.n7.nabble.com/WSO2-Development-f3.html
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: [APIM 2.x.y] Can I plug a Custom File System Provider ?

Youcef HILEM
Hi,

I found this implementation :
https://github.com/Alsan/turing-chunk07/tree/master/components/deployment-synchronizer/org.wso2.carbon.deployment.synchronizer.s3/src/main/java/org/wso2/carbon/deployment/synchronizer/s3

See also :
http://wso2-oxygen-tank.10903.n7.nabble.com/S3-based-deployment-synchronizer-td68778.html

But how to deploy this solution since APIM 2.2.0 : « Publisher can play the
role of the Gateway Manager. Therefore, now, the Gateway Manager profile
(-Dprofile=gateway-manager) has been deprecated as it is redundant”
(https://docs.wso2.com/display/AM220/About+this+Release#AboutthisRelease-Deprecatedfeaturesandfunctionalities)


Thanks
Youcef HILEM



--
Sent from: http://wso2-oxygen-tank.10903.n7.nabble.com/WSO2-Development-f3.html
_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev