File Connector Version2 for WSO2 ESB.

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

File Connector Version2 for WSO2 ESB.

vivekananthan
Hi All,

I am going to implement File Connector Version2, So I have planned to improve existing file connector as well as add more operations with existing one.In current file connector, more parameters are required to do some operations (e.g.move operation).

I found some issues with existing one.

1) The Copy operations does not work for zip files (after coping zip files , that file will be corrupted) .
2)Lot of parameters are required.
3)Archive operation does not work for remote files.

So I am going to fix this problems in version2.

And I have planned to add more functionalities. Such as

1)List all files name inside a folder for a given folder(In current File Connector , This operation is working for a zip files).
2)Compress a folder including remote file(ftp).
3)Arrange the files in ascending or descending order.
4)Hide a folder.

Please give me some suggestions and what are the operations I have to consider in File Connector version 2.



Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:+94752786138

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Connector Version2 for WSO2 ESB.

Sriskandarajah Suhothayan
We should be able to read a large xml file in a streaming mode and extract several xml sub elements. 
and read line by line 

Suho

On Fri, Sep 4, 2015 at 2:21 PM, Vivekananthan Sivanayagam <[hidden email]> wrote:
Hi All,

I am going to implement File Connector Version2, So I have planned to improve existing file connector as well as add more operations with existing one.In current file connector, more parameters are required to do some operations (e.g.move operation).

I found some issues with existing one.

1) The Copy operations does not work for zip files (after coping zip files , that file will be corrupted) .
2)Lot of parameters are required.
3)Archive operation does not work for remote files.

So I am going to fix this problems in version2.

And I have planned to add more functionalities. Such as

1)List all files name inside a folder for a given folder(In current File Connector , This operation is working for a zip files).
2)Compress a folder including remote file(ftp).
3)Arrange the files in ascending or descending order.
4)Hide a folder.

Please give me some suggestions and what are the operations I have to consider in File Connector version 2.



Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138



--
S. Suhothayan
Technical Lead & Team Lead of WSO2 Complex Event Processor
lean . enterprise . middleware



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Connector Version2 for WSO2 ESB.

Cyril Rognon-2
In reply to this post by vivekananthan
Hi,

this is good news. We also have planned to enhance FileConnector :

- add capability to process non xml files
- correct some errors whith xml prologue
- add capability to read from an archive (zip) - this one is in your scope I suppose

And more my be coming soon.



Cyril

2015-09-04 10:51 GMT+02:00 Vivekananthan Sivanayagam <[hidden email]>:
Hi All,

I am going to implement File Connector Version2, So I have planned to improve existing file connector as well as add more operations with existing one.In current file connector, more parameters are required to do some operations (e.g.move operation).

I found some issues with existing one.

1) The Copy operations does not work for zip files (after coping zip files , that file will be corrupted) .
2)Lot of parameters are required.
3)Archive operation does not work for remote files.

So I am going to fix this problems in version2.

And I have planned to add more functionalities. Such as

1)List all files name inside a folder for a given folder(In current File Connector , This operation is working for a zip files).
2)Compress a folder including remote file(ftp).
3)Arrange the files in ascending or descending order.
4)Hide a folder.

Please give me some suggestions and what are the operations I have to consider in File Connector version 2.



Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:+94752786138

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Connector Version2 for WSO2 ESB.

malakasilva
Hi Vivekananthan,

We have the requirement to process a large file by processing one record at a time and the mediation of that record/message, then to proceed with the next. 

This is a valid and very common use case that can implement at File connector new version. 

Can we come up with an approach to address all such requirements? 

On Mon, Sep 7, 2015 at 7:48 PM, Vivekananthan Sivanayagam <[hidden email]> wrote:
Hi all,
Thank you. I will consider these suggestion.




Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

On Mon, Sep 7, 2015 at 3:32 PM, Aiyadurai Rajeevan <[hidden email]> wrote:
Hi,

We should be able to copy files as blocks and aggregate them together, This functionality would be more useful when deal with fairly big files.

Thanks & Regards,
S.A.Rajeevan
Software Engineer WSO2 Inc
E-Mail: [hidden email] | Mobile : <a href="tel:%2B94776411636" value="+94776411636" target="_blank">+94776411636

On Fri, Sep 4, 2015 at 3:03 PM, Cyril Rognon <[hidden email]> wrote:
Hi,

this is good news. We also have planned to enhance FileConnector :

- add capability to process non xml files
- correct some errors whith xml prologue
- add capability to read from an archive (zip) - this one is in your scope I suppose

And more my be coming soon.



Cyril

2015-09-04 10:51 GMT+02:00 Vivekananthan Sivanayagam <[hidden email]>:
Hi All,

I am going to implement File Connector Version2, So I have planned to improve existing file connector as well as add more operations with existing one.In current file connector, more parameters are required to do some operations (e.g.move operation).

I found some issues with existing one.

1) The Copy operations does not work for zip files (after coping zip files , that file will be corrupted) .
2)Lot of parameters are required.
3)Archive operation does not work for remote files.

So I am going to fix this problems in version2.

And I have planned to add more functionalities. Such as

1)List all files name inside a folder for a given folder(In current File Connector , This operation is working for a zip files).
2)Compress a folder including remote file(ftp).
3)Arrange the files in ascending or descending order.
4)Hide a folder.

Please give me some suggestions and what are the operations I have to consider in File Connector version 2.



Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture






--

Best Regards,

Malaka Silva
Senior Tech Lead
M: +94 777 219 791
Fax :94 11 2145300 
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77

WSO2, Inc. 
lean . enterprise . middleware 
http://www.wso2.com/

Save a tree -Conserve nature & Save the world for your future. Print this email only if it is absolutely necessary.

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Connector Version2 for WSO2 ESB.

Viraj Senevirathne
Hi Malaka,

This STAX API [1]can be used to parse any xml file using streams which can be help full is we are processing a large xml file record after record.


Thank you,

On Fri, Sep 25, 2015 at 11:36 AM, Vivekananthan Sivanayagam <[hidden email]> wrote:
Hi Malaka,

This means , given a file should be divided into multiple parts and then read one after the another each divided part?





Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

On Fri, Sep 25, 2015 at 10:41 AM, Malaka Silva <[hidden email]> wrote:
Hi Vivekananthan,

We have the requirement to process a large file by processing one record at a time and the mediation of that record/message, then to proceed with the next. 

This is a valid and very common use case that can implement at File connector new version. 

Can we come up with an approach to address all such requirements? 

On Mon, Sep 7, 2015 at 7:48 PM, Vivekananthan Sivanayagam <[hidden email]> wrote:
Hi all,
Thank you. I will consider these suggestion.




Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

On Mon, Sep 7, 2015 at 3:32 PM, Aiyadurai Rajeevan <[hidden email]> wrote:
Hi,

We should be able to copy files as blocks and aggregate them together, This functionality would be more useful when deal with fairly big files.

Thanks & Regards,
S.A.Rajeevan
Software Engineer WSO2 Inc
E-Mail: [hidden email] | Mobile : <a href="tel:%2B94776411636" value="+94776411636" target="_blank">+94776411636

On Fri, Sep 4, 2015 at 3:03 PM, Cyril Rognon <[hidden email]> wrote:
Hi,

this is good news. We also have planned to enhance FileConnector :

- add capability to process non xml files
- correct some errors whith xml prologue
- add capability to read from an archive (zip) - this one is in your scope I suppose

And more my be coming soon.



Cyril

2015-09-04 10:51 GMT+02:00 Vivekananthan Sivanayagam <[hidden email]>:
Hi All,

I am going to implement File Connector Version2, So I have planned to improve existing file connector as well as add more operations with existing one.In current file connector, more parameters are required to do some operations (e.g.move operation).

I found some issues with existing one.

1) The Copy operations does not work for zip files (after coping zip files , that file will be corrupted) .
2)Lot of parameters are required.
3)Archive operation does not work for remote files.

So I am going to fix this problems in version2.

And I have planned to add more functionalities. Such as

1)List all files name inside a folder for a given folder(In current File Connector , This operation is working for a zip files).
2)Compress a folder including remote file(ftp).
3)Arrange the files in ascending or descending order.
4)Hide a folder.

Please give me some suggestions and what are the operations I have to consider in File Connector version 2.



Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture






--

Best Regards,

Malaka Silva
Senior Tech Lead
M: <a href="tel:%2B94%20777%20219%20791" value="+94777219791" target="_blank">+94 777 219 791
Fax :94 11 2145300 
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77

WSO2, Inc. 
lean . enterprise . middleware 
http://www.wso2.com/

Save a tree -Conserve nature & Save the world for your future. Print this email only if it is absolutely necessary.




--
Viraj Senevirathne
Software Engineer; WSO2, Inc.


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Connector Version2 for WSO2 ESB.

malakasilva
Hi Philippe,

Thx for the input. But mediation(mediators) and connectors are at different layers. These are not tight coupled.

Tran formation is part of ESB where as connector is an input or output component. 

Inline image 1

On Fri, Sep 25, 2015 at 4:09 PM, Philippe Sevestre <[hidden email]> wrote:
Hi,

Since large files also come in non-XML formats, maybe you should also consider using smooks at the file connector level. With that, formats like EDI, CSV, etc would also benefit from the new file connector.

Philippe

On Fri, Sep 25, 2015 at 4:08 AM, Viraj Senevirathne <[hidden email]> wrote:
Hi Malaka,

This STAX API [1]can be used to parse any xml file using streams which can be help full is we are processing a large xml file record after record.


Thank you,

On Fri, Sep 25, 2015 at 11:36 AM, Vivekananthan Sivanayagam <[hidden email]> wrote:
Hi Malaka,

This means , given a file should be divided into multiple parts and then read one after the another each divided part?





Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

On Fri, Sep 25, 2015 at 10:41 AM, Malaka Silva <[hidden email]> wrote:
Hi Vivekananthan,

We have the requirement to process a large file by processing one record at a time and the mediation of that record/message, then to proceed with the next. 

This is a valid and very common use case that can implement at File connector new version. 

Can we come up with an approach to address all such requirements? 

On Mon, Sep 7, 2015 at 7:48 PM, Vivekananthan Sivanayagam <[hidden email]> wrote:
Hi all,
Thank you. I will consider these suggestion.




Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

On Mon, Sep 7, 2015 at 3:32 PM, Aiyadurai Rajeevan <[hidden email]> wrote:
Hi,

We should be able to copy files as blocks and aggregate them together, This functionality would be more useful when deal with fairly big files.

Thanks & Regards,
S.A.Rajeevan
Software Engineer WSO2 Inc
E-Mail: [hidden email] | Mobile : <a href="tel:%2B94776411636" value="+94776411636" target="_blank">+94776411636

On Fri, Sep 4, 2015 at 3:03 PM, Cyril Rognon <[hidden email]> wrote:
Hi,

this is good news. We also have planned to enhance FileConnector :

- add capability to process non xml files
- correct some errors whith xml prologue
- add capability to read from an archive (zip) - this one is in your scope I suppose

And more my be coming soon.



Cyril

2015-09-04 10:51 GMT+02:00 Vivekananthan Sivanayagam <[hidden email]>:
Hi All,

I am going to implement File Connector Version2, So I have planned to improve existing file connector as well as add more operations with existing one.In current file connector, more parameters are required to do some operations (e.g.move operation).

I found some issues with existing one.

1) The Copy operations does not work for zip files (after coping zip files , that file will be corrupted) .
2)Lot of parameters are required.
3)Archive operation does not work for remote files.

So I am going to fix this problems in version2.

And I have planned to add more functionalities. Such as

1)List all files name inside a folder for a given folder(In current File Connector , This operation is working for a zip files).
2)Compress a folder including remote file(ftp).
3)Arrange the files in ascending or descending order.
4)Hide a folder.

Please give me some suggestions and what are the operations I have to consider in File Connector version 2.



Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture






--

Best Regards,

Malaka Silva
Senior Tech Lead
M: <a href="tel:%2B94%20777%20219%20791" value="+94777219791" target="_blank">+94 777 219 791
Fax :<a href="tel:94%2011%C2%A02145300" value="+94112145300" target="_blank">94 11 2145300 
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77

WSO2, Inc. 
lean . enterprise . middleware 
http://www.wso2.com/

Save a tree -Conserve nature & Save the world for your future. Print this email only if it is absolutely necessary.




--
Viraj Senevirathne
Software Engineer; WSO2, Inc.


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

www.lighthouse.com.br

Philippe Sevestre ([hidden email])
Tel.: <a href="tel:%2B55%20%2811%29%204063-0968" value="+551140630968" target="_blank">+55 (11) 4063-0968
Cel.: +55 (11) 9-9735-2586
Av. Brig. Faria Lima, 1461 - 4o andar 01452-002 São Paulo, SP - Brasil



--

Best Regards,

Malaka Silva
Senior Tech Lead
M: +94 777 219 791
Fax :94 11 2145300 
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77

WSO2, Inc. 
lean . enterprise . middleware 
http://www.wso2.com/

Save a tree -Conserve nature & Save the world for your future. Print this email only if it is absolutely necessary.

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: File Connector Version2 for WSO2 ESB.

malakasilva
Hi Philippe,

Thank you for the explanation. Will think about this as well.

On Fri, Sep 25, 2015 at 6:23 PM, Philippe Sevestre <[hidden email]> wrote:
Hi Malaka,

I´m quite aware of this distinction. My point is that smooks already provides for multiple input formats and also operates in a streaming way - this is why it is so widely used in ETL scenarios.

The idea is that a "smooks"connector/transport" would hook into its processing loop and feed the synapse engine with a new message for each record. In this sense, both engines would operate more or less like a coroutine pattern.

Philippe.


On Fri, Sep 25, 2015 at 9:27 AM, Malaka Silva <[hidden email]> wrote:
Hi Philippe,

Thx for the input. But mediation(mediators) and connectors are at different layers. These are not tight coupled.

Tran formation is part of ESB where as connector is an input or output component. 

Inline image 1

On Fri, Sep 25, 2015 at 4:09 PM, Philippe Sevestre <[hidden email]> wrote:
Hi,

Since large files also come in non-XML formats, maybe you should also consider using smooks at the file connector level. With that, formats like EDI, CSV, etc would also benefit from the new file connector.

Philippe

On Fri, Sep 25, 2015 at 4:08 AM, Viraj Senevirathne <[hidden email]> wrote:
Hi Malaka,

This STAX API [1]can be used to parse any xml file using streams which can be help full is we are processing a large xml file record after record.


Thank you,

On Fri, Sep 25, 2015 at 11:36 AM, Vivekananthan Sivanayagam <[hidden email]> wrote:
Hi Malaka,

This means , given a file should be divided into multiple parts and then read one after the another each divided part?





Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

On Fri, Sep 25, 2015 at 10:41 AM, Malaka Silva <[hidden email]> wrote:
Hi Vivekananthan,

We have the requirement to process a large file by processing one record at a time and the mediation of that record/message, then to proceed with the next. 

This is a valid and very common use case that can implement at File connector new version. 

Can we come up with an approach to address all such requirements? 

On Mon, Sep 7, 2015 at 7:48 PM, Vivekananthan Sivanayagam <[hidden email]> wrote:
Hi all,
Thank you. I will consider these suggestion.




Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

On Mon, Sep 7, 2015 at 3:32 PM, Aiyadurai Rajeevan <[hidden email]> wrote:
Hi,

We should be able to copy files as blocks and aggregate them together, This functionality would be more useful when deal with fairly big files.

Thanks & Regards,
S.A.Rajeevan
Software Engineer WSO2 Inc
E-Mail: [hidden email] | Mobile : <a href="tel:%2B94776411636" value="+94776411636" target="_blank">+94776411636

On Fri, Sep 4, 2015 at 3:03 PM, Cyril Rognon <[hidden email]> wrote:
Hi,

this is good news. We also have planned to enhance FileConnector :

- add capability to process non xml files
- correct some errors whith xml prologue
- add capability to read from an archive (zip) - this one is in your scope I suppose

And more my be coming soon.



Cyril

2015-09-04 10:51 GMT+02:00 Vivekananthan Sivanayagam <[hidden email]>:
Hi All,

I am going to implement File Connector Version2, So I have planned to improve existing file connector as well as add more operations with existing one.In current file connector, more parameters are required to do some operations (e.g.move operation).

I found some issues with existing one.

1) The Copy operations does not work for zip files (after coping zip files , that file will be corrupted) .
2)Lot of parameters are required.
3)Archive operation does not work for remote files.

So I am going to fix this problems in version2.

And I have planned to add more functionalities. Such as

1)List all files name inside a folder for a given folder(In current File Connector , This operation is working for a zip files).
2)Compress a folder including remote file(ftp).
3)Arrange the files in ascending or descending order.
4)Hide a folder.

Please give me some suggestions and what are the operations I have to consider in File Connector version 2.



Thank you
Vivekananthan Sivanayagam
Associate Software Engineer | WSO2
M:<a href="tel:%2B94752786138" value="+94752786138" target="_blank">+94752786138

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture






--

Best Regards,

Malaka Silva
Senior Tech Lead
M: <a href="tel:%2B94%20777%20219%20791" value="+94777219791" target="_blank">+94 777 219 791
Fax :<a href="tel:94%2011%C2%A02145300" value="+94112145300" target="_blank">94 11 2145300 
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77

WSO2, Inc. 
lean . enterprise . middleware 
http://www.wso2.com/

Save a tree -Conserve nature & Save the world for your future. Print this email only if it is absolutely necessary.




--
Viraj Senevirathne
Software Engineer; WSO2, Inc.


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--

www.lighthouse.com.br

Philippe Sevestre ([hidden email])
Tel.: <a href="tel:%2B55%20%2811%29%204063-0968" value="+551140630968" target="_blank">+55 (11) 4063-0968
Cel.: +55 (11) 9-9735-2586
Av. Brig. Faria Lima, 1461 - 4o andar 01452-002 São Paulo, SP - Brasil



--

Best Regards,

Malaka Silva
Senior Tech Lead
M: <a href="tel:%2B94%20777%20219%20791" value="+94777219791" target="_blank">+94 777 219 791
Fax :<a href="tel:94%2011%C2%A02145300" value="+94112145300" target="_blank">94 11 2145300 
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77

WSO2, Inc. 
lean . enterprise . middleware 
http://www.wso2.com/

Save a tree -Conserve nature & Save the world for your future. Print this email only if it is absolutely necessary.



--

www.lighthouse.com.br

Philippe Sevestre ([hidden email])
Tel.: <a href="tel:%2B55%20%2811%29%204063-0968" value="+551140630968" target="_blank">+55 (11) 4063-0968
Cel.: +55 (11) 9-9735-2586
Av. Brig. Faria Lima, 1461 - 4o andar 01452-002 São Paulo, SP - Brasil



--

Best Regards,

Malaka Silva
Senior Tech Lead
M: +94 777 219 791
Fax :94 11 2145300 
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77

WSO2, Inc. 
lean . enterprise . middleware 
http://www.wso2.com/

Save a tree -Conserve nature & Save the world for your future. Print this email only if it is absolutely necessary.

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture