[RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Hasunie Adikari
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Isuru Udana
Hi Hasunie,

Are we gaining a significant performance improvement in this approach by making mediators content-unaware ?  
To perform these validations messages should be built and we are internally using woodstox in our builders too.  

At mediator level generally we shouldn't mess-up with input-streams, buffers and these various internal properties in Passthrough Transport.  
If it is inevitable, we need to be extra careful as these are very sensitive.

Thanks.

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: +94 77 3791887
blog: http://mytecheye.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Hasunie Adikari
Hi Isuru,

Here we are trying to avoid building the message in gateway itself since building the entire malicious payload may lead gateway crashes. Instead of counting limits after building the tree, We manually build the message through third-party library and parser will keep counting nodes while parsing object. So when the limit hit, it will terminate the process and stop further processing. We have planned to thwart following attacks[1].Here we provide both depth limits and schema validations.

[1]
XML Bomb
Coercive Parsing
XXE (External Entity Attacks)  
Schema Poisoning
Buffer Overflows


Regards
Hasunie

On Tue, Jan 2, 2018 at 8:22 PM, Isuru Udana <[hidden email]> wrote:
Hi Hasunie,

Are we gaining a significant performance improvement in this approach by making mediators content-unaware ?  
To perform these validations messages should be built and we are internally using woodstox in our builders too.  

At mediator level generally we shouldn't mess-up with input-streams, buffers and these various internal properties in Passthrough Transport.  
If it is inevitable, we need to be extra careful as these are very sensitive.

Thanks.

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:+94713095876


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Vinod Kavinda
In reply to this post by Hasunie Adikari
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Dushan Abeyruwan
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago). To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:(001)408-791-9312


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Isuru Udana
Hi Dushan,

On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[hidden email]> wrote:
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago).
No. I think it was force.passthrough.builder property which you introduced for ELB requirement.  

To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: +94 77 3791887
blog: http://mytecheye.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Hasunie Adikari
Hi Isuru,

As we discussed, I cloned the input stream by consuming the passthrough pipe as in below.


if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());

        }
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buffer)) > -1 ) {
    byteArrayOutputStream.write(buffer, 0, len);
}
byteArrayOutputStream.flush();


InputStream is1 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 
InputStream is2 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 

consume the clones for the validation and set another clone as a buffereInputstream property in the axis2messagecontext.

BufferedInputStream bufferedInputStreamOriginal = new BufferedInputStream(inputStreamOriginal);
axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, bufferedInputStreamOriginal);

I'm still getting the stream closed issue only for correct the messages which have been passed through multiple validators. If the validators throw an exception, the request is getting build and generate the custom response as expected. It seems like we implemented a way that gets the inputstream from the passthrough pipe for the content unaware flows. Unless it uses to get the inputstream from the messagecontext. It was proven once I attached an empty content aware mediator and test the same flow. I was able to observe the expected behaviour for the same implementation with the content aware mediator.

Do we have a way to define cloned input stream as an original inputstream in passthrough pipe?


Regards
Hasunie






On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[hidden email]> wrote:
Hi Dushan,

On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[hidden email]> wrote:
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago).
No. I think it was force.passthrough.builder property which you introduced for ELB requirement.  

To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:+94713095876


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Isuru Udana
Hi Hasunie,

As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM has no effect on the flow in this case and Passthough Sender still seek content from the original input stream which got empty due to this cloning logic. That's the reason for this behaviour.

Thanks.



On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[hidden email]> wrote:
Hi Isuru,

As we discussed, I cloned the input stream by consuming the passthrough pipe as in below.


if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());

        }
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buffer)) > -1 ) {
    byteArrayOutputStream.write(buffer, 0, len);
}
byteArrayOutputStream.flush();


InputStream is1 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 
InputStream is2 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 

consume the clones for the validation and set another clone as a buffereInputstream property in the axis2messagecontext.

BufferedInputStream bufferedInputStreamOriginal = new BufferedInputStream(inputStreamOriginal);
axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, bufferedInputStreamOriginal);

I'm still getting the stream closed issue only for correct the messages which have been passed through multiple validators. If the validators throw an exception, the request is getting build and generate the custom response as expected. It seems like we implemented a way that gets the inputstream from the passthrough pipe for the content unaware flows. Unless it uses to get the inputstream from the messagecontext. It was proven once I attached an empty content aware mediator and test the same flow. I was able to observe the expected behaviour for the same implementation with the content aware mediator.

Do we have a way to define cloned input stream as an original inputstream in passthrough pipe?


Regards
Hasunie






On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[hidden email]> wrote:
Hi Dushan,

On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[hidden email]> wrote:
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago).
No. I think it was force.passthrough.builder property which you introduced for ELB requirement.  

To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: +94 77 3791887
blog: http://mytecheye.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Dimuthu Leelarathne-2
Hi All,

I am ok as long as we are invoking a method and not copying.

thanks,
Dimuthu

On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[hidden email]> wrote:
Hi Hasunie,

As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM has no effect on the flow in this case and Passthough Sender still seek content from the original input stream which got empty due to this cloning logic. That's the reason for this behaviour.

Thanks.



On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[hidden email]> wrote:
Hi Isuru,

As we discussed, I cloned the input stream by consuming the passthrough pipe as in below.


if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());

        }
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buffer)) > -1 ) {
    byteArrayOutputStream.write(buffer, 0, len);
}
byteArrayOutputStream.flush();


InputStream is1 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 
InputStream is2 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 

consume the clones for the validation and set another clone as a buffereInputstream property in the axis2messagecontext.

BufferedInputStream bufferedInputStreamOriginal = new BufferedInputStream(inputStreamOriginal);
axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, bufferedInputStreamOriginal);

I'm still getting the stream closed issue only for correct the messages which have been passed through multiple validators. If the validators throw an exception, the request is getting build and generate the custom response as expected. It seems like we implemented a way that gets the inputstream from the passthrough pipe for the content unaware flows. Unless it uses to get the inputstream from the messagecontext. It was proven once I attached an empty content aware mediator and test the same flow. I was able to observe the expected behaviour for the same implementation with the content aware mediator.

Do we have a way to define cloned input stream as an original inputstream in passthrough pipe?


Regards
Hasunie






On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[hidden email]> wrote:
Hi Dushan,

On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[hidden email]> wrote:
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago).
No. I think it was force.passthrough.builder property which you introduced for ELB requirement.  

To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Dimuthu Leelarathne 
Director, Solutions Architecture

WSO2, Inc. (http://wso2.com)
email: 
[hidden email]
Mobile: +94773661935 
Blog: http://muthulee.blogspot.com

Lean . Enterprise . Middleware

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Hasunie Adikari
In reply to this post by Isuru Udana
Hi all,

As I discussed with Isuru, There are some possible approaches to overcome the issue.

1. Create a new pass through pipe.
   - The data will be written to the pipe by a spawned thread and current thread will be consuming the data and continuing the message flow. We went through the pipe creation logic and seemed it tightly coupled with encoding, decoding methodologies so that it can't be 
      implemented at APIM level. 

2. Change the synapse level logic to get the ByteArrayInputstream and write it into the response.
   - It can be but have to thoroughly go through and do it carefully unless the default message flow would be affected.

3. Build the message by invoking RelayUtils.buildMessage() at the validator mediator after successfully parsing all the validators.
    - It will be slightly affected the performance but this is the straightforward solution at this moment. 

I have improved the code by applying the 3rd option as we discussed. Setting the PassThroughConstants.BUFFERED_INPUT_STREAM has an effect now onwards since we changed a way that building the message to achieve the content aware behavior which seeks the
inputream from the axis2 message context instead of the original inputstream.


Regards
Hasunie

      

On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[hidden email]> wrote:
Hi Hasunie,

As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM has no effect on the flow in this case and Passthough Sender still seek content from the original input stream which got empty due to this cloning logic. That's the reason for this behaviour.

Thanks.



On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[hidden email]> wrote:
Hi Isuru,

As we discussed, I cloned the input stream by consuming the passthrough pipe as in below.


if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());

        }
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buffer)) > -1 ) {
    byteArrayOutputStream.write(buffer, 0, len);
}
byteArrayOutputStream.flush();


InputStream is1 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 
InputStream is2 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 

consume the clones for the validation and set another clone as a buffereInputstream property in the axis2messagecontext.

BufferedInputStream bufferedInputStreamOriginal = new BufferedInputStream(inputStreamOriginal);
axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, bufferedInputStreamOriginal);

I'm still getting the stream closed issue only for correct the messages which have been passed through multiple validators. If the validators throw an exception, the request is getting build and generate the custom response as expected. It seems like we implemented a way that gets the inputstream from the passthrough pipe for the content unaware flows. Unless it uses to get the inputstream from the messagecontext. It was proven once I attached an empty content aware mediator and test the same flow. I was able to observe the expected behaviour for the same implementation with the content aware mediator.

Do we have a way to define cloned input stream as an original inputstream in passthrough pipe?


Regards
Hasunie






On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[hidden email]> wrote:
Hi Dushan,

On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[hidden email]> wrote:
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago).
No. I think it was force.passthrough.builder property which you introduced for ELB requirement.  

To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:+94713095876


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Hasunie Adikari
Hi all,

I encounter an issue while writing a unit test to clone the input stream. I implemented the cloning method[1] a way that gets the input stream from the passthrough pipe. As I discussed with the EI team, We can't mock the pipe and also there were some obstacles for creating the pipe and I addressed the issue in the same mail thread. So I am gonna create packeage private clone method to continue the unit test flow.

[1] Pipe pipe = (Pipe) axis2MC.getProperty(PassThroughConstants.PASS_THROUGH_PIPE);
        BufferedInputStream bufferedInputStream = null;
        Map<String, InputStream> inputStreams = null;
        InputStream inputStreamSchema;
        InputStream inputStreamXml;
       ---------
        if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());
        }
        if (bufferedInputStream != null) {
            ByteArrayOutputStream byteArrayOutputStreamSchema = new ByteArrayOutputStream();
            byte[] buffer = new byte[1024];
            int length;

            while ((length = bufferedInputStream.read(buffer)) > -1) {
                byteArrayOutputStreamSchema.write(buffer, 0, length);
            }
            byteArrayOutputStreamSchema.flush();
            inputStreamMap = new HashMap<>();
            inputStreamSchema = new ByteArrayInputStream(byteArrayOutputStreamSchema.toByteArray());
            inputStreamXml = new ByteArrayInputStream(byteArrayOutputStreamSchema.toByteArray());
            inputStreamSql = new ByteArrayInputStream(byteArrayOutputStreamSchema.toByteArray());
          ------
            inputStreams.put("Schema", inputStreamSchema);
            inputStreams.put("XML", inputStreamXml);
          ---
        }

Regards
Hasunie

On Tue, Jan 9, 2018 at 11:08 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

As I discussed with Isuru, There are some possible approaches to overcome the issue.

1. Create a new pass through pipe.
   - The data will be written to the pipe by a spawned thread and current thread will be consuming the data and continuing the message flow. We went through the pipe creation logic and seemed it tightly coupled with encoding, decoding methodologies so that it can't be 
      implemented at APIM level. 

2. Change the synapse level logic to get the ByteArrayInputstream and write it into the response.
   - It can be but have to thoroughly go through and do it carefully unless the default message flow would be affected.

3. Build the message by invoking RelayUtils.buildMessage() at the validator mediator after successfully parsing all the validators.
    - It will be slightly affected the performance but this is the straightforward solution at this moment. 

I have improved the code by applying the 3rd option as we discussed. Setting the PassThroughConstants.BUFFERED_INPUT_STREAM has an effect now onwards since we changed a way that building the message to achieve the content aware behavior which seeks the
inputream from the axis2 message context instead of the original inputstream.


Regards
Hasunie

      

On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[hidden email]> wrote:
Hi Hasunie,

As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM has no effect on the flow in this case and Passthough Sender still seek content from the original input stream which got empty due to this cloning logic. That's the reason for this behaviour.

Thanks.



On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[hidden email]> wrote:
Hi Isuru,

As we discussed, I cloned the input stream by consuming the passthrough pipe as in below.


if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());

        }
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buffer)) > -1 ) {
    byteArrayOutputStream.write(buffer, 0, len);
}
byteArrayOutputStream.flush();


InputStream is1 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 
InputStream is2 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 

consume the clones for the validation and set another clone as a buffereInputstream property in the axis2messagecontext.

BufferedInputStream bufferedInputStreamOriginal = new BufferedInputStream(inputStreamOriginal);
axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, bufferedInputStreamOriginal);

I'm still getting the stream closed issue only for correct the messages which have been passed through multiple validators. If the validators throw an exception, the request is getting build and generate the custom response as expected. It seems like we implemented a way that gets the inputstream from the passthrough pipe for the content unaware flows. Unless it uses to get the inputstream from the messagecontext. It was proven once I attached an empty content aware mediator and test the same flow. I was able to observe the expected behaviour for the same implementation with the content aware mediator.

Do we have a way to define cloned input stream as an original inputstream in passthrough pipe?


Regards
Hasunie






On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[hidden email]> wrote:
Hi Dushan,

On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[hidden email]> wrote:
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago).
No. I think it was force.passthrough.builder property which you introduced for ELB requirement.  

To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:+94713095876


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Dimuthu Leelarathne-2


On Fri, Jan 12, 2018 at 3:21 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I encounter an issue while writing a unit test to clone the input stream. I implemented the cloning method[1] a way that gets the input stream from the passthrough pipe. As I discussed with the EI team, We can't mock the pipe and also there were some obstacles for creating the pipe and I addressed the issue in the same mail thread. So I am gonna create packeage private clone method to continue the unit test flow.

+1. That is one reason for test classes and implementation to have the same package.

thanks,
Dimuthu


[1] Pipe pipe = (Pipe) axis2MC.getProperty(PassThroughConstants.PASS_THROUGH_PIPE);
        BufferedInputStream bufferedInputStream = null;
        Map<String, InputStream> inputStreams = null;
        InputStream inputStreamSchema;
        InputStream inputStreamXml;
       ---------
        if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());
        }
        if (bufferedInputStream != null) {
            ByteArrayOutputStream byteArrayOutputStreamSchema = new ByteArrayOutputStream();
            byte[] buffer = new byte[1024];
            int length;

            while ((length = bufferedInputStream.read(buffer)) > -1) {
                byteArrayOutputStreamSchema.write(buffer, 0, length);
            }
            byteArrayOutputStreamSchema.flush();
            inputStreamMap = new HashMap<>();
            inputStreamSchema = new ByteArrayInputStream(byteArrayOutputStreamSchema.toByteArray());
            inputStreamXml = new ByteArrayInputStream(byteArrayOutputStreamSchema.toByteArray());
            inputStreamSql = new ByteArrayInputStream(byteArrayOutputStreamSchema.toByteArray());
          ------
            inputStreams.put("Schema", inputStreamSchema);
            inputStreams.put("XML", inputStreamXml);
          ---
        }

Regards
Hasunie

On Tue, Jan 9, 2018 at 11:08 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

As I discussed with Isuru, There are some possible approaches to overcome the issue.

1. Create a new pass through pipe.
   - The data will be written to the pipe by a spawned thread and current thread will be consuming the data and continuing the message flow. We went through the pipe creation logic and seemed it tightly coupled with encoding, decoding methodologies so that it can't be 
      implemented at APIM level. 

2. Change the synapse level logic to get the ByteArrayInputstream and write it into the response.
   - It can be but have to thoroughly go through and do it carefully unless the default message flow would be affected.

3. Build the message by invoking RelayUtils.buildMessage() at the validator mediator after successfully parsing all the validators.
    - It will be slightly affected the performance but this is the straightforward solution at this moment. 

I have improved the code by applying the 3rd option as we discussed. Setting the PassThroughConstants.BUFFERED_INPUT_STREAM has an effect now onwards since we changed a way that building the message to achieve the content aware behavior which seeks the
inputream from the axis2 message context instead of the original inputstream.


Regards
Hasunie

      

On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[hidden email]> wrote:
Hi Hasunie,

As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM has no effect on the flow in this case and Passthough Sender still seek content from the original input stream which got empty due to this cloning logic. That's the reason for this behaviour.

Thanks.



On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[hidden email]> wrote:
Hi Isuru,

As we discussed, I cloned the input stream by consuming the passthrough pipe as in below.


if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());

        }
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buffer)) > -1 ) {
    byteArrayOutputStream.write(buffer, 0, len);
}
byteArrayOutputStream.flush();


InputStream is1 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 
InputStream is2 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 

consume the clones for the validation and set another clone as a buffereInputstream property in the axis2messagecontext.

BufferedInputStream bufferedInputStreamOriginal = new BufferedInputStream(inputStreamOriginal);
axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, bufferedInputStreamOriginal);

I'm still getting the stream closed issue only for correct the messages which have been passed through multiple validators. If the validators throw an exception, the request is getting build and generate the custom response as expected. It seems like we implemented a way that gets the inputstream from the passthrough pipe for the content unaware flows. Unless it uses to get the inputstream from the messagecontext. It was proven once I attached an empty content aware mediator and test the same flow. I was able to observe the expected behaviour for the same implementation with the content aware mediator.

Do we have a way to define cloned input stream as an original inputstream in passthrough pipe?


Regards
Hasunie






On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[hidden email]> wrote:
Hi Dushan,

On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[hidden email]> wrote:
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago).
No. I think it was force.passthrough.builder property which you introduced for ELB requirement.  

To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Dimuthu Leelarathne 
Director, Solutions Architecture

WSO2, Inc. (http://wso2.com)
email: 
[hidden email]
Mobile: +94773661935 
Blog: http://muthulee.blogspot.com

Lean . Enterprise . Middleware

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Dushan Abeyruwan
In reply to this post by Hasunie Adikari
Hi 
 Pls provide the diff of the changes you have done.

@ESB Team / PPT experts, since there are PPT level changes you need keep watch on performance impact, memory blueprint impact, how the heap usage varies per message size since  (smallest to the largest) + per how the behavior for complex interaction which has clone/iterators etc etc.

Cheers,
Dushan

On Tue, Jan 9, 2018 at 9:38 AM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

As I discussed with Isuru, There are some possible approaches to overcome the issue.

1. Create a new pass through pipe.
   - The data will be written to the pipe by a spawned thread and current thread will be consuming the data and continuing the message flow. We went through the pipe creation logic and seemed it tightly coupled with encoding, decoding methodologies so that it can't be 
      implemented at APIM level. 

2. Change the synapse level logic to get the ByteArrayInputstream and write it into the response.
   - It can be but have to thoroughly go through and do it carefully unless the default message flow would be affected.

3. Build the message by invoking RelayUtils.buildMessage() at the validator mediator after successfully parsing all the validators.
    - It will be slightly affected the performance but this is the straightforward solution at this moment. 

I have improved the code by applying the 3rd option as we discussed. Setting the PassThroughConstants.BUFFERED_INPUT_STREAM has an effect now onwards since we changed a way that building the message to achieve the content aware behavior which seeks the
inputream from the axis2 message context instead of the original inputstream.


Regards
Hasunie

      

On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[hidden email]> wrote:
Hi Hasunie,

As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM has no effect on the flow in this case and Passthough Sender still seek content from the original input stream which got empty due to this cloning logic. That's the reason for this behaviour.

Thanks.



On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[hidden email]> wrote:
Hi Isuru,

As we discussed, I cloned the input stream by consuming the passthrough pipe as in below.


if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());

        }
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buffer)) > -1 ) {
    byteArrayOutputStream.write(buffer, 0, len);
}
byteArrayOutputStream.flush();


InputStream is1 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 
InputStream is2 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 

consume the clones for the validation and set another clone as a buffereInputstream property in the axis2messagecontext.

BufferedInputStream bufferedInputStreamOriginal = new BufferedInputStream(inputStreamOriginal);
axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, bufferedInputStreamOriginal);

I'm still getting the stream closed issue only for correct the messages which have been passed through multiple validators. If the validators throw an exception, the request is getting build and generate the custom response as expected. It seems like we implemented a way that gets the inputstream from the passthrough pipe for the content unaware flows. Unless it uses to get the inputstream from the messagecontext. It was proven once I attached an empty content aware mediator and test the same flow. I was able to observe the expected behaviour for the same implementation with the content aware mediator.

Do we have a way to define cloned input stream as an original inputstream in passthrough pipe?


Regards
Hasunie






On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[hidden email]> wrote:
Hi Dushan,

On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[hidden email]> wrote:
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago).
No. I think it was force.passthrough.builder property which you introduced for ELB requirement.  

To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:(001)408-791-9312


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Dimuthu Leelarathne-2
Hi Dushan,

It is an optional mediator. We haven't touched the core (not a single line)

We will provide the performance results on Monday.

thanks,
Dimuthu


On Mon, Jan 15, 2018 at 10:26 PM, Dushan Abeyruwan <[hidden email]> wrote:
Hi 
 Pls provide the diff of the changes you have done.

@ESB Team / PPT experts, since there are PPT level changes you need keep watch on performance impact, memory blueprint impact, how the heap usage varies per message size since  (smallest to the largest) + per how the behavior for complex interaction which has clone/iterators etc etc.

Cheers,
Dushan

On Tue, Jan 9, 2018 at 9:38 AM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

As I discussed with Isuru, There are some possible approaches to overcome the issue.

1. Create a new pass through pipe.
   - The data will be written to the pipe by a spawned thread and current thread will be consuming the data and continuing the message flow. We went through the pipe creation logic and seemed it tightly coupled with encoding, decoding methodologies so that it can't be 
      implemented at APIM level. 

2. Change the synapse level logic to get the ByteArrayInputstream and write it into the response.
   - It can be but have to thoroughly go through and do it carefully unless the default message flow would be affected.

3. Build the message by invoking RelayUtils.buildMessage() at the validator mediator after successfully parsing all the validators.
    - It will be slightly affected the performance but this is the straightforward solution at this moment. 

I have improved the code by applying the 3rd option as we discussed. Setting the PassThroughConstants.BUFFERED_INPUT_STREAM has an effect now onwards since we changed a way that building the message to achieve the content aware behavior which seeks the
inputream from the axis2 message context instead of the original inputstream.


Regards
Hasunie

      

On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[hidden email]> wrote:
Hi Hasunie,

As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM has no effect on the flow in this case and Passthough Sender still seek content from the original input stream which got empty due to this cloning logic. That's the reason for this behaviour.

Thanks.



On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[hidden email]> wrote:
Hi Isuru,

As we discussed, I cloned the input stream by consuming the passthrough pipe as in below.


if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());

        }
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buffer)) > -1 ) {
    byteArrayOutputStream.write(buffer, 0, len);
}
byteArrayOutputStream.flush();


InputStream is1 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 
InputStream is2 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 

consume the clones for the validation and set another clone as a buffereInputstream property in the axis2messagecontext.

BufferedInputStream bufferedInputStreamOriginal = new BufferedInputStream(inputStreamOriginal);
axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, bufferedInputStreamOriginal);

I'm still getting the stream closed issue only for correct the messages which have been passed through multiple validators. If the validators throw an exception, the request is getting build and generate the custom response as expected. It seems like we implemented a way that gets the inputstream from the passthrough pipe for the content unaware flows. Unless it uses to get the inputstream from the messagecontext. It was proven once I attached an empty content aware mediator and test the same flow. I was able to observe the expected behaviour for the same implementation with the content aware mediator.

Do we have a way to define cloned input stream as an original inputstream in passthrough pipe?


Regards
Hasunie






On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[hidden email]> wrote:
Hi Dushan,

On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[hidden email]> wrote:
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago).
No. I think it was force.passthrough.builder property which you introduced for ELB requirement.  

To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Dimuthu Leelarathne 
Director, Solutions Architecture

WSO2, Inc. (http://wso2.com)
email: 
[hidden email]
Mobile: +94773661935 
Blog: http://muthulee.blogspot.com

Lean . Enterprise . Middleware

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: [RRT] XML, JSON, Shema validation threat protectors in APIM 2.1.x

Dushan Abeyruwan
Hi Dimuthu,
 Understood, thanks for the prompted response

Cheers,
Dushan

On Tue, Jan 16, 2018 at 5:12 AM, Dimuthu Leelarathne <[hidden email]> wrote:
Hi Dushan,

It is an optional mediator. We haven't touched the core (not a single line)

We will provide the performance results on Monday.

thanks,
Dimuthu


On Mon, Jan 15, 2018 at 10:26 PM, Dushan Abeyruwan <[hidden email]> wrote:
Hi 
 Pls provide the diff of the changes you have done.

@ESB Team / PPT experts, since there are PPT level changes you need keep watch on performance impact, memory blueprint impact, how the heap usage varies per message size since  (smallest to the largest) + per how the behavior for complex interaction which has clone/iterators etc etc.

Cheers,
Dushan

On Tue, Jan 9, 2018 at 9:38 AM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

As I discussed with Isuru, There are some possible approaches to overcome the issue.

1. Create a new pass through pipe.
   - The data will be written to the pipe by a spawned thread and current thread will be consuming the data and continuing the message flow. We went through the pipe creation logic and seemed it tightly coupled with encoding, decoding methodologies so that it can't be 
      implemented at APIM level. 

2. Change the synapse level logic to get the ByteArrayInputstream and write it into the response.
   - It can be but have to thoroughly go through and do it carefully unless the default message flow would be affected.

3. Build the message by invoking RelayUtils.buildMessage() at the validator mediator after successfully parsing all the validators.
    - It will be slightly affected the performance but this is the straightforward solution at this moment. 

I have improved the code by applying the 3rd option as we discussed. Setting the PassThroughConstants.BUFFERED_INPUT_STREAM has an effect now onwards since we changed a way that building the message to achieve the content aware behavior which seeks the
inputream from the axis2 message context instead of the original inputstream.


Regards
Hasunie

      

On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[hidden email]> wrote:
Hi Hasunie,

As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM has no effect on the flow in this case and Passthough Sender still seek content from the original input stream which got empty due to this cloning logic. That's the reason for this behaviour.

Thanks.



On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[hidden email]> wrote:
Hi Isuru,

As we discussed, I cloned the input stream by consuming the passthrough pipe as in below.


if (pipe != null) {
            bufferedInputStream = new BufferedInputStream(pipe.getInputStream());

        }
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buffer)) > -1 ) {
    byteArrayOutputStream.write(buffer, 0, len);
}
byteArrayOutputStream.flush();


InputStream is1 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 
InputStream is2 = new ByteArrayInputStream(byteArrayOutputStream.toByteArray()); 

consume the clones for the validation and set another clone as a buffereInputstream property in the axis2messagecontext.

BufferedInputStream bufferedInputStreamOriginal = new BufferedInputStream(inputStreamOriginal);
axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, bufferedInputStreamOriginal);

I'm still getting the stream closed issue only for correct the messages which have been passed through multiple validators. If the validators throw an exception, the request is getting build and generate the custom response as expected. It seems like we implemented a way that gets the inputstream from the passthrough pipe for the content unaware flows. Unless it uses to get the inputstream from the messagecontext. It was proven once I attached an empty content aware mediator and test the same flow. I was able to observe the expected behaviour for the same implementation with the content aware mediator.

Do we have a way to define cloned input stream as an original inputstream in passthrough pipe?


Regards
Hasunie






On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[hidden email]> wrote:
Hi Dushan,

On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[hidden email]> wrote:
Hi Hasunie,
  Current PTT design would build the message whenever if there is content aware mediator available. However IIRC, I did this message.builder.the invoked thing to cope with the WSO2 ELB we had (a few years ago).
No. I think it was force.passthrough.builder property which you introduced for ELB requirement.  

To be honest, that looks ugly isn it (in terms of overall picture). Basically, what it does; even if there are content-aware mediators, the engine would forcefully ignore that (it was ELB requirement :) ) but for APIM I don't think that would be the same, cos we may have to deal with many use cases sometimes of cause with content aware flows with API compositions etc etc.

So, let's think what we can do here; regex and XML threat protectors equally important if security is priority thus,  we would no longer able to achieve the same core basic aspect (content none awareness) because, such protections required you to walk through the nodes and verify some aspects (basically, you need to expand the xml node tree to get result set) in that way, it is required the message to be build. Anyway, what I would think the best approach here is not to change complete synapse content awareness logic rather I would think you may have mediator in place but only if such protection engaged that may build the message to get XML inforset (rather build through root, may be you can mark this meditor as content-aware false, then build if message not already build prior to process)

IMO, lets just not complicate the what we try to build around message validation. I mean if we need such protection we may need to sacrify some aspects am I?

Cheers,
Dushan

On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[hidden email]> wrote:
Hi Hasunie,

This is expected since the synapse engine now expecting an already built message. If I understood your requirement correctly, one option is to use a Builder Mediator before using any content aware mediator. Even though we do not recommend the Builder mediators now, still we can use it for your specific use case. Or you have to revert the message.builder.invoked property to false again.

Regards,
Vinod

On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[hidden email]> wrote:
I'm trying to combine SQL injection(Regex) threat protector with the XML threat protector. So I created a sequence[1] with XMLthreatprotector mediator and regex mediator consecutively and uploaded it to be able to validate the request message through both the xml validator and regex validators. If I set the message.builder.invoked property to TRUE in xml validator mediator to avoid sending the content in pass-through pipe(request message) as the response, Regex mediators is getting failed. The regex mediator was designed a way that the incoming messages are built in synapse level and eveluate the message content at the mediator level. It seems like we can't continue any mediators which are required to get the message content, after we manually set the aforementioned property to true in the previous mediator. If I set it to true, RelayUtill will skip building the message as in here [2]. Any thoughts regarding the issue. I'm currently working on the issue to be able to combine regex and XML threat protectors.
[2]
if (pipe != null
&& !Boolean.TRUE.equals(messageContext
.getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && forcePTBuild) {
InputStream in = pipe.getInputStream();

Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
if (http_sc != null && http_sc instanceof Integer && http_sc.equals(202)) {
messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
Boolean.TRUE);
return;
}

builldMessage(messageContext, earlyBuild, in);
return;
}


[1] 
<sequence xmlns="http://ws.apache.org/ns/synapse" name="validator">
    <property name="threatType" expression="get-property('threatType')" value="SQL-Injection and XML validator"/>
    <property>
    <property>
     --------
    <switch source="get-property('To')">
       --------------
    </switch>
    <class name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>
    <property name="enabledCheckPathParams" expression="get-property('enabledCheckPathParams')" value="true"/>
    <property>
    <property>
     --
    <class name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>
    <filter source="get-property('threat')" regex=".*error.*">
        <then>
            <sequence key="_threat_fault_"></sequence>
        </then>
    </filter>
</sequence>

Regards
Hasunie

On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[hidden email]> wrote:
Hi all,

I'm working on threat protector feature in APIM. We're actually trying to achieve here is to protect both backend resources and gateway from the XML and JSON based attacks. The Balerina based APIM 3 gateway will be protected by threat handlers. But In here
APIM 2.1.x we have implemented mediators to achieve it.

If we allow building the request message at the synapse level, It will definitely affect the gateway, All the request messages which go through the mediators are built since the Abstarctemediator is designed a way that the isContentAware method always returns true. So we set it to false in both XML and JSON validator mediators and allow to parse the XML request via a third party StAX parser called woodstox hence it was the best option among other StAX parsers for threat protection features. It will keep counting the given limits and when the limit is exceeded, It will terminate the process and throw a meaningful exception. I have created a custom threat sequence(thrat_fault) and If a threat is detected by getting an exception I configured to direct the response through the custom error sequence.
I reuse the same custom sequence which was implemented for the regex threat protector [1]

Woodstox parser covers most of the vulnerabilities as in here

Vulnerablity:

xml bomb - DTD disabling 

external entity attack - disabling external entities. 

Note :
Apart from the mediator level, The external entity reference property was disabled from the DOM parsers at the synapse level as well.

import org.apache.xerces.impl.Constants;

private static final int ENTITY_EXPANSION_LIMIT = 0;
private static final DocumentBuilderFactory documentBuilderFactory =     
   DocumentBuilderFactory.newInstance();

static {
   documentBuilderFactory.setNamespaceAware(true);
   documentBuilderFactory.setXIncludeAware(false);
   documentBuilderFactory.setExpandEntityReferences(false);

   try {
       documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX +
           Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
   } catch (ParserConfigurationException e) {

Buffer overflow attack - by limiting the count of elements, children and length of attributes/keys/values.

woodstox properties:
dtdEnabled
externalEntitiesEnabled
maxDepth
maxElementCount
maxAttributeCount
maxAttributeLength
entityExpansionLimit
maxChildrenPerElement


For thwart cohesive attacks, we use both schema validator and depth limits. Ideally, only the woodstox validator should detect the cohesive attacks by exceeding the defined depth limit. But the schema validator will protect the schema poising attacks in the second step as well.

I observed an issue when It comes to combining each other(woodstox+ schema validator). We have designed the feature in such a way that gets the inputstream from the message context and consumes it in the woodstox validator. but in here we have to consume the input stream again for the schema validation just after passing through the woodstox.That was the issue and I tried the following methodologies to resolved the issue

1. try to get the XML object from the woodstox parser to be able to avoid using the input stream again.
2. deep clone the inputstream and use cloned input stream for the schema validation.
3. reset, mark the buffered input stream(synapse engine also has done rest, mark)

1st one was taken time and much complex to get the XML object since Woodstock is based on the StAX parsers and also deep cloning was not working properly and experienced the same issue after cloning the inputstream. But the 3rd option makes life easy so I implemented a way that returning the buffered input stream, after doing the rest, mark,  then It works properly. I went through the RelayUtil message builders [2] and It also uses the mark and reset methodology and return InputStream.


I observed another issue once the validator throws an exception, the server hanged and didn't get any response and getting timeout issue. I was able to figure it out and Issue occurred while trying to build the request message in Relayutil.buildmessage().But Ideally, If we get an error we don't need the request message anymore. As I discussed offline with the APIM team, I used the consumeAndDiscardMessage method to discard the request message from the message context and set message.builder.invoked property to TRUE. It needs to be set to avoid sending the content in pass-through pipe (request message) as the response.




Regards
Hasunie


















--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Vinod Kavinda
Senior Software Engineer 
Mobile : <a href="tel:%2B94%20%280%29%20712%20415544" value="+94773451194" style="color:rgb(17,85,204)" target="_blank">+94 (0) 712 415544
http://wso2.com/signature




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Isuru Udana
Senior Technical Lead
WSO2 Inc.; http://wso2.com
email: [hidden email] cell: <a href="tel:+94%2077%20379%201887" value="+94773791887" target="_blank">+94 77 3791887
blog: http://mytecheye.blogspot.com/



--
Hasunie Adikari
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
Mobile:<a href="tel:+94%2071%20309%205876" value="+94713095876" target="_blank">+94713095876




--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:<a href="tel:+1%20408-791-9312" value="+14087919312" target="_blank">(001)408-791-9312




--
Dimuthu Leelarathne 
Director, Solutions Architecture

WSO2, Inc. (http://wso2.com)
email: 
[hidden email]
Mobile: <a href="tel:+94%2077%20366%201935" value="+94773661935" target="_blank">+94773661935 
Blog: http://muthulee.blogspot.com

Lean . Enterprise . Middleware



--
Dushan Abeyruwan | Architect
Technical Support,MV
PMC Member Apache Synpase
WSO2 Inc. http://wso2.com/
Mobile:(001)408-791-9312


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture