Micro Gateway CLI - Hashing Resources (APIs/Policies) for change detection

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Micro Gateway CLI - Hashing Resources (APIs/Policies) for change detection

Malintha Amarasinghe
Hi,

Micro gateway CLI works completely separately to API manager; whenever a new API is added for a label, whenever there is a change happens to an existing label there won't be any events published etc like previously. The CLI needs to regenerate the source build it and push the artifacts to the deployment and the full process needs to complete. In most occasions, the CLI can be configured to run periodically to generate sources and do above job. 

But in this case, most of the time, the CLI will be uselessly generating sources building it and pushing the artifacts to deployment. Comparatively, building and pushing artifacts to deployment have a huge overhead compared to generating sources.

This effort is to avoid that as much as possible by change-detection; i.e. 

1. The CLI will check if any of the required resources has changed vs the previous build and notify the user after a successful "setup" (source generate) command using the command line output and the exit code of the command. 
2. Using the exit code, a user can write a shell script etc to decide whether he should proceed with "build" or not. 


Proposed implementation:

API Publisher APIs does not have ETag feature. Even if it is there, the ETag will be generated for the whole resource. For code generation, we will be only using few attributes of the resource, hence using a global ETag for a resource may lead to unnecessary changes for the ETag. Hence the proposed implementation will be using a CLI-side hash generation for used attributes of the resource (API/Policies) only.

To mark the attributes which are used for generating the code, a newly introduced annotation "@Hash" can be used. 

Ex:

public class APIDetailedDTO extends APIInfoDTO {

/**
* Swagger definition of the APIDetailedDTO which contains details about URI templates and scopes\n
**/
@Hash
@JsonProperty("apiDefinition")
public String getApiDefinition() {
return apiDefinition;
}

public void setApiDefinition(String apiDefinition) {
this.apiDefinition = apiDefinition;
}


/**
* WSDL URL if the APIDetailedDTO is based on a WSDL endpoint\n
**/
@JsonProperty("wsdlUri")
public String getWsdlUri() {
return wsdlUri;
}

public void setWsdlUri(String wsdlUri) {
this.wsdlUri = wsdlUri;
}

@Hash
@JsonProperty("responseCaching")
public String getResponseCaching() {
return responseCaching;
}


The methods marked with @Hash will be automatically extracted from the code and will be used to generate the hashes for each resource.

The generated hashes will be stored inside the CLI's temp folder against each resources' UUID, which will be used to compare the hash changes between next runs.


Highly appreciate your ideas on this.

Thanks!
Malintha


--
Malintha Amarasinghe
WSO2, Inc. - lean | enterprise | middleware

Mobile : +94 712383306

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Micro Gateway CLI - Hashing Resources (APIs/Policies) for change detection

Malintha Amarasinghe
+ IsuruH

On Tue, Jun 19, 2018 at 12:41 PM, Malintha Amarasinghe <[hidden email]> wrote:
List of fields planned to be added as of now; kindly let me know if any field is missing.

API
name
context
version
apiDefinition
responseCaching
isDefaultVersion
type - (http vs ws) 
transport - (http/https)
endpointConfig
endpointSecurity
corsConfiguration
authorizationHeader


SubscriptionThrottlePolicy
policyName
defaultLimit (throttling limits)
stopOnQuotaReach

ApplicationThrottlePolicy
policyName
defaultLimit (throttling limits)


Thanks!
Malintha

On Tue, Jun 19, 2018 at 12:00 PM, Harsha Kumara <[hidden email]> wrote:


On Tue, Jun 19, 2018 at 11:45 AM Malintha Amarasinghe <[hidden email]> wrote:
Hi,

Micro gateway CLI works completely separately to API manager; whenever a new API is added for a label, whenever there is a change happens to an existing label there won't be any events published etc like previously. The CLI needs to regenerate the source build it and push the artifacts to the deployment and the full process needs to complete. In most occasions, the CLI can be configured to run periodically to generate sources and do above job. 

But in this case, most of the time, the CLI will be uselessly generating sources building it and pushing the artifacts to deployment. Comparatively, building and pushing artifacts to deployment have a huge overhead compared to generating sources.

This effort is to avoid that as much as possible by change-detection; i.e. 

1. The CLI will check if any of the required resources has changed vs the previous build and notify the user after a successful "setup" (source generate) command using the command line output and the exit code of the command. 
2. Using the exit code, a user can write a shell script etc to decide whether he should proceed with "build" or not. 


Proposed implementation:

API Publisher APIs does not have ETag feature. Even if it is there, the ETag will be generated for the whole resource. For code generation, we will be only using few attributes of the resource, hence using a global ETag for a resource may lead to unnecessary changes for the ETag. Hence the proposed implementation will be using a CLI-side hash generation for used attributes of the resource (API/Policies) only.

To mark the attributes which are used for generating the code, a newly introduced annotation "@Hash" can be used. 

Ex:

public class APIDetailedDTO extends APIInfoDTO {

/**
* Swagger definition of the APIDetailedDTO which contains details about URI templates and scopes\n
**/
@Hash
@JsonProperty("apiDefinition")
public String getApiDefinition() {
return apiDefinition;
}

public void setApiDefinition(String apiDefinition) {
this.apiDefinition = apiDefinition;
}


/**
* WSDL URL if the APIDetailedDTO is based on a WSDL endpoint\n
**/
@JsonProperty("wsdlUri")
public String getWsdlUri() {
return wsdlUri;
}

public void setWsdlUri(String wsdlUri) {
this.wsdlUri = wsdlUri;
}

@Hash
@JsonProperty("responseCaching")
public String getResponseCaching() {
return responseCaching;
}


The methods marked with @Hash will be automatically extracted from the code and will be used to generate the hashes for each resource.

The generated hashes will be stored inside the CLI's temp folder against each resources' UUID, which will be used to compare the hash changes between next runs.
What are the fields which we have added to the hash? 


Highly appreciate your ideas on this.

Thanks!
Malintha


--
Malintha Amarasinghe
WSO2, Inc. - lean | enterprise | middleware

Mobile : +94 712383306


--
Harsha Kumara
Associate Technical Lead, WSO2 Inc.
Mobile: +94775505618
Blog:harshcreationz.blogspot.com



--
Malintha Amarasinghe
WSO2, Inc. - lean | enterprise | middleware

Mobile : +94 712383306



--
Malintha Amarasinghe
WSO2, Inc. - lean | enterprise | middleware

Mobile : +94 712383306

_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Micro Gateway CLI - Hashing Resources (APIs/Policies) for change detection

Isuru Haththotuwa
+1 for this approach.

On Tue, Jun 19, 2018 at 1:47 PM, Malintha Amarasinghe <[hidden email]> wrote:
+ IsuruH

On Tue, Jun 19, 2018 at 12:41 PM, Malintha Amarasinghe <[hidden email]> wrote:
List of fields planned to be added as of now; kindly let me know if any field is missing.

API
name
context
version
apiDefinition
responseCaching
isDefaultVersion
type - (http vs ws) 
transport - (http/https)
endpointConfig
endpointSecurity
corsConfiguration
authorizationHeader


SubscriptionThrottlePolicy
policyName
defaultLimit (throttling limits)
stopOnQuotaReach

ApplicationThrottlePolicy
policyName
defaultLimit (throttling limits)


Thanks!
Malintha

On Tue, Jun 19, 2018 at 12:00 PM, Harsha Kumara <[hidden email]> wrote:


On Tue, Jun 19, 2018 at 11:45 AM Malintha Amarasinghe <[hidden email]> wrote:
Hi,

Micro gateway CLI works completely separately to API manager; whenever a new API is added for a label, whenever there is a change happens to an existing label there won't be any events published etc like previously. The CLI needs to regenerate the source build it and push the artifacts to the deployment and the full process needs to complete. In most occasions, the CLI can be configured to run periodically to generate sources and do above job. 

But in this case, most of the time, the CLI will be uselessly generating sources building it and pushing the artifacts to deployment. Comparatively, building and pushing artifacts to deployment have a huge overhead compared to generating sources.

This effort is to avoid that as much as possible by change-detection; i.e. 

1. The CLI will check if any of the required resources has changed vs the previous build and notify the user after a successful "setup" (source generate) command using the command line output and the exit code of the command. 
2. Using the exit code, a user can write a shell script etc to decide whether he should proceed with "build" or not. 


Proposed implementation:

API Publisher APIs does not have ETag feature. Even if it is there, the ETag will be generated for the whole resource. For code generation, we will be only using few attributes of the resource, hence using a global ETag for a resource may lead to unnecessary changes for the ETag. Hence the proposed implementation will be using a CLI-side hash generation for used attributes of the resource (API/Policies) only.

To mark the attributes which are used for generating the code, a newly introduced annotation "@Hash" can be used. 

Ex:

public class APIDetailedDTO extends APIInfoDTO {

/**
* Swagger definition of the APIDetailedDTO which contains details about URI templates and scopes\n
**/
@Hash
@JsonProperty("apiDefinition")
public String getApiDefinition() {
return apiDefinition;
}

public void setApiDefinition(String apiDefinition) {
this.apiDefinition = apiDefinition;
}


/**
* WSDL URL if the APIDetailedDTO is based on a WSDL endpoint\n
**/
@JsonProperty("wsdlUri")
public String getWsdlUri() {
return wsdlUri;
}

public void setWsdlUri(String wsdlUri) {
this.wsdlUri = wsdlUri;
}

@Hash
@JsonProperty("responseCaching")
public String getResponseCaching() {
return responseCaching;
}


The methods marked with @Hash will be automatically extracted from the code and will be used to generate the hashes for each resource.

The generated hashes will be stored inside the CLI's temp folder against each resources' UUID, which will be used to compare the hash changes between next runs.
What are the fields which we have added to the hash? 


Highly appreciate your ideas on this.

Thanks!
Malintha


--
Malintha Amarasinghe
WSO2, Inc. - lean | enterprise | middleware

Mobile : +94 712383306


--
Harsha Kumara
Associate Technical Lead, WSO2 Inc.
Mobile: +94775505618
Blog:harshcreationz.blogspot.com



--
Malintha Amarasinghe
WSO2, Inc. - lean | enterprise | middleware

Mobile : +94 712383306



--
Malintha Amarasinghe
WSO2, Inc. - lean | enterprise | middleware

Mobile : +94 712383306



--
Thanks and Regards,

Isuru H.
+94 716 358 048



_______________________________________________
Dev mailing list
[hidden email]
http://wso2.org/cgi-bin/mailman/listinfo/dev