Old access token cleanup when token generating, renewing and revoking.

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Old access token cleanup when token generating, renewing and revoking.

Nalaka Senarathna


A solution for the access token table filled up with EXPIRED, INACTIVE and REVOKED tokens in the Access token table, old access token can move to the Audit table when the new token is generating, renewing or token revoking.


Old Access tokens will be stored in newly created Audit table for audit purposes.


*Token cleanup feature can be turn enable / disable.

*When the token cleanup feature is enabled old access token can be stored in the audit table 

  For audit purposes or can disable that feature too, not to store old access tokens(this also can enable/disable).


This project avoids the access token table growing and improve the token lookup. For existing users also can get advantage from this.


Growing Audit table is no problem if the token cleanup feature and retain old access token feature is enabled .because writing operation to Audit table is not increased when the token table is growing(Audit table has no constraint).This property was tested until 14 million token generation.








regards.

--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Old access token cleanup when token generating, renewing and revoking.

Harshan Liyanage
Hi Nalaka,

You could do the same with a configurable periodic task instead of modifying existing token request flows. What you have to do is to implement the token cleanup feature as a periodic task which scans the token related tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit table. There will be a configuration to configure the CRON expression for that task.

WDYT?

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <[hidden email]> wrote:


A solution for the access token table filled up with EXPIRED, INACTIVE and REVOKED tokens in the Access token table, old access token can move to the Audit table when the new token is generating, renewing or token revoking.


Old Access tokens will be stored in newly created Audit table for audit purposes.


*Token cleanup feature can be turn enable / disable.

*When the token cleanup feature is enabled old access token can be stored in the audit table 

  For audit purposes or can disable that feature too, not to store old access tokens(this also can enable/disable).


This project avoids the access token table growing and improve the token lookup. For existing users also can get advantage from this.


Growing Audit table is no problem if the token cleanup feature and retain old access token feature is enabled .because writing operation to Audit table is not increased when the token table is growing(Audit table has no constraint).This property was tested until 14 million token generation.








regards.

--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Old access token cleanup when token generating, renewing and revoking.

Nuwan Dias
A periodic task won't work for this because when the system runs on tables with millions of records, the task will lock the table for the clean up process. This will impact other queries being executed on the table and hence block user flows.

On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage <[hidden email]> wrote:
Hi Nalaka,

You could do the same with a configurable periodic task instead of modifying existing token request flows. What you have to do is to implement the token cleanup feature as a periodic task which scans the token related tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit table. There will be a configuration to configure the CRON expression for that task.

WDYT?

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <[hidden email]> wrote:


A solution for the access token table filled up with EXPIRED, INACTIVE and REVOKED tokens in the Access token table, old access token can move to the Audit table when the new token is generating, renewing or token revoking.


Old Access tokens will be stored in newly created Audit table for audit purposes.


*Token cleanup feature can be turn enable / disable.

*When the token cleanup feature is enabled old access token can be stored in the audit table 

  For audit purposes or can disable that feature too, not to store old access tokens(this also can enable/disable).


This project avoids the access token table growing and improve the token lookup. For existing users also can get advantage from this.


Growing Audit table is no problem if the token cleanup feature and retain old access token feature is enabled .because writing operation to Audit table is not increased when the token table is growing(Audit table has no constraint).This property was tested until 14 million token generation.








regards.

--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Old access token cleanup when token generating, renewing and revoking.

Nalaka Senarathna
hi harshan,

Also if there are multiple nodes then those nodes also may attempt to clean up the same access token at the same time.

related mail thread:[1]
[1]Access token Table cleaning and keeping the access token data for future purposes

regards.


On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias <[hidden email]> wrote:
A periodic task won't work for this because when the system runs on tables with millions of records, the task will lock the table for the clean up process. This will impact other queries being executed on the table and hence block user flows.

On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage <[hidden email]> wrote:
Hi Nalaka,

You could do the same with a configurable periodic task instead of modifying existing token request flows. What you have to do is to implement the token cleanup feature as a periodic task which scans the token related tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit table. There will be a configuration to configure the CRON expression for that task.

WDYT?

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <[hidden email]> wrote:


A solution for the access token table filled up with EXPIRED, INACTIVE and REVOKED tokens in the Access token table, old access token can move to the Audit table when the new token is generating, renewing or token revoking.


Old Access tokens will be stored in newly created Audit table for audit purposes.


*Token cleanup feature can be turn enable / disable.

*When the token cleanup feature is enabled old access token can be stored in the audit table 

  For audit purposes or can disable that feature too, not to store old access tokens(this also can enable/disable).


This project avoids the access token table growing and improve the token lookup. For existing users also can get advantage from this.


Growing Audit table is no problem if the token cleanup feature and retain old access token feature is enabled .because writing operation to Audit table is not increased when the token table is growing(Audit table has no constraint).This property was tested until 14 million token generation.








regards.

--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Old access token cleanup when token generating, renewing and revoking.

Harshan Liyanage
Hi all,

@Nuwan: That's why I suggested having a configurable cron expression so that users can configure the task to run on an optimal schedule instead of blocking vital functionalities. Also in that way, we could do a batch deletes and insertions instead of single rows. For example, they could let it run every mid-night so then the access token tables won't grow for millions of records and not affecting the user traffic. 

@Nalaka: You could let the task to run only on a manager node in a distributed setup using a configuration. 

BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed design. :)

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna <[hidden email]> wrote:
hi harshan,

Also if there are multiple nodes then those nodes also may attempt to clean up the same access token at the same time.

related mail thread:[1]
[1]Access token Table cleaning and keeping the access token data for future purposes

regards.


On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias <[hidden email]> wrote:
A periodic task won't work for this because when the system runs on tables with millions of records, the task will lock the table for the clean up process. This will impact other queries being executed on the table and hence block user flows.

On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage <[hidden email]> wrote:
Hi Nalaka,

You could do the same with a configurable periodic task instead of modifying existing token request flows. What you have to do is to implement the token cleanup feature as a periodic task which scans the token related tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit table. There will be a configuration to configure the CRON expression for that task.

WDYT?

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <[hidden email]> wrote:


A solution for the access token table filled up with EXPIRED, INACTIVE and REVOKED tokens in the Access token table, old access token can move to the Audit table when the new token is generating, renewing or token revoking.


Old Access tokens will be stored in newly created Audit table for audit purposes.


*Token cleanup feature can be turn enable / disable.

*When the token cleanup feature is enabled old access token can be stored in the audit table 

  For audit purposes or can disable that feature too, not to store old access tokens(this also can enable/disable).


This project avoids the access token table growing and improve the token lookup. For existing users also can get advantage from this.


Growing Audit table is no problem if the token cleanup feature and retain old access token feature is enabled .because writing operation to Audit table is not increased when the token table is growing(Audit table has no constraint).This property was tested until 14 million token generation.








regards.

--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

updated design.png (371K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Old access token cleanup when token generating, renewing and revoking.

Nuwan Dias


On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage <[hidden email]> wrote:
Hi all,

@Nuwan: That's why I suggested having a configurable cron expression so that users can configure the task to run on an optimal schedule instead of blocking vital functionalities. Also in that way, we could do a batch deletes and insertions instead of single rows. For example, they could let it run every mid-night so then the access token tables won't grow for millions of records and not affecting the user traffic. 

Our objective is to run this clean up process without anyone having to configure anything. Even today we have instructions and scripts to clean up the tables. But in practice, nobody even notices this and never does these kind of stuff until they hit an issue and consult us through support. So if we design this in such a way that someone has to turn on something or configure something, I am positive our efforts will be in waste. 

@Nalaka: You could let the task to run only on a manager node in a distributed setup using a configuration. 

There is no such thing as a "manager" node in our architecture. All nodes are equal. Plus, we cannot introduce such changes as well due to numerous complexities such as how to HA the manager node, too many varying configurations, etc.

BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed design. :)

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna <[hidden email]> wrote:
hi harshan,

Also if there are multiple nodes then those nodes also may attempt to clean up the same access token at the same time.

related mail thread:[1]
[1]Access token Table cleaning and keeping the access token data for future purposes

regards.


On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias <[hidden email]> wrote:
A periodic task won't work for this because when the system runs on tables with millions of records, the task will lock the table for the clean up process. This will impact other queries being executed on the table and hence block user flows.

On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage <[hidden email]> wrote:
Hi Nalaka,

You could do the same with a configurable periodic task instead of modifying existing token request flows. What you have to do is to implement the token cleanup feature as a periodic task which scans the token related tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit table. There will be a configuration to configure the CRON expression for that task.

WDYT?

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <[hidden email]> wrote:


A solution for the access token table filled up with EXPIRED, INACTIVE and REVOKED tokens in the Access token table, old access token can move to the Audit table when the new token is generating, renewing or token revoking.


Old Access tokens will be stored in newly created Audit table for audit purposes.


*Token cleanup feature can be turn enable / disable.

*When the token cleanup feature is enabled old access token can be stored in the audit table 

  For audit purposes or can disable that feature too, not to store old access tokens(this also can enable/disable).


This project avoids the access token table growing and improve the token lookup. For existing users also can get advantage from this.


Growing Audit table is no problem if the token cleanup feature and retain old access token feature is enabled .because writing operation to Audit table is not increased when the token table is growing(Audit table has no constraint).This property was tested until 14 million token generation.








regards.

--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Old access token cleanup when token generating, renewing and revoking.

Harshan Liyanage
Hi Nuwan,

Thanks for your detailed clarifications. Both explanations are perfectly valid. 

Regards,

Harshan Liyanage
lean.enterprise.middleware.


On Thu, Jul 19, 2018 at 8:35 PM Nuwan Dias <[hidden email]> wrote:


On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage <[hidden email]> wrote:
Hi all,

@Nuwan: That's why I suggested having a configurable cron expression so that users can configure the task to run on an optimal schedule instead of blocking vital functionalities. Also in that way, we could do a batch deletes and insertions instead of single rows. For example, they could let it run every mid-night so then the access token tables won't grow for millions of records and not affecting the user traffic. 

Our objective is to run this clean up process without anyone having to configure anything. Even today we have instructions and scripts to clean up the tables. But in practice, nobody even notices this and never does these kind of stuff until they hit an issue and consult us through support. So if we design this in such a way that someone has to turn on something or configure something, I am positive our efforts will be in waste. 

@Nalaka: You could let the task to run only on a manager node in a distributed setup using a configuration. 

There is no such thing as a "manager" node in our architecture. All nodes are equal. Plus, we cannot introduce such changes as well due to numerous complexities such as how to HA the manager node, too many varying configurations, etc.

BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed design. :)

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna <[hidden email]> wrote:
hi harshan,

Also if there are multiple nodes then those nodes also may attempt to clean up the same access token at the same time.

related mail thread:[1]
[1]Access token Table cleaning and keeping the access token data for future purposes

regards.


On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias <[hidden email]> wrote:
A periodic task won't work for this because when the system runs on tables with millions of records, the task will lock the table for the clean up process. This will impact other queries being executed on the table and hence block user flows.

On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage <[hidden email]> wrote:
Hi Nalaka,

You could do the same with a configurable periodic task instead of modifying existing token request flows. What you have to do is to implement the token cleanup feature as a periodic task which scans the token related tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit table. There will be a configuration to configure the CRON expression for that task.

WDYT?

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <[hidden email]> wrote:


A solution for the access token table filled up with EXPIRED, INACTIVE and REVOKED tokens in the Access token table, old access token can move to the Audit table when the new token is generating, renewing or token revoking.


Old Access tokens will be stored in newly created Audit table for audit purposes.


*Token cleanup feature can be turn enable / disable.

*When the token cleanup feature is enabled old access token can be stored in the audit table 

  For audit purposes or can disable that feature too, not to store old access tokens(this also can enable/disable).


This project avoids the access token table growing and improve the token lookup. For existing users also can get advantage from this.


Growing Audit table is no problem if the token cleanup feature and retain old access token feature is enabled .because writing operation to Audit table is not increased when the token table is growing(Audit table has no constraint).This property was tested until 14 million token generation.








regards.

--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729
_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Old access token cleanup when token generating, renewing and revoking.

Ishara Cooray
Our objective is to run this clean up process without anyone having to configure anything. Even today we have instructions and scripts to clean up the tables. But in practice, nobody even notices this and never does these kind of stuff until they hit an issue and consult us through support. So if we design this in such a way that someone has to turn on something or configure something, I am positive our efforts will be in waste.

So by default are we going to enable the token cleanup feature and storing old access tokens in audit table?

@Nalaka,
Have we done a load test to verify whether this feature does not affect to the functionalities when there is a high load in a high concurrency situation?
If so appreciate if you can share that details as well.



Thanks & Regards,
Ishara Cooray
Senior Software Engineer
Mobile : +9477 262 9512
WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

On Fri, Jul 20, 2018 at 10:04 AM, Harshan Liyanage <[hidden email]> wrote:
Hi Nuwan,

Thanks for your detailed clarifications. Both explanations are perfectly valid. 

Regards,

Harshan Liyanage
lean.enterprise.middleware.


On Thu, Jul 19, 2018 at 8:35 PM Nuwan Dias <[hidden email]> wrote:


On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage <[hidden email]> wrote:
Hi all,

@Nuwan: That's why I suggested having a configurable cron expression so that users can configure the task to run on an optimal schedule instead of blocking vital functionalities. Also in that way, we could do a batch deletes and insertions instead of single rows. For example, they could let it run every mid-night so then the access token tables won't grow for millions of records and not affecting the user traffic. 

Our objective is to run this clean up process without anyone having to configure anything. Even today we have instructions and scripts to clean up the tables. But in practice, nobody even notices this and never does these kind of stuff until they hit an issue and consult us through support. So if we design this in such a way that someone has to turn on something or configure something, I am positive our efforts will be in waste. 

@Nalaka: You could let the task to run only on a manager node in a distributed setup using a configuration. 

There is no such thing as a "manager" node in our architecture. All nodes are equal. Plus, we cannot introduce such changes as well due to numerous complexities such as how to HA the manager node, too many varying configurations, etc.

BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed design. :)

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna <[hidden email]> wrote:
hi harshan,

Also if there are multiple nodes then those nodes also may attempt to clean up the same access token at the same time.

related mail thread:[1]
[1]Access token Table cleaning and keeping the access token data for future purposes

regards.


On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias <[hidden email]> wrote:
A periodic task won't work for this because when the system runs on tables with millions of records, the task will lock the table for the clean up process. This will impact other queries being executed on the table and hence block user flows.

On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage <[hidden email]> wrote:
Hi Nalaka,

You could do the same with a configurable periodic task instead of modifying existing token request flows. What you have to do is to implement the token cleanup feature as a periodic task which scans the token related tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit table. There will be a configuration to configure the CRON expression for that task.

WDYT?

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <[hidden email]> wrote:


A solution for the access token table filled up with EXPIRED, INACTIVE and REVOKED tokens in the Access token table, old access token can move to the Audit table when the new token is generating, renewing or token revoking.


Old Access tokens will be stored in newly created Audit table for audit purposes.


*Token cleanup feature can be turn enable / disable.

*When the token cleanup feature is enabled old access token can be stored in the audit table 

  For audit purposes or can disable that feature too, not to store old access tokens(this also can enable/disable).


This project avoids the access token table growing and improve the token lookup. For existing users also can get advantage from this.


Growing Audit table is no problem if the token cleanup feature and retain old access token feature is enabled .because writing operation to Audit table is not increased when the token table is growing(Audit table has no constraint).This property was tested until 14 million token generation.








regards.

--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729
_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Old access token cleanup when token generating, renewing and revoking.

Nuwan Dias


On Tue, Jul 24, 2018 at 3:17 PM Ishara Cooray <[hidden email]> wrote:
Our objective is to run this clean up process without anyone having to configure anything. Even today we have instructions and scripts to clean up the tables. But in practice, nobody even notices this and never does these kind of stuff until they hit an issue and consult us through support. So if we design this in such a way that someone has to turn on something or configure something, I am positive our efforts will be in waste.

So by default are we going to enable the token cleanup feature and storing old access tokens in audit table?

Yes, otherwise there's no point of this feature IMO because we currently have a solution that cleans data but in reality nobody uses it unless they hit a issue and request our assistance to find and run this. 

@Nalaka,
Have we done a load test to verify whether this feature does not affect to the functionalities when there is a high load in a high concurrency situation?
If so appreciate if you can share that details as well.

Yes, we did. [hidden email] please point to the relevant thread. 



Thanks & Regards,
Ishara Cooray
Senior Software Engineer
Mobile : +9477 262 9512
WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

On Fri, Jul 20, 2018 at 10:04 AM, Harshan Liyanage <[hidden email]> wrote:
Hi Nuwan,

Thanks for your detailed clarifications. Both explanations are perfectly valid. 

Regards,

Harshan Liyanage
lean.enterprise.middleware.


On Thu, Jul 19, 2018 at 8:35 PM Nuwan Dias <[hidden email]> wrote:


On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage <[hidden email]> wrote:
Hi all,

@Nuwan: That's why I suggested having a configurable cron expression so that users can configure the task to run on an optimal schedule instead of blocking vital functionalities. Also in that way, we could do a batch deletes and insertions instead of single rows. For example, they could let it run every mid-night so then the access token tables won't grow for millions of records and not affecting the user traffic. 

Our objective is to run this clean up process without anyone having to configure anything. Even today we have instructions and scripts to clean up the tables. But in practice, nobody even notices this and never does these kind of stuff until they hit an issue and consult us through support. So if we design this in such a way that someone has to turn on something or configure something, I am positive our efforts will be in waste. 

@Nalaka: You could let the task to run only on a manager node in a distributed setup using a configuration. 

There is no such thing as a "manager" node in our architecture. All nodes are equal. Plus, we cannot introduce such changes as well due to numerous complexities such as how to HA the manager node, too many varying configurations, etc.

BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed design. :)

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna <[hidden email]> wrote:
hi harshan,

Also if there are multiple nodes then those nodes also may attempt to clean up the same access token at the same time.

related mail thread:[1]
[1]Access token Table cleaning and keeping the access token data for future purposes

regards.


On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias <[hidden email]> wrote:
A periodic task won't work for this because when the system runs on tables with millions of records, the task will lock the table for the clean up process. This will impact other queries being executed on the table and hence block user flows.

On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage <[hidden email]> wrote:
Hi Nalaka,

You could do the same with a configurable periodic task instead of modifying existing token request flows. What you have to do is to implement the token cleanup feature as a periodic task which scans the token related tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit table. There will be a configuration to configure the CRON expression for that task.

WDYT?

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <[hidden email]> wrote:


A solution for the access token table filled up with EXPIRED, INACTIVE and REVOKED tokens in the Access token table, old access token can move to the Audit table when the new token is generating, renewing or token revoking.


Old Access tokens will be stored in newly created Audit table for audit purposes.


*Token cleanup feature can be turn enable / disable.

*When the token cleanup feature is enabled old access token can be stored in the audit table 

  For audit purposes or can disable that feature too, not to store old access tokens(this also can enable/disable).


This project avoids the access token table growing and improve the token lookup. For existing users also can get advantage from this.


Growing Audit table is no problem if the token cleanup feature and retain old access token feature is enabled .because writing operation to Audit table is not increased when the token table is growing(Audit table has no constraint).This property was tested until 14 million token generation.








regards.

--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729
_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Reply | Threaded
Open this post in threaded view
|

Re: Old access token cleanup when token generating, renewing and revoking.

Nalaka Senarathna

hi all,



@Nalaka,
Have we done a load test to verify whether this feature does not affect to the functionalities when there is a high load in a high concurrency situation?
If so appreciate if you can share that details as well.

Yes, we did. [hidden email] please point to the relevant thread. 
 

Earlier we tested this feature to verify the growing Audit table is reduced performance or not( [1] )



This feature is further tested with followings


1.Generate refresh revoke

2.Generate revoke

3.Generate refresh

4.Generate



Testing Phase 1: with feature(Storing the old access tokens in audit table )


Tested with 100 threads using 100 users

Token generate: average throughput 200/s


Generate and refresh: average throughput 170/s


Generate refresh revoke: average throughput 188/s


Tested with 10 threads 

Generate revoke: average throughput 840/s


Tested until 2 million in the audit table, 20000 in access token table.

Latency doesn’t increase when the audit table is large.



Testing phase 2: without this feature(Storing the old access tokens in the same table)


Perform well in starting and reduce the throughputs when the table is growing.

Clearly observed in token generate then refresh then revoke: throughput  55/s.

Latency increasing when access token table is large.



[1]Access token Table cleaning and keeping the access token data for future purposes



thanks.



On Sun, Jul 29, 2018 at 5:11 AM, Nuwan Dias <[hidden email]> wrote:


On Tue, Jul 24, 2018 at 3:17 PM Ishara Cooray <[hidden email]> wrote:
Our objective is to run this clean up process without anyone having to configure anything. Even today we have instructions and scripts to clean up the tables. But in practice, nobody even notices this and never does these kind of stuff until they hit an issue and consult us through support. So if we design this in such a way that someone has to turn on something or configure something, I am positive our efforts will be in waste.

So by default are we going to enable the token cleanup feature and storing old access tokens in audit table?

Yes, otherwise there's no point of this feature IMO because we currently have a solution that cleans data but in reality nobody uses it unless they hit a issue and request our assistance to find and run this. 

@Nalaka,
Have we done a load test to verify whether this feature does not affect to the functionalities when there is a high load in a high concurrency situation?
If so appreciate if you can share that details as well.

Yes, we did. [hidden email] please point to the relevant thread. 



Thanks & Regards,
Ishara Cooray
Senior Software Engineer
Mobile : +9477 262 9512
WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

On Fri, Jul 20, 2018 at 10:04 AM, Harshan Liyanage <[hidden email]> wrote:
Hi Nuwan,

Thanks for your detailed clarifications. Both explanations are perfectly valid. 

Regards,

Harshan Liyanage
lean.enterprise.middleware.


On Thu, Jul 19, 2018 at 8:35 PM Nuwan Dias <[hidden email]> wrote:


On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage <[hidden email]> wrote:
Hi all,

@Nuwan: That's why I suggested having a configurable cron expression so that users can configure the task to run on an optimal schedule instead of blocking vital functionalities. Also in that way, we could do a batch deletes and insertions instead of single rows. For example, they could let it run every mid-night so then the access token tables won't grow for millions of records and not affecting the user traffic. 

Our objective is to run this clean up process without anyone having to configure anything. Even today we have instructions and scripts to clean up the tables. But in practice, nobody even notices this and never does these kind of stuff until they hit an issue and consult us through support. So if we design this in such a way that someone has to turn on something or configure something, I am positive our efforts will be in waste. 

@Nalaka: You could let the task to run only on a manager node in a distributed setup using a configuration. 

There is no such thing as a "manager" node in our architecture. All nodes are equal. Plus, we cannot introduce such changes as well due to numerous complexities such as how to HA the manager node, too many varying configurations, etc.

BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed design. :)

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna <[hidden email]> wrote:
hi harshan,

Also if there are multiple nodes then those nodes also may attempt to clean up the same access token at the same time.

related mail thread:[1]
[1]Access token Table cleaning and keeping the access token data for future purposes

regards.


On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias <[hidden email]> wrote:
A periodic task won't work for this because when the system runs on tables with millions of records, the task will lock the table for the clean up process. This will impact other queries being executed on the table and hence block user flows.

On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage <[hidden email]> wrote:
Hi Nalaka,

You could do the same with a configurable periodic task instead of modifying existing token request flows. What you have to do is to implement the token cleanup feature as a periodic task which scans the token related tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit table. There will be a configuration to configure the CRON expression for that task.

WDYT?

Thanks,

Harshan Liyanage
lean.enterprise.middleware.


On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <[hidden email]> wrote:


A solution for the access token table filled up with EXPIRED, INACTIVE and REVOKED tokens in the Access token table, old access token can move to the Audit table when the new token is generating, renewing or token revoking.


Old Access tokens will be stored in newly created Audit table for audit purposes.


*Token cleanup feature can be turn enable / disable.

*When the token cleanup feature is enabled old access token can be stored in the audit table 

  For audit purposes or can disable that feature too, not to store old access tokens(this also can enable/disable).


This project avoids the access token table growing and improve the token lookup. For existing users also can get advantage from this.


Growing Audit table is no problem if the token cleanup feature and retain old access token feature is enabled .because writing operation to Audit table is not increased when the token table is growing(Audit table has no constraint).This property was tested until 14 million token generation.








regards.

--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474


_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729
_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture




--
Nuwan Dias

Director - WSO2, Inc. http://wso2.com
Phone : +94 777 775 729



--
Nalaka Senarathna
Associate Software Engineer | WSO2

Mobile : +94714118474



_______________________________________________
Architecture mailing list
[hidden email]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture