Indusface – Product Release & Rollout SOP

Business continuity is at the forefront of most systems and process design at Indusface. In a recent blog, we discussed how Indusface follows design-for-failure principles a powerful approach that enables us to deploy faster. In this blog, I will talk about the processes we have to ensure that our code and rule deployments do not cause widespread downtime to our protected assets.  

For Indusface, the hot paths include the protection path (Web Application and API Protection Platform) and other features such as CDN acceleration that have an impact on customer applications and APIs.  

The protection path includes security updates to traffic inspection rules, detection plugins, and code that supports and enhances the security rules.  

Let’s take a few examples of strategies we use for seamless deployment in the protection and traffic flow. 

Minimal Scope Then Rolling Update 

Deployment scope is as tight as possible depending on the update in question. We deploy at the UAT site (if provided by the customer) -> individual site -> Block -> Region -> Global with monitoring at every step. This ensures that any deployment issues are found, rolled back if needed and fixed before widespread issues.  Not all changes fit all scopes e.g. WAF software update must be at a block level not on an individual site level. 

A/B Deployment 

Larger scopes are rolled out using an A/B strategy. Here only a small percentage of traffic uses the new deployment and as confidence grows, more and more of the traffic is routed into new deployment until all traffic passes through the new system. If any issues are encountered, we can instantly move 100% traffic back to the original system.

Extensive automation including automated rollback ensures that this process is extremely fast. This was successfully used in a recent migration where we moved all our customers to new architecture. A/B deployment helped us move our customers’ applications to the new architecture with minimal disruption. 

Our rollout process includes notifications to customers at least four working days before we release. These notifications give exact details on what is being deployed and whether there is any risk of downtime. 

Monitoring 

Post deployments, the team responsible for the release process uses automated monitoring alerts and manual sampling to ensure that the feature is doing what was designed and no unexpected interaction causing issues are noticed. They are empowered to take corrective actions including rolling back the release.

Edge Cases 

Where potentially high impact changes must be made at larger scopes e.g. the above-mentioned WAF software update, we start with our own ‘production’ test block running in production fronting only Indusface test sites and covering the gamut of customer and license types, e.g. API, Web application, CDN, self-service rules, and so on. Such systems are already in place like the production test block OR spun up as needed for specific releases. 

Post Deployment 

Our design for failure system referenced earlier is always on to take corrective action when any issues are encountered. 

We follow software development best practices where changes go through dev, qa, staging, and release stages with all the checks and balances at each stage e.g. unit tests, automated QA tests, regression suites, and staging with tapped live traffic. All releases go through a rigorous process prior to release, but we do even more for releases that can affect the protection and availability of customer assets.  

Stay tuned for more relevant and interesting security articles. Follow Indusface on Facebook, Twitter, and LinkedIn.