Best Practices: Production Change Procedures

To minimize the risk of a change or system upgrade causing a problem with an existing production flow,  Forum Systems recommends the following procedures for all production changes (version upgrades, policy changes, or policy additions):

1. Schedule the downtime or maintenance window for a time when there is low traffic volumes expected (weekends, after hours, etc.)
2. Remove the Sentry instance to be updated from the live production environment. Typically this is done by the Sentry from the upstream Load Balancer pool. While redundant Sentry instances handle the current production traffic, one system is updated.
3. Export/Backup the existing Sentry configuration prior to an update. This can be scheduled to happen automatically on a daily basis to an FTP, SFTP, or Database location.
4. Upgrade the device and/or make the intended policy changes.
5. Manually inspect/review the configuration as best as possible in the WebAdmin interface.
6. Test the specific policies that were added or changed thoroughly.
7. Run the standard Automated Regression Tests against ALL services deployed on the Sentry instance. This may uncover any other inadvertent policy changes.
8. If there are no issues put the device back into production and monitor.
9. If there are no issues, take the other Sentry instance(s) out of production and follow the same procedures to update and test.

Additional Notes:

Forum Sentry configuration changes are implemented immediately upon clicking 'Apply' or 'Save' on a policy object without requiring any restart.  This allows for adding new policies without impacting existing production flows.

Administrators should be aware of possible inadvertent  configuration changes that could impact seemingly unrelated policies.  

For instance, importing a new REST policy will also import all of the dependencies of the policy. This includes the listener, ssl policies, task lists, keys, etc. While importing, any existing policy objects that are have the same name may be overwritten. This could mean inadvertently  overwriting an existing policy object that is associated to another runtime policy.