How To: Capturing Request/Response Payloads with Forum Sentry


Sentry administrators may need to capture request and/or response payload data for the API traffic flows processed by Sentry. This article outlines the various ways to capture the request and response data.


Message Payload Capture Options

There are multiple ways to capture the request and response payloads processed by Sentry. The best option depends on the use case requirements.

The following are all options for capturing request/response payload data with Forum Sentry.


1. Debug Level Logging - With debug level logging enabled either globally or at the policy level, the default behavior is to log the request and response payloads.  Note that the local Sentry logs can be viewed in multiple ways: through the WebAdmin interface on the Logging page, through the Log Transaction Viewer, through the CLI interface with appliances, and finally by downloading the full log or a specific transaction and opening in a text editor.

To capture payloads in the local Sentry logs, enable debug logging globally (for all policies) on the Diagnostics>>Logging>>Settings page.



Alternatively, enable debug level logging on individual WSDL or Content Policies, on the Logging tab.




2. Log Override Codes - Sentry allows for always logging certain messages regardless of the global or policy level log level.  For instance, even if debug level is not used globally or on the policy, the payloads can still be logged out using the "Override log level for the following codes" option either globally or on individual policies. 

To capture the request and response payload in the logs use the codes 08407 and 0840D, respectively.

The image below shows how to enable the override log level option for both request and response payloads on an individual policy.




3. Remote Syslog Logging - The same system log data (including request and response payloads) that is collected and stored locally in Sentry can also be sent off to remote log aggregation tools using the industry standard Syslog protocol. The Syslog policies support TCP with TLS to ensure privacy of the logging data while in transit.


With many Sentry use cases the logging information (especially the payloads) may contain sensitive information. Please be aware of where this data is being sent and how it is secured once received.




4. Archive Document Task - The Archive Document task in Sentry allows for storing the full request/response payloads as well as storing individual parts of an XML message using different Data Types. The Archive Document task utilizes a Data Source policy and the corresponding Sentry schema for each database type supported. The database schema files (available on the Data Source policy) outline the tables, references, and formats for the stored data.  The Archive Document task can be applied to request or response processing of various policies. This allows for targeting specific payloads to be archived - rather than all payloads for a specific policy.



5. Query Database Task - The Query Database task allows for storing and retrieving data, to and from a database. While this task utilizes a Data Source policy, the Sentry schema is not used and the data can be stored in any table. The SQL can reference variables that can be pulled from a variety of sources.  Use the ? character in the query and then select the appropriate Source Type and SQL Type.  


Note: To insert a full document, first use a Map Attributes and Headers task to map the Full Document to User Attribute. Then choose that User Attribute as the Source Type in the Query Database task.



6. Log Task - The Log task allows for customized logging of various request and response data, including the payloads.  The log task can use an XPath expression to log a full XML document (or a part of an XML document) or it can log header and attribute values. If the payload is first mapped to an attribute, the attribute can be logged out at any Logging Level.




7. WS Reports - With the WS Reports feature in Sentry, there is an option to save each request and response. The WS Reports utilize a Data Source policy and the corresponding Sentry schema for each database type supported. The database schema files (available on the Data Source policy) outline the tables, references, and formats for the stored data.


Note: This is a global feature and enabling it means storing all request and response data for all traffic processed by Sentry. In heavy volume environments, this may create significant network overhead and could impact overall performance. Please contact Forum Systems Support to discuss the use case requirements prior to enabling.




8. Packet Captures - If there is a need to view a payload at a network trace level, prior to it being processed by Sentry, enable a packet capture on the Diagnostics>>Logging>>Packet Captures page of the WebAdmin interface. While filters can be used, this should be a troubleshooting mechanism only. This is not a recommend solution for collecting and storing payload data for an extended period of time.  Additionally, if the services are using TLS the payload information in the capture will be encrypted.  The captures created by Sentry can be viewed with Wireshark.








Article is closed for comments.