Administrators monitoring the Sentry memory usage over time may notice drops in the usage percentage occurring at somewhat regular intervals. Sentry is a Java application and the regular drops in memory usage are due to the JVM Garbage Collection (GC). The GC routine is handled by the underlying JVM and the GC parameters are not user configurable. For a detailed explanation of the JVM GC see:
http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html
http://www.cubrid.org/blog/dev-platform/understanding-java-garbage-collection/
FAQs - The following are frequently asked questions related to GC with Sentry:
1. Can the Sentry use case and runtime patterns (frequency, transaction times, document sizes, processing done in Sentry, etc...) impact how the GC works?
Yes. Ultimately, how the GC works depends on how an application is coded. In the case of Sentry, different types of processing invoke different code paths which will indicate the frequency of the GC.
For instance, for services with low volume but longer running transactions, the memory usage stat may climb high as the GC happens infrequently. In contrast, with smaller faster transactions there may be more frequent GC occurrences keeping the overall memory usage stat lower.
There are no definitive rules regarding when the GC will run for any given Sentry instance. There are many different use case options with Sentry, with many single instances processing a variety of message patterns, sizes, frequencies in variety of different ways. Even Sentry instances that are deployed with the same configurations Active/Active may not show the same memory usage stats, indicating the GC is working differently on these systems.
2. Is there any GC tuning available with Forum Sentry?
Not with Sentry appliances, possibly with Sentry software instances though this is not advised and unsupported.
With the Forum Sentry appliances (hardware and virtual) the JVM (and inherent GC) run within the locked ForumOS operating systems. Therefore there is no tuning of the GC with Sentry appliances.
With Sentry software instances, running on Windows or Linux as a service, it may be possible to tune the GC. Note that Forum Support does not advise or support GC tuning.
3. Will increasing the amount of RAM with Sentry virtual appliances or software impact the GC?
Possibly. While the overall memory usage stat may be lower, the frequency of the GC will not change significantly based on the RAM available. Rather, changes in the Sentry configuration or runtime patterns (frequency, size, etc...) would have a greater impact.
4. Is it possible to monitor the GC?
Monitoring the GC is not currently possible through the Sentry interface or via the manual "runDiagnostics" CLI command. Monitoring the Memory usage can be automated via the Sentry SNMP or JMX APIs.
5. What if memory usage continues to climb steadily, even with the periodic GC running to drop the usage regularly?
While at first glance this may seem to indicate a memory leak, it should be noted that memory leaks have been very rare with the Sentry product since it's launch in 2001. Rather, the increase in memory usage may indicate a few other scenarios, which typically only show up after several months of uptime:
- slow, stuck, or hung threads adding up over time - hung threads can result from failed transactions, specifically when there are remote server issues and no timeouts on the remote policies, rebooting clears hung threads
- internal Sentry caches growing over time - for instance when Sentry is generating session cookies and storing user attribute data to these sessions these are stored in a local cache, there are also other internal caches, all of which are cleared on a reboot
- slow but regular increases in message volume and message sizes - in these cases a reboot won't likely resolve the problem but rather it may be time to scale
- a combination of all of the above
Note that while rebooting may resolve a couple of the issues that grow over time (as noted in #5), Forum Systems does not necessarily recommend a scheduled reboot scheme. Rather we recommend reporting and discussing the issues experienced in each environment with Forum Systems Support so a permanent solution can be achieved.
0 Comments