Feedback
Did this article resolve your question/issue?

   

Article

Large performance decrease occurs after applying Sun Solaris patches

« Go Back

Information

 
TitleLarge performance decrease occurs after applying Sun Solaris patches
URL NameP147903
Article Number000140789
EnvironmentProduct: OpenEdge
Version: 10.x ,11.x
OS: Solaris 10
Question/Problem Description
Large performance decrease occurs after applying Sun Solaris patches
System CPU is being substantially utilized
Progress processes consume much more System CPU than would normally occur, sometimes over 50%
Progress user response times are extremely slow
All CPUs in the machine occasionally spike to 100% System CPU
When all CPUs spike to 100% System CPU, there are no system calls being made
Lots of cross calls (xcalls) are occurring
Machine performance degrades as more users log on and more processes start up.
Steps to Reproduce
Clarifying Information
Machine has lots of memory.  e.g. 64GB
Machine has lots of CPUs.  e.g. 16 CPUs
A large number of Progress processes are being run on the machine, including Progress clients, AppServers, WebSpeed agents, Sonic Adapters
Error Message
Defect/Enhancement Number
Cause
A change in one of Sun's patches has caused more aggressive memory management to occur.  The exact Sun patch that caused the behavior has not been identified due to the number of patches that were applied at the same time.

Sun bug number 6642475 exists for this issue.

The performance problem is caused by Sun's aggressive page coalescing which is turned on by default.  Even though this feature was enabled (by default) prior to the performance problem occurring, after applying the patch the algorithm changed with relation to how it searches for large contiguous pages.

To explain in more detail;  When a process like _progres is spawned it will create it's process heap which is used for it's own work area (not shared with other processes).  It allocates it's memory in 8kb pages until it a time when it crosses a 4MB boundary.  At that time, the operating system will try to coalesce (consolidate) the 8kb pages of memory into a 4MB page.

The reason for doing this coalescing is to improve performance.  Instead of having 8kb pages scattered over many different system boards in the machine, it tries to coalesce them onto the same system board as the CPU being used by the process.  This makes the path from the CPU to memory much shorter.

The problem that is occurring is that this task of coalescing the memory consumes an increasing amount of system CPU resources as the number of processes on the machine increases.

The reason why many CPUs simultaneously go to 100% System CPU is because the _progres process performs mmap to acquire memory.  This operation can be quite expensive on a machine with a number of CPUs, as each time mmap is done by a process it results in cross calls (xcalls) to all CPUs on which the process has run.  This essentially means that it locks all of those processors for a short period of time.  You will not see these mmaps as sys calls (system calls), so while you see that System CPU has gone to 100% there will be no sys calls. 
Resolution

Disable aggressive page coalescing.

This can be done both online or offline.

Offline method (takes effect after a reboot):

Add the following entry to the /etc/system file:
set pg_contig_disable=1


Online method (NOT RECOMMENDED - please discuss with Sun before using this method):
This method is not supported or recommended by Sun as it updates the running kernel directly.  A mistake can crash the system or cause other issues.


echo "pg_contig_disable/W 1" | mdb -kw

Only processes started AFTER setting pg_contig_disable will use the new setting.


In most environments, leaving aggressive page coalescing enabled will improve performance.  It is not recommended to disable the feature unless this specific issue is being encountered.

Workaround
Notes
Last Modified Date3/30/2017 8:01 PM
Attachment 
Files
Disclaimer The origins of the information on this site may be internal or external to Progress Software Corporation (“Progress”). Progress Software Corporation makes all reasonable efforts to verify this information. However, the information provided is for your information only. Progress Software Corporation makes no explicit or implied claims to the validity of this information.

Any sample code provided on this site is not supported under any Progress support program or service. The sample code is provided on an "AS IS" basis. Progress makes no warranties, express or implied, and disclaims all implied warranties including, without limitation, the implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample code is borne by the user. In no event shall Progress, its employees, or anyone else involved in the creation, production, or delivery of the code be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample code, even if Progress has been advised of the possibility of such damages.