It is possible that the operating system kernel configuration (ulimit -c) may not be set to allow core file generation.
- In releases prior to OpenEdge 10.2B this would also inhibit protrace generation.
- When the core file size is not large enough a core dump may not be created in the case of an application crash.
The protrace calls 'backtrace' to unwind the stack. To create that stack trace the program calls 'backtrace' OS function, gets call stack from that and tries to map those addresses into program or external library symbol names.
- Suffering from memory conditions on the server could also result in the stack not being able to be unwound. For example associated memory errors like "stget: out of storage. (1450)", could additionally be why the protrace is not created.
- The fork errors can occur for many different reasons. Associated memory errors like: "(358) SYSTEM ERROR: Too many subprocesses, cannot fork. Errno=11"
- Error number 11 means "No more processes
- Memory allocation is one reason this can occur, others of course are:
- Too many processes on the system.
- Too many processes for the user
Since OpenEdge 10.2B and later, disabling core files
does not prevent protrace files from being generated.
However, when there is a memory violation and for whatever other reason the protrace is not generated at all or the CStack is incomplete when a process crashes, enabling the core files may be required to get the necessary data for troubleshooting. For example prior to OpenEdge 11.6.4, 11.7 async unsafe calls in signal handling can cause no C-stack in a protrace. For further information refer to Article 000064858, Can KILL -SIGUSR1 cause a process to crash?
The core file has more information than just a C-stack The core file also provides information about the state of local and global variables when the application crashed, which can be useful on tracking down the cause of the crash.
In many operating systems core file generation is disabled by default. Consult the operating system administration guides on how to enable core file generation.
Ensure that the Operating System is configured to generate system dumps:
ulimit -c is not disabled and there are no suid or hardlimits preventing core file generation;
- ulimit -a can be used to identify if core size has been set.
- ulimit -c can be used to set the core file size to unlimited or some value like 524288.
- ulimit -c 0 is set, will not generate any core file at all.
Commonly, core file generation may be enabled on a per user bases by putting the following statement in the user .profile
(or equivalent profile configuration file depending on operating system).
ulimit -c unlimited
In some operating systems such as RedHat Linux this may be performed on a global basis by adding the following ulimit command in /etc/profile:
ulimit -S -c unlimited > /dev/null 2>&1
If both the client and the database server crash and each generates a core file we may need to identify which core file belong to which process; in order for LINUX kernel to have unique core file in form of “core.<pid>”, one need to make sure the value in “/proc/sys/kernel/core_uses_pid”
is greater than 0.
Review the system log files for additional kernel exhaustion and get the UNIX administrator to get these sorted out.