Feedback
Did this article resolve your question/issue?

   

Article

Linux Database crashes when semaphores get removed

« Go Back

Information

 
TitleLinux Database crashes when semaphores get removed
URL Namesemaphores-removed-when-user-logs-out-on-Linux-000086484
Article Number000115567
EnvironmentProduct: OpenEdge
Version: 10.2B,11.x
OS: Red Hat Linux 7.2, 7.3
Question/Problem Description
Database crashes when its semaphores get removed.

Whenever any command is run against the database using as the same user account that started the databases, semaphores get removed when the user logs off and the database crashes.

The database crashes with semaphore errors every time a user account that runs a database administrative task cron session finishes

The database is started with a user account where the corresponding semaphores are owned by this user:

$  sudo -u sysadmin proserve dbname
$ ipcs -s 

------ Semaphore Arrays -------- 
key semid owner perms nsems 
0x00000000 983040 sysadmin 666 7 
0x00000000 1015809 sysadmin 666 39 
0x00000000 1048578 sysadmin 666 39

When someone logs on to the system with this user account and connects to the database, for example: using ssh or sudo
$  proshut dbname -C list

When that user logs off from the system, semaphore errors occur and the database crashes

/var/log/messages reports the following: 
     
systemd: Created slice user-1000.slice. 
systemd: Starting user-1000.slice. 
systemd: Started Session 759 of user sysadmin. 
systemd-logind: New session 759 of user sysadmin. 
systemd: Starting Session 759 of user sysadmin. 
systemd-logind: Removed session 759. 
systemd: Removed slice user-1000.slice. 
systemd: Stopping user-1000.slice. 
Steps to Reproduce
Clarifying Information
As long as the same user account that owns the semaphores does not log off the database remains up and running.

The database starts with appropriate semaphores sets (-semsets) and the kernel has been tuned with SEMMSL 250, SEMMNI 256 and SEMMNS 32000. Article 000020723, Calculating SEMMSL, SEMMNS and SEMMNU  
Error Message(1075) Semaphore id -32767 was removed
(6517) SYSTEM ERROR: Unexpected error return from semAdd -1

(1132) Invalid semaphore id
(10839) SYSTEM ERROR: Unable to set semaphore set : semid = 7, errno = 22.
(-----) semLockLog_2: semValue val = -1
Defect/Enhancement Number
Cause
The default value for RemoveIPC in RHEL 7.2 is yes
  • The Linux parameter RemoveIPC=yes in etc/systemd/logind.conf is causing the issue when the user is the non-system user.
  • The RemoveIPC= logic only applies to non-system users. It is disabled entirely for system users
When user accounts start Applications using IPC (semaphores, shared memory, message queues) do not take this into account, they will experience problems after update to RHEL 7.2 or later when that user account logs off. 

There is a difference in behavior between system and regular users. IPC objects of the root user and system users are excluded from the effect of this setting, otherwise it is a configurable setting as documented in the Linux man pages. Any user that starts processes & daemons requiring communication & synchronization mechanisms need to be started by system users which are excluded from IPC removal (shared memory and semaphores). Otherwise the default behavior for systemd is to “clean slate on logout” ensuring the user no longer consumes IPC resources.
 
(**) man pages: logind.conf

RemoveIPC=
           Controls whether System V and POSIX IPC objects belonging to the user shall be removed when the user fully logs out.
Takes a boolean argument.
If enabled, the user may not consume IPC resources after the last of the user's sessions terminated.
This covers System V semaphores, shared memory and message queues, as well as POSIX shared memory and message queues.
Note that IPC objects of the root user and other system users are excluded from the effect of this setting. Defaults to "yes"
Resolution
Set up accounts that start databases appropriately.

OPTION 1: Create the user as a system user.

The RemoveIPC logic only applies to non-system users. It is disabled entirely for system users determined at systemd compile time from the SYS_UID_MAX setting in /etc/login.defs. Ensure the user is registered properly as system user (i.e. with a UID < 1000) instead of regular user.

A user created with the system option (adduser -r or adduser --system)

OPTION 2: For regular user accounts prevent semaphores being removed when the user account logs out.

Edit /etc/systemd/logind.conf, change RemoveIPC=no, save and reboot the system or restart systemd-logind service.     

Additional Login manager configuration considerations:

Another configuration setting is KillUserProcesses, which controls whether all processes of a user's session should be killed when the user logs out of that session.

KillUserProcesses defaults to yes since systemd version 230 and will close background processes started from a user shell for example.
The logind.conf man page explains how to configure the default behavior with: KillOnlyUsers, KillExcludeUsers, KillUserProcesses, loginctl enable-linger, systemd-run
Workaround
Notes
Last Modified Date5/21/2018 8:35 AM
Attachment 
Files
Disclaimer The origins of the information on this site may be internal or external to Progress Software Corporation (“Progress”). Progress Software Corporation makes all reasonable efforts to verify this information. However, the information provided is for your information only. Progress Software Corporation makes no explicit or implied claims to the validity of this information.

Any sample code provided on this site is not supported under any Progress support program or service. The sample code is provided on an "AS IS" basis. Progress makes no warranties, express or implied, and disclaims all implied warranties including, without limitation, the implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample code is borne by the user. In no event shall Progress, its employees, or anyone else involved in the creation, production, or delivery of the code be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample code, even if Progress has been advised of the possibility of such damages.