Feedback
 
Did this article resolve your question/issue?

   

Your feedback is appreciated.

Please tell us how we can make this article more useful. Please provide us a way to contact you, should we need clarification on the feedback provided or if you need further assistance.

Characters Remaining: 1025

 


Article

Linux Database crashes when semaphores get removed

« Go Back

Information

 
Article Number000086484
EnvironmentProduct: OpenEdge
Version: 10.2B,11.x
OS: Red Hat Linux 7.2, 7.3
Question/Problem Description
Database crashes when its semaphores get removed.

Whenever any command is run against the database using as the same user account that started the databases, semaphores get removed when the user logs off and the database crashes.

The database crashes with semaphore errors every time a user account that runs a database administrative task cron session finishes

The database is started with a user account where the corresponding semaphores are owned by this user:

$  sudo -u sysadmin proserve dbname
$ ipcs -s 

------ Semaphore Arrays -------- 
key semid owner perms nsems 
0x00000000 983040 sysadmin 666 7 
0x00000000 1015809 sysadmin 666 39 
0x00000000 1048578 sysadmin 666 39

When someone logs on to the system with this user account and connects to the database, for example: using ssh or sudo
$  proshut dbname -C list

When that user logs off from the system, semaphore errors occur and the database crashes

/var/log/messages reports the following: 
     
systemd: Created slice user-1000.slice. 
systemd: Starting user-1000.slice. 
systemd: Started Session 759 of user sysadmin. 
systemd-logind: New session 759 of user sysadmin. 
systemd: Starting Session 759 of user sysadmin. 
systemd-logind: Removed session 759. 
systemd: Removed slice user-1000.slice. 
systemd: Stopping user-1000.slice. 
Steps to Reproduce
Clarifying Information
As long as the same user account that owns the semaphores does not log off the database remains up and running.

The database starts with appropriate semaphores sets (-semsets) and the kernel has been tuned with SEMMSL 250, SEMMNI 256 and SEMMNS 32000. Article 000020723, Calculating SEMMSL, SEMMNS and SEMMNU  
Error Message(1075) Semaphore id -32767 was removed
(6517) SYSTEM ERROR: Unexpected error return from semAdd -1

(1132) Invalid semaphore id
(10839) SYSTEM ERROR: Unable to set semaphore set : semid = 7, errno = 22.
(-----) semLockLog_2: semValue val = -1
Defect/Enhancement Number
Cause
The default value for RemoveIPC in RHEL 7.2 is yes
  • The Linux parameter RemoveIPC=yes in etc/systemd/logind.conf is causing the issue when the user is the non-system user.
  • The RemoveIPC= logic only applies to non-system users. It is disabled entirely for system users
When user accounts start Applications using IPC (semaphores, shared memory, message queues) do not take this into account, they will experience problems after update to RHEL 7.2 or later when that user account logs off. 

There is a difference in behavior between system and regular users. IPC objects of the root user and system users are excluded from the effect of this setting, otherwise it is a configurable setting as documented in the Linux man pages. Any user that starts processes & daemons requiring communication & synchronization mechanisms need to be started by system users which are excluded from IPC removal (shared memory and semaphores). Otherwise the default behavior for systemd is to “clean slate on logout” ensuring the user no longer consumes IPC resources.
 
(**) man pages: logind.conf

RemoveIPC=
           Controls whether System V and POSIX IPC objects belonging to the user shall be removed when the user fully logs out.
Takes a boolean argument.
If enabled, the user may not consume IPC resources after the last of the user's sessions terminated.
This covers System V semaphores, shared memory and message queues, as well as POSIX shared memory and message queues.
Note that IPC objects of the root user and other system users are excluded from the effect of this setting. Defaults to "yes"
Resolution
Set up accounts that start databases appropriately.

OPTION 1: Create the user as a system user.

The RemoveIPC logic only applies to non-system users. It is disabled entirely for system users determined at systemd compile time from the SYS_UID_MAX setting in /etc/login.defs. Ensure the user is registered properly as system user (i.e. with a UID < 1000) instead of regular user.

A user created with the system option (adduser -r or adduser --system)

OPTION 2: For regular user accounts prevent semaphores being removed when the user account logs out.

Edit /etc/systemd/logind.conf, change RemoveIPC=no, save and reboot the system or restart systemd-logind service.     

Additional Login manager configuration considerations:

Another configuration setting is KillUserProcesses, which controls whether all processes of a user's session should be killed when the user logs out of that session.

KillUserProcesses defaults to yes since systemd version 230 and will close background processes started from a user shell for example.
The logind.conf man page explains how to configure the default behavior with: KillOnlyUsers, KillExcludeUsers, KillUserProcesses, loginctl enable-linger, systemd-run
Workaround
Notes
Attachment 
Last Modified Date5/21/2018 8:35 AM