Feedback
 
Did this article resolve your question/issue?

   

Your feedback is appreciated.

Please tell us how we can make this article more useful. Please provide us a way to contact you, should we need clarification on the feedback provided or if you need further assistance.

Characters Remaining: 1025

 


Article

Investigating why the database takes so long to shut down

Information

 
Article Number000021134
EnvironmentProduce: OpenEdge
Version: All supported versions
OS: All supported platforms
Question/Problem Description
Investigating why the database takes so long to shut down.
Understanding why proshut hangs database shutdown.
Steps to Reproduce
Clarifying Information
Error Message
Defect/Enhancement Number
Cause
Resolution
During the shutdown process, a complete flush of the dirty buffers takes place and all connected processes are given time to end. If the database needs to be more aggressively shut down, issue an emergency shutdown:
$   proshut [dbname] -F -by

Any active connections will be terminated and PROSHUT will not have to wait for these to exit themselves. For further detail on the differences between a normal and an emergency shutdown, refer to Article:  Since OpenEdge 10.2B shutdown improvements also introduced the -shutdownTimeout parameter to limit the time period the PROSHUT process will wait for the database activity to end. Investigating why the database takes so long to shut down

If this scenario is happening often, it is worth considering the following the next time a shutdown is needed:

0.) Since OpenEdge 10.2B, once the -shutdowntimeout has been reached, this results in an emergency shutdown of the database.  During an emergency shutdown of a database, all writes to the database are stopped.  Processes will be killed if they do not disconnect or terminate on their own.

1.) Before killing the process or the Broker, execute the following:
 
a) Run a PROMON session before PROSHUT to gather all information on user activity. Ideally, users should first be asked to log off before issuing a shutdown. Alternatively, the following Article describes a programmatic method of disconnecting all currently connected users:
b) Check the system for any rogue processes that haven't shut down and produce a stacktrace against them. There could be a process locking the shutdown.

c) If After-imaging is in use; run "rfutil dbname -C aimage list" to get the current status of all ai extents
 
d) Produce a stacktrace against the _mprshut process itself with kill -16 <process-id> to generate a protrace file. Since 10.1C these can be generated with the proGetStack command. For further information, refer to Article:
e) Send Progress Technical Support the stacktraces that result from the above actions in order to further analyse which processes were locking the shutdown and why.

2. Once the database has successfully completed shutdown, does a "truncate bi" perform a crash recovery without any error messages?

Errors would indicate file corruption which would need to be investigated at a System level.

3. Is the shutdown action being performed from the same_user_account / environment every time?

It could be either a permissions issue, the DLC environment variable not being set correctly or a different version of proshut executable being used than that which the database was started with.

4. What are the -B and DB block size?

When shared memory is not tuned properly, the shutdown process is affected.
For example:
  • Even for small databases a -B value under 3000 is considered to be too small when there are many users still connected or even a few users still running intensive transaction processing at shutdown time. Try increasing the -B to at least 20000 and then tune this until ‘Buffer Hits’ in PROMON, 5.  Activity, equate to > 85% or paging, under ‘typical' database load.
  • For larger databases with -B values larger than realistically needed, the shutdown process is hindered flushing buffers to disk. Further investigations into improving i/o on the system are needed.
5. During shutdown:
  • What is the disk wait activity like?
  • How long is the Buffer Pool flushing allowed to complete?
  • What other activity is running on the system?
  • Are only Progress processes hanging or are there other processes hanging as well?
If these cannot be explained, then it may be worth getting the system checked out by the Operating System vendor.

6. In the database log file, check to see if any of the following messages are recorded:
 
++ bkread: missing bkflsx call (611)

This indicates a contention situation for buffers in the -B buffer pool: (there probably aren't enough to go around). Increasing the -B parameter will improve matters. If the force option is needed, (proshut dbname -F) then this could indicate a bigger problem and a full database scan followed by database repair actions if there are data corruption in the report.

++ 1124 errors

If issued by “BROKER”, then this indicates a defect because, Brokers do not read blocks, it flushes the buffer pool to disk during shutdown.

++ SYSTEM ERROR: Unable to kill parent process, errno= . (1680)

Review -minport, -maxport settings. Progress recommends not using port < 2000, as these are reserved for UNIX processes. Changing these ports to any unused range above 3000 will circumvent system errors concerning being unable to kill parent processes.

All of the above could be direct or indirect contributing factors to the delaying of the completion of the database shutdown.
Workaround
  
Notes
References to Other Documentation:

Progress article(s):
000010773, What is UNIX truss command?
Attachment 
Last Modified Date4/24/2019 7:17 PM