Feedback
Did this article resolve your question/issue?

   

Article

Investigating why the database takes so long to shut down

Information

 
TitleInvestigating why the database takes so long to shut down
URL NameP55254
Article Number000139300
EnvironmentProduce: OpenEdge
Version: All supported versions
OS: All supported platforms
Question/Problem Description
Investigating why the database takes so long to shut down.
Understanding why proshut hangs database shutdown.
Steps to Reproduce
Clarifying Information
Error Message
Defect/Enhancement Number
Cause
Resolution
During the shutdown process, a complete flush of the dirty buffers takes place and all connected processes are given time to end. If the database needs to be more aggressively shut down, issue an emergency shutdown after the initial graceful shutdown:
$   proshut [dbname] -F -by

Any active connections will be terminated and PROSHUT will not have to wait for these to exit themselves. For further detail on the differences between a normal and an emergency shutdown, refer to Article:  Since OpenEdge 10.2B shutdown improvements also introduced the -shutdownTimeout parameter to limit the time period the PROSHUT process will wait for the database activity to end. Investigating why the database takes so long to shut down

If this scenario is happening often, it is worth considering the following the next time a shutdown is needed:

0.) Since OpenEdge 10.2B, once the -shutdowntimeout has been reached, this results in an emergency shutdown of the database.  During an emergency shutdown of a database, all writes to the database are stopped.  Processes will be killed if they do not disconnect or terminate on their own.

1.) Before killing the process or the Broker, execute the following:
 
a) Run a PROMON session before PROSHUT to gather all information on user activity. Ideally, users should first be asked to log off before issuing a shutdown. Alternatively, the following Article describes a programmatic method of disconnecting all currently connected users:
b) Check the system for any rogue processes that haven't shut down and produce a stacktrace against them. There could be a process locking the shutdown.

c) If After-imaging is in use; run "rfutil dbname -C aimage list" to get the current status of all ai extents
 
d) Produce a stacktrace against the _mprshut process itself with kill -16 <process-id> to generate a protrace file. Since 10.1C these can be generated with the proGetStack command. For further information, refer to Article:
e) Send Progress Technical Support the stacktraces that result from the above actions in order to further analyse which processes were locking the shutdown and why.

2. Once the database has successfully completed shutdown, does a "truncate bi" perform a crash recovery without any error messages?

Errors would indicate file corruption which would need to be investigated at a System level.

3. Is the shutdown action being performed from the same_user_account / environment every time?

It could be either a permissions issue, the DLC environment variable not being set correctly or a different version of proshut executable being used than that which the database was started with.

4. What are the -B and DB block size?

When shared memory is not tuned properly, the shutdown process is affected.
For example:
  • Even for small databases a -B value under 3000 is considered to be too small when there are many users still connected or even a few users still running intensive transaction processing at shutdown time. Try increasing the -B to at least 32000 and then tune this until ‘Buffer Hits’ in PROMON, 5.  Activity, equate to > 85% or paging, under ‘typical' database load.
  • For larger databases with -B values larger than realistically needed, the shutdown process is hindered flushing buffers to disk. Further investigations into improving i/o on the system are needed.
5. During shutdown:
  • What is the disk wait activity like?
  • How long is the Buffer Pool flushing allowed to complete?
  • What other activity is running on the system?
  • Are only Progress processes hanging or are there other processes hanging as well?
If these cannot be explained, then it may be worth getting the system checked out by the Operating System vendor.

6. In the database log file, check to see if any of the following messages are recorded:
 
++ bkread: missing bkflsx call (611)

This indicates a contention situation for buffers in the -B buffer pool: (there probably aren't enough to go around). Increasing the -B parameter will improve matters. If the force option is needed, (proshut dbname -F) then this could indicate a bigger problem and a full database scan followed by database repair actions if there are data corruption in the report.

++ 1124 errors

If issued by “BROKER”, then this indicates a defect because, Brokers do not read blocks, it flushes the buffer pool to disk during shutdown. 
When 1124 block corruption messages are evident, then subsequent investigation finds no block corruption on disk, these block bkioread / bkiowrite errors are due to memory, cache issues at the time and can also be related to any non-Progress utilities accessing the runtime application environment at the time outside of database quiescense (proquiet).

++ SYSTEM ERROR: Unable to kill parent process, errno= . (1680)

Review -minport, -maxport settings. Progress recommends not using port < 5000, as these are typically reserved for other processes. Changing these remote server ports to any unused range is one known fix that will circumvent system errors concerning being unable to kill parent processes.

All of the above could be direct or indirect contributing factors to the delaying of the completion of the database shutdown.
Workaround
  
Notes
Last Modified Date6/17/2020 7:28 AM
Attachment 
Files
Disclaimer The origins of the information on this site may be internal or external to Progress Software Corporation (“Progress”). Progress Software Corporation makes all reasonable efforts to verify this information. However, the information provided is for your information only. Progress Software Corporation makes no explicit or implied claims to the validity of this information.

Any sample code provided on this site is not supported under any Progress support program or service. The sample code is provided on an "AS IS" basis. Progress makes no warranties, express or implied, and disclaims all implied warranties including, without limitation, the implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample code is borne by the user. In no event shall Progress, its employees, or anyone else involved in the creation, production, or delivery of the code be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample code, even if Progress has been advised of the possibility of such damages.