During the shutdown process, a complete flush of the dirty buffers takes place and all connected processes are given time to end. If the database needs to be more aggressively shut down, issue an emergency shutdown after the initial graceful shutdown:
$ proshut [dbname] -F -by
Any active connections will be terminated and PROSHUT will not have to wait for these to exit themselves. For further detail on the differences between a normal and an emergency shutdown, refer to Article:
Since OpenEdge 10.2B
shutdown improvements also introduced the -shutdownTimeout
parameter to limit the time period the PROSHUT process will wait for the database activity to end.
Investigating why the database takes so long to shut down
If this scenario is happening often, it is worth considering the following the next time a shutdown is needed:
0.) Since OpenEdge 10.2B, once the -shutdowntimeout
has been reached, this results in an emergency shutdown of the database. During an emergency shutdown of a database, all writes to the database are stopped. Processes will be killed if they do not disconnect or terminate on their own.
1.) Before killing the process or the Broker, execute the following:
a) Run a PROMON session before PROSHUT to gather all information on user activity. Ideally, users should first be asked to log off before issuing a shutdown. Alternatively, the following Article describes a programmatic method of disconnecting all currently connected users:
b) Check the system for any rogue processes that haven't shut down and produce a stacktrace against them. There could be a process locking the shutdown.
c) If After-imaging is in use; run "rfutil dbname -C aimage list" to get the current status of all ai extents
d) Produce a stacktrace against the _mprshut process itself with kill -16 <process-id> to generate a protrace file. Since 10.1C these can be generated with the proGetStack command. For further information, refer to Article:
e) Send Progress Technical Support the stacktraces that result from the above actions in order to further analyse which processes were locking the shutdown and why.
2. Once the database has successfully completed shutdown, does a "truncate bi
" perform a crash recovery without any error messages?
Errors would indicate file corruption which would need to be investigated at a System level.
3. Is the shutdown action being performed from the same_user_account / environment every time?
It could be either a permissions issue, the DLC environment variable not being set correctly or a different version of proshut executable being used than that which the database was started with.
4. What are the -B and DB block size?
When shared memory is not tuned properly, the shutdown process is affected.
- Even for small databases a -B value under 3000 is considered to be too small when there are many users still connected or even a few users still running intensive transaction processing at shutdown time. Try increasing the -B to at least 32000 and then tune this until ‘Buffer Hits’ in PROMON, 5. Activity, equate to > 85% or paging, under ‘typical' database load.
- For larger databases with -B values larger than realistically needed, the shutdown process is hindered flushing buffers to disk. Further investigations into improving i/o on the system are needed.
5. During shutdown:
- What is the disk wait activity like?
- How long is the Buffer Pool flushing allowed to complete?
- What other activity is running on the system?
- Are only Progress processes hanging or are there other processes hanging as well?
If these cannot be explained, then it may be worth getting the system checked out by the Operating System vendor.
6. In the database log file, check to see if any of the following messages are recorded:
++ bkread: missing bkflsx call (611)
This indicates a contention situation for buffers in the -B buffer pool: (there probably aren't enough to go around). Increasing the -B parameter will improve matters. If the force option is needed, (proshut dbname -F
) then this could indicate a bigger problem and a full database scan followed by database repair actions if there are data corruption in the report.++ 1124 errors
If issued by “BROKER”, then this indicates a defect because, Brokers
do not read blocks
, it flushes the buffer pool to disk during shutdown.
When 1124 block corruption messages are evident, then subsequent investigation finds no block corruption on disk, these block bkioread / bkiowrite errors are due to memory, cache issues at the time and can also be related to any non-Progress utilities accessing the runtime application environment at the time outside of database quiescense (proquiet).++ SYSTEM ERROR: Unable to kill parent process, errno= . (1680)
Review -minport, -maxport
settings. Progress recommends not using port < 5000, as these are typically reserved for other processes. Changing these remote server ports to any unused range is one known fix that will circumvent system errors concerning being unable to kill parent processes.
All of the above could be direct or indirect contributing factors to the delaying of the completion of the database shutdown.