Article

Why is my bi file growing so large

« Go Back

Information

 
Article Number000001274
EnvironmentProduct: Progress
Version: 8.x, 9.x
Product: OpenEdge
Version 10.x, 11.x
OS: All supported platforms
Question/Problem Description
Why is my .bi file growing so large ?
Why is my .bi file hitting the 2Gb limit ?
How to troubleshoot an excessive Before Image file growth ?
How can I stop the BI file from growing excessively ?
Steps to Reproduce
Clarifying Information
Error Message
Defect/Enhancement Number
Cause
Resolution
Before-Image File Size Limits
  • In versions prior to Progress 9 limited the size of the entire bi file to no more than 2 gig, no matter how the bi extents were physically arranged. In other words, the sum of the bi extent sizes could not exceed 2 gig.
  • In Progress 9 (with the introduction of Storage Areas) database bi extents are treated differently: the sum of the bi extents can exceed 2Gb, and
  • Starting Progress 9.1C (with the introduction of Large Files Support for Enterprise Database Licenses), each extent can exceed 2Gb when largefiles are enabled at database level and the OS supports largefiles.
How the .bi file works:

When a before-image (.bi) file is created initially, 4 bi-clusters are automatically allocated. Prior to Progress 9.x the default bicluster size was 16KB, since Progress 9.x it is 512KB by default. The bi cluster size can be changed offline with PROUTIL -C truncate bi -bi <size in KB).

Before-image file clusters are managed as a doubly linked list, containing forward and backward pointers.  When a transaction starts, transaction information is stored in the first cluster.  When this cluster is full, transaction notes are then written to the next cluster.  Finally, when the initial four clusters are filled with pending transactions, a decision needs to be made to either extend the .bi file or reuse space, if possible.

If the first cluster has no active transactions, then this cluster is reused, marked as the "last" cluster, thus making cluster 2 the "first" cluster in the .bi file for the next "extend or reuse" decision.

However, if the first cluster still contains incomplete transactions, then this cluster cannot be re-used. When this happens, the .bi file grows by another cluster - in this example, cluster 5.  When cluster 5 is full, the same decision process is exercised again.

If cluster 1 still contains pending transactions, then the .bi file is again extended.  Eventually, the transactions in the first cluster will be committed, and will then be available for re-use, so no additional clusters need to be added at this checkpoint.

Why would pending transactions in bicluster 1 sit there for so long?

A bi file can grow extremely large when for example a schema change is applied to a large database table (1 Gb or more).

To address this, starting in Progress 8.2A, "Fast Schema Change" was introduced to make schema changes faster and not use as much .bi space. In Progress 9, "Schema Versioning" was introduced, which only uses .bi space when an index is added as active to or deleted from a database table, but not when fields are added or deleted.

Excessive bi file growth only ever occurs when a transaction is started and remains active or open for a long period of time.

One simple example is that the user who initiated the transaction walks away from the desk while the application is waiting for user input needed to complete the transaction.  Since this cluster can never be "skipped over" (it is managed as a doubly linked list), the .bi file will continue to grow as new clusters are added when needed at each checkpoint.

Sometimes there is no way to avoid the .bi file being extended; there just isn't enough time to commit transactions and make the cluster available before it is needed again by other transactions. Typically this scenario would be supported by excessive Checkpoints (LE 1 minute) and alleviated by increasing the bi cluster size.

Transactions have their before-image transaction notes recorded in the bi file so that we can undo during online processing and complete the Redo and Physical and Logical Undo bi recovery. As described above, he bi file is organized in sections called clusters, specifically biclusters.

As each cluster is filled with before-image transaction notes, the bi file is extended to provide more clusters.  To minimize bi file growth, clusters are re-used, but only when all Transactions to which the bi notes belong, are committed to the database and closed. As bi notes are written serially to the bi file then clusters can only be re-used serially.  This means it only takes one active transaction to remain open for Progress to keep adding new clusters as other users open and close transactions.

Long duration transactions are caused by:

Use of ESQL or ODBC.

The way in which the ABL application transaction scope is coded.
"A transaction is a unit of work which must complete in its entirety or not at all"

Bi Growth will therefore occur when:
  • Data is imported from a file in one transaction, as opposed to importing with one transaction per record 
  • The scope of the transaction all-or-nothing, one transaction, as opposed to splitting into smaller transactions
  • The size of the transaction is a function of the size of the underlying database.  As more records are added to tables over time, so the transaction size increases.
  • The application code (e.g. triggers) get into a loop that blocks other parts of the code and never completes.
  • A transaction is started at login and does not end until logout, possibly extending for the entire day.
  • Users start a transaction and then leave to do something else, or they wait for the next caller to ring.
In case of excessive bi growth, the first step is to identify users running long transactions. Start promon; R&D; 1(Status display); 4(Processes/client...); 3(Active transaction).  A list of users with open active transactions will be visible. A transaction time of over 10 minutes would be unusual.

Users who have open transactions can be traced using PROMON as above and asked about those time spans. Finding these users, knowing exactly why their transaction takes so long and cutting this transaction short is the best solution to addressing excessive bi growth once the associated application code responsible has been isolated.

Starting from Progress 8.3, using the Recovery log Threshold (-bithold) startup parameter sets the maximum size to which BI files can grow. Once the threshold is reached, the database performs an emergency shut down unless the bistall database startup parameter is also used. This mechanism ensures (unexpected) excessive bi growth can be throttled.


 
Workaround
Notes
Attachment 
Last Modified Date5/3/2017 3:56 PM
Disclaimer

The origins of the information on this site may be internal or external to Progress Software Corporation (“Progress”). Progress Software Corporation makes all reasonable efforts to verify this information. However, the information provided is for your information only. Progress Software Corporation makes no explicit or implied claims to the validity of this information.

Any sample code provided on this site is not supported under any Progress support program or service. The sample code is provided on an "AS IS" basis. Progress makes no warranties, express or implied, and disclaims all implied warranties including, without limitation, the implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample code is borne by the user. In no event shall Progress, its employees, or anyone else involved in the creation, production, or delivery of the code be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample code, even if Progress has been advised of the possibility of such damages.



Feedback
 
Was this article helpful?

   

Your feedback is appreciated.

Please tell us how we can make this article more useful. Please provide us a way to contact you, should we need clarification on the feedback provided or if you need further assistance.

Characters Remaining: 1025