Feedback
Did this article resolve your question/issue?

   

Article

Cyclic Redundancy Check Explained

« Go Back

Information

 
TitleCyclic Redundancy Check Explained
URL Name20245
Article Number000120687
EnvironmentProduct: Progress
Version: 8.x, 9.x
Product: OpenEdge
Version: 10.x, 11.x, 12.x
OS: All supported platforms
Question/Problem Description
Cyclic Redundancy Check Explained.
How does CRC work within different versions of Progress?
After dumping and loading database, is a recompilation of application programs needed?
How to avoid CRC errors when deploying application code after schema changes?
R-code CRCs and procedure integrity.
Deployment with the Three Database Rule.
The 3 database deployment method.
Steps to Reproduce
Clarifying Information
Error Message** CRC for <filn> does not match CRC in <procedure>. Try recompiling. (1896)
Defect Number
Enhancement Number
Cause
Resolution
Starting with Version 6.3F, Progress incorporated a powerful feature to support the deployment of object code, the Cyclic Redundancy Check (CRC).

With CRC, the same object code can run against different database tables that have identical structures, even when these tables are created at different times and with different methods.

Each table in a Progress database has it's own CRC number that is stored in the metaschema field _File._CRC.

Deployment strategies are based on the CRC feature.

Why does compiled code fail to run when application schema are identical?

After a migration application code that was not recompiled against the new OpenEdge version may fail to run:

    ** CRC for <filn> does not match CRC in <procedure>. Try recompiling. (1896).

A subsequent incremental application schema dump (delta.df) does not reveal any difference between the Development database and the Production database. How can you get this error if the application schema are identical?
  • The answer is in the word "identical". The CRC calculation for a file (R-CODE) contains components that do not show up in an incremental .df file.
  • Some of these components (for example, _Data-Type, _Extent, and _Unique) are fundamental user-defined properties of the database and certainly affect its "structure".
  • Others (for example, _order and _Decimals) can be changed at any time and appear to be much less fundamental.
  • Still others, (for example, _field-rpos) remain completely hidden to a developer but are important to Progress internally.
  • It is the Field Record Position (_field-rpos) that is responsible for "CRC issues".
  • The following table lists the different CRC issues by Progress Version:

History of CRC in Progress:

Version Number   Feature                   CRC 
===============================================================
6.3F             CRCs introduced           * _File._File-number
                                           * _Index._idx-num
                                           * _Field._Field-Rpos
                                           * byte swapping

7                Sequences Introduced      * _Field._Field-Rpos
                                           * Sequence CRC

8.1              Sequence issue resolved   * _Field._Field-Rpos

9                Index CRC Introduced      * _Field._Field-Rpos
                                           * Index CRC


There are also CRCs for schema trigger procedures, but these serve a completely different purpose and are not responsible for any of the CRC issues discussed here. For further information refer to Article:
Discussion of the _Field._field-rpos:

There are two reasons why the _Field._field-rpos is important to Progress internally:
  • Every Progress application database contains fields.
  • Every Progress version is affected by this CRC issue.

The following table lists all the schema fields involved in the CRC calculation for files.

Fields that take part in the CRC Calculation for Files:

_File        _Field           _Index            _Index–Field
=============================================================
_File-Name   _Field-Name      _Index-Name       _Field-Name (*2)
_DB-lang     _Data-Type       _Unique           _Ascending
             _dtype           _num-comp         _Abbreviate
             _sys-field       _idx-num(*3)      _Unsorted
             _field-rpos
             _Decimals
             _Order
             _Extent
             _Fld-stdtype
             _Fld-stlen(*1)
             _Fld-stoff
             _Fld-case


(*1)    _Field._Fld-stlen is not involved in the CRC calculation for BLOB and CLOB fields.
(*2)   _Index–Field._Field-Name for index fields is referenced in the _Field table using _Field-recid in _Index-Field.
(*3)   _Index._idx-num component was used prior to Version 7.x.

  • All of the listed schema fields participate in the CRC calculation for each database table. A change to any of these fields automatically changes the value of _File._CRC for that table.
  • Not all of these schema fields can be modified by the developer. The most notorious schema field in this respect is the _field-rpos (Field Record Position) field.


What is the Field Record Position field (_Field._field-rpos) ?

The Field Record Position holds the value of a field's physical position within a record. This value is automatically supplied by Progress when new fields are created. Freed-up field positions are reused, but it is also possible to create gaps by deleting fields. The _field-rpos has nothing to do with the _order field used for the default display order.

Examples demonstrating _Field._field-rpos pitfalls:

Example 1:  The Effect of CRC when adding and deleting fields

1. Create a new Database Sports1 from the Sports Database

$  prodb sports1 sports

Run procedure rpos.p against it:

/* rpos.p */

OUTPUT TO rpos.txt.
FIND _file WHERE _file._file-name = "Salesrep" NO-LOCK.

PUT UNFORMATTED LDBNAME("dictdb") + "." + _File-name + " (CRC " + STRING(_File._crc) + ")" SKIP.

FOR EACH _field OF _file BY _field-rpos:
  DISPLAY _field-name FORMAT "x(18)" _field-rpos WITH STREAM-IO.
END.

The output of rpos.p shows that the highest value for _field-rpos == 5
The _field-rpos values always start at 2 because Progress reserves the first field record position to store the table number.

Sports1.Salesrep (CRC 43047)

Field-Name............._field-rpos
----------------------.-----------
Rep-Name........................2
Region..........................3
Sales-Rep.......................4
Month-Quota.....................5

2. Add two fields to the Salesrep table:

  • field1 , field2.
  • Leave all other field attributes at their default values.
  • Commit the Database transaction.
  • Run rpos.p again


The output of rpos.p shows that _field-rpos values 6 and 7 are assigned to the new fields, respectively:

Sports1.Salesrep (CRC 8331)

Field-Name............._field-rpos
----------------------.-----------
Rep-Name........................2
Region..........................3
Sales-Rep.......................4
Month-Quota.....................5
field1..........................6
field2..........................7


3.  Delete field1.

  • Commit the transaction.
  • Run rpos.p again.
The output of rpos.p shows that there is now a gap in the _field-rpos sequence where field1 used to be (5 - 7):

 

Sports1.Salesrep (CRC 36130)

Field-Name............._field-rpos
----------------------.-----------
Rep-Name........................2
Region..........................3
Sales-Rep.......................4
Month-Quota.....................5
field2..........................7


4  Create a second Database (Sports2) from Sports:

$  prodb sports2 sports

  • Create an incremental df file from Sports1 to Sports2.
  • The resulting delta.df file contains an ADD FIELD instruction for field2 only.
  • The _field-rpos attribute is not mentioned in the delta.df, this is the source of CRC deployment issues.
  • Apply delta.df file to Sports2 and run rpos.p against Sports2:
The output of rpos.p shows that field2 has a _field-rpos value of 6 in the 'deployment' database

Sports2.Salesrep (CRC 52448)

Field-Name............._field-rpos
----------------------.-----------
Rep-Name.........................2
Region...........................3
Sales-Rep........................4
Month-Quota......................5
field2...........................6


Both Schemas appear to be identical in the Data Dictionary and if another incremental dump between Sports1 and Sports2 were taken, it would not reveal any differences.

The CRCs however, are different:

  • Sports2.Salesrep.field2 has a _field-rpos value of 6
  • Sports1.Salesrep.field2 still has a _field-rpos value of 7.


Example 2:  _field-rpos numbers are reused:

1. Using the Sports1 Database:

  • Delete Salesrep.Region (_field-rpos = 3) .
  • Commit the transaction,
  • Run rpos.p against Sports1.
The output of rpos.p shows there is another gap in the _field-rpos sequence, where Salesrep.Region used to be (2 - 4):

 

Sports1.Salesrep (CRC 7618)

Field-Name............._field-rpos
----------------------.-----------
Rep-Name.........................2
Sales-Rep........................4
Month-Quota......................5
field2...........................7


2. Delete the current Sports2 database and create a new Sports2 Database from Sports again.

$  echo y | prodel sports2
$  prodb sports2 sports

  • Create an incremental dump file delta2.df from Sports1 to Sports2.
  • During the incremental definitions creation, Progress asks if the Region field was renamed or deleted.
  • Select deleted, because you do not want to put the data of the old Salesrep.Region field in the new field2.
  • The delta2.df file now contains an ADD FIELD instruction for field2 and a DROP FIELD instruction for Region.
  • Apply delta2.df to Sports2.
  • Run rpos.p against Sports2
The output of rpos.p shows that the _field-rpos value of 3 has been reused and has been assigned to field2:

 

Sports2.Salesrep (CRC 23241)

Field-Name............._field-rpos
----------------------.-----------
Rep-Name.........................2
field2...........................3
Sales-Rep........................4
Month-Quota......................5


The above examples demonstrate it is virtually impossible to keep two Database's CRC compatible when manual changes are made to one of them through the Data Dictionary.

While the physical position of a field can't be controlled, the logical position can. When the definitions of a table are dumped via the Data Dictionary, there’s the option that is checked by default:

'Include POSITION for .r / Binary Load Compatibility".

When this option is checked, the value of _Field-rpos is dumped into the .df file and then when it is loaded back into a database, the _Field-rpos will be kept as it was and the r-code compiled will be compatible.  

Activating “Include POSITION for .r / Binary Load Compatibility” option ensures that is not required to recompile .r code against any new schema that is created by loading a new *.df file, and that data dumped with PROUTIL (binary dump) can be loaded into the database.

Please refer to Progress Documentation for more details: 

OpenEdge Development,  Basic Database Tools Database Tasks, “Dumping data definitions”
OpenEdge Deployment,  Managing ABL Applications R-code Features and Functions , Database CRCs and time stampstime stamps, “Database CRC changes that do not prevent r-code execution”

Solution: Deployment with the Three Database Rule (3 database deployment method):

To not make manual changes might sound impractical at first, but manual changes can be accomplished with the three database method:

  • This method introduces a third database in-between the Development and Production database.
  • This database is referred to as Basic database (or sometimes the deployment database).
  • The Basic database does not contain data, only application schema and has the same CRCs as the Production Database.
  • Make the database schema changes and compile application code using the logical name of the Production Database against the Basic Database.
  • Now the .df file can be distributed together with the new application code.
  • When the same df file is applied to the Production database(s), you are guaranteed to get the same CRC(s) as the Basic database, and the compiled code will run against it.

Three Database Rule Deployment:

  1. Make manual schema changes as required in the Development database.
  2. Create an incremental .df file from the Development database to the Basic database.
  3. Apply the delta.df file to the Basic database.
  4. Use the -ld parameter to connect the Basic database with the logical name of the Production database.
  5. Compile all affected source code against the Basic database.
  6. Deploy the new application code together with the delta.df file to the Production database.
  7. Apply the delta.df file to the Production database.
  8. The application runs against the Production database without CRC errors.
With this 3 database deployment method,  the Development database can be changed freely as developers add new application schema elements. During the development stage, new elements may even be deleted before they ever make it to production.

Every time a .df is produced from the Development database and applied to the Basic database, it is also applied to all Production database(s), and new compiled code for all affected programs against the Basic database is distributed to all Production sites as well.

Affected Programs include all those who use any objects whose CRCs have been changed.

Additional information for Progress Versions 8.x and 9.x:

Progress versions earlier than 9.x:
  • Include indices in the CRC calculation. 
  • Indexes of a table do not have their own CRCs, a change to an index is considered a change to the table.
  • Index changes affect the the table CRC.
  • Applications that use the modified table do not run without recompiling, even if the index is not used
Since Progress 9.x
  • Table and index CRCs are separate. Indexes have their own CRCs.  
  • Indexes can be added/dropped/modified without modifying the table CRC.
  • Only applications that use a changed index must be recompiled.
Impact of index changes on CRC values:

In earlier versions of Progress, indexes were included in the calculation of CRC values for a table. Whenever an index was added or deleted, the CRC value for the table changed. Without recompiling, an application that referenced the table, even if it did not use the index, would not execute.

To rebuild an index after it has been changed required the database to be off-line. The length of time the database was off-line depended on the which approach you used.

The approach that involved the least downtime required the index to be inactive:

The state of the index does not affect the CRC value.
The index is rebuilt in the background while the database is online, using PROUTIL IDXFIX.
However, the database must be taken offline in order to make the index active with PROUTIL IDXBUILD.
While the index is inactive, an application that references that index will not execute, even if the application has a matching CRC value.

This reduces flexibility in terms of determining when to take the database off-line to activate the indices.

The Progress 9.x Solution:

In Progress 9.x, indices were removed from the table's CRC calculation and given their own CRC values.
Once the applications are converted to Progress 9.x and the database is migrated to Progress 9.x, a change to the indices does not necessitate a recompilation of applications because the table's CRC value is unchanged.

Temporary and intermittently active indices can be added without changing the CRC and therefore without impact to applications that do not reference the indices. If indexes are referenced, at run time, the application index CRC values for the indices it references are compared to the index CRCs in the database. If there is a mismatch, the application does not run and an error is issued.

Fast schema update:

Version 8.2A introduced the Fast Schema Update for added fields. Fast Schema Update appends new fields to the end of the record, provided they do not have an initial value. This is done at run time as records are accessed by users, rather than in the off-line batch mode.

Each time a record is accessed, it is scanned to count the number of fields to verify that it has the correct number. If the field count is not correct for the  current version of the database, the database manager appends the missing fields to the end of the record.

CRC is not used for sequences. The r-code won’t be affected by changes to the sequence.

Progress 9.x introduces two changes to the schema update process:
  • Fast Schema Update is extended to deleted fields.
When a record is accessed at run time, a deleted field marker replaces the deleted field.
  • Schema versioning reduces the amount of time spent scanning records.
Workaround
Notes

References to Other Documentation: 

OpenEdge Deployment: Managing ABL Applications, Database CRCs and time stamps
OpenEdge Deployment: Managing ABL Applications, R-code CRCs and procedure integrity
OpenEdge Development: Basic Database Tools Database Tasks, “Dumping data definitions”
OpenEdge Deployment:  Managing ABL Applications R-code Features and Functions , Database CRCs and time stampstime stamps, “Database CRC changes that do not prevent r-code execution”

Progress Articles:

 What are the different types of CRC stored in the database's metaschema ?  
 What dictionary changes force recompiling in OpenEdge?  
 How to display the CRC values for tables & indexes in the database  

Last Modified Date11/20/2020 6:54 AM
Attachment 
Files
Disclaimer The origins of the information on this site may be internal or external to Progress Software Corporation (“Progress”). Progress Software Corporation makes all reasonable efforts to verify this information. However, the information provided is for your information only. Progress Software Corporation makes no explicit or implied claims to the validity of this information.

Any sample code provided on this site is not supported under any Progress support program or service. The sample code is provided on an "AS IS" basis. Progress makes no warranties, express or implied, and disclaims all implied warranties including, without limitation, the implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample code is borne by the user. In no event shall Progress, its employees, or anyone else involved in the creation, production, or delivery of the code be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample code, even if Progress has been advised of the possibility of such damages.