Feedback
Did this article resolve your question/issue?

   

Article

Replication synchronizes after failover using offline transition then fails error 1028

Information

 
TitleReplication synchronizes after failover using offline transition then fails error 1028
URL NameReplication-still-synchronized-after-failover-using-offline-transition
Article Number000173692
EnvironmentProduct: OpenEdge
Version: 11.0x, 11.1x, 11.2x, 11.3x, 11.4x, 11.5x, 11.6x, 11.70, 11.7.1, 11.7.2, 11.7.3
OS: All supported platforms
Question/Problem Description
Databases were transitioned off-line, appeared to synchronize, and then have issues with replication processing later
Replication synchronized after failover using offline transition : dsrutil -C transition
Databases re-started in their original primary-source and secondary-target roles, but failed to apply transaction notes
 
 
Steps to Reproduce
Clarifying Information
Manual transition against the online secondary target to transition the database to a source database succeeded:
dsrutil target -C transition agent1

The primary-target database was successfully synchronised from a copy of the new secondary-source database:
S: (13875) This database is enabled for OpenEdge Replication as a Target database.
T: (13875) This database is enabled for OpenEdge Replication as a Source database.

Initiate failback with controlled transition:
proshut source -by
proshut target -by
dsrutil source -C transition

S: (13958) Beginning Replication Transition operation: Updating database master block.
S: (13958) Beginning Replication Transition operation: Backing up database.
S: (18555) Database successfully transitioned from a Target database to a Source database.
S: (13551) The transition of this database has completed normally.

dsrutil target -C transition

T: (13958) Beginning Replication Transition operation: Opening database.
T: (13958) Beginning Replication Transition operation: Setting up transition.
T: (13958) Beginning Replication Transition operation: Switching AI Extents.
T: (13958) Beginning Replication Transition operation: Backing up database.
T: (18555) Database successfully transitioned from a Source database to a Target database.
T: (13551) The transition of this database has completed normally.
T: (451)   Mark backedup session begin for root on /dev/pts/2.
T: (13958) Beginning Replication Transition operation: Updating database master block.

Start source and target databases:

T: (333)   Multi-user session begin.
T: (13875) This database is enabled for OpenEdge Replication as a Target database.
S: (333)   Multi-user session begin.
S: (13875) This database is enabled for OpenEdge Replication as a Source database.

Synchronisation succeeds:
T: (10392) Database <target> is being replicated from database <source> on host <IP_source>.
T: (10489) Fathom Replication Agent agent1 is beginning Startup Synchronization at block 0.
S: (10507) The Fathom Replication Server has successfully connected to the Fathom Replication Agent agent1 on host <IP_target>.
S: (11251) The Replication Server successfully connected to all of its configured Agents.
S: (10439) The Fathom Replication Agent agent1 has been properly configured and the target database has been properly sourced.
S: (10508) Beginning Fathom Replication synchronization for the Fathom Replication Agent agent1.
T: (10668) The Source and Target databases are synchronized.
T: (10490) Fathom Replication Agent agent1 is beginning normal processing at block 9.
S: (10436) The source database and the target database on host <hostname> are synchronized.

Forward processing fails:
T: (1028)  SYSTEM ERROR: Rollforward Apply AI note, BKUPDCTR=19580112, note updctr=19580099.
T: (8999)  area 6, dbkey 192, record type 39
T: (10428) Function dsaAIApplyNote failed in rpNOT_ProcessNoteBlock with error -30000.
T: (10495) Block processing ended in error at block 17 in area 70.  Fathom Replication cannot continue.
T: (10504) Unexpected error -30000 returned to function rpAGT_AgentLoop.
S: (10699) The Fathom Replication Agent agent1 is requesting to be terminated.
S: (10700) The Fathom Replication Agent agent1 is being terminated.
S: (10702) All Fathom Replication Agents have been terminated or have ended.
S: (10698) The Fathom Replication Server will shutdown but the source database will remain active.
S: (10505) The Fathom Replication Server is ending.
T: (10506) The Fathom Replication Agent agent1 is ending.
 
Error Message(1028) SYSTEM ERROR: Rollforward Apply AI note, BKUPDCTR=19580112, note updctr=19580099
(10495) Block processing ended in error at block <blk> in area <area>. Fathom Replication cannot continue.
Defect/Enhancement NumberDefect PSC00363647 / OCTA-3947
Cause
Shutting down the source database and target databases does not ensure in all situations that the databases are entirely synchronized.
For example, AI block changes that are flushed to disk at shut down time may not be replicated to the target databases.

The syntax "dsrutil -C transition" does not support the ability to "fail over".

That syntax should only be used on target databases when a source database (or server) is lost (disaster recovery) then requires that the target(s) are reseeded from a backup of the secondary-source database.
Resolution
In OpenEdge 11.7.4 Service Pack and later, when Initiating failback with controlled transition, the primary-target will transition to a primary-source database, but the target database must then be reseeded from a backup of the primary-source database.
dsrutil source -C transition
probkup [online] source source_new.back -ReplTargetCreation

Fail Over/Fail back is always an on-line transition (both source and target(s) are on-line) and is executed only against the source while both the source and target databases remain online:
dsrutil <dbname> -C transition failover

For further information refer to Article  000096616, 11.7.4 Replication synchronisation failure after failback processing using controlled transition  
 
Workaround
Notes
Last Modified Date6/14/2019 1:25 PM
Attachment 
Files
Disclaimer The origins of the information on this site may be internal or external to Progress Software Corporation (“Progress”). Progress Software Corporation makes all reasonable efforts to verify this information. However, the information provided is for your information only. Progress Software Corporation makes no explicit or implied claims to the validity of this information.

Any sample code provided on this site is not supported under any Progress support program or service. The sample code is provided on an "AS IS" basis. Progress makes no warranties, express or implied, and disclaims all implied warranties including, without limitation, the implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample code is borne by the user. In no event shall Progress, its employees, or anyone else involved in the creation, production, or delivery of the code be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample code, even if Progress has been advised of the possibility of such damages.