Article

What does the information in error 9407 found in the appserver log signify?

« Go Back

Information

 
Article Number000011630
EnvironmentProduct: Progress
Version: 9.x
Product: OpenEdge
Version: 10.x, 11.x
OS: All supported platforms
Question/Problem Description
What does the port number and hostname in error 9407 signify in an Appserver.server.log ?
Connection failure for host <host_name> port <port> transport <transport_name>. (9407)

Where does the port being referenced in error 9407 reside in the network communication?
Is the port number being referenced in error 9407 user assigned?
Which machine is the IP-address in error 9407 referring to? 

How to identify connection endpoints for error 9407
Steps to Reproduce
Clarifying Information
Error MessageConnection failure for host <host_name> port <port> transport <transport_name>. (9407)
Defect/Enhancement Number
Cause
Resolution

What does the  "Connection failure for host <host_name> port <port> transport <transport_name>. (9407)"  mean ?

The error "Connection failure for host <host_name> port <port> transport <transport_name>. (9407)" is very generic, and it signifies that for some reason, the other endpoint of the connection can no longer be reached. This can apply to any socket connection made by the AppServer Agent, including, but not limited to:

  • The connection to the owning Broker process
  • A connection from the Agent to another AppServer Broker
  • A connection from the Agent to a WebService
  • For state-aware and state-reset operating modes, the connection to the client that the agent is currently bound to.

It can be caused by the other endpoint terminating without closing the connection cleanly, or because the network connection was interrupted.

Without knowing what the other endpoint is, and what that was doing at the time of the issue, it's impossible to identify the root cause of this error.

How to identify the endpoint ?

  1. What's the hostname/address reported in the error message?
  • If host is "localhost" or 127.0.0.1, per IP protocol standards means the connection was within the same machine local machine. Note: Modern OS'es will shortcut traffic to this address so that it never reaches the actual network, so for this name/address the network itself does not need to be investigated.
  • Otherwise, trace this back to specific machines or network segments.
  1. What's the port number reported in the error message? 

This must be checked on the machine found in Step 1 above.

  • If there is an OpenEdge installation, cross-reference with ubroker.properties & conmgr.properties files to see if this is in the defined port -minport -maxport range of a specific database/appserver/other component. If yes, check logs for those additional components for additional symptoms.
  • Otherwise, use netstat -b or equivalent OS tools if end-point of connection can still be identified.

Note that these steps are not full-proof. 

  • Ports may be outside of any -minport -maxport range, as there are cases where the AVM will just claim any available port from the OS if it needs one. So-called ephemeral ports, may or may not be within a set range depending on OS:
  • Netstat may fail to detect anything if the failing connection has been cleaned up by the OS network layers already. 
  • If the issue occurs frequently enough, repeated snapshots while the environment is still running can help identify this. But a certain level of uncertainty will remain. 

What can be done to get more data out of the AppServer Agent reporting error 9407? 

Add -debugalert to the AppServer Agents' startup parameters. This will make the Agent include the ABL trace for any error message that gets written to the log to assist in identifying if the error occurs while the Agent is running ABL code, and if yes, where in the code.  That can help identify the nature of the other endpoint:

  • If there is no ABL trace, the Agent is not running any ABL code at the time of the error, so it is likely receiving a request or sending the response. At this point, the endpoint is likely the AppServer Broker (for stateless/state-free operating modes) or the AppServer client (for state-aware/state-reset operating modes).
  • If there is an ABL trace, the line referenced by the trace likely has a statement that would cause TCP activity - writing data to a socket, or something similar. Analysing the ABL code in detail can help determine the type of connection established, and with that the nature of the other endpoint.
  • If the issue is reproducible, increase the logging level and add more log entry types for the Broker and Agent. Additionally, engage Network Specialists to add enhanced tracing with tools like Wireshark for example.

    brkrLogEntryTypes=UBroker.Basic,Ubroker.ClientMsgStream, Ubroker.ServerMsgStream,Ubroker.Debug,Ubroker.UBNET
    brkrLoggingLevel=4

    srvrLogEntryTypes=4GLTrace,UBNET,UBS,UBC,ASPlumbing,DB.Connects 
    srvrLoggingLevel=4 
Workaround
Notes
Attachment 
Last Modified Date1/22/2018 9:14 AM
Disclaimer

The origins of the information on this site may be internal or external to Progress Software Corporation (“Progress”). Progress Software Corporation makes all reasonable efforts to verify this information. However, the information provided is for your information only. Progress Software Corporation makes no explicit or implied claims to the validity of this information.

Any sample code provided on this site is not supported under any Progress support program or service. The sample code is provided on an "AS IS" basis. Progress makes no warranties, express or implied, and disclaims all implied warranties including, without limitation, the implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample code is borne by the user. In no event shall Progress, its employees, or anyone else involved in the creation, production, or delivery of the code be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample code, even if Progress has been advised of the possibility of such damages.



Feedback
 
Was this article helpful?

   

Your feedback is appreciated.

Please tell us how we can make this article more useful. Please provide us a way to contact you, should we need clarification on the feedback provided or if you need further assistance.

Characters Remaining: 1025