What does the "Connection failure for host <host_name> port <port> transport <transport_name>. (9407)" mean ?
The error "Connection failure for host <host_name> port <port> transport <transport_name>. (9407)" is very generic, and it signifies that for some reason, the other endpoint of the connection can no longer be reached. This can apply to any socket connection made by the AppServer Agent, including, but not limited to:
- The connection to the owning Broker process
- A connection from the Agent to another AppServer Broker
- A connection from the Agent to a WebService
- For state-aware and state-reset operating modes, the connection to the client that the agent is currently bound to.
It can be caused by the other endpoint terminating without closing the connection cleanly, or because the network connection was interrupted.
Without knowing what the other endpoint is, and what that was doing at the time of the issue, it's impossible to identify the root cause of this error.
How to identify the endpoint ?
- What's the hostname/address reported in the error message?
- If host is "localhost" or 127.0.0.1, per IP protocol standards means the connection was within the same machine local machine. Note: Modern OS'es will shortcut traffic to this address so that it never reaches the actual network, so for this name/address the network itself does not need to be investigated.
- Otherwise, trace this back to specific machines or network segments.
- What's the port number reported in the error message?
This must be checked on the machine found in Step 1 above.
- If there is an OpenEdge installation, cross-reference with ubroker.properties & conmgr.properties files to see if this is in the defined port -minport -maxport range of a specific database/appserver/other component. If yes, check logs for those additional components for additional symptoms.
- Otherwise, use netstat -b or equivalent OS tools if end-point of connection can still be identified.
Note that these steps are not full-proof.
- Ports may be outside of any -minport -maxport range, as there are cases where the AVM will just claim any available port from the OS if it needs one. So-called ephemeral ports, may or may not be within a set range depending on OS:
- Netstat may fail to detect anything if the failing connection has been cleaned up by the OS network layers already.
- If the issue occurs frequently enough, repeated snapshots while the environment is still running can help identify this. But a certain level of uncertainty will remain.
What can be done to get more data out of the AppServer Agent reporting error 9407?
Add -debugalert to the AppServer Agents' startup parameters. This will make the Agent include the ABL trace for any error message that gets written to the log to assist in identifying if the error occurs while the Agent is running ABL code, and if yes, where in the code. That can help identify the nature of the other endpoint:
- If there is no ABL trace, the Agent is not running any ABL code at the time of the error, so it is likely receiving a request or sending the response. At this point, the endpoint is likely the AppServer Broker (for stateless/state-free operating modes) or the AppServer client (for state-aware/state-reset operating modes).
- If there is an ABL trace, the line referenced by the trace likely has a statement that would cause TCP activity - writing data to a socket, or something similar. Analysing the ABL code in detail can help determine the type of connection established, and with that the nature of the other endpoint.
- If the issue is reproducible, increase the logging level and add more log entry types for the Broker and Agent. Additionally, engage Network Specialists to add enhanced tracing with tools like Wireshark for example.