Can't create cluster Node (recv failed)

bmark

<p>Hello,</p>
<p> </p>
<p>I have problems creating a cluster node on the default cluster. I installed iHub 16 on a Windows Server 2012 environment. The iHub server itself is running well. Now I installed the system console, because I want to add an additional volume. System Console is reachable and seems to work. All necessary ports are open to all nic's</p>
<pre class="_prettyXprint">
TCP    0.0.0.0:8010           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8011           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8080           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8100           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8101           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8380           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8382           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8390           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8391           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8392           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8433           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8500           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8700           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8701           0.0.0.0:0              LISTENING
TCP    0.0.0.0:8709           0.0.0.0:0              LISTENING</pre>
<p>To add an additional volume first I have to add a cluster node.</p>
<p>But this is failing with error below. Firewall is turned off and the soap endpoint on port 8101 (see stacktracke) seems to be reachable.</p>
<p> </p>
<p>Also the dns name "ihubdemo.emea1.cis.trcloud" is resolvable by the dns</p>
<p> </p>
<p>Error:</p>
<pre class="_prettyXprint">
Software caused connection abort: recv failed</pre>
<p>In the stdout.log I can find this:</p>
<pre class="_prettyXprint">
ERROR 15 Feb 2017 10:11:16,245 [http-bio-8500-exec-1] adapter.BaseAdapter - class com.actuate.umc.ihub.provisioning.adapter.GetServiceListAdapter===RetryCount: 5 Hostname: iHubDemo.emea1.cis.trcloud Exception: java.net.SocketException: Software caused connection abort: recv failed (?:?)
ERROR 15 Feb 2017 10:11:24,403 [http-bio-8500-exec-1] adapter.BaseAdapter -
class com.actuate.umc.ihub.provisioning.adapter.GetServiceListAdapter::execute()
hostname: iHubDemo.emea1.cis.trcloud
endpoint: https://iHubDemo.emea1.cis.trcloud:8101/ihub
RetryCount: 1
nextHost: null
AxisFault
 faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
 faultSubcode:
 faultString: java.net.SocketException: Software caused connection abort: recv failed
 faultActor:
 faultNode:
 faultDetail:
    {http://xml.apache.org/axis/}stackTrace:java.net.SocketException: Software caused connection abort: recv failed
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:911)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
    at org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:583)
    at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:143)
    at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
    at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
    at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
    at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
    at org.apache.axis.client.Call.invokeEngine(Call.java:2765)
    at org.apache.axis.client.Call.invoke(Call.java:2748)
    at org.apache.axis.client.Call.invoke(Call.java:2424)
    at org.apache.axis.client.Call.invoke(Call.java:2347)
    at org.apache.axis.client.Call.invoke(Call.java:1804)
    at com.actuate.schemas.provisioning.ProvisioningPmdSoapBindingStub.systemLogin(ProvisioningPmdSoapBindingStub.java:2866)
    at com.actuate.umc.ihub.provisioning.adapter.SystemLoginAdapter.d(Unknown Source)
    at com.actuate.umc.ihub.provisioning.adapter.SystemLoginAdapter.execute(Unknown Source)
    at com.actuate.umc.ihub.provisioning.adapter.BaseAdapter.a(Unknown Source)
    at com.actuate.umc.ihub.provisioning.adapter.BaseAdapter.execute(Unknown Source)
    at com.actuate.umc.mc.ajax.handler.ServiceHelper.a(Unknown Source)
    at com.actuate.umc.mc.ajax.handler.ConfigureServer.b(Unknown Source)
    at com.actuate.umc.mc.ajax.handler.ConfigureServer.a(Unknown Source)
    at com.actuate.umc.mc.ajax.handler.ConfigureServer.processServer(Unknown Source)
    at com.actuate.umc.mc.ajax.handler.ConfigureServer.getResult(Unknown Source)
    at com.actuate.umc.mc.ajax.handler.AbstractHandler.processRequest(Unknown Source)
    at com.actuate.umc.mc.ajax.Processor.processRequest(Unknown Source)
    at com.actuate.umc.mc.servlet.AjaxServlet.handleRequest(Unknown Source)
    at com.actuate.umc.mc.servlet.AjaxServlet.doPost(Unknown Source)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

    {http://xml.apache.org/axis/}hostname:iHubDemo
 (?:?)
ERROR 15 Feb 2017 10:11:25,403 [http-bio-8500-exec-1] adapter.BaseAdapter - class com.actuate.umc.ihub.provisioning.adapter.GetServiceListAdapter===RetryCount: 5 Hostname: iHubDemo.emea1.cis.trcloud Exception: java.net.SocketException: Software caused connection abort: recv failed (?:?)</pre>
<p>Thank you for your help</p>
<p> </p>
<p>Benjamin Mark</p>

RSiegler

<p>bmark,</p>
<p> </p>
<p>Can you upload the contents of the ihubd and ihubc log files?</p>
<p>I would be interested in reviewing to see if there are any indicators in those files as to what may be occurring.</p>

bmark

<p>Hello RSiegler,</p>
<p> </p>
<p>here are the files.</p>
<p> </p>
<p>Thank you</p>
<p> </p>
<p>Benjamin</p>

RSiegler

<p>Benjamin,</p>
<p> </p>
<p>Looking at the ihubd log file, there are indicators that the system is being overwhelmed with SOAP messages, or there is a critical resource limit being met (CPU/RAM).</p>
<p> </p>
<p>How much available RAM is there when the iHub is completely offline, and what is the CPU load of the system as a whole during the time when the iHub is running?</p>