Error while upgrading secondary node of CS 7.1 in high availability mode

Hi,
we are facing issue while upgrade CS 7.1 to 7.3, the second content server node is unable to communicate with DB. Below is the error.

15:17:40,109 INFO [Thread-87] - Installer executes dm_rcs_copyfiles.ebs to get files from primary server
15:17:43,480 INFO [Thread-89] - The installer is creating folders for the selected repository.
15:17:43,480 INFO [Thread-89] - The installer stops service/process for the server dm_USLogistics1
15:17:43,480 INFO [Thread-89] - Installer updates server ini file.
15:17:43,480 INFO [Thread-89] - Installer creates necessary directory structure.
15:17:43,496 INFO [Thread-89] - The installer tests database connection.
15:17:43,558 ERROR [Thread-89] com.documentum.install.multinode.cfs.installanywhere.actions.DiWAServerUpgradeContentFileServer - Repository could not connect to the database. Detailed error:

com.documentum.install.shared.common.error.DiException: Repository could not connect to the database. Detailed error:

at com.documentum.install.multinode.cfs.common.services.DiServerConfigureCS.testServerConnection(DiServerConfigureCS.java:291)
at com.documentum.install.multinode.cfs.installanywhere.actions.DiWAServerUpgradeContentFileServer.setup(DiWAServerUpgradeContentFileServer.java:67)
at com.documentum.install.shared.installanywhere.actions.InstallWizardAction.install(InstallWizardAction.java:75)
at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
at com.zerog.ia.installer.util.GenericInstallPanel$2.run(Unknown Source)

can anyone guide any troubleshooting steps for this issue?

Comments

  • to add further we are using CFS program to upgrade the node 2

  • hi,

    can you double check whether you can connect to database via sqlplus or sqlserver developer studio tool?

    if work, please run dmdbtest to test db connection

    dmdbtest -docbase_name docbasename -init_file /xxxxxx/dba/config/****/server.ini

    Tony

    Tony Qiu
    Opentext ECD - SME

  • MYDCTM
    edited September 6, 2018 #4

    @hqiu said:
    hi,

    can you double check whether you can connect to database via sqlplus or sqlserver developer studio tool?

    if work, please run dmdbtest to test db connection

    dmdbtest -docbase_name docbasename -init_file /xxxxxx/dba/config/****/server.ini

    Tony

    Hi Tony,
    Thanks for the reply, I am able to proceed further in the installation. There was wrong aek.key name mentioned in the server.ini file. after mentioning correct name in the server.ini was able to proceed further in the installation. However, now another issue occurred at the last leg. please see below and guide on this.

    Installer log

    13:07:54,355 INFO [Thread-62] - The installer is updating windows registry.
    13:07:54,355 INFO [Thread-62] - The installer is configuring content-file server for repositoryUSLogistics1.
    13:07:57,359 INFO [Thread-62] - The installer is starting service for the repository USLogistics1.
    13:09:13,276 INFO [Thread-68] - Waiting for repository USLogistics1.sg2pedmsa14_dm_USLogistics1 to start up.
    13:09:59,898 INFO [Thread-35] com.documentum.fc.client.security.internal.CreateIdentityCredential - timed destroy thread destroyed credential now, this is expected behavior
    13:17:50,429 ERROR [Thread-68] com.documentum.install.server.installanywhere.actions.DiWAServerTestDocbaseConnection2 - Fail to connect to the repository. It could be that the server is in progress of starting. Press Yes, if you want to wait for it to start.
    com.documentum.install.shared.common.error.DiException: Fail to connect to the repository. It could be that the server is in progress of starting. Press Yes, if you want to wait for it to start.
    at com.documentum.install.server.installanywhere.actions.DiWAServerTestDocbaseConnection.setup(DiWAServerTestDocbaseConnection.java:116)
    at com.documentum.install.server.installanywhere.actions.DiWAServerTestDocbaseConnection2.setup(DiWAServerTestDocbaseConnection2.java:38)
    at com.documentum.install.shared.installanywhere.actions.InstallWizardAction.install(InstallWizardAction.java:75)
    at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
    at com.zerog.ia.installer.util.GenericInstallPanel$2.run(Unknown Source)

    docbase log

    2018-09-06T13:09:06.080000 1516[4928] 0000000000000000 TRACE LAUNCH: .\dm_agent_exec.exe -enable_ha_setup 1 -docbase_name USLogistics1.sg2tedmsa14_dm_USLogistics1 -docbase_owner svc-dmsadmin -sleep_duration 0 -parent_handle 7756
    dmExec::Launch(E:\Documentum\product\7.3\bin.\dm_agent_exec.exe -enable_ha_setup 1 -docbase_name USLogistics1.sg2pedmsa14_dm_USLogistics1 -docbase_owner svc-dmsadmin -sleep_duration 0 -parent_handle 7756)
    Working Directory(E:\Documentum\product\7.3\bin)
    Thu Sep 06 13:09:08 2018 [INFORMATION] [AGENTEXEC 720] Detected during program initialization: Version: 7.3.0000.0214 Win64
    Thu Sep 06 13:09:08 2018 [ERROR] [AGENTEXEC 720] Detected during program initialization: Command Failed: connect,USLogistics1.sg2pedmsa14_dm_USLogistics1,svc-dmsadmin,'',,,try_native_first, status: 0, with error message [DM_SESSION_E_CLIENT_AUTHENTICATION_FAILURE]error: "Failed to authenticate client. Please check server log for more detail."

    .
    Thu Sep 06 13:09:08 2018 [FATAL ERROR] [AGENTEXEC 720] Detected during program initialization: Command Failed: connect,USLogistics1.sg2pedmsa14_dm_USLogistics1,svc-dmsadmin,'',,,try_native_first, status: 0, with error message .
    The dm_agent_exec utility is exiting due to a fatal error.
    2018-09-06T13:10:09.074000 1516[1676] 0101b3f080000002 [DM_SESSION_W_RESTART_AGENT_EXEC]warning: "The agent exec program has stopped running. It will be restarted."

  • You may want to remove your dfc.keystore and let it recreate on startup

    Russell Kavanagh
    Documentum SME | Opentext

  • MYDCTM
    edited September 6, 2018 #6

    @rkavanagh said:
    You may want to remove your dfc.keystore and let it recreate on startup

    On every run of the installer it is creating a new keystore and till now it has created 8 keystores. I have approved them as privileged clients also.

  • This is log of dm_rcs_copyfiles_out.log

    2018-09-06 11:38:29 AM: DocbaseName=USLogistics1.USLogistics1: UserName=svc-dmsadmin: Len(Password)=10: TempDir=C:\Users\svc-dmsadmin\AppData\Local\Temp\687681.tmp\50d995bf-9e66-4702-b52b-29751039dd3c
    2018-09-06 11:38:29 AM: Step 1: Attempting to connect to docbase USLogistics1 as user svc-dmsadmin
    2018-09-06 11:38:30 AM: Failed to connect to docbase USLogistics1.USLogistics1 as user svc-dmsadmin

  • hi, you need to add dfc to trust list, not approve client. need to check following two option for privileges client:

    Enable trusted login
    Enable Trusted server privilege

    Tony

    Tony Qiu
    Opentext ECD - SME

  • @hqiu said:
    hi, you need to add dfc to trust list, not approve client. need to check following two option for privileges client:

    Enable trusted login
    Enable Trusted server privilege

    Tony

    Hi Tony,
    I just mentioned privileged client since Russell was asking about deleting keystore file. There is no problem with authentication. I have enabled authentication trace log. Below is the statement that I found for every attempt which I think is successful authentication.

    2018-09-06T14:45:43.761000 2236[1332] 0101b3f081288958 [AUTH] Final Auth Result=T

  • hi,

    can you use iapi from secondary machine to connect to USLogistics1.USLogistics1 as trust logon

    > iapi USLogistics1.USLogistics1

    Tony Qiu
    Opentext ECD - SME

  • Hi All,
    Thanks for your time on this thread..... Issue is resolved. There seems to be some cache issue which we are unsure of, after clearing Documentum cache and recreating keystore file issue got resolved.

  • Always delete cache whenever you restart anything Documentum related :)

  • @DCTM_Guru said:
    Always delete cache whenever you restart anything Documentum related :)

    Agree