SSL configured Content Server could not connect to SSL configured Docbroker

Leendert

Self signed certificates have been generated as described in OpenText documentation. When Content Server is started and is trying to connect to the SSL configured Docbroker errors are reported in the Docbroker log file as shown hereafter:

2019-06-04T15:57:05.732460 [DM_DOCBROKER_I_LISTENING]info: "The Docbroker is listening on network address: (INET_ADDR: family: 2, port: 1494, host: lrv1448r (10.192.225.140, 8ce1c00a))"
Using ciphers AES128-SHA
[DM_SERVER_SSL_TRACE] R_SSL_get_error() returned 1 on R_SSL_do_handshake() returned code -1 in dm_nl_ssl_accept().
[DM_SERVER_SSL_TRACE] Error description is : error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol.
[DM_SERVER_SSL_TRACE] R_SSL_do_handshake() failed even after retry. R_SSL_get_error() returned 1 on R_SSL_do_handshake() returned code 2 in dm_nl_ssl_accept().
2019-06-04T15:57:58.961553 [DM_DOCBROKER_W_SSL_HANDSHAKE_FAILED]warning: "Failed to establish a secure connection. Secure port: 1494. Client address: 10.192.225.140:53266. Check that clients and servers have the correct docbroker port configuration."

However, the validation to see if Docbroker will return certificates by issuing command "openssl s_client -showcerts -debug -connect lrv1448r.europe.intranet:1494" works fine as is shown below. The cipher used is AES128-SHA (the only one that could work according to OpenText). Seems as if the Content Server is using the wrong protocol but I don't see any steering parameters defined to chamge protocol.
Who knows the solution to get this all working?

CONNECTED(00000003)
write to 0x159d880 [0x15c89f0] (247 bytes => 247 (0xF7))
0000 - 16 03 01 00 f2 01 00 00-ee 03 03 5c f6 7c c4 83 ............|..
0010 - 22 f4 1a 36 12 2a 07 95-86 d2 8c 1b 33 3a 05 6a "..6.......3:.j
0020 - 4b 53 92 fa 81 57 09 91-15 03 64 00 00 84 c0 30 KS...W....d....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a3 00 9f 00 6b .,.(.$.........k
etc.etc.
00e0 - 04 01 04 02 04 03 03 01-03 02 03 03 02 01 02 02 ................
00f0 - 02 03 00 0f 00 01 01 .......
read from 0x159d880 [0x15cdf50] (7 bytes => 7 (0x7))
0000 - 16 03 03 00 51 02 ....Q.
0007 - <SPACES/NULS>
read from 0x159d880 [0x15cdf5a] (79 bytes => 79 (0x4F))
0000 - 00 4d 03 03 5c f6 7c c4-81 a7 d0 d3 94 9e b1 55 .M...|........U
0010 - 4b b8 78 17 c5 48 e6 09-be 77 40 3e 79 ae 29 08 K.x..H...w@>y.).
0020 - fe d6 d7 c0 20 f4 38 ad-6f b3 87 7f a2 32 db 6d .... .8.o....2.m
0030 - 78 d0 eb 26 2f 9b 9e 8f-52 56 44 27 5c 6c ac 6c x..&/...RVD'\l.l
0040 - e8 43 f4 77 c1 00 2f 00-00 05 ff 01 00 01 .C.w../.......
004f - <SPACES/NULS>
read from 0x159d880 [0x15cdf53] (5 bytes => 5 (0x5))
0000 - 16 03 03 03 f9 .....
read from 0x159d880 [0x15cdf58] (1017 bytes => 1017 (0x3F9))
0000 - 0b 00 03 f5 00 03 f2 00-03 ef 30 82 03 eb 30 82 ..........0...0.
0010 - 02 d3 a0 03 02 01 02 02-09 00 ea 26 bc 3f 5a 7c ...........&.?Z|
0020 - e6 53 30 0d 06 09 2a 86-48 86 f7 0d 01 01 0b 05 .S0....H.......
etc.etc.etc.
03d0 - 38 13 a1 85 d2 62 d4 ca-6a d6 99 95 ee 36 fc 7a 8....b..j....6.z
03e0 - c5 ab 6a b4 c6 71 79 82-eb 81 c3 c3 e0 38 31 95 ..j..qy......81.
03f0 - 54 d0 e3 38 dc 07 56 7b-bd T..8..V{.
depth=0 C = NL, ST = NH, L = Amsterdam, O = OIB GS, OU = Tech/GS/Corporate Applications, CN = sdecsd-p.europe.intranet
verify error:num=18:self signed certificate
verify return:1
depth=0 C = NL, ST = NH, L = Amsterdam, O = OIB GS, OU = Tech/GS/Corporate Applications, CN = sdecsd-p.europe.intranet
verify return:1
read from 0x159d880 [0x15cdf53] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 04 .....
read from 0x159d880 [0x15cdf58] (4 bytes => 4 (0x4))
0000 - 0e .
0004 - <SPACES/NULS>
write to 0x159d880 [0x15d8400] (267 bytes => 267 (0x10B))
0000 - 16 03 03 01 06 10 00 01-02 01 00 80 c7 13 7a 49 ..............zI
0010 - a8 4d 18 04 30 f2 ea 1d-7d 15 af 3f b7 5d 3d de .M..0...}..?.]=.
0020 - 25 2e 8a 30 db e3 86 eb-16 dd e8 d7 97 ef 72 f9 %..0..........r.
etc.etc.
write to 0x159d880 [0x15d8400] (6 bytes => 6 (0x6))
0000 - 14 03 03 00 01 01 ......
write to 0x159d880 [0x15d8400] (69 bytes => 69 (0x45))
0000 - 16 03 03 00 40 28 ca db-f4 e4 49 1f 57 9e c2 6a ....@(....I.W..j
0010 - ec 38 bc f6 9c 0f ec c4-ae 1c ae 43 4e 11 85 9f .8.........CN...
0020 - f7 0e 81 3b 64 0f e0 cd-a6 07 df 35 93 06 4f 09 ...;d......5..O.
0030 - aa 08 a8 9f 19 7e 28 cc-4b f0 08 7f e0 76 79 0d .....~(.K....vy.
0040 - ed 08 1a cd ed .....
read from 0x159d880 [0x15cdf53] (5 bytes => 5 (0x5))
0000 - 14 03 03 00 01 .....
read from 0x159d880 [0x15cdf58] (1 bytes => 1 (0x1))
0000 - 01 .
read from 0x159d880 [0x15cdf53] (5 bytes => 5 (0x5))

read from 0x159d880 [0x15cdf58] (64 bytes => 64 (0x40))
0000 - cd a1 11 8e c7 4f de 10-77 76 b3 30 ae 87 ec 1f .....O..wv.0....
0010 - b9 d9 7c 12 91 d0 59 02-e9 dc af 1b d2 f2 34 14 ..|...Y.......4.
0020 - f6 e3 3d 9f 09 db ff a5-7b 5d 24 78 65 32 c1 22 ..=.....{]$xe2."

0030 - 48 ec 30 20 6a 04 86 49-31 cd d4 ea 60 80 39 80 H.0 j..I1...`.9.

Certificate chain
0 s:/C=NL/ST=NH/L=Amsterdam/O=OIB GS/OU=Tech/GS/Corporate Applications/CN=sdecsd-p.europe.intranet
i:/C=NL/ST=NH/L=Amsterdam/O=OIB GS/OU=Tech/GS/Corporate Applications/CN=sdecsd-p.europe.intranet
-----BEGIN CERTIFICATE-----
MIID6zCCAtOgAwIBAgIJAOomvD9afOZTMA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD
VQQGEwJOTDELMAkGA1UECAwCTkgxEjAQBgNVBAcMCUFtc3RlcmRhbTEPMA0GA1UE
etc.etc.
yh9azlGqOBOhhdJi1Mpq1pmV7jb8esWrarTGcXmC64HDw+A4MZVU0OM43AdWe70=

-----END CERTIFICATE-----

Server certificate
subject=/C=NL/ST=NH/L=Amsterdam/O=OIB GS/OU=Tech/GS/Corporate Applications/CN=sdecsd-p.europe.intranet

issuer=/C=NL/ST=NH/L=Amsterdam/O=OIB GS/OU=Tech/GS/Corporate Applications/CN=sdecsd-p.europe.intranet

No client certificate CA names sent

SSL handshake has read 1192 bytes and written 589 bytes

New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : AES128-SHA
Session-ID: F438AD6FB3877FA232DB6D78D0EB262F9B9E8F525644275C6CAC6CE843F477C1
Session-ID-ctx:
Master-Key: 122FA85B36D2B40767BE7EC877F41DE7E590DB7876C0E0855EB76051056DC04146870CF5E921820DE87E8A55EDDCB504
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1559657668
Timeout : 300 (sec)

Verify return code: 18 (self signed certificate)

^[[A
write to 0x159d880 [0x15d24a3] (53 bytes => 53 (0x35))
0000 - 17 03 03 00 30 62 33 b7-cf 4d 57 a1 7a f5 1e 94 ....0b3..MW.z...
0010 - c3 27 8b 4d bd 6b 8d c3-f4 af 49 51 cb 90 44 79 .'.M.k....IQ..Dy
0020 - 67 8b 51 0d 34 db 99 d3-d0 d0 7d 0d b4 50 d4 2d g.Q.4.....}..P.-
0030 - 8e 9a ff 0f c7 .....
read from 0x159d880 [0x15cdf53] (5 bytes => 0 (0x0))
read:errno=0
write to 0x159d880 [0x15d24a3] (53 bytes => 53 (0x35))
0000 - 15 03 03 00 30 ca 7e 7d-71 dd ef 48 0d 7d 90 a1 ....0.~}q..H.}..
0010 - 28 7f df 79 b2 af 7c db-7e 05 7b d3 58 e0 a1 f2 (..y..|.~.{.X...
0020 - f3 78 b5 ee 41 6e ab bb-26 ff 21 f3 cd 73 2a 52 .x..An..&.!..s*R
0030 - 47 4e e0 cb 50

Leendert

And Documentum version is 7.2, running on Linux machines.

rkavanagh

Hi, in your server.ini, did you place the ssl information in the Server Startup section and not in the docbroker section?
That is a common issue with the server not connecting.
The following entries should be above the docbroker projections:
keystore_file=server.p12
keystore_pwd_file=server.pwd
truststore_file=server-trust.p7b

And the files need to be in the Documentum/dba/secure folder.

Leendert

Hello Russel, thank you for the comment. I surely added those entries to the server.ini file, see fragment from server.ini below (the truststore file contains the Docbroker certificate).

........
crypto_mode = AES256_RSA1024_SHA256
crypto_keystore = Local
crypto_keyname = aek.key
crypto_lockbox=lockbox.lb

Above values cannot be changed once docbase is created

#

SSL Required configuration entries

keystore_file=/appl/dadctm/dba/secure/server.p12
keystore_pwd_file=/appl/dadctm/dba/secure/server.pwd

truststore_file=/appl/dadctm/dba/secure/server-trust.p7b

cipherlist=AES128-SHA
#
#
[DOCBROKER_PROJECTION_TARGET]
host = lrv1448r.europe.intranet
port = 1493
................