I just ran into this same problem. The LAPI documents fail to provide a critical hint that is needed to make the NTLM authentication from Java work. I had to figure out the problem by decompiling the LLNTLM.class file from within LAPI.JAR.
Fortunately the fix is very easy.
NTLM within a Java client is done by using JNI to load the main Windows DLL for LAPI (LAPI_BASE.DLL). This is where the NTLM work is actually done. If the Java app can't load the LAPI_BASE.DLL then you get the "unknown authentication" problem.
JNI loads DLLs based on a system property that you can inspect: System.getProperty("java.library.path");. This is usually your windows path unless you change it on the java command line.
So the solution is to make sure that the Livelink LAPI DLLs are all present in the path somewhere. The authentication starts working immediately once it can load the DLL.
You can also define the java.library.path with a define in the command line ( -D ).
-- john