nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Teamsite 6.1, GUI Does Not Restart (Solaris 9)

jimk351c

Our new installation of Teamsite 6.1 started just fine for the first couple of re-boots. Now, the GUI does not come up and we get the following error:

iwstart: Setting LD_PRELOAD to work around Sun bug 5075525
iwstart: Spawning iwserver.sol (trace file: /var/adm/iwtrace.log)...
iwstart: Waiting up to 60 seconds for iwserver to come up... READY
iwmount: Mounting /iwmnt/default...
iwmount: Mounting /.iwmnt/default...
iwmount: Mounted /iwmnt/default and /.iwmnt/default.
Interwoven OpenAPI starting ...
usage:
nohup command [argument ...]
nohup -p [-Fa] pid [pid ...]
nohup -g [-Fa] pgid [pgid ...]
ERROR: Failed to start. Check /var/adm/iwui/openapi.log

Here is the log entry for openapi.log:

iwopenapiboot: ---------- STARTING: Fri Nov 12 14:46:02 GMT 2004
iwopenapiboot: ---------- STOPPING: Sun Nov 14 02:53:07 GMT 2004
iwopenapiboot: ---------- STOPPED: Sun Nov 14 02:53:07 GMT 2004
iwopenapiboot: ---------- STARTING: Sun Nov 14 02:55:28 GMT 2004

I have a trouble ticked into Support. Any ideas on how to proceed?

TIA.

Senior IT Specialist, The Library Of Congress
jika@loc.gov

Find more posts tagged with

Comments

Migrateduser

Hi,
Can you try running the shell script "<iw-home>/private/bin/iwopenapiboot start" independently and see whether openapi starts up? If that fails, can you see if "<iw-home>/iwopenapi/iwopenapi" and see if that starts up? If you can using any of these methods, there might be a problem with one of the scripts that are used for restarting TeamSite services. Please note that <iw-home> refers to the location of your TeamSite installation.

Best Regards,
Narendra

jimk351c

Sorry, I have been away on Paternity leave :-)

I just tried this:

private/bin/iwopenapiboot start

and received the following error:

private/bin/iwopenapiboot: /var/adm/iwui/openapi.log: cannot create

Then I tried this:

iwopenapi/iwopenapi

and it appears to start fine. Now the following process is running:

iwadmin 15840 15795 0 09:03:21 pts/1 0:03 iwopenapi -server -Xss4000000 -Djava.library.path=/iw-home/iwopenapi -Djava.rmi

Yet, the GUI still doesn't start. So, I re-tried starting iw server and now I get the following when I try to start the GUI:

The Interwoven Web Daemon was unable to contact the Interwoven Servlet Engine. The Interwoven Servlet Engine may be down, or running on an unexpected host or port.

Please contact your system administrator.

but I know I'm going to the proper port.

Senior IT Specialist, The Library Of Congress
jika@loc.gov

Migrateduser

Hi,
It seems as though there are insufficient permissions to create /var/adm/iwui/openapi.log. Can you create a file in that location named "openapi.log" and give write permissions to everybody, and see if "private/bin/iwopenapiboot start" works? If that works, I am pretty sure that the main startup script for Interwoven services will also work.

Best Regards,
Narendra

jimk351c

The file already exists:

$ pwd
/var/adm/iwui
$ ls -l openapi.log
-rw-rw-r-- 1 iwui nobody 10593 Nov 29 13:15 openapi.log

Are you recommending I open up "writing" to that file to "world"?

Thanks.

Senior IT Specialist, The Library Of Congress
jika@loc.gov

Adam Stoller

When you ran 'private/bin/iwopenapiboot start' - who did you run it as? root? sudo? or yourself?

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

jimk351c

My mistake, I was running as myself. I just re-ran as root and got the following:

Interwoven OpenAPI starting ...
usage:
nohup command [argument ...]
nohup -p [-Fa] pid [pid ...]
nohup -g [-Fa] pgid [pgid ...]
java.rmi.server.ExportException: Port already in use: 1099; nested exception is
java.net.BindException: Address already in use
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:447)
at java.net.ServerSocket.<init>(ServerSocket.java:165)
at java.net.ServerSocket.<init>(ServerSocket.java:77)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createServerSocket(RM)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createServerSocket(RM)
at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:5)
at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:217)
at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:17)
at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:314)
at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:114)
at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:1)
at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:90)
at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:76)
at sun.rmi.registry.RegistryImpl.main(RegistryImpl.java:315)

ERROR: Failed to start. Check /var/adm/iwui/openapi.log

Here is the error that it creates in the log file:

Catalina.stop: java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:350)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:137)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:124)
at java.net.Socket.<init>(Socket.java:268)
at java.net.Socket.<init>(Socket.java:95)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:581)
at org.apache.catalina.startup.Catalina.execute(Catalina.java:402)
at org.apache.catalina.startup.Catalina.process(Catalina.java:180)
at java.lang.reflect.Method.invoke(Native Method)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:203)

Senior IT Specialist, The Library Of Congress
jika@loc.gov

Migrateduser

Can you use "lsof" to find out what program is using the port 1099? OpenAPI needs the port 1099 to start RMI registry (rmiregistry). If that port is already in use by some other program you need, you can override the default port 1099 in <iw-home>/iwopenapi/openapi.cfg. Uncomment the entry - "interwoven.api.locator: <IWRMINamingServiceLocator><localhostPort>1099</localhostPort></IWRMINamingServiceLocator>" and the change the value from 1099 to some other open port.

Best Regards,
Narendra

jimk351c

There is nothing else running on the port. See attached.

Senior IT Specialist, The Library Of Congress
jika@loc.gov

Migrateduser

I am running out of ideas. I think your best option now is to pursue this with Interwoven support.

Best Regards,
Narendra

jimk351c

Thanks for your help. I have been discussing the situation with IW support. They have no resolution yet.

Senior IT Specialist, The Library Of Congress
jika@loc.gov

jimk351c

OK, here is the latest:

I shut down iw.server (iw.server stop). Next, I started open api manually (pbrun iwopenapiboot start &). Next, I started iw.server (iw.server start). As usual, I got a failed to start message. But, when I go to the site, I actually get an error message now, instead of connection refused I get this: (The Interwoven Web Daemon was unable to contact the Interwoven Servlet Engine. The Interwoven Servlet Engine may be down, or running on an unexpected host or port. Please contact your system administrator. )

Here is the failed start message for iw.server:
Interwoven TeamSite starting ...
ERROR: Failed to start. Check /var/adm/iwserver.log

contents of iwserver.log:
iw.server: ---------- STARTING: Wed Dec 1 13:18:11 GMT 2004
iwetcboot: Running iwstart /iw-store in /iw-home
iwstart: The events log file is /var/adm/iwevents.log
iwstart: resetting wfs queue state ... done
iwstart: Resetting aborted state of kernel queue
iwstart: checking for standard services
iwstart: nfsd running
iwstart: mountd running
iwstart: Starting iwprocdstart script (log file: /var/adm/iwprocessd.log)... DONE
iwstart: Setting LD_PRELOAD to work around Sun bug 5075525
iwstart: Spawning iwserver.sol (trace file: /var/adm/iwtrace.log)...
iwstart: Waiting up to 60 seconds for iwserver to come up... READY
iwmount: Mounting /iwmnt/default...
iwmount: Mounting /.iwmnt/default...
iwmount: Mounted /iwmnt/default and /.iwmnt/default.
Interwoven Web Daemon starting ...
Interwoven Proxy Server starting ...
Interwoven Servlet Engine starting ...
ERROR: Failed to start Interwoven Servlet Engine. Check /var/adm/iwui/servletd.log

Contents of servletd.log:
Catalina.stop: java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:350)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:137)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:124)
at java.net.Socket.<init>(Socket.java:268)
at java.net.Socket.<init>(Socket.java:95)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:581)
at org.apache.catalina.startup.Catalina.execute(Catalina.java:402)
at org.apache.catalina.startup.Catalina.process(Catalina.java:180)
at java.lang.reflect.Method.invoke(Native Method)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:203)

$ lsof | grep 1099
java 5397 root 5u IPv4 0x300072104c0 0t0 TCP *:1099 (LISTEN)

But, if I do a shutdown (iw.server stop)
I get this error:
Interwoven TeamSite stopping. This may take a while...
ERROR: iwuiboot Error stopping TeamSite. Check /var/adm/iwserver.log

contents of iwserver.log:
iw.server: ---------- STOPPING: Wed Dec 1 13:31:33 GMT 2004
Interwoven Servlet Engine stopping ...
ERROR: Failed to stop Interwoven Servlet Engine. Check /var/adm/iwui/servletd.lo
g
Interwoven Proxy Server stopping ...
Interwoven Web Daemon stopping ...
/iw-home/iw-webd/bin/iw.webd stop: iwwebd stopped
iwuiboot: ---------- STOPPED: Wed Dec 1 13:31:35 GMT 2004
Interwoven OpenAPI stopping ...
iwserver: 6.1.0 Build 39268 Interwoven 20040819
iwetcboot: Writing unsaved changes to disk...
All stores frozen for 86400 seconds.
iwetcboot: Done saving changes.
iwmount: Unmounting /iwmnt/default...
iwmount: Unmounting /.iwmnt/default...
iwetcboot: Running iwstop ...
iwstop: Looking for the wfs kernel module id... found 137
iwstop: aborting the current queue... done
iwstop: waiting for the server to shutdown... done
iw.server: ---------- STOPPED: Wed Dec 1 13:31:39 GMT 2004

and, when I do this:
lsof | grep 1099

nothing is running.

IW support is escalating the case.........

Senior IT Specialist, The Library Of Congress
jika@loc.gov

nipper

Have you bounced the box ?

We run 552 on Solaris 2.8. Sometimes iwreset -a fails & servletd will not restart. Restarting TS does not take care & bouncing the box is required.

Andy

jimk351c

Yea, we have tried bouncing. Thanks...

Senior IT Specialist, The Library Of Congress
jika@loc.gov

nipper

Just a thought, there is a KB article that mentions iwwebd will not start when access_log is > 2 GB. Can you check that size ?

jimk351c

-rw-r--r-- 1 root root 6716617 Dec 1 15:43 access_log

Not surprising since this is a brand new install. Thanks...

Senior IT Specialist, The Library Of Congress
jika@loc.gov

nipper

This is a new install, has the GUI ever come up ?

Andy

jimk351c

Yes. Interwoven came out and installed and everything worked through several re-boots.

Senior IT Specialist, The Library Of Congress
jika@loc.gov

nipper

Any networking changes ? IP/hostname/domain ? Install any other products/patches in the meantime (like MetaTagger) ?

May need to rerun the GUI install piece, but I would wait to see what support says.

Andy

jimk351c

No, no changes.

Is it possible to only install the GUI portion without re-installing the underlying server?

Thanks.

Senior IT Specialist, The Library Of Congress
jika@loc.gov

nipper

Yes, not for the faint of heart but since it is not working anyway....

Note: THIS IS FOR 5.5.2, I do not have 6.1 loaded to check how it is done.

Peruse the file: $iwhome/install/iwinstall - look for the part about Setup Interwoven Web Daemon

There should be 2 scripts run in there. One (on 552) is install_webd.ipl with a lot of parms. You can use the
log file to determine the appropriate parameters. The other run is install_webapps.ipl which will unpack
and build the GUI.

Again, this is how it works on 5.5.2. If I have a chance I will take a quick look at 6.1.

HTH

Andy

jimk351c

Interesting. I will talk to tech support about the possibility of doing this.

Thanks.

Senior IT Specialist, The Library Of Congress
jika@loc.gov

jimk351c

Good news folks, tech support was able to figure it out. We have an application called "powerbroker" installed on our server, which is used to grant SU privelages to users. Powerbroker installs its owns version of SU which is not compatible with Teamsite. The trick is to replace Powerbrokers SU with that of your UX OS.

We're back up!

Thanks to everyone for their input and to support for figuring out this issue.

Senior IT Specialist, The Library Of Congress
jika@loc.gov