TS won't start up

Hi all,

We are having a problem with TeamSite and I'm hoping someone here might be able to point me in the right direction as to where I need to look to troubleshoot this...

For seemingly no reason, all of our presentation templates started generating compile errors this morning. As our sys admin is on holidays, I SSH'd in and ran an "iwrest -a".

This made things worse; before I did that, the PT generation errors were the only problem. After the reset, TeamSite won't come back up. I really have no idea what's going on here.

I've attached some of the lines from servletd_out.log; I hope something in there is helpful.

Thank you kindly for your time!

Cheers,
Matt

TS 6.7.2 SP1 on RH Linux

Find more posts tagged with

Comments

Crsb

java.net.BindException: Address already in use:8080

Verify what process is using port 8080, is it teamsite tomcat instance? or something else?

Try shutting down teamsite /etc/init.d/iw.server stop, when it finishes stopping, check for connections on 8080 and make sure they are none. If a connection is still holding to 8080, check if it is teamsite or something else. kill the process if it is teamsite process. once there are no connections on 8080, start teamsite /etc/init.d/iw.server start.

admin.jpg

nipper

Well that does not seem to help a great deal. Complete restart would be on my list of todos.

BTW, is there a process that strips out setuid bits running on your server ?

Also check the iwserver and iwtrace logs after you restart and see if anything is helpful.

Finally, run df -h and post the output.

Since you are linux, you should not be hitting the null iwperl file that happens on DOS.

matty_mouse

Thanks Andy, Crsb,

A complete restart of the server has been done. The log I posted is from after that.

I'm sorry - I can't answer the question about the setuid process; I'm not the admin, and there are a lot of processes running right now (150+). Well, seems like a lot to me, anyways.

I did attach a couple of errors from iwtrace, and a tail of iwserver. Hoping those shed some more light on this. It's funny - this is our sys admin's first day of vacation. Great timing, haha!

Thanks again for any suggestions!

Cheers,
Matt

matty_mouse

Oh, forgot the df stats (I initially checked this before my first iwreset, as we've had problems in the past with the event subsystem filling up our disk so TS would crash/be unable to restart. Here they are:

Filesystem Size Used Avail Use% Mounted on
/dev/sda1 123G 18G 99G 16% /
/dev/sdb1 824G 319G 464G 41% /cms
none 8.0G 0 8.0G 0% /dev/shm
/dev/sda6 11G 58M 9.6G 1% /tmp
/dev/sda2 64G 4.0G 57G 7% /usr
/dev/sda3 64G 783M 60G 2% /var
none 824G 319G 464G 41% /iwmnt/default
none 824G 319G 464G 41% /iwmnt/iwadmin

linesizeissue.png

nipper

OK, those seem to have come up correctly

DO a df -h

Also type locate iw_cgi_wrapper.cgi

then cut & paste that output into

ls -l ....path/to/iw_cgi_wrapper.cgi

sample-data-clustering-live-map.jpg

matty_mouse

ls -l /cms/iw-home/httpd/iw-bin results:

Most files here have -rwxrwxr-x permissions, but the ones below do not. However, I thought I've seen these permissions in use before:

-rwsrwxr-x 1 root daemon 269128 Mar 20 2009 iwbsauth.cgi
-rwsrwxr-x 1 root daemon 239252 Mar 20 2009 iwbslogin.cgi
-rwsrwxr-x 1 root daemon 4387048 Mar 20 2009 iw_cgi_wrapper.cgi
-rwsrwxr-x 1 root daemon 4445188 Mar 20 2009 iwfsckcgi.cgi
-rwsrwxr-x 1 root daemon 4401020 Mar 20 2009 iwrcopy
-rwsrwxr-x 1 root daemon 262540 Mar 20 2009 iwrcopywrap.cgi
-rwsrwxr-x 1 root daemon 3328100 Mar 20 2009 iwrecover.cgi
-rwsrwxr-x 1 root daemon 4483348 Mar 20 2009 iwrlist
-rwsrwxr-x 1 root daemon 282004 Mar 20 2009 iwrlistwrap.cgi

Cheers,
Matt

nipper

OK the rws means read/write/sticky so the script/program will execute as the owner of the program (root) rather than the person who executed it (Matt)

It seems as if TeamSite is running correctly. If so, look at any other logs in local/logs/iwui that have changed and tail them.

mart2001

Great timing indeed! Are you sure that it is the teamsite services that is not up or is it the servletd that is not up?
You can try to do a netstat and find the pid of the program that is using port 8080. Can't recall what the switches are off the top of my head since they are different on windows, linux, and solaris.

Thanks Andy, Crsb,

A complete restart of the server has been done. The log I posted is from after that.

I'm sorry - I can't answer the question about the setuid process; I'm not the admin, and there are a lot of processes running right now (150+). Well, seems like a lot to me, anyways.

I did attach a couple of errors from iwtrace, and a tail of iwserver. Hoping those shed some more light on this. It's funny - this is our sys admin's first day of vacation. Great timing, haha!

Thanks again for any suggestions!

Cheers,
Matt

matty_mouse

Thanks mart2001 - there was nothing running on port 8080, so I guess it was servletd that was not running.

The problem has been resolved, however. My boss contacted A/I support, and it turns out our sys admin was not rotating the log files, so we had one that was almost 1.5 gigs in size.

All this time wasted troubleshooting something that should never have happened... oh well. Thanks all for your time and suggestions. Have a Merry Christmas and a Happy New Year!

Cheers,
Matt