Problem with apache all of a sudden

All,

I am experiencing an issue all of a sudden where apache is not starting up after a reboot because the port is already in use. Here is the scenario:

1) I issue the TeamSite shutdown command (iw.server stop) and the system shuts down properly.
2) I issue the apache shut down command (apachectl stop) and it shuts down properly.

When I try to start up Apache (apachectl start), I get the message "(125)Address already in use: make_sock: could not bind to address [::]:80".

In doing some research, I find that something on the server is still listening on port 80, and here is what it shows is running:

iwui 1613 1598 0 Mar 16 ? 0:01 /lcl/sit/apps/TeamSite/iw-webd/bin/iwwebd -DUNIX -d /lcl/sit/apps/TeamSite/iw-w
iwui 1614 1598 0 Mar 16 ? 0:00 /lcl/sit/apps/TeamSite/iw-webd/bin/iwwebd -DUNIX -d /lcl/sit/apps/TeamSite/iw-w
iwui 1612 1598 0 Mar 16 ? 0:00 /lcl/sit/apps/TeamSite/iw-webd/bin/iwwebd -DUNIX -d /lcl/sit/apps/TeamSite/iw-w
root 1598 1 0 Mar 16 ? 2:08 /lcl/sit/apps/TeamSite/iw-webd/bin/iwwebd -DUNIX -d /lcl/sit/apps/TeamSite/iw-w

Just as a little background, here is what changed on my server. Yesterday, they changed the server authentication from using NIS to LDAP. Things stopped working right after the switch was made. In working with the server admins, they say that this should have little, if any affect. However, I have been finding out that there has been other unique challenges after the change was made. I asked the server team to switch my server back to NIS authentication (which they did) until I could get this resolved, but it did not resolve the situation. I have tried just about everything I can find on the web, but still not having any luck.

Would anyone have any suggestions on why iwwebd is running while the server even after I shut the server down or how to get this resolved? There is some urgency for me to fix because it is our production instance (and users are not able to preview pages within TeamSite).

Thanks

TeamSite 7.2.1 on Solaris 10.

Find more posts tagged with

Comments

Rick Poulin

Check what process is actually using that port:
# netstat -ap | grep http
or
# netstat -ap | grep 80

ttriemst

I started the TS server up (but not apache), and here is what I get:

darter# netstat -ap | grep http
darter# netstat -ap | grep 80

However, when I do a telnet localhost 80, here is what I get:

/etc> telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET/

302 Found

Found

The document has moved here.

Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.8e Server at darter Port 80

Connection to localhost closed by foreign host.
/etc>

This tells me that something is listening on port 80, and I was able to determine it was iw-webd (or iwwebd).

Adam Stoller

Normally - on a TeamSite server - iwwebd is listening on Port 80 and any other web services running on that server (e.g.: apache being used as the "customer web server") should be running on a different port - usually 81.

Stopping iwserver does not stop iwwebd (as you noted)
However, that's not the issue here - the issue is why are you trying to run (non-iwwebd) apache on port 80 on the TeamSite server?

Side note: Switching to LDAP authentication requires a number of steps - both in terms of setting up the authentication protocol and in terms of making sure that all your customizations have been converted from Perl to Java (or XSL in the case of PTs) -- unless you're still listing all the accounts in /etc/passwd and /etc/shadow and have something like PAM configured to perform the actual authentication against the LDAP server - in which case you're existing Perl code should be fine.

ttriemst

Thanks for the post Gohti.

I have been working on trying to figure out the problem, and I think I have it narrowed down. It appears that when I try to preview a file, I found the following error in the apache log file:

File does not exist: File does not exist: /iwmnt/default/main, referer: http://guppy.grhq.gfs.com/iw-cc/Site/Component....

So, on the server (using putty) I navigated to /iwmnt/default and did an "ls -lrt", and there is nothing there. I thought that maybe it was a permissions issue, but verified with my server administrator that there is nothing under the directory.

Any thoughts on what prevents the directory from being available once the server starts up?

I'm trying to noodle through all of the server logs as it has been a few weeks since my last TeamSite reboot, so any help would be appreciated.

nipper

So there is nothing in iwmnt/default ? OK, that is bad. Putting it mildly.

Is TS server running ? Check iwtrace and iwserver logs.

run df -h on the server and post the results.

Check your license

ttriemst

Yes, TS is running and my users are working on it. They can still edit files, save, and everything else. They just can not preview the files (that is the only thing that they are experiencing issues with). I have validated that the license is still valid (and have dealt with an expired license before), so I feel pretty good about that. I did the df -h, and the space looks good. Here are the results:

/lcl/apps/apache/httpd/logs> df -h
Filesystem size used avail capacity Mounted on
rpool/ROOT/p20120516_JDW
67G 5.5G 45G 11% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 775M 448K 775M 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
sharefs 0K 0K 0K 0% /etc/dfs/sharetab
fd 0K 0K 0K 0% /dev/fd
swap 775M 272K 775M 1% /tmp
swap 775M 56K 775M 1% /var/run
rpool/export 67G 23K 45G 1% /export
rpool/export/home 67G 21K 45G 1% /export/home
rpool/lcl 67G 1.9G 45G 5% /lcl
rpool/lcl/apps 67G 201M 45G 1% /lcl/apps
rpool/lcl/apps/apache
512M 234M 278M 46% /lcl/apps/apache
rpool/lcl/apps/java 1.2G 540M 688M 44% /lcl/apps/java
rpool/lcl/apps/oracle1020
67G 806M 45G 2% /lcl/apps/oracle1020
rpool/lcl/apps/perl 256M 84M 172M 33% /lcl/apps/perl
rpool/lcl/prd 67G 25K 45G 1% /lcl/prd
rpool/lcl/prd/apps 67G 485M 45G 2% /lcl/prd/apps
teamsitepool/ApplicationContainer
138G 1.1G 89G 2% /lcl/prd/apps/ApplicationContainer
teamsitepool/CommonServices
138G 43M 89G 1% /lcl/prd/apps/CommonServices
teamsitepool/CommonServicesClient
138G 15M 89G 1% /lcl/prd/apps/CommonServicesClient
rpool/lcl/prd/apps/Cyfin
67G 264M 45G 1% /lcl/prd/apps/Cyfin
teamsitepool/FoundationLibraries
138G 188M 89G 1% /lcl/prd/apps/FoundationLibraries
teamsitepool/TeamSite
138G 2.7G 89G 3% /lcl/prd/apps/TeamSite
rpool/lcl/prd/apps/apache
67G 23K 45G 1% /lcl/prd/apps/apache
rpool/lcl/prd/apps/apache/httpd
67G 247M 45G 1% /lcl/prd/apps/apache/httpd
rpool/lcl/prd/apps/esm
67G 124K 45G 1% /lcl/prd/apps/esm
teamsitepool/iw-home 138G 2.7G 89G 3% /lcl/prd/apps/iw-home
teamsitepool/iw-search
138G 21K 89G 1% /lcl/prd/apps/iw-search
rpool/lcl/prd/data 67G 36M 45G 1% /lcl/prd/data
rpool/lcl/prd/data/apache
67G 23K 45G 1% /lcl/prd/data/apache
rpool/lcl/prd/data/apache/support
50M 31K 50M 1% /lcl/prd/data/apache/support
teamsitepool/iw-store
138G 41G 89G 32% /lcl/prd/data/iw-store
teamsitepool/iw-apache
2.0G 85M 1.9G 5% /lcl/prd/data/iw_apache
rpool/lcl/prd/logs 67G 25K 45G 1% /lcl/prd/logs
rpool/lcl/prd/logs/apache
67G 23K 45G 1% /lcl/prd/logs/apache
rpool/lcl/prd/logs/apache/support
512M 121K 512M 1% /lcl/prd/logs/apache/support
teamsitepool/iw-logs 138G 640M 89G 1% /lcl/prd/logs/iw-logs
rpool 67G 97K 45G 1% /rpool
teamsitepool 138G 21K 89G 1% /teamsitepool
rpool/lcl/apps/openv 67G 828M 45G 2% /usr/openv
/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1
50G 5.5G 45G 11% /platform/sun4u-us3/lib/libc_psr.so.1
/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
50G 5.5G 45G 11% /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
homedir.grhq.gfs.com:/homedir/gfshome/ttriemst
64G 35G 29G 56% /gfs/home/ttriemst
infr:/prd/BE/storage 4.7T 4.1T 656G 87% /gfs/infr/storage
homedir.grhq.gfs.com:/homedir/gfshome/srm
64G 35G 29G 56% /gfs/home/srm
custtransprd:/prd/BE/custtrans/apps
4.7T 4.1T 656G 87% /gfs/prd/apps/custtrans
/lcl/apps/apache/httpd/logs>

ttriemst

Sorry, should have added that I am working on digging through the logs right now.

nipper

OK, if people are working, then TS is running.

There is a virtual directory in Apache, is that created ?

When you logged on the server, what login did you have ? By chance do you have branch security on ? And maybe have the main branch marked private ?

Rick Poulin

iwserver.log/iwtrace.log would be the first place to look. You can try bouncing the mount via `iwstoreadm -d default` `iwstoreadm -a default`. I'd hope/expect that this would either resolve the issue or generate some kind of warning in those logs. If that doesn't work, go the next level down and fiddle with the iwmount CLT. If still nothing, try running an iwfsck.

Another thought.. maybe /iwmnt/default is a "plain" folder that got created by accident and your virtual mount is actually elsewhere? what does iwgetmount return?

nipper

You realize that the DF you show does not have /iwmnt listed.

I have to ask, but are you certain you are on the right server ? It does not see to be running TeamSite.

run:

mount

You should see something like this

none on /iwmnt/default type pmfs (rw,store=/iwserver/default)
none on /iwmnt/iwadmin type pmfs (rw,store=/iwserver/iwadmin)

but then df should have shown you this:

none 34G 15G 19G 44% /iwmnt/default
none 34G 15G 19G 44% /iwmnt/iwadmin

ttriemst

I did double check and I am on my production server. I did run the iwgetmount, and it did return the right value. I am having my server admin run

/lcl/prd/apps/TeamSite/bin> ./iwgetmount
/iwmnt
/lcl/prd/apps/TeamSite/bin>

I am having my server admin run the "iwstoreadm -d default" command and will see what that yields (I work in a pretty restricted environment).

Rick Poulin

I am having my server admin run the "iwstoreadm -d default" command and will see what that yields (I work in a pretty restricted environment).

Umm.. you'll want to have him run the "-a" immediately afterwards, and should probably be doing these tests off hours...

nipper

I would site with the admin and run some commands.

iwstat
ps -ef | grep TeamSite

in addition to the other commands I suggested

ttriemst

All,

Thanks for the quick and great feedback. I was able to get this resolved. Come to find out, the server administrators were doing some tasks that "should not have any impact on the server", but in fact did have an impact. The did something that caused all of the drives on the server to remount, and TeamSite did not like this. Once I rebooted the TeamSite application, everything came up without any issues.

Thanks again for the quick responses.