nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Problem With the Backing Store

cute_guy

I am facing a problem with my teamsite ,The backing store is not getting mounted and the iwserver is starting

######################################
i have attached the extract from iwtrace.log
##################################
Cache size: 30000
TEnvironment: fd limit 8192 max 8192
Store /local/iw-store/workflow: 10 objects found in the log file.
TMetafileIoEngine::GetHeader: Unable to read index 0x0:Bad file number
TModification::Commit: WritePoint failed: object ID = 3200, data size = 9
84
Store /local/iw-store/workflow: recovery failed, store is disabled.
Failed to start store manager.
EXITED:
###################################################################################################
I have also attached extract from iwserver.log
###################################################################################################
iwstop: Looking for the wfs kernel module id... found 110
iwstop: checking for standard services
iwstop: nfsd stopping...
iwstop: mountd not running, ok
iwstop: aborting the current queue... done
iwstop: waiting for the server to shutdown... done
iwstop: unloading kernel module... done
iwstop: checking for standard services
iwstop: nfsd starting
iwstop: mountd running
iwevent: Stopping report daemons...
iwevent: Closed db...
iwstart: The events log file is /var/iwov/iwevents.log
iwstart: looking for the service pointer in the kernel ... found 70fdcb98
iwstart: checking for standard services
iwstart: nfsd stopping...
iwstart: mountd not running, ok
iwstart: reloading kernel module ... done
iwstart: patching kernel service pointer ... done
iwstart: checking for standard services
iwstart: nfsd starting
iwstart: mountd running
iwstart: Starting iwprocdstart script (log file: /var/adm/iwprocessd.log)... DONE
iwstart: Spawning iwserver.2.8 (trace file: /var/iwov/iwtrace.log)...
iwstart: Waiting up to 60 seconds for iwserver to come up... TIMED OUT
ERROR:iwstart: iwserver did not come up in 60 seconds
ERROR:iwstart: Please look at /var/iwov/iwtrace.log for more information
ERROR: Failed to start.

Find more posts tagged with

Comments

iwovGraduate

I would recommend you open a case with support.

Have you tried using iwfsck ?

cute_guy

This is the output of iwfsck, i couldn't infer anything from this.

bash-2.03# ./iwfsck
Command line: ./iwfsck
iwfsck: 5.5.2 Build 17976 Interwoven 20030323

***************************************
Walk branches and areas.
***************************************

[Wed Jul 28 08:16:46 2004]

***************************************
Walk directories, match with areas.
***************************************

[Wed Jul 28 08:16:48 2004]
[Wed Jul 28 08:16:48 2004]
Total points walked: 3950
Directories walked: 0
Files walked: 0
Symlinks walked: 0
TeamXML files walked: 0
Zombies walked: 0
Total points read from disk: 1359
FINISHED 0 errors found

Adam Stoller

Run it with the '-d' flag

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

new_report_3.rptdesign

cute_guy

This is output obtained::-

[root@tsdev1] /app/iw-home/private/bin # ./iwfsck -d
Command line: ./iwfsck -d
iwfsck: 5.5.2 Build 17976 Interwoven 20030323

***************************************
Walk branches and areas.
***************************************

[Wed Jul 28 12:50:26 2004]

***************************************
Walk directories, match with areas.
***************************************

[Wed Jul 28 12:50:28 2004]
[Wed Jul 28 12:50:28 2004]

***************************************
Walk directories and files.
***************************************

Looking up starting vpath /
[Wed Jul 28 12:50:28 2004]
STATUS: 0 points walked
[Wed Jul 28 12:50:30 2004]
STATUS: 1000 points walked
[Wed Jul 28 12:50:32 2004]
STATUS: 2000 points walked
[Wed Jul 28 12:50:35 2004]
STATUS: 3000 points walked
[Wed Jul 28 12:50:35 2004]
STATUS: 4000 points walked
[Wed Jul 28 12:50:36 2004]
STATUS: 5000 points walked
STATUS: 5672 points walked
Finished walking unwalked points.
[Wed Jul 28 12:50:36 2004]
Total points walked: 11102
Directories walked: 1443
Files walked: 3533
Symlinks walked: 0
TeamXML files walked: 0
Zombies walked: 712
Total points read from disk: 6709
FINISHED 0 errors found

I have also tried the following

[root@tsdev1] / # mount
/ on /dev/dsk/c0t0d0s0 read/write/setuid/intr/largefiles/onerror=panic/dev=2200000 on Thu Jul 22 13:
41:26 2004
/proc on /proc read/write/setuid/dev=3d40000 on Thu Jul 22 13:41:25 2004
/dev/fd on fd read/write/setuid/dev=3e00000 on Thu Jul 22 13:41:27 2004
/etc/mnttab on mnttab read/write/setuid/dev=3f00000 on Thu Jul 22 13:41:28 2004
/var on /dev/dsk/c0t0d0s3 read/write/setuid/intr/largefiles/onerror=panic/dev=2200003 on Thu Jul 22
13:41:29 2004
/var/run on swap read/write/setuid/dev=1 on Thu Jul 22 13:41:29 2004
/app on /dev/dsk/c0t0d0s4 read/write/setuid/intr/largefiles/onerror=panic/dev=2200004 on Thu Jul 22
13:41:31 2004
/home on /dev/dsk/c0t0d0s7 read/write/setuid/intr/largefiles/onerror=panic/dev=2200007 on Thu Jul 22
13:41:31 2004
/local on /dev/dsk/c0t1d0s7 read/write/setuid/intr/largefiles/onerror=panic/dev=220000f on Thu Jul 2
2 13:41:31 2004
/tmp on swap read/write/setuid/dev=2 on Thu Jul 22 13:41:31 2004

[root@tsdev1] / # iwstat -c
ERROR:00930: RPC connection failure

[root@tsdev1] / # iwgethome
/app/iw-home

[root@tsdev1] / # iwgetstore
/local/iw-store

[root@tsdev1] / # df -k `iwgetstore` `iwgethome`
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t1d0s7 17143530 92114 16879981 1% /local
/dev/dsk/c0t0d0s4 4030518 3025129 965084 76% /app

Adam Stoller

Make sure nfsd is running on the server - TeamSite won't start up if nfsd isn't already running (though I thought the error messages were different)

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

cute_guy

[root@tsdev1] / # ps -ef|grep nfsd
root 388 1 0 Jul 22 ? 0:00 /usr/lib/nfs/nfsd -a 16

skip11

Is the entry in /etc/name_to_sysnum still correct ? The absense of the this entry will
throw a different error message than the one you're getting, so it's probably ok. Patching
a Solaris OS with the Recommened and Security cluster (which includes a kernel update)
will overwrite this file and render your teamsite unable to start.

Anything in /var/adm/messages ? Did you run dmesg to look at the latest OS messages,
might be something in there. Also take a look at /etc/mnttab, maybe you have a old munged
entry in there. I see you did run mount, but a look at this file may yield more.

You also might want to truss the start script.

Regards,

RB

Edited by skip on 07/28/04 06:53 AM (server time).

cute_guy

output of /etc/name_to_sysnum

nosys 0
rexit 1
fork 2
read 3
write 4
open 5
close 6
wait 7
creat 8
link 9
unlink 10
exec 11
chdir 12
gtime 13
mknod 14
chmod 15
chown 16
brk 17
stat 18
lseek 19
getpid 20
mount 21
umount 22
setuid 23
getuid 24
stime 25
alarm 27
fstat 28
pause 29
utime 30
stty 31
gtty 32
access 33
nice 34
statfs 35
syssync 36
kill 37
fstatfs 38
setpgrp 39
dup 41
pipe 42
times 43
profil 44
setgid 46
getgid 47
ssig 48
msgsys 49
sysacct 51
shmsys 52
semsys 53
ioctl 54
uadmin 55
uexch 56
utssys 57
fdsync 58
exece 59
umask 60
chroot 61
fcntl 62
ulimit 63
tasksys 70
acctctl 71
exacctsys 72
reserved_73 73
reserved_74 74
reserved_75 75
reserved_76 76
lwp_park 77
sendfilev 78
rmdir 79
mkdir 80
getdents 81
sysfs 84
getmsg 85
putmsg 86
poll 87
lstat 88
symlink 89
readlink 90
setgroups 91
getgroups 92
fchmod 93
fchown 94
sigprocmask 95
sigsuspend 96
sigaltstack 97
sigaction 98
sigpending 99
setcontext 100
statvfs 103
fstatvfs 104
getloadavg 105
nfs 106
waitsys 107
sigsendsys 108
priocntlsys 112
pathconf 113
mincore 114
mmap 115
mprotect 116
munmap 117
fpathconf 118
vfork 119
fchdir 120
readv 121
writev 122
xstat 123
lxstat 124
fxstat 125
xmknod 126
setrlimit 128
getrlimit 129
lchown 130
memcntl 131
getpmsg 132
putpmsg 133
rename 134
uname 135
setegid 136
sysconfig 137
adjtime 138
systeminfo 139
seteuid 141
vtrace 142
fork1 143
sigwait 144
lwp_info 145
yield 146
lwp_sema_wait 147
lwp_sema_post 148
lwp_sema_trywait 149
lwp_detach 150
corectl 151
modctl 152
fchroot 153
utimes 154
vhangup 155
gettimeofday 156
getitimer 157
setitimer 158
lwp_create 159
lwp_exit 160
lwp_suspend 161
lwp_continue 162
lwp_kill 163
lwp_self 164
lwp_setprivate 165
lwp_getprivate 166
lwp_wait 167
lwp_mutex_wakeup 168
lwp_mutex_enter 169
lwp_cv_wait 170
lwp_cv_signal 171
lwp_cv_broadcast 172
pread 173
pwrite 174
llseek 175
inst_sync 176
srmlimitsys 177
kaio 178
cpc 179
tsolsys 184
acl 185
c2audit 186
processor_bind 187
processor_info 188
p_online 189
sigqueue 190
clock_gettime 191
clock_settime 192
clock_getres 193
timer_create 194
timer_delete 195
timer_settime 196
timer_gettime 197
timer_getoverrun 198
nanosleep 199
facl 200
doorfs 201
setreuid 202
setregid 203
install_utrap 204
signotify 205
schedctl 206
pset 207
sparc_utrap_install 208
resolvepath 209
signotifywait 210
lwp_sigredirect 211
lwp_alarm 212
getdents64 213
mmap64 214
stat64 215
lstat64 216
fstat64 217
statvfs64 218
fstatvfs64 219
setrlimit64 220
getrlimit64 221
pread64 222
pwrite64 223
creat64 224
open64 225
rpcmod 226
so_socket 230
so_socketpair 231
bind 232
listen 233
accept 234
connect 235
shutdown 236
recv 237
recvfrom 238
recvmsg 239
send 240
sendmsg 241
sendto 242
getpeername 243
getsockname 244
getsockopt 245
setsockopt 246
sockconfig 247
ntp_gettime 248
ntp_adjtime 249
lwp_mutex_unlock 250
lwp_mutex_trylock 251
lwp_mutex_init 252
cladm 253
lwp_sigtimedwait 254
umount2 255
wfs 181

##################################################
OUTPUT of /var/adm/messages
#####################################
Jul 25 22:00:00 tsdev1 sendmail[14242]: [ID 702911 mail.crit] My unqualified host name (tsdev1) unkn
own; sleeping for retry
Jul 25 22:01:00 tsdev1 sendmail[14242]: [ID 702911 mail.alert] unable to qualify my own domain name
(tsdev1) -- using short name
Jul 25 22:01:00 tsdev1 syslogd: /var/adm/utmpx not owned by root or not mode 644.\nThis file must be
owned by root and not writable by\nanyone other than root. This alert is being dropped because of\
nthis problem.
Jul 25 22:01:00 tsdev1 last message repeated 1 time
Jul 26 22:00:00 tsdev1 sendmail[18097]: [ID 702911 mail.crit] My unqualified host name (tsdev1) unkn
own; sleeping for retry
Jul 26 22:01:00 tsdev1 sendmail[18097]: [ID 702911 mail.alert] unable to qualify my own domain name
(tsdev1) -- using short name
Jul 26 22:01:00 tsdev1 syslogd: /var/adm/utmpx not owned by root or not mode 644.\nThis file must be
owned by root and not writable by\nanyone other than root. This alert is being dropped because of\
nthis problem.
Jul 26 22:01:00 tsdev1 last message repeated 1 time
Jul 27 19:06:57 tsdev1 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Jul 27 19:06:57 tsdev1 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Jul 27 22:00:00 tsdev1 sendmail[22142]: [ID 702911 mail.crit] My unqualified host name (tsdev1) unkn
own; sleeping for retry
Jul 27 22:01:00 tsdev1 sendmail[22142]: [ID 702911 mail.alert] unable to qualify my own domain name
(tsdev1) -- using short name
Jul 27 22:01:00 tsdev1 syslogd: /var/adm/utmpx not owned by root or not mode 644.\nThis file must be
owned by root and not writable by\nanyone other than root. This alert is being dropped because of\
nthis problem.
Jul 27 22:01:00 tsdev1 last message repeated 1 time

###############################################
OUTPUT of dmesg
################################################
Jul 27 19:06:57 tsdev1 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Jul 27 19:06:57 tsdev1 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Jul 27 22:00:00 tsdev1 sendmail[22142]: [ID 702911 mail.crit] My unqualified host name (tsdev1) unkn
own; sleeping for retry
Jul 27 22:01:00 tsdev1 sendmail[22142]: [ID 702911 mail.alert] unable to qualify my own domain name
(tsdev1) -- using short name
Jul 27 22:01:00 tsdev1 syslogd: /var/adm/utmpx not owned by root or not mode 644.\nThis file must be
owned by root and not writable by\nanyone other than root. This alert is being dropped because of\
nthis problem.
Jul 27 22:01:00 tsdev1 last message repeated 1 time
########################################################
OUTPUT of /etc/mnttab
###########################################
/dev/dsk/c0t0d0s0 / ufs rw,intr,largefiles,onerror=panic,suid,dev=2200000 1090
483886
/proc /proc proc dev=3d40000 1090483885
fd /dev/fd fd rw,suid,dev=3e00000 1090483887
mnttab /etc/mnttab mntfs dev=3f00000 1090483888
/dev/dsk/c0t0d0s3 /var ufs rw,intr,largefiles,onerror=panic,suid,dev=2200003 1090
483889
swap /var/run tmpfs dev=1 1090483889
/dev/dsk/c0t0d0s4 /app ufs rw,intr,largefiles,onerror=panic,suid,dev=2200004 1090
483891
/dev/dsk/c0t0d0s7 /home ufs rw,intr,largefiles,onerror=panic,suid,dev=2200007 1090
483891
/dev/dsk/c0t1d0s7 /local ufs rw,intr,largefiles,onerror=panic,suid,dev=220000f 1090
483891
swap /tmp tmpfs dev=2 1090483891
tsdev1:vold(pid223) /vol nfs ignore,dev=3f80001 1090483922

skip11

I suppose you have rebooted the Sun and the filesystem your BS resides on fsck's fine ?

At this point I would truss the iw.server start and post it.

RB

cute_guy

#####################################################
OUTPUT of /usr/bin/truss /etc/init.d/iw.server start
########################################################

execve("/etc/init.d/iw.server", 0xFFBEF9D4, 0xFFBEF9E4) argc = 3
resolvepath("/usr/lib/ld.so.1", "/usr/lib/ld.so.1", 1023) = 16
open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT
stat("/usr/lib/libgen.so.1", 0xFFBEF0F4) = 0
open("/usr/lib/libgen.so.1", O_RDONLY) = 3
fstat(3, 0xFFBEF0F4) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF3A0000
mmap(0x00000000, 98304, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF380000
mmap(0xFF396000, 2335, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 24576) = 0xFF396000
munmap(0xFF386000, 65536) = 0
memcntl(0xFF380000, 6932, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/usr/lib/libc.so.1", 0xFFBEF0F4) = 0
open("/usr/lib/libc.so.1", O_RDONLY) = 3
fstat(3, 0xFFBEF0F4) = 0
mmap(0xFF3A0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF3A0000
mmap(0x00000000, 802816, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF280000
mmap(0xFF33C000, 24764, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 704512) = 0xFF33C0
00
munmap(0xFF32C000, 65536) = 0
memcntl(0xFF280000, 113504, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
stat("/usr/lib/libdl.so.1", 0xFFBEF0F4) = 0
open("/usr/lib/libdl.so.1", O_RDONLY) = 3
fstat(3, 0xFFBEF0F4) = 0
mmap(0xFF3A0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF3A0000
close(3) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFF370000
stat("/usr/platform/SUNW,Ultra-5_10/lib/libc_psr.so.1", 0xFFBEEF04) = 0
open("/usr/platform/SUNW,Ultra-5_10/lib/libc_psr.so.1", O_RDONLY) = 3
fstat(3, 0xFFBEEF04) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF360000
mmap(0x00000000, 16384, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF350000
close(3) = 0
munmap(0xFF360000, 8192) = 0
getpid() = 28511 [28510]
getpgid(28511) = 28510
getsid(28511) = 28504
brk(0x0003A0F0) = 0
sysconfig(_CONFIG_SIGRT_MIN) = 38
sysconfig(_CONFIG_SIGRT_MAX) = 45
sigaltstack(0xFFBEF8C4, 0x00000000) = 0
sigaction(SIGHUP, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGHUP, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGINT, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGINT, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGQUIT, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGQUIT, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGILL, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGILL, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGTRAP, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGTRAP, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGABRT, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGABRT, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGEMT, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGEMT, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGFPE, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGFPE, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGBUS, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGBUS, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGSEGV, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGSYS, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGSYS, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGPIPE, 0x00000000, 0xFFBEF840) = 0
sigaction(SIGPIPE, 0xFFBEF7A0, 0xFFBEF820) = 0
sigaction(SIGALRM, 0x00000000, 0xFFBEF840) = 0
brk(0x0003A0F8) = 0
brk(0x0003A2F8) = 0
stat("/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.2", 0xFFBEE8EC) = 0
open("/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.2", O_RDONLY) = 3
fstat(3, 0xFFBEE8EC) = 0
mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF360000
mmap(0x00000000, 90112, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF260000
mmap(0xFF272000, 8622, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 8192) = 0xFF272000
munmap(0xFF264000, 57344) = 0
memcntl(0xFF260000, 6740, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3) = 0
munmap(0xFF360000, 8192) = 0
brk(0x0003A4F8) = 0
brk(0x0003A5E0) = 0
brk(0x0003A7E0) = 0
stat("/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.2", 0xFFBEE9AC) = 0
brk(0x0003A9E0) = 0
brk(0x0003ABE0) = 0
getuid() = 0 [0]
getuid() = 0 [0]
getgid() = 1 [1]
getgid() = 1 [1]
open64("/etc/init.d/iw.server", O_RDONLY) = 3
close(19) Err#9 EBADF
fcntl(3, F_DUPFD, 0x00000013) = 19
close(3) = 0
fcntl(19, F_SETFD, 0x00000001) = 0
ioctl(2, TCGETA, 0xFFBEF7EC) = 0
ioctl(19, TCGETA, 0xFFBEF7EC) Err#25 ENOTTY
read(19, " # ! / b i n / s h\n #\n".., 128) = 128
brk(0x0003ADE0) = 0
getuid() = 0 [0]
stat64("/usr/sbin/basename", 0xFFBEF188) Err#2 ENOENT
stat64("/sbin/basename", 0xFFBEF188) Err#2 ENOENT
stat64("/usr/bin/basename", 0xFFBEF188) = 0
access("/usr/bin/basename", 9) = 0
pipe() = 3 [4]
fork() = 28512
close(4) = 0
read(3, " i w . s e r v e r\n", 128) = 10
read(3, 0xFFBEF520, 128) = 0
ioctl(3, TCGETA, 0xFFBEF3A4) Err#22 EINVAL
ioctl(3, TCGETA, 0xFFBEF404) Err#22 EINVAL
close(3) = 0
waitid(P_PID, 28512, 0xFFBEF400, WEXITED|WTRAPPED) = 0
read(19, " "\n V E R S I O N = " 5".., 128) = 128
read(19, " d o w n i w s e r v e".., 128) = 128
stat64("/usr/sbin/iwgethome", 0xFFBEF188) Err#2 ENOENT
stat64("/sbin/iwgethome", 0xFFBEF188) Err#2 ENOENT
stat64("/usr/bin/iwgethome", 0xFFBEF188) = 0
access("/usr/bin/iwgethome", 9) = 0
pipe() = 3 [4]
fork() = 28513
close(4) = 0
read(3, " / a p p / i w - h o m e".., 128) = 13
read(3, 0xFFBEF520, 128) = 0
ioctl(3, TCGETA, 0xFFBEF3A4) Err#22 EINVAL
ioctl(3, TCGETA, 0xFFBEF404) Err#22 EINVAL
close(3) = 0
waitid(P_PID, 28513, 0xFFBEF400, WEXITED|WTRAPPED) = 0
read(19, " " E R R O R : R u n".., 128) = 128
read(19, " $ I W _ H O M E / b i n".., 128) = 128
brk(0x0003B0E0) = 0
brk(0x0003AEE0) = 0
read(19, " S I O N "\n\t\t e x i t".., 128) = 128
brk(0x0003B2E0) = 0
read(19, " G = $ 1\n\t\t e l s e\n".., 128) = 128
brk(0x0003B0E0) = 0
brk(0x0003AEE0) = 0
read(19, " p i d = ` U N I X 9 5".., 128) = 128
read(19, " " $ p i d " ! = "".., 128) = 128
brk(0x0003B3E0) = 0
brk(0x0003B1E0) = 0
read(19, " 1 2 > / d e v / n u l".., 128) = 128
stat64("/usr/bin/iwgetlog", 0xFFBEF568) Err#2 ENOENT
stat64("/app/iw-home/private/bin/iwgetlog", 0xFFBEF568) Err#2 ENOENT
stat64("/app/iw-home/bin/iwgetlog", 0xFFBEF568) = 0
access("/app/iw-home/bin/iwgetlog", 9) = 0
fork() = 28514
waitid(P_PID, 28514, 0xFFBEF6F0, WEXITED|WTRAPPED|WNOWAIT) = 0
ioctl(0, TIOCGPGRP, 0xFFBEF6AC) = 0
ioctl(0, TCGETS, 0x000391B8) = 0
waitid(P_PID, 28514, 0xFFBEF6F0, WEXITED|WTRAPPED) = 0
brk(0x0003B7E0) = 0
read(19, " t R E S U L T\n i f ".., 128) = 128
brk(0x0003B5E0) = 0
brk(0x0003B3E0) = 0
pipe() = 3 [4]
fork() = 28515
close(4) = 0
read(3, " / v a r / i w o v / i w".., 128) = 23
read(3, 0xFFBEF520, 128) = 0
ioctl(3, TCGETA, 0xFFBEF3A4) Err#22 EINVAL
ioctl(3, TCGETA, 0xFFBEF404) Err#22 EINVAL
close(3) = 0
waitid(P_PID, 28515, 0xFFBEF400, WEXITED|WTRAPPED) = 0
read(19, " r n a m e $ I W L O G".., 128) = 128
stat64("/usr/bin/dirname", 0xFFBEF188) = 0
access("/usr/bin/dirname", 9) = 0
pipe() = 3 [4]
fork() = 28516
close(4) = 0
read(3, " / v a r / i w o v\n", 128) = 10
read(3, 0xFFBEF520, 128) = 0
ioctl(3, TCGETA, 0xFFBEF3A4) Err#22 EINVAL
ioctl(3, TCGETA, 0xFFBEF404) Err#22 EINVAL
close(3) = 0
waitid(P_PID, 28516, 0xFFBEF400, WEXITED|WTRAPPED) = 0
stat64("/usr/bin/mkdir", 0xFFBEF3E8) = 0
access("/usr/bin/mkdir", 9) = 0
brk(0x0003BAE0) = 0
stat64("/var/iwov", 0xFFBEF3C8) = 0
brk(0x0003B8E0) = 0
brk(0x0003B6E0) = 0
stat64("/usr/bin/iwetcboot", 0xFFBEF568) Err#2 ENOENT
stat64("/app/iw-home/private/bin/iwetcboot", 0xFFBEF568) = 0
access("/app/iw-home/private/bin/iwetcboot", 9) = 0
fork() = 28517
waitid(P_PID, 28517, 0xFFBEF6F0, WEXITED|WTRAPPED|WNOWAIT) = 0
ioctl(0, TIOCGPGRP, 0xFFBEF6AC) = 0
ioctl(0, TCGETS, 0x000391B8) = 0
waitid(P_PID, 28517, 0xFFBEF6F0, WEXITED|WTRAPPED) = 0
brk(0x0003B4E0) = 0
read(19, " f [ $ R E S U L T ".., 128) = 128
read(19, " e c h o " S t a r t i".., 128) = 128
read(19, " r w o v e n T e a m S".., 128) = 128
brk(0x0003BCE0) = 0
read(19, " & 1 ;\n\t R E S U L".., 128) = 128
read(19, "\t f i\n\t i w e v e".., 128) = 128
stat64("/usr/bin/iwevent", 0xFFBEEF08) Err#2 ENOENT
stat64("/app/iw-home/private/bin/iwevent", 0xFFBEEF08) = 0
access("/app/iw-home/private/bin/iwevent", 9) = 0
read(19, " t . C h e c k $ I W".., 128) = 128
read(19, " - n e 0 ] ; t".., 128) = 128
read(19, " a l l " ] ; t h e".., 128) = 128
read(19, " " $ I W _ H O M E / i n".., 128) = 128
stat64("/usr/bin/mv", 0xFFBEEBA8) = 0
access("/usr/bin/mv", 9) = 0
stat64("/usr/bin/iwuiboot", 0xFFBEED28) Err#2 ENOENT
stat64("/app/iw-home/private/bin/iwuiboot", 0xFFBEED28) = 0
access("/app/iw-home/private/bin/iwuiboot", 9) = 0
read(19, " h e n\n\t\t e c h o "".., 128) = 128
read(19, " L O G F I L E 2 > & 1".., 128) = 128
brk(0x0003C4E0) = 0
read(19, " G : ` d a t e ` " 1".., 128) = 128
read(19, "\t\t e c h o " E R R O".., 128) = 128
read(19, " ? ; e x p o r t R".., 128) = 128
read(19, " i\n\n\t i w e t c b o".., 128) = 128
read(19, " r r o r s t o p p i n".., 128) = 128
read(19, " 3 - n e 0 ] ; ".., 128) = 128
read(19, " a t e ` " 1 > > $ I W".., 128) = 128
read(19, " i t $ R E S U L T 1\n".., 128) = 128
brk(0x0003CCE0) = 0
read(19, " e p t f o r r m i r".., 128) = 84
Interwoven TeamSite starting ...
write(1, " I n t e r w o v e n T".., 33) = 33
stat64("/usr/bin/date", 0xFFBEF050) = 0
access("/usr/bin/date", 9) = 0
pipe() = 3 [4]
fork() = 28519
close(4) = 0
read(3, " T h u J u l 2 9 1".., 128) = 29
read(3, 0xFFBEF3E8, 128) = 0
ioctl(3, TCGETA, 0xFFBEF26C) Err#22 EINVAL
ioctl(3, TCGETA, 0xFFBEF2CC) Err#22 EINVAL
close(3) = 0
waitid(P_PID, 28519, 0xFFBEF2C8, WEXITED|WTRAPPED) = 0
fcntl(1, F_DUPFD, 0x0000000A) = 10
open64("/var/iwov/iwserver.log", O_WRONLY) = 3
llseek(3, 0, SEEK_END) = 122790
fcntl(1, F_GETFD, 0x00000000) = 0
close(1) = 0
fcntl(3, F_DUPFD, 0x00000001) = 1
close(3) = 0
fcntl(2, F_DUPFD, 0x0000000A) = 11
dup(1) = 3
fcntl(2, F_GETFD, 0x00000000) = 0
close(2) = 0
fcntl(3, F_DUPFD, 0x00000002) = 2
close(3) = 0
write(1, " i w . s e r v e r : -".., 61) = 61
fcntl(2, F_GETFD, 0x00000000) = 0
close(2) = 0
fcntl(11, F_DUPFD, 0x00000002) = 2
close(11) = 0
fcntl(1, F_GETFD, 0x00000000) = 0
close(1) = 0
fcntl(10, F_DUPFD, 0x00000001) = 1
close(10) = 0
fork() = 28520
ERROR: Failed to start. Check /var/iwov/iwserver.log
waitid(P_PID, 28520, 0xFFBEF558, WEXITED|WTRAPPED|WNOWAIT) = 0
ioctl(0, TIOCGPGRP, 0xFFBEF514) = 0
ioctl(0, TCGETS, 0x000391B8) = 0
waitid(P_PID, 28520, 0xFFBEF558, WEXITED|WTRAPPED) = 0
brk(0x0003CAE0) = 0
read(19, 0x000394D8, 128) = 0
ioctl(19, TCGETA, 0xFFBEF784) Err#25 ENOTTY
ioctl(19, TCGETA, 0xFFBEF7E4) Err#25 ENOTTY
close(19) = 0
llseek(0, 0, SEEK_CUR) = 12460
_exit(9)

skip11

Hi,

nothing out of the ordinary in this truss when compared to a successful
startup on my system. If a backup of the store exists, you might want to apply
it.

Sorry I can't help more,

RB

Adam Stoller

Uhm - you might want to look at this line:ERROR: Failed to start. Check /var/iwov/iwserver.logor more specifically - look in /var/iwov/iwserver.log and see what it says there...

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

skip11

yeah, post the iwserver.log ! I dying to know what this problem is Smiley Wink

cute_guy

##############################################
tail -f /var/iwov/iwserver.log
##############################################
iw.server: ---------- STARTING: Thu Jul 29 17:11:42 IST 2004
iwetcboot: Running iwstart /local/iw-store in /app/iw-home
iwstart: The events log file is /var/iwov/iwevents.log
iwstart: looking for the service pointer in the kernel ... found 70fdcb98
iwstart: checking for standard services
iwstart: nfsd stopping...
iwstart: mountd not running, ok
iwstart: reloading kernel module ... done
iwstart: patching kernel service pointer ... done
iwstart: checking for standard services
iwstart: nfsd starting
iwstart: mountd running
iwstart: Starting iwprocdstart script (log file: /var/adm/iwprocessd.log)... DONE
iwstart: Spawning iwserver.2.8 (trace file: /var/iwov/iwtrace.log)...
iwstart: Waiting up to 60 seconds for iwserver to come up... TIMED OUT
ERROR:iwstart: iwserver did not come up in 60 seconds
ERROR:iwstart: Please look at /var/iwov/iwtrace.log for more information
ERROR: Failed to start.
###########################################################
tail -f /var/iwov/iwtrace.log
#############################################################
STARTING: Thu Jul 29 17:11:45 IST 2004
[Thu Jul 29 17:11:45 2004]
*** IWServer Locale is --> English_UnitedStates.Latin1@Default
ERROR: no valid autoprivate file found in the current configuration (/app/iw-home/conf/events/autoprv.txt, /app/iw-home/local/config
/autoprivate.cfg, /app/iw-home/config/autoprivate.cfg, /app/iw-home/local/config/autoprivate.cfg) (2)
[Thu Jul 29 17:11:45 2004] iwserver: 5.5.2 Build 20638 Interwoven 20030604
UtsInfo Sys SunOS Node tsdev1 Release 5.8 Vers Generic_108528-22 Machine sun4u
TeamSite Executable Version checked OS 5 Exe 5
TeamSite Executable Version checked OS 8 Exe 8
[Thu Jul 29 17:11:45 2004] SMS disabled
[Thu Jul 29 17:11:45 2004] Reading users and groups...
[Thu Jul 29 17:11:45 2004] done. 17 users and 23 groups read.
[Thu Jul 29 17:11:45 2004] Error while reading config file "/app/iw-home/local/config/submit.cfg" for file expression ".*" in area "
^/default/main/portal/shared/WORKAREA/.*$": "iw_portal_share_admin" is an invalid group name for setting gid
[Thu Jul 29 17:11:45 2004] Error while reading config file "/app/iw-home/local/config/submit.cfg" for file expression ".*" in area "
^/default/main/portal/iptreference11/shared/WORKAREA/.*$": "iw_iptref11_share_admin" is an invalid group name for setting gid
[Thu Jul 29 17:11:45 2004] Error while reading config file "/app/iw-home/local/config/submit.cfg" for file expression ".*" in area "
^/default/main/portal/iptreference11/iptcountry11/shared/WORKAREA/.*$": "iw_iptcountry11_share_admin" is an invalid group name for s
etting gid
[Thu Jul 29 17:11:45 2004] Error while reading config file "/app/iw-home/local/config/submit.cfg" for file expression ".*" in area "
^/default/main/portal/iptreference11/iptcountry11/lagcb/WORKAREA/.*$": "iw_iptcountry11_lagcb_users" is an invalid group name for se
tting gid
[Thu Jul 29 17:11:45 2004] Error while reading config file "/app/iw-home/local/config/submit.cfg" for file expression ".*" in area "
^/default/main/nonportal/shared/WORKAREA/.*$": "iw_nonportal_share_admin" is an invalid group name for setting gid
EVENT HANDLER: "/app/iw-home/local/config/submit.cfg"
CASE-SENSITIVE = YES
RULES:
AREA EXPRESSION: "^/default/main/portal/shared/WORKAREA/.*$"
FILE EXPRESSION: ".*"
ACTION: "perm=0775"
AREA EXPRESSION: "^/default/main/portal/iptreference11/shared/WORKAREA/.*$"
FILE EXPRESSION: ".*"
ACTION: "perm=0775"
AREA EXPRESSION: "^/default/main/portal/iptreference11/iptcountry11/shared/WORKAREA/.*$"
FILE EXPRESSION: ".*"
ACTION: "perm=0775"
AREA EXPRESSION: "^/default/main/portal/iptreference11/iptcountry11/lagcb/WORKAREA/.*$"
FILE EXPRESSION: ".*"
ACTION: "perm=0775"
AREA EXPRESSION: "^/default/main/nonportal/shared/WORKAREA/.*$"
FILE EXPRESSION: ".*"
ACTION: "perm=0775"
AREA EXPRESSION: "^/default/main/jfp/asia/irgcb/WORKAREA/.*$"
FILE EXPRESSION: ".*"
ACTION: "gid=iwjfp"
ACTION: "perm=0775"
AREA EXPRESSION: "^/default/main/jfp/asia/pkgcb/WORKAREA/.*$"
FILE EXPRESSION: ".*"
ACTION: "gid=iwjfp"
ACTION: "perm=0775"
New webserver_uid nobody (60001)
New vannotate_uid iwui (102)
[Thu Jul 29 17:11:45 2004] Cache size: 30000
[Thu Jul 29 17:11:45 2004] TEnvironment: fd limit 8192 max 8192
[Thu Jul 29 17:11:45 2004] Store /local/iw-store/workflow: 10 objects found in the log file.
[Thu Jul 29 17:11:45 2004] TMetafileIoEngine::GetHeader: Unable to read index 0x0:Bad file number
[Thu Jul 29 17:11:45 2004] TModification::Commit: WritePoint failed: object ID = 3200, data size = 984
[Thu Jul 29 17:11:45 2004] Store /local/iw-store/workflow: recovery failed, store is disabled.
[Thu Jul 29 17:11:45 2004] Failed to start store manager.
EXITED: Thu Jul 29 17:11:45 IST 2004
#########################################################
/etc/init.d/iw.server start
####################################################
Interwoven TeamSite starting ...
ERROR: Failed to start. Check /var/iwov/iwserver.log

lhdavis

A few questions:

1) Has your TS 5.5.2 server been working and this occurred all of a sudden in the past couple of days? Or is this a new install? My guess is it's the former rather than the latter but wanted clarification.

2) Is your backing store frozen? I would have expected iwstat would have mentioned it but it seems like you were only checking the cache.

3) Has anything been installed on this box over the past week?

4) Did you open a case with Support?

Luke Davis
Open Technology Group, Inc.
luke.davis@med.va.gov

skip11

this is a problem:

Thu Jul 29 17:11:45 2004] Error while reading config file "/app/iw-home/local/config/submit.cfg" for file expression ".*" in area "
^/default/main/portal/shared/WORKAREA/.*$": "iw_portal_share_admin" is an invalid group name for setting gid
[Thu Jul 29 17:11:45 2004] Error while reading config file "/app/iw-home/local/config/submit.cfg" for file expression ".*" in area "
^/default/main/portal/iptreference11/shared/WORKAREA/.*$": "iw_iptref11_share_admin" is an invalid group name for setting gid
[Thu Jul 29 17:11:45 2004] Error while reading config file "/app/iw-home/local/config/submit.cfg" for file expression ".*" in area "
^/default/main/portal/iptreference11/iptcountry11/shared/WORKAREA/.*$": "iw_iptcountry11_share_admin" is an invalid group name for s
etting gid
[Thu Jul 29 17:11:45 2004] Error while reading config file "/app/iw-home/local/config/submit.cfg" for file expression ".*" in area "
^/default/main/portal/iptreference11/iptcountry11/lagcb/WORKAREA/.*$": "iw_iptcountry11_lagcb_users" is an invalid group name for se
tting gid
[Thu Jul 29 17:11:45 2004] Error while reading config file "/app/iw-home/local/config/submit.cfg" for file expression ".*" in area "
^/default/main/nonportal/shared/WORKAREA/.*$": "iw_nonportal_share_admin" is an invalid group name for setting gid

if the files do exist, are they readable ? I moved mine out of the way, but my server started (with the above errors) AND
mounted the backing store. So at this point I'd say the logs are faithfully reporting syntax errors in the .cfg files.

RB

smg

Hi,

I am working with cute_guy to resolve this issue. :-)

I have performed the following operations just to understand the problem in better manner.

STEP 1: Taken the backup of this folder contents /local/iw-store/workflow of System1

STEP 2: Copied the /local/iw-store/workflow folder contents from System2 (TeamSite system) to System1.

STEP 3: # /etc/init.d/iw.server start
Interwoven TeamSite starting ...
#

Now the system1 is up and running with the workflows of system2. (Which is the expected behavior. But not the solution for this issue.)

So, Now my question is

How to repair the existing contents of /local/iw-store/workflow system1 (which is available in backup)?
So that to get the actual workflows of the system1.

Here, I have given the log files difference. (Before and after the above steps)

======================================================================
BEFORE

#/etc/init.d/iw.server start
Interwoven TeamSite starting ...
ERROR: Failed to start. Check /var/iwov/iwserver.log
--------------------------------------------------------------------------------
AFTER

#/etc/init.d/iw.server start
Interwoven TeamSite starting ...
#
======================================================================
BEFORE

#tail -f /var/iwov/iwserver.log
…..
iwstart: Waiting up to 60 seconds for iwserver to come up... TIMED OUT
ERROR:iwstart: iwserver did not come up in 60 seconds
ERROR:iwstart: Please look at /var/iwov/iwtrace.log for more information
ERROR: Failed to start.
--------------------------------------------------------------------------------
AFTER

#tail -f /var/iwov/iwserver.log
…..
iwstart: Waiting up to 60 seconds for iwserver to come up... READY
iwmount: Mounting /iwmnt/default...
iwmount: Mounting /.iwmnt/default...
iwmount: Mounted /iwmnt/default and /.iwmnt/default.
iwevent: Running reporting database
iwevent: Running TS GRC daemon
Interwoven Web Daemon starting ...
Jul 29 17:38:34 IST 2004
tarting ...
/iw-webd/bin/iw.webd start: iwwebd started
iwuiboot: ---------- STARTED: Thu Jul 29 17:39:32 IST 2004
iw.server: ---------- STARTED: Thu Jul 29 17:39:32 IST 2004
======================================================================
BEFORE

#tail -f /var/iwov/iwtrace.log
…..
[Thu Jul 29 17:38:30 2004] Cache size: 30000
[Thu Jul 29 17:38:30 2004] TEnvironment: fd limit 8192 max 8192
[Thu Jul 29 17:38:30 2004] TEnvironment: fd limit 8192 max 8192
[Thu Jul 29 17:38:31 2004] TFSEnvironment: inode cache 30000
[Thu Jul 29 17:38:31 2004] Batch Jobs Thread started
/etc/iw.cfg: rpc_threadcount=64
SCI Base Thread 21 started (max thread count 64)
--------------------------------------------------------------------------------
AFTER

#tail -f /var/iwov/iwtrace.log
…..
[Thu Jul 29 17:11:45 2004] Cache size: 30000
[Thu Jul 29 17:11:45 2004] TEnvironment: fd limit 8192 max 8192
[Thu Jul 29 17:11:45 2004] Store /local/iw-store/workflow: 10 objects found in the log file.
[Thu Jul 29 17:11:45 2004] TMetafileIoEngine::GetHeader: Unable to read index 0x0:Bad file number
[Thu Jul 29 17:11:45 2004] TModification::Commit: WritePoint failed: object ID = 3200, data size = 984
[Thu Jul 29 17:11:45 2004] Store /local/iw-store/workflow: recovery failed, store is disabled.
[Thu Jul 29 17:11:45 2004] Failed to start store manager.
EXITED: Thu Jul 29 17:11:45 IST 2004
======================================================================

Thanks
S. Muthu Ganesh

skip11

check the permissions on the backing store directory tree on the failing system1.

cute_guy

Has your TS 5.5.2 server been working and this occurred all of a sudden in the past couple of days? YES

Is your backing store frozen? NO

Has anything been installed on this box over the past week?NO

Did you open a case with Support?

NO

Migrateduser

I recommend you open a case with support.
Regards,

lissa

Adam Stoller

Either I'm missing something - or everyone else is - the error messages look pretty clear to me:

Do the groups:

iw_portal_share_admin
iw_iptref11_share_admin
iw_iptcountry11_share_admin
iw_iptcountry11_lagcb_users, and
iw_nonportal_share_admin

exist on your TeamSite server in /etc/group (or in LDAP if your server is configured to pull groups from LDAP)?

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

smg

Hi,

We have checked the file permissions (user, group & mode) they were ok.

Thanks
S. Muthu Ganesh

cute_guy

Yes , These groups are present in /etc/group file
[ iw_portal_share_admin,iw_iptref11_share_admin,iw_iptcountry11_share_admin,iw_iptcountry11_lagcb_users, andiw_nonportal_share_admin ]

my 0.02 eurocents ...

while these error messages are weird in light of the fact that user claims that the groups exist, I believe the error messages only indicate that some of the parsing of submit.cfg has failed, and that submit filters for the corresponding lines will not be applied (but the rest of submit.cfg will be applied), I don't think that these messages relate to the non start-up of the iw.server, which seems to relate solely to the corrupted workflow store.

on the whole though, it seems I can't help ;-)