TeamSite fails to startup - segmentation fault error

Hi All,

We are facing an issue, wherein TeamSite just fails to start. iwserver log states:

iwstart: Loading pmfs kernel module...
iwstart: Starting iwutildstart script (User: root)...
iwstart: Starting iwauthdstart script...
iwstart: Spawning iwserver.linux (trace file: /usr1/app/interwoven/TeamSite/local/logs/iwtrace.log)...
iwstart: Waiting up to 60 seconds for iwserver to come up...\c
no element found at line 1^M
bash: line 1: 4755 Segmentation fault iwserver.linux -e "/usr1/app/interwoven/TeamSite/local/logs/iwevents.log" /usr1/app/interwoven/iw-store
TIMED OUT
iwstart: ERROR: iwserver did not come up in 60 seconds
iwstart: ERROR: Please look at /usr1/app/interwoven/TeamSite/local/logs/iwtrace.log for more information
ERROR: Failed to start.

While pmfs kernel module error resolved after we re-booted the system (This has happened earlier also), but we are unable to know the cause of "segmentation fault" error.

We checked the store health also:
iwfsck returns:
Command line: /usr1/app/interwoven/TeamSite/private/bin/iwfsck
iwfsck: 7.1.0 Build 1255 Interwoven
Format for type 'BtreeMapLeafParentRecordPointID' isn't defined
Format for type 'BtreeNonLeafParentRecordParentRecord' isn't defined
Format for type 'BtreeNonLeafParentRecordPointID' isn't defined
Format for type 'BtreeSetLeafParentRecord' isn't defined
Format for type 'DependencyRecord' isn't defined
Error: The FORMAT directory is missing files. Please try -nofmtchk option.And when run with nofmtchk option, it finishes with 0 errors:

Total Run time: 2 secs -> 2 Secs
FINISHED 0 error(s) found

Any help in this regard is much appreciated.

Thanks
Payal

Environment:
TS 7.1
RHEL 5.4

Find more posts tagged with

Comments

interDev

Have you checked iwtrace logs?

PayalGarg

iwtrace logs:

[Thu Jan 26 02:02:39 2012] iwserver: 7.1.0 Build 1255 Interwoven
ERROR: no valid submit config file found in the current configuration (/usr1/app/interwoven/TeamSite/conf/events/submit.conf, /usr1/app/interwoven/TeamSite/local/config/submit.cfg, /usr1/app/interwoven/TeamSite/config/submit.cfg, /usr1/app/interwoven/TeamSite/local/config/submit.cfg) (2)
ERROR: no valid autoprivate file found in the current configuration (/usr1/app/interwoven/TeamSite/conf/events/autoprv.txt, /usr1/app/interwoven/TeamSite/local/config/autoprivate.cfg, /usr1/app/interwoven/TeamSite/config/autoprivate.cfg, /usr1/app/interwoven/TeamSite/local/config/autoprivate.cfg) (2)
[Thu Jan 26 02:02:39 2012] *** IWServer Locale is --> English_UnitedStates.Latin1@Default
[Thu Jan 26 02:02:39 2012] IW Global User = iwguser and ID = 20027
[Thu Jan 26 02:02:39 2012] IW Global Group = iwglobal and ID = 20027
[Thu Jan 26 02:02:39 2012] Reading userops file /usr1/app/interwoven/TeamSite/private/etc/userops.xml ...
[Thu Jan 26 02:02:39 2012] Done reading userops file /usr1/app/interwoven/TeamSite/private/etc/userops.xml
[Thu Jan 26 02:02:39 2012] Reading userops file /usr1/app/interwoven/TeamSite/private/etc/userops_ui.xml ...
[Thu Jan 26 02:02:39 2012] Done reading userops file /usr1/app/interwoven/TeamSite/private/etc/userops_ui.xml
[Thu Jan 26 02:02:39 2012] Reading users and groups...
[Thu Jan 26 02:02:39 2012] Loading disk cache from Wed Jan 25 20:14:33 2012, revision = 1.
[Thu Jan 26 02:02:39 2012] done. 143 users and 57 groups read.
[Thu Jan 26 02:02:39 2012] [iwserver] branch_default_perm=775
[Thu Jan 26 02:02:39 2012] [iwserver] directory_default_perm=775
[Thu Jan 26 02:02:39 2012] [iwserver] file_default_perm=664
[Thu Jan 26 02:02:39 2012] [iwserver] workarea_default_perm=775
[Thu Jan 26 02:02:39 2012] New webserver_uid nobody (99)
[Thu Jan 26 02:02:39 2012] Log Permissions Disabled
[Thu Jan 26 02:02:39 2012] Log Authentication Disabled
[Thu Jan 26 02:02:39 2012] User database enabled
[Thu Jan 26 02:02:39 2012] Log Config Mirror Disabled
[Thu Jan 26 02:02:39 2012] Login Daemon Enabled
[Thu Jan 26 02:02:39 2012] Group Buffer size set to 8192
[Thu Jan 26 02:02:39 2012] group_list_update_interval is set to 10 minutes
[Thu Jan 26 02:02:39 2012] Ldap user data cache thread delay = 1440
[Thu Jan 26 02:02:39 2012] Cache size: 30000
[Thu Jan 26 02:02:39 2012] setting rollover threshold to 10485760
[Thu Jan 26 02:02:39 2012] setting num rollovers to 10
[Thu Jan 26 02:02:39 2012] TEnvironment: fd limit 1024 max 1024
[Thu Jan 26 02:02:39 2012] Initialized store 0x1 at "/usr1/app/interwoven/iw-store/workflow" (version=300011, next id=0xef100).
[Thu Jan 26 02:02:39 2012] TEnvironment: fd limit 1024 max 1024
[Thu Jan 26 02:02:39 2012] Initialized store 0x64 at "/usr1/app/interwoven/iw-store/default" (version=300011, next id=0x5a00).
[Thu Jan 26 02:02:39 2012] Utility Daemon Enabled
[Thu Jan 26 02:02:39 2012] TFSEnvironment: inode cache 30000
[Thu Jan 26 02:02:39 2012] Inode table was not shut down properly.
[Thu Jan 26 02:02:39 2012] Creating new inode tables.
[Thu Jan 26 02:02:39 2012] TEnvironment: fd limit 1024 max 1024
[Thu Jan 26 02:02:39 2012] Initialized store 0x65 at "/usr1/app/interwoven/iw-store/iwadmin" (version=300011, next id=0x66500).
[Thu Jan 26 02:02:39 2012] TFSEnvironment: inode cache 30000
[Thu Jan 26 02:02:39 2012] Inode table was not shut down properly.
[Thu Jan 26 02:02:39 2012] Creating new inode tables.
[Thu Jan 26 02:02:39 2012] TEnvironment: fd limit 1024 max 1024
[Thu Jan 26 02:02:39 2012] Initialized store 0x67 at "/usr1/app/interwoven/iw-store/hip" (version=300011, next id=0x1770800).
[Thu Jan 26 02:02:39 2012] TFSEnvironment: inode cache 30000
[Thu Jan 26 02:02:39 2012] Inode table was not shut down properly.
[Thu Jan 26 02:02:39 2012] Creating new inode tables.
[Thu Jan 26 02:02:39 2012] TEnvironment: fd limit 1024 max 1024
[Thu Jan 26 02:02:39 2012] Initialized store 0x68 at "/usr1/app/interwoven/iw-store/elt" (version=300011, next id=0x130d600).
[Thu Jan 26 02:02:39 2012] TBkgJobsSet::Cleanup Job 0x130b98c cancelled
EXITED: \c
Thu Jan 26 02:02:43 UTC 2012

In these logs, errors related to submit.cfg and autoprivate.cfg can be ignored, because, we are having those errors in the logs as always..

Thanks
Payal

nipper

Can you vi iwhome/private/bin/iwstart ?

It *seems* as if you have a DOS formatted file (the control-M), which messes up a script's shebang line.

VI will show that to you

Adam Stoller

Can you vi iwhome/private/bin/iwstart ?

It *seems* as if you have a DOS formatted file (the control-M), which messes up a script's shebang line.

VI will show that to you

So, usually, will cat -vte file

PayalGarg

Thanks for your replies.

/iwhome/TeamSite/private/bin/iwstart is not having any ^M characters. Also this file was last changed in 2009.

To resolve this issue, we had to re-build our TS content store from some other server. We were not able to get the previous one back.

IWOV support suggest us to increase cachesize:
"

[color=navy][font=Arial]The error means that TS has run out of available cache slots to store data points in memory. This indicates that your cachesize setting may be too low. The current setting in iw.cfg is:" />

[/color][/font]

[color=navy][font=Arial]cachesize=30000"

[/color][/font]

Thanks
Payal