adminResponse status="SUSPENDED"

Environment[indent]TS/OD/LS 7.2.1 on Linux 5[/indent]
Issue[indent]Things were working well last week. Sometime over the weekend, one of our two DEV LSCS servers started complaining (in the idol/log/application.log) about a shortage of space.
I restarted idol on that app server (/etc/init.d/iwlscscontent stop & start) and that seemed to have fixed the problem.
Then sometime over lunch - LSCS deployments started failing again - but this time there's nothing in the IDOL log to correspond with it, and restarting the local IDOL server has minimal effect.

When we go to the URL http://appserver:15060/lscs/v1/admin/status we get back:[php]

[/php]At first, the status for Search Server Repository was "DOWN" - but after restarting IDOL on the app server, it went back to being "UP" - however the general adminResponse stays at "SUSPENDED" and we are unable to telnet to the IDOL ports 10010 and 10011 (10010 = searchService.export.port, searchService.query.port, and searchService.status.port; 10011 = searchService.index.port)

We're trying to stop / start a bunch of services - but we either haven't done the right one(s) or we haven't done them in the right order, or something - so I'm looking for some guidance here because I cannot seem to find anything in the documentation (surprise).[/indent]

Find more posts tagged with

Comments

nipper

Is the DB still available ? Any issues/locks on it ?

Adam Stoller

Is the DB still available ? Any issues/locks on it ?

The DB appears to be available (I can access it with SQL Developer)
And there is a HI_VALUE table in it (I saw that mentioned in one of the threads that seemed to come close to my issue).

How can I verify if there are locking issues on the DB?
Or do I need to contact the DBA for that?
If it helps any:[html]

buildDate="2010-October-26 " apiVersion="v1"/>
Mon Oct 22 14:00:59 EDT 2012

11.3992

true
true
/usr/LSCSRT-Store
oracle.jdbc.driver.OracleDriver
ojdbc6.jar
200

jdbc Smiley Surprised

racle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL = TCP)(HOST=dbservername.company.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=INTWVD)))

MM/dd/yyyy
//tsservername.company.com/default/main/WWO/WVO/WVR
xml
org.hibernate.dialect.Oracle10gDialect
/usr/Interwoven/LiveSiteCSRT/runtime/webapps/lscs/WEB-INF/classes/LSCS.lic
1000
500
10
true
true
10010
localhost
10011
localhost
10010
localhost
10010
3600000

application/xml
application/json

application/xml
application/json

[/html]

Rick Poulin

But your database has status "UP".. Your status is saying that only the LSCS webapp is unhappy. I think an expired license is a more likely cause. Either way, your lscs.log should have complaints logged.

Adam Stoller

But your database has status "UP".. Your status is saying that only the LSCS webapp is unhappy. I think an expired license is a more likely cause. Either way, your lscs.log should have complaints logged.

I'm working with Support - they said the same thing - but the license looks like it's okay as far as I can tell (I ran the java command to spit out the info, and I provided Support a copy of the actual license file)
There are some errors in the lscs.log which Support is taking a look at - right now I'm trying to set up a contingency solution with the second of two app servers as the target instead of the both the first and second - and unfortunately, the second was only recently added so I have to get all the content sync'd from TS...)

Bowker

Make sure you have enough disk space.

This is from personal experience. I have not seen this in any manual but to be honest, who reads every word in those manuals.

IDOL will not be happy and will suspend collecting/indexing data unless it has 50% free space on it's indexing drive. We had 70% free and when it started indexing it got to above 50% utilization due to temp files and it suspended indexing.

Please note this is our experience, your's may vary. This doesn't sound like a similar issue, but it may be worth a look.

Adam Stoller

Make sure you have enough disk space.

This is from personal experience. I have not seen this in any manual but to be honest, who reads every word in those manuals.

IDOL will not be happy and will suspend collecting/indexing data unless it has 50% free space on it's indexing drive. We had 70% free and when it started indexing it got to above 50% utilization due to temp files and it suspended indexing.

Please note this is our experience, your's may vary. This doesn't sound like a similar issue, but it may be worth a look.

Did you get any messages in any of the logs when this happened or did IDOL just up and die?

In our case, we start the LSDS app in WebSphere, then we start the LSCS app in WebSphere, then I start IDOL (/etc/init.d/iwlscsscontent start) and everything is "happy".

Then I do an LSCS deployment via a PLC-derivitive workflow and IDOL dies and LSCS goes into a suspended mode. If I restart IDOL, LSCS remains in a suspended mode.

However, the only errors I've noticed thus far is afterIDOL died. I don't see any error messages in any of the log files indicating why it's croaking.

We did just add disk space to the '/' partition - I don't think it's quite 50% free - but then again, before this whole episode it was never anywhere near 50% free and was working just fine.

Adam Stoller

It may seem anti-climactic or even counter-intuitive, but Support's most recent suggestion (after a number of back-and-firths) seems to have done the trick:[indent]Please run IDOL maintenance tool DRECOMPACT and then do a fresh deployment to see if this tool can fix the problem.

Here's the guide:

How to run DRECOMPACT?
DRECOMPACT is a memory/cpu consuming process so it is "strongly" recommended to run this during off business hours so that it doesn't impact your production environment. DRECOMPACT can be issued in the browser using the following URL:
http://localhost:<index-port>/DRECOMPACT
You can check the value of the Index Port in IWLSCSContent.cfg.
When you run this command you will get a INDEXID in the browser. You can check the status of this by doing the following:
Create an output directory in the IDOL folder then execute:
http://localhost:<lscs-port>/lscs/v1/util-command/idolpt?action=indexergetstatus&output=file&filename=./output/myfile.xml
Find the job for your indexid and check it's status periodically to make sure it finished.[/indent]For verifying the status of the process I suggest just running the URL (

http://<lscs-server>:<lscs-port>/lscs/v1/util-command/idolpt?action=indexergetstatus

) in your browser, assuming you can access that port of that server from your browser - my results were:[html]
INDEXERGETSTATUS
SUCCESS

69782
10.53.91.74
2012/11/02 07:45:15
2012/11/02 07:45:16
2012/11/02 08:00:13
897
100
0
0
-1
Finished

/DRECOMPACT

[/html]Since then, my IDOL services stay up and LSCS is now "UP" instead of "SUSPENDED" - I'm doing a bunch of test deployments now but thus far it's looking pretty good.

Rick Poulin

Have they told you what was the suspected problem that DRECOMPACT is resolving? It sounds to me like a "we don't know so try this" kind of answer...

Adam Stoller

Have they told you what was the suspected problem that DRECOMPACT is resolving? It sounds to me like a "we don't know so try this" kind of answer...

No, at least not yet. If it's still working Monday, I'll try to get that information out of them...

Adam Stoller

Here's the explanation:

For your issue, you keep getting idol fail right after a deployment and the all the logs that you submit showed that idol on runtime was refusing connection. So that gives me a clue that your idol may be the issue.

I looked at all your runtime systemout logs. It doesn't tell me much but keep trying to connect to idol(even after you enabled the debug mode).

However, when you did a idolpt?action=GetStatus, I saw :[php]31544
9240575[/php]The "committed documents" should be as same as "document sections".
In your case, yours are not.

To me, it seems you guys didn't do idol maintenance much or even never did it.

So at this point, I thought the issue might be something related to idol's database or index due to the poor maintenance.

So I went ahead and gave you the guide to run that to see if that helps.

Personally, I don't have the expertise to explain how idol works, but I think the last time you successfully deploy a file into LSCS, which relates to idol on runtime. It "jammed" up idol and thus can't accept incoming new deployments. To solve that, you need to run DRECOMPACT to let idol solve it.

An honest and informative answer IMO - it was a confusing issue, but ultimately they came through - which is what mattered to me.

I'm not sure what regular IDOL maintenance is recommended, but for the new servers that we're setting up I'm going to make sure that DRECOMPACT is scheduled to run regularly (there's a KB for that - just search for "DRECOMPACT" and you should be able to find it (the name of the config file is different, but everything else should be the same).