iwtrace.log getting maxed out with Checksum errors

Our iwtrace.log grew to almost 8 gig with tons of the below errors...devnet search didnt turn up much, and the logger setting was set to default in /iw-cc/command/iw.base.loggers - i couldnt find any other log files that controlled debug settings (i.e. error vs fatal vs debug) for the iwtrace.log.

performance on the server has been incredibly slow...any help would be greatly appreciated

[Tue Dec 01 21:18:56 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_
[Tue Dec 01 21:18:56 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:56 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:56 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:56 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:56 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

[Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a [Tue Dec 01 21:18:57 2009] Restore(0xc5d3a) failed, internal error 2 [Tue Dec 01 21:18:57 2009] Checksum failure for e:\iw-store\iwadmin\d0\dc\d5d\f3a_

Find more posts tagged with

Comments

Adam Stoller

I suggest you stop the server, run iwfsck -d on the store, and open a case with Support with the output from that command.

Migrateduser

Thanks as always Fish, we will try that tonight hopefully off hours so there are no delays with the support case. the log is still getting hammered this morning...

[Wed Dec 02 08:40:30 2009] Checksum failure for e:\iw-store\iwadmin\d0\dd\dce\fd2_
[Wed Dec 02 08:40:30 2009] Checksum failure for e:\iw-store\iwadmin\d0\dd\dce\fd2
[Wed Dec 02 08:40:30 2009] Restore(0xdced2) failed, internal error 2

nipper

Thanks as always Fish, we will try that tonight hopefully off hours so there are no delays with the support case. the log is still getting hammered this morning...

Good luck with that. I got a frantic call from an old customer, they have a P1 case open with support & were told to wait in line with everyone else.

Migrateduser

iwfsck results attached - im not sure our it was ran properly (our IT guy did a freeze before hand) as i see:

\Interwoven\TeamSite\private\bin>iwfsck -d
Command line: iwfsck -d
iwfsck: 6.7.2 Build 69622 Interwoven 20080818
[Wed Dec 02 21:51:34 2009] Backing store e:\iw-store\default is currently locked by process 7372 on CSCMSADM01
The backing store is locked by another process. For example, the TeamSite server may be running.
Reported results may not be accurate.

nipper

When was the last time the server was rebooted ? That probably won't make much of a difference, but you should try it.

The iwfsck results do not look good. Try to access some of the files referenced with the errors, are they complete ? If they are corrupt, how about the version in Staging.

Adam Stoller

iwfsck results attached - im not sure our it was ran properly (our IT guy did a freeze before hand) as i see:

\Interwoven\TeamSite\private\bin>iwfsck -d
Command line: iwfsck -d
iwfsck: 6.7.2 Build 69622 Interwoven 20080818
[Wed Dec 02 21:51:34 2009] Backing store e:\iw-store\default is currently locked by process 7372 on CSCMSADM01
The backing store is locked by another process. For example, the TeamSite server may be running.
Reported results may not be accurate.

That message you can pretty much ignore - it's true - but it's not particularly relevant to the information contained in the log file.

As nipper said - the log does not look good. You should open a case with Support immediately and send them that log file

Migrateduser

thanks guys - we have our ticket in with the results, but as you guessed, its in support limbo and hasnt been assigned yet. we are clearing the iwtrace.log out daily but its already at 320meg by EOD business

from my experience, checksum errors usually happen when building my own PCS and are bios/motherboard related. i know its a shot in the dark, but any chance this is hardware/server related on the TeamSite prod box?

nipper

from my experience, checksum errors usually happen when building my own PCS and are bios/motherboard related. i know its a shot in the dark, but any chance this is hardware/server related on the TeamSite prod box?

I am certain IWOV would like to blame it on the hardware. While it *could* be a disk problem, it is certainly not bios/motherboard. But you can replace the hardware and ship it to me if it makes you feel better.

The checksum is also used in many applications for data consistency. Performing a full set of disk diagnostics would not be a bad idea. Support will probably want that anyway.

Migrateduser

ahh you guys as usual were right, this time about support! as for disk diagnostics, you mean a full scandisk w/ sector check and a defrag?

Cases are worked in the order in which they are received. At the moment all engineers are currently working with other customers. As they free up they are taking the next issue. We are working as quickly as possible to address all the issue including yours.

Thanks for your understanding,

Chris Loesch
Customer Interaction Engineer
Autonomy, Inc.

nipper

ahh you guys as usual were right, this time about support! as for disk diagnostics, you mean a full scandisk w/ sector check and a defrag?

Yep. & BTW, that is the email a customer of mine received as well.

BirtEx.zip

img1.png

img2.png

Migrateduser

Chkdsk complete on our prod box-- it took about 2 hours. It doesn't look there were any major errors. We cleared out the iwtrace, and its still getting hammered...so the chkdsk didnt help

Checking file system on E:
The type of the file system is NTFS.
Volume label is TeamSiteStore.

A disk check has been scheduled.
Windows will now check the disk.
Cleaning up minor inconsistencies on the drive.
CHKDSK is recovering lost files.
Recovering orphaned file f32_ (692776) into directory file 691787.
Recovering orphaned file f46 (692777) into directory file 691787.
Recovering orphaned file f34_ (692778) into directory file 691787.
Recovering orphaned file f48 (692779) into directory file 691787.
Recovering orphaned file f35_ (692780) into directory file 691787.
Recovering orphaned file f49 (692781) into directory file 691787.
Recovering orphaned file f38_ (692782) into directory file 691787.
Recovering orphaned file f4c (692783) into directory file 691787.
Recovering orphaned file f46_ (692784) into directory file 691787.
Recovering orphaned file f48_ (692785) into directory file 691787.
Recovering orphaned file f5d (692786) into directory file 691787.
Recovering orphaned file f49_ (692787) into directory file 691787.
Recovering orphaned file f5e (692788) into directory file 691787.
Recovering orphaned file f4c_ (692789) into directory file 691787.
Cleaning up 15 unused index entries from index $SII of file 0x9.
Cleaning up 15 unused index entries from index $SDH of file 0x9.
Cleaning up 15 unused security descriptors.
CHKDSK is verifying file data (stage 4 of 5)...
File data verification completed.
CHKDSK is verifying free space (stage 5 of 5)...
Free space verification is complete.
Windows has made corrections to the file system.

Adam Stoller

Until you get the backing store fixed you're probably not going to get rid of this problem.

This is one of the reasons I recommend customers run iwfsck -d at least monthly if not more frequently -- the sooner you find a problem cropping up, the easier it is to get it fixed. Doesn't help you "now" - but hopefully once you get this fixed it will help you down the road...

nipper

This is one of the reasons I recommend customers run iwfsck -d at least monthly if not more frequently -- the sooner you find a problem cropping up, the easier it is to get it fixed. Doesn't help you "now" - but hopefully once you get this fixed it will help you down the road...

Thank you Mr Hindsight. Smiley Very Happy

Migrateduser

Well we finally heard from back from IWOV, their "recommendation" is to restore the entire

//CSCMSADM01/iwadmin

branch from tape backup before started seeing the checksum errors...among the many issues i see with this are

1) we will lose all our current workflow instances
2) this is an OOTB installed directory/branch, im not sure its as simple as a dropping in the old tape backup and everything will be OK
3) due to the size at which the log grew (at 8 gig it was obviously un-openable), we are unable to pinpoint 100% when the errors started

thoughts? suggestions?

nipper

Well we finally heard from back from IWOV, their "recommendation" is to restore the entire

//CSCMSADM01/iwadmin

branch from tape backup before started seeing the checksum errors...among the many issues i see with this are

1) we will lose all our current workflow instances
2) this is an OOTB installed directory/branch, im not sure its as simple as a dropping in the old tape backup and everything will be OK
3) due to the size at which the log grew (at 8 gig it was obviously un-openable), we are unable to pinpoint 100% when the errors started

thoughts? suggestions?

Get rid of that stupid pink bear.

Smiley Very Happy

Take a backup of everything in the admin drive. You really do not need the history, just the latest changes (or maybe WORKAREA and STAGING)

I would write a little perl script (I love perl) using Archive::Zip and have it take every branch, every staging area and every workarea and create a ZIP file.

I do not know if you can delete the whole iwadmin store and reinstall it (maybe IWOV can point you to a way to make it automatically)

Do you use SitePublisher ? That is the only reason I can think of to keep the history.

HTH

Andy