nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

iwmigrate TS 6.1 into 6.7.1 - store is 7x bigger

Bill Klish

TeamSite 6.1 Solaris
TeamSite 6.7.1 Linux

We are in the process of doing some initial iwmigrate tests and I have to admit I am a bit shocked with the results.

Our 6.1 backing store is 14 GB. I copied it over to our 6.7.1 server and did nothing else except start iwmigrate.

After running 1 branch (+ sub branches) through the process (medium size branch with some branches), it took up 8 GB of space. I thought this was way too much.

So I decided to run it through iwfsshrink and this is what happened:
Not currently running.
Last started Thu Feb 15 06:54:47 2007
Last completed Thu Feb 15 08:05:57 2007
Files examined: 81326
Bytes examined: 8390572981
Files found to be duplicates: 76456
Files converted: 42959
Bytes removed: 7339193519

Basically, 7Gb of the 8Gb was waste. And 76000 files were duplicates. So, if I was to migrate my entire store at once, will we need to have 100 GB of space available? Do we need to run iwfsshrink on the first or is this some sort of bug?

Find more posts tagged with

Comments

ccprofessional

Obviously, you are doing something wrong

you killed iwmigrate and ran iwfsshrink?! good luck

gzevin

Obviously, you are doing something wrong

you killed iwmigrate and ran iwfsshrink?! good luck

why is this so obvious to you? or you think that iwmigrate is a tool made in heaven?

ccprofessional

well, you don't kill a data-conversion process before it completes, then run something like iwfsshrink that has the potential to change the partially-f*ked store.

In my book, this is obvious.

Adam Stoller

well, you don't kill a data-conversion process before it completes, then run something like iwfsshrink that has the potential to change the partially-f*ked store.

In my book, this is obvious.

I'm not sure the wording in the original post was meant to indicate that iwmigrate was killed in order to run iwfsshrink - I read it as a subset of the total number of branches to be migrated was fully processed with iwmigrate, and then iwfsshrink was run. That shouldn't cause any problems. (BK - care to clarify?)

As for the data explosion - iwmigrate (and iwconvert before it) have always had this problem - for some reason the re-playing of events causes mass amounts of duplicated data (which can then be cleaned up by iwfsshrink).

A lot depends, I believe, on the number of editions being converted and the number of times a given file was changed in between editions - especially with regard to metadata changes ... though this is all just conjecture.

Based upon the results though, I think you either need to re-think the amount of editions you want to migrate, or be prepared to do it over the course of a bunch of time interchanging runs of iwmigrate with iwfsshrink -- or make sure you have gobs of disk space for the new store so that you can do the iwmigrate in one (or at least fewer) passes before running iwfsshrink, and then look for ways to reduce the size of the partition you used for the store (e.g.: create new partition, copy it over, remove the old partition, rename the new partition into place or change the specification for the path to the store - or, if the system allows for it - just shrink the size of the partition ;-)

Frederik

FYI, KB 57161 - Possible Reasons For Increase in Backing Store Size After Iwmigrate (only available on Support). Although it's written for the upgrade 5.5.2 -> 6.x, I guess it still applies.

... or be prepared to do it over the course of a bunch of time interchanging runs of iwmigrate with iwfsshrink

I would use this option with extreme caution, only for the reason that KB article 55358 warns against activating/changing the store in the middle of incremental migrates (and activation/changing is what iwfsshrink does).

Before doing anything else (especially activating the new store), take a backup of the backing store created by the migration that has just finished. This is required because any changes to the backing store created by the first migration pass will prevent the incremental migrate from working properly.

From my understanding of how iwmigrate and iwfsshrink work, the above seems unnecessarily conservative, and I would be inclined to think that interchanging iwmigrate and iwfsshrink would work. But I can't recommend going against the Support article's wisdom, without extra warning Smiley Wink

BTW, to my knowledge, you can safely "abort" the iwmigrate ( KB 53673 )

Adam Stoller

FYI, KB 57161 - Possible Reasons For Increase in Backing Store Size After Iwmigrate (only available on Support). Although it's written for the upgrade 5.5.2 -> 6.x, I guess it still applies.

[alternate iwmigrate / iwfsshrink]

I would use this option with extreme caution, only for the reason that KB article 55358 warns against activating/changing the store in the middle of incremental migrates (and activation/changing is what iwfsshrink does).

In my mind, incremental migrates means that I migrate branch x, then later migrate branch x again.

What I thought was being discussed was migrating branch x, then later migrating branch y

I don't believe that they are the same things.

Also, though I didn't see the specific warning you mention - if it was there, I believe it means making additional changes (new versions, new editions) on the target server in between incremental migration efforts - not running iwfsshrink, which does not delete versions of files, but instead restructures the backing store such that if a file's change between two versions is due only to metadata changes - then the physical copy of the content is deleted and a reference pointer is put in place that points to the "shared" content and the unique metadata.

Doing this (running iwfsshrink between runs of iwmigrate) should not cause any problems - and I'm sure I've done this without running into any.

From my understanding of how iwmigrate and iwfsshrink work, the above seems unnecessarily conservative, and I would be inclined to think that interchanging iwmigrate and iwfsshrink would work. But I can't recommend going against the Support article's wisdom, without extra warning

BTW, to my knowledge, you can safely "abort" the iwmigrate ( KB 53673 )

Yes, you can abort it, but then the next run wants to pick up from where it left of by using a modified version of the command line (which it will tell you about if you try to use the exact same command after the abortion as you did before.

In order to get around that - there's a file you can remove from the backing store (I think it's on the target side, but perhaps the source side). Sorry, cannot remember the name of the file right now - but if you look, you can usually find it.

Bill Klish

I did let iwmigrate run to completion. It was not aborted, but thanks for jumping to conclusions and telling me I am doing something wrong. I guess I need to be crystal clear in my post next time Smiley Happy

We were just doing this as a simple test to see what would happen. There were quite a few sub branches underneath this main branch as well, not too many files, editions or EAs set on files. It was surprising that it took up so much extra space. We are in the process of testing our most complicated branch (multi-lingual, many editions, lots of files, tons of EAs). This branch never ran to completion when we did the last migration (4.5 to 5.5.2).

As for limiting the number of editions, that is somewhat of a harder sell as for compliance reasons my customer would like to have all their history in case future legal action occurs. So if we just move the whole store over, it is converted during the install process, but not optimized. If we want it optimized we need to pare back the history. Seems like this is something Interwoven should have fixed from the last go round, so this would all just work.

I will let you guys know the results of this huge branch and its sub branches.

Incidentally, the 8GB that was created took only about 2 hours and 20 minutes to process, which I thought was encouraging since this is running inside vmware and only has 2 1 GHZ cpus and 1 GB of memory allocated.

CalculateTimeBetweenRows.rptdesign