Migrate Content Job

kjimenez · June 8, 2009

Hello,

I am migratign a large amount of content from one file store to another using the Migrate Content Admin job in Documentum Webtop.

However, when I run this job it makes Webtop very unstable. I would like to schedule this job to run in the evenings, migrating a month of data at a time. I have data going back as far as 2007. Can anyone steer me in the right direction to create this? Also, do you think it is the fact I am running the job from webtop that makes webtop unstable, or is it that I am moving data on the store and the SQL server? We have users on almost 24 hrs a day and I do not want to inconveniance them. Thanks.

karnam · June 8, 2009

Do you mean to say you are using MIGRATE CONTENT Admin method to migrate content from one store to another store, in that case you cannot schedule it, since it is a method. You can use Migration policies to migrate content using either DQL query or a particular type to a target store. Migration polices allows you to schedule it so that you can run the job during off peak hours. Migration polices can be created using Document Administrator.

HTH.

venkas6 · June 8, 2009

Hello,

Content Server supports three ways to move content files.

1. MIGRATE_CONTENT administration method
This method allows you to move one content file or multiple content files.

MIGRATE_CONTENT lets you move files from a file store, ca store, blobstore or
distributed store.

2. Content migration policies
Content migration policies are implemented as jobs. The jobs automate the
movement of content files from one storage area to another.

You can specify the storage area to which you want to move the selected content files.

You can specify the selection criteria for the content to be moved.

or How many content files you want to move in one execution.

or The maximum number of files to move in one execution.

3. Records migration jobs
A records migration job moves batches of content files based on criteria you define.

Records migration jobs are an alternate way to move a large number of content files if
you do not want to execute MIGRATE_CONTENT manually or if you do not have a
Content Storage Services license.

Based on either of these options, you can schedule the jobs so that the application doesn't become very unstable.

For more info, please look @ Content Server Administration Guide which provides more details on the same.

Thanks,

Sekhar

April · June 9, 2009

Simply, what about using either IDAL32 or IAPI32, saved command as a batch file and scehduled it on Windows Scheduled Job? It will do the job.

kjimenez · June 22, 2009

I have reviewed these possible solutions. I think a migration policy is the best solution. However, I am a little confused with one thing. I went to create the job, and I picked the store to move documents to, but it looks like it is trying to make me select document type, but we have a lot of custom types.

I also don't see how to add a date value in as a query.

For instance if I wanted to migrate ALL objects from a store called secure to a store called secure02 older than 6 months. How would I create that job? The Admin Guide is very vague.

venkas6 · June 23, 2009

You can use the selection criteria in the rules section of migration policy job to migrate documents from one store to another.

You would have an option of migrating objects older than months based on creation date, modify date or access date of the document.

Or you could specify a DQL predicate that could be a valid where clause to migrate specific objects older than 6 months.

Thanks,

Sekhar

AnithaSeshadri · June 23, 2009

I think the MIGRATE_CONTENT method could be used to move all the objects from one filestore to another filestore.

For eg : You can do the follwoing in IAPI:

API> apply,c,NULL,MIGRATE_CONTENT,SOURCE_STORE,S,fl1,TARGET_STORE,S,filestore_01,LOG_FILE,S,C:\temp\gen1.log,REMOVE_ORIGINAL,B,T

This query moves all the objects from filstore fl1 to filstore_01. The log file identifies the messages generated by the method.

You can even specify a query to move objects based on a criteria:
eg : API> apply,c,NULL,MIGRATE_CONTENT,TARGET_STORE,S,fl1,QUERY,S,r_creation_date > DATE('23.06.200900:00:01'),SYSOBJECT_QUERY,B,T,LOG_FILE,S,C:\temp\gen.log,REMOVE_ORIGINAL,B,F

The above query moves all objects that were created after 23.06.2009 from filestore filestore_01 to target store fl1.

Please see the dql quide for more information and other options for MIGRATE_CONTENT method.I hope this helps.

Thanks,

Anitha

kjimenez · June 23, 2009

I am pretty novice with Documentum. When I go into the rules section of the job, I do not see where to input a query? It looks like I have to select a query from a drop down. Do I have to write and save the query elsewhere?

kjimenez · June 23, 2009

Also, besides date, I need to do this in small batches because users are on the system 24/7.

Chaithanya · June 24, 2009

If you wan't to migrate the content in batches you can do it by creating seperate migration policies for each and every batch that needs to migrated(migration rules are defined such that each and every migration policy picks up small set of content for migration) and can schedule at different timings. Also you are saying that you have lot of custom types and you want migrate a small set of content in each run, then the best solution is to create a migration policy corresponding to each custom type and specifying the where clause of the query which selects the set of objects of that type needs to be migrated.

I will take a simple example and explain in detail how it can be done, lets say we have there custom types type1, type2 type3 and content corresponding to these types needs to be migrated in batches such that type1 related content gets migrated in batch1, type2 related content needs to migrated in batch2, subset of type3 related content gets migrated in batch3. In order to achieve the above behavior follow the steps below.

-> Go to Migration Policies node under storemanagement in DA and create a new migration policy,for each migration policy configure the migration rules as explained below

    ->Go to Rules Tab in migration policy properites container and select the "DQL Query selection radio button".There are options to configure the migration policies from DA one is simple selection and other is DQL query selection(It was labelled as custom selection before), the simple selection allows one to configure the migration rules by using pre defined rule options provided in DA. Where as using the DQL query selection option one can specify instance of particular type that needs to be migratied as well as the custom query that selects subset instance associated with that type.

    ->As you are planning to specify custom query such that small set of content gets migrated on each run of migration policy it is recommened to use the DQL query option.

   ->Referring to above example cited by me if you want to create migration policy rule such that content related to instances of type1 gets migrated follow the below steps

        ->Select custom type that needs to be considered by clicking the select Link next to "Move specified option" and provide the where clause of the custom query in the text below under "Where" label. For the DQL query selection it is mandatory to specify the where clause. i.e if you want to select the objects whose creation date is before 04/19/2009 date then content of the where clause text box goes as r_creation_date <= DATE('04/19/2009','mm/dd/yyyy')

Also you have other options where you can limit the number of objects that can be migrated, and also can configure the batch size where you can set of size the batch that gets migrated in one go.

kjimenez · June 24, 2009

I do not see the migration policies tab. I am using an older version of DA. 5.3.

Is it different in this version? I can get to Records Migration Job under job management, but This doesn't even ask me to pass arguments.

If I create a normal job and use the dm_MoveContent method it asks me to pass arguments so I think this would work, but I have no idea how to pass arguments to do what I want it to do.

Chaithanya · June 25, 2009

If you have contente storage services license enabled on the repository you will see migration policies node under storemanegement, just go that node rather than job management node. You can create migration policies only if the content storage service license is enabled. If you are using 5.3 sp5 and above you will see the custom type changes.

kjimenez · June 25, 2009

I think I do not have the content storage services license. How do I check this? Content Server was installed long before I have workedc on this.

Chaithanya · June 26, 2009

it is simple if you don't see migration policies and assignment policies nodes under storagemanagement in DA when you connect to corresponding repository, then content storage services is not enabled in the repository. You need CSS licence enabled in the repository to create and execute migration poicies.

Pie · June 26, 2009

If you are still having issues, I'm migrating about 8.5 Terabytes of content from one file store to multiple file stores. I expect to be doing this for a while, but it is working fine. Right now I am 3TB into the process and I have almost filled one of my target filestores.

I run everything in RepoInt running on the Content Server directly. The command (with different filestore and object type names) that is running is:

Execute MIGRATE_CONTENT With
target_store='filestore_new',
max_migrate_count=100000,
parallel_degree=10,
remove_original=True,
sysobject_query=True,
log_file='c:\temp\ContentMoveLog37.tst',
query='r_object_id in (select r_object_id from custom_document_type where a_storage_type = ''filestore_01'')'

As you can see, I am on my 37th batch. I have played with the max_migrate_count to get it to run how long I desire. With my content profile, the 100,000 documents will take me 10 hours, but your mileage will vary. I would migrate 10,000 items first and time it. That 10K will be a longer 10K on average, but it will give you a baseline.

Only run one of these queries at a time. I had some minor issues. I think I could do it safely, but in general, one at a time should allow you to monitor progress quite readily.

In general, I am trying to move static images from my primary filestore to a new filestore that is designed to better handle my static content. This allows my primary content filestore to focus on just the basic Documentum reports and miscellaneous standard content.

I hope this helps.

-Pie

kjimenez · July 20, 2009

I never used repoint before...

This sounds like a good idea though. I am just a little wary. This basically runs the same migration job via command line right?

So far I have moved 20GB of data only in batches of 2k files. However if I do 5k it causes latency but I am doing it via webtop so this may be why. Do you experience any latency?

Pie · July 20, 2009

Running it through Webtop isn't a good idea as the process takes time and it could casue issues like latency on your web application. I would run it on the Content Server through RepoInt or the DQL tool. I've had great success running it on RepoInt on the server with 10 threads at a time. I've migrated over 3 million objects with this method and can process about 10K/hour running 10 processes in the Migration query.

-Pie

kjimenez · July 23, 2009

I am not finding the file DFC.jar, so I am not sure which directory the DFC Jar folder is found in. I have never used anything but Content Server. Can you give me any tips in installing this? Thanks.

kjimenez · July 23, 2009

Nervermind, I figured itout.

kjimenez · July 28, 2009

Ok, I have it all setup and ready to run the query, and I tried to add in a date value because we want to keep at leat 6 months of data on the current store. Here is the query I wrote. Can you tell me what is wrong with the query value?

Execute MIGRATE_CONTENT With

target_store='secure_02',

max_migrate_count=1000,

parallel_degree=10,

remove_original=True,

sysobject_query=True,

log_file='C:\MigrationLogs\migrate.txt',

query='a_storage_type="Secure" and r_creation_date<"31-December-07"'

Thanks.

AnithaSeshadri · July 28, 2009

Can you let us know what the error is?What is the outcome of this DQL?

Thanks,

Anitha

kjimenez · July 28, 2009

Query Error:[DM_API_E_NOQUERY]error: "No query has been run. Cannot use a query collection id."

AnithaSeshadri · July 28, 2009

execute migrate_content with target_store='store1' , query='a_storage_type=''Secure'' and r_creation_date < DATE(31-December-07)',sysobject_query=1,type_to_query='t1_document',max_migrate_count=500,log_file='/tmp/mig/mig3/mig.log'

Can you try with the above query?

Thanks,

Anitha

kjimenez · July 28, 2009

I receive the same error:

Query Error:[DM_API_E_NOQUERY]error: "No query has been run. Cannot use a query collection id."

kjimenez · July 28, 2009

Now I tried the following query:

Execute MIGRATE_CONTENT With

target_store='secure_02',

max_migrate_count=100,

parallel_degree=10,

remove_original=True,

sysobject_query=True,

log_file='C:\MigrationLogs\migrate.txt',

query='r_object_id in (select r_object_id from custom_type where a_storage_type=''Secure'')'

It does not error out but it says"Executing Query" and stays there. I don't think 100 documents should take longer than 2 or 3 minutes but according to the log it is moving them

My goal here is that I want to move all documents from Secure to secure_02. I need to use a date range as well, but I ran this query just to see if I can successfully move docs at all.

If anyone has a test enviornment and is able to take this last query and add a date value let me know Thanks everyone, this has been a great help.

chaitk · July 28, 2009

To add date condition, you can try the below query:

query='r_object_id in (select r_object_id from custom_type where a_storage_type='Secure' and r_creation_date < DATE('12/31/2007','mm/dd/yyyy'))'

Pie · July 28, 2009

I might be able to try out a date range tomorrow on my ongoing migration. I'll let you know.

As to why it is taking a few minutes, it takes a while to actually run the query and get the list together. I find that on my system, mind you there are a lot of objects, it takes 10+ minutes to run the query sometimes. So don't worry about the time. The larger the batches, the better the average processing time will be for you.

-Pie

Pie · July 29, 2009

I tried with a Date condition. Try this:

Execute migrate_content With
source_store='Secure',
target_store='secure_02',
max_migrate_count=40000,
parallel_degree=10,
remove_original=True,
sysobject_query=True,
log_file='c:\temp\ContentMoveLog72.tst',
query='r_object_id in (select r_object_id from custom_type where r_creation_date >''1/11/2008'' and r_creation_date <''1/13/2008'')'

This moved documents created on 1/11/2008 and 1/12/2008 to the new storage. The key difference is that I added source_store as a parameter and then putin date parameters. I find that the "query" field seems to have a length limit and you need to keep it shorter. Moving the source_store as a parameter allows you that room.

-Pie

YingXing · November 28, 2011

Laurence,

You thread is very helpful for me to prepare my TCS migration. I have a couple of questions for you:

1. How long did it take take you to migrate a batch (40000 objects)?

2. With the query specified, were all the versions of the objects moved to the new store?

Thank you

Ying

DavidHaveman · November 12, 2012

With the "query", what if you specify the target_store as "A" but don't have a filter for things that are in "A" already? will documents in target_store be retrieved and "migrated" as well or is the job smart enough to exclude documents already in the target file_store?

( curious for Documentum 5.3 here.. )

YingXing · November 13, 2012

The following is the query gets executed for the migration job in our process:

EXECUTE migrate_content WITH target_store='my_target_store',
renditions='all',remove_original=true,max_migrate_count=1000,
batch_size=100,sysobject_query=true,type_to_query='my_custom_type',
query='FOLDER(''/my_cabinet/28'', DESCEND)
AND a_storage_type=''filestore_01''
AND r_lock_owner='' '' and r_modify_date>=DATE(''07/15/2011'')
and r_modify_date<DATE(''07/01/2012'')',
log_file='D:/tcs_migration/logs/REGIONS/27/TCS_MIGRATION_11082012_100453_REGION_28_01.log'

You can specify in the query to migrate only the document stored in the source store (bold above).

Migrate Content Job

Comments

Categories