nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

[DFC] Folder creation in multi-thread environment

vincentklock

We have a method which will recursively create docbase folder, base on some parameters received as input.

This method runs well in mono-thread environment, but when coming to multi-thread - which is required accounting for the needed throughput, as our method checks the existence of the folder, then tries creating it we may get the following error:


Error: [DM_FOLDER_E_PATH_EXISTS]error:  "Cannot save (or link) 'MYFOLDER' folder with path name '/Client Documents/2,013/11/MYPARENT/MYFOLDER' because one already exists." 
Exception: DfException:: THREAD: pool-1-thread-5; MSG: [DM_FOLDER_E_PATH_EXISTS]

We thought about 2 ways out:

synchronize the whole method, which is possible but implies a performance cost
catch this exception when it occurs, parse the message for the key of this error and, if found, return the existing folder

I assume this kind of problem is frequent when working with business objects and service mode, so does anyone know of a better way to handle it?

Please find below the code of our method:


// DocumentFolder object is a bean which defines the Path, the acl name and the acl domain
public String getOrCreateFolder(DocumentFolder folderToCreate) throws DfException {
    IDfSession session = null;
    try {
        session = dctmDataSource.getSession();
        String path = folderToCreate.getPath();
        IDfId folderId = session.getIdByQualification("dm_folder WHERE ANY r_folder_path = '" + path + "'");
        if (!folderId.isNull()) {
            LOG.debug("Documentum folder {} was found. ID is {}", path, folderId.toString());
            return folderId.getId();
        }
        LOG.debug("Documentum folder {} was not found. Creating path recursively.", path);

        // Create folder hierarchy
        String parentFolderName = "";
        StringBuilder pathBuilder = new StringBuilder();
        boolean isCabinet = true;

        if (path.startsWith("/") && path.length() > 1) {
            path = path.substring(1);
        }

        String[] folderNames = path.split("/");
        for (String folderName : folderNames) {
            pathBuilder.append("/");
            pathBuilder.append(folderName);

            folderId = session.getIdByQualification("dm_folder WHERE ANY r_folder_path = '" + pathBuilder + "'");
            if (folderId.isNull()) {
                LOG.info("Folder {} does not exist. Creating it..", pathBuilder.toString());

                IDfFolder folder = null;
                if (isCabinet) {
                    // The root folder is a cabinet. Everything below is a
                    // folder.
                    folder = (IDfFolder) session.newObject("dm_cabinet");
                } else {
                    folder = (IDfFolder) session.newObject("dm_folder");
                    folder.link(parentFolderName);
                }

                folder.setObjectName(folderName);

                if (!StringUtils.isBlank(folderToCreate.getAclName())
                    && !StringUtils.isBlank(folderToCreate.getAclDomain())) {
                    folder.setACLName(folderToCreate.getAclName());
                    folder.setACLDomain(folderToCreate.getAclDomain());
                }

                folder.save();
                folderId = folder.getObjectId();
            } else {
                LOG.debug("Folder {} already exist. Skipping creation.", pathBuilder.toString());
            }

            parentFolderName = pathBuilder.toString();
            isCabinet = false;
        }

        return folderId.getId();

    } finally {
        if (session != null) {
            dctmDataSource.releaseSession(session);
        }
    }
}

Find more posts tagged with

Note:

If there are Accepted Answers, those will be shown by default. You can switch to 'All Replies' by selecting the tab below.

Button

Accepted answers

Haroon_A

Once I had to write a multi-threaded utility to migrate content from one environment to another. In the process i had to decide whether to create the destination folder or not.

I ended up synchronizing the createFolder method. Synchronizing the method will reduce the performance, but try to put the minimum logic in your synchronized method. i.e. verify if a folder already exist, create if not. Try obtaining your session, getting the name of your folder, etc... outside your synchronize method.

public static synchronized IDfFolder verifyAndCreateFolder (IDfSession s, String folderName) {

if folder exist then

return folder

else

create folder

return folder

}

All comments

PanfilovAB

Smth. like:


private static IDfFolder createFolder(IDfSession session, String path)
    throws DfException {
    try {
        IDfFolder folder = session.getFolderByPath(path);
        if (folder != null) {
            return folder;
        }
        int slashIndex = path.lastIndexOf('/');
        String name = path.substring(slashIndex + 1);
        String parent = path.substring(0, slashIndex);
        if ("".equals(parent)) {
            folder = (IDfFolder) session.newObject("dm_cabinet");
            folder.setObjectName(name);
            folder.save();
        } else {
            IDfFolder parentFolder = createFolder(session, parent);
            session.beginTrans();
            parentFolder.lock();
            folder = session.getFolderByPath(path);
            if (folder != null) {
                return folder;
            }
            folder = (IDfFolder) session.newObject("dm_folder");
            folder.setObjectName(name);
            folder.link(parent);
            folder.save();
            session.commitTrans();
        }
        return folder;
    } finally {
        if (session.isTransactionActive()) {
            session.abortTrans();
        }
    }
}

vincentklock

Hi Panfilov,

Thanks for this example. I may be wrong, but I am not sure this code would solve the multi-thread issue. Locking the parent folder will cause concurrent thread to be unable to create the subfolder - which is desirable, but a DfException would be raised, too. It may be worth digging this way, however.

Concerning the management of the transaction it would not work when using sessions obtained from the SessionManager.

Vincent

PanfilovAB

So, you can use java-based locks like:


private static IDfFolder createFolder(IDfSession session, String path)
    throws DfException {
    IDfFolder folder = session.getFolderByPath(path);
    if (folder != null) {
        return folder;
    }
    int slashIndex = path.lastIndexOf('/');
    String name = path.substring(slashIndex + 1);
    String parent = path.substring(0, slashIndex);
    synchronized (("lock-" + path).intern()) {
        folder = session.getFolderByPath(path);
        if (folder != null) {
            return folder;
        }
        if ("".equals(parent)) {
            folder = (IDfFolder) session.newObject("dm_cabinet");
            folder.setObjectName(name);
            folder.save();
        } else {
            createFolder(session, parent);
            folder = (IDfFolder) session.newObject("dm_folder");
            folder.setObjectName(name);
            folder.link(parent);
            folder.save();
        }
    }
    return folder;
}

but it will work for single JVM only, if you run multiple JVMs that will not work - only database locks guarantee consistency.


Locking the parent folder will cause concurrent thread to be unable to create the subfolder

Actually, when you create folder CS updates r_link_count attribute for parent folder, so it does put lock too:

2013-11-08T19:47:30.474210	28750[28750]	0101d920800b214e	197	*update dm_sysobject_s set i_vstamp=i_vstamp where r_object_id='0c01d92080000105' and i_vstamp =1*
2013-11-08T19:47:30.474879	28750[28750]	0101d920800b214e	197	EXEC	0.0006800000
2013-11-08T19:47:30.474971	28750[28750]	0101d920800b214e	197	select count(*) from dm_folder_r where r_folder_path = :p0	PARAMS :p0=/dmadmin/x6
2013-11-08T19:47:30.475666	28750[28750]	0101d920800b214e	197	EXEC	0.0007150000
2013-11-08T19:47:30.475713	28750[28750]	0101d920800b214e	197	FETCH(0)	0.0000120000
2013-11-08T19:47:30.475771	28750[28750]	0101d920800b214e	197	commit


Concerning the management of the transaction it would not work when using sessions obtained from the SessionManager.

It's an restriction of old dfc.

Haroon_A

Once I had to write a multi-threaded utility to migrate content from one environment to another. In the process i had to decide whether to create the destination folder or not.

public static synchronized IDfFolder verifyAndCreateFolder (IDfSession s, String folderName) {

if folder exist then

return folder

else

create folder

return folder

}

lastnitescurry

Consider a third way


vincentklock wrote:

We thought about 2 ways out:

synchronize the whole method, which is possible but implies a performance cost
catch this exception when it occurs, parse the message for the key of this error and, if found, return the existing folder

Break up the import and filing of the documents into separate processes
- Import, load the document into repository and set metadata
- Filing, for each document not filed......

vincentklock

Thank you all for your help, we will synchronize a basic and fast "getFolder" method.
The cabinet we will use is specific for this process, so a Documentum-based lock might be overkill. However I keep the idea in case we need this kind of feature in SBO / TBO.