Programmatically configure HotFolders

We are in a situation where we would like to programmatically configure HotFolders and also address the silly password connected to a user issue.

I noticed that all the HotFolder configurations seemed to be stored in the HotFolder config .xml file. Can I programmatically stop the mbpublication service, alter the xml in the configuration, and start the mbpublication service?

Are there any gotchas to watch out for?

Find more posts tagged with

Comments

msnider

Hi Stacy,

While not necessarily recommended, yes, you can programmatically modify the HotFolders configuration file and restart the publication service. I say this isn't necessarily recommended because the HotFolder Editor tries to prevent creating a configuration that either won't work (i.e. placing a folder id in the config file in which the user does not have change permission) or has unintended sideeffects (such as using the same source folder for two different HotFolders).

To help me list some gotchas, can you explain further what modifications you are expecting to do and what you mean by "the silly issue connected to a user"?

Mark

Migrateduser

Hi Mark,

Thanks for responding. The problem we have right now, is that we get a feed of images from one of image provider news agencies (and soon to be many more). We are ingesting roughly 2,000-2,500 assets a day, both images and xml side car files.

We have an issue where we hit a little bit more than 10,000 assets in a container and the MediaBin server stops putting request into the insertion queue.

To rectify this, we create a new container, and point the HotFolder to the new container. Something like this $/Media Repository/Feeds/1, $/Media Repository/Feeds/2, $/Media Repository/Feeds/3.

We have created our own Windows service application which fires multiple times a day to assess the container size and date at which point we make a programmatic logical decision where images should be fed.

This is where the programmatic manipulation of the HotFolderConfig.xml file comes into play. We need to alter the destination containerGuid without having human intervention.

If you have another approach, that would be great!

Thanks for your advice!

Stacy

msnider

I believe there is an easier way to accomplish this.

A HotFolder should respond to events within a subfolder as well....i.e. it will mirror the filesystem folder structure of the source folder into the folder structure of the MediaBin folder. So I would suggest leaving the HotFolder's destination MediaBin folder as $/Media Repository/Feeds and instead, modify the feed mechanism to write to a new subfolder of the filesystem based on the number of assets in the current MediaBin folder.

Thus, when the HotFolder detects a new file event it will recreate it's folder structure within MediaBin...in effect, automatically doing what you are proposing to do yourself.

Mark

lyman

"We have an issue where we hit a little bit more than 10,000 assets in a container and the MediaBin server stops putting request into the insertion queue."

This statement concerns me. There should not be any such limitation. Now prior to MB 4.6 there was a reduction in efficiency when a container held too many assets but with folder pagination a built-in feature in MediaBin 4.6 that should no longer be an issue.

Cheers,

Lyman Hurd

Migrateduser

Hi Guys,

Thanks for responding. Yes, it's a pain in the buns. We don't know the exact number however, when we hit about 10,800 images or sometimes more, the folder stops responding.

The HotFolder cannot queue any more jobs to that container. If we use the web client or windows client, I cannot manually drag and drop an image into the container as it will timeout.

Any ideas how to diagnose or troubleshoot the issue? Have you guys actually stress tested MediaBin with let's say 200,000 images and xml files? In our situation, that is how we would like to use the application. We'd like to dump roughly that many xml files and images into a single container.

Migrateduser

Stacy-

I understand why the image and XML files/pair should be together, but is there a specific need to have all the files in a single container? Regarding Mark's suggestion above, is there a requirement that would prevent using sub-folders? They can inherit permissions and metadata from the parent when they are created.

If creating the sub-folders on the source isn't an option, it would be an easy matter to write a script that would monitor the Hot Folder destination container and move assets to a sub-folder (creating a new one as the current sub-folder reaches a threshold). This could be scheduled to run nightly or hourly or however often you want (using the Window Task Scheduler). And it could move assets always or only when there are too many or by date or whatever.

Search performance shouldn't be impacted. The only issues (at first thought) would be browsing (which might be an issue anyways with 10,000+ images) and changes to permissions (which can be helped by using the Permissions Manager - an app for 4.5.3 and built in to 4.6).

I know of a company that has over 2 million assets. They ingest a very large # of images per day and they divide their assets into sub-folders. They only have a search interface, though, so they don't even notice how the files are stored in MediaBin. But it works really well for them. They manage permissions via groups so that isn't an option either (which is good, as they have 5,000+ folders!).

I obviously don't know all your requirements, but thought I would throw that out there as an option (one that should be pretty easy to implement).

Matt

--
Matt Thesing
Rapid Software Solutions, LLC
matt@rapid-soft.com