Walking a folder tree in Web Services

Hi,

This probably is a job for the GetNodesInContainer() but I need some help with the options, and how to paginate the call. I'm working with CS 10.0 so please no solutions that involve upgrading coz that ain't going to happen.

I'm trying to write an app that will "discover" what is contained within Content Server (and no, I'm not trying to pull data *out* of Content server to go to an OT competitor - this actually has to do with a chunked move). I noticed that the way GetNodesInContainer works is that you could theoretically go multiple depths, which is an interesting feature but not one I need right now. What I do need is the ability to set a page limit, but know when I have more pages. Example, say I want to get the children of a big email folder that has 10,000+ objects. If I try and return an array of 10,000 items that is going to time out my request, and also take up a ton of memory. But since the call returns a Node[], if I set a reasonable maxItems, I will miss a whole bunch and don't have a good way of paging this.

What are others doing to fake out pagination for the retrieval of children ofa folder? Also, what are folks doing to limit the results to just container objects (folders, email folders, projects, etc)?

Thanks in advance

-Hugh

Find more posts tagged with

Comments

eLink User

Hugh,

ListNodesByPage dont know which version introduced it .Does that show up in your wsdl? the pageSpec is long winded but somewhere in this forum me or Kyle has put a working sample.I think I was just playing with SOAPUI .This has that lapi equivalent of saying select * from dtree where parentid=1234 and subtype in(0,144,136) etc.

.....appu appnair appunair

Well, if I called the wrongnumber, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937

On Wed, May 18, 2016 at 8:24 AM, eLink Entry: Content Web Services Forum <otdncontentwebservices971forum@elinkkc.opentext.com> wrote:

Walking a folder tree in Web Services Posted byFerguson, HughOn 05/18/2016 09:17 AM

Hi, This probably is a job for the GetNodesInContainer() but I need some help with the options, and how to paginate the call. I'm working with CS 10.0 so please no solutions that involve upgrading coz that ain't going to happen.I'm trying to write an app that will "discover" what is contained within Content Server (and no, I'm not trying to pull data *out* of Content server to go to an OT competitor - this actually has to do with a chunked move). I noticed that the way GetNodesInContainer works is that you could theoretically go multiple depths, which is an interesting feature but not one I need right now. What I do need is the ability to set a page limit, but know when I have more pages. Example, say I want to get the children of a big email folder that has 10,000+ objects. If I try and return an array of 10,000 items that is going to time out my request, and also take up a ton of memory. But since the call returns a Node[], if I set a reasonable maxItems, I will miss a whole bunch and don't have a good way of paging this.What are others doing to fake out pagination for the retrieval of children ofa folder? Also, what are folks doing to limit the results to just container objects (folders, email folders, projects, etc)?Thanks in advance-Hugh
[To post a comment, use the normal reply function]
Forum: Content Web Services Forum
Content Server: Knowledge Center

eLink User

Here is that post for reference

https://knowledge.opentext.com/knowledge/cs.dll?func=ll&objId=62149123&objAction=viewincontainer&ShowReplyEntry=62150538#forum_topic_62150538

.....appu appnair appunair

Well, if I called the wrongnumber, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937

On Wed, May 18, 2016 at 8:24 AM, eLink Entry: Content Web Services Forum <otdncontentwebservices971forum@elinkkc.opentext.com> wrote:

Walking a folder tree in Web Services Posted byFerguson, HughOn 05/18/2016 09:17 AM

Hi, This probably is a job for the GetNodesInContainer() but I need some help with the options, and how to paginate the call. I'm working with CS 10.0 so please no solutions that involve upgrading coz that ain't going to happen.I'm trying to write an app that will "discover" what is contained within Content Server (and no, I'm not trying to pull data *out* of Content server to go to an OT competitor - this actually has to do with a chunked move). I noticed that the way GetNodesInContainer works is that you could theoretically go multiple depths, which is an interesting feature but not one I need right now. What I do need is the ability to set a page limit, but know when I have more pages. Example, say I want to get the children of a big email folder that has 10,000+ objects. If I try and return an array of 10,000 items that is going to time out my request, and also take up a ton of memory. But since the call returns a Node[], if I set a reasonable maxItems, I will miss a whole bunch and don't have a good way of paging this.What are others doing to fake out pagination for the retrieval of children ofa folder? Also, what are folks doing to limit the results to just container objects (folders, email folders, projects, etc)?Thanks in advance-Hugh
[To post a comment, use the normal reply function]
Forum: Content Web Services Forum
Content Server: Knowledge Center

Ferdinand Prantl

I can't see a fitting method in CS SOAP WS. GetNodesInContainer provides no paging. There is ListNodesByPage which provides paging, but lists only children. If there were ListNodesByPageAndFacet, you could page through all nodes within the specified container as three hierarchy root, which is what you need.

You could write your own service. If you're able to formulate a faceted filter to browse through the nodes, you could try CS REST API. For example, getting first 100 documents and folders (subtypes 144 and 0):

GET .../api/v1/nodes/390426/nodes?limit=100&page=1&sort=asc_name&extra=false&where_facet=3440%3A144%7C0

You could continue with the page parameter incremented, until you get less then 100 childern. The trouble is, that the data source ID of the node type facet (3440 on my server) is different on every machine. If you have a lot of subtypes and the expression where_facet=uuencode(3440:144|0) would be too long, you can take other facet with less values to seect or nodes.

HFCDev

The ListNodesByPage() will suit my needs. Can anyone point me to a good example of how to use it? It's not immediately clear from the Javadocs that are provided how one goes about properly iterating the results. For instance, you create a NodePageSpecification for your initial request, where you set various params (i.e. containers only, show hidden nodes, page size, etc). And you get back a NodePageResults, which is presumably set to page 0 and has a bunch of nodes attached, but how do you get subsequent pages? Do I pass just the page context to a new NodepageSpecification object or do I have to rebuild that object? This is one area where a clear example would be REALLY helpful.

-Hugh

eLink User

Do you have SOAP UI? I was able to send a command to it and that was that screenshot I included.I never went deep into understanding it.

.....appu appnair appunair

Well, if I called the wrongnumber, why did you answer the phone?
James Thurber, New Yorker cartoon caption, June 5, 1937

On Wed, May 18, 2016 at 9:56 AM, eLink Entry: Content Web Services Forum <otdncontentwebservices971forum@elinkkc.opentext.com> wrote:

Walking a folder tree in Web Services Posted byFerguson, HughOn 05/18/2016 10:46 AM

The ListNodesByPage() will suit my needs. Can anyone point me to a good example of how to use it? It's not immediately clear from the Javadocs that are provided how one goes about properly iterating the results. For instance, you create a NodePageSpecification for your initial request, where you set various params (i.e. containers only, show hidden nodes, page size, etc). And you get back a NodePageResults, which is presumably set to page 0 and has a bunch of nodes attached, but how do you get subsequent pages? Do I pass just the page context to a new NodepageSpecification object or do I have to rebuild that object? This is one area where a clear example would be REALLY helpful.-Hugh
[To post a comment, use the normal reply function]Topic:Walking a folder tree in Web ServicesForum:Content Web Services ForumContent Server:Knowledge Center

HFCDev

I don't have an easy way of using that. Has no one used ListNodesByPage before? I would have thought this would have been one of the essential code samples, but that's not to be.

-Hugh

HFCDev

I also looked at the SoapUI screenshot and I'm missing some key pieces of information.

1) The NodepageResults has a page context which you are presumably supposed to pass in to the next call. When I do that, I keep getting the same page repeat itself. Let's say I do the following call:

public NodePageResult GetChildren(Node parent, int pageSize)
{

LogDebug("Getting children for " + parent.Name);
NodePageSpecification nps = new NodePageSpecification();
nps.IncludeHiddenNodes = true;
nps.ContainersOnly = true;
nps.PageSize = pageSize;
nps.PageNumber = 1;

return GetDocmanClient().ListNodesByPage(ref otAuth, parent.ID, nps);

}

From my NodePageResults, how do I get the next page???

2) Assuming I'm supposed to pass in the page context, am I just creating a new NodePageSpecification that if I add the page context, it will "remember" what I put in previously? This may be moot because when I do the call above, the page context is always null.

3) If not the page context, how am I supposed to be paging through the results from this call??

-Hugh

jdonpaul

Here is a very basic C# example of what I have used in the past:

DocumentManagement.DocumentManagementClient docManClient = new DocumentManagement.DocumentManagementClient();
DocumentManagement.OTAuthentication docOTAuth = new DocumentManagement.OTAuthentication();

docManClient.Endpoint.Address = new EndpointAddress(DocumentManagement_addr);

docOTAuth.AuthenticationToken = token;
int currentPageNumber = 1;
DocumentManagement.NodePageResult currentPageDocs = null;

DocumentManagement.NodePageSpecification nodePageSpec = new DocumentManagement.NodePageSpecification();
nodePageSpec.PageSize = 50;
nodePageSpec.ContainersOnly = false;
nodePageSpec.IncludeTypes = new string[] { "Document" };

do
{
nodePageSpec.PageNumber = currentPageNumber; // page number has to be greater than zero
currentPageDocs = docManClient.ListNodesByPage(ref docOTAuth, node, nodePageSpec);

// set total number of pages if we are on the first page
Console.WriteLine("Page " + currentPageNumber);

   foreach (var _node in currentPageDocs.Nodes)
   {
       Console.WriteLine(_node.ID);
   }

} while (++currentPageNumber <= currentPageDocs.NumberOfPages);

As you can see, the page number is not 0 based. Also, I keep calling the ListNodesbyPage using the same NodePageSpecification while incrementing the page number that I want returned. This is by no means a perfect example, but at least it shows how to use ListNodesbyPage and NodePageSpecification.

HFCDev

Through the process of reverse engineering, I have an answer. It appears that the page context is irrelevant and the node page specification is where you invariably set the page you want to retrieve. My algorithm to get a paginated browse of a folder this has the following:

NodePageResult npr = connector.GetChildren(currNode, pageSize);
if (npr != null)
{

Node node;


for (int i=1; i <= npr.NumberOfPages; i++)
{
LogDebug(String.Format("Processing page {0} with context {1}", i, npr.PageContext));
if (i>1)
npr = connector.GetChildren(currNode, pageSize, i);
for (int j = 0; j < npr.Nodes.Length; j++)
{
node = npr.Nodes[j];
//LogDebug("Queuing node " + node.Name);
GetQueue().Enqueue(node);
}

npr.PageNumber++;
}

}

My connector.GetChildren() is as follows - I have a defaulted 3rd param which is the page I'm after which defaults to 1:

public NodePageResult GetChildren(Node parent, int pageSize, int pageNumber=1)
{

LogDebug("Getting children for " + parent.Name);
NodePageSpecification nps = new NodePageSpecification();
nps.IncludeHiddenNodes = true;
nps.ContainersOnly = true;
nps.PageSize = pageSize;
nps.PageNumber = pageNumber;

return GetDocmanClient().ListNodesByPage(ref otAuth, parent.ID, nps);

}

The above seems to work reasonably well.

-Hugh