startdeploy.sh removes REST data?

Environment[indent]TS/OD/LS 7.2.1 (vanilla) on Linux with WebSphere 7[/indent]Issue[indent]I ran the following command today:[indent]

./startdeploy.sh -StartDeploy_TsAreaDiffArea /default/main/branch/STAGING -StartDeploy_TsAreaDiffPreviousArea /default/main/branch/EDITION/INITIAL -StartDeploy_DeploymentMode production

[/indent]After doing this, we found results from a REST query against LSCS was returning no documents, but running a query in the DB still returned 100+ results.

I ended up selecting all the files in the particular directory that we were doing the REST query on, and submitting them through our PLC-variant workflow - and then they showed up in the query.[/indent]Query for the Forum[indent]Can anyone see anything wrong in that command?
Can anyone think of a reason why LSCS would not return any results when the DB would?[/indent]

Find more posts tagged with

Comments

Rick Poulin

[indent]Can anyone see anything wrong in that command?
Can anyone think of a reason why LSCS would not return any results when the DB would?[/indent]

Can't see anything wrong with that command. The first thing to check is whether the generated .idx file (under LSCSAuth/tmp) contains entries for the files that have been deployed. No .idx entry means no LSCS entry.

Doesn't apply here, but the reason I'm pointing at that is that the PLC does not traverse directories when computing the .idx. Thus, submitting a directory leads to no updates in LSCS, whereas submitting all files within a directory does. I'm not sure how the CLT calculates which files have changed (I don't use it much if ever).

Adam Stoller

Hmm - the response I got back from Support seems to indicate the behavior is expected and that startdeploy.sh doesn't do what I thought / wanted it to do.

startdeploy script picks up less metadata than PLC Work-Flow. This is working as designed like you have experienced in your testing as well. This is because startdeploy script has a limitation that it cannot grab the TS MetaData (Extended Attributes) that are stored in the database. The recommended way of doing all deployments is to use the PLC Work-Flow.

I questioned the above[indent]You're saying that it makes sense that running startdeploy.sh will clear out all the REST data from LSCS (even though the data is in the DB)?
The fact that a query like:
http://server:port/lscs/v1/document?q=ast_relative_path:*/Resort_Content*&project=//tsserver/default/main/branch&max=500
which prior to startdeploy.sh returns 148 results, but after running startdeploy.sh returns 0 results - seems like a rather serious problem.

Also - since we're adding additional servers - running everything through the PLC workflow involves the following short-comings / side-effects:I haven't had the opportunity yet to try to run through the above test - it's difficult when you're talking about a production TeamSite server and a replication farm that touches all of your post-development receivers -- having "accidently" deleted all that REST data before, I'm not keep on incurring the wrath of my customer by doing it again "intentionally"...

Also - I'm not sure what section of the LSCS Technical Reference guide they're talking about - I assume it's the 7.1 version (since I don't recall finding it in any other release) - I did a quick scan and didn't see anything particularly enlightening.

Thoughts?

Topic37469_Group interval__cwong.rptdesign

Rick Poulin

Well, the startDeploy CLT was deprecated for a reason, and I think you've found it. Smiley Happy

I knew it had some shortcomings, but I didn't think it was quite that bad.

Not that it solves your problem per se, but that example query of yours might work if you match against the actual path (which is a kind of PK) rather than the metadata computed only by the PLC:
?q=path:*/Resort_thingy*

The ref he's talking about is the (poorly documented of course) note about parameters on p.24 of the 7.1 tech ref, in combination with Chapter 5 (p.65, specifically).. I think his suggestion is to pretty much rewrite the useful part of the CLT for yourself, except make it work this time. Personally I think I'd start by trying to figure out the piece that my mental model is missing -- how the CLT computes the file list. I can't imagine the EA processing is that different once the list is computed, but I could be wrong. In your case, you might be able to whip up a quick perl script to build that override XML, if you figure out what the format is supposed to be (the table on p.21 seems to be the "DTD" of sorts).

I haven't investigated much personally, since I've had no need to redeploy STAGING, but I think your use case would be fairly common and a good admin tool to have in your back pocket. Not that it helps you much now, but I think you just found me a small side project. Smiley Happy

Rick Poulin

UPDATE: did some digging, and I believe I understand the difference.

First, the answer to my own question, is that the CLT uses CSSDK call CSArea#compare() to compare the two areas. Interesting points that this led to, is that the CLT appears to support specifying any folder path within a workarea as the 'StartDeploy_TsAreaDiffArea' and 'StartDeploy_TsAreaDiffPreviousArea' parameters.. thus you could in theory use this to deploy only specific folder trees (read: haven't tested this). Second interesting point is that if you're redeploying the whole source area, specifying <emptystring> as the previous area will be faster than specifying the 'INITIAL' edition's path, because it'll skip the call to compare() altogether and just assume it has to deploy everything. Don't know how much faster, since I imagine the compare() compares version numbers, not the actual binary content.

Anyway, getting back to your question.. The CLT will only deploy extended attributes and nothing else. Thus, if you modify your query to use "path:" (a required PK) instead of "ast_relative_path" (a field populated only by the PLC), you should indeed get the 148 results you were expecting. What the PLC does is that it has Spring-configured additional metadata processors which, for the most part, are configured using the field mappings found under ApplicationContainer/server/default/deploy/iw-cc.war/WEB-INF/conf/livesite/fieldmappings. Thus, any field you find in these files will NOT be there for any files (re)deployed using the CLT instead of the PLC.

So the bottom line is, only "id", "path" and extended attributes are always "safe" for use in your queries if you plan on using the CLT for deployments. If you want to use the ones computed by the PLC's extra metadata processors, then you either need to keep using the PLC, or you can compute the whole of the attributes you want by generating that override manifest as mentioned by A/I support.

I've got a clear picture of this in my mind, but I feel I've done a terrible job of explaining it, so let me know if I should clarify.

output _got after placing cross tab in grid.xlsx

Expected output--Sample.xlsx

Adam Stoller

Rick - much to think about (thanks) - not sure when I'll get to really play with it (as I believe I mentioned - it's a production server so I have little leeway in terms of when I can "break" things).

What I've done thus far is written a simple script to traverse the workarea, finding all the files that are not currently locked, modified, or deleted - and lock them by me - so I can then go into the UI and do View > My Locked Files - select them and submit them through the PLC workflow. It leaves some gaps as far as bootstrapping a new target environment, but it gets a significant majority taken care of...