Problem using the HandleZipFile script

Using the HandleZipFile script/workflow using MIM 8.1.8.5 I'm having very infrequent issues with the Transfer of the unzipped file to its final destination.

Four large zip files (usually 300-350mb) are pulled simultaneously from a Linux file system initiated by a Directory Monitor. Each file goes through the HandleZipFile script/wf and each is successfully unzipped and written to a temporary Windows file system (usually 3-4 gig each). However, the subsequent Transfer_File processes for each that move the unzipped files to a zOS file system occassionally do not run . The result is that the unzipped files are not sent to the mainframe and there is no notification of failure (since there is no failure.....the Transfer_File just doesn't run). The unzipped files remain in the temporary directory having never been transferred to the mainframe.

I'm not sure what 'triggers' the file transfer to run once the unzipped files are written to the temporary file system. The CreateXfers step successfully completes all steps.

mvh

Find more posts tagged with

BPM

Comments

Mike_Policarpio

I am unfamiliar with the HandleZipFile script and workflow. If you are able to post it, we can take a look at it to see what the issue may be.

MARK HUBAND

I've attached the HandleZipFiles xmscript. Included below are the Request Template that invokes the script and the Policy that defines the final transfer of the unzipped file (which uses a normal Transfer_File). Let me know if you want the workflow that accompanies the HandleZipFiles.xms. mvh

Request Template:

<?xml version="1.0" encoding="UTF-8"?>

$XMDM_SRC_QMGR_NAME$
$XMDM_SRC_QMGR_NAME$
${wdEnvOfficeHubQM}
${wdEnvOfficeHubQM}
$XMDM_PATH$
f:\ARCHIVE\ORDERFCAST\$XMDM_ORIGINAL_NAME$
false
TRUE
true
true
BINARY
false
3072
FIXED

1

31
$XMDM_ORIGINAL_NAME$
ORDERFCAST

admin

HandleZipFile Policy:

${queueManager}

CQ.COORDINATOR

yes
MQSTR

yes
yes

ORDERFCAST
RDF CGO ORDER FORECAST
fdo.orderfcst

$XMDM_PROCESS_ID$
Transfer_File
RDF_ORDER_FCAST
${spathbasename}
RDF CGO ORDER FORECAST::${wdEnvSAFnode}::RADFP.UTDBPAS.XFER.FUTR.ORD

RDF CGO ORDER FORECAST::${wdEnvSAFnode}::RADFP.UTDBPAS.XFER.FUTR.ORD

${wdEnvOfficeHubQM}
${wdEnvOfficeHubQM}
QQM0
${wdEnvOfficeHubQM}
F:\IN\ORDERFCAST\orderforecast_${wdstoreNum}
RADFQ.UTDBPAS.XFER.FUTR.ORD(+1)
PS
FB
64
300
100
CYL
true
2048
TEXT
PAD
40
9
true
FIXED

1

admin

MARK HUBAND

Here's the HandleZipFile.xms script which uses a workflow (not attached).

/*-----------------------------------------------------------------------------
* File : HandleZipFiles.xms
*-----------------------------------------------------------------------------
* Copyright : WinnDixie
*-----------------------------------------------------------------------------
*
*-----------------------------------------------------------------------------*/

#include "xmputils.inc"
#include "wdstatus.inc"
#include "wdVarSubstitution.inc"
#include "wdLog.inc"
#include "wdconstants.inc"
#include "xmopm.inc"

var xmdir;
var log;

UnzipFile(xmoWorkflow)
{
var rc=0;
log=new wdLog();
log.component="HandleZipFiles";
log.logToScreen("UnzipFile()");
var xmpUtils = new XMPUtils();
var guid=xmpUtils.GetUUID();

try
{
if(!xmoWorkflow.contains("XMOFTSReq.UserArgs.zipType"))
{
throw("Missing zipType in UserArgs section.");
}
var xmoPM = new XMOPM();
var srcFilename=xmoWorkflow.XMOFTSReq.dpath;
var tmpDir=WD_TEMP_DIR+"\\"+guid;
var outFile=WD_TEMP_DIR+"\\"+guid+".txt";
var syscall="unzip "+srcFilename+" -d "+tmpDir;
var xmoProc=XMObject::create("Process");
xmoWorkflow.wdUnzipDir=tmpDir;
xmoProc.commandLine=syscall;
var rc=xmoPM.LaunchAndWait(xmoProc);
if(rc==0)
{
if(xmoProc.ProcessInfo.returnCode==0)
{
xmoWorkflow.createAndInsert("wdUnzipResult");
xmoWorkflow.wdUnzipResult.insert(xmoProc.deepCopy());
xmoWorkflow.result="OK";
}
else
{
xmoWorkflow.result="Exception";
var exc=XMObject::create("Exception");
exc.exceptionText="Unzip failed, see Unzip Output";
exc.insert(xmoProc.deepCopy());
insertException(xmoWorkflow, exc);
}
}
else
{
xmoWorkflow.result="Exception";
var exc=XMObject::create("Exception");
exc.exceptionText="Unable to start unzip process, see Unzip Output";
insertException(xmoWorkflow, exc);
}
addReferenceStatus("Unzip Output", "XML", "Unzip", xmoProc, xmoWorkflow);
}
catch(ex)
{
log.logToScreen("Exception in UnzipFile()");
log.logToScreen(ex);
insertException(xmoWorkflow, ex);
xmoWorkflow.result="Exception";
rc=99;
}
log.logToScreen("Finished");
return(rc);
}

CreateXfers(xmoWorkflow)
{
var xmdir = new XMDirectory();
var curs = new XMOCursor(xmoWorkflow.wdUnzipResult.Process.Stdout);
var elem;
var offset;
var fileName;
var zipFileName;
var tmpstr="";
var xmFileIn;
var xmFileOut;
var strBuffer=allocate(65536);
var strFileData="";
var regex="[A-Za-z][A-Za-z][0-9][0-9]\\.[0-9][0-9]";
var ftfTemplate;
var count=0;
var putOptions;
var putDescriptor;
var myDate=new Date();
var xmoDate=XML(myDate.toXML());
var qmgr;
var processId="";
log=new wdLog();
log.component="HandleZipFiles";

try
{
//var xmoPolicy = xmdir.getObject("XML/Policies/HandleZipFilesPolicy.xml");
var zipType=xmoWorkflow.XMOFTSReq.UserArgs.zipType;
var zipPolicy=xmdir.getObject("XML/Policies/HandleZipFiles_"+zipType+".xml");
var zipReq;
qmgr=new XMQQueueManager(zipPolicy.get("MyConnection"), XMQQueueManager::exceptionOnError);
qmgr.connect();
var outputQueue=new XMQOutputQ(qmgr, zipPolicy.get("MyOutputQueue"));
outputQueue.open();

if (xmoWorkflow.XMOFTSReq.UserArgs.wdStoreNumOffset > 0)
{
xmoWorkflow.XMOFTSReq.UserArgs.storeNum=
getBasename(xmoWorkflow.XMOFTSReq.spath).subString(xmoWorkflow.XMOFTSReq.UserArgs.wdStoreNumOffset, 4);
}
else
{
xmoWorkflow.XMOFTSReq.UserArgs.storeNum="9999";
}

// For each file in the archive
for(curs.setToFirst(); curs.isValid(); curs.setToNext())
{
elem = curs.element().asString();
offset=elem.indexOf("ing:", 1);
if(offset>0)
{
fileName=elem.subString(offset+5);
zipReq=findMatchingRequest(zipPolicy, fileName, xmoWorkflow);

if(zipReq!=0)
{
processId=zipReq.name + "_" +
xmoWorkflow.XMOFTSReq.UserArgs.storeNum + "_" +
xmoDate.year.asString().rightJustify(4, "0")+
xmoDate.month.asString().rightJustify(2, "0")+
xmoDate.day.asString().rightJustify(2, "0")+ "_" +
xmoDate.hour.asString().rightJustify(2, "0")+
xmoDate.minute.asString().rightJustify(2, "0")+
xmoDate.second.asString().rightJustify(2, "0")+ "_" +
xmoDate.ticks.asString().rightJustify(3, "0")+ "_" +
count.asString().rightJustify(4, "0");
if(processId.length()>50)
{
processId=processId.subString(processId.length()-49, 50);
}
ftfTemplate = zipReq.ftfTemplate.Workflow.deepCopy();
var xmoReport=XMObject::create("ReportRecord");

findReplaceInObject(ftfTemplate, "${zipFileName}", xmoWorkflow.XMOFTSReq.UserArgs.zipFileName);

tmpstr=getBasename(fileName);
findReplaceInObject(ftfTemplate, "${srcFileName}", tmpstr.strip());

tmpstr=getBasename(ftfTemplate.XMOFTSReq.dpath);
findReplaceInObject(ftfTemplate, "${dpathbasename}", tmpstr.strip());

tmpstr=getBasename(ftfTemplate.XMOFTSReq.spath);
findReplaceInObject(ftfTemplate, "${spathbasename}", tmpstr.strip());

doSubOnObject(ftfTemplate, xmoWorkflow);

xmoReport.fileType=ftfTemplate.docType;
xmoReport.fileName=ftfTemplate.docId;
xmoReport.processId=processId;
xmFileIn = new XMFile(fileName,XMFile::readMode, XMFile::stream);
xmFileIn.open();

// Now that we've read the header, copy the rest of the file
xmFileOut=new XMFile(ftfTemplate.XMOFTSReq.spath,XMFile::writeMode, XMFile::stream);
log.logToScreen("Writing "+ftfTemplate.XMOFTSReq.spath);
var totBytes=0;
do
{
xmFileIn.read(&strBuffer);
if(strBuffer.length()>0)
{
xmFileOut.write(strBuffer);
totBytes+=strBuffer.length();
}
}
while(!xmFileIn.eof());
xmFileIn.close();
xmFileOut.close();

xmoReport.totBytes=totBytes;
xmoReport.timeCreated=xmoDate.year.asString().rightJustify(4, "0")+"/"+
xmoDate.month.asString().rightJustify(2, "0")+"/"+
xmoDate.day.asString().rightJustify(2, "0")+ " " +
xmoDate.hour.asString().rightJustify(2, "0")+":"+
xmoDate.minute.asString().rightJustify(2, "0")+":"+
xmoDate.second.asString().rightJustify(2, "0");

putOptions=new XMQPutOptions(zipPolicy.get("MyPutOptions"));
putDescriptor=new XMQDescriptor(zipPolicy.get("MyPutDescriptor"));
count++;

ftfTemplate.processId=processId;
if(totBytes>0)
{
outputQueue.put(ftfTemplate.asString(), putDescriptor, putOptions);
}

addReferenceStatus("Store File Report", "XML", "Report Record", xmoReport, xmoWorkflow);
}
}
}
xmoWorkflow.result="OK";
}
catch(ex)
{
log.logToScreen("Exception in CreateXfers()");
log.logToScreen(ex);
insertException(xmoWorkflow, ex);
xmoWorkflow.result="Exception";
}
}

findMatchingRequest(zipPolicy, fileName, xmoWorkflow)
{
var zipMatch=0;
var regex;
var elem;
log=new wdLog();
log.component="HandleZipFiles";
var curs = new XMOCursor(zipPolicy.expectedFiles);
for(curs.setToFirst(); curs.isValid(); curs.setToNext())
{
elem=curs.element();
if(elem.tagName()=="file")
{
regex = elem.regex;
if(fileName.matches(regex))
{
zipMatch=elem.deepCopy();
break;
}
}
}
return zipMatch;
}

CleanupTempFiles(xmoWorkflow)
{
var rc=0;
log=new wdLog();
log.component="HandleZipFiles";
try
{
var xmoPM = new XMOPM();
var syscall="cmd.exe /c rmdir /s/q "+xmoWorkflow.wdUnzipDir;
var xmoProc=XMObject::create("Process");
xmoProc.commandLine=syscall;
var rc=xmoPM.LaunchAndWait(xmoProc);
if(rc==0)
{
if(xmoProc.ProcessInfo.returnCode==0)
{
xmoWorkflow.result="OK";
}
else
{
xmoWorkflow.result="Exception";
var exc=XMObject::create("Exception");
exc.exceptionText="Cleanup failed, see output";
exc.insert(xmoProc.deepCopy());
insertException(xmoWorkflow, exc);
}
}
else
{
xmoWorkflow.result="Exception";
var exc=XMObject::create("Exception");
exc.exceptionText="Unable to start cleanup process, see output";
exc.insert(xmoProc.deepCopy());
insertException(xmoWorkflow, exc);
}
addReferenceStatus("Cleanup Output", "XML", "Cleanup", xmoProc, xmoWorkflow);
}
catch(ex)
{
log.logToScreen("Exception in CleanupTempFiles()");
log.logToScreen(ex);
insertException(xmoWorkflow, ex);
xmoWorkflow.result="Exception";
rc=99;
}
return(rc);
}

MARK HUBAND

Still waiting to hear back on this one. Here's the workflow xml for the HandleZipFile script (without the presentation layer). All steps in this script are successful when we have the problem.

<?xml version="1.0" encoding="UTF-8"?>

TransferToArchive
Activity
false
Coordinator
Transfer_File
Yes
Application
ExceptionHandler

Exception

UnzipFile

UnzipFile
ScriptActivity
true
UnzipFile
HandleZipFiles.xms
Yes
Application
Unzip file
ExceptionHandler

Exception

CreateXfers

CreateXfers
ScriptActivity
true
CreateXfers
HandleZipFiles.xms
Yes
Application
Create Transfer_File Messages
ExceptionHandler

Exception

CleanupTempFiles

CleanupTempFiles
ScriptActivity
false
CleanupTempFiles
HandleZipFiles.xms
Yes
Application
ExceptionHandler

Exception

XMC_EndState

<XMC_EndState/>
<Start_1/>

ExceptionHandler
MQActivity
false
ExceptionHandler
CQ.REJECTED.MESSAGES
notifyTX();
xmoWorkflow.result = "Failed";
No
defaultExceptionHandler

Exception

XMC_EndState

Mike_Policarpio

Do you see any messages on the CQ.REJECTED.MESSAGES queue and/or dead letter queues on any of nodes involved in the corresponding to the date/timestamp when the Transfer_File portion would normally run?

MARK HUBAND

There are three queue managers involved in the entire process- there are no messages in the DLQ or CQ.REJECTED.MESSAGES queues on any of the three queue managers.

Jeff_Yowell

Hi MarkV I was reviewing this post and this is something we should really open a Customer Support ticket for.

Whenever you are ready please email us at BPSSupport@opentext.com.