Problem using the HandleZipFile script

MARK HUBAND
MARK HUBAND
in AppWorks #1

Using the HandleZipFile script/workflow using MIM 8.1.8.5 I'm having very infrequent issues with the Transfer of the unzipped file to its final destination. 

 

Four large zip files (usually 300-350mb) are pulled simultaneously from a Linux file system initiated by a Directory Monitor. Each file goes through the HandleZipFile script/wf and each is successfully unzipped and written to a temporary Windows file system (usually 3-4 gig each).  However, the subsequent Transfer_File processes for each that move the unzipped files to a zOS file system occassionally do not run . The result is that the unzipped files are not sent to the mainframe and there is no notification of failure (since there is no failure.....the Transfer_File just doesn't run).  The unzipped files remain in the temporary directory having never been transferred to the mainframe.

 

I'm not sure what 'triggers' the file transfer to run once the unzipped files are written to the temporary file system.  The CreateXfers step successfully completes all steps.  

 

mvh

Tagged:

Comments

  • Mike_Policarpio
    Mike_Policarpio
    #2

    I am unfamiliar with the HandleZipFile script and workflow.  If you are able to post it, we can take a look at it to see what the issue may be. 

     

     

  • MARK HUBAND
    MARK HUBAND
    #3

    I've attached the HandleZipFiles xmscript.  Included below are the Request Template that invokes the script and the Policy that defines the final transfer of the unzipped file (which uses a normal Transfer_File).  Let me know if you want the workflow that accompanies the HandleZipFiles.xms.  mvh

     

    Request Template:

    <?xml version="1.0" encoding="UTF-8"?>


     


      $XMDM_SRC_QMGR_NAME$
      $XMDM_SRC_QMGR_NAME$
      ${wdEnvOfficeHubQM}
      ${wdEnvOfficeHubQM}
      $XMDM_PATH$
      f:\ARCHIVE\ORDERFCAST\$XMDM_ORIGINAL_NAME$
      false
      TRUE
      true
      true
      BINARY
      false
      3072
      FIXED
     
          1
     
     
          31
          $XMDM_ORIGINAL_NAME$
          ORDERFCAST
     
      admin

     

    HandleZipFile Policy:

     


     
     
     
          ${queueManager}
     
     
          CQ.COORDINATOR
     
     
          yes
          MQSTR
     
     
           yes
           yes
     
     
         
         ORDERFCAST
         RDF CGO ORDER FORECAST
         fdo.orderfcst
         
         
          $XMDM_PROCESS_ID$
          Transfer_File
          RDF_ORDER_FCAST
          ${spathbasename}
          RDF CGO ORDER FORECAST::${wdEnvSAFnode}::RADFP.UTDBPAS.XFER.FUTR.ORD
         
           
           


            ${wdEnvOfficeHubQM}
            ${wdEnvOfficeHubQM}
            QQM0
            ${wdEnvOfficeHubQM}
            F:\IN\ORDERFCAST\orderforecast_${wdstoreNum}
            RADFQ.UTDBPAS.XFER.FUTR.ORD(+1)
            PS
            FB
            64
            300
            100
            CYL
            true
            2048
            TEXT
            PAD
            40 
            9
            true
            FIXED
           
                 1
           
            admin
         
       
       
       
     
  • MARK HUBAND
    MARK HUBAND
    #4

    Here's the HandleZipFile.xms script which uses a workflow (not attached).

     

    /*-----------------------------------------------------------------------------
    * File : HandleZipFiles.xms
    *-----------------------------------------------------------------------------
    * Copyright : WinnDixie
    *-----------------------------------------------------------------------------
    *
    *-----------------------------------------------------------------------------*/

    #include "xmputils.inc"
    #include "wdstatus.inc"
    #include "wdVarSubstitution.inc"
    #include "wdLog.inc"
    #include "wdconstants.inc"
    #include "xmopm.inc"

    var xmdir;
    var log;

    UnzipFile(xmoWorkflow)
    {
    var rc=0;
    log=new wdLog();
    log.component="HandleZipFiles";
    log.logToScreen("UnzipFile()");
    var xmpUtils = new XMPUtils();
    var guid=xmpUtils.GetUUID();

    try
    {
    if(!xmoWorkflow.contains("XMOFTSReq.UserArgs.zipType"))
    {
    throw("Missing zipType in UserArgs section.");
    }
    var xmoPM = new XMOPM();
    var srcFilename=xmoWorkflow.XMOFTSReq.dpath;
    var tmpDir=WD_TEMP_DIR+"\\"+guid;
    var outFile=WD_TEMP_DIR+"\\"+guid+".txt";
    var syscall="unzip "+srcFilename+" -d "+tmpDir;
    var xmoProc=XMObject::create("Process");
    xmoWorkflow.wdUnzipDir=tmpDir;
    xmoProc.commandLine=syscall;
    var rc=xmoPM.LaunchAndWait(xmoProc);
    if(rc==0)
    {
    if(xmoProc.ProcessInfo.returnCode==0)
    {
    xmoWorkflow.createAndInsert("wdUnzipResult");
    xmoWorkflow.wdUnzipResult.insert(xmoProc.deepCopy());
    xmoWorkflow.result="OK";
    }
    else
    {
    xmoWorkflow.result="Exception";
    var exc=XMObject::create("Exception");
    exc.exceptionText="Unzip failed, see Unzip Output";
    exc.insert(xmoProc.deepCopy());
    insertException(xmoWorkflow, exc);
    }
    }
    else
    {
    xmoWorkflow.result="Exception";
    var exc=XMObject::create("Exception");
    exc.exceptionText="Unable to start unzip process, see Unzip Output";
    insertException(xmoWorkflow, exc);
    }
    addReferenceStatus("Unzip Output", "XML", "Unzip", xmoProc, xmoWorkflow);
    }
    catch(ex)
    {
    log.logToScreen("Exception in UnzipFile()");
    log.logToScreen(ex);
    insertException(xmoWorkflow, ex);
    xmoWorkflow.result="Exception";
    rc=99;
    }
    log.logToScreen("Finished");
    return(rc);
    }

    CreateXfers(xmoWorkflow)
    {
    var xmdir = new XMDirectory();
    var curs = new XMOCursor(xmoWorkflow.wdUnzipResult.Process.Stdout);
    var elem;
    var offset;
    var fileName;
    var zipFileName;
    var tmpstr="";
    var xmFileIn;
    var xmFileOut;
    var strBuffer=allocate(65536);
    var strFileData="";
    var regex="[A-Za-z][A-Za-z][0-9][0-9]\\.[0-9][0-9]";
    var ftfTemplate;
    var count=0;
    var putOptions;
    var putDescriptor;
    var myDate=new Date();
    var xmoDate=XML(myDate.toXML());
    var qmgr;
    var processId="";
    log=new wdLog();
    log.component="HandleZipFiles";

    try
    {
    //var xmoPolicy = xmdir.getObject("XML/Policies/HandleZipFilesPolicy.xml");
    var zipType=xmoWorkflow.XMOFTSReq.UserArgs.zipType;
    var zipPolicy=xmdir.getObject("XML/Policies/HandleZipFiles_"+zipType+".xml");
    var zipReq;
    qmgr=new XMQQueueManager(zipPolicy.get("MyConnection"), XMQQueueManager::exceptionOnError);
    qmgr.connect();
    var outputQueue=new XMQOutputQ(qmgr, zipPolicy.get("MyOutputQueue"));
    outputQueue.open();

    if (xmoWorkflow.XMOFTSReq.UserArgs.wdStoreNumOffset > 0)
    {
    xmoWorkflow.XMOFTSReq.UserArgs.storeNum=
    getBasename(xmoWorkflow.XMOFTSReq.spath).subString(xmoWorkflow.XMOFTSReq.UserArgs.wdStoreNumOffset, 4);
    }
    else
    {
    xmoWorkflow.XMOFTSReq.UserArgs.storeNum="9999";
    }

    // For each file in the archive
    for(curs.setToFirst(); curs.isValid(); curs.setToNext())
    {
    elem = curs.element().asString();
    offset=elem.indexOf("ing:", 1);
    if(offset>0)
    {
    fileName=elem.subString(offset+5);
    zipReq=findMatchingRequest(zipPolicy, fileName, xmoWorkflow);

    if(zipReq!=0)
    {
    processId=zipReq.name + "_" +
    xmoWorkflow.XMOFTSReq.UserArgs.storeNum + "_" +
    xmoDate.year.asString().rightJustify(4, "0")+
    xmoDate.month.asString().rightJustify(2, "0")+
    xmoDate.day.asString().rightJustify(2, "0")+ "_" +
    xmoDate.hour.asString().rightJustify(2, "0")+
    xmoDate.minute.asString().rightJustify(2, "0")+
    xmoDate.second.asString().rightJustify(2, "0")+ "_" +
    xmoDate.ticks.asString().rightJustify(3, "0")+ "_" +
    count.asString().rightJustify(4, "0");
    if(processId.length()>50)
    {
    processId=processId.subString(processId.length()-49, 50);
    }
    ftfTemplate = zipReq.ftfTemplate.Workflow.deepCopy();
    var xmoReport=XMObject::create("ReportRecord");

    findReplaceInObject(ftfTemplate, "${zipFileName}", xmoWorkflow.XMOFTSReq.UserArgs.zipFileName);

    tmpstr=getBasename(fileName);
    findReplaceInObject(ftfTemplate, "${srcFileName}", tmpstr.strip());

    tmpstr=getBasename(ftfTemplate.XMOFTSReq.dpath);
    findReplaceInObject(ftfTemplate, "${dpathbasename}", tmpstr.strip());

    tmpstr=getBasename(ftfTemplate.XMOFTSReq.spath);
    findReplaceInObject(ftfTemplate, "${spathbasename}", tmpstr.strip());

    doSubOnObject(ftfTemplate, xmoWorkflow);

    xmoReport.fileType=ftfTemplate.docType;
    xmoReport.fileName=ftfTemplate.docId;
    xmoReport.processId=processId;
    xmFileIn = new XMFile(fileName,XMFile::readMode, XMFile::stream);
    xmFileIn.open();

    // Now that we've read the header, copy the rest of the file
    xmFileOut=new XMFile(ftfTemplate.XMOFTSReq.spath,XMFile::writeMode, XMFile::stream);
    log.logToScreen("Writing "+ftfTemplate.XMOFTSReq.spath);
    var totBytes=0;
    do
    {
    xmFileIn.read(&strBuffer);
    if(strBuffer.length()>0)
    {
    xmFileOut.write(strBuffer);
    totBytes+=strBuffer.length();
    }
    }
    while(!xmFileIn.eof());
    xmFileIn.close();
    xmFileOut.close();

    xmoReport.totBytes=totBytes;
    xmoReport.timeCreated=xmoDate.year.asString().rightJustify(4, "0")+"/"+
    xmoDate.month.asString().rightJustify(2, "0")+"/"+
    xmoDate.day.asString().rightJustify(2, "0")+ " " +
    xmoDate.hour.asString().rightJustify(2, "0")+":"+
    xmoDate.minute.asString().rightJustify(2, "0")+":"+
    xmoDate.second.asString().rightJustify(2, "0");

    putOptions=new XMQPutOptions(zipPolicy.get("MyPutOptions"));
    putDescriptor=new XMQDescriptor(zipPolicy.get("MyPutDescriptor"));
    count++;

    ftfTemplate.processId=processId;
    if(totBytes>0)
    {
    outputQueue.put(ftfTemplate.asString(), putDescriptor, putOptions);
    }

    addReferenceStatus("Store File Report", "XML", "Report Record", xmoReport, xmoWorkflow);
    }
    }
    }
    xmoWorkflow.result="OK";
    }
    catch(ex)
    {
    log.logToScreen("Exception in CreateXfers()");
    log.logToScreen(ex);
    insertException(xmoWorkflow, ex);
    xmoWorkflow.result="Exception";
    }
    }

    findMatchingRequest(zipPolicy, fileName, xmoWorkflow)
    {
    var zipMatch=0;
    var regex;
    var elem;
    log=new wdLog();
    log.component="HandleZipFiles";
    var curs = new XMOCursor(zipPolicy.expectedFiles);
    for(curs.setToFirst(); curs.isValid(); curs.setToNext())
    {
    elem=curs.element();
    if(elem.tagName()=="file")
    {
    regex = elem.regex;
    if(fileName.matches(regex))
    {
    zipMatch=elem.deepCopy();
    break;
    }
    }
    }
    return zipMatch;
    }

    CleanupTempFiles(xmoWorkflow)
    {
    var rc=0;
    log=new wdLog();
    log.component="HandleZipFiles";
    try
    {
    var xmoPM = new XMOPM();
    var syscall="cmd.exe /c rmdir /s/q "+xmoWorkflow.wdUnzipDir;
    var xmoProc=XMObject::create("Process");
    xmoProc.commandLine=syscall;
    var rc=xmoPM.LaunchAndWait(xmoProc);
    if(rc==0)
    {
    if(xmoProc.ProcessInfo.returnCode==0)
    {
    xmoWorkflow.result="OK";
    }
    else
    {
    xmoWorkflow.result="Exception";
    var exc=XMObject::create("Exception");
    exc.exceptionText="Cleanup failed, see output";
    exc.insert(xmoProc.deepCopy());
    insertException(xmoWorkflow, exc);
    }
    }
    else
    {
    xmoWorkflow.result="Exception";
    var exc=XMObject::create("Exception");
    exc.exceptionText="Unable to start cleanup process, see output";
    exc.insert(xmoProc.deepCopy());
    insertException(xmoWorkflow, exc);
    }
    addReferenceStatus("Cleanup Output", "XML", "Cleanup", xmoProc, xmoWorkflow);
    }
    catch(ex)
    {
    log.logToScreen("Exception in CleanupTempFiles()");
    log.logToScreen(ex);
    insertException(xmoWorkflow, ex);
    xmoWorkflow.result="Exception";
    rc=99;
    }
    return(rc);
    }

  • MARK HUBAND
    MARK HUBAND
    #5

    Still waiting to hear back on this one.  Here's the workflow xml for the HandleZipFile script (without the presentation layer).  All steps in this script are successful when we have the problem.

     

    <?xml version="1.0" encoding="UTF-8"?>





    TransferToArchive
    Activity
    false
    Coordinator
    Transfer_File
    Yes
    Application
    ExceptionHandler

    Exception

    UnzipFile


    UnzipFile
    ScriptActivity
    true
    UnzipFile
    HandleZipFiles.xms
    Yes
    Application
    Unzip file
    ExceptionHandler

    Exception

    CreateXfers


    CreateXfers
    ScriptActivity
    true
    CreateXfers
    HandleZipFiles.xms
    Yes
    Application
    Create Transfer_File Messages
    ExceptionHandler

    Exception

    CleanupTempFiles


    CleanupTempFiles
    ScriptActivity
    false
    CleanupTempFiles
    HandleZipFiles.xms
    Yes
    Application
    ExceptionHandler

    Exception

    XMC_EndState

    <XMC_EndState/>
    <Start_1/>

    ExceptionHandler
    MQActivity
    false
    ExceptionHandler
    CQ.REJECTED.MESSAGES
    notifyTX();
    xmoWorkflow.result = "Failed";
    No
    defaultExceptionHandler

    Exception

    XMC_EndState

     

  • Mike_Policarpio
    Mike_Policarpio
    #6

    Do you see any messages on the CQ.REJECTED.MESSAGES queue and/or dead letter queues on any of nodes involved in the corresponding to the date/timestamp when the Transfer_File portion would normally run? 

  • MARK HUBAND
    MARK HUBAND
    #7

    There are three queue managers involved in the entire process- there are no messages in the DLQ or CQ.REJECTED.MESSAGES queues on any of the three queue managers.

  • Jeff_Yowell
    Jeff_Yowell
    #8

    Hi MarkV I was reviewing this post and this is something we should really open a Customer Support ticket for.

    Whenever you are ready please email us at BPSSupport@opentext.com.

     

Categories

image
OpenText CE Products
AppWorks
Gateway & Platform