The product uses messaging to send instructions to various components and to deliver data for file transfer requests. Websphere MQ utilizes message priority to determine which messages are processed first. In order to understand the prioritization process, it is important to first understand the message flow in a file transfer process.
In the case of a typical file transfer request, the following describes the file transfer process:
- User submits a request via the Process Monitor or FTF command line program. These programs submit the request to the Process Coordinator in the form of a WebSphere MQ message that contains the Workflow. This message is sometimes referred to as the control message.
- The Process Coordinator picks up the request and routes the message to the sending File Transfer component.
- The Sender processes the file and emits Data Messages to the Receiving File Transfer component's data queue. The Sender then returns the Workflow to the Process Coordinator.
- The Process Coordinator picks up the Workflow and routes the message to the Receiving File Transfer component.
- The Receiver picks up the Workflow and processes the Data Messages that were previously sent by the Sender. The Receiver returns the Workflow to the Process Coordinator.
- The Process Coordinator picks up the Workflow and completes the File Transfer business process.
The product provides the ability to control the priority of the messages used by the system. Priority is controlled in the file transfer request by setting -priority on the request. The possible values are 1 (lowest) - 9 (highest). There are some important differences between versions 7.x and versions 8.1.8 and above regarding which messages are affected by priority.
In version 7.x, priority controls the priority of the Data Messages only. The Workflow message is not affected. If the user submits the request from the Process Monitor or the FTF Command Line program, the priority setting of the message is set according to the default priority setting on the Process Coordinator's input queue. For example, if the default priority of the CQ.COORDINATOR queue is set to 5, all requests submitted by the FTF Command Line Program to that queue will have the Workflow message set to 5.
Example display of CQ.COORDINATOR queue settings, showing the default priority setting: 1 : dis ql(CQ.COORDINATOR) all AMQ8409: Display Queue details. QUEUE(CQ.COORDINATOR) TYPE(QLOCAL) ACCTQ(QMGR) ALTDATE(2007-06-18) ALTTIME(09.26.29) BOQNAME( ) BOTHRESH(0) CLUSNL( ) CLUSTER( ) CLWLPRTY(0) CLWLRANK(0) CLWLUSEQ(QMGR) CRDATE(2007-06-18) CRTIME(09.26.29) CURDEPTH(0) DEFBIND(OPEN) DEFPRTY(5) DEFPSIST(YES) DEFSOPT(SHARED) DEFTYPE(PREDEFINED) DESCR( ) DISTL(NO) GET(ENABLED) HARDENBO INITQ( ) IPPROCS(1) MAXDEPTH(5000) MAXMSGL(4194304) MONQ(QMGR) MSGDLVSQ(PRIORITY) NOTRIGGER NPMCLASS(NORMAL) OPPROCS(0) PROCESS( ) PUT(ENABLED) QDEPTHHI(80) QDEPTHLO(20) QDPHIEV(DISABLED) QDPLOEV(DISABLED) QDPMAXEV(ENABLED) QSVCIEV(NONE) QSVCINT(999999999) RETINTVL(999999999) SCOPE(QMGR) SHARE STATQ(QMGR) TRIGDATA( ) TRIGDPTH(1) TRIGMPRI(0) TRIGTYPE(FIRST) USAGE(NORMAL)
The user also has the choice of creating a custom request program that submits a properly formatted message to the Coordinator. In this case, the user has the opportunity to submit the request with the desired priority setting on the message. There are several other techniques that can be used to control the priority of the Workflow message. These include utilization of queue remote definitions for the Process Coordinator input queue, creating new instances of the Process Coordinator on the same queue manager in combination with alternate config.xml files, and by establishing Process Coordinators on separate queue managers to handle different classes of service and priority.
In version 8.1.8, priority processing was changed to provide better control of priority and to better process in-flight transactions.
When a user submits a request using the 8.1.8 FTF Command Line or the Process Coordinator, the -priority option affects both the Workflow message and the Data Messages. Recall that in 7.x priority only controlled the Data Messages. A user can utilize priority to provide the ability to process a certain class of requests ahead of normal transactions. For example, the majority of the daily processing can be sent at priority 1. This is the lowest priority setting. In the course of a business day, there is a need to submit and process a class of requests that are time sensitive in nature. These requests could be defined with a priority of 5 to ensure that they are processed ahead of the majority of daily transfers.
When the Process Coordinator pulls messages off of its input queue, it does so first by priority and then FIFO. The initial Workflow and all subsequent responses from components, such as the File Transfer Service and Exit Processor, will be processed using priority and FIFO.
In-flight transaction processing was also enhanced in version 8.1.8. Prior to this change, all newly submitted transactions and all in-flight transactions were treated with the same priority as the initial Workflow. The Process Coordinator's configuration can now be changed to increase the priority of in-flight transactions. When the Process Coordinator picks up the Workflow message to start a new process, the priority of the message is increased by 1. This has the affect of ensuring that in-flight transactions of the same priority class will be processed ahead of newly submitted work. If no in-flight (higher priority) messages are in the queue, then new work will be started in the system. Every time a component such as the Process Coordinator or File Transfer Service is ready for a new message, it will retrieve the highest priority message.
In order to configure this option in the Process Coordinator you must alter the Coordinator.xml policy found in the Registry under Services/Config. If the Coordinator.xml policy does not contain a tag called then add it. The value of can be set to TRUE or FALSE.
Example Coordinator Policy with prioritize set:
SUPGEMBAR
SUPGEMBAR
CQ.COORDINATOR
TRUE
.
.
.
FALSE means that new and in-process transactions will be executed with the same priority.
TRUE means the Coordinator will complete in-process transactions before taking on new work. In-process transactions will receive a higher priority.
To illustrate the benefits of using the prioritize setting in the Process Coordinator, consider the following scenario with prioritize set to FALSE (normal operation):
If 10,000 requests were submitted to the system at the same time, the Process Coordinator would try to start all of the processes at the same time. In-flight traffic would queue up behind the original requests, therefore it would be possible to see all requests in progress at the same time. You may also see that large numbers of processes would be on the same steps at the same time. For example, many processes would be in the Send step together and subsequently the Receive step together. The total amount of resources required to store Data Messages will be high because Data Messages have been delivered to the data queue but won't be processed until the Workflow is sent to the Receiver. This Workflow could be delayed because the Process Coordinator is trying to service all requests at the same time.
Contrast the above scenario to the following with prioritize set to TRUE (new behavior):
If 10,000 requests were submitted to the system at the same time, the Process Coordinator would start and increase the priority of many of the initial requests - these are now in-flight requests. The Workflows would be routed by the Process Coordinator to the Sender. The Sender would do its work and then return the Workflow back to the Process Coordinator. The response would show up in the Process Coordinator's input queue as a higher priority message and would be processed ahead of any new request that has not been processed. When there are no in-flight messages in the input queue, the Process Coordinator will accept and initiate new work. You will see that the first requests to be submitted will complete sooner and closer to the submission order. In addition, Data Messages should be processed sooner, therefore reducing the length of time that they occupy space in the queues. New work is naturally paced due to priority.
Please consult the product documentation for more information on how to configure the Process Coordinator and how to add -priority to transfer requests.
