Best practice for handling 400/503 errors from IIS due to queuing during high load?

I'm encountering an issue where large volumes of SOAP requests are being sent to IIS, accepted, and then waiting in queue for processing by the CWS service(s), which are not able to process the requests as quickly as they are arriving.

Eventually IIS starts returning 400/503 statuses to the callers, presumably as the request has expired in queue. The IIS time-taken shown in the logs is typically averaging in the 50 to 70 second range when 503's start being thrown.

Presumably I need to reduce the queue depth upstream of the CWS application(s) in order to return the 503 to the client application in a more timely manner so as to allow it to take steps to back-off the request submission rate? That's to say; by the time 503's start being emitted presumably there is already a very large queue of work behind the timing out request that will, in turn, also time out.

Any suggestions on how to debug/tune a default IIS/CWS installation for this condition? Also, is there any request queuing internal to the CWS apps that needs to be adjusted?

Find more posts tagged with

Comments

Kyle_Swidrovich

Hi Kris,

There’s nothing internal to the CWS services themselves that could be tweaked here.

Any throttling/queuing would need to be done at the web server / app server level (or possibly within WCF, somehow).

If you don’t know the web/app server configuration ahead of time, then I suspect you’ll be coding up a dynamic throttling mechanism within your client application. Otherwise, keep your concurrent threads in your client to a level that won’t kill the web server.

Kyle

Kris Boutilier

Turns out this was my mistake - the queuing was occurring on the IIS side for completed requests going back to the client, not because of a flood of new/pending requests coming in. The responses were pending as the client was not servicing the socket reads for each open connection in a timely manner (suspended threads).

While the client was running I was able to issue 'appcmd list requests' on the IIS server and noted that the majority of the backlogged requests had all reached the 'stage:SendResponse' but their timers were still steadily incrementing.

With a little rethinking all seems well again.