DA Slowdown

I am seeing a slowdown in DA performance, wondering if others see the same sort of thing.

Background - The overall goal is to create a reporting table of all records (around 22 million objects). I start everything off with an agent. This agent creates a "temporary table", identifies all of the records (min id, max id, count to process), and starts the DA job to insert all of the records into the "temporary table". Finally there is a 2nd agent that checks to see if the DA job(s) are done and when they are this 2nd agent builds indexes, removes the reporting table, and renames the "temporary table". The DA job(s) are simple job type and either split the work into smaller subtasks or perform the work to insert into the temporary table (based on the count of objects to process).

One of the fields I insert is a date/time stamp. I use this field to count the number of records inserted per hour. Here is what I am seeing:

17-Nov-16 10:00:00   293,383
17-Nov-16 11:00:00   298,162
17-Nov-16 12:00:00   300,782
17-Nov-16 13:00:00   307,939
17-Nov-16 14:00:00   301,982
17-Nov-16 15:00:00   293,684
17-Nov-16 16:00:00   298,283
17-Nov-16 17:00:00   276,290
...
19-Nov-16 17:00:00   166,753
19-Nov-16 18:00:00   158,958
19-Nov-16 19:00:00   159,071
19-Nov-16 20:00:00   156,390
19-Nov-16 21:00:00   148,981
19-Nov-16 22:00:00   153,021
19-Nov-16 23:00:00   155,318
...
21-Nov-16 3:00:00   99,664
21-Nov-16 4:00:00   98,617
21-Nov-16 5:00:00   96,545
21-Nov-16 6:00:00   97,847
21-Nov-16 7:00:00   193,197
21-Nov-16 8:00:00   294,711
21-Nov-16 9:00:00   291,232

I've made numerous changes to try and improve the performance but I always see the same sort of behavior...a slowing over time, fixed by a restart of the CS service (as seen at 21-Nov 7AM), only to start slowing all over again. To try and eliminate my specific code as the culprit I created another set of Simple Jobs that split into smaller tasks eventually running a task that counts from 1 to 100,000 (so no DB activity other than normal DA processing). It exhibits this same sort of behavior.

I have noticed that before the restart of services the CS service is consuming a large amount of memory (one server this morning it was around 12 GB). Does this suggest some sort of memory leak?

Cheers...

Find more posts tagged with

Comments

David Templeton

John

This reporting table is your table? If so, what are the indexes on the table? Do you have thread logs showing the overall time for the tasks? Are they slowing down? Do you have connects of the insert/update of that table. Could it be slowing down due to the database expanding to handle the new rows being added?

Just some thoughts

David

John Simon

David,

Thanks for the reply!!!

Yes the record gathering jobs is using a custom table. I create it (in the agent that kicks the whole process off) with no indexes, add them later after inserting is done (in the cleanup agent). I considered the slowdown might be because the table continually expands and adding new extents might be an issue so now I preallocate space for the table during creation. Also, if it were the size of the table causing a slowdown I wouldn't expect a stop/start of the service to have any effect.

Yes on thread logs, No connect logs (but definitely willing to gather them if you think it might help). Each task acts on 34/35 records and at startup takes between 1700ms and 2400ms. Over time that slows to between 4500ms and 5500ms (and goes right back to between 1700ms and 2400ms upon a restart). Obviously each task acts on a unique set of records and that would account for some difference in timing between tasks, but not that much.

A colleague suggested perfmon to track server peformance, haven't done that yet as I am not sure what specific thing(s) to track (is it possible to watch just the llserver process?!?).

Cheers...

David Templeton

John

I agree the restart shouldn’t have the significant impact you are seeing. I am trying to think what is consuming the memory etc. Okay, lets flip the question around, if are NOT running your tasks, does the ram and timing increase still?

As for watching, it will either be llserver or more recently llserverworker. But watching the process, while interesting, wouldn’t provide much more than validating what you are seeing of the increase. It may be worthwhile from that perspective but it would be lacking details as to what is going on

David

John Simon

Quoted David Templeton on 11/22/2016 07:58 AM:

...Okay, lets flip the question around, if are NOT running your tasks, does the ram and timing increase still?

I mentioned another job type where all it does is split/count (duplicating the rm reporting job but replacing all of the data gathering and inserting with a simple count to 100K). I started it on a VM on my home machine with the idea of letting it run a couple of days. I'll post the results later today or tomorrow.

Cheers...

John Simon

DT,

The results I promised...from the splitting/counting job...do not show the same slowdown. The total number of tasks processed in an hour bounce around within a fairly narrow range. I think that suggests that something in my records gathering job is causing a memory leak, now to go about tracking it down. There is a thread from a few years ago that mentions using webadmin.testargs&refcounts, so I'll need to track that down and see how to incorprate it in a way that makes sense.

Cheers...