Troubleshooting distributed agent

On another post, I found that I can execute a distributed agent worker by doing the following:

In opentext.ini, set [Options] runWithoutLogin=true
Run func=distributedagent.rundistagent
Run func=distributedagent.runagent

I've recently developed a solution that uses the distributed agent, and I've had it working fine when running in Builder using these manual instructions. But when runnning OTCS as a service, sometimes it seems to just give up on me and never execute my jobs. When I dive back into the manual execution, after a couple tries it seems to start working fine again.

If I execute func=distributedagent.runagent while running OTCS as a service, it seems to never complete. Just hangs forever. Doesn't even timeout. I've tried looking for some control on the admin page, and all I found was a distributed agent worker thread status page. This page shows some worker threads, and they are all idle. I'm not really sure what I'm looking for there, but it seems to me that my jobs are not even being executed.

What are my options for troubleshooting this? Is there some quirkiness with distributed agents that makes them not run? How can I tell what the agent is doing when running as a service when the only logs are 0kb?

This is a dev CS10 system.

Find more posts tagged with

Comments

John W. Simon, Jr.

There is a document named “Distributed Agent Monitoring” that shows you some SQL you can run to give you an idea about what is going on behind the scene. I believe it is on the KC but support will have it if you can’t find it on the KC.

Do you have multiple servers running distributed agents? The DAAgent should only be running on 1 server (so running func=distributedagent.runDistAgent when the service is running will cause unexpected results).

Questions:

What does the [loader] section of your servers look like?

What priority did you make your new DA task?

You mention logs with 0KB, is it the distributedAgent logs you are looking at?

From: eLink Entry: Content Server IDE Development Forum [mailto:contentserver-ide@elinkkc.opentext.com]
Sent: Monday, September 15, 2014 11:13 AM
To: eLink Recipient
Subject: [EXTERNAL]Troubleshooting distributed agent

Troubleshooting distributed agent

Posted bybrettc@infograph.com (Conley, Brett) On 09/15/2014 12:09 PM

On another post, I found that I can execute a distributed agent worker by doing the following:

In opentext.ini, set [Options] runWithoutLogin=true
Run func=distributedagent.rundistagent
Run func=distributedagent.runagent

This is a dev CS10 system.

[To post a comment, use the normal reply function]
Forum:	Content Server IDE Development Forum
Content Server:	Knowledge Center

Brett_Conley

A little confused. You say it should only run on one server, but then say that it will be unpredictable if I run OTCS as a service. A service still only runs on one server. Did you mean to say it will be unpredictable if it runs on more than one thread? Even so, I can't see how that would be unpredictable, since that's how they're supposed to run.

The [loaders] section looks like any other default installation:

[loader]
load=sockserv;javaserver;agents;notify;wfagent
load_relagent=relagent
load_distributedagent=distributedagent
load_daagent=daagent
load_verify=verifyAgent

I've given all 5 of my new task types a priority of 10.

The logs I'm talking about are the distributedagent_run###.html logs. They are always at 0K. I assume it's because the output hasn't been flushed, but stopping the OTCS service (or closing Builder) doesn't cause any of these to contain any accessible data.

There's also a daagent_run###.html log that is pretty much useless since all it says is "Distributed Agent executing... Distributed Agent ended."

On a system where the distributed agent is still running along swimmingly, there is also a daagent_###.out log, but the info it contains doesn't appear to be very useful, and like I said, it's not there when I'm having problems with the DA.

I'll look at that monitoring doc and see if it has anything helpful.

Thanks.

John W. Simon, Jr.

Sorry for the confusion...

The DAAGENT runs on its own thread (the loader section is telling you this). If you have multiple servers it is possible (though not advised) to run it on more than one server. What I was trying to say is that when you are running it as a service AND running it as a request handler (func=distributedAgnt.RunDistAgent) you are in effect running it twice.

Their should also be a distributedAgent_run###.out log. This will tell you if the DAs are actually picking anything up.

Your priority will place your job/task behind just about all other jobs/tasks. With DAs the higher the number the more priority you are giving it (for example the new workflow stuff using DAs have a priority of 60, populate column/facet have a priority of 75, purge deleted items has a priority of 85).

Cheers...

Brett_Conley

Do I have to set debug= something specific to get the distributedAgent_run###.out logs?

The priority is intentional. The tasks are things that people can probably wait for, and besides, I don't want Dave upset with me for messing up his facet stuff.

Thanks for your help.

Kyle_Swidrovich

Hey Brett,

Had a chat here with Mark Simm.

The agent.runagent only controls the Agents executing from the AgentSchedule table and because the DA Framework is outside of it you have to call it separately. Here’s some information on it.

Running Agents in Builder/CSIDE

Shut down Builder/CSIDE/CS

Open opentext.ini file

Set RunWithoutLogin=TRUE in the options section

Save the changes to opentext.ini

Restart Builder/CSIDE.

Then hit the ?func=agent.runagent request handler which should run the agents

This is great for debugging your agent.

You should also ensure that it works via schedule though, which will mean setting up something for agent to do, using the real server, waiting for the time for it to occur, and then checking the side effects to ensure that your agent ran and did what it should.

Running Distributed Agents in Builder

Do the above configuration, then…

Run DA Agent (moves records from WorkerQueue to WorkerQueuePending)

?func=distributedagent.RunDistAgent

Run DA Worker (moves records from WorkerQueuePending to WorkerQueueCurrent and works on them)

?func=distributedagent.RunAgent

That helps with debugging stuff running manually.
Debug=2 + wantLogs=TRUE should be all you need to do your service-version debugging.

John W. Simon, Jr.

For us your priority would never work. Our system was so busy trying to keep up with the facet updating that it would never get to a priority that low. Hopefully you have a different experience (here is a query taken from that documentation I mentioned that will show you all the tasks waiting for a worker).

--
-- Tasks in WorkerQueuePending waiting for workers
--
SELECT TYPE,
         Priority,
         HandlerID,
         COUNT (*) AS "Total",
         MAX (QueueTime) AS "MaxQueueTime",
         MIN (QueueTime) AS "MinQueueTime"
    FROM WorkerQueuePending
GROUP BY TYPE, Priority, HandlerID
ORDER BY TYPE, Priority DESC, COUNT (*) DESC;

Kyle mentioned WorkerQueue and WorkerQueuePending. Attached is a drawing that gives a high level overview (in case you don't already have this information).

I thought of something else as well. The worker(s) begins and runs for 60 minutes even if there are no tasks to process, and the HTML file is not written until the worker is done. Then 30 seconds later the process starts over (this sleep time is controlled with an INI setting named "SleepIntervalSec"). Catching the HTML file with something other than 0KB might be quite a challenge!

When running the worker as a request handler you can pass in some extra parameters. I found this one most helpful ?func=distributedAgent.RunAgent&runIntervalSec=5 (it tells the worker to only run for 5 seconds instead of that 60 minutes mentioned earlier).

Cheers...

Distribute Agent - Drawing.pdf