dm_job.run_now=1 is ignored

Hi. Our app UI needs to trigger a long-running, resource-consuming activity within 5 minutes after the button click. We created a job for that and set run_now=1 on the job object.
However, sometimes the job only starts on its schedule which is many hours ahead ignoring run_now.

The job is not disabled and it's not like we're out of runners.

select a_last_invocation from dm_job order by 1 desc enable (RETURN_TOP 15);

08.2020 10:25:14
08.2020 10:25:13
08.2020 10:24:44
08.2020 10:24:43
08.2020 10:24:14
08.2020 10:24:13
08.2020 10:22:43
08.2020 10:22:42
08.2020 10:22:13
08.2020 10:22:12
08.2020 10:21:42 - this seems to be a good slot to start it
08.2020 10:20:42
08.2020 10:20:12
08.2020 10:20:11
08.2020 10:19:11

Job dump.

USER ATTRIBUTES

  object_name                     : EfxOnix3RequestExportJob
  title                           : 
  subject                         : 
  authors                       []: <none>
  keywords                      []: <none>
  resolution_label                : 
  owner_name                      : efxam
  owner_permit                    : 7
  group_name                      : docu
  group_permit                    : 1
  world_permit                    : 6
  log_entry                       : 
  acl_domain                      : efxa
  acl_name                        : efx_acl_write_all
  language_code                   : 
  method_name                     : EfxOnix3RequestExportJob
  method_arguments             [0]: -docbase_name efxabase
  pass_standard_arguments         : F
  start_date                      : 11.08.2020 1:00:00
  expiration_date                 : 11.08.2056 1:00:00
  max_iterations                  : 0
  run_interval                    : 1
  run_mode                        : 3
  is_inactive                     : F
  inactivate_after_failure        : F
  target_server                   : 
  method_trace_level              : 0
  run_now                         : T
  method_data                   []: <none>

SYSTEM ATTRIBUTES

  r_object_type                   : dm_job
  r_creation_date                 : 17.08.2020 5:19:23
  r_modify_date                   : 17.08.2020 9:54:25
  r_modifier                      : Iris
  r_access_date                   : nulldate
  r_composite_id                []: <none>
  r_composite_label             []: <none>
  r_component_label             []: <none>
  r_order_no                    []: <none>
  r_link_cnt                      : 0
  r_link_high_cnt                 : 0
  r_assembled_from_id             : 0000000000000000
  r_frzn_assembly_cnt             : 0
  r_has_frzn_assembly             : F
  r_is_virtual_doc                : 0
  r_page_cnt                      : 0
  r_content_size                  : 0
  r_lock_owner                    : 
  r_lock_date                     : nulldate
  r_lock_machine                  : 
  r_version_label              [0]: 1.0
                               [1]: CURRENT
  r_immutable_flag                : F
  r_frozen_flag                   : F
  r_has_events                    : F
  r_creator_name                  : efxam
  r_is_public                     : T
  r_policy_id                     : 0000000000000000
  r_resume_state                  : 0
  r_current_state                 : 0
  r_alias_set_id                  : 0000000000000000
  r_full_content_size             : 0
  r_aspect_name                 []: <none>
  r_object_id                     : 0802254b807f3fdc

APPLICATION ATTRIBUTES

  a_application_type              : 
  a_status                        : 
  a_is_hidden                     : F
  a_retention_date                : nulldate
  a_archive                       : F
  a_compound_architecture         : 
  a_link_resolved                 : F
  a_content_type                  : 
  a_full_text                     : T
  a_storage_type                  : 
  a_special_app                   : 
  a_effective_date              []: <none>
  a_expiration_date             []: <none>
  a_publish_formats             []: <none>
  a_effective_label             []: <none>
  a_effective_flag              []: <none>
  a_category                      : 
  a_is_template                   : F
  a_controlling_app               : 
  a_extended_properties         []: <none>
  a_is_signed                     : F
  a_last_review_date              : nulldate
  a_continuation_interval         : 0
  a_next_continuation             : nulldate
  a_is_continued                  : F
  a_last_invocation               : 17.08.2020 5:19:55
  a_last_completion               : 17.08.2020 5:19:56
  a_last_return_code              : 0
  a_last_process_id               : 26111
  a_last_document_id              : 0902254b807f3515
  a_iterations                    : 1
  a_current_status                : COMPLETED
  a_next_invocation               : 18.08.2020 1:00:00

INTERNAL ATTRIBUTES

  i_is_deleted                    : F
  i_reference_cnt                 : 1
  i_has_folder                    : T
  i_folder_id                  [0]: 0b02254b80033505
  i_contents_id                   : 0000000000000000
  i_cabinet_id                    : 0c02254b80000106
  i_antecedent_id                 : 0000000000000000
  i_chronicle_id                  : 0802254b807f3fdc
  i_latest_flag                   : T
  i_branch_cnt                    : 0
  i_direct_dsc                    : F
  i_is_reference                  : F
  i_retain_until                  : nulldate
  i_retainer_id                 []: <none>
  i_partition                     : 0
  i_is_replica                    : F
  i_vstamp                        : 11

Find more posts tagged with

Comments

Hicham Bahi

The agent exec process is responsible for launching the jobs. By default, it polls jobs every minute so you shouldn't have to wait more than a minute, unless there's more jobs to launch than the maximum configured number, which is 3 by default. Perhaps you should enable trace_launch on the associated method to check if perhaps it launches but some other errors occurs ?

ilya basin

I found the reason: the scheduler implementation in the agent has no fairness, i.e. instead of selecting all pending jobs and starting them one-by-one as the runners become available it always selects the first few pending jobs ordered by a_next_invocation.

If during the upcoming queue checks there are enough scheduled jobs then a job with run_now=1 has no chance to be picked.

The workaround is to set a_next_invocation to some date in the past. However, this has a side effect of changing the configured schedule of the job.

Hicham Bahi

You can increase the number of jobs that agentexec will launch each poll interval. You need to update the method_verb of the dm_method called agent_exec_method, e.g.:

.\dm_agent_exec -max_concurrent_jobs 5

Checkout the server admin guide for more details.

DCTM_Guru

Alternatively, instead of waiting for agent exec to trigger your job, you can manually execute method that corresponds to the job. Presumably, you have access to Documentum Administrator when you set the job to run now. If that's the case, you can run apply API and trigger method to be launched independent of agent exec.

crazytraveller

As @DCTM_Guru suggested, its better to invoke a method directly for such use case instead of waiting for job execution

ilya basin

I've just learned that the problem only occurs in cluster configurations. Compare the agentexec.log of clustered:

AND (target_server = ' ' OR target_server is NULL) ORDER BY a_next_invocation, r_object_id ENABLE (RETURN_TOP 3)

and non-clustered agent:

ORDER BY run_now DESC, a_next_invocation, r_object_id

So it's a bug after all. The developers wanted to prioritize run_now, but they forgot to do that for clusters.