Round Robin Job Prioritization

This page describes round robin job queueing in PiCache.

Overview

PiCache Round Robin job prioritization feature is intended to address scaling issues in which a large number of Perforce IPLM end-users create many long-running Data Management (Perforce) jobs.

The primary use case is a user-based round-robin mode. In this case, each user is assigned a separate job queue for the user's requested IPVs. The queue manager picks up a single job from the top of each user's queue, going through all queued up users, one at a time. This means that each users job queues receive an equal priority.

In default FIFO-like mode, if user A queues up 100 jobs, and user B queues 1 job afterwards for the same IPV, then user A's jobs will be prioritized until all of that user's 100 jobs are completed. After that user B's 1 job will be processed.

In Round Robin mode the same scenario is processed by processing one job from user A, then one job from user B, then back to user A and so on. In this scenario user B will only have to wait for user A's first job to complete.

Note that IPVs already in the cache should not be re-populated.

Round Robin job prioritization mode can be enabled with PiCache configuration settings.

A queuing setting was added for control of PiCache's queuing behavior. 

# queuing behavior
# queuing has 2 options:
#   ipv: (default) jobs are put into queues based on IPV names, and an IPV
#        queue is worked on by a PiCache back-end worker until it is drained;
#        this can be thought of as "IPV-concentration" behavior
#   user_rr: for user-based round-robin queuing behavior, where jobs are put
#            into queues based on the users (user names) that enqueued the jobs
#            and jobs are worked on in a round-robin manner so no one user
#            dominates PiCache work
#queuing = user_rr

Changing Queuing Behavior

When configured for IPV-Concentration queuing behavior, the PiCache front-end enqueues jobs into IPV job queues and the PiCache back-end dequeues and executes jobs from the IPV job queues. Likewise, when configured for User-Based Round-Robin queuing behavior, the front-end enqueues jobs into user job queues and the back-end dequeues and executes jobs from the user job queues. 

To change from IPV-Concentration queuing behavior to User-Based Round-Robin queuing behavior, or vice versa, before changing the queuing behavior the currently enqueued jobs should be finished executing, then PiCache stopped, reconfigured with the new queuing setting, and started. To determine when the jobs are finished executing, the following commands can be run:

$ /usr/share/mdx/redis/mdx-redis-bin-5.0.7/bin/redis-cli KEYS rq:queue:\*
(empty list or set)
$ /usr/share/mdx/redis/mdx-redis-bin-5.0.7/bin/redis-cli KEYS \*wip\*
(empty list or set)
$ /usr/share/mdx/redis/mdx-redis-bin-5.0.7/bin/redis-cli KEYS \*lock\*
(empty list or set)
$ /usr/share/mdx/redis/mdx-redis-bin-5.0.7/bin/redis-cli KEYS \*recovery\*
(empty list or set)

When all these commands return (empty list or set), all jobs have finished executing.

A multi-instance PiCache deployment is one in which the PiCache front-end is on a different machine than the PiCache back-end, or where there are multiple front-end instances and/or multiple back-end instances residing on different machines in a High Availability deployment. It is important not to mix different queuing behaviors between the instances. For example, if the front-end is configured for IPV-Concentration queuing behavior and the back-end is configured for User-Based Round-Robin queuing behavior, the front-end will enqueue jobs into IPV job queues which the back-end will not access and no jobs will be dequeued and executed.