DataMover Allocation & Load Balancing¶

When OpenECPDS needs to store or forward a file, it must select:

1. A Transfer Group — a logical cluster of DataMovers sharing the same storage pool.
2. A storage volume (filesystem index) within that group.
3. An ordered list of DataMover servers to try, from most to least preferred.

This process is handled by the TransferServerProvider class. Each provider instance resolves these three choices once at construction time and is immutable thereafter.

Transfer Group Resolution¶

The group is resolved through a fixed priority chain:

Cluster-Level Weighted Round Robin (WRR)¶

Transfer Groups can be grouped into named clusters. When a group belongs to a cluster and no explicit filesystem index was pre-allocated, the provider may switch to a different group in the same cluster using weighted round-robin (WRR) scheduling.

Eligibility¶

A group is a WRR candidate if:

• It belongs to the same cluster as the resolved group (clusterName field).
• Its clusterWeight is strictly positive.
• It is currently available: marked active and has at least one active, reachable DataMover.

Scheduling¶

Candidates are sorted alphabetically to ensure a stable order independent of database return ordering. An atomic counter is incremented per allocation; the counter slot maps into the weighted sum of candidates to select the next group.

When the set of eligible candidates changes (groups are enabled/disabled or weights change), the counter resets to avoid skew from stale rotation state.

When Fallback is Triggered¶

Cluster fallback occurs when:

• No filesystem was pre-allocated (allocatedFileSystem = null) — the scheduler deliberately allows load-balancing across cluster peers.
• The resolved group is not currently available — it is inactive or all its DataMovers are unreachable.

If the resolved group has no cluster name, or no eligible peer exists, the original group is returned unchanged.

Graceful Group Fallback¶

If, after cluster WRR, the selected group is still unavailable and an explicit group name was provided, the provider attempts one further fallback: it looks up the destination's own transferGroupName FK directly (bypassing any dissemination-host lookup).

This covers the common operational scenario where an operator has updated the Transfer Group in the Destination edit form, but the scheduler option or dissemination host has not yet been updated:

Warning: TransferGroup group-old not available for destination DEST_NAME;
         falling back to destination's configured TransferGroup group-new
         — update scheduler.transfergroup in the Destination properties
         to remove this warning

The warning is a reminder to resolve the underlying inconsistency.

Volume (Filesystem) Allocation¶

Each Transfer Group has one or more storage volumes (indexed from 0). A volume index is selected once per provider instance and used consistently for all operations with that instance.

Pre-allocated Index¶

If the caller supplies an explicit allocatedFileSystem value (e.g., during a retry of an existing transfer), that index is used directly and volume allocation is skipped.

Weighted Allocator¶

When no explicit index is given, the WeightedAllocator selects a volume:

The coefficient-of-variation (CV) test applies in both the standard and size-aware modes: if the standard deviation of volume weights divided by their mean falls below cvThreshold, uniform selection is used even when statistics are available.

Size-aware penalty formula¶

effective_free_space[i] = max(1, free_space[i] - min(free_space[i], α × fileSize))

Setting fileSizePenaltyFactor = 0 disables the penalty entirely. Setting fileSizePenaltyFactor = 1 (the default) subtracts the full file size from each volume's free space before selection.

Background Volume Updater¶

Volume usage statistics are refreshed by a background daemon thread. On each tick:

1. All Transfer Groups in the database are iterated.
2. Groups with no available DataMover are skipped.
3. For each remaining group, all DataMovers are contacted once via RMI to retrieve per-volume used/total bytes and per-volume mover ordering.
4. The WeightedAllocator statistics and the VOLUME_USAGE_CACHE are updated atomically.

Group updates within a single tick run concurrently (thread pool capped at 16), but per-volume calls within a single group remain sequential to avoid overwhelming the RMI layer.

The update frequency and other tuning parameters are described in the Configuration Parameters section below.

DataMover Server Ordering¶

Within the selected group and volume, active DataMover servers are ordered by a least-activity policy computed once per provider construction:

Pipeline¶

Access methods¶

Callers that want to retry on failure iterate the returned list from the beginning, attempting each server in order until one succeeds.

Configuration Parameters¶

All parameters are in the [TransferServerProvider] configuration section unless otherwise noted.

Summary: Selection Flow¶

┌─────────────────────────────────────────────────────────────────┐
│                TransferServerProvider construction              │
├─────────────────────────────────────────────────────────────────┤
│ 1. Resolve TransferGroup                                        │
│      a. Explicit groupName (scheduler.transfergroup) ?          │
│      b. Destination's dissemination host's group ?              │
│      c. Destination's own transferGroupName ?                   │
│      d. Server.defaultTransferGroup                             │
│                                                                 │
│ 2. Cluster WRR fallback                                         │
│      (if allocatedFileSystem=null OR group unavailable)         │
│      → select peer group within cluster by weighted RR          │
│                                                                 │
│ 3. Graceful group fallback                                       │
│      (if still unavailable AND explicit groupName was given)    │
│      → try destination's own transferGroupName directly         │
│                                                                 │
│ 4. Allocate volume                                              │
│      a. Explicit allocatedFileSystem → use as-is               │
│      b. fileSize > 0 → size-aware WeightedAllocator             │
│      c. otherwise   → free-space WeightedAllocator              │
│      (both b/c fall back to uniform random if CV ≤ threshold)  │
│                                                                 │
│ 5. Order DataMovers                                             │
│      Least-downloads on selected volume, rotation tie-breaker,  │
│      preferred server reinserted at position 0 if active        │
└─────────────────────────────────────────────────────────────────┘

Related¶

• Components
• Failover Mechanism
• OpenECPDS Entities
• Destination Options