Class TransferServerProvider

java.lang.Object
ecmwf.ecpds.master.transfer.TransferServerProvider
public final class TransferServerProvider extends Object
Central provider for selecting and ordering TransferServer instances in ECPDS. The provider is responsible for multi-level load distribution: cluster-level weighted round-robin (WRR), per-group server rotation, and storage-volume-aware allocation.

Scheduling Layers

  1. Group resolution: The provider resolves a TransferGroup from explicit name, destination mapping, or default configuration.
  2. Cluster WRR: If enabled by construction semantics, a group within the same cluster can be chosen via WRR. The RR counter resets on topology changes (weights / activation / availability) to avoid skew.
  3. Volume allocation: The selected group’s volume index is chosen either explicitly (pre-allocated) or via TransferServerProvider.WeightedAllocator, with optional size-aware penalties and CV-based uniform fallback.
  4. Server ordering: Within the group, active and reachable DataMovers are ordered by least filesystem activity on the selected volume, with caller-stable rotation as a tie-breaker. A preferred server, if provided, is reinserted at index 0.

State invalid input: '&' Caches

  • Instances are immutable w.r.t. selected group, volume index, and server ordering.
  • Static caches maintain cluster RR state, per-group RR counters, volume usage, and allocator statistics.
  • A background daemon refreshes volume usage and mover ordering.

Thread Safety

  • All public methods are read-only and safe for concurrent calls.
  • Static schedulers use ConcurrentHashMap and atomics.
  • Allocator uses per-group locking for consistent snapshots.

Failure Semantics

  • Constructor Details

    • TransferServerProvider

      public TransferServerProvider(String caller, String groupName, String destination, long fileSize) throws TransferServerProvider.TransferServerException, DataBaseException
      Convenience constructor that delegates to the canonical constructor with:
      • server = null
      • allocatedFileSystem = null
      See TransferServerProvider(String, String, String, long, TransferServer, Integer) for detailed behaviour.
      Parameters:
      caller - identifier of the caller (used for stable rotation seeding)
      groupName - explicit group name; if null, destination and/or default are used
      destination - optional destination name used for group resolution
      fileSize - expected file size in bytes (non-positive disables size-aware allocation)
      Throws:
      TransferServerProvider.TransferServerException - if no suitable group/server can be selected
      DataBaseException - on database errors

    • TransferServerProvider

      public TransferServerProvider(String caller, String groupName, String destination, TransferServer server, Integer allocatedFileSystem) throws TransferServerProvider.TransferServerException, DataBaseException
      Convenience constructor that delegates to the canonical constructor with:
      • fileSize = -1 (disables size-aware allocation)
      See TransferServerProvider(String, String, String, long, TransferServer, Integer) for detailed behaviour.
      Parameters:
      caller - identifier of the caller (used for stable rotation seeding)
      groupName - explicit group name; if null, destination and/or default are used
      destination - optional destination name used for group resolution
      server - optional preferred server to reinsert at index 0 if active/reachable
      allocatedFileSystem - optional explicit volume index; when null, the allocator is used
      Throws:
      TransferServerProvider.TransferServerException - if no suitable group/server can be selected
      DataBaseException - on database errors

    • TransferServerProvider

      public TransferServerProvider(String caller, String groupName, String destination) throws TransferServerProvider.TransferServerException, DataBaseException
      Convenience constructor that delegates to the canonical constructor with:
      • fileSize = -1
      • server = null
      • allocatedFileSystem = null
      See TransferServerProvider(String, String, String, long, TransferServer, Integer) for detailed behaviour.
      Parameters:
      caller - identifier of the caller (used for stable rotation seeding)
      groupName - explicit group name; if null, destination and/or default are used
      destination - optional destination name used for group resolution
      Throws:
      TransferServerProvider.TransferServerException - if no suitable group/server can be selected
      DataBaseException - on database errors

    • TransferServerProvider

      public TransferServerProvider(String caller, String groupName, String destination, Integer allocatedFileSystem) throws TransferServerProvider.TransferServerException, DataBaseException
      Convenience constructor that delegates to the canonical constructor with:
      • fileSize = -1 (disables size‑aware volume allocation)
      • server = null (no preferred server to reinsert)
      • allocatedFileSystem explicitly provided by the caller

      This overload is used when the caller wants to:

      • select a specific transfer group (via groupName or destination lookup),
      • optionally resolve the group based on destination,
      • force the use of a specific storage volume index (bypassing weighted allocation),
      • and does not wish to use size‑aware volume selection.

      The behaviour of group resolution, availability checks, and cluster‑level fallback follows the rules documented in the canonical constructor TransferServerProvider(String, String, String, long, TransferServer, Integer).

      Parameters:
      caller - identifier of the caller (used for stable rotation seeding)
      groupName - explicit group name; if null, group is resolved from destination or configuration
      destination - optional destination used for group resolution
      allocatedFileSystem - explicit volume index to use instead of weighted allocation
      Throws:
      TransferServerProvider.TransferServerException - if the resolved/selected group is not available, has no volumes, or if no active DataMover can be selected
      DataBaseException - if database queries for group, destination, or servers fail

    • TransferServerProvider

      public TransferServerProvider(String caller, String groupName) throws TransferServerProvider.TransferServerException, DataBaseException
      Convenience constructor that delegates to the canonical constructor with:
      • destinationName = null
      • fileSize = -1
      • server = null
      • allocatedFileSystem = null
      See TransferServerProvider(String, String, String, long, TransferServer, Integer) for detailed behaviour.
      Parameters:
      caller - identifier of the caller (used for stable rotation seeding)
      groupName - explicit group name; if null, the default is used
      Throws:
      TransferServerProvider.TransferServerException - if no suitable group/server can be selected
      DataBaseException - on database errors

    • TransferServerProvider

      public TransferServerProvider(String caller, String groupName, String destinationName, long fileSize, TransferServer server, Integer allocatedFileSystem) throws TransferServerProvider.TransferServerException, DataBaseException
      Constructs a provider and pre-computes:
      • the TransferGroup to use (explicit, destination-mapped, or default; may be adjusted within the cluster by WRR fallback),
      • the storage volume index (getFileSystem()), using either an explicit index or the TransferServerProvider.WeightedAllocator,
      • the ordered TransferServer list according to the least-activity policy (FS activity primary; rotation tie-breaker; optional preferred server on top).

      Group resolution invalid input: '&' fallback: The constructor first resolves the group via resolveTransferGroup(String, String). If allocatedFileSystem is null or if the resolved group is not available, it may select a different group within the same cluster using tryClusterFallback(TransferGroup), which in turn leverages cluster-wide WRR through selectGroupFromClusterRoundRobin(TransferGroup, TransferGroup[]).

      Volume selection: If allocatedFileSystem is non-null, it is used as-is. Otherwise, the TransferServerProvider.WeightedAllocator is invoked:

      • when fileSize > 0, TransferServerProvider.WeightedAllocator.allocate(TransferGroup, long) is used;
      • otherwise, TransferServerProvider.WeightedAllocator.allocate(TransferGroup) is used.

      Server ordering: The final least-activity ordering is computed once during construction via computeLeastActivityOrdering(String, TransferGroup, TransferServer, int). If the resulting active list is empty, construction fails.

      Thread-safety: The constructed instance is immutable with respect to selection decisions (group, volume index, server ordering). Static caches and background updaters are concurrency-safe.

      Parameters:
      caller - identifier of the caller (used for stable rotation seeding)
      groupName - explicit group name; if null, destinationName and/or the global default are used
      destinationName - optional destination used for group resolution (see resolveTransferGroup(String, String))
      fileSize - expected file size in bytes; non-positive values disable size-aware allocation
      server - optional preferred server to reinsert at index 0 if active/reachable
      allocatedFileSystem - optional explicit volume index; when null, the allocator is used
      Throws:
      TransferServerProvider.TransferServerException - if the resolved/selected group is not available, has no volumes, or if no active DataMover can be selected
      DataBaseException - if the database lookups fail for group, destination, or servers

  • Method Details

    • getVolumeSnapshot

      public static Map<String,long[][]> getVolumeSnapshot(String groupName)
      Returns a snapshot of the current volume disk usage for all registered transfer groups, or for a single group if groupName is non-null.

      Delegates to TransferServerProvider.WeightedAllocator.getVolumeSnapshot(String). The data comes from the in-memory cache maintained by the background usage updater — no DataMover RMI calls are made at query time.

      Parameters:
      groupName - the transfer group to query, or null for all groups
      Returns:
      a map of group name to long[volumeIndex][2] where [i][0] is used bytes and [i][1] is total bytes; never null; empty if no data has been collected yet

    • getFileSystem

      public int getFileSystem()

      How the index is obtained:

      • If the caller provided an explicit allocatedFileSystem in the constructor, that exact index is used.
      • Otherwise, the index is selected during construction by the TransferServerProvider.WeightedAllocator, using either size-aware weights (when a positive fileSize is given) or free-space weights.

      This value is immutable for the lifetime of this provider instance and is guaranteed to be within [0, group.getVolumeCount()).

      Returns:
      the selected (0-based) volume index for this provider instance

    • getTransferGroup

      public TransferGroup getTransferGroup()
      Returns the resolved and validated TransferGroup for this provider instance.

      The group is resolved at construction time using (in order):

      1. Explicit groupName, if provided
      2. Destination mapping (if a destinationName was provided)
      3. Global default transfer group (from configuration)

      After resolution, the provider may switch to a different group within the same cluster through WRR-based fallback if either:

      • the resolved group is not currently available, or
      • no explicit allocatedFileSystem was provided and cluster selection is enabled

      The returned group is guaranteed to be active and to have at least one active/connected DataMover at the time of construction; otherwise construction fails.

      Returns:
      the selected and available TransferGroup; never null

    • getTransferGroupName

      public String getTransferGroupName()
      Convenience accessor for getTransferGroup().getName().

      Equivalent to getTransferGroup().getName(), provided for call sites that only need the name.

      Returns:
      the name of the selected transfer group; never null

    • getTransferServersByLeastActivity

      public List<TransferServer> getTransferServersByLeastActivity()
      Returns a snapshot of the active TransferServer list for this provider instance, ordered by the “least activity” policy that was computed during construction.

      Ordering policy (computed at construction time):

      1. Caller-stable base rotation (deterministic hash of caller, group name, and runtime salt) combined with a per-group round-robin counter to mitigate burst skew.
      2. Filtering of inactive or unreachable servers (only active servers with a reachable DataMover interface remain).
      3. Primary sort by per-volume activity: servers with the fewest downloads on getFileSystem() come first.
      4. Tie-breaker by rotation order: among equal loads, the rotated declared order decides.
      5. If a preferred server was provided to the constructor and is available, it is reinserted at index 0.

      The returned list is an independent copy and can be freely modified by the caller. Subsequent calls return a fresh copy of the same internal ordering (the provider does not recompute ordering after construction).

      Thread-safety: This method performs no mutation and is safe for concurrent calls. The underlying ordering is immutable within this provider instance.

      Returns:
      a new List of active TransferServer instances, ordered by least activity

    • getTransferServersByMostFreeSpace

      public List<TransferServer> getTransferServersByMostFreeSpace()
      Returns the active TransferServer list ordered by highest free space on the selected volume (getFileSystem()), if a cache entry is available.

      Behaviour:

      • If a VolumeUsageResult is present in the internal cache for groupName + ":" + fileSystem, the method uses the mover ordering computed by MasterServer.computeAllVolumeUsageAndMovers(TransferGroup) (descending free space), intersected with the provider's active set.
      • If no cache entry exists, the method returns a shuffled copy of the provider's least-activity list as a best-effort fallback to avoid persistent hot-spotting.

      Scope: This method does not modify the provider's internal least-activity ordering; it returns a new list each time. The cache is populated asynchronously by the background usage updater.

      Thread-safety: Safe for concurrent callers; relies on immutable snapshots and concurrent maps.

      Returns:
      a new List of active TransferServer instances sorted by descending free space on the selected volume when cache is present; otherwise a shuffled best-effort ordering