Class TransferScheduler

java.lang.Object

ecmwf.common.technical.ThreadService.ConfigurableRunnable

ecmwf.common.technical.WakeupThread

ecmwf.common.ecaccess.ECaccessScheduler

ecmwf.common.ecaccess.MBeanScheduler

ecmwf.ecpds.master.transfer.TransferScheduler

All Implemented Interfaces:

MBeanService, MonitorCallback, Closeable, AutoCloseable, Runnable

public final class TransferScheduler
extends MBeanScheduler

The Class TransferScheduler.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	TransferScheduler.AcquisitionResult	The Class AcquisitionResult.
static final class	TransferScheduler.BackupResult	The Class BackupResult.
final class	TransferScheduler.DestinationThread	The Class DestinationThread.
static final class	TransferScheduler.DownloadResult	The Class DownloadResult.
static final class	TransferScheduler.ExecutionResult	The Class ExecutionResult.
static final class	TransferScheduler.FilterResult	The Class FilterResult.
class	TransferScheduler.MonitoringThread	The Class MonitoringThread.
static final class	TransferScheduler.PurgeResult	The Class PurgeResult.
static final class	TransferScheduler.ReplicateResult	The Class ReplicateResult.

Field Summary

Fields inherited from class ECaccessScheduler

NEXT_STEP_ABORT, NEXT_STEP_CONTINUE, NEXT_STEP_DELAY, NEXT_STEP_INIT, SCHEDULER_STATE_CLOSING, SCHEDULER_STATE_JAMMED, SCHEDULER_STATE_OFFLINE, SCHEDULER_STATE_ONHOLD, SCHEDULER_STATE_ONLINE, SCHEDULER_STATE_OPENING, states, steps

Constructor Summary

Constructors

Constructor	Description
TransferScheduler(String name)	Instantiates a new transfer scheduler.

Method Summary

Modifier and Type	Method	Description
static TransferScheduler.AcquisitionResult	acquisition(StatusUpdate out, String destinationName, Host host, String source, String pattern, boolean synchronous)	Version 1 is getting the list as an array of String and version 2 is getting the list as an InputStream.
static TransferScheduler.BackupResult	backup(Host hostForBackup, TransferServer[] servers, DataTransfer transfer)	Backup.
static void	check(TransferServer server, long ticket)	Check.
static void	check(ProxySocket proxySocket)	Check.
static void	del(TransferServer server, Host host, String source)	Del.
static TransferScheduler.DownloadResult	download(DataTransfer transfer, Host hostForSource, List<DataTransfer> relatedTransfers)	Download.
static TransferScheduler.ExecutionResult	execution(StatusUpdate out, String destinationName, Host host, String script)	Executing script against data mover (this is because the script might open a connection to a remote site to get the list of files to retrieve).
static String	fetchContent(Host host, String source, int maxBytes)	Fetch the raw content of a remote URL/path via the host's configured ECtrans module on a DataMover.
static TransferScheduler.FilterResult	filter(TransferServer[] serversList, DataFile dataFile, boolean remove)	Filter.
static ProxySocket	get(DataTransfer transfer, long remotePosn, long length)	Gets a ProxySocket to download a DataTransfer.
static ProxySocket	get(TransferServer server, Host host, String source, long remotePosn, long length, boolean removeOriginal)	Gets the.
String	getActivity()	Gets the activity.
Object	getAttribute(String attributeName)	Gets the attribute.
DataTransfer	getDestinationLastTransfer(String destinationName, boolean ok)	Gets the destination last transfer.
Destination[]	getDestinations()	Gets the destinations.
String	getDestinationStatus(String destinationName)	Gets the destination status.
String	getDestinationStatus(String destinationName, String statusCode)	Gets the destination status.
TransferScheduler.DestinationThread	getDestinationThread(String destinationName)	Gets the destination thread.
static Map<String,int[]>	getDownloadMetricsSnapshot()	Returns a snapshot of the current in-flight download counts for all data movers and volumes.
static String	getHostReport(Host proxyHost, Host host)	Gets the host report.
MBeanInfo	getMBeanInfo()	Gets the MBean info.
TransferScheduler.MonitoringThread	getMonitoringThread()	Gets the monitoring thread.
static String	getMoverReport(Host proxyHost)	Gets the mover report.
static int	getNumberOfDownloadsFor(TransferServer server, int fileSystem)	Allow getting the current number of downloads for the provided moverName and fileSystem.
int	getPendingDataTransfersCount(String destinationName)	Gets the pending data transfers count.
int	getPoolSize()	Gets the pool size.
static TransferServer	getTransferServerName(boolean debug, String transferGroupName, TransferServer originalMover, String moverList)	Randomly select a DataMover from a list in the format ({operator} TransferGroup) mover1,mover2 ...
static String	getTransferServerUsedForRetrieval(DataTransfer transfer)	Get the name of the TransferServer which was used to retrieve the file from the source host.
void	holdAllDestinations(String userName, String status, boolean graceful)	Hold all destinations.
void	holdDestination(String userName, String destinationName, String status, boolean graceful)	Hold destination.
void	initialize()	Initialize.
static boolean	isAcquisition(DataTransfer transfer)	Checks if is acquisition.
static String[]	list(TransferServer server, Host host, String source, String pattern)	List.
static void	mkdir(TransferServer server, Host host, String dir)	Mkdir.
static void	move(TransferServer server, Host host, String source, String target)	Move.
int	nextStep()	Next step.
void	notifyCompletion(DataTransfer transfer)	Notify completion.
void	notifyRequeue(DataTransfer transfer)	Notify requeue.
static TransferScheduler.PurgeResult	purge(TransferServer[] transfers, DataFile dataFile)	Purge.
static TransferScheduler.PurgeResult	purge(TransferServer[] transfers, Host proxyHost, DataFile dataFile)	Purge.
static void	purge(TransferServer[] servers, Host proxyHost, List<ExistingStorageDirectory> directories)	Purge.
static boolean	put(TransferServer[] servers, DataTransfer transfer, Host hostForSource)	Puts the.
static ProxySocket	put(TransferServer server, Host host, String target, long remotePosn, long size)	Puts the.
static void	registerMoverMetrics(String groupName, String serverName, int volumeCount)	Pre-registers a data mover in the metrics map so it appears in the download activity view immediately on connect, even before its first download.
static TransferScheduler.ReplicateResult	replicate(String sourceMoverName, TransferServer[] serversList, DataTransfer transfer)	Replicate.
void	restartAllDestinations(String userName, boolean graceful)	Restart all destinations.
void	restartDestination(String userName, String comment, String destinationName, boolean graceful, boolean reset)	Restart destination.
static void	rmdir(TransferServer server, Host host, String dir)	Rmdir.
boolean	setAttribute(String name, Object value)	Sets the attribute.
void	setPoolSize(int poolSize)	Sets the pool size.
void	shutdown()	Shutdown.
void	shutdown(boolean graceful)	Shutdown.
static long	size(TransferServer server, Host host, String source)	Size.
void	updateHost(Host host)	Update host.

Methods inherited from class MBeanScheduler

close, getMonitorName, invoke

Methods inherited from class ECaccessScheduler

configurableRun, deactivateMonitor, getDelay, getJammedTimeout, getLastStepTime, getMonitorManager, getNextStep, getSchedulerState, getStartDate, getStepTime, isInitialized, isJammed, isOnHold, isRunning, onHold, setDelay, setJammedTimeout, setOnhold, setTimeRanges, start, wakeup

Methods inherited from class WakeupThread

isSleeping, waitFor

Methods inherited from class ThreadService.ConfigurableRunnable

execute, execute, execute, getContextClassLoader, getPriority, getThreadName, interrupt, interrupted, isAlive, isStarted, join, join, run, setContextClassLoader, setForceCookie, setInheritCookie, setPriority, setThreadNameAndCookie

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TransferScheduler

public TransferScheduler(String name)

Instantiates a new transfer scheduler.

Parameters:

name - the name

Method Details

notifyCompletion

public void notifyCompletion(DataTransfer transfer)

Notify completion.

Parameters:

transfer - the transfer

getPoolSize

public int getPoolSize()

Gets the pool size.

Returns:

the pool size

setPoolSize

public void setPoolSize(int poolSize)

Sets the pool size.

Parameters:

poolSize - the new pool size

notifyRequeue

public void notifyRequeue(DataTransfer transfer)

Notify requeue.

initialize

public void initialize()

Initialize.

Overrides:

initialize in class ECaccessScheduler

getDestinationThread

public TransferScheduler.DestinationThread getDestinationThread(String destinationName)
                                                         throws MasterException

Gets the destination thread.

Parameters:

destinationName - the destination name

Returns:

the destination thread

Throws:

MasterException - the master exception

getDestinations

public Destination[] getDestinations()

Gets the destinations.

Returns:

the destinations

getDestinationLastTransfer

public DataTransfer getDestinationLastTransfer(String destinationName,
 boolean ok)

Gets the destination last transfer.

Parameters:

destinationName - the destination name

ok - the ok

Returns:

the destination last transfer

updateHost

public void updateHost(Host host)
                throws DataBaseException

Update host.

Parameters:

host - the host

Throws:

DataBaseException - the data base exception

getPendingDataTransfersCount

public int getPendingDataTransfersCount(String destinationName)

Gets the pending data transfers count.

Parameters:

destinationName - the destination name

Returns:

the pending data transfers count

holdDestination

public void holdDestination(String userName,
 String destinationName,
 String status,
 boolean graceful)
                     throws DataBaseException

Hold destination.

Parameters:

userName - the user name

destinationName - the destination name

status - the status

graceful - the graceful

Throws:

DataBaseException - the data base exception

holdAllDestinations

public void holdAllDestinations(String userName,
 String status,
 boolean graceful)
                         throws DataBaseException

Hold all destinations.

Parameters:

userName - the user name

status - the status

graceful - the graceful

Throws:

DataBaseException - the data base exception

getDestinationStatus

public String getDestinationStatus(String destinationName)
                            throws DataBaseException

Gets the destination status.

Parameters:

destinationName - the destination name

Returns:

the destination status

Throws:

DataBaseException - the data base exception

getDestinationStatus

public String getDestinationStatus(String destinationName,
 String statusCode)
                            throws DataBaseException

Gets the destination status.

Parameters:

destinationName - the destination name

statusCode - the Status Code

Returns:

the destination status

Throws:

DataBaseException - the data base exception

restartDestination

public void restartDestination(String userName,
 String comment,
 String destinationName,
 boolean graceful,
 boolean reset)
                        throws DataBaseException

Restart destination.

Parameters:

userName - the user name

comment - the comment

destinationName - the destination name

graceful - the graceful

reset - the reset

Throws:

DataBaseException - the data base exception

restartAllDestinations

public void restartAllDestinations(String userName,
 boolean graceful)
                            throws DataBaseException

Restart all destinations.

Parameters:

userName - the user name

graceful - the graceful

Throws:

DataBaseException - the data base exception

getMonitoringThread

public TransferScheduler.MonitoringThread getMonitoringThread()
                                                       throws MasterException

Gets the monitoring thread.

Returns:

the monitoring thread

Throws:

MasterException - the master exception

nextStep

public int nextStep()

Next step.

Specified by:

nextStep in class ECaccessScheduler

Returns:

the int

getActivity

public String getActivity()

Gets the activity.

Overrides:

getActivity in class ECaccessScheduler

Returns:

the activity

getTransferServerUsedForRetrieval

public static String getTransferServerUsedForRetrieval(DataTransfer transfer)

Get the name of the TransferServer which was used to retrieve the file from the source host.

Parameters:

transfer - the transfer

Returns:

the transfer server used for retrieval

getTransferServerName

public static TransferServer getTransferServerName(boolean debug,
 String transferGroupName,
 TransferServer originalMover,
 String moverList)
                                            throws IOException

Randomly select a DataMover from a list in the format ({operator} TransferGroup) mover1,mover2 ... (e.g. (== internet) ecpds-dm1,ecpds-dm2). If there is no DataMover found then return the original DataMover.

Parameters:

debug - the debug

transferGroupName - the transfer group name

originalMover - the original mover

moverList - the mover list

Returns:

the transfer server name

Throws:

IOException - Signals that an I/O exception has occurred.

put

public static ProxySocket put(TransferServer server,
 Host host,
 String target,
 long remotePosn,
 long size)
                       throws MasterException

Puts the.

Parameters:

server - the server

host - the host

target - the target

remotePosn - the remote posn

size - the size

Returns:

the proxy socket

Throws:

MasterException - the master exception

get

public static ProxySocket get(TransferServer server,
 Host host,
 String source,
 long remotePosn,
 long length,
 boolean removeOriginal)
                       throws MasterException

Gets the.

Parameters:

server - the server

host - the host

source - the source

remotePosn - the remote posn

length - the length

removeOriginal - the remove original

Returns:

the proxy socket

Throws:

MasterException - the master exception

get

public static ProxySocket get(DataTransfer transfer,
 long remotePosn,
 long length)
                       throws DataBaseException,
MasterException,
TransferServerProvider.TransferServerException

Gets a ProxySocket to download a DataTransfer.

Parameters:

transfer - the transfer

remotePosn - the remote posn

length - the length

Returns:

the proxy socket

Throws:

DataBaseException - the data base exception

MasterException - the master exception

TransferServerProvider.TransferServerException

check

public static void check(ProxySocket proxySocket)
                  throws IOException

Check.

Parameters:

proxySocket - the proxy socket

Throws:

IOException - Signals that an I/O exception has occurred.

size

public static long size(TransferServer server,
 Host host,
 String source)
                 throws MasterException

Size.

Parameters:

server - the server

host - the host

source - the source

Returns:

the long

Throws:

MasterException - the master exception

list

public static String[] list(TransferServer server,
 Host host,
 String source,
 String pattern)
                     throws MasterException

List.

Parameters:

server - the server

host - the host

source - the source

pattern - the pattern

Returns:

the string[]

Throws:

MasterException - the master exception

del

public static void del(TransferServer server,
 Host host,
 String source)
                throws MasterException

Del.

Parameters:

server - the server

host - the host

source - the source

Throws:

MasterException - the master exception

check

public static void check(TransferServer server,
 long ticket)
                  throws MasterException

Check.

Parameters:

server - the server

ticket - the ticket

Throws:

MasterException - the master exception

mkdir

public static void mkdir(TransferServer server,
 Host host,
 String dir)
                  throws MasterException

Mkdir.

Parameters:

server - the server

host - the host

dir - the dir

Throws:

MasterException - the master exception

rmdir

public static void rmdir(TransferServer server,
 Host host,
 String dir)
                  throws MasterException

Rmdir.

Parameters:

server - the server

host - the host

dir - the dir

Throws:

MasterException - the master exception

isAcquisition

public static boolean isAcquisition(DataTransfer transfer)

Checks if is acquisition.

Parameters:

transfer - the transfer

Returns:

true, if is acquisition

move

public static void move(TransferServer server,
 Host host,
 String source,
 String target)
                 throws MasterException

Move.

Parameters:

server - the server

host - the host

source - the source

target - the target

Throws:

MasterException - the master exception

put

public static boolean put(TransferServer[] servers,
 DataTransfer transfer,
 Host hostForSource)

Puts the.

Parameters:

servers - the servers

transfer - the transfer

hostForSource - the host for source

Returns:

true, if successful

filter

public static TransferScheduler.FilterResult filter(TransferServer[] serversList,
 DataFile dataFile,
 boolean remove)

Filter.

Parameters:

serversList - the servers list

dataFile - the data file

remove - the remove

Returns:

the filter result

replicate

public static TransferScheduler.ReplicateResult replicate(String sourceMoverName,
 TransferServer[] serversList,
 DataTransfer transfer)

Replicate.

Parameters:

serversList - the servers list

transfer - the transfer

Returns:

the replicate result

backup

public static TransferScheduler.BackupResult backup(Host hostForBackup,
 TransferServer[] servers,
 DataTransfer transfer)
                                             throws DataBaseException

Backup.

Parameters:

hostForBackup - the host for backup

servers - the servers

transfer - the transfer

Returns:

the backup result

Throws:

DataBaseException - the data base exception

acquisition

public static TransferScheduler.AcquisitionResult acquisition(StatusUpdate out,
 String destinationName,
 Host host,
 String source,
 String pattern,
 boolean synchronous)
                                                       throws TransferServerProvider.TransferServerException,
DataBaseException,
IOException

Version 1 is getting the list as an array of String and version 2 is getting the list as an InputStream.

Parameters:

out - the out

destinationName - the destination name

host - the host

source - the source

pattern - the pattern

synchronous - the synchronous

Returns:

the acquisition result

Throws:

TransferServerProvider.TransferServerException - the transfer server exception

DataBaseException - the data base exception

IOException - Signals that an I/O exception has occurred.

execution

public static TransferScheduler.ExecutionResult execution(StatusUpdate out,
 String destinationName,
 Host host,
 String script)
                                                   throws TransferServerProvider.TransferServerException,
DataBaseException,
IOException

Executing script against data mover (this is because the script might open a connection to a remote site to get the list of files to retrieve).

Parameters:

out - the out

destinationName - the destination name

host - the host

script - the script

Returns:

the execution result

Throws:

TransferServerProvider.TransferServerException - the transfer server exception

DataBaseException - the data base exception

IOException - Signals that an I/O exception has occurred.

fetchContent

public static String fetchContent(Host host,
 String source,
 int maxBytes)
                           throws TransferServerProvider.TransferServerException,
DataBaseException,
IOException

Fetch the raw content of a remote URL/path via the host's configured ECtrans module on a DataMover. Uses the same DataMover selection logic as execution(StatusUpdate, String, Host, String).

Parameters:

host - the host whose ECtrans module is used for retrieval

source - the remote URL or path to retrieve

maxBytes - maximum number of bytes to return

Returns:

GZIPped InputStream of the content

Throws:

TransferServerProvider.TransferServerException - the transfer server exception

DataBaseException - the data base exception

IOException - if no DataMover available or retrieval fails

getNumberOfDownloadsFor

public static int getNumberOfDownloadsFor(TransferServer server,
 int fileSystem)

Allow getting the current number of downloads for the provided moverName and fileSystem.

Parameters:

server - the server

fileSystem - the file system

Returns:

the number of downloads for

registerMoverMetrics

public static void registerMoverMetrics(String groupName,
 String serverName,
 int volumeCount)

Pre-registers a data mover in the metrics map so it appears in the download activity view immediately on connect, even before its first download. If an entry for this mover already exists, it is left untouched.

Parameters:

groupName - the transfer group name

serverName - the transfer server (data mover) name

volumeCount - the number of volumes (filesystems) on this mover

getDownloadMetricsSnapshot

public static Map<String,int[]> getDownloadMetricsSnapshot()

Returns a snapshot of the current in-flight download counts for all data movers and volumes.

The returned map is keyed by "groupName.moverName" (e.g. "group-04.bodh1ecpdmv-04"). The value is an int[] where index i is the current download count on volume i. The array is sized to maxVolumeIndex + 1 for that mover; missing volume indices default to 0.

The snapshot is safe for RMI serialisation — it is a deep copy of the live concurrent map.

Returns:

a sorted snapshot of download counts; never null; may be empty if no transfers have started yet

download

public static TransferScheduler.DownloadResult download(DataTransfer transfer,
 Host hostForSource,
 List<DataTransfer> relatedTransfers)
                                                 throws TransferServerProvider.TransferServerException,
DataBaseException

Download.

Parameters:

transfer - the transfer

hostForSource - the host for source

relatedTransfers - the related transfers

Returns:

the download result

Throws:

TransferServerProvider.TransferServerException - the transfer server exception

DataBaseException - the data base exception

purge

public static TransferScheduler.PurgeResult purge(TransferServer[] transfers,
 DataFile dataFile)

Purge.

Parameters:

transfers - the transfers

dataFile - the data file

Returns:

the purge result

purge

public static TransferScheduler.PurgeResult purge(TransferServer[] transfers,
 Host proxyHost,
 DataFile dataFile)

Purge.

Parameters:

transfers - the transfers

proxyHost - the proxy host

dataFile - the data file

Returns:

the purge result

purge

public static void purge(TransferServer[] servers,
 Host proxyHost,
 List<ExistingStorageDirectory> directories)

Purge.

Parameters:

servers - the servers

proxyHost - the proxy host

directories - the directories

getMoverReport

public static String getMoverReport(Host proxyHost)
                             throws DataBaseException,
IOException

Gets the mover report.

Parameters:

proxyHost - the proxy host

Returns:

the mover report

Throws:

DataBaseException - the data base exception

IOException - Signals that an I/O exception has occurred.

getHostReport

public static String getHostReport(Host proxyHost,
 Host host)
                            throws DataBaseException,
IOException

Gets the host report.

Parameters:

proxyHost - the proxy host

host - the host

Returns:

the host report

Throws:

DataBaseException - the data base exception

IOException - Signals that an I/O exception has occurred.

shutdown

public void shutdown()

Shutdown.

Overrides:

shutdown in class MBeanScheduler

shutdown

public void shutdown(boolean graceful)

Shutdown.

Parameters:

graceful - the graceful

getMBeanInfo

public MBeanInfo getMBeanInfo()

Gets the MBean info.

Specified by:

getMBeanInfo in interface MBeanService

Overrides:

getMBeanInfo in class MBeanScheduler

Returns:

the MBean info

getAttribute

public Object getAttribute(String attributeName)
                    throws AttributeNotFoundException,
MBeanException

Gets the attribute.

Specified by:

getAttribute in interface MBeanService

Overrides:

getAttribute in class MBeanScheduler

Parameters:

attributeName - the attribute name

Returns:

the attribute

Throws:

AttributeNotFoundException - the attribute not found exception

MBeanException - the MBean exception

setAttribute

public boolean setAttribute(String name,
 Object value)
                     throws InvalidAttributeValueException,
MBeanException

Sets the attribute.

Specified by:

setAttribute in interface MBeanService

Overrides:

setAttribute in class MBeanScheduler

Parameters:

name - the name

value - the value

Returns:

true, if successful

Throws:

InvalidAttributeValueException - the invalid attribute value exception

MBeanException - the MBean exception