Acquisition Directory¶

How it works¶

Scheduler cycle¶

The AcquisitionScheduler runs continuously inside the MasterServer. On each cycle it queries every destination that has acquisition enabled, then iterates over the Acquisition hosts associated with those destinations. For each (destination, host) pair that is not already running, it spawns an AcquisitionThread to process that host.

A host is only picked up when its Acquisition Time has elapsed since the last run. The thread count is capped by Scheduler.maxAcquisitionThreads (default 100). If the cap is reached the scheduler waits until threads finish before starting new ones.

AcquisitionThread lifecycle¶

1. Read the Directory field. Each non-empty line is a listing spec. If the entire content is wrapped in $(...) it is evaluated as a script on a DataMover first; the output of the script is then treated as the listing spec lines.
2. Parse each line — extract inline [options], trailing {regex} filename filter, $date tokens, and optional wildcard filter — and build the final remote path.
3. List the remote directory via a DataMover using the configured transfer module (FTP, SFTP, S3, HTTP, …). The raw listing output is stored in the Directory Listings panel.
4. Parse each file entry using an FTP-style parser (systemKey, date formats, language code). Each entry is checked against the file selection filters.
5. Act on matched entries: schedule them for retrieval (action=queue) or delete them from the remote server (action=delete).
6. Write progress to the Progress panel (the console log above) after each listing step.

Progress panel vs Directory Listings¶

Directory configuration¶

Plain text mode¶

Each non-empty line describes one remote directory to scan. All parts except the path are optional. Blank lines and lines starting with # are ignored.

Line syntax¶

[option1=value1;option2=value2]/path/to/$date/{filename-regex}

Each non-empty line in the Directory field describes one remote path to list. All parts except the path are optional.

The host form guide also shows this equivalent line structure:

Plain text examples¶

/data/ecmwf/$date/{.*\.grib2}
[fileage=<6h]/data/ecmwf/$date[datedelta=-1d]/{.*\.grib2}
# older data with size filter
[filesize=>1024;fileage=<48h]/archive/$date[dateformat=yyyy/MM/dd]/

Date tokens¶

Host field tokens¶

Available in the Directory field for dynamic path construction:

$host[name] $host[nickname] $host[host] $host[login] $host[comment] $host[userMail] $host[networkCode] $host[networkName] $transferMethod[name] $ectransModule[name]

The host form guide also lists these tokens as substituted before plain-text lines are processed or before the script runs.

Host¶

$host[name] $host[comment] $host[host] $host[login] $host[passwd] $host[userMail] $host[networkCode] $host[networkName] $host[nickname]

Transfer Method & Module¶

$transferMethod[name] $transferMethod[comment] $transferMethod[value] $ectransModule[name]

Date¶

$date $date[dateformat=…,datedelta=…] $dirdate

Script mode¶

If the entire Directory field content is wrapped in $( ... ), it is executed as a script on an allocated DataMover. The standard output of the script is then interpreted as the listing spec lines (one path per line). This allows dynamic directory list generation.

JavaScript example¶

// Build directory lines dynamically — one per output line
var today = new Date();
var yyyy  = today.getFullYear();
var mm    = String(today.getMonth() + 1).padStart(2, '0');
var dd    = String(today.getDate()).padStart(2, '0');
var base  = "/data/feed/" + yyyy + "/" + mm + "/" + dd + "/";
base + "{.*\\.nc}\n" +
"[fileage=<48h]" + base.replace(dd, String(today.getDate()-1).padStart(2,'0')) + "{.*\\.nc}";

Python example¶

from datetime import datetime, timedelta

base = "/data/feed/"
for delta in range(3):  # today and 2 previous days
    day = datetime.utcnow() - timedelta(days=delta)
    age = delta * 24
    print("[fileage=<" + str(age + 24) + "h]" + base + day.strftime("%Y/%m/%d") + "/{.*\\.nc}")

Use the Test on Server button to execute the script and inspect its output before saving.

Wildcard filter¶

The acquisition.wildcardFilter option appends a glob-style wildcard to the path sent to the remote server before listing. This lets the transfer module filter server-side, reducing the number of entries returned. The acquisition.regexPattern option applies an additional server-side regex filter at the listing call level.

File selection¶

Selection pipeline¶

Each file entry returned by the remote listing passes through the following checks in order. A file is selected only if all checks pass.

Listing parser options¶

When the transfer module returns a raw FTP-style listing, the following options control how it is parsed:

Scheduling¶

Action¶

Uniqueness & deduplication¶

Before scheduling, the scheduler checks whether the file already exists in the database using a unique key built from the destination name, effective target filename, and optionally the modification time.

Re-queueing¶

If a file already exists in the database, the scheduler evaluates whether it should be re-scheduled.

Target & metadata¶

Retrieval parameters¶

Advanced¶

Performance & parallelism¶

Timeout & interruption¶

Pattern matching¶

Diagnostics¶

Related¶

• Host Directory Field
• Dissemination Directory
• Replication, Source, Backup & Proxy Directory
• Transfer Modules