Plugin Infrastructure¶

OpenECPDS supports complementary extension points for identity and data exchange:

• Identity Extensions — alternative Master WebUser authentication and external user lookup
• Remote Service Extensions — optional Master-provided services for trusted RMI peers
• Transfer Modules — extend outgoing connections (acquisition and dissemination)
• Server Plugins — extend incoming connections (Data Portal)

Implement the relevant interface and explicitly register its implementation. Transfer modules and server plugins retain their existing loaders; identity and remote-service extensions use explicit startup-only registration.

Identity extensions¶

The ecmwf.ecpds.extension package provides the initial OpenECPDS Extension SPI:

These are Master-side hooks for account credential validation. Authentication does not grant permissions, create a WebUser or bypass the active WebUser check. The core retains authorization and database persistence.

Installation and configuration¶

Compile providers against the matching ecmwf:ecpds-core artifact. Do not bundle copies of OpenECPDS classes in the extension jar. Install trusted provider jars and their dependencies on the Master application classpath. A [PluginList] entry or transfer-module archive does not make an identity provider available.

Enable providers in the Master's local ecmwf.properties override:

[AuthenticationExtension]
class=org.example.ecpds.CompanyAuthentication
endpoint=https://identity.example.invalid
required=yes

[UserDirectoryExtension]
class=org.example.ecpds.CompanyDirectory
endpoint=https://directory.example.invalid

The example class names illustrate administrator-supplied implementations; they are not bundled providers. The bundled ecmwf.ecpds.extension.ecmwf.EcmwfTotpAuthenticationProvider adapts the existing ECMWF [TOTP] endpoint for Master WebUser and synchronized FTP/SFTP/Data Portal credential checks. With a blank class, that provider is selected automatically when [TOTP] active=true; otherwise blank preserves local WebUser password validation. Setting a class explicitly replaces the automatic ECMWF selection. Each provider class must be public, implement the appropriate SPI interface, and have a public no-argument constructor. Missing or blank class disables an optional role unless documented automatic selection applies. Installing a jar alone never activates it.

Set required=yes in [AuthenticationExtension] to fail Master startup if automatic provider selection leaves authentication unconfigured. It defaults to no; an explicitly configured provider that fails to load always fails startup regardless of this setting.

Providers receive an immutable configuration snapshot through ExtensionContext.option(name) and the local service identity through root(). Additional keys belong to the provider. Protect configuration containing secrets; never log credentials or option values.

One instance is created per configured role, even if both roles name the same class. Authentication starts before the directory; shutdown is in reverse order. A failed startup closes instantiated providers and fails Master startup, rather than silently reverting to built-in authentication. An incompatible apiVersion() is rejected before start is called.

Providers must be thread-safe, bound their network waits and release resources in close, including resources allocated during failed initialization. Configuration and implementation changes require a Master restart. Extensions are trusted in-process Java code, not sandboxed applications.

Authentication policy¶

The existing checks and authentication ordering remain in place:

1. Empty credentials are rejected; the ECUser must exist or be imported, and a corresponding active WebUser must already exist locally.
2. Existing configured static/anonymous and certificate authentication paths retain precedence for WebUser login.
3. An active AuthenticationProvider validates ordinary WebUser credentials. The ECMWF provider auto-selected by [TOTP] active=true preserves the existing password/passcode endpoint behavior. Otherwise a configured provider replaces local WebUser password validation; with no provider, local validation is unchanged.
4. For active synchronized incoming users, credential validation is delegated to the same provider. Inactive providers or rejected credentials do not fall back to a local password. Self-service subscriber-password handling and open-access users retain their existing policy.

An authentication provider returns ACCEPTED, REJECTED or UNSUPPORTED. Only ACCEPTED succeeds. Neither rejection, an unsupported request, a null result nor a provider outage falls back to a local password. Throw IOException for an outage; return REJECTED for invalid credentials. Provider failures are reported to clients as a generic authentication failure.

During login, directory lookup runs only when the ECUser is missing locally. Return an ECUser with exactly the requested name, or null when absent; throw IOException for an outage. Imported users do not automatically receive a WebUser or permissions. Providers can override authenticateIdentity to return an accepted result with fresh ECUser data. The core verifies the exact requested name and persists the update before accepting login; failed persistence fails authentication. Providers that implement only authenticate retain their original behavior. Existing scheduled ECUser refreshes, when configured, also use the directory lookup hook.

Minimal authentication implementation¶

public final class CompanyAuthentication implements AuthenticationProvider {
    private IdentityClient client;

    @Override
    public int apiVersion() {
        return API_VERSION;
    }

    @Override
    public void start(ExtensionContext context) throws IOException {
        client = new IdentityClient(context.option("endpoint"));
    }

    @Override
    public Result authenticate(String user, String credentials, String origin)
            throws IOException {
        return client.validate(user, credentials) ? Result.ACCEPTED : Result.REJECTED;
    }

    @Override
    public void close() throws IOException {
        if (client != null) {
            client.close();
        }
    }
}

This sketch uses an integration-specific IdentityClient; it is not an OpenECPDS class. Its implementation must set connection/read timeouts and keep credentials out of logs and exception messages.

ECaccess integration boundary¶

The optional ecpds-ecaccess module implements ECCMD directory lookup without copying MasterServer. It is built only with the ecaccess-extension Maven profile and a separately extracted client-contract jar. See ECaccess directory for build and configuration. Credentials remain with core TOTP authentication; this adapter does not validate passwords or passcodes. The same optional jar supplies ECaccess gateway transfers through the independent remote-service SPI below. Gateway operations and legacy wire types remain entirely in that jar; the identity SPI does not dispatch them.

Remote service extensions¶

RemoteServiceProvider supplies Master-side services needed by trusted Movers. It extends the same versioned Extension lifecycle and returns java.rmi.Remote from open(target). The provider defines and validates the target, and owns its remote interface and exported objects. Core has no knowledge of the service's operations, protocol or implementation-specific data types.

Register stable service IDs and their configuration section names explicitly:

[RemoteServiceExtensions]
company=CompanyService

[CompanyService]
class=org.example.ecpds.CompanyRemoteService
endpoint=...

The provider must be public with a public no-argument constructor. Each registration creates an independent instance with its section's immutable options and the Master root. IDs match [A-Za-z][A-Za-z0-9_.-]{0,63}. Providers start in service-ID order before Master schedulers start and close in reverse order, including when startup fails. Missing configuration, an invalid class or an incompatible API fails startup; remove a registration to disable it. With no registrations, no provider is loaded.

Movers call MasterProxy.getExtensionService(serviceId, target) over Master RMI and cast the result to the extension's remote interface. Install the trusted interface jar on both peers' application classpaths; runtime class download is not required. A provider returns an exported object/stub or null for unavailability; errors fail the request without core retries. Remote operations are declared on the extension interface, not dispatched through reflective method names or arbitrary class names supplied by callers.

This is a trusted in-process SPI using the existing Master RMI trust boundary, not a public REST API or an authorization sandbox. The REST Master proxy rejects this lookup explicitly. Providers must bound waits, support concurrent calls, and release exports on close and when clients abandon them. The extension defines its peer-interface compatibility; deploy matching jars on both sides.

Interactive login extensions¶

Interactive login is independent of credential validation. An implementation of InteractiveLoginProvider supplies a display name, an HTTPS authorization URL and protocol validation returning ExternalIdentity(issuer, subject).

The Master owns random state, nonce, S256 PKCE secrets, expiry, single-use transactions and explicit local-account mapping. The Monitor keeps a random binding in its HTTPS browser session and uses the existing management connection to start and complete login. Neither the Monitor nor a browser may submit a trusted username or identity in place of protocol verification.

Enable one provider with [InteractiveLoginExtension] class on the Master. Map verified identities through [InteractiveLoginMapping], with the local WebUser ID as key and a JSON object containing issuer and subject as value. Duplicate identities fail startup. Unmapped identities and missing, disabled or anonymous local accounts cannot log in. No account provisioning or permission assignment occurs.

The provider uses the same API-version and startup/shutdown lifecycle as other identity extensions. It starts after authentication and directory providers. An active interactive provider adds a separate button; existing credential, certificate and TOTP login behavior is unchanged. Interactive login deliberately does not run through the credential-validation SPI or require a local password.

The initial bundled example is ecmwf.ecpds.extension.oidc.OidcLoginProvider, implementing OIDC authorization-code login. See OpenID Connect login for setup, supported provider features and operational requirements.

Extension implementations receive protocol secrets and are trusted application code. They must verify the external protocol before returning an identity, bound network waits and avoid logging codes, tokens or credentials. Local logout clears the pending browser transaction and the application user; federated provider logout is not part of this initial interface.

Transfer Modules (outgoing)¶

Transfer modules plug into the ECtrans transfer engine used by Data Acquisition and Data Dissemination. Each module implements a protocol adapter that the engine calls to connect, authenticate, transfer data, list directories, and clean up.

Built-in modules cover:

Adding a new transfer module¶

A new outgoing protocol is added by implementing the ECtrans module interface and referencing the class from a host configuration. No changes to the core platform are required. The module receives connection parameters, handles the protocol lifecycle, and returns results to the transfer engine.

Extension-owned transfer options¶

Optional modules can define their own typed options without adding constants to the core ECtransOptions enum. Built-ins and extensions share the ECtransOptionDefinition contract, accepted by ECtransSetup getters, setters, script lookup and normalization. Existing enum-based methods remain available for already compiled modules.

Implement ecmwf.common.ectrans.ECtransOptionProvider in the extension:

package example.transfer;

import java.time.Duration;
import java.util.List;
import java.util.Properties;
import ecmwf.common.ectrans.*;

public final class ExampleOptions implements ECtransOptionProvider {
    private static final Properties HELP = new Properties();
    static {
        HELP.setProperty("example.timeout.tips", "Maximum time to wait for the remote service.");
        HELP.setProperty("example.token.secured", "true");
    }

    public static final ECtransOption<Duration> TIMEOUT = new ECtransOption<>(
        ECtransGroups.HOST, "example", "timeout", Duration.class,
        List.of(Duration.ofSeconds(30)), List.of(), List.of(), false, HELP);
    public static final ECtransOption<String> TOKEN = new ECtransOption<>(
        ECtransGroups.HOST, "example", "token", String.class,
        List.of(), List.of(), List.of(), false, HELP);

    @Override
    public List<ECtransOptionDefinition> options() {
        return List.of(TIMEOUT, TOKEN);
    }
}

In the extension JAR, include META-INF/services/ecmwf.common.ectrans.ECtransOptionProvider containing:

example.transfer.ExampleOptions

The module can then use setup.getDuration(ExampleOptions.TIMEOUT) and setup.getString(ExampleOptions.TOKEN). Host Properties remain ordinary example.timeout = "PT1M" and example.token = "..." entries. An empty default list means no default. Choices and SubOption definitions describe validation; the boolean constructor argument enables free-form nested options. Metadata can instead be loaded from an extension-owned properties resource using <module>.<name>.tips, .visible and .secured keys. Mark credentials .secured=true so redacted setup output masks them; visibility alone does not protect a secret.

Install the extension on the application classpath of the Master, Monitor and participating Movers/Proxies, then restart those services. Discovery is metadata-only and does not activate a transfer module or start external connections. Providers should therefore have a public no-argument constructor and avoid operational work or optional server dependencies during initialization. Archive-only transfer-module loading does not register application-wide options.

ECtransOptionRegistry.get(group) supplies the combined, immutable catalogue for editor completion/help, property validation, hidden-option handling and normalization; secret redaction also includes registered definitions. ECtransOptions.get(group) retains its built-in-enum-only API. Discovery uses the core application classloader at first catalogue access and is cached for the process lifetime, not the request/thread context classloader. Malformed service providers, invalid names and duplicate group/parameter names (including attempts to override built-ins) fail discovery. Use a unique lowercase alphanumeric module name starting with a letter, and alphanumeric option names starting with a letter.

The ECaccess extension uses this mechanism for its four ecaccess.* options. Their property names and defaults are unchanged, but their definitions and help are absent from the catalogue when the extension is not installed. Existing stored properties are not migrated or deleted by discovery.

Server Plugins (incoming)¶

Server plugins extend the Data Portal with new incoming protocols. Each plugin runs as an independent server, listening on its own port, authenticating users against the OpenECPDS user database, and delegating data operations to the mover's virtual filesystem.

The base classes in ecmwf.common.plugin provide:

Built-in server plugins on the Data Mover:

Plugins are registered in ecmwf.properties under [PluginList]:

[PluginList]
http=ecmwf.ecpds.mover.plugin.http.HttpPlugin
mqtt=ecmwf.ecpds.mover.plugin.mqtt.MqttPlugin
ftp=ecmwf.common.ftpd.FtpPlugin,maxConnections=2000,inverseResolution=no
ssh=ecmwf.ecpds.mover.plugin.ssh.SshPlugin
ecproxy=ecmwf.ecpds.mover.plugin.ecproxy.ECproxyPlugin,maxConnections=2000,inverseResolution=no

Adding a new server plugin¶

To add a new incoming protocol:

1. Extend ServerPlugin (or SimplePlugin for text-command protocols) from ecmwf.common.plugin.
2. Implement the required abstract methods:
3. getPort() — the TCP port to listen on
4. newInstance() — factory method called per accepted connection
5. startConnection(Socket) — handle one client session
6. refuseConnection(Socket, int) — called when connection limits are exceeded
7. Register the class in [PluginList] in ecmwf.properties.

The framework handles TCP accept loop, per-connection thread allocation, maximum connection enforcement, JMX MBean registration, and graceful shutdown automatically.

Extending OpenECPDS¶

These extension points are designed to be extended. If you need a protocol that is not yet supported — whether outgoing or incoming — ECMWF is happy to help guide or collaborate on integration and development.

Related¶

• Transfer Modules overview
• Protocols & Connections
• Data Portal
• Architecture Overview
• Contributing