Skip to content

OpenECPDS Documentation

OpenECPDS

Our mission with OpenECPDS is to keep data moving.

Inspired by operational excellence. Powered by open-source innovation. Acquire from anywhere. Deliver everywhere. Connect with confidence. Share without limits.

OpenECPDS is a multi-purpose data repository — the Data Store — that delivers three strategic data-related services:

  • Data Acquisition — the automatic discovery and retrieval of data from data providers.
  • Data Dissemination — the automatic distribution of data products to remote sites.
  • Data Portal — the pulling and pushing of data initiated by remote sites.

Data Acquisition and Data Dissemination are active services initiated by OpenECPDS, whereas the Data Portal is a passive service triggered by incoming requests from remote sites. The Data Portal provides interactive access to the Dissemination and Acquisition services.

OpenECPDS enhances data services by integrating innovative technologies to streamline the acquisition, dissemination, and storage of data across diverse environments and protocols.

OpenECPDS is domain-agnostic, not specific to meteorology. Its origins at ECMWF reflect demanding operational requirements, not restrictions on the data it can handle. The core moves and stores files without requiring a particular scientific format or subject matter. Research, industry, public services and enterprise teams can use the same acquisition, dissemination and portal services to exchange their own datasets, automate partner delivery and bridge on-premises and cloud storage. Domain-specific integrations, such as WMO WIS2, are optional.

Why OpenECPDS

  • Acquire from anywhere

    Automatically discover and retrieve data from providers over FTP, SFTP, FTPS, HTTP/S, WebDAV, Amazon S3, Azure and Google Cloud Storage.

  • Deliver everywhere

    Disseminate products to more than 1,000 destinations across 80+ countries with a fully customisable, retry-aware transfer scheduler.

  • Data Portal for all users

    A built-in portal lets anonymous, self-registered, or authorised users push and pull data over FTP, SFTP, SCP, HTTPS, WebDAV and S3 — no custom integration needed. One platform handles both incoming and outgoing traffic.

  • Real-time notifications

    An embedded MQTT broker and client enable instant notifications and integration with the WMO WIS2 infrastructure.

  • Container-native

    The standalone image runs the full stack on a laptop for instant evaluation. For production, each component (Master, Data Mover, Monitor) has its own container and can be deployed independently or orchestrated at scale with Kubernetes.

  • Open and extensible

    New outgoing protocols plug into the transfer module framework; new incoming protocols plug into the server plugin framework (SshPlugin, HttpPlugin, FtpPlugin…). Open-source and actively developed by ECMWF.

Why OpenECPDS? →

Architecture at a glance

ECMWF Product Data Store (ECPDS)

Component Role
Master Server Central coordinator — authentication, metadata, scheduling, Data Mover allocation.
Mover Server (Data Mover) Moves bytes — connects to remote systems via transfer modules, stores/streams content.
Monitor Server Web monitoring interface for destinations, transfers and hosts.
Data Portal Incoming FTP/SFTP/SCP/HTTPS/WebDAV/S3 access for remote sites to push and pull data.
Database Persists metadata, destinations, hosts, transfers and history.

See the Architecture Overview for how these components work together, and Continental Data Movers for geographically distributed dissemination.

Core capabilities

Data exchange

  • Multiple protocols — FTP, SFTP, FTPS, SCP, HTTP/S, WebDAV, Amazon S3, Azure Blob and Google Cloud Storage. See Protocols & Connections.
  • Data Portal — incoming and outgoing in one platform — remote users can push data in and pull data out over FTP, SFTP, SCP, HTTPS, WebDAV and S3. Access can be restricted to authorised users, opened anonymously, or offered as self-service registration — all without any custom integration. Dissemination and Acquisition data is accessible through the same portal, making OpenECPDS a complete two-way data exchange hub. See Data Portal.
  • Automated acquisition and notifications — discover and filter provider files, parse listings and catalogues, and control deduplication. An embedded MQTT broker and client support delivery notifications and event-driven acquisition; WMO WIS2 integration is optional. See Acquisition and MQTT Overview.

Reliability and data lifecycle

  • Distributed delivery — scale transfer capacity across multiple Data Movers, with allocation, load balancing and geographically distributed delivery. See DataMover Allocation and Continental Data Movers.
  • Scheduling and transfer resilience — delivery windows, queue controls, retries, fallback Hosts, concurrency limits and throttling let each destination follow its own delivery policy. Host failover is not a guarantee of whole-platform high availability. See Transfer Scheduling & Queue Control and Host Failover.
  • Data lifecycle and integrity — hierarchy-free object storage with directory emulation, compression, checksums, configurable expiry, automatic cleanup and backup mapping to existing archives. Optional Storage Lifecycle Policies provide dry-run estimates and controlled reduction of redundant replicas while protecting originals and active work. See Object Storage, Additional Features and Storage Lifecycle Policies.

Operations and observability

  • Operational monitoring — inspect destinations, queues, transfer history and infrastructure through the Monitor, including a live 2D/3D Earth view of data flows. See Monitoring.
  • Logs, traces and metrics — structured operational logs, JMX/Hawtio and Prometheus integration, plus optional OpenTelemetry logs, distributed transfer traces and application metrics. An optional database journal safeguards delivery of persisted operational events; it is not a transactional audit trail. See Event Logging and JMX Monitoring.
  • Network diagnostics and tuning — per-connection TCP statistics and per-Host socket tuning where supported. Opt-in Network Weather combines transfer evidence with optional TCP-connect and MTR probes for approved paths. See Transfer Network Statistics, TCP & Network Tuning and Network Weather.

Security and access control

  • Controlled access — user permissions, secure transfer protocols, central TLS certificate administration, Monitor TOTP authentication and optional Monitor-only OpenID Connect login mapped to existing local accounts. See Users & Access Control, TLS Certificates, TOTP and OpenID Connect.

Integration and extensibility

  • APIs and automation — manage resources through the versioned REST API, submit files with the CLI, and integrate transfer workflows using scripts and handlers. See REST API, ECPDS CLI and Transfer Reliability & Automation.
  • Extensible plugin architecture — new outgoing protocols are added via the transfer module framework; new incoming (Data Portal) protocols via the server plugin framework (SshPlugin, HttpPlugin, FtpPlugin, MqttPlugin…). Both sides can be extended without modifying the core platform. Optional extensions can also provide identity integration and typed transfer-option metadata.

See Additional Features for a full breakdown of platform capabilities.

Support & resources