Why OpenECPDS¶

OpenECPDS is an open-source, ECMWF-backed, distributed Managed File Transfer and data distribution platform built for organisations that cannot treat data exchange as an afterthought. It combines acquisition, dissemination, portal access, monitoring, and automation in one operational system designed for sustained, mission-critical use.

OpenECPDS is domain-agnostic: meteorology is its origin, not its boundary. Its core acquisition, storage, dissemination and portal services do not require a particular scientific format or subject matter. The same capabilities can support research datasets, industrial exports, public-sector data publishing and enterprise partner exchange. Domain-specific integrations, such as WMO WIS2, are optional; they are not prerequisites for using the platform.

Many platforms can transfer files securely. OpenECPDS is designed for teams that also need to scale distribution, orchestrate multiple data movers, integrate with business, scientific and operational workflows, and operate the whole system as infrastructure rather than as a single gateway.

If you are evaluating OpenECPDS against traditional MFT products, this page explains where it overlaps, where it differentiates, and why those differences matter.

The advantages go beyond moving bytes: operators can diagnose delivery problems, control storage growth, coordinate incident communications, and connect transfer activity to their observability tools from the same platform.

Introduction¶

OpenECPDS originated from operational needs at ECMWF, where large-scale, time-sensitive, international data exchange is part of daily production. It is now available as open source, giving organisations access to the same architectural ideas that support demanding real-world workflows in meteorology, science, and operational services.

The platform is centred on a distributed Data Store and three complementary services: Data Acquisition, Data Dissemination, and the Data Portal. Together, these cover provider-side retrieval, consumer-side delivery, and self-service push/pull access without forcing you to assemble separate products for each function.

Because OpenECPDS is open source, teams can inspect how it works, adapt it to their own operating model, and integrate it into existing tooling without being locked into a proprietary workflow. At the same time, it is documented, container-ready, and structured enough to be evaluated like an enterprise platform rather than like an internal prototype.

Common Ground with MFT Platforms¶

If you come from a commercial MFT evaluation, you will recognise many core capabilities. OpenECPDS covers the fundamentals you would expect from a serious transfer platform, including secure protocols, automation, scheduling, monitoring, and API access.

That overlap is important. OpenECPDS is not asking organisations to trade away mainstream MFT requirements in order to get specialised scientific-data capabilities. Instead, it starts from the same operational baseline and extends it into areas that matter when data volumes, distribution fan-out, and workflow integration become harder.

Where OpenECPDS Differentiates¶

The strongest reasons to choose OpenECPDS usually appear after the basic feature checklist. This is where architecture, scale model, workflow fit, and openness become more important than simple protocol support.

Multi-node Data Mover infrastructure¶

OpenECPDS is built around a clear separation between the Master Server, which coordinates transfers, and the Data Movers, which move the bytes. That design matters because it lets you scale transfer capacity, place movers near networks or storage zones, and route work according to transfer group, role, or geography instead of forcing everything through a single node.

In portal workflows, OpenECPDS even distinguishes between the User Data Mover that accepts a user connection and the Target Data Mover that stores the content. This gives the platform flexibility in how it handles ingress, storage placement, and load balancing. For organisations with multiple data centres, segmented networks, or regional distribution points, that is a practical architectural advantage rather than just an implementation detail. See Components and Continental Data Movers.

Sustained high-volume dissemination¶

Some transfer products are excellent at orchestrating business-file exchange but are not optimised for sustained, large-scale dissemination to many remote destinations. OpenECPDS is designed for exactly that kind of workload: repeated operational delivery, custom scheduling, per-host tuning, Data Mover allocation, and queue behaviour that can be controlled per destination.

The platform also treats data lifecycle, compression, checksums, and expiry as built-in operational concerns. That makes it a better fit for organisations that need to move large datasets repeatedly and predictably, not just complete isolated one-off transfers. See Additional Features and Data Transfer Lifecycle.

MQTT and WIS2 integration¶

This is one of the clearest areas where OpenECPDS differs from a typical general-purpose MFT platform. OpenECPDS includes an embedded MQTT broker for publication, an MQTT client for acquisition, and documented integration patterns for WMO WIS2. That means the same platform can publish notifications to downstream consumers, subscribe to upstream notifications, and turn those events into transfer activity without bolting on a separate messaging system.

For organisations in meteorology, environmental monitoring, research dissemination, or other event-driven data ecosystems, this is a major advantage. It aligns the transfer platform with modern publish/subscribe workflows instead of forcing polling-only integration patterns. See MQTT Overview and WMO WIS2 Integration.

Operational monitoring designed for distributed transfer systems¶

OpenECPDS includes a dedicated monitoring interface for destinations, hosts, transfers, users, and infrastructure activity. This is not only an audit log or dashboard layer: operators can inspect transfer state, manage queues, review event histories, and work with concepts that map directly to how a distributed transfer estate is run. A live 2D/3D Earth view shows transfer activity aggregated by Host, with status colours and filters for acquisition, dissemination, replication, source, backup and proxy traffic. Continental Mover markers, cloud-region locations and 24-hour traffic totals help operators understand where data is flowing, not just how many jobs are queued. Scheduler Traffic views complement that picture with current replication, backup and proxy work.

The documentation also reflects this operational focus, with dedicated sections for transfer statistics, structured event categories, global reach, and physical infrastructure examples. That matters when the evaluation question is not only “can it move files?” but also “can we operate this reliably every day?” See Monitoring, Live Earth, Event Logging, and Global Reach.

Logs, traces and metrics that fit your observability stack¶

OpenECPDS supports structured operational logs, optional distributed transfer traces, and Prometheus/JMX metrics. Teams can keep the Splunk-format file output, export native OpenTelemetry Logs, or use both. Sampled traces connect Master scheduling and completion to Mover execution, with links from dissemination logs to the corresponding attempt.

Only the Master needs access to the telemetry backend: Movers, including Continental Movers, forward trace data through their existing control connections. This keeps backend credentials and external connectivity out of remote transfer sites. An optional database journal retains successfully persisted operational events through Collector outages and Master restarts, with backlog and persistence-health indicators in the Monitor.

The advantage is a choice of diagnostic depth without committing to one monitoring vendor. Traces remain best effort, and the log journal is a delivery safeguard, not a transactional audit trail. See Event Logging and Hawtio, Prometheus and Grafana.

Network evidence alongside transfer outcomes¶

A slow or failed transfer does not, by itself, explain whether the problem is the application, an endpoint or the network path. Network Weather brings bounded passive delivery and retrieval observations together with optional TCP-connect and MTR probes for administrator-approved paths. Operators can inspect the evidence in administration tables and a Live Earth overlay, including for Continental Movers.

Transfer-level socket statistics provide additional evidence where supported, with acquisition and dissemination kept distinct. Together, these tools help teams focus investigations and share concrete observations with network operators rather than relying only on a failed-job count. Collection and probes are opt-in; missing measurements are shown as unavailable coverage, not as a healthy network. See Network Weather and Transfer Network Statistics.

Storage efficiency with explicit retention safeguards¶

Distributed delivery needs redundancy, but keeping every replica until file expiry can consume substantial capacity. Storage Lifecycle Policies let administrators evaluate and selectively reduce redundant copies by destination, group or file pattern, using dry-run estimates and savings counters before enabling enforcement.

Configured requeue windows, active-work protection, verified originals and per-file holds constrain removal. Originals remain until normal expiry; policies do not silently discard the last verified original to meet a storage target. Optional pressure handling and lazy replication provide further control, with explicit redundancy trade-offs.

This gives operators a way to balance disk use and recovery requirements within the transfer platform rather than relying on separate cleanup scripts. Enforcement is off by default. See Storage Lifecycle Policies.

Operational knowledge and incident communication in context¶

Knowing that delivery is delayed is only part of the operator's job. OpenECPDS also keeps destination contacts, ownership information and runbooks next to the data flows they describe. Structured metadata can supply contact lists and export selected procedures to Opsview notes, helping operators find the right response and escalation path without searching a separate document store.

Delay Notification groups affected products and cycles into a consistent notice, derives recipients from the relevant destinations, and tracks updates and resolutions. System Messages publish time-bounded maintenance or warning banners across the Monitor and Data Portal. These features help reduce inconsistent messages and repeated manual recipient selection during an incident; they do not replace operator judgement or confirm that recipients have read a notice. See Destination Metadata, Delay Notification and System Messages.

Safer configuration and central certificate management¶

Acquisition and dissemination settings can be tested against real server-side behaviour before they are saved. Directory previews, sample-content inspection and acquisition parsing previews help validate paths, listing metadata and size extraction without scheduling a full production flow. This is particularly useful for heterogeneous HTTP, cloud and scientific data sources whose listings do not all follow the same conventions.

Administrators can also inspect certificate health, generate certificate-signing requests, import CA-signed certificates, and deploy or reload certificates across connected Monitors and Movers from the UI. Expiry warnings make certificate maintenance visible alongside other operational tasks, while hot reload avoids a service restart for supported certificate changes.

Together, these capabilities reduce repetitive per-server administration and make configuration problems easier to catch before they interrupt delivery. See Acquisition testing, Dissemination testing and TLS Certificate Management.

Data Portal: incoming and outgoing in one platform¶

Many organisations end up combining separate products for scheduled outbound delivery and user-driven inbound/outbound access. OpenECPDS keeps those worlds together. Its Data Portal supports user-initiated download and upload workflows, while the same platform also runs background dissemination and acquisition services.

This matters operationally because governance, identity, logging, quota management, and storage policy can remain consistent. You do not have to maintain one product for scheduled delivery and another for partner-facing portal exchange if your use case spans both. See Data Portal and Data User Options.

Container-native deployment and fast evaluation path¶

OpenECPDS is designed to run in containers, with documented Docker-based workflows, a Kubernetes deployment path, and a standalone image that packages the full stack for quick evaluation. This makes it easier to test on a laptop, run in a development environment, and then move towards more realistic deployment topologies without switching technologies.

For evaluators, that shortens time-to-first-impression. For operators, it supports reproducible builds and infrastructure-as-code style deployment practices. The same platform can start small and still align with container-centric operating models. See System Requirements, Installation, and Deploying on Kubernetes.

Open REST API¶

OpenECPDS exposes a versioned JSON REST API for programmatic control over incoming users, destinations, metadata, monitoring data, and backup/restore workflows. That makes it easier to integrate with provisioning systems, internal portals, CMDBs, workflow engines, and automation pipelines.

Because the API is documented openly, it can also be evaluated alongside the UI and transfer workflows rather than hidden behind a separate commercial agreement or private SDK. For organisations that care about platform integration, that transparency lowers adoption friction. See REST API.

Extensible transfer module architecture¶

OpenECPDS uses a modular transfer architecture. That gives the platform a stable operational core while still allowing protocol-specific behaviour and future extension. It already includes modules for FTP, FTPS, SFTP, HTTP/HTTPS, Amazon S3, Google Cloud Storage, Azure Blob Storage, ECauth, and portal-oriented workflows.

This matters for two reasons. First, it lets operators tune each protocol appropriately rather than flattening all endpoints into a lowest-common-denominator abstraction. Second, it gives technical teams a cleaner path for adding or adapting integrations when their environment has specialised requirements. See Transfer Modules and Protocols & Connections.

Who is it for?¶

OpenECPDS is especially relevant for organisations whose transfer requirements are shaped by scale, operational continuity, multi-protocol distribution, or automated partner and user-facing data exchange. That includes environments where datasets are large, distribution is repeated, and the same platform must serve both automated systems and external users.

Typical candidates include:

• Industrial and enterprise teams that acquire files from providers, distribute exports to partners, or connect on-premises storage with cloud services
• Public-data publishers offering scheduled dataset delivery and self-service downloads, regardless of the subject matter
• National meteorological services that disseminate operational products to many partners and also ingest data from upstream providers
• Scientific data centres that need a blend of scheduled delivery, archive-facing storage, acquisition, and user portal access
• Operational data hubs that coordinate data exchange across regions, networks, and consuming systems
• Research infrastructures distributing large machine-readable datasets via web, MQTT, and cloud-compatible interfaces
• Government or intergovernmental services that need policy controls, auditability, and geographically distributed delivery
• Organisations consolidating multiple transfer tools into a single operational platform
• Individual researchers or small teams who want to store local datasets and share them with collaborators or the public — the Data Portal provides a self-hosted, multi-protocol file system interface (FTP, SFTP, SCP, HTTPS, WebDAV, S3) without the need for a third-party storage service
• Data publishers who need to distribute files on a schedule, notify subscribers via MQTT when new data arrives, and give end users a simple portal to download at their own pace

OpenECPDS scales down as well as up — the standalone container runs comfortably on a laptop, and a minimal single-node deployment works well as a centralised file gateway or personal data store. Whether the challenge is a small internal hub, a self-hosted file sharing service, or large-scale multi-node dissemination to thousands of destinations, the same platform and the same operational model apply.

Proven at Scale¶

OpenECPDS is not positioned as a theoretical architecture. The documentation describes an ECMWF operational deployment and supporting infrastructure that illustrate the scale the platform is intended to handle. That is one of the strongest signals for evaluators who want evidence of operational maturity.

Examples documented in this site include:

• more than 1,000 destinations across 80+ countries
• deployments that scale to hundreds of systems managing petabytes of data
• an illustrative ECMWF infrastructure of 80 bare-metal systems with 2 petabytes of storage
• a geographically distributed model including a Continental Data Mover in the United States to support transatlantic distribution patterns

These figures matter because they show that OpenECPDS is aimed at sustained, large operational estates rather than only lab-scale demonstrations. Even if your deployment is smaller, that design headroom can reduce the risk of outgrowing the platform later. See Global Reach and Illustrative Physical Infrastructure.

Why organisations choose it¶

When teams choose OpenECPDS, it is often because they want more than “secure transfer software.” They want a platform that can sit at the centre of data exchange, connect to varied protocols, handle scheduled and user-driven access, scale out through multiple movers, and participate in event-driven ecosystems.

Common reasons include:

• avoiding fragmentation across separate acquisition, dissemination, portal, and notification tools
• keeping operational control over tuning, scheduling, monitoring, and lifecycle policy
• investigating delivery problems with linked logs and traces, transfer statistics, and network-path evidence
• managing redundant storage with dry-run estimates, retention safeguards and holds
• keeping contacts, runbooks and delay communications connected to the affected data flows
• centralising certificate administration and telemetry backend access across a distributed fleet
• supporting scientific or mission-critical workflows that do not map neatly to generic business-file patterns
• preferring an open, inspectable, ECMWF-backed codebase over a closed appliance model
• wanting a realistic path from local evaluation to containerised production deployment

Getting Started¶

The fastest way to understand whether OpenECPDS fits your environment is to run it and inspect how the pieces work together. Start with the requirements, bring up the platform, and then explore the monitoring UI, portal, MQTT flows, and protocol modules that matter to your use case.

Recommended path:

1. Review System Requirements
2. Follow Installation
3. Launch the stack with First Run
4. Explore Architecture Overview
5. Compare Additional Features and the Transfer Modules with your current MFT requirements
6. Assess observability, Network Weather and Storage Lifecycle Policies against your operational needs; enable optional collection and enforcement deliberately

Related¶

• System Requirements
• Installation
• First Run
• Architecture Overview
• Additional Features
• Transfer Modules
• REST API
• Global Reach
• Centralised vs. Federated
• Data Ownership & Catalogue