Class RemoteListingEngine

java.lang.Object
ecmwf.common.parser.core.RemoteListingEngine
public class RemoteListingEngine extends Object
Generic remote listing ingestion engine.

This engine provides a unified framework for parsing and normalizing heterogeneous remote catalog/listing formats into a common FtpParser.FileEntry model.

The goal of this framework is to allow OpenECPDS users to dynamically configure remote listing integrations through the UI without requiring custom parser development or software recompilation.

Supported formats

  • FTP directory listings (UNIX, Windows, etc.)
  • CSV feeds
  • Generic JSON feeds
  • STAC catalogs (SpatioTemporal Asset Catalog)
  • XML feeds
  • S3 inventory listings

Architecture

The engine delegates the parsing work to specialized RemoteListingParser implementations registered in the RemoteListingParserRegistry.

Each parser is responsible for:

  • understanding a specific remote format
  • extracting file metadata
  • mapping extracted values into FtpParser.FileEntry

The extraction logic is driven at runtime through the ParserConfig configuration object, allowing users to customize:

  • field mappings
  • CSV delimiters
  • JSON paths
  • XML node mappings
  • FTP parser settings
  • STAC asset selection

Example usage

ParserConfig config = new ParserConfig("csv",
        Map.of("delimiter", ",", "header", "false", "nameCol", "0", "timeCol", "2", "urlCol", "4"));

String content = remoteConnection.download(url);

FtpParser.FileEntry[] entries = new RemoteListingEngine().process(config, content);

Typical OpenECPDS workflow

  1. User configures remote parser type and mapping options in the UI
  2. Configuration is stored in the OpenECPDS database
  3. Remote content is downloaded dynamically at runtime
  4. The engine selects the appropriate parser
  5. The parser extracts and normalizes the entries
  6. The resulting FtpParser.FileEntry objects are used by OpenECPDS workflows

Extensibility

New remote formats can easily be supported by implementing the RemoteListingParser interface and registering the parser in the registry.

Thread safety

The engine itself is stateless and thread-safe assuming parser implementations are also thread-safe.

Since:
8.0.0
Author:
Laurent Gougeon - syi@ecmwf.int, ECMWF