Monitoring¶
The Monitoring section provides a real-time overview of the OpenECPDS system. It is the first page shown after login and the primary operational dashboard.
Dashboard (admin view)¶
The dashboard shows all destinations and their current transfer status. Each row represents a destination with colour-coded status indicators for queued, running, done, and failed transfers. The toolbar at the top provides quick access to filtering and refresh controls.
Live Earth¶
Live Earth is a real-time 3D globe visualisation of the transfers currently in progress across every Data Mover, available from the Monitoring card on the start page (/do/monitoring/globe). An in-page ⓘ button next to the page title expands a short built-in explanation of the whole page — the globe itself, its view/filter controls, and every figure shown in the stats panel described below — so this reference doc and the page stay in sync without needing to be read side-by-side. Transfers are aggregated per destination Host: each Host with at least one active transfer is drawn as a single pulsing arc/marker from the Master Server to that Host, coloured by status (blue while active, red if any of its transfers is failing, green fading out just after completion), with the arc's thickness reflecting the Host's combined throughput and the marker growing slightly with its number of concurrent transfers. Clicking a Host marker opens a panel summarising its aggregate activity (active transfer count, protocol(s) in use, total throughput, total bytes transferred, longest-running transfer) together with a per-transfer breakdown table (Data Mover, protocol, rate, bytes, status).
The globe is powered by a lightweight WebSocket feed (/ws/globe) pushed from the Master Server, which aggregates live samples reported periodically by every Data Mover while at least one globe page is open — no additional load is added when nobody is watching. Transfers relayed through a ProxyHost (a Data Mover with no direct RMI connection to the Master Server, reachable only via another Data Mover's REST interface) are included as well: their samples are relayed to the Master Server through that intermediary Data Mover's REST endpoint, exactly like their transfer status and progress updates already are. Each active ProxyHost is additionally drawn as its own marker (a distinct violet dot) at its resolved geolocation, with the arc for its transfers starting from that marker rather than from the Master Server's own location — making it visually clear that the data left from a different place. Ordinary, directly-connected Data Movers are not shown as separate markers, since they are considered co-located with the Master Server itself. The map imagery and 3D rendering are provided by a self-hosted copy of CesiumJS; no external network access or account/token is required.
A floating stats panel overlaid in the bottom-right corner of the globe itself (styled like the legend/country-breakdown panels, and kept clear of Cesium's own attribution logo, which sits bottom-left) shows, in its Network row, the number of active transfers and hosts, a combined-throughput speed-meter, and the total bytes transferred over the rolling last 24 hours. The 24h total is maintained on the Master Server itself (not in the browser), so every open globe page — and every reconnect — shows the exact same figure. It is kept in memory for accuracy, but its underlying data is also periodically saved to the database (SYS_CONFIG table, every 2 minutes by default, plus once more on a graceful shutdown) and reloaded at startup, so a normal Master Server restart only loses at most a couple of minutes of history instead of resetting the whole 24h figure to zero. This can be disabled with liveTransferBytesScheduler=no in the [Server] section of the Master Server's ecmwf.properties (the save interval can also be adjusted with a liveTransferBytesScheduler duration entry in the [Scheduler] section), in which case the figure reverts to being memory-only.
A second row of that same panel, Data Portal, covers the separate world of end-user Data Portal traffic (FTP, HTTP, SFTP, S3, WebDAV) that the transfer arcs above do not — since incoming/outgoing user sessions are not tied to a specific destination Host and so cannot be drawn as an arc. It shows the number of currently open Data Portal sessions across every Data Mover, plus a data-in and a data-out speed-meter computed from a live 5-second rolling average of bytes uploaded/downloaded by those sessions, all refreshed by the same WebSocket feed and requiring no extra configuration. A fourth tile, Storage, uses the same semicircular speed-meter style (rather than a literal capacity/percentage gauge) to show the aggregate used/total disk space across every volume of every Data Mover as a percentage — reusing the same cached figures already shown per-Mover on the Data Movers page, so it never triggers extra disk I/O of its own; hovering the tile reveals the exact used/total figures, and it only turns amber/red once the aggregate crosses 75%/90% respectively, as an early warning that some Data Movers may be running low on space.
By default, the globe is centred on the Master Server's own location, auto-detected via a GeoIP lookup of its hostname. If this location is inaccurate or cannot be resolved (for example, when the Master Server runs behind NAT/VPN or in a cloud/datacenter network range), it can be pinned explicitly with a forced.<ip-or-prefix> entry in the [GeoIP] section of the Master Server's ecmwf.properties — the same mechanism already used to force the location of any Host (e.g. for the traceroute/Nmap map on a Host's report page), so no separate/duplicate setting is needed. For example:
Because the globe's origin is resolved once by the Master Server itself (not by the Monitor plugin), it stays consistent even when several Monitor plugins are connected to the same Master Server.
The base map imagery is the low-resolution "Natural Earth II" imagery bundled with CesiumJS — sharp for a whole-Earth view, but it blurs once zoomed into a country/region, since no extra detail is bundled. Optionally, the Monitor plugin can transparently build and serve a sharper, higher-resolution layer (Natural Earth's 50m-resolution hypsometric/relief/water raster) instead: on the first "Live Earth" page opened after the Monitor starts, it downloads the (public-domain) source raster and tiles it locally — in pure Java, no GDAL or other native dependency required — into a small (a few MB) tile pyramid cached on disk. This happens entirely in the background and never blocks the page: it simply keeps using the bundled low-resolution imagery until the higher-resolution one finishes building, and silently falls back to it again if it can never be built (e.g. no internet access). If it fails (typically because the Monitor server has no internet access at the time), it keeps retrying every 15 minutes indefinitely, so connectivity becoming available only later is still picked up automatically without needing a restart. The cache location defaults to a subdirectory of the JVM's temp directory, and can be pointed at a persistent location (so it survives restarts and isn't rebuilt every time) with a globeImageryCacheDir entry in the [Server] section of the Monitor's ecmwf.properties:
Building the higher-resolution tile pyramid briefly needs a few hundred MB of heap (to hold the full source raster and the largest resized zoom level at once), on top of whatever the Monitor plugin otherwise uses. To avoid this optional enhancement ever destabilising a memory-constrained deployment (e.g. the standalone all-in-one image's default 512MB heap per service), it is automatically skipped — falling back to the bundled low-resolution imagery, with a one-off log message — if the JVM's max heap is below a configurable threshold (1024MB by default), set via globeImageryMinHeapMB in the same [Server] section:
Country and major city names can similarly be shown on the globe, using the same lazy/background/no-account approach: the Monitor plugin downloads Natural Earth's small (public-domain) country boundary and populated-place datasets on first use and converts them into a compact labels file, cached on disk (a few tens of KB). The default whole-globe view shows no names at all to stay uncluttered; country names join in first as the view zooms in a little, then city names join in even later as it zooms in further still (largest cities first, progressively smaller/less populated ones as it gets closer) - the same progressive-reveal technique used by most web map providers, shown as bold, uppercase, amber-bordered pills for countries versus plainer grey pills for cities so the two kinds of labels are easy to tell apart at a glance. Like the imagery layer, this never blocks the page — labels simply appear once (if ever) the background download finishes — and retries indefinitely every 15 minutes if it fails. They are hidden by default; a small tag icon next to the fullscreen button opens a dropdown with independent "Country names" / "Town names" switches, so each user can show either, both, or neither, remembered separately on their browser between visits. Its cache location can likewise be pointed at a persistent path with a globeLabelsCacheDir entry in the same [Server] section:
System Topology¶
System Topology is a live node/edge diagram of the OpenECPDS deployment itself, available from the Monitoring card on the start page (/do/monitoring/topology), right next to Live Earth. Where Live Earth visualises data in transit, System Topology visualises the infrastructure that moves it: the Master Server, the database it relies on, every registered Data Mover, and the Monitor instance you are currently using — grouped visually into dashed boxes by host, so it is easy to see at a glance which components share a machine (or container) and which run on separate ones, whatever your specific deployment layout happens to be. An in-page ⓘ button next to the title explains the diagram in full, including every symbol and colour used.
Components are drawn as coloured boxes (blue for the Master, cyan for this Monitor, purple for the database, green for an up Data Mover, grey/dashed for a down one), connected by labelled arrows showing the direction of each control connection (Monitor → Master over RMI, Master → Database over JDBC, Master → each Data Mover, labelled with its registration port). Clicking any component opens a details panel with its host, port(s) and status; for the Master and this Monitor, every network plugin currently loaded in that JVM (e.g. ecpds, ftp, http) is listed together with its live port and status, read directly from that JVM's own plugin container — no extra configuration needed. A 15s/30s/1m/5m/Off pill selector (defaulting to 30s, remembered per-browser) controls how often the diagram re-fetches itself in the background with no page reload; a manual refresh button is also available regardless of that setting. The diagram follows the page's light/dark theme like the rest of the Monitor UI.
All the data behind the diagram is fetched by the Monitor from the Master over the same RMI management interface already used elsewhere in the UI (e.g. for the Data Movers page) — the Monitor never contacts a Data Mover directly. Figures shown for a Data Mover (host, port, enabled flag, up/down state, plus its live per-plugin port list when connected) are all obtained by the Master itself: the up/down state reuses the same per-minute availability snapshots (MOVER_AVAILABILITY_SNAPSHOT table) already powering the uptime history on the Data Movers page, while the plugin/port list is read by the Master over its existing control connection to that Data Mover — the same connection it already uses for every other management operation, so this page adds no extra channel and no meaningful extra load. One thing is not yet shown, as a first iteration: a list of every other connected Monitor instance if more than one is deployed — this would need new dedicated RMI plumbing between the Master and its Monitors and is left for a future iteration.