Skip to content

Amazon S3 Transfer Module

Info

AWS SDK v2 — this module uses the AWS Java SDK v2 (software.amazon.awssdk). Credentials are taken from the host Login (Access Key ID) and Password (Secret Access Key) fields. IAM role assumption is also supported.

Connection

AWS endpoint

The module connects to standard AWS endpoints. The endpoint is resolved automatically — no manual configuration is needed in most cases.

s3.region = "eu-west-1"          # AWS region (optional — auto-discovered from bucket if omitted)
s3.url = "https://custom.example.com:9000"  # explicit endpoint override (optional, see below)
s3.scheme = "https"              # http | https  (default: https, ignored when url is set)
s3.port = "443"                  # port (default: 443, ignored when url is set)

Endpoint auto-detection

The module inspects the ECtrans destination host to determine the endpoint automatically:

  • AWS host (*.amazonaws.com): no endpoint override is applied — the AWS SDK constructs the correct regional endpoint from s3.region (e.g. bucket.s3.eu-west-1.amazonaws.com).
  • IP address (IPv4 or IPv6 — happens when ectrans.usednsname=no resolves the hostname before connecting) and s3.region is explicitly set: treated the same as an AWS host — SDK uses s3.region to construct the endpoint. For non-AWS services with a resolved IP, set s3.url explicitly to keep the IP as the endpoint.
  • Non-AWS hostname (MinIO, Ceph, NIRD, private S3 service): the destination host is automatically used as the endpoint override. No s3.url needed.
  • Explicit s3.url: always takes precedence over all cases above.

Region is optional for standard AWS regions

If s3.region is not set and s3.bucketName is configured, the module automatically discovers the bucket's region at connect time using GetBucketLocation. An INFO log entry records the discovered region. If discovery fails, us-east-1 is used as a fallback (with a WARN log).

Opt-in AWS regions require explicit s3.region

For AWS opt-in regions (eu-south-1, eu-south-2, ap-east-1, me-south-1, etc.), auto-discovery via GetBucketLocation may fail because the account must have explicitly enabled the region. Set s3.region explicitly to avoid falling back to us-east-1: 

s3.region = "eu-south-1"

Info

When s3.url is set, s3.scheme and s3.port are ignored. The URL is used as-is as the endpoint override.

SSL / TLS

s3.sslValidation = "yes"   # validate server certificate (default: no - disabled)
s3.strict = "yes"          # strict hostname verification (default: no)
s3.protocol = "TLS"        # SSL context protocol: TLS | TLSv1.2 | TLSv1.3 (default: TLS)

Advanced connectivity

s3.listenAddress = "192.168.1.10"  # local IP to bind outgoing connections to
s3.dualstack = "yes"               # use IPv4/IPv6 dual-stack endpoint (default: no)
s3.acceleration = "yes"            # S3 Transfer Acceleration (default: no)
                                   # Note: acceleration ignores s3.url

Cross-region access

s3.crossRegionAccess = "yes"   # enable cross-region redirect following (default: no)
                               # When enabled: the SDK follows 301 redirects to the
                               # bucket's actual region transparently.
                               # Region is still auto-discovered at connect time
                               # (same as when s3.region is omitted).

Warning

s3.crossRegionAccess is incompatible with s3.url (custom endpoint). Use it only with standard AWS endpoints.

Auto-discovery vs crossRegionAccess

Region auto-discovery (from GetBucketLocation) is independent of s3.crossRegionAccess. Auto-discovery runs whenever s3.region is blank and a s3.bucketName is set. s3.crossRegionAccess additionally tells the SDK to follow cross-region 301 redirects at the HTTP layer, which can help in edge cases where a request is initially routed incorrectly.

Auth & IAM

1. Static credentials (Access Key / Secret Key)

Set Login = Access Key ID and Password = Secret Access Key in the host Identity card. No additional properties are required.

2. IAM Role assumption (STS AssumeRole)

Useful when the access key belongs to an IAM user or role that is allowed to assume a more privileged role. The module calls STS AssumeRole before connecting to S3.

s3.roleArn = "arn:aws:iam::123456789012:role/MyS3Role"
s3.roleSessionName = "ecpds-session"   # session name tag (required with roleArn)
s3.durationSeconds = "3600"            # STS session lifetime in seconds (default: 3600)
s3.externalId = "my-external-id"       # ExternalId condition (if required by the role trust policy)

Info

The STS call always uses the global us-east-1 endpoint, so s3.region does not need to be set solely for role assumption. The base credentials (Login / Password) must have sts:AssumeRole permission on the target role ARN.

Bucket

Bucket name & key prefix

The bucket name can be set here or derived from the host Directory field (first path segment). The prefix is prepended to every object key.

s3.bucketName = "my-data-bucket"  # fixed bucket name (overrides the Directory field)
s3.prefix = "incoming/data/"      # key prefix for all objects (default: empty)
s3.allowEmptyBucketName = "yes"   # allow connecting without a bucket (default: no)

Listing & path style

s3.recursiveLevel = "0"            # listing depth: 0 = flat (default), -1 = unlimited, N = N levels
s3.enablePathStyleAccess = "yes"   # use path-style URLs: https://endpoint/bucket/key
                                   # (required for MinIO, Ceph and most S3-compatible services)

Info

Standard AWS uses virtual-hosted style (bucket.s3.amazonaws.com). Path style is mandatory for most non-AWS S3-compatible services.

Bucket creation

s3.mkBucket = "yes"   # create the bucket automatically if it does not exist (default: no)

Listing ownership metadata

These values appear in the FTP-style directory listing returned to the acquisition engine. They do not affect S3 object ownership.

s3.ftpuser = "myuser"    # owner name shown in directory listings (default: login name)
s3.ftpgroup = "mygroup"  # group name shown in directory listings (default: login name)

Transfer

Multipart uploads

Large objects are automatically split into parts and uploaded using a pipelined thread pool. While one part is being transmitted to S3, the next part is already being buffered from the source — significantly improving throughput for large files.

s3.partSize = "10"              # part size in MB for each multipart part (default: 10, minimum: 5)
s3.multipartSize = "5GB"        # threshold above which multipart is used (default: disabled/MAX)
                                # Example: "100MB" triggers multipart for files over 100 MB
s3.numUploadThreads = "2"       # number of threads sending parts to S3 simultaneously (default: 2)
                                # reading from source and uploading to S3 overlap in time, so
                                # increasing this allows more parts to be in-flight to S3 at once.
s3.queueCapacity = "4"          # max number of filled part buffers queued for upload (default: 4)
                                # together with numUploadThreads, bounds peak heap usage to
                                # (numUploadThreads + queueCapacity) × partSize MB.
s3.singlepartSize = "9223372036854775807"  # max size for single-part streaming (default: Long.MAX)
                                           # Lower this to force in-memory buffering for small files

Tuning upload throughput

s3.numUploadThreads controls how many parts can be transmitted to S3 simultaneously. A single thread always reads data from the source; as soon as a part buffer is full it is handed off to the upload pool, so reading the next part from source and sending the previous part to S3 happen at the same time instead of back-to-back.

For high-throughput workloads over fast links, increase both s3.numUploadThreads and s3.partSize:

s3.multipartSize = "100MB"   # use multipart for files over 100 MB
s3.partSize = "25"           # 25 MB per part
s3.numUploadThreads = "4"    # up to 4 parts sent to S3 simultaneously
s3.queueCapacity = "4"       # 4 additional parts may queue before back-pressure kicks in

Part buffers are pre-allocated once and reused for the lifetime of the upload — no per-part heap allocation. Peak memory per transfer = (numUploadThreads + queueCapacity) × partSize MB (e.g. 4 + 4 = 8 slots × 25 MB = 200 MB). When all slots are occupied the write thread blocks instead of allocating more heap.

Memory usage with concurrent transfers

Each concurrent S3 transfer allocates its own buffer pool. With N simultaneous transfers the total multipart buffer memory is N × (numUploadThreads + queueCapacity) × partSize MB. For example, 10 concurrent transfers with the high-throughput config above = 10 × 200 MB = 2 GB.

To reduce per-transfer memory without significantly impacting throughput, lower s3.partSize (minimum 5 MB) and s3.numUploadThreads (3–4 is typically sufficient for most links).

Memory buffering

s3.useByteArrayInputStream = "yes"  # buffer the object in memory before upload (default: no)
                                    # Useful when the stream size is unknown; enables retries.
                                    # Only applied when file size < s3.singlepartSize.

Warning

Enabling s3.useByteArrayInputStream for large files may exhaust heap memory. Use in combination with a low s3.singlepartSize.

Checksums

AWS SDK v2 calculates and validates checksums by default. Override if your S3-compatible service rejects checksum headers.

s3.requestChecksumCalculation = "WHEN_REQUIRED"   # WHEN_SUPPORTED | WHEN_REQUIRED
s3.responseChecksumValidation = "WHEN_REQUIRED"   # WHEN_SUPPORTED | WHEN_REQUIRED
# Leave unset to use the SDK default (WHEN_SUPPORTED)

Chunked encoding

s3.disableChunkedEncoding = "yes"  # disable HTTP chunked transfer encoding (default: no)
                                   # Required by some S3-compatible services that do not
                                   # support chunked encoding (e.g. older MinIO versions).

SDK retry behaviour

The AWS SDK normally retries on transient errors (5xx responses, throttling) by replaying the request body stream. Because ECpds feeds a non-resettable pipeline stream, a retry attempt by the SDK results in:

Content input stream does not support mark/reset, and was already read once

ECpds handles its own retry logic at a higher level, so SDK-level retries are redundant. Enable this option for any S3-compatible endpoint (non-Amazon) that triggers SDK retries:

s3.disableSdkRetries = "yes"   # disable AWS SDK retry logic (default: no)
                               # Recommended for non-Amazon S3-compatible endpoints
                               # (NIRD/Sigma2, MinIO, Ceph, etc.)

Note

This setting has no effect on ECpds-level retries — transfers will still be retried according to the configured retry policy.

Typical setups

Standard AWS S3 (minimal)

s3.region is auto-discovered from the bucket — no need to specify it:

s3.bucketName = "my-bucket"
s3.sslValidation = "yes"
# Login = Access Key ID
# Password = Secret Key
# Region is auto-discovered at connect time from GetBucketLocation

Standard AWS S3 (explicit region)

s3.region = "eu-west-1"
s3.bucketName = "my-bucket"
s3.sslValidation = "yes"

AWS opt-in region (eu-south-1, ap-east-1, etc.)

Auto-discovery may fail for opt-in regions — set s3.region explicitly:

s3.region = "eu-south-1"
s3.bucketName = "my-bucket"
s3.sslValidation = "yes"

MinIO / S3-compatible

No s3.url needed — the destination host is automatically used as the endpoint override:

s3.region = "us-east-1"
s3.enablePathStyleAccess = "yes"
s3.bucketName = "my-bucket"
s3.sslValidation = "yes"
s3.disableSdkRetries = "yes"   # prevent stream-replay errors on non-Amazon endpoints
# Destination host (e.g. minio.example.com) is auto-detected as a custom endpoint

Use s3.url only if you need to override to a different URL than the destination host:

s3.url = "https://minio.example.com:9000"
s3.region = "us-east-1"
s3.enablePathStyleAccess = "yes"
s3.bucketName = "my-bucket"
s3.disableSdkRetries = "yes"

NIRD / Sigma2 (and similar research storage)

NIRD is a non-AWS S3-compatible service. The destination host is automatically used as the endpoint, but always set s3.enablePathStyleAccess = "yes" — virtual-hosted-style (bucket.hostname) does not work on NIRD:

s3.bucketName = "my-project-bucket"
s3.region = "us-east-1"                # any valid string; unused for routing with a custom endpoint
s3.enablePathStyleAccess = "yes"       # required — NIRD does not support virtual-hosted-style
s3.disableSdkRetries = "yes"           # required — NIRD may return responses the SDK considers retryable
s3.disableChunkedEncoding = "yes"
s3.useByteArrayInputStream = "yes"
# Destination host: s3.nird.sigma2.no (set in the host editor)
# The host is auto-used as the endpoint override — no s3.url needed
# unless you want to override to a different URL

IAM Role assumption

No s3.region needed — the STS call uses the global endpoint and the bucket region is auto-discovered:

s3.bucketName = "my-bucket"
s3.roleArn = "arn:aws:iam::131544655113:role/ECPDS-ExternalRole"
s3.roleSessionName = "ECPDS-Session"
s3.externalId = "my-external-id"       # if required by the role trust policy
s3.sslValidation = "yes"

IAM Role assumption with large multipart uploads

s3.bucketName = "aiu-aws-weather-1"
s3.roleArn = "arn:aws:iam::131544655113:role/ECPDS-ExternalRole"
s3.roleSessionName = "ECPDS-Session"
s3.externalId = "my-external-id"
s3.multipartSize = "100MB"             # use multipart for files over 100 MB
s3.partSize = "25"                     # 25 MB per part
s3.numUploadThreads = "4"              # 4 threads in-flight → ~200 MB pipelined to S3
s3.queueCapacity = "4"                 # 4 additional parts may queue → peak = (4+4)×25 = 200 MB/transfer
s3.sslValidation = "yes"
# s3.region is omitted — auto-discovered from the bucket at connect time