Amazon S3 Transfer Module¶
Info
AWS SDK v2 — this module uses the AWS Java SDK v2 (software.amazon.awssdk). Credentials are taken from the host Login (Access Key ID) and Password (Secret Access Key) fields. IAM role assumption is also supported.
Connection¶
AWS endpoint¶
By default the module connects to standard AWS endpoints using the region. Use s3.url for S3-compatible services (MinIO, Ceph, etc.) or private endpoints.
s3.region = "eu-west-1" # AWS region (required for standard AWS)
s3.url = "https://minio.example.com:9000" # custom endpoint (S3-compatible)
s3.scheme = "https" # http | https (default: http, ignored when url is set)
s3.port = "443" # port (default: 80, ignored when url is set)
Info
When s3.url is set, s3.scheme and s3.port are ignored. The URL is used as-is as the endpoint override.
SSL / TLS¶
s3.sslValidation = "yes" # validate server certificate (default: no - disabled)
s3.strict = "yes" # strict hostname verification (default: no)
s3.protocol = "TLS" # SSL context protocol: TLS | TLSv1.2 | TLSv1.3 (default: TLS)
Advanced connectivity¶
s3.listenAddress = "192.168.1.10" # local IP to bind outgoing connections to
s3.dualstack = "yes" # use IPv4/IPv6 dual-stack endpoint (default: no)
s3.acceleration = "yes" # S3 Transfer Acceleration (default: no)
# Note: acceleration ignores s3.url
Cross-region access¶
s3.crossRegionAccess = "yes" # auto-discover the bucket's true region (default: no)
# When enabled: s3.url is ignored; region is resolved
# by probing s3.region (default us-east-1) first.
# Set s3.bucketName so discovery works at connect time.
Warning
s3.crossRegionAccess is incompatible with s3.url (custom endpoint). Use it only with standard AWS endpoints.
Auth & IAM¶
1. Static credentials (Access Key / Secret Key)¶
Set Login = Access Key ID and Password = Secret Access Key in the host Identity card. No additional properties are required.
2. IAM Role assumption (STS AssumeRole)¶
Useful when the access key belongs to an IAM user or role that is allowed to assume a more privileged role. The module calls STS AssumeRole before connecting to S3.
s3.roleArn = "arn:aws:iam::123456789012:role/MyS3Role"
s3.roleSessionName = "ecpds-session" # session name tag (default: none)
s3.durationSeconds = "3600" # STS session lifetime in seconds (default: 3600)
s3.externalId = "my-external-id" # ExternalId condition (if required by the role trust policy)
Info
The base credentials (Login / Password) must have sts:AssumeRole permission on the target role ARN.
Bucket¶
Bucket name & key prefix¶
The bucket name can be set here or derived from the host Directory field (first path segment). The prefix is prepended to every object key.
s3.bucketName = "my-data-bucket" # fixed bucket name (overrides the Directory field)
s3.prefix = "incoming/data/" # key prefix for all objects (default: empty)
s3.allowEmptyBucketName = "yes" # allow connecting without a bucket (default: no)
Listing & path style¶
s3.recursiveLevel = "0" # listing depth: 0 = flat (default), -1 = unlimited, N = N levels
s3.enablePathStyleAccess = "yes" # use path-style URLs: https://endpoint/bucket/key
# (required for MinIO, Ceph and most S3-compatible services)
Info
Standard AWS uses virtual-hosted style (bucket.s3.amazonaws.com). Path style is mandatory for most non-AWS S3-compatible services.
Bucket creation¶
Listing ownership metadata¶
These values appear in the FTP-style directory listing returned to the acquisition engine. They do not affect S3 object ownership.
s3.ftpuser = "myuser" # owner name shown in directory listings (default: login name)
s3.ftpgroup = "mygroup" # group name shown in directory listings (default: login name)
Transfer¶
Multipart uploads¶
Large objects are automatically split into parts and uploaded in parallel. Tune thresholds and part sizes to match your network and S3 service limits.
s3.partSize = "10" # part size in MB for multipart uploads (default: 10)
s3.multipartSize = "5GB" # threshold above which multipart is used (default: disabled/MAX)
# Example: "100MB" triggers multipart for files over 100 MB
s3.singlepartSize = "9223372036854775807" # max size for single-part streaming (default: Long.MAX)
# Lower this to force in-memory buffering for small files
Memory buffering¶
s3.useByteArrayInputStream = "yes" # buffer the object in memory before upload (default: no)
# Useful when the stream size is unknown; enables retries.
# Only applied when file size < s3.singlepartSize.
Warning
Enabling s3.useByteArrayInputStream for large files may exhaust heap memory. Use in combination with a low s3.singlepartSize.
Checksums¶
AWS SDK v2 calculates and validates checksums by default. Override if your S3-compatible service rejects checksum headers.
s3.requestChecksumCalculation = "WHEN_REQUIRED" # WHEN_SUPPORTED | WHEN_REQUIRED
s3.responseChecksumValidation = "WHEN_REQUIRED" # WHEN_SUPPORTED | WHEN_REQUIRED
# Leave unset to use the SDK default (WHEN_SUPPORTED)
Chunked encoding¶
s3.disableChunkedEncoding = "yes" # disable HTTP chunked transfer encoding (default: no)
# Required by some S3-compatible services that do not
# support chunked encoding (e.g. older MinIO versions).
Typical setups¶
Standard AWS S3¶
s3.region = "eu-west-1"
s3.bucketName = "my-bucket"
s3.sslValidation = "yes"
# Login = Access Key ID
# Password = Secret Key
MinIO / S3-compatible¶
s3.url = "https://minio.example.com"
s3.region = "us-east-1"
s3.enablePathStyleAccess = "yes"
s3.bucketName = "my-bucket"
s3.sslValidation = "yes"
Cross-region bucket¶
s3.bucketName = "bucket-in-eu-south-1"
s3.crossRegionAccess = "yes"
s3.sslValidation = "yes"
# Region auto-discovered from bucket
IAM Role assumption¶
s3.region = "us-east-1"
s3.roleArn = "arn:aws:iam::123:role/R"
s3.roleSessionName = "ecpds"
s3.bucketName = "my-bucket"