Repository: cobbzilla/s3s3mirror
Branch: master
Commit: 17c94a28ae19
Files: 33
Total size: 99.6 KB
Directory structure:
gitextract_cx814f8m/
├── .gitignore
├── LICENSE.txt
├── README.md
├── pom.xml
├── s3s3mirror.bat
├── s3s3mirror.sh
└── src/
├── main/
│ ├── java/
│ │ └── org/
│ │ └── cobbzilla/
│ │ └── s3s3mirror/
│ │ ├── CopyMaster.java
│ │ ├── DeleteMaster.java
│ │ ├── KeyCopyJob.java
│ │ ├── KeyDeleteJob.java
│ │ ├── KeyFingerprint.java
│ │ ├── KeyJob.java
│ │ ├── KeyLister.java
│ │ ├── KeyMaster.java
│ │ ├── MirrorConstants.java
│ │ ├── MirrorContext.java
│ │ ├── MirrorMain.java
│ │ ├── MirrorMaster.java
│ │ ├── MirrorOptions.java
│ │ ├── MirrorStats.java
│ │ ├── MultipartKeyCopyJob.java
│ │ ├── Sleep.java
│ │ └── comparisonstrategies/
│ │ ├── ComparisonStrategy.java
│ │ ├── ComparisonStrategyFactory.java
│ │ ├── EtagComparisonStrategy.java
│ │ ├── SizeAndLastModifiedComparisonStrategy.java
│ │ └── SizeOnlyComparisonStrategy.java
│ └── resources/
│ └── log4j.xml
└── test/
└── java/
└── org/
└── cobbzilla/
└── s3s3mirror/
├── MirrorMainTest.java
├── MirrorTest.java
├── S3Asset.java
├── SyncStrategiesTest.java
└── TestFile.java
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
*.iml
.idea
tmp
logs
dependency-reduced-pom.xml
*~
target
!target/*.jar
*.log
.settings
.classpath
.project
================================================
FILE: LICENSE.txt
================================================
Copyright 2017-2021 Jonathan Cobb
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: README.md
================================================
s3s3mirror
==========
A utility for mirroring content from one S3 bucket to another.
Designed to be lightning-fast and highly concurrent, with modest CPU and memory requirements.
An object will be copied if and only if at least one of the following holds true:
* The object does not exist in the destination bucket.
* The "sync strategy" triggers (by default uses the Etag sync strategy)
* Etag Strategy (Default): If the size or Etags don't match between the source and destination bucket.
* Size Strategy: If the sizes don't match between the source and destination bucket.
* Size and Last Modified Strategy: If the source and destination objects have a different size, or the source bucket object has a more recent last modified date.
When copying, the source metadata and ACL lists are also copied to the destination object.
Note: [the 2.1-stable branch](https://github.com/cobbzilla/s3s3mirror/tree/2.1-stable) supports copying to/from local directories.
### Motivation
I started with "s3cmd sync" but found that with buckets containing many thousands of objects, it was incredibly slow
to start and consumed *massive* amounts of memory. So I designed s3s3mirror to start copying immediately with an intelligently
chosen "chunk size" and to operate in a highly-threaded, streaming fashion, so memory requirements are much lower.
Running with 100 threads, I found the gating factor to be *how fast I could list items from the source bucket* (!?!)
Which makes me wonder if there is any way to do this faster. I'm sure there must be, but this is pretty damn fast.
### AWS Credentials
* s3s3mirror will first look for credentials in your system environment. If variables named AWS\_ACCESS\_KEY\_ID and AWS\_SECRET\_ACCESS\_KEY are defined, then these will be used.
* Next, it checks for a ~/.s3cfg file (which you might have for using s3cmd). If present, the access key and secret key are read from there.
* IAM Roles can be used on EC2 instances by specifying the --iam flag
* If none of the above is found, it will error out and refuse to run.
### System Requirements
* Java 8 or higher
### Building
mvn package
Note that s3s3mirror now has a prebuilt jar checked in to github, so you'll only need to do this if you've been playing with the source code.
The above command requires that Maven 3 is installed.
### License
s3s3mirror is available under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
### Usage
s3s3mirror.sh [options] <source-bucket>[/src-prefix/path/...] <destination-bucket>[/dest-prefix/path/...]
### Versions
The 1.x branch (currently master) has been in use by the most number of people and is the most battle tested.
The 2.x branch supports copying between S3 and any local filesystem. It has seen heavy use and performs well, but is not as widely used as the 1.x branch.
**In the near future, the 1.x branch will offshoot from master, and the 2.x branch will be merged into master.** There are a handful of features
on the 1.x branch that have not yet been ported to 2.x. If you can live without them, I encourage you to use the 2.x branch. If you really need them,
I encourage you to port them to the 2.x branch, if you have the ability.
### Options
-c (--ctime) N : Only copy objects whose Last-Modified date is younger than this many days
For other time units, use these suffixes: y (years), M (months), d (days), w (weeks),
h (hours), m (minutes), s (seconds)
-i (--iam) : Attempt to use IAM Role if invoked on an EC2 instance
-P (--profile) VAL : Use a specific profile from your credential file (~/.aws/config)
-m (--max-connections) N : Maximum number of connections to S3 (default 100)
-n (--dry-run) : Do not actually do anything, but show what would be done (default false)
-r (--max-retries) N : Maximum number of retries for S3 requests (default 5)
-p (--prefix) VAL : Only copy objects whose keys start with this prefix
-d (--dest-prefix) VAL : Destination prefix (replacing the one specified in --prefix, if any)
-e (--endpoint) VAL : AWS endpoint to use (or set AWS_ENDPOINT in your environment)
-X (--delete-removed) : Delete objects from the destination bucket if they do not exist in the source bucket
-t (--max-threads) N : Maximum number of threads (default 100)
-v (--verbose) : Verbose output (default false)
-z (--proxy) VAL : host:port of proxy server to use.
Defaults to proxy_host and proxy_port defined in ~/.s3cfg,
or no proxy if these values are not found in ~/.s3cfg
-u (--upload-part-size) N : The upload size (in bytes) of each part uploaded as part of a multipart request
for files that are greater than the max allowed file size of 5368709120 bytes (5 GB)
Defaults to 4294967296 bytes (4 GB)
-C (--cross-account-copy) : Copy across AWS accounts. Only Resource-based policies are supported (as
specified by AWS documentation) for cross account copying
Default is false (copying within same account, preserving ACLs across copies)
If this option is active, the owner of the destination bucket will receive full control
-s (--ssl) : Use SSL for all S3 api operations (default false)
-E (--server-side-encryption) : Enable AWS managed server-side encryption (default false)
-l (--storage-class) : S3 storage class "Standard" or "ReducedRedundancy" (default Standard)
-S (--size-only) : Only takes size of objects in consideration when determining if a copy is required.
-L (--size-and-last-modified) : Uses size and last modified to determine if files have change like the AWS CLI and ignores etags. If -S (--size-only) is also specified that strategy is selected over this strategy.
### Examples
Copy everything from a bucket named "source" to another bucket named "dest"
s3s3mirror.sh source dest
Copy everything from "source" to "dest", but only copy objects created or modified within the past week
s3s3mirror.sh -c 7 source dest
s3s3mirror.sh -c 7d source dest
s3s3mirror.sh -c 1w source dest
s3s3mirror.sh --ctime 1w source dest
Copy everything from "source/foo" to "dest/bar"
s3s3mirror.sh source/foo dest/bar
s3s3mirror.sh -p foo -d bar source dest
Copy everything from "source/foo" to "dest/bar" and delete anything in "dest/bar" that does not exist in "source/foo"
s3s3mirror.sh -X source/foo dest/bar
s3s3mirror.sh --delete-removed source/foo dest/bar
s3s3mirror.sh -p foo -d bar -X source dest
s3s3mirror.sh -p foo -d bar --delete-removed source dest
Copy within a single bucket -- copy everything from "source/foo" to "source/bar"
s3s3mirror.sh source/foo source/bar
s3s3mirror.sh -p foo -d bar source source
BAD IDEA: If copying within a single bucket, do *not* put the destination below the source
s3s3mirror.sh source/foo source/foo/subfolder
s3s3mirror.sh -p foo -d foo/subfolder source source
*This might cause recursion and raise your AWS bill unnecessarily*
###### If you've enjoyed using s3s3mirror and are looking for a warm-fuzzy feeling, consider dropping a little somethin' into my [tip jar](https://cobbzilla.org/tipjar.html)
================================================
FILE: pom.xml
================================================
<?xml version="1.0" encoding="UTF-8"?>
<!--
(c) Copyright 2013-2021 Jonathan Cobb
This code is available under the Apache License, version 2: http://www.apache.org/licenses/LICENSE-2.0.html
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.cobbzilla</groupId>
<artifactId>s3s3mirror</artifactId>
<version>1.2.8-SNAPSHOT</version>
<packaging>jar</packaging>
<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
<properties>
<org.slf4j.version>1.7.30</org.slf4j.version>
<junit.version>4.13.1</junit.version>
</properties>
<dependencies>
<!-- Logging -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${org.slf4j.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>${org.slf4j.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${org.slf4j.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.17.1</version>
<scope>runtime</scope>
</dependency>
<!-- auto-generate java boilerplate -->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.16</version>
<scope>compile</scope>
</dependency>
<!-- Testing -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.8.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>3.7.7</version>
<scope>test</scope>
</dependency>
<!-- command line argument handling -->
<dependency>
<groupId>args4j</groupId>
<artifactId>args4j</artifactId>
<version>2.33</version>
</dependency>
<!-- for ctime argument -->
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.10.9</version>
</dependency>
<!-- Amazon SDK -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.12.261</version>
</dependency>
</dependencies>
<build>
<plugins>
<!-- Force Java 1.8 at all times -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<showWarnings>true</showWarnings>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.cobbzilla.s3s3mirror.MirrorMain</mainClass>
</transformer>
</transformers>
<!-- Exclude signed jars to avoid errors
see: http://stackoverflow.com/a/6743609/1251543
-->
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
================================================
FILE: s3s3mirror.bat
================================================
@echo off
java -Dlog4j.configuration=file:target/classes/log4j.xml -Ds3s3mirror.version=1.2.8 -jar target/s3s3mirror-1.2.7-SNAPSHOT.jar %*
================================================
FILE: s3s3mirror.sh
================================================
#!/bin/bash
THISDIR=$(cd "$(dirname $0)" && pwd)
VERSION=1.2.8
JARFILE="${THISDIR}/target/s3s3mirror-${VERSION}-SNAPSHOT.jar"
VERSION_ARG="-Ds3s3mirror.version=${VERSION}"
DEBUG=$1
if [ "${DEBUG}" = "--debug" ] ; then
# Run in debug mode
shift # remove --debug from options
java -Dlog4j.configuration=file:target/classes/log4j.xml -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 ${VERSION_ARG} -jar "${JARFILE}" "$@"
else
# Run in regular mode
java ${VERSION_ARG} -Dlog4j.configuration=file:target/classes/log4j.xml -jar "${JARFILE}" "$@"
fi
exit $?
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/CopyMaster.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import org.cobbzilla.s3s3mirror.comparisonstrategies.ComparisonStrategy;
import org.cobbzilla.s3s3mirror.comparisonstrategies.ComparisonStrategyFactory;
import org.cobbzilla.s3s3mirror.comparisonstrategies.SizeOnlyComparisonStrategy;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
public class CopyMaster extends KeyMaster {
private final ComparisonStrategy comparisonStrategy;
public CopyMaster(AmazonS3Client client, MirrorContext context, BlockingQueue<Runnable> workQueue, ThreadPoolExecutor executorService) {
super(client, context, workQueue, executorService);
comparisonStrategy = ComparisonStrategyFactory.getStrategy(context.getOptions());
}
protected String getPrefix(MirrorOptions options) { return options.getPrefix(); }
protected String getBucket(MirrorOptions options) { return options.getSourceBucket(); }
protected KeyCopyJob getTask(S3ObjectSummary summary) {
if (summary.getSize() > MirrorOptions.MAX_SINGLE_REQUEST_UPLOAD_FILE_SIZE) {
return new MultipartKeyCopyJob(client, context, summary, notifyLock, new SizeOnlyComparisonStrategy());
}
return new KeyCopyJob(client, context, summary, notifyLock, comparisonStrategy);
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/DeleteMaster.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
public class DeleteMaster extends KeyMaster {
public DeleteMaster(AmazonS3Client client, MirrorContext context, BlockingQueue<Runnable> workQueue, ThreadPoolExecutor executorService) {
super(client, context, workQueue, executorService);
}
protected String getPrefix(MirrorOptions options) {
return options.hasDestPrefix() ? options.getDestPrefix() : options.getPrefix();
}
protected String getBucket(MirrorOptions options) { return options.getDestinationBucket(); }
@Override
protected KeyJob getTask(S3ObjectSummary summary) {
return new KeyDeleteJob(client, context, summary, notifyLock);
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyCopyJob.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;
import lombok.extern.slf4j.Slf4j;
import org.cobbzilla.s3s3mirror.comparisonstrategies.ComparisonStrategy;
import org.slf4j.Logger;
import java.util.Date;
/**
* Handles a single key. Determines if it should be copied, and if so, performs the copy operation.
*/
@Slf4j
public class KeyCopyJob extends KeyJob {
protected String keydest;
protected ComparisonStrategy comparisonStrategy;
public KeyCopyJob(AmazonS3Client client, MirrorContext context, S3ObjectSummary summary, Object notifyLock, ComparisonStrategy comparisonStrategy) {
super(client, context, summary, notifyLock);
keydest = summary.getKey();
final MirrorOptions options = context.getOptions();
if (options.hasDestPrefix()) {
keydest = keydest.substring(options.getPrefixLength());
keydest = options.getDestPrefix() + keydest;
}
this.comparisonStrategy = comparisonStrategy;
}
@Override public Logger getLog() { return log; }
@Override
public void run() {
final MirrorOptions options = context.getOptions();
final String key = summary.getKey();
try {
if (!shouldTransfer()) return;
final ObjectMetadata sourceMetadata = getObjectMetadata(options.getSourceBucket(), key, options);
final AccessControlList objectAcl = getAccessControlList(options, key);
if (options.isDryRun()) {
log.info("Would have copied " + key + " to destination: " + keydest);
} else {
if (keyCopied(sourceMetadata, objectAcl)) {
context.getStats().objectsCopied.incrementAndGet();
} else {
context.getStats().copyErrors.incrementAndGet();
}
}
} catch (Exception e) {
log.error("error copying key: " + key + ": " + e);
} finally {
synchronized (notifyLock) {
notifyLock.notifyAll();
}
if (options.isVerbose()) log.info("done with " + key);
}
}
boolean keyCopied(ObjectMetadata sourceMetadata, AccessControlList objectAcl) {
String key = summary.getKey();
MirrorOptions options = context.getOptions();
boolean verbose = options.isVerbose();
int maxRetries= options.getMaxRetries();
MirrorStats stats = context.getStats();
for (int tries = 0; tries < maxRetries; tries++) {
if (verbose) log.info("copying (try #" + tries + "): " + key + " to: " + keydest);
final CopyObjectRequest request = new CopyObjectRequest(options.getSourceBucket(), key, options.getDestinationBucket(), keydest);
request.setStorageClass(StorageClass.valueOf(options.getStorageClass()));
if (options.isEncrypt()) {
request.putCustomRequestHeader("x-amz-server-side-encryption", "AES256");
}
request.setNewObjectMetadata(sourceMetadata);
if (options.isCrossAccountCopy()) {
request.setAccessControlList(buildCrossAccountAcl(objectAcl));
} else {
request.setAccessControlList(objectAcl);
}
try {
stats.s3copyCount.incrementAndGet();
client.copyObject(request);
stats.bytesCopied.addAndGet(sourceMetadata.getContentLength());
if (verbose) log.info("successfully copied (on try #" + tries + "): " + key + " to: " + keydest);
return true;
} catch (AmazonS3Exception s3e) {
log.error("s3 exception copying (try #" + tries + ") " + key + " to: " + keydest + ": " + s3e);
} catch (Exception e) {
log.error("unexpected exception copying (try #" + tries + ") " + key + " to: " + keydest + ": " + e);
}
try {
Thread.sleep(10);
} catch (InterruptedException e) {
log.error("interrupted while waiting to retry key: " + key);
return false;
}
}
return false;
}
private boolean shouldTransfer() {
final MirrorOptions options = context.getOptions();
final String key = summary.getKey();
final boolean verbose = options.isVerbose();
if (options.hasCtime()) {
final Date lastModified = summary.getLastModified();
if (lastModified == null) {
if (verbose) log.info("No Last-Modified header for key: " + key);
} else {
if (lastModified.getTime() < options.getMaxAge()) {
if (verbose) log.info("key "+key+" (lastmod="+lastModified+") is older than "+options.getCtime()+" (cutoff="+options.getMaxAgeDate()+"), not copying");
return false;
}
}
}
final ObjectMetadata metadata;
try {
metadata = getObjectMetadata(options.getDestinationBucket(), keydest, options);
} catch (AmazonS3Exception e) {
if (e.getStatusCode() == 404) {
if (verbose) log.info("Key not found in destination bucket (will copy): "+ keydest);
return true;
} else {
log.warn("Error getting metadata for " + options.getDestinationBucket() + "/" + keydest + " (not copying): " + e);
return false;
}
} catch (Exception e) {
log.warn("Error getting metadata for " + options.getDestinationBucket() + "/" + keydest + " (not copying): " + e);
return false;
}
if (summary.getSize() > MirrorOptions.MAX_SINGLE_REQUEST_UPLOAD_FILE_SIZE) {
return metadata.getContentLength() != summary.getSize();
}
final boolean objectChanged = comparisonStrategy.sourceDifferent(summary, metadata);
if (verbose && !objectChanged) log.info("Destination file is same as source, not copying: "+ key);
return objectChanged;
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyDeleteJob.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;
import lombok.extern.slf4j.Slf4j;
import org.slf4j.Logger;
@Slf4j
public class KeyDeleteJob extends KeyJob {
private String keysrc;
public KeyDeleteJob (AmazonS3Client client, MirrorContext context, S3ObjectSummary summary, Object notifyLock) {
super(client, context, summary, notifyLock);
final MirrorOptions options = context.getOptions();
keysrc = summary.getKey(); // NOTE: summary.getKey is the key in the destination bucket
if (options.hasPrefix()) {
keysrc = keysrc.substring(options.getDestPrefixLength());
keysrc = options.getPrefix() + keysrc;
}
}
@Override public Logger getLog() { return log; }
@Override
public void run() {
final MirrorOptions options = context.getOptions();
final MirrorStats stats = context.getStats();
final boolean verbose = options.isVerbose();
final int maxRetries = options.getMaxRetries();
final String key = summary.getKey();
try {
if (!shouldDelete()) return;
final DeleteObjectRequest request = new DeleteObjectRequest(options.getDestinationBucket(), key);
if (options.isDryRun()) {
log.info("Would have deleted "+key+" from destination because "+keysrc+" does not exist in source");
} else {
boolean deletedOK = false;
for (int tries=0; tries<maxRetries; tries++) {
if (verbose) log.info("deleting (try #"+tries+"): "+key);
try {
stats.s3deleteCount.incrementAndGet();
client.deleteObject(request);
deletedOK = true;
if (verbose) log.info("successfully deleted (on try #"+tries+"): "+key);
break;
} catch (AmazonS3Exception s3e) {
log.error("s3 exception deleting (try #"+tries+") "+key+": "+s3e);
} catch (Exception e) {
log.error("unexpected exception deleting (try #"+tries+") "+key+": "+e);
}
try {
Thread.sleep(10);
} catch (InterruptedException e) {
log.error("interrupted while waiting to retry key: "+key);
break;
}
}
if (deletedOK) {
context.getStats().objectsDeleted.incrementAndGet();
} else {
context.getStats().deleteErrors.incrementAndGet();
}
}
} catch (Exception e) {
log.error("error deleting key: "+key+": "+e);
} finally {
synchronized (notifyLock) {
notifyLock.notifyAll();
}
if (verbose) log.info("done with "+key);
}
}
private boolean shouldDelete() {
final MirrorOptions options = context.getOptions();
final boolean verbose = options.isVerbose();
// Does it exist in the source bucket
try {
ObjectMetadata metadata = getObjectMetadata(options.getSourceBucket(), keysrc, options);
return false; // object exists in source bucket, don't delete it from destination bucket
} catch (AmazonS3Exception e) {
if (e.getStatusCode() == 404) {
if (verbose) log.info("Key not found in source bucket (will delete from destination): "+ keysrc);
return true;
} else {
log.warn("Error getting metadata for " + options.getSourceBucket() + "/" + keysrc + " (not deleting): " + e);
return false;
}
} catch (Exception e) {
log.warn("Error getting metadata for " + options.getSourceBucket() + "/" + keysrc + " (not deleting): " + e);
return false;
}
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyFingerprint.java
================================================
package org.cobbzilla.s3s3mirror;
import lombok.*;
@EqualsAndHashCode(callSuper=false) @AllArgsConstructor
public class KeyFingerprint {
@Getter private final long size;
@Getter private final String etag;
public KeyFingerprint(long size) {
this(size, null);
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyJob.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;
import org.slf4j.Logger;
public abstract class KeyJob implements Runnable {
protected final AmazonS3Client client;
protected final MirrorContext context;
protected final S3ObjectSummary summary;
protected final Object notifyLock;
public KeyJob(AmazonS3Client client, MirrorContext context, S3ObjectSummary summary, Object notifyLock) {
this.client = client;
this.context = context;
this.summary = summary;
this.notifyLock = notifyLock;
}
public abstract Logger getLog();
@Override public String toString() { return summary.getKey(); }
protected ObjectMetadata getObjectMetadata(String bucket, String key, MirrorOptions options) throws Exception {
Exception ex = null;
for (int tries=0; tries<options.getMaxRetries(); tries++) {
try {
context.getStats().s3getCount.incrementAndGet();
return client.getObjectMetadata(bucket, key);
} catch (AmazonS3Exception e) {
if (e.getStatusCode() == 404) throw e;
} catch (Exception e) {
ex = e;
if (options.isVerbose()) {
if (tries >= options.getMaxRetries()) {
getLog().error("getObjectMetadata(" + key + ") failed (try #" + tries + "), giving up");
break;
} else {
getLog().warn("getObjectMetadata("+key+") failed (try #"+tries+"), retrying...");
}
}
}
}
throw ex;
}
protected AccessControlList getAccessControlList(MirrorOptions options, String key) throws Exception {
Exception ex = null;
for (int tries=0; tries<=options.getMaxRetries(); tries++) {
try {
context.getStats().s3getCount.incrementAndGet();
return client.getObjectAcl(options.getSourceBucket(), key);
} catch (Exception e) {
ex = e;
if (tries >= options.getMaxRetries()) {
// Annoyingly there can be two reasons for this to fail. It will fail if the IAM account
// permissions are wrong, but it will also fail if we are copying an item that we don't
// own ourselves. This may seem unusual, but it occurs when copying AWS Detailed Billing
// objects since although they live in your bucket, the object owner is AWS.
getLog().warn("Unable to obtain object ACL, copying item without ACL data.");
return new AccessControlList();
}
if (options.isVerbose()) {
if (tries >= options.getMaxRetries()) {
getLog().warn("getObjectAcl(" + key + ") failed (try #" + tries + "), giving up.");
break;
} else {
getLog().warn("getObjectAcl("+key+") failed (try #"+tries+"), retrying...");
}
}
}
}
throw ex;
}
AccessControlList buildCrossAccountAcl(AccessControlList original) {
AccessControlList result = new AccessControlList();
for (Grant grant : original.getGrantsAsList()) {
// Covers all 3 types: Everyone, Authenticate User, Log Delivery
if (grant.getGrantee() instanceof GroupGrantee) {
result.grantPermission(grant.getGrantee(), grant.getPermission());
}
}
// Equal to the canned way: request.setCannedAccessControlList(CannedAccessControlList.BucketOwnerFullControl);
result.grantPermission(new CanonicalGrantee(context.getOwner().getId()), Permission.FullControl);
result.setOwner(context.getOwner());
return result;
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyLister.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.AmazonS3Exception;
import com.amazonaws.services.s3.model.ListObjectsRequest;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import lombok.extern.slf4j.Slf4j;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
@Slf4j
public class KeyLister implements Runnable {
private AmazonS3Client client;
private MirrorContext context;
private int maxQueueCapacity;
private final List<S3ObjectSummary> summaries;
private final AtomicBoolean done = new AtomicBoolean(false);
private ObjectListing listing;
public boolean isDone () { return done.get(); }
public KeyLister(AmazonS3Client client, MirrorContext context, int maxQueueCapacity, String bucket, String prefix) {
this.client = client;
this.context = context;
this.maxQueueCapacity = maxQueueCapacity;
final MirrorOptions options = context.getOptions();
int fetchSize = options.getMaxThreads();
this.summaries = new ArrayList<S3ObjectSummary>(10*fetchSize);
final ListObjectsRequest request = new ListObjectsRequest(bucket, prefix, null, null, fetchSize);
listing = s3getFirstBatch(client, request);
synchronized (summaries) {
final List<S3ObjectSummary> objectSummaries = listing.getObjectSummaries();
summaries.addAll(objectSummaries);
context.getStats().objectsRead.addAndGet(objectSummaries.size());
if (options.isVerbose()) log.info("added initial set of "+objectSummaries.size()+" keys");
}
}
@Override
public void run() {
final MirrorOptions options = context.getOptions();
final boolean verbose = options.isVerbose();
int counter = 0;
log.info("starting...");
try {
while (true) {
while (getSize() < maxQueueCapacity) {
if (listing.isTruncated()) {
listing = s3getNextBatch();
if (++counter % 100 == 0) context.getStats().logStats();
synchronized (summaries) {
final List<S3ObjectSummary> objectSummaries = listing.getObjectSummaries();
summaries.addAll(objectSummaries);
context.getStats().objectsRead.addAndGet(objectSummaries.size());
if (verbose) log.info("queued next set of "+objectSummaries.size()+" keys (total now="+getSize()+")");
}
} else {
log.info("No more keys found in source bucket, exiting");
return;
}
}
try {
Thread.sleep(50);
} catch (InterruptedException e) {
log.error("interrupted!");
return;
}
}
} catch (Exception e) {
log.error("Error in run loop, KeyLister thread now exiting: "+e);
} finally {
if (verbose) log.info("KeyLister run loop finished");
done.set(true);
}
}
private ObjectListing s3getFirstBatch(AmazonS3Client client, ListObjectsRequest request) {
final MirrorOptions options = context.getOptions();
final boolean verbose = options.isVerbose();
final int maxRetries = options.getMaxRetries();
Exception lastException = null;
for (int tries=0; tries<maxRetries; tries++) {
try {
context.getStats().s3getCount.incrementAndGet();
ObjectListing listing = client.listObjects(request);
if (verbose) log.info("successfully got first batch of objects (on try #"+tries+")");
return listing;
} catch (Exception e) {
lastException = e;
log.warn("s3getFirstBatch: error listing (try #"+tries+"): "+e);
if (Sleep.sleep(50)) {
log.info("s3getFirstBatch: interrupted while waiting for next try");
break;
}
}
}
throw new IllegalStateException("s3getFirstBatch: error listing: "+lastException, lastException);
}
private ObjectListing s3getNextBatch() {
final MirrorOptions options = context.getOptions();
final boolean verbose = options.isVerbose();
final int maxRetries = options.getMaxRetries();
for (int tries=0; tries<maxRetries; tries++) {
try {
context.getStats().s3getCount.incrementAndGet();
ObjectListing next = client.listNextBatchOfObjects(listing);
if (verbose) log.info("successfully got next batch of objects (on try #"+tries+")");
return next;
} catch (AmazonS3Exception s3e) {
log.error("s3 exception listing objects (try #"+tries+"): "+s3e);
} catch (Exception e) {
log.error("unexpected exception listing objects (try #"+tries+"): "+e);
}
if (Sleep.sleep(50)) {
log.info("s3getNextBatch: interrupted while waiting for next try");
break;
}
}
throw new IllegalStateException("Too many errors trying to list objects (maxRetries="+maxRetries+")");
}
private int getSize() {
synchronized (summaries) {
return summaries.size();
}
}
public List<S3ObjectSummary> getNextBatch() {
List<S3ObjectSummary> copy;
synchronized (summaries) {
copy = new ArrayList<S3ObjectSummary>(summaries);
summaries.clear();
}
return copy;
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyMaster.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import lombok.extern.slf4j.Slf4j;
import java.util.List;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
@Slf4j
public abstract class KeyMaster implements Runnable {
public static final int STOP_TIMEOUT_SECONDS = 10;
private static final long STOP_TIMEOUT = TimeUnit.SECONDS.toMillis(STOP_TIMEOUT_SECONDS);
protected AmazonS3Client client;
protected MirrorContext context;
private AtomicBoolean done = new AtomicBoolean(false);
public boolean isDone () { return done.get(); }
private BlockingQueue<Runnable> workQueue;
private ThreadPoolExecutor executorService;
protected final Object notifyLock = new Object();
private Thread thread;
public KeyMaster(AmazonS3Client client, MirrorContext context, BlockingQueue<Runnable> workQueue, ThreadPoolExecutor executorService) {
this.client = client;
this.context = context;
this.workQueue = workQueue;
this.executorService = executorService;
}
protected abstract String getPrefix(MirrorOptions options);
protected abstract String getBucket(MirrorOptions options);
protected abstract KeyJob getTask(S3ObjectSummary summary);
public void start () {
this.thread = new Thread(this);
this.thread.start();
}
public void stop () {
final String name = getClass().getSimpleName();
final long start = System.currentTimeMillis();
log.info("stopping "+ name +"...");
try {
if (isDone()) return;
this.thread.interrupt();
while (!isDone() && System.currentTimeMillis() - start < STOP_TIMEOUT) {
if (Sleep.sleep(50)) return;
}
} finally {
if (!isDone()) {
try {
log.warn(name+" didn't stop within "+STOP_TIMEOUT_SECONDS+" after interrupting it, forcibly killing the thread...");
this.thread.stop();
} catch (Exception e) {
log.error("Error calling Thread.stop on " + name + ": " + e, e);
}
}
if (isDone()) log.info(name+" stopped");
}
}
public void run() {
final MirrorOptions options = context.getOptions();
final boolean verbose = options.isVerbose();
final int maxQueueCapacity = MirrorMaster.getMaxQueueCapacity(options);
int counter = 0;
try {
final KeyLister lister = new KeyLister(client, context, maxQueueCapacity, getBucket(options), getPrefix(options));
executorService.submit(lister);
List<S3ObjectSummary> summaries = lister.getNextBatch();
if (verbose) log.info(summaries.size()+" keys found in first batch from source bucket -- processing...");
while (true) {
for (S3ObjectSummary summary : summaries) {
while (workQueue.size() >= maxQueueCapacity) {
try {
synchronized (notifyLock) {
notifyLock.wait(50);
}
Thread.sleep(50);
} catch (InterruptedException e) {
log.error("interrupted!");
return;
}
}
executorService.submit(getTask(summary));
counter++;
}
summaries = lister.getNextBatch();
if (summaries.size() > 0) {
if (verbose) log.info(summaries.size()+" more keys found in source bucket -- continuing (queue size="+workQueue.size()+", total processed="+counter+")...");
} else if (lister.isDone()) {
if (verbose) log.info("No more keys found in source bucket -- ALL DONE");
return;
} else {
if (verbose) log.info("Lister has no keys queued, but is not done, waiting and retrying");
if (Sleep.sleep(50)) return;
}
}
} catch (Exception e) {
log.error("Unexpected exception in MirrorMaster: "+e, e);
} finally {
while (workQueue.size() > 0 || executorService.getActiveCount() > 0) {
// wait for the queue to be empty
if (Sleep.sleep(100)) break;
}
// this will wait for currently executing tasks to finish
executorService.shutdown();
done.set(true);
}
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorConstants.java
================================================
package org.cobbzilla.s3s3mirror;
public class MirrorConstants {
public static final long KB = 1024L;
public static final long MB = KB * 1024L;
public static final long GB = MB * 1024L;
public static final long TB = GB * 1024L;
public static final long PB = TB * 1024L;
public static final long EB = PB * 1024L;
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorContext.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.model.Owner;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.Setter;
@AllArgsConstructor
public class MirrorContext {
@Getter @Setter private MirrorOptions options;
@Getter @Setter private Owner owner;
@Getter private final MirrorStats stats = new MirrorStats();
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorMain.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.auth.InstanceProfileCredentialsProvider;
import com.amazonaws.auth.BasicSessionCredentials;
import com.amazonaws.services.s3.model.AccessControlList;
import com.amazonaws.services.s3.model.Owner;
import lombok.Cleanup;
import lombok.Getter;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.kohsuke.args4j.CmdLineParser;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
/**
* Provides the "main" method. Responsible for parsing options and setting up the MirrorMaster to manage the copy.
*/
@Slf4j
public class MirrorMain {
@Getter @Setter private String[] args;
@Getter private final MirrorOptions options = new MirrorOptions();
private final CmdLineParser parser = new CmdLineParser(options);
private final Thread.UncaughtExceptionHandler uncaughtExceptionHandler = new Thread.UncaughtExceptionHandler() {
@Override public void uncaughtException(Thread t, Throwable e) {
log.error("Uncaught Exception (thread "+t.getName()+"): "+e, e);
}
};
@Getter private AmazonS3Client client;
@Getter private MirrorContext context;
@Getter private MirrorMaster master;
public MirrorMain(String[] args) { this.args = args; }
public static void main (String[] args) {
MirrorMain main = new MirrorMain(args);
main.run();
}
public void run() {
init();
master.mirror();
}
public void init() {
if (client == null) {
try {
parseArguments();
} catch (Exception e) {
System.err.println(e.getMessage());
parser.printUsage(System.err);
System.exit(1);
}
client = getAmazonS3Client();
context = new MirrorContext(options, getTargetBucketOwner(client));
master = new MirrorMaster(client, context);
Runtime.getRuntime().addShutdownHook(context.getStats().getShutdownHook());
Thread.setDefaultUncaughtExceptionHandler(uncaughtExceptionHandler);
}
}
protected AmazonS3Client getAmazonS3Client() {
ClientConfiguration clientConfiguration = new ClientConfiguration().withProtocol((options.isSsl() ? Protocol.HTTPS : Protocol.HTTP))
.withMaxConnections(options.getMaxConnections());
if (options.getHasProxy()) {
clientConfiguration = clientConfiguration
.withProxyHost(options.getProxyHost())
.withProxyPort(options.getProxyPort());
}
AmazonS3Client client = null;
if(System.getenv("AWS_SECURITY_TOKEN") != null) {
BasicSessionCredentials basicSessionCredentials = new BasicSessionCredentials(System.getenv("AWS_ACCESS_KEY_ID"), System.getenv("AWS_SECRET_ACCESS_KEY"), System.getenv("AWS_SECURITY_TOKEN"));
client = new AmazonS3Client(basicSessionCredentials, clientConfiguration);
} else if (options.hasAwsKeys()) {
client = new AmazonS3Client(options, clientConfiguration);
} else if (options.isUseIamRole()) {
client = new AmazonS3Client(new InstanceProfileCredentialsProvider(), clientConfiguration);
} else {
throw new IllegalStateException("No authenication method available, please specify IAM Role usage or AWS key and secret");
}
if (options.hasEndpoint()) client.setEndpoint(options.getEndpoint());
return client;
}
protected void parseArguments() throws Exception {
parser.parseArgument(args);
// for credentials, check for IAM role usage if not then...
// try the .aws/config file first if there is a profile specified, otherwise defer to
// .s3cfg before using the default .aws/config credentials
// (this may attempt .aws/config twice for no reason, but maintains backward compatibility)
if (options.isUseIamRole() == false) {
if (!options.hasAwsKeys() && options.getProfile() != null) loadAwsKeysFromAwsConfig();
if (!options.hasAwsKeys()) loadAwsKeysFromS3Config();
if (!options.hasAwsKeys()) loadAwsKeysFromAwsConfig();
if (!options.hasAwsKeys()) loadAwsKeysFromAwsCredentials();
if (!options.hasAwsKeys()) {
throw new IllegalStateException("Could not find credentials, IAM Role usage not specified and ENV vars not defined: " + MirrorOptions.AWS_ACCESS_KEY + " and/or " + MirrorOptions.AWS_SECRET_KEY);
}
} else {
InstanceProfileCredentialsProvider client = new InstanceProfileCredentialsProvider();
if (client.getCredentials() == null) {
throw new IllegalStateException("Could not find IAM Instance Profile credentials from the AWS metadata service.");
}
}
options.initDerivedFields();
}
private void loadAwsKeysFromS3Config() {
try {
// try to load from ~/.s3cfg
@Cleanup BufferedReader reader = new BufferedReader(new FileReader(System.getProperty("user.home")+File.separator+".s3cfg"));
String line;
while ((line = reader.readLine()) != null) {
if (line.trim().startsWith("access_key")) {
options.setAWSAccessKeyId(line.substring(line.indexOf("=") + 1).trim());
} else if (line.trim().startsWith("secret_key")) {
options.setAWSSecretKey(line.substring(line.indexOf("=") + 1).trim());
} else if (!options.getHasProxy() && line.trim().startsWith("proxy_host")) {
options.setProxyHost(line.substring(line.indexOf("=") + 1).trim());
} else if (!options.getHasProxy() && line.trim().startsWith("proxy_port")){
options.setProxyPort(Integer.parseInt(line.substring(line.indexOf("=") + 1).trim()));
}
}
} catch (Exception e) {
// ignore - let other credential-discovery processes have a crack
}
}
private void loadAwsKeysFromAwsConfig() {
try {
// try to load from ~/.aws/config
@Cleanup BufferedReader reader = new BufferedReader(new FileReader(
System.getProperty("user.home") + File.separator + ".aws" + File.separator + "config"));
String line;
boolean skipSection = true;
while ((line = reader.readLine()) != null) {
line = line.trim();
if (line.startsWith("[")) {
// if no defined profile, use '[default]' otherwise use profile with matching name
if ((options.getProfile() == null && line.equals("[default]"))
|| (options.getProfile() != null && line.equals("[profile " + options.getProfile() + "]"))) {
skipSection = false;
} else {
skipSection = true;
}
continue;
}
if (skipSection) continue;
if (line.startsWith("aws_access_key_id")) {
options.setAWSAccessKeyId(line.substring(line.indexOf("=") + 1).trim());
} else if (line.startsWith("aws_secret_access_key")) {
options.setAWSSecretKey(line.substring(line.indexOf("=") + 1).trim());
}
}
} catch (Exception e) {
// ignore - let other credential-discovery processes have a crack
}
}
private void loadAwsKeysFromAwsCredentials() {
try {
// try to load from ~/.aws/config
@Cleanup BufferedReader reader = new BufferedReader(new FileReader(
System.getProperty("user.home") + File.separator + ".aws" + File.separator + "credentials"));
String line;
boolean skipSection = true;
while ((line = reader.readLine()) != null) {
line = line.trim();
if (line.startsWith("[")) {
// if no defined profile, use '[default]' otherwise use profile with matching name
if ((options.getProfile() == null && line.equals("[default]"))
|| (options.getProfile() != null && line.equals("[" + options.getProfile() + "]"))) {
skipSection = false;
} else {
skipSection = true;
}
continue;
}
if (skipSection) continue;
if (line.startsWith("aws_access_key_id")) {
options.setAWSAccessKeyId(line.substring(line.indexOf("=") + 1).trim());
} else if (line.startsWith("aws_secret_access_key")) {
options.setAWSSecretKey(line.substring(line.indexOf("=") + 1).trim());
}
}
} catch (Exception e) {
// ignore - let other credential-discovery processes have a crack
}
}
private Owner getTargetBucketOwner(AmazonS3Client client) {
AccessControlList targetBucketAcl = client.getBucketAcl(options.getDestinationBucket());
return targetBucketAcl.getOwner();
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorMaster.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import lombok.extern.slf4j.Slf4j;
import java.util.concurrent.*;
/**
* Manages the Starts a KeyLister and sends batches of keys to the ExecutorService for handling by KeyJobs
*/
@Slf4j
public class MirrorMaster {
public static final String VERSION = System.getProperty("s3s3mirror.version");
private final AmazonS3Client client;
private final MirrorContext context;
public MirrorMaster(AmazonS3Client client, MirrorContext context) {
this.client = client;
this.context = context;
}
public void mirror() {
log.info("version "+VERSION+" starting");
final MirrorOptions options = context.getOptions();
if (options.isVerbose() && options.hasCtime()) log.info("will not copy anything older than "+options.getCtime()+" (cutoff="+options.getMaxAgeDate()+")");
final int maxQueueCapacity = getMaxQueueCapacity(options);
final BlockingQueue<Runnable> workQueue = new LinkedBlockingQueue<Runnable>(maxQueueCapacity);
final RejectedExecutionHandler rejectedExecutionHandler = new RejectedExecutionHandler() {
@Override
public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
log.error("Error submitting job: "+r+", possible queue overflow");
}
};
final ThreadPoolExecutor executorService = new ThreadPoolExecutor(options.getMaxThreads(), options.getMaxThreads(), 1, TimeUnit.MINUTES, workQueue, rejectedExecutionHandler);
final KeyMaster copyMaster = new CopyMaster(client, context, workQueue, executorService);
KeyMaster deleteMaster = null;
try {
copyMaster.start();
if (context.getOptions().isDeleteRemoved()) {
deleteMaster = new DeleteMaster(client, context, workQueue, executorService);
deleteMaster.start();
}
while (true) {
if (copyMaster.isDone() && (deleteMaster == null || deleteMaster.isDone())) {
log.info("mirror: completed");
break;
}
if (Sleep.sleep(100)) return;
}
} catch (Exception e) {
log.error("Unexpected exception in mirror: "+e, e);
} finally {
try { copyMaster.stop(); } catch (Exception e) { log.error("Error stopping copyMaster: "+e, e); }
if (deleteMaster != null) {
try { deleteMaster.stop(); } catch (Exception e) { log.error("Error stopping deleteMaster: "+e, e); }
}
}
}
public static int getMaxQueueCapacity(MirrorOptions options) {
return 10 * options.getMaxThreads();
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorOptions.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.auth.AWSCredentials;
import lombok.Getter;
import lombok.Setter;
import org.joda.time.DateTime;
import org.kohsuke.args4j.Argument;
import org.kohsuke.args4j.Option;
import java.util.Date;
import static org.cobbzilla.s3s3mirror.MirrorConstants.*;
public class MirrorOptions implements AWSCredentials {
public static final String S3_PROTOCOL_PREFIX = "s3://";
public static final String AWS_ACCESS_KEY = "AWS_ACCESS_KEY_ID";
public static final String AWS_SECRET_KEY = "AWS_SECRET_ACCESS_KEY";
@Getter @Setter private String aWSAccessKeyId = System.getenv().get(AWS_ACCESS_KEY);
@Getter @Setter private String aWSSecretKey = System.getenv().get(AWS_SECRET_KEY);
public boolean hasAwsKeys() { return aWSAccessKeyId != null && aWSSecretKey != null; }
public static final String USAGE_USE_IAM_ROLE = "Use IAM role from EC2 instance, can only be used in AWS";
public static final String OPT_USE_IAM_ROLE = "-i";
public static final String LONGOPT_USE_IAM_ROLE = "--iam";
@Option(name=OPT_USE_IAM_ROLE, aliases=LONGOPT_USE_IAM_ROLE, usage=USAGE_USE_IAM_ROLE)
@Getter @Setter private boolean useIamRole = false;
public static final String USAGE_PROFILE= "Use a specific profile from your credential file (~/.aws/config)";
public static final String OPT_PROFILE= "-P";
public static final String LONGOPT_PROFILE = "--profile";
@Option(name=OPT_PROFILE, aliases=LONGOPT_PROFILE, usage=USAGE_PROFILE)
@Getter @Setter private String profile = null;
public static final String USAGE_DRY_RUN = "Do not actually do anything, but show what would be done";
public static final String OPT_DRY_RUN = "-n";
public static final String LONGOPT_DRY_RUN = "--dry-run";
@Option(name=OPT_DRY_RUN, aliases=LONGOPT_DRY_RUN, usage=USAGE_DRY_RUN)
@Getter @Setter private boolean dryRun = false;
public static final String USAGE_VERBOSE = "Verbose output";
public static final String OPT_VERBOSE = "-v";
public static final String LONGOPT_VERBOSE = "--verbose";
@Option(name=OPT_VERBOSE, aliases=LONGOPT_VERBOSE, usage=USAGE_VERBOSE)
@Getter @Setter private boolean verbose = false;
public static final String USAGE_SSL = "Use SSL for all S3 api operations";
public static final String OPT_SSL = "-s";
public static final String LONGOPT_SSL = "--ssl";
@Option(name=OPT_SSL, aliases=LONGOPT_SSL, usage=USAGE_SSL)
@Getter @Setter private boolean ssl = false;
public static final String USAGE_ENCRYPT = "Enable AWS managed server-side encryption";
public static final String OPT_ENCRYPT = "-E";
public static final String LONGOPT_ENCRYPT = "--server-side-encryption";
@Option(name=OPT_ENCRYPT, aliases=LONGOPT_ENCRYPT, usage=USAGE_ENCRYPT)
@Getter @Setter private boolean encrypt = false;
public static final String USAGE_STORAGE_CLASS = "Specify the S3 StorageClass (Standard | ReducedRedundancy)";
public static final String OPT_STORAGE_CLASS = "-l";
public static final String LONGOPT_STORAGE_CLASS = "--storage-class";
@Option(name=OPT_STORAGE_CLASS, aliases=LONGOPT_STORAGE_CLASS, usage=USAGE_STORAGE_CLASS)
@Getter @Setter private String storageClass = "Standard";
public static final String USAGE_PREFIX = "Only copy objects whose keys start with this prefix";
public static final String OPT_PREFIX = "-p";
public static final String LONGOPT_PREFIX = "--prefix";
@Option(name=OPT_PREFIX, aliases=LONGOPT_PREFIX, usage=USAGE_PREFIX)
@Getter @Setter private String prefix = null;
public boolean hasPrefix () { return prefix != null && prefix.length() > 0; }
public int getPrefixLength () { return prefix == null ? 0 : prefix.length(); }
public static final String USAGE_DEST_PREFIX = "Destination prefix (replacing the one specified in --prefix, if any)";
public static final String OPT_DEST_PREFIX= "-d";
public static final String LONGOPT_DEST_PREFIX = "--dest-prefix";
@Option(name=OPT_DEST_PREFIX, aliases=LONGOPT_DEST_PREFIX, usage=USAGE_DEST_PREFIX)
@Getter @Setter private String destPrefix = null;
public boolean hasDestPrefix() { return destPrefix != null && destPrefix.length() > 0; }
public int getDestPrefixLength () { return destPrefix == null ? 0 : destPrefix.length(); }
public static final String AWS_ENDPOINT = "AWS_ENDPOINT";
public static final String USAGE_ENDPOINT = "AWS endpoint to use (or set "+AWS_ENDPOINT+" in your environment)";
public static final String OPT_ENDPOINT = "-e";
public static final String LONGOPT_ENDPOINT = "--endpoint";
@Option(name=OPT_ENDPOINT, aliases=LONGOPT_ENDPOINT, usage=USAGE_ENDPOINT)
@Getter @Setter private String endpoint = System.getenv().get(AWS_ENDPOINT);
public boolean hasEndpoint () { return endpoint != null && endpoint.trim().length() > 0; }
public static final String USAGE_MAX_CONNECTIONS = "Maximum number of connections to S3 (default 100)";
public static final String OPT_MAX_CONNECTIONS = "-m";
public static final String LONGOPT_MAX_CONNECTIONS = "--max-connections";
@Option(name=OPT_MAX_CONNECTIONS, aliases=LONGOPT_MAX_CONNECTIONS, usage=USAGE_MAX_CONNECTIONS)
@Getter @Setter private int maxConnections = 100;
public static final String USAGE_MAX_THREADS = "Maximum number of threads (default 100)";
public static final String OPT_MAX_THREADS = "-t";
public static final String LONGOPT_MAX_THREADS = "--max-threads";
@Option(name=OPT_MAX_THREADS, aliases=LONGOPT_MAX_THREADS, usage=USAGE_MAX_THREADS)
@Getter @Setter private int maxThreads = 100;
public static final String USAGE_MAX_RETRIES = "Maximum number of retries for S3 requests (default 5)";
public static final String OPT_MAX_RETRIES = "-r";
public static final String LONGOPT_MAX_RETRIES = "--max-retries";
@Option(name=OPT_MAX_RETRIES, aliases=LONGOPT_MAX_RETRIES, usage=USAGE_MAX_RETRIES)
@Getter @Setter private int maxRetries = 5;
public static final String USAGE_SIZE_ONLY = "Only use object size when checking for equality and ignore etags";
public static final String OPT_SIZE_ONLY = "-S";
public static final String LONGOPT_SIZE_ONLY = "--size-only";
@Option(name=OPT_SIZE_ONLY, aliases=LONGOPT_SIZE_ONLY, usage=USAGE_SIZE_ONLY)
@Getter @Setter private boolean sizeOnly = false;
public static final String USAGE_SIZE_LAST_MODIFIED = "Uses size and last modified to determine if files have change like the AWS CLI and ignores etags. If size only is also specified that strategy is selected.";
public static final String OPT_SIZE_LAST_MODIFIED = "-L";
public static final String LONGOPT_SIZE_LAST_MODIFIED = "--size-and-last-modified";
@Option(name=OPT_SIZE_LAST_MODIFIED, aliases=LONGOPT_SIZE_LAST_MODIFIED, usage=USAGE_SIZE_LAST_MODIFIED)
@Getter @Setter private boolean sizeAndLastModified = false;
public static final String USAGE_CTIME = "Only copy objects whose Last-Modified date is younger than this many days. " +
"For other time units, use these suffixes: y (years), M (months), d (days), w (weeks), h (hours), m (minutes), s (seconds)";
public static final String OPT_CTIME = "-c";
public static final String LONGOPT_CTIME = "--ctime";
@Option(name=OPT_CTIME, aliases=LONGOPT_CTIME, usage=USAGE_CTIME)
@Getter @Setter private String ctime = null;
public boolean hasCtime() { return ctime != null; }
private static final String PROXY_USAGE = "host:port of proxy server to use. " +
"Defaults to proxy_host and proxy_port defined in ~/.s3cfg, or no proxy if these values are not found in ~/.s3cfg";
public static final String OPT_PROXY = "-z";
public static final String LONGOPT_PROXY = "--proxy";
@Option(name=OPT_PROXY, aliases=LONGOPT_PROXY, usage=PROXY_USAGE)
public void setProxy(String proxy) {
final String[] splits = proxy.split(":");
if (splits.length != 2) {
throw new IllegalArgumentException("Invalid proxy setting ("+proxy+"), please use host:port");
}
proxyHost = splits[0];
if (proxyHost.trim().length() == 0) {
throw new IllegalArgumentException("Invalid proxy setting ("+proxy+"), please use host:port");
}
try {
proxyPort = Integer.parseInt(splits[1]);
} catch (Exception e) {
throw new IllegalArgumentException("Invalid proxy setting ("+proxy+"), port could not be parsed as a number");
}
}
@Getter @Setter public String proxyHost = null;
@Getter @Setter public int proxyPort = -1;
public boolean getHasProxy() {
boolean hasProxyHost = proxyHost != null && proxyHost.trim().length() > 0;
boolean hasProxyPort = proxyPort != -1;
return hasProxyHost && hasProxyPort;
}
private long initMaxAge() {
DateTime dateTime = new DateTime(nowTime);
// all digits -- assume "days"
if (ctime.matches("^[0-9]+$")) return dateTime.minusDays(Integer.parseInt(ctime)).getMillis();
// ensure there is at least one digit, and exactly one character suffix, and the suffix is a legal option
if (!ctime.matches("^[0-9]+[yMwdhms]$")) throw new IllegalArgumentException("Invalid option for ctime: "+ctime);
if (ctime.endsWith("y")) return dateTime.minusYears(getCtimeNumber(ctime)).getMillis();
if (ctime.endsWith("M")) return dateTime.minusMonths(getCtimeNumber(ctime)).getMillis();
if (ctime.endsWith("w")) return dateTime.minusWeeks(getCtimeNumber(ctime)).getMillis();
if (ctime.endsWith("d")) return dateTime.minusDays(getCtimeNumber(ctime)).getMillis();
if (ctime.endsWith("h")) return dateTime.minusHours(getCtimeNumber(ctime)).getMillis();
if (ctime.endsWith("m")) return dateTime.minusMinutes(getCtimeNumber(ctime)).getMillis();
if (ctime.endsWith("s")) return dateTime.minusSeconds(getCtimeNumber(ctime)).getMillis();
throw new IllegalArgumentException("Invalid option for ctime: "+ctime);
}
private int getCtimeNumber(String ctime) {
return Integer.parseInt(ctime.substring(0, ctime.length() - 1));
}
@Getter private long nowTime = System.currentTimeMillis();
@Getter private long maxAge;
@Getter private String maxAgeDate;
public static final String USAGE_DELETE_REMOVED = "Delete objects from the destination bucket if they do not exist in the source bucket";
public static final String OPT_DELETE_REMOVED = "-X";
public static final String LONGOPT_DELETE_REMOVED = "--delete-removed";
@Option(name=OPT_DELETE_REMOVED, aliases=LONGOPT_DELETE_REMOVED, usage=USAGE_DELETE_REMOVED)
@Getter @Setter private boolean deleteRemoved = false;
@Argument(index=0, required=true, usage="source bucket[/source/prefix]") @Getter @Setter private String source;
@Argument(index=1, required=true, usage="destination bucket[/dest/prefix]") @Getter @Setter private String destination;
@Getter private String sourceBucket;
@Getter private String destinationBucket;
/**
* Current max file size allowed in amazon is 5 GB. We can try and provide this as an option too.
*/
public static final long MAX_SINGLE_REQUEST_UPLOAD_FILE_SIZE = 5 * GB;
private static final long DEFAULT_PART_SIZE = 4 * GB;
private static final String MULTI_PART_UPLOAD_SIZE_USAGE = "The upload size (in bytes) of each part uploaded as part of a multipart request " +
"for files that are greater than the max allowed file size of " + MAX_SINGLE_REQUEST_UPLOAD_FILE_SIZE + " bytes ("+(MAX_SINGLE_REQUEST_UPLOAD_FILE_SIZE/GB)+"GB). " +
"Defaults to " + DEFAULT_PART_SIZE + " bytes ("+(DEFAULT_PART_SIZE/GB)+"GB).";
private static final String OPT_MULTI_PART_UPLOAD_SIZE = "-u";
private static final String LONGOPT_MULTI_PART_UPLOAD_SIZE = "--upload-part-size";
@Option(name=OPT_MULTI_PART_UPLOAD_SIZE, aliases=LONGOPT_MULTI_PART_UPLOAD_SIZE, usage=MULTI_PART_UPLOAD_SIZE_USAGE)
@Getter @Setter private long uploadPartSize = DEFAULT_PART_SIZE;
private static final String CROSS_ACCOUNT_USAGE ="Copy across AWS accounts. Only Resource-based policies are supported (as " +
"specified by AWS documentation) for cross account copying. " +
"Default is false (copying within same account, preserving ACLs across copies). " +
"If this option is active, we give full access to owner of the destination bucket.";
private static final String OPT_CROSS_ACCOUNT_COPY = "-C";
private static final String LONGOPT_CROSS_ACCOUNT_COPY = "--cross-account-copy";
@Option(name=OPT_CROSS_ACCOUNT_COPY, aliases=LONGOPT_CROSS_ACCOUNT_COPY, usage=CROSS_ACCOUNT_USAGE)
@Getter @Setter private boolean crossAccountCopy = false;
public void initDerivedFields() {
if (hasCtime()) {
this.maxAge = initMaxAge();
this.maxAgeDate = new Date(maxAge).toString();
}
String scrubbed;
int slashPos;
scrubbed = scrubS3ProtocolPrefix(source);
slashPos = scrubbed.indexOf('/');
if (slashPos == -1) {
sourceBucket = scrubbed;
} else {
sourceBucket = scrubbed.substring(0, slashPos);
if (hasPrefix()) throw new IllegalArgumentException("Cannot use a "+OPT_PREFIX+"/"+LONGOPT_PREFIX+" argument and source path that includes a prefix at the same time");
prefix = scrubbed.substring(slashPos+1);
}
scrubbed = scrubS3ProtocolPrefix(destination);
slashPos = scrubbed.indexOf('/');
if (slashPos == -1) {
destinationBucket = scrubbed;
} else {
destinationBucket = scrubbed.substring(0, slashPos);
if (hasDestPrefix()) throw new IllegalArgumentException("Cannot use a "+OPT_DEST_PREFIX+"/"+LONGOPT_DEST_PREFIX+" argument and destination path that includes a dest-prefix at the same time");
destPrefix = scrubbed.substring(slashPos+1);
}
}
protected String scrubS3ProtocolPrefix(String bucket) {
bucket = bucket.trim();
if (bucket.startsWith(S3_PROTOCOL_PREFIX)) {
bucket = bucket.substring(S3_PROTOCOL_PREFIX.length());
}
return bucket;
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorStats.java
================================================
package org.cobbzilla.s3s3mirror;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import static org.cobbzilla.s3s3mirror.MirrorConstants.*;
@Slf4j
public class MirrorStats {
@Getter private final Thread shutdownHook = new Thread() {
@Override public void run() { logStats(); }
};
private static final String BANNER = "\n--------------------------------------------------------------------\n";
public void logStats() {
log.info(BANNER + "STATS BEGIN\n" + toString() + "STATS END " + BANNER);
}
private long start = System.currentTimeMillis();
public final AtomicLong objectsRead = new AtomicLong(0);
public final AtomicLong objectsCopied = new AtomicLong(0);
public final AtomicLong copyErrors = new AtomicLong(0);
public final AtomicLong objectsDeleted = new AtomicLong(0);
public final AtomicLong deleteErrors = new AtomicLong(0);
public final AtomicLong s3copyCount = new AtomicLong(0);
public final AtomicLong s3deleteCount = new AtomicLong(0);
public final AtomicLong s3getCount = new AtomicLong(0);
public final AtomicLong bytesCopied = new AtomicLong(0);
public static final long HOUR = TimeUnit.HOURS.toMillis(1);
public static final long MINUTE = TimeUnit.MINUTES.toMillis(1);
public static final long SECOND = TimeUnit.SECONDS.toMillis(1);
public String toString () {
final long durationMillis = System.currentTimeMillis() - start;
final double durationMinutes = durationMillis / 60000.0d;
final String duration = String.format("%d:%02d:%02d", durationMillis / HOUR, (durationMillis % HOUR) / MINUTE, (durationMillis % MINUTE) / SECOND);
final double readRate = objectsRead.get() / durationMinutes;
final double copyRate = objectsCopied.get() / durationMinutes;
final double deleteRate = objectsDeleted.get() / durationMinutes;
return "read: "+objectsRead+ "\n"
+ "copied: "+objectsCopied+"\n"
+ "copy errors: "+copyErrors+"\n"
+ "deleted: "+objectsDeleted+"\n"
+ "delete errors: "+deleteErrors+"\n"
+ "duration: "+duration+"\n"
+ "read rate: "+readRate+"/minute\n"
+ "copy rate: "+copyRate+"/minute\n"
+ "delete rate: "+deleteRate+"/minute\n"
+ "bytes copied: "+formatBytes(bytesCopied.get())+"\n"
+ "GET operations: "+s3getCount+"\n"
+ "COPY operations: "+ s3copyCount+"\n"
+ "DELETE operations: "+ s3deleteCount+"\n";
}
private String formatBytes(long bytesCopied) {
if (bytesCopied > EB) return ((double) bytesCopied) / ((double) EB) + " EB ("+bytesCopied+" bytes)";
if (bytesCopied > PB) return ((double) bytesCopied) / ((double) PB) + " PB ("+bytesCopied+" bytes)";
if (bytesCopied > TB) return ((double) bytesCopied) / ((double) TB) + " TB ("+bytesCopied+" bytes)";
if (bytesCopied > GB) return ((double) bytesCopied) / ((double) GB) + " GB ("+bytesCopied+" bytes)";
if (bytesCopied > MB) return ((double) bytesCopied) / ((double) MB) + " MB ("+bytesCopied+" bytes)";
if (bytesCopied > KB) return ((double) bytesCopied) / ((double) KB) + " KB ("+bytesCopied+" bytes)";
return bytesCopied + " bytes";
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/MultipartKeyCopyJob.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;
import lombok.extern.slf4j.Slf4j;
import org.cobbzilla.s3s3mirror.comparisonstrategies.ComparisonStrategy;
import java.util.ArrayList;
import java.util.List;
@Slf4j
public class MultipartKeyCopyJob extends KeyCopyJob {
public MultipartKeyCopyJob(AmazonS3Client client, MirrorContext context, S3ObjectSummary summary, Object notifyLock, ComparisonStrategy comparisonStrategy) {
super(client, context, summary, notifyLock, comparisonStrategy);
}
@Override
boolean keyCopied(ObjectMetadata sourceMetadata, AccessControlList objectAcl) {
long objectSize = summary.getSize();
MirrorOptions options = context.getOptions();
String sourceBucketName = options.getSourceBucket();
int maxPartRetries = options.getMaxRetries();
String targetBucketName = options.getDestinationBucket();
List<CopyPartResult> copyResponses = new ArrayList<CopyPartResult>();
if (options.isVerbose()) {
log.info("Initiating multipart upload request for " + summary.getKey());
}
InitiateMultipartUploadRequest initiateRequest = new InitiateMultipartUploadRequest(targetBucketName, keydest)
.withObjectMetadata(sourceMetadata);
if (options.isCrossAccountCopy()) {
initiateRequest.withAccessControlList(buildCrossAccountAcl(objectAcl));
} else {
initiateRequest.withAccessControlList(objectAcl);
}
InitiateMultipartUploadResult initResult = client.initiateMultipartUpload(initiateRequest);
long partSize = options.getUploadPartSize();
long bytePosition = 0;
for (int i = 1; bytePosition < objectSize; i++) {
long lastByte = bytePosition + partSize - 1 >= objectSize ? objectSize - 1 : bytePosition + partSize - 1;
String infoMessage = "copying : " + bytePosition + " to " + lastByte;
if (options.isVerbose()) {
log.info(infoMessage);
}
CopyPartRequest copyRequest = new CopyPartRequest()
.withDestinationBucketName(targetBucketName)
.withDestinationKey(keydest)
.withSourceBucketName(sourceBucketName)
.withSourceKey(summary.getKey())
.withUploadId(initResult.getUploadId())
.withFirstByte(bytePosition)
.withLastByte(lastByte)
.withPartNumber(i);
for (int tries = 1; tries <= maxPartRetries; tries++) {
try {
if (options.isVerbose()) log.info("try :" + tries);
context.getStats().s3copyCount.incrementAndGet();
CopyPartResult copyPartResult = client.copyPart(copyRequest);
copyResponses.add(copyPartResult);
if (options.isVerbose()) log.info("completed " + infoMessage);
break;
} catch (Exception e) {
if (tries == maxPartRetries) {
client.abortMultipartUpload(new AbortMultipartUploadRequest(
targetBucketName, keydest, initResult.getUploadId()));
log.error("Exception while doing multipart copy", e);
return false;
}
}
}
bytePosition += partSize;
}
CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest(targetBucketName, keydest,
initResult.getUploadId(), getETags(copyResponses));
client.completeMultipartUpload(completeRequest);
if(options.isVerbose()) {
log.info("completed multipart request for : " + summary.getKey());
}
context.getStats().bytesCopied.addAndGet(objectSize);
return true;
}
private List<PartETag> getETags(List<CopyPartResult> copyResponses) {
List<PartETag> eTags = new ArrayList<PartETag>();
for (CopyPartResult response : copyResponses) {
eTags.add(new PartETag(response.getPartNumber(), response.getETag()));
}
return eTags;
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/Sleep.java
================================================
package org.cobbzilla.s3s3mirror;
import lombok.extern.slf4j.Slf4j;
@Slf4j
public class Sleep {
public static boolean sleep(int millis) {
try {
Thread.sleep(millis);
} catch (InterruptedException e) {
log.error("interrupted!");
return true;
}
return false;
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/ComparisonStrategy.java
================================================
package org.cobbzilla.s3s3mirror.comparisonstrategies;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.S3ObjectSummary;
public interface ComparisonStrategy {
boolean sourceDifferent(S3ObjectSummary source, ObjectMetadata destination);
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/ComparisonStrategyFactory.java
================================================
package org.cobbzilla.s3s3mirror.comparisonstrategies;
import lombok.AccessLevel;
import lombok.NoArgsConstructor;
import org.cobbzilla.s3s3mirror.MirrorOptions;
@NoArgsConstructor(access = AccessLevel.PRIVATE)
public class ComparisonStrategyFactory {
public static ComparisonStrategy getStrategy(MirrorOptions mirrorOptions) {
if (mirrorOptions.isSizeOnly()) {
return new SizeOnlyComparisonStrategy();
} else if (mirrorOptions.isSizeAndLastModified()) {
return new SizeAndLastModifiedComparisonStrategy();
} else {
return new EtagComparisonStrategy();
}
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/EtagComparisonStrategy.java
================================================
package org.cobbzilla.s3s3mirror.comparisonstrategies;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.S3ObjectSummary;
public class EtagComparisonStrategy extends SizeOnlyComparisonStrategy {
@Override
public boolean sourceDifferent(S3ObjectSummary source, ObjectMetadata destination) {
return super.sourceDifferent(source, destination) || !source.getETag().equals(destination.getETag());
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/SizeAndLastModifiedComparisonStrategy.java
================================================
package org.cobbzilla.s3s3mirror.comparisonstrategies;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.S3ObjectSummary;
public class SizeAndLastModifiedComparisonStrategy extends SizeOnlyComparisonStrategy {
@Override
public boolean sourceDifferent(S3ObjectSummary source, ObjectMetadata destination) {
return super.sourceDifferent(source, destination) || source.getLastModified().after(destination.getLastModified());
}
}
================================================
FILE: src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/SizeOnlyComparisonStrategy.java
================================================
package org.cobbzilla.s3s3mirror.comparisonstrategies;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.S3ObjectSummary;
public class SizeOnlyComparisonStrategy implements ComparisonStrategy {
@Override
public boolean sourceDifferent(S3ObjectSummary source, ObjectMetadata destination) {
return source.getSize() != destination.getContentLength();
}
}
================================================
FILE: src/main/resources/log4j.xml
================================================
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration PUBLIC "-//LOGGER" "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<!-- Appenders -->
<appender name="console" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.err" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%t %-5p: %c - %m%n" />
</layout>
</appender>
<!-- our loggers -->
<logger name="org.cobbzilla">
<level value="info" />
</logger>
<!-- Root Logger -->
<root>
<priority value="info" />
<appender-ref ref="console" />
</root>
</log4j:configuration>
================================================
FILE: src/test/java/org/cobbzilla/s3s3mirror/MirrorMainTest.java
================================================
package org.cobbzilla.s3s3mirror;
import org.junit.Test;
import static org.junit.Assert.*;
public class MirrorMainTest {
public static final String SOURCE = "s3://from-bucket";
public static final String DESTINATION = "s3://to-bucket";
@Test
public void testBasicArgs() throws Exception {
final MirrorMain main = new MirrorMain(new String[]{SOURCE, DESTINATION});
main.parseArguments();
final MirrorOptions options = main.getOptions();
assertFalse(options.isDryRun());
assertEquals(SOURCE, options.getSource());
assertEquals(DESTINATION, options.getDestination());
}
@Test
public void testDryRunArgs() throws Exception {
final MirrorMain main = new MirrorMain(new String[]{MirrorOptions.OPT_DRY_RUN, SOURCE, DESTINATION});
main.parseArguments();
final MirrorOptions options = main.getOptions();
assertTrue(options.isDryRun());
assertEquals(SOURCE, options.getSource());
assertEquals(DESTINATION, options.getDestination());
}
@Test
public void testMaxConnectionsArgs() throws Exception {
int maxConns = 42;
final MirrorMain main = new MirrorMain(new String[]{MirrorOptions.OPT_MAX_CONNECTIONS, String.valueOf(maxConns), SOURCE, DESTINATION});
main.parseArguments();
final MirrorOptions options = main.getOptions();
assertFalse(options.isDryRun());
assertEquals(maxConns, options.getMaxConnections());
assertEquals(SOURCE, options.getSource());
assertEquals(DESTINATION, options.getDestination());
}
@Test
public void testInlinePrefix() throws Exception {
final String prefix = "foo";
final MirrorMain main = new MirrorMain(new String[]{SOURCE + "/" + prefix, DESTINATION});
main.parseArguments();
final MirrorOptions options = main.getOptions();
assertEquals(prefix, options.getPrefix());
assertNull(options.getDestPrefix());
}
@Test
public void testInlineDestPrefix() throws Exception {
final String destPrefix = "foo";
final MirrorMain main = new MirrorMain(new String[]{SOURCE, DESTINATION + "/" + destPrefix});
main.parseArguments();
final MirrorOptions options = main.getOptions();
assertEquals(destPrefix, options.getDestPrefix());
assertNull(options.getPrefix());
}
@Test
public void testInlineSourceAndDestPrefix() throws Exception {
final String prefix = "foo";
final String destPrefix = "bar";
final MirrorMain main = new MirrorMain(new String[]{SOURCE + "/" + prefix, DESTINATION + "/" + destPrefix});
main.parseArguments();
final MirrorOptions options = main.getOptions();
assertEquals(prefix, options.getPrefix());
assertEquals(destPrefix, options.getDestPrefix());
}
@Test
public void testInlineSourcePrefixAndPrefixOption() throws Exception {
final String prefix = "foo";
final MirrorMain main = new MirrorMain(new String[]{MirrorOptions.OPT_PREFIX, prefix, SOURCE + "/" + prefix, DESTINATION});
try {
main.parseArguments();
fail("expected IllegalArgumentException");
} catch (Exception e) {
assertTrue(e instanceof IllegalArgumentException);
}
}
@Test
public void testInlineDestinationPrefixAndPrefixOption() throws Exception {
final String prefix = "foo";
final MirrorMain main = new MirrorMain(new String[]{MirrorOptions.OPT_DEST_PREFIX, prefix, SOURCE, DESTINATION + "/" + prefix});
try {
main.parseArguments();
fail("expected IllegalArgumentException");
} catch (Exception e) {
assertTrue(e instanceof IllegalArgumentException);
}
}
/**
* When access keys are read from environment then the --proxy setting is valid.
* If access keys are ready from s3cfg file then proxy settings are picked from there.
* @throws Exception
*/
@Test
public void testProxyHostAndProxyPortOption() throws Exception {
final String proxy = "localhost:8080";
final MirrorMain main = new MirrorMain(new String[]{MirrorOptions.OPT_PROXY, proxy, SOURCE, DESTINATION});
main.getOptions().setAWSAccessKeyId("accessKey");
main.getOptions().setAWSSecretKey("secretKey");
main.parseArguments();
assertEquals("localhost", main.getOptions().getProxyHost());
assertEquals(8080, main.getOptions().getProxyPort());
}
@Test
public void testInvalidProxyOption () throws Exception {
for (String proxy : new String[] {"localhost", "localhost:", ":1234", "localhost:invalid", ":", ""} ) {
testInvalidProxySetting(proxy);
}
}
private void testInvalidProxySetting(String proxy) throws Exception {
final MirrorMain main = new MirrorMain(new String[]{MirrorOptions.OPT_PROXY, proxy, SOURCE, DESTINATION});
main.getOptions().setAWSAccessKeyId("accessKey");
main.getOptions().setAWSSecretKey("secretKey");
try {
main.parseArguments();
fail("Invalid proxy setting ("+proxy+") should have thrown exception");
} catch (IllegalArgumentException expected) {}
}
}
================================================
FILE: src/test/java/org/cobbzilla/s3s3mirror/MirrorTest.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.AmazonS3Exception;
import com.amazonaws.services.s3.model.ObjectMetadata;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.RandomStringUtils;
import org.junit.After;
import org.junit.Test;
import java.util.ArrayList;
import java.util.List;
import static org.cobbzilla.s3s3mirror.MirrorOptions.*;
import static org.cobbzilla.s3s3mirror.TestFile.Clean;
import static org.cobbzilla.s3s3mirror.TestFile.Copy;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.fail;
@Slf4j
public class MirrorTest {
public static final String SOURCE_ENV_VAR = "S3S3_TEST_SOURCE";
public static final String DEST_ENV_VAR = "S3S3_TEST_DEST";
public static final String SOURCE = System.getenv(SOURCE_ENV_VAR);
public static final String DESTINATION = System.getenv(DEST_ENV_VAR);
private List<S3Asset> stuffToCleanup = new ArrayList<S3Asset>();
// Every individual test *must* initialize the "main" instance variable, otherwise NPE gets thrown here.
private MirrorMain main = null;
private TestFile createTestFile(String key, Copy copy, Clean clean) throws Exception {
return TestFile.create(key, main.getClient(), stuffToCleanup, copy, clean);
}
public static String random(int size) {
return RandomStringUtils.randomAlphanumeric(size) + "_" + System.currentTimeMillis();
}
private boolean checkEnvs() {
if (SOURCE == null || DESTINATION == null) {
log.warn("No "+SOURCE_ENV_VAR+" and/or no "+DEST_ENV_VAR+" found in enviroment, skipping test");
return false;
}
return true;
}
@After
public void cleanupS3Assets () {
// Every individual test *must* initialize the "main" instance variable, otherwise NPE gets thrown here.
if (checkEnvs()) {
AmazonS3Client client = main.getClient();
for (S3Asset asset : stuffToCleanup) {
try {
log.info("cleanupS3Assets: deleting "+asset);
client.deleteObject(asset.bucket, asset.key);
} catch (Exception e) {
log.error("Error cleaning up object: "+asset+": "+e.getMessage());
}
}
main = null;
}
}
@Test
public void testSimpleCopy () throws Exception {
if (!checkEnvs()) return;
final String key = "testSimpleCopy_"+random(10);
final String[] args = {OPT_VERBOSE, OPT_PREFIX, key, SOURCE, DESTINATION};
testSimpleCopyInternal(key, args);
}
@Test
public void testSimpleCopyWithInlinePrefix () throws Exception {
if (!checkEnvs()) return;
final String key = "testSimpleCopyWithInlinePrefix_"+random(10);
final String[] args = {OPT_VERBOSE, SOURCE + "/" + key, DESTINATION};
testSimpleCopyInternal(key, args);
}
private void testSimpleCopyInternal(String key, String[] args) throws Exception {
main = new MirrorMain(args);
main.init();
final TestFile testFile = createTestFile(key, Copy.SOURCE, Clean.SOURCE_AND_DEST);
main.run();
assertEquals(1, main.getContext().getStats().objectsCopied.get());
assertEquals(testFile.data.length(), main.getContext().getStats().bytesCopied.get());
final ObjectMetadata metadata = main.getClient().getObjectMetadata(DESTINATION, key);
assertEquals(testFile.data.length(), metadata.getContentLength());
}
@Test
public void testSimpleCopyWithDestPrefix () throws Exception {
if (!checkEnvs()) return;
final String key = "testSimpleCopyWithDestPrefix_"+random(10);
final String destKey = "dest_testSimpleCopyWithDestPrefix_"+random(10);
final String[] args = {OPT_PREFIX, key, OPT_DEST_PREFIX, destKey, SOURCE, DESTINATION};
testSimpleCopyWithDestPrefixInternal(key, destKey, args);
}
@Test
public void testSimpleCopyWithInlineDestPrefix () throws Exception {
if (!checkEnvs()) return;
final String key = "testSimpleCopyWithInlineDestPrefix_"+random(10);
final String destKey = "dest_testSimpleCopyWithInlineDestPrefix_"+random(10);
final String[] args = {SOURCE+"/"+key, DESTINATION+"/"+destKey };
testSimpleCopyWithDestPrefixInternal(key, destKey, args);
}
private void testSimpleCopyWithDestPrefixInternal(String key, String destKey, String[] args) throws Exception {
main = new MirrorMain(args);
main.init();
final TestFile testFile = createTestFile(key, Copy.SOURCE, Clean.SOURCE);
stuffToCleanup.add(new S3Asset(DESTINATION, destKey));
main.run();
assertEquals(1, main.getContext().getStats().objectsCopied.get());
assertEquals(testFile.data.length(), main.getContext().getStats().bytesCopied.get());
final ObjectMetadata metadata = main.getClient().getObjectMetadata(DESTINATION, destKey);
assertEquals(testFile.data.length(), metadata.getContentLength());
}
@Test
public void testDeleteRemoved () throws Exception {
if (!checkEnvs()) return;
final String key = "testDeleteRemoved_"+random(10);
main = new MirrorMain(new String[]{OPT_VERBOSE, OPT_PREFIX, key,
OPT_DELETE_REMOVED, SOURCE, DESTINATION});
main.init();
// Write some files to dest
final int numDestFiles = 3;
final String[] destKeys = new String[numDestFiles];
final TestFile[] destFiles = new TestFile[numDestFiles];
for (int i=0; i<numDestFiles; i++) {
destKeys[i] = key + "-dest" + i;
destFiles[i] = createTestFile(destKeys[i], Copy.DEST, Clean.DEST);
}
// Write 1 file to source
final String srcKey = key + "-src";
final TestFile srcFile = createTestFile(srcKey, Copy.SOURCE, Clean.SOURCE_AND_DEST);
// Initiate copy
main.run();
// Expect only 1 copy and numDestFiles deletes
assertEquals(1, main.getContext().getStats().objectsCopied.get());
assertEquals(numDestFiles, main.getContext().getStats().objectsDeleted.get());
// Expect none of the original dest files to be there anymore
for (int i=0; i<numDestFiles; i++) {
try {
main.getClient().getObjectMetadata(DESTINATION, destKeys[i]);
fail("testDeleteRemoved: expected "+destKeys[i]+" to be removed from destination bucket "+DESTINATION);
} catch (AmazonS3Exception e) {
if (e.getStatusCode() != 404) {
fail("testDeleteRemoved: unexpected exception (expected statusCode == 404): "+e);
}
}
}
// Expect source file to now be present in both source and destination buckets
ObjectMetadata metadata;
metadata = main.getClient().getObjectMetadata(SOURCE, srcKey);
assertEquals(srcFile.data.length(), metadata.getContentLength());
metadata = main.getClient().getObjectMetadata(DESTINATION, srcKey);
assertEquals(srcFile.data.length(), metadata.getContentLength());
}
}
================================================
FILE: src/test/java/org/cobbzilla/s3s3mirror/S3Asset.java
================================================
package org.cobbzilla.s3s3mirror;
import lombok.AllArgsConstructor;
import lombok.ToString;
@AllArgsConstructor @ToString
class S3Asset {
public String bucket;
public String key;
}
================================================
FILE: src/test/java/org/cobbzilla/s3s3mirror/SyncStrategiesTest.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import org.cobbzilla.s3s3mirror.comparisonstrategies.EtagComparisonStrategy;
import org.cobbzilla.s3s3mirror.comparisonstrategies.SizeAndLastModifiedComparisonStrategy;
import org.cobbzilla.s3s3mirror.comparisonstrategies.SizeOnlyComparisonStrategy;
import org.junit.Test;
import java.sql.Timestamp;
import java.time.LocalDateTime;
import java.util.Random;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
public class SyncStrategiesTest {
private final EtagComparisonStrategy etagComparisonStrategy = new EtagComparisonStrategy();
private final SizeOnlyComparisonStrategy sizeOnlyComparisonStrategy = new SizeOnlyComparisonStrategy();
private final SizeAndLastModifiedComparisonStrategy sizeAndLastModifiedComparisonStrategy = new SizeAndLastModifiedComparisonStrategy();
private static final String ETAG_A = "ETAG_A";
private static final String ETAG_B = "ETAG_B";
private static final long SIZE_A = 0;
private static final long SIZE_B = 1;
private static final LocalDateTime TIME_EARLY = LocalDateTime.of(2020, 1, 1, 0, 0);
private static final LocalDateTime TIME_LATER = TIME_EARLY.plusDays(1);
@Test
public void testEtaStrategygEtagAndSizeMatch() {
S3ObjectSummary source = createTestS3ObjectSummary(ETAG_A, SIZE_A);
ObjectMetadata destination = createTestObjectMetadata(ETAG_A, SIZE_A);
assertFalse(etagComparisonStrategy.sourceDifferent(source, destination));
}
@Test
public void testEtagStrategySizeMatchEtagDoesNot() {
S3ObjectSummary source = createTestS3ObjectSummary(ETAG_A, SIZE_A);
ObjectMetadata destination = createTestObjectMetadata(ETAG_B, SIZE_A);
assertTrue(etagComparisonStrategy.sourceDifferent(source, destination));
}
@Test
public void testEtagStrategyEtagMatchSizeDoesNot() {
S3ObjectSummary source = createTestS3ObjectSummary(ETAG_A, SIZE_A);
ObjectMetadata destination = createTestObjectMetadata(ETAG_A, SIZE_B);
assertTrue(etagComparisonStrategy.sourceDifferent(source, destination));
}
@Test
public void testSizeStrategySizeMatches() {
S3ObjectSummary source = createTestS3ObjectSummary(SIZE_A);
ObjectMetadata destination = createTestObjectMetadata(SIZE_A);
assertFalse(sizeOnlyComparisonStrategy.sourceDifferent(source, destination));
}
@Test
public void testSizeStrategySizeDoesNotMatches() {
S3ObjectSummary source = createTestS3ObjectSummary(SIZE_A);
ObjectMetadata destination = createTestObjectMetadata(SIZE_B);
assertTrue(sizeOnlyComparisonStrategy.sourceDifferent(source, destination));
}
@Test
public void testSizeAndLastModifiedStrategySizeAndLastModifiedMatch() {
S3ObjectSummary source = createTestS3ObjectSummary(SIZE_A, TIME_EARLY);
ObjectMetadata destination = createTestObjectMetadata(SIZE_A, TIME_EARLY);
assertFalse(sizeAndLastModifiedComparisonStrategy.sourceDifferent(source, destination));
}
@Test
public void testSizeAndLastModifiedStrategyLastModifiedMatchSizeDoesNot() {
S3ObjectSummary source = createTestS3ObjectSummary(SIZE_A, TIME_EARLY);
ObjectMetadata destination = createTestObjectMetadata(SIZE_B, TIME_EARLY);
assertTrue(sizeAndLastModifiedComparisonStrategy.sourceDifferent(source, destination));
}
@Test
public void testSizeAndLastModifiedStrategySizeMatchDestinationAfterSource() {
S3ObjectSummary source = createTestS3ObjectSummary(SIZE_A, TIME_EARLY);
ObjectMetadata destination = createTestObjectMetadata(SIZE_A, TIME_LATER);
assertFalse(sizeAndLastModifiedComparisonStrategy.sourceDifferent(source, destination));
}
@Test
public void testSizeAndLastModifiedStrategySizeMatchSourceAfterDestination() {
S3ObjectSummary source = createTestS3ObjectSummary(SIZE_A, TIME_LATER);
ObjectMetadata destination = createTestObjectMetadata(SIZE_A, TIME_EARLY);
assertTrue(sizeAndLastModifiedComparisonStrategy.sourceDifferent(source, destination));
}
private S3ObjectSummary createTestS3ObjectSummary(long size) {
return createTestS3ObjectSummary(randomString(), size);
}
private S3ObjectSummary createTestS3ObjectSummary(String etag, long size) {
return createTestS3ObjectSummary(etag, size, LocalDateTime.now());
}
private S3ObjectSummary createTestS3ObjectSummary(long size, LocalDateTime lastModifiedDate) {
return createTestS3ObjectSummary(randomString(), size, lastModifiedDate);
}
private S3ObjectSummary createTestS3ObjectSummary(String etag, long size, LocalDateTime lastModified) {
S3ObjectSummary summary = new S3ObjectSummary();
summary.setETag(etag);
summary.setSize(size);
summary.setLastModified(Timestamp.valueOf(lastModified));
return summary;
}
private ObjectMetadata createTestObjectMetadata(long size) {
return createTestObjectMetadata(randomString(), size);
}
private ObjectMetadata createTestObjectMetadata(String etag, long size) {
return createTestObjectMetadata(etag, size, LocalDateTime.now());
}
private ObjectMetadata createTestObjectMetadata(long size, LocalDateTime lastModified) {
return createTestObjectMetadata(randomString(), size, lastModified);
}
private ObjectMetadata createTestObjectMetadata(String etag, long size, LocalDateTime lastModified) {
ObjectMetadata metadata = mock(ObjectMetadata.class);
doReturn(etag).when(metadata).getETag();
doReturn(size).when(metadata).getContentLength();
doReturn(Timestamp.valueOf(lastModified)).when(metadata).getLastModified();
return metadata;
}
private String randomString() {
return Integer.toString(new Random().nextInt(1000));
}
}
================================================
FILE: src/test/java/org/cobbzilla/s3s3mirror/TestFile.java
================================================
package org.cobbzilla.s3s3mirror;
import com.amazonaws.services.s3.AmazonS3Client;
import lombok.Cleanup;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.RandomUtils;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.util.List;
class TestFile {
public static final int TEST_FILE_SIZE = 1024;
enum Copy { SOURCE, DEST, SOURCE_AND_DEST }
enum Clean { SOURCE, DEST, SOURCE_AND_DEST }
public File file;
public String data;
public TestFile() throws Exception{
file = File.createTempFile(getClass().getName(), ".tmp");
data = MirrorTest.random(TEST_FILE_SIZE + (RandomUtils.nextInt() % 1024));
@Cleanup FileOutputStream out = new FileOutputStream(file);
IOUtils.copy(new ByteArrayInputStream(data.getBytes()), out);
file.deleteOnExit();
}
public static TestFile create(String key, AmazonS3Client client, List<S3Asset> stuffToCleanup, Copy copy, Clean clean) throws Exception {
TestFile testFile = new TestFile();
switch (clean) {
case SOURCE:
stuffToCleanup.add(new S3Asset(MirrorTest.SOURCE, key));
break;
case DEST:
stuffToCleanup.add(new S3Asset(MirrorTest.DESTINATION, key));
break;
case SOURCE_AND_DEST:
stuffToCleanup.add(new S3Asset(MirrorTest.SOURCE, key));
stuffToCleanup.add(new S3Asset(MirrorTest.DESTINATION, key));
break;
}
switch (copy) {
case SOURCE:
client.putObject(MirrorTest.SOURCE, key, testFile.file);
break;
case DEST:
client.putObject(MirrorTest.DESTINATION, key, testFile.file);
break;
case SOURCE_AND_DEST:
client.putObject(MirrorTest.SOURCE, key, testFile.file);
client.putObject(MirrorTest.DESTINATION, key, testFile.file);
break;
}
return testFile;
}
}
gitextract_cx814f8m/
├── .gitignore
├── LICENSE.txt
├── README.md
├── pom.xml
├── s3s3mirror.bat
├── s3s3mirror.sh
└── src/
├── main/
│ ├── java/
│ │ └── org/
│ │ └── cobbzilla/
│ │ └── s3s3mirror/
│ │ ├── CopyMaster.java
│ │ ├── DeleteMaster.java
│ │ ├── KeyCopyJob.java
│ │ ├── KeyDeleteJob.java
│ │ ├── KeyFingerprint.java
│ │ ├── KeyJob.java
│ │ ├── KeyLister.java
│ │ ├── KeyMaster.java
│ │ ├── MirrorConstants.java
│ │ ├── MirrorContext.java
│ │ ├── MirrorMain.java
│ │ ├── MirrorMaster.java
│ │ ├── MirrorOptions.java
│ │ ├── MirrorStats.java
│ │ ├── MultipartKeyCopyJob.java
│ │ ├── Sleep.java
│ │ └── comparisonstrategies/
│ │ ├── ComparisonStrategy.java
│ │ ├── ComparisonStrategyFactory.java
│ │ ├── EtagComparisonStrategy.java
│ │ ├── SizeAndLastModifiedComparisonStrategy.java
│ │ └── SizeOnlyComparisonStrategy.java
│ └── resources/
│ └── log4j.xml
└── test/
└── java/
└── org/
└── cobbzilla/
└── s3s3mirror/
├── MirrorMainTest.java
├── MirrorTest.java
├── S3Asset.java
├── SyncStrategiesTest.java
└── TestFile.java
SYMBOL INDEX (149 symbols across 26 files)
FILE: src/main/java/org/cobbzilla/s3s3mirror/CopyMaster.java
class CopyMaster (line 12) | public class CopyMaster extends KeyMaster {
method CopyMaster (line 15) | public CopyMaster(AmazonS3Client client, MirrorContext context, Blocki...
method getPrefix (line 20) | protected String getPrefix(MirrorOptions options) { return options.get...
method getBucket (line 21) | protected String getBucket(MirrorOptions options) { return options.get...
method getTask (line 23) | protected KeyCopyJob getTask(S3ObjectSummary summary) {
FILE: src/main/java/org/cobbzilla/s3s3mirror/DeleteMaster.java
class DeleteMaster (line 9) | public class DeleteMaster extends KeyMaster {
method DeleteMaster (line 11) | public DeleteMaster(AmazonS3Client client, MirrorContext context, Bloc...
method getPrefix (line 15) | protected String getPrefix(MirrorOptions options) {
method getBucket (line 19) | protected String getBucket(MirrorOptions options) { return options.get...
method getTask (line 21) | @Override
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyCopyJob.java
class KeyCopyJob (line 14) | @Slf4j
method KeyCopyJob (line 20) | public KeyCopyJob(AmazonS3Client client, MirrorContext context, S3Obje...
method getLog (line 32) | @Override public Logger getLog() { return log; }
method run (line 34) | @Override
method keyCopied (line 63) | boolean keyCopied(ObjectMetadata sourceMetadata, AccessControlList obj...
method shouldTransfer (line 106) | private boolean shouldTransfer() {
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyDeleteJob.java
class KeyDeleteJob (line 8) | @Slf4j
method KeyDeleteJob (line 13) | public KeyDeleteJob (AmazonS3Client client, MirrorContext context, S3O...
method getLog (line 24) | @Override public Logger getLog() { return log; }
method run (line 26) | @Override
method shouldDelete (line 82) | private boolean shouldDelete() {
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyFingerprint.java
class KeyFingerprint (line 5) | @EqualsAndHashCode(callSuper=false) @AllArgsConstructor
method KeyFingerprint (line 11) | public KeyFingerprint(long size) {
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyJob.java
class KeyJob (line 7) | public abstract class KeyJob implements Runnable {
method KeyJob (line 14) | public KeyJob(AmazonS3Client client, MirrorContext context, S3ObjectSu...
method getLog (line 21) | public abstract Logger getLog();
method toString (line 23) | @Override public String toString() { return summary.getKey(); }
method getObjectMetadata (line 25) | protected ObjectMetadata getObjectMetadata(String bucket, String key, ...
method getAccessControlList (line 50) | protected AccessControlList getAccessControlList(MirrorOptions options...
method buildCrossAccountAcl (line 83) | AccessControlList buildCrossAccountAcl(AccessControlList original) {
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyLister.java
class KeyLister (line 14) | @Slf4j
method isDone (line 25) | public boolean isDone () { return done.get(); }
method KeyLister (line 27) | public KeyLister(AmazonS3Client client, MirrorContext context, int max...
method run (line 46) | @Override
method s3getFirstBatch (line 86) | private ObjectListing s3getFirstBatch(AmazonS3Client client, ListObjec...
method s3getNextBatch (line 112) | private ObjectListing s3getNextBatch() {
method getSize (line 138) | private int getSize() {
method getNextBatch (line 144) | public List<S3ObjectSummary> getNextBatch() {
FILE: src/main/java/org/cobbzilla/s3s3mirror/KeyMaster.java
class KeyMaster (line 13) | @Slf4j
method isDone (line 23) | public boolean isDone () { return done.get(); }
method KeyMaster (line 31) | public KeyMaster(AmazonS3Client client, MirrorContext context, Blockin...
method getPrefix (line 38) | protected abstract String getPrefix(MirrorOptions options);
method getBucket (line 39) | protected abstract String getBucket(MirrorOptions options);
method getTask (line 41) | protected abstract KeyJob getTask(S3ObjectSummary summary);
method start (line 43) | public void start () {
method stop (line 48) | public void stop () {
method run (line 71) | public void run() {
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorConstants.java
class MirrorConstants (line 3) | public class MirrorConstants {
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorContext.java
class MirrorContext (line 8) | @AllArgsConstructor
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorMain.java
class MirrorMain (line 23) | @Slf4j
method uncaughtException (line 33) | @Override public void uncaughtException(Thread t, Throwable e) {
method MirrorMain (line 42) | public MirrorMain(String[] args) { this.args = args; }
method main (line 44) | public static void main (String[] args) {
method run (line 49) | public void run() {
method init (line 54) | public void init() {
method getAmazonS3Client (line 73) | protected AmazonS3Client getAmazonS3Client() {
method parseArguments (line 96) | protected void parseArguments() throws Exception {
method loadAwsKeysFromS3Config (line 120) | private void loadAwsKeysFromS3Config() {
method loadAwsKeysFromAwsConfig (line 141) | private void loadAwsKeysFromAwsConfig() {
method loadAwsKeysFromAwsCredentials (line 172) | private void loadAwsKeysFromAwsCredentials() {
method getTargetBucketOwner (line 203) | private Owner getTargetBucketOwner(AmazonS3Client client) {
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorMaster.java
class MirrorMaster (line 11) | @Slf4j
method MirrorMaster (line 19) | public MirrorMaster(AmazonS3Client client, MirrorContext context) {
method mirror (line 24) | public void mirror() {
method getMaxQueueCapacity (line 73) | public static int getMaxQueueCapacity(MirrorOptions options) {
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorOptions.java
class MirrorOptions (line 15) | public class MirrorOptions implements AWSCredentials {
method hasAwsKeys (line 24) | public boolean hasAwsKeys() { return aWSAccessKeyId != null && aWSSecr...
method hasPrefix (line 74) | public boolean hasPrefix () { return prefix != null && prefix.length()...
method getPrefixLength (line 75) | public int getPrefixLength () { return prefix == null ? 0 : prefix.len...
method hasDestPrefix (line 83) | public boolean hasDestPrefix() { return destPrefix != null && destPref...
method getDestPrefixLength (line 84) | public int getDestPrefixLength () { return destPrefix == null ? 0 : de...
method hasEndpoint (line 94) | public boolean hasEndpoint () { return endpoint != null && endpoint.tr...
method hasCtime (line 132) | public boolean hasCtime() { return ctime != null; }
method setProxy (line 139) | @Option(name=OPT_PROXY, aliases=LONGOPT_PROXY, usage=PROXY_USAGE)
method getHasProxy (line 159) | public boolean getHasProxy() {
method initMaxAge (line 166) | private long initMaxAge() {
method getCtimeNumber (line 186) | private int getCtimeNumber(String ctime) {
method initDerivedFields (line 228) | public void initDerivedFields() {
method scrubS3ProtocolPrefix (line 259) | protected String scrubS3ProtocolPrefix(String bucket) {
FILE: src/main/java/org/cobbzilla/s3s3mirror/MirrorStats.java
class MirrorStats (line 11) | @Slf4j
method run (line 15) | @Override public void run() { logStats(); }
method logStats (line 19) | public void logStats() {
method toString (line 40) | public String toString () {
method formatBytes (line 62) | private String formatBytes(long bytesCopied) {
FILE: src/main/java/org/cobbzilla/s3s3mirror/MultipartKeyCopyJob.java
class MultipartKeyCopyJob (line 11) | @Slf4j
method MultipartKeyCopyJob (line 14) | public MultipartKeyCopyJob(AmazonS3Client client, MirrorContext contex...
method keyCopied (line 18) | @Override
method getETags (line 88) | private List<PartETag> getETags(List<CopyPartResult> copyResponses) {
FILE: src/main/java/org/cobbzilla/s3s3mirror/Sleep.java
class Sleep (line 5) | @Slf4j
method sleep (line 8) | public static boolean sleep(int millis) {
FILE: src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/ComparisonStrategy.java
type ComparisonStrategy (line 6) | public interface ComparisonStrategy {
method sourceDifferent (line 7) | boolean sourceDifferent(S3ObjectSummary source, ObjectMetadata destina...
FILE: src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/ComparisonStrategyFactory.java
class ComparisonStrategyFactory (line 7) | @NoArgsConstructor(access = AccessLevel.PRIVATE)
method getStrategy (line 9) | public static ComparisonStrategy getStrategy(MirrorOptions mirrorOptio...
FILE: src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/EtagComparisonStrategy.java
class EtagComparisonStrategy (line 6) | public class EtagComparisonStrategy extends SizeOnlyComparisonStrategy {
method sourceDifferent (line 7) | @Override
FILE: src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/SizeAndLastModifiedComparisonStrategy.java
class SizeAndLastModifiedComparisonStrategy (line 6) | public class SizeAndLastModifiedComparisonStrategy extends SizeOnlyCompa...
method sourceDifferent (line 7) | @Override
FILE: src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/SizeOnlyComparisonStrategy.java
class SizeOnlyComparisonStrategy (line 6) | public class SizeOnlyComparisonStrategy implements ComparisonStrategy {
method sourceDifferent (line 7) | @Override
FILE: src/test/java/org/cobbzilla/s3s3mirror/MirrorMainTest.java
class MirrorMainTest (line 7) | public class MirrorMainTest {
method testBasicArgs (line 12) | @Test
method testDryRunArgs (line 24) | @Test
method testMaxConnectionsArgs (line 36) | @Test
method testInlinePrefix (line 50) | @Test
method testInlineDestPrefix (line 61) | @Test
method testInlineSourceAndDestPrefix (line 72) | @Test
method testInlineSourcePrefixAndPrefixOption (line 84) | @Test
method testInlineDestinationPrefixAndPrefixOption (line 96) | @Test
method testProxyHostAndProxyPortOption (line 113) | @Test
method testInvalidProxyOption (line 125) | @Test
method testInvalidProxySetting (line 132) | private void testInvalidProxySetting(String proxy) throws Exception {
FILE: src/test/java/org/cobbzilla/s3s3mirror/MirrorTest.java
class MirrorTest (line 20) | @Slf4j
method createTestFile (line 34) | private TestFile createTestFile(String key, Copy copy, Clean clean) th...
method random (line 38) | public static String random(int size) {
method checkEnvs (line 42) | private boolean checkEnvs() {
method cleanupS3Assets (line 50) | @After
method testSimpleCopy (line 67) | @Test
method testSimpleCopyWithInlinePrefix (line 76) | @Test
method testSimpleCopyInternal (line 85) | private void testSimpleCopyInternal(String key, String[] args) throws ...
method testSimpleCopyWithDestPrefix (line 101) | @Test
method testSimpleCopyWithInlineDestPrefix (line 110) | @Test
method testSimpleCopyWithDestPrefixInternal (line 119) | private void testSimpleCopyWithDestPrefixInternal(String key, String d...
method testDeleteRemoved (line 135) | @Test
FILE: src/test/java/org/cobbzilla/s3s3mirror/S3Asset.java
class S3Asset (line 6) | @AllArgsConstructor @ToString
FILE: src/test/java/org/cobbzilla/s3s3mirror/SyncStrategiesTest.java
class SyncStrategiesTest (line 19) | public class SyncStrategiesTest {
method testEtaStrategygEtagAndSizeMatch (line 32) | @Test
method testEtagStrategySizeMatchEtagDoesNot (line 40) | @Test
method testEtagStrategyEtagMatchSizeDoesNot (line 48) | @Test
method testSizeStrategySizeMatches (line 56) | @Test
method testSizeStrategySizeDoesNotMatches (line 64) | @Test
method testSizeAndLastModifiedStrategySizeAndLastModifiedMatch (line 72) | @Test
method testSizeAndLastModifiedStrategyLastModifiedMatchSizeDoesNot (line 80) | @Test
method testSizeAndLastModifiedStrategySizeMatchDestinationAfterSource (line 88) | @Test
method testSizeAndLastModifiedStrategySizeMatchSourceAfterDestination (line 96) | @Test
method createTestS3ObjectSummary (line 104) | private S3ObjectSummary createTestS3ObjectSummary(long size) {
method createTestS3ObjectSummary (line 108) | private S3ObjectSummary createTestS3ObjectSummary(String etag, long si...
method createTestS3ObjectSummary (line 112) | private S3ObjectSummary createTestS3ObjectSummary(long size, LocalDate...
method createTestS3ObjectSummary (line 116) | private S3ObjectSummary createTestS3ObjectSummary(String etag, long si...
method createTestObjectMetadata (line 126) | private ObjectMetadata createTestObjectMetadata(long size) {
method createTestObjectMetadata (line 130) | private ObjectMetadata createTestObjectMetadata(String etag, long size) {
method createTestObjectMetadata (line 134) | private ObjectMetadata createTestObjectMetadata(long size, LocalDateTi...
method createTestObjectMetadata (line 138) | private ObjectMetadata createTestObjectMetadata(String etag, long size...
method randomString (line 148) | private String randomString() {
FILE: src/test/java/org/cobbzilla/s3s3mirror/TestFile.java
class TestFile (line 13) | class TestFile {
type Copy (line 17) | enum Copy { SOURCE, DEST, SOURCE_AND_DEST }
type Clean (line 18) | enum Clean { SOURCE, DEST, SOURCE_AND_DEST }
method TestFile (line 23) | public TestFile() throws Exception{
method create (line 31) | public static TestFile create(String key, AmazonS3Client client, List<...
Condensed preview — 33 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (108K chars).
[
{
"path": ".gitignore",
"chars": 108,
"preview": "*.iml\n.idea\ntmp\nlogs\ndependency-reduced-pom.xml\n*~\ntarget\n!target/*.jar\n*.log\n.settings\n.classpath\n.project\n"
},
{
"path": "LICENSE.txt",
"chars": 559,
"preview": "Copyright 2017-2021 Jonathan Cobb\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this "
},
{
"path": "README.md",
"chars": 7607,
"preview": "s3s3mirror\n==========\n\nA utility for mirroring content from one S3 bucket to another.\n\nDesigned to be lightning-fast and"
},
{
"path": "pom.xml",
"chars": 5806,
"preview": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n<!--\n (c) Copyright 2013-2021 Jonathan Cobb\n This code is available under the "
},
{
"path": "s3s3mirror.bat",
"chars": 139,
"preview": "@echo off\njava -Dlog4j.configuration=file:target/classes/log4j.xml -Ds3s3mirror.version=1.2.8 -jar target/s3s3mirror-1.2"
},
{
"path": "s3s3mirror.sh",
"chars": 589,
"preview": "#!/bin/bash\n\nTHISDIR=$(cd \"$(dirname $0)\" && pwd)\n\nVERSION=1.2.8\nJARFILE=\"${THISDIR}/target/s3s3mirror-${VERSION}-SNAPSH"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/CopyMaster.java",
"chars": 1414,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport com.amazonaws.services.s3.mod"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/DeleteMaster.java",
"chars": 888,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport com.amazonaws.services.s3.mod"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/KeyCopyJob.java",
"chars": 6171,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport com.amazonaws.services.s3.mod"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/KeyDeleteJob.java",
"chars": 4094,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport com.amazonaws.services.s3.mod"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/KeyFingerprint.java",
"chars": 294,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport lombok.*;\n\n@EqualsAndHashCode(callSuper=false) @AllArgsConstructor\npublic clas"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/KeyJob.java",
"chars": 3962,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport com.amazonaws.services.s3.mod"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/KeyLister.java",
"chars": 5943,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport com.amazonaws.services.s3.mod"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/KeyMaster.java",
"chars": 4860,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport com.amazonaws.services.s3.mod"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/MirrorConstants.java",
"chars": 341,
"preview": "package org.cobbzilla.s3s3mirror;\n\npublic class MirrorConstants {\n\n public static final long KB = 1024L;\n public s"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/MirrorContext.java",
"chars": 370,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.model.Owner;\nimport lombok.AllArgsConstructor;\nimpor"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/MirrorMain.java",
"chars": 9486,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.ClientConfiguration;\nimport com.amazonaws.Protocol;\nimport com.a"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/MirrorMaster.java",
"chars": 2789,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport lombok.extern.slf4j.Slf4j;\n\ni"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/MirrorOptions.java",
"chars": 14363,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.auth.AWSCredentials;\n\nimport lombok.Getter;\nimport lombok.Setter"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/MirrorStats.java",
"chars": 3434,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport lombok.Getter;\nimport lombok.extern.slf4j.Slf4j;\n\nimport java.util.concurrent."
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/MultipartKeyCopyJob.java",
"chars": 4316,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport com.amazonaws.services.s3.mod"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/Sleep.java",
"chars": 341,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport lombok.extern.slf4j.Slf4j;\n\n@Slf4j\npublic class Sleep {\n\n public static boo"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/ComparisonStrategy.java",
"chars": 289,
"preview": "package org.cobbzilla.s3s3mirror.comparisonstrategies;\n\nimport com.amazonaws.services.s3.model.ObjectMetadata;\nimport co"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/ComparisonStrategyFactory.java",
"chars": 637,
"preview": "package org.cobbzilla.s3s3mirror.comparisonstrategies;\n\nimport lombok.AccessLevel;\nimport lombok.NoArgsConstructor;\nimpo"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/EtagComparisonStrategy.java",
"chars": 463,
"preview": "package org.cobbzilla.s3s3mirror.comparisonstrategies;\n\nimport com.amazonaws.services.s3.model.ObjectMetadata;\nimport co"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/SizeAndLastModifiedComparisonStrategy.java",
"chars": 491,
"preview": "package org.cobbzilla.s3s3mirror.comparisonstrategies;\n\nimport com.amazonaws.services.s3.model.ObjectMetadata;\nimport co"
},
{
"path": "src/main/java/org/cobbzilla/s3s3mirror/comparisonstrategies/SizeOnlyComparisonStrategy.java",
"chars": 418,
"preview": "package org.cobbzilla.s3s3mirror.comparisonstrategies;\n\nimport com.amazonaws.services.s3.model.ObjectMetadata;\nimport co"
},
{
"path": "src/main/resources/log4j.xml",
"chars": 732,
"preview": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE log4j:configuration PUBLIC \"-//LOGGER\" \"log4j.dtd\">\n\n<log4j:configurati"
},
{
"path": "src/test/java/org/cobbzilla/s3s3mirror/MirrorMainTest.java",
"chars": 5327,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport org.junit.Test;\n\nimport static org.junit.Assert.*;\n\npublic class MirrorMainTes"
},
{
"path": "src/test/java/org/cobbzilla/s3s3mirror/MirrorTest.java",
"chars": 7301,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport com.amazonaws.services.s3.mod"
},
{
"path": "src/test/java/org/cobbzilla/s3s3mirror/S3Asset.java",
"chars": 191,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport lombok.AllArgsConstructor;\nimport lombok.ToString;\n\n@AllArgsConstructor @ToStr"
},
{
"path": "src/test/java/org/cobbzilla/s3s3mirror/SyncStrategiesTest.java",
"chars": 6142,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.model.ObjectMetadata;\nimport com.amazonaws.services."
},
{
"path": "src/test/java/org/cobbzilla/s3s3mirror/TestFile.java",
"chars": 2079,
"preview": "package org.cobbzilla.s3s3mirror;\n\nimport com.amazonaws.services.s3.AmazonS3Client;\nimport lombok.Cleanup;\nimport org.ap"
}
]
About this extraction
This page contains the full source code of the cobbzilla/s3s3mirror GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 33 files (99.6 KB), approximately 23.2k tokens, and a symbol index with 149 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.