[ { "path": ".gitignore", "content": "!.gitignore\n*.iml\ntarget\n.idea\n*.log\n\n*.class\n/.classpath\n/.project\n/.settings\n\ndependency-reduced-pom.xml\n\n" }, { "path": ".travis.yml", "content": "sudo: required\nnotifications:\n email: false\nlanguage: java\njdk:\n- oraclejdk8\nos:\n- linux\nservices:\n- docker\nenv:\n- MONGOLASTIC_FULL=\"ozlerhakan/mongolastic:1.4.4\"\nscript: mvn install\nbefore_install:\n- curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.1.deb && sudo dpkg -i --force-confnew elasticsearch-6.1.1.deb\n- sudo service elasticsearch start\n- sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6\n- echo \"deb http://repo.mongodb.org/apt/ubuntu precise/mongodb-org/3.4 multiverse\" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list\n- sudo apt-get update\n- sudo apt-get install -y mongodb-org\n- sudo service mongod start\n- mongod --version\nbefore_script:\n- wget https://dl.dropbox.com/s/jvngnitppao8hay/tweets.zip\n- unzip tweets.zip\n- sleep 15\n- curl http://localhost:9200/\n- mongorestore -h 127.0.0.1 --port 27017 dump\n- mvn clean\nafter_success:\n- docker build -t $MONGOLASTIC_FULL .\n- docker tag $MONGOLASTIC_FULL ozlerhakan/mongolastic:1.4;\n- docker tag $MONGOLASTIC_FULL ozlerhakan/mongolastic:latest;\n- docker images\n- if [ \"$TRAVIS_BRANCH\" == \"master\" ]; then\n docker login -u=\"$DOCKER_USERNAME\" -p=\"$DOCKER_PASSWORD\";\n docker push $MONGOLASTIC_FULL;\n docker push ozlerhakan/mongolastic:1.4;\n docker push ozlerhakan/mongolastic:latest;\n fi\ndeploy:\n provider: releases\n skip_cleanup: true\n api_key:\n secure: E8iX+y1svn/88ftT5fnMIfECNUNek0T/VtrkdacuFriC/cNcopcZUuIEqmCTuMIvo+iL/h4F4wYzGLadRU6iOhnPk8oMPadcR+2Pll/mz7/I4zcj/iZL8KMJGlJxLpnTKBw3QEGnWflg/S8I6g6bOcGnPhQW0AQZ6bj7FBR59Gd7abtqAkwZVki+zxXbMzMqA3VYRaMc2UEdTU6AlcR7GeCdcDFhQ/4uqmMySGGGKLAROZ+0r9BhbaZ0ivuR7Uza9l68d0FhN2wRxDhP86RM3hbZttGQw+CBw7+3IcHoJ9MpSGgatxWd3eudWypDoocVs9FxYKS3zJcVzzLWtmB1yRdY/zOcLkr7Tkz2z+LIZbjPJFjeDUgpRJPqw8ckPOEbsdmp+/0zVBp+aDdMgpa3hfXkchTa4hXUDWJCZiuN4oByWxnBqoALUAjRdIflCeZuukbmHzZb6QWNTfRSGAZtQHO+sdJIvMFFC9v0k6T2WXvlEWDXt0cA8Gid9iBZLZfemJiBRNRP94TDcakMZzwnBH/G3l1nc7tk8AR7gX0b/biLeloSdxdI7UGV+uLvy+LgrsJp5TeLvBaylCRJ6672IvroGZwZm/GG3NsPS1vb2WoZfbjiS+paWUxreeFumJyg43Vuv0LiDND2y2P8zTtnC7HaM0k54ewt86Kx5w86Ma0=\n file: /home/travis/build/ozlerhakan/mongolastic/target/mongolastic.jar\n on:\n tags: true\n all_branches: true\n repo: ozlerhakan/mongolastic\n\n" }, { "path": "Contributors.adoc", "content": "== Contributors\n\nThank you for our contributors to make this project more usable! ✨\n\n* https://github.com/hakdogan[Hüseyin Hakdoğan]\n* https://github.com/winder[Will Winder]\n* https://github.com/wareninja[Yılmaz Güleryüz]" }, { "path": "Dockerfile", "content": "FROM openjdk:8-jdk-alpine\nMAINTAINER Hakan Ozler \nADD target/mongolastic.jar .\nENTRYPOINT [\"java\",\"-jar\",\"mongolastic.jar\",\"-f\"]" }, { "path": "LICENSE", "content": "The MIT License (MIT)\n\nCopyright (c) 2015-2016 Kodcu.com\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.adoc", "content": "= Mongolastic\n:version: v1.4.3\n\nimage:https://travis-ci.org/ozlerhakan/mongolastic.svg?branch=master[\"Build Status\", link=\"https://travis-ci.org/ozlerhakan/mongolastic\"]\nimage:https://api.codacy.com/project/badge/Grade/8d768c2fc37246019115e4d090a33b98[\"Codacy code quality\", link=\"https://www.codacy.com/app/ozlerhakan/mongolastic?utm_source=github.com&utm_medium=referral&utm_content=ozlerhakan/mongolastic&utm_campaign=Badge_Grade\"]\nimage:https://img.shields.io/docker/pulls/ozlerhakan/mongolastic.svg[\"Docker Pulls,link=\"https://hub.docker.com/r/ozlerhakan/mongolastic\"]\nimage:https://img.shields.io/github/release/ozlerhakan/mongolastic.svg[]\nimage:https://img.shields.io/badge/mongo.java.driver-3.4.2-brightgreen.svg[] image:https://img.shields.io/badge/elastic.java.driver-6.2.4-brightgreen.svg[]\nimage:https://img.shields.io/badge/license-MIT-blue.svg[]\n\nMongolastic enables you to migrate your datasets from a mongod node to an elasticsearch node and vice versa. Since mongo and elastic servers can run with different characteristics, the tool provides several optional and required features to ably connect them. Mongolastic works with a yaml or json configuration file to begin a migration process. It reads your demand on the file and start syncing data in the specified direction.\n\n== How it works\n\nFirst, you can either pull the corresponding image of the app from https://hub.docker.com/r/ozlerhakan/mongolastic/[Docker Hub]\n\nSupported tags and respective Dockerfile links:\n\n* `_1.4_`, `_1.4.4_`, `_latest_` https://github.com/ozlerhakan/mongolastic/blob/master/Dockerfile[_(master/Dockerfile)_]\n* `_1.4.3_` https://github.com/ozlerhakan/mongolastic/blob/0dacd80cbdf7b5b7b282bf6dd89ede8558021577/Dockerfile[_(1.4.2/Dockerfile)_]\n\n\nor download the latest https://github.com/ozlerhakan/mongolastic/releases/download/{version}/mongolastic.jar[mongolastic.jar] file.\n\nSecond, create a yaml or json file which must contain the following structure:\n\n[source,yaml]\n----\nmisc:\n dindex:\n name: <1>\n as: <2>\n ctype:\n name: <3>\n as: <4>\n direction: (em | me) <5>\n batch: <6>\n dropDataset: <7>\nmongo:\n host: <8>\n port: <9>\n query: \"mongo-query\" <10>\n project: \"projection\" <11>\n auth: <12>\n user: \n pwd: \"password\"\n source: \n mechanism: ( plain | scram-sha-1 | x509 | gssapi | cr )\nelastic:\n host: <13>\n port: <14>\n dateFormat: \"\" <15>\n longToString: <16>\n clusterName: <17>\n auth: <18>\n user: \n pwd: \"password\"\n----\n<1> the _database/index name_ to connect to.\n<2> another _database/index name_ in which documents will be located in the target service (*Optional*)\n<3> the _collection/type name_ to export.\n<4> another _collection/type name_ in which indexed/collected documents will reside in the target service (*Optional*)\n<5> direction of the data transfer. the default direction is me (that is, mongo to elasticsearch). You can skip this option if your data move from mongo to es.\n<6> Override the default batch size which is normally 200. (*Optional*)\n<7> configures whether or not the target table should be dropped prior to loading data. Default value is true (*Optional*)\n<8> the name of the host machine where the `mongod` is running.\n<9> the port where the `mongod` instance is listening.\n<10> data will be transferred based on a json mongodb query (*Optional*)\n<11> with 1.4.1, you can manipulate documents that will be migrated from mongo to es based on the https://docs.mongodb.com/manual/reference/operator/aggregation/project/[`$project`] operator (*Optional*)\n<12> as of v1.3.5, you can access an auth mongodb by giving auth configuration. (*Optional*)\n<13> the name of the host machine where the `elastic node` is running.\n<14> the *transport* port where the transport module will communicate with the running elastic node. E.g. *9300* for node-to-node communication.\n<15> a custom formatter for Date fields rather than the default DateCodec (*Optional*)\n<16> serialize long value as a string for backwards compatibility with other tools (*Optional*)\n<17> connect to a spesific elastic cluster (*Optional*)\n<18> as of v1.3.9, you can access an auth elastic search by giving auth configuration. (*Optional*)\n\n---\n\nAlternatively, a JSON file can be specified as a mongolastic configuration file including the same YAML file structure above.\n\n[source,json]\n----\n{\n\t\"misc\": {\n\t\t\"dindex\": {\n\t\t\t\"name\": \"twitter\",\n\t\t\t\"as\": \"media\"\n\t\t},\n\t\t\"ctype\": {\n\t\t\t\"name\": \"tweets\",\n\t\t\t\"as\": \"posts\"\n\t\t},\n\t\t\"direction\": \"me\",\n\t\t\"batch\": 400,\n\t\t\"dropDataset\": true\n\t},\n\t\"mongo\": {\n\t\t\"host\": \"127.0.0.1\",\n\t\t\"port\": 27017,\n\t\t\"query\": \"{ lang: 'en' }\",\n\t\t\"project\": \"{ user:1, name:'$user.name', location: { $substr: [ '$user.location', 10, 15 ] }}\",\n\t\t\"auth\": {\n\t\t\t\"user\": \"joe\",\n\t\t\t\"pwd\": \"1234\",\n\t\t\t\"source\": \"twitter\",\n\t\t\t\"mechanism\": \"scram-sha-1\"\n\t\t}\n\t},\n\t\"elastic\": {\n\t\t\"host\": \"127.0.0.1\",\n\t\t\"port\": 9300,\n\t\t\"dateFormat\": \"yyyy-MM-dd\",\n\t\t\"longToString\": true,\n\t\t\"auth\": {\n\t\t\t\"user\": \"joe\",\n\t\t\t\"pwd\": \"4321\"\n\t\t}\n\t}\n}\n----\n\n== Example #1\n\nThe following files have the same configuration details:\n\n.yaml file\n[source,yaml]\n----\nmisc:\n dindex:\n name: twitter\n as: kodcu\n ctype:\n name: tweets\n as: posts\nmongo:\n host: localhost\n port: 27017\n query: \"{ 'user.name' : 'kodcu.com'}\"\nelastic:\n host: localhost\n port: 9300\n----\n\n.json file\n[source,json]\n----\n{\n\t\"misc\": {\n\t\t\"dindex\": {\n\t\t\t\"name\": \"twitter\",\n\t\t\t\"as\": \"kodcu\"\n\t\t},\n\t\t\"ctype\": {\n\t\t\t\"name\": \"tweets\",\n\t\t\t\"as\": \"posts\"\n\t\t}\n\t},\n\t\"mongo\": {\n\t\t\"host\": \"localhost\",\n\t\t\"port\": 27017,\n\t\t\"query\": \"{ 'user.name' : 'kodcu.com'}\"\n\t},\n\t\"elastic\": {\n\t\t\"host\": \"localhost\",\n\t\t\"port\": 9300\n\t}\n}\n----\n\nthe config says that the transfer direction is from mongodb to elasticsearch, mongolastic first looks at the _tweets_ collection, where the _user name_ is _kodcu.com_, of the _twitter_ database located on a mongod server running on default host interface and port number. If It finds the corresponding data, It will start copying those into an elasticsearch environment running on default host and transport number. After all, you should see a type called _\"posts\"_ in an index called _\"kodcu\"_ in the current elastic node. Why the index and type are different is because \"dindex.as\" and \"ctype.as\" options were set, these indicates that your data being transferred exist in _posts_ type of the _kodcu_ index.\n\nAfter downloading the jar or pulling the image and providing a conf file, you can either run the tool as:\n\n $ java -jar mongolastic.jar -f config.file\n\n__or__\n\n $ docker run --rm -v $(PWD)/config.file:/config.file --net host ozlerhakan/mongolastic: config.file\n\n== Example #2\n\nUsing the project field, you are able to manipulate documents when migrating them from mongodb to elasticsearch. For more examples about the `$project` operator of the aggregation pipeline, take a look at its https://docs.mongodb.com/manual/reference/operator/aggregation/project/[documentation].\n\n[source,yaml]\n----\nmisc:\n dindex:\n name: twitter\n ctype:\n name: tweets\nmongo:\n host: 192.168.10.151\n port: 27017\n project: \"{ user: 1, name: '$user.name', location: { $substr: [ '$user.location', 10, 15 ] }}\" <1>\nelastic:\n host: 192.168.10.152\n port: 9300\n----\n<1> the migrated documents will include the user field and contain new fields `name` and `location`.\n\nNOTE: Every attempt of running the tool drops the mentioned db/index in the target environment unless the dropDataset parameter is configured otherwise.\n\n== License\n\nMongolastic is released under http://showalicense.com/?hide_explanations=false&year=2015&fullname=Kodcu.com#license-mit[MIT].\n" }, { "path": "pom.xml", "content": "\n\n 4.0.0\n\n com.kodcu.mongolastic\n mongolastic\n 1.4.4\n\n \n UTF-8\n 1.8\n 6.2.4\n 3.4.2\n 2.8\n \n\n 2015\n\n mongolastic\n \n A tool that migrates data from mongodb to elasticsearch and vice versa.\n \n http://github.com/ozlerhakan/mongolastic\n\n \n Kodcu.com\n \n\n \n \n ozlerhakan\n Hakan Ozler\n ozler.hakan@gmail.com\n http://github.com/ozlerhakan\n \n developer\n \n \n \n hakdogan\n Hüseyin Akdoğan\n huseyin.akdogan@kodcu.com\n http://github.com/hakdogan\n \n software evangelist - developer\n \n \n \n\n \n 3.0\n \n\n \n github.com\n https://github.com/ozlerhakan/mongolastic/issues\n \n\n \n \n \n elasticsearch-releases\n https://artifacts.elastic.co/maven\n \n true\n \n \n false\n \n \n \n\n \n \n org.mongodb\n mongodb-driver\n ${mongo.version}\n \n \n org.slf4j\n slf4j-log4j12\n \n \n log4j\n log4j\n \n \n \n \n org.elasticsearch.client\n transport\n ${elastic.version}\n \n \n org.slf4j\n slf4j-log4j12\n \n \n log4j\n log4j\n \n \n \n \n org.apache.logging.log4j\n log4j-slf4j-impl\n ${logger.version}\n \n \n org.apache.logging.log4j\n log4j-api\n ${logger.version}\n \n \n org.apache.logging.log4j\n log4j-core\n ${logger.version}\n \n \n com.fasterxml.jackson.core\n jackson-databind\n 2.8.6\n \n \n junit\n junit\n 4.12\n test\n \n \n\n\n \n ${project.artifactId}\n \n \n org.apache.maven.plugins\n maven-shade-plugin\n 3.0.0\n \n \n package\n \n shade\n \n \n \n \n com.kodcu.main.Mongolastic\n \n \n \n \n \n \n \n \n \n \n *:*\n \n META-INF/*.SF\n META-INF/*.DSA\n META-INF/*.RSA\n \n \n \n \n \n \n \n \n org.apache.maven.plugins\n maven-compiler-plugin\n 3.6.1\n \n ${java.version}\n ${java.version}\n \n \n \n \n\n" }, { "path": "src/main/java/com/kodcu/config/ElasticConfiguration.java", "content": "package com.kodcu.config;\n\nimport java.net.InetAddress;\nimport java.net.UnknownHostException;\nimport java.util.Objects;\nimport java.util.Optional;\n\nimport org.apache.logging.log4j.LogManager;\nimport org.apache.logging.log4j.Logger;\nimport org.elasticsearch.client.Client;\nimport org.elasticsearch.common.settings.Settings;\nimport org.elasticsearch.common.settings.Settings.Builder;\nimport org.elasticsearch.common.transport.TransportAddress;\nimport org.elasticsearch.transport.client.PreBuiltTransportClient;\n\n/**\n * Created by hakdogan on 21/05/15.\n */\npublic class ElasticConfiguration {\n\n private final Logger logger = LogManager.getLogger(ElasticConfiguration.class);\n private final YamlConfiguration config;\n private Client client;\n\n public ElasticConfiguration(final YamlConfiguration config) {\n this.config = config;\n this.prepareClient();\n }\n\n private void prepareClient() {\n\n Builder settingsBuilder = applySettings();\n try {\n TransportAddress ista = new TransportAddress(InetAddress.getByName(config.getElastic().getHost()), config.getElastic().getPort());\n client = new PreBuiltTransportClient(settingsBuilder.build())\n .addTransportAddress(ista);\n\n } catch (UnknownHostException ex) {\n logger.error(ex.getMessage(), ex);\n System.exit(-1);\n }\n }\n\n private Builder applySettings() {\n Builder settingsBuilder = Settings.builder();\n\n settingsBuilder.put(\"client.transport.ping_timeout\", \"15s\");\n settingsBuilder.put(\"client.transport.nodes_sampler_interval\", \"5s\");\n // YG: to ensure reliable connection & resolve NoNodeAvailableException\n settingsBuilder.put(\"client.transport.sniff\", true);\n settingsBuilder.put(\"network.bind_host\", 0);\n\n // YG: for supporting ES Auth with ES Shield\n Optional.ofNullable(config.getElastic().getAuth())\n .ifPresent(auth -> settingsBuilder.put(\"xpack.security.user\", String.join(\":\", auth.getUser(), auth.getPwd())));\n\n if (Objects.nonNull(config.getElastic().getClusterName())) {\n settingsBuilder.put(\"cluster.name\", config.getElastic().getClusterName());\n } else {\n settingsBuilder.put(\"client.transport.ignore_cluster_name\", true);\n }\n return settingsBuilder;\n }\n\n public void closeNode() {\n client.close();\n }\n\n public Client getClient() {\n return client;\n }\n\n}\n" }, { "path": "src/main/java/com/kodcu/config/FileConfiguration.java", "content": "package com.kodcu.config;\r\n\r\nimport com.fasterxml.jackson.databind.ObjectMapper;\r\nimport org.slf4j.Logger;\r\nimport org.slf4j.LoggerFactory;\r\nimport org.yaml.snakeyaml.Yaml;\r\n\r\nimport java.io.File;\r\nimport java.io.FileInputStream;\r\nimport java.io.IOException;\r\nimport java.util.Objects;\r\n\r\n/**\r\n * Created by Hakan on 5/19/2015.\r\n */\r\npublic class FileConfiguration {\r\n\r\n private final Logger logger = LoggerFactory.getLogger(FileConfiguration.class);\r\n private final String parameter;\r\n\r\n public FileConfiguration(String parameter) {\r\n this.parameter = parameter;\r\n }\r\n\r\n public YamlConfiguration getFileContent() {\r\n YamlConfiguration config = null;\r\n File ymlFile = new File(parameter);\r\n try {\r\n Yaml yaml = new Yaml();\r\n if (ymlFile.isFile()) {\r\n FileInputStream configFile = new FileInputStream(ymlFile);\r\n config = yaml.loadAs(configFile, YamlConfiguration.class);\r\n } else {\r\n // we expect that this is just a string including yaml format\r\n config = yaml.loadAs(parameter, YamlConfiguration.class);\r\n }\r\n } catch (Exception e) {\r\n try {\r\n ObjectMapper mapper = new ObjectMapper();\r\n config = mapper.readValue(ymlFile, YamlConfiguration.class);\r\n } catch (IOException ex) {\r\n logger.error(e.getMessage(), e);\r\n System.exit(0);\r\n }\r\n }\r\n\r\n logger.info(System.lineSeparator() + \"Config Output:\" + System.lineSeparator() + config.toString() + System.lineSeparator());\r\n config = this.controlAsSettings(config);\r\n return config;\r\n }\r\n\r\n private YamlConfiguration controlAsSettings(YamlConfiguration config) {\r\n String dIndexAs = config.getMisc().getDindex().getAs();\r\n String cTypeAs = config.getMisc().getCtype().getAs();\r\n if (Objects.isNull(dIndexAs))\r\n config.getMisc().getDindex().setAs(config.getMisc().getDindex().getName());\r\n if (Objects.isNull(cTypeAs))\r\n config.getMisc().getCtype().setAs(config.getMisc().getCtype().getName());\r\n if (config.getMisc().getBatch() < 200)\r\n config.getMisc().setBatch(200);\r\n\r\n return config;\r\n }\r\n\r\n}\r\n" }, { "path": "src/main/java/com/kodcu/config/MongoConfiguration.java", "content": "package com.kodcu.config;\r\n\r\nimport com.mongodb.MongoClient;\r\nimport com.mongodb.MongoClientOptions;\r\nimport com.mongodb.MongoCredential;\r\nimport com.mongodb.ReadPreference;\r\nimport com.mongodb.ServerAddress;\r\nimport com.mongodb.client.MongoCollection;\r\nimport com.mongodb.client.MongoDatabase;\r\nimport org.bson.Document;\r\nimport org.slf4j.Logger;\r\nimport org.slf4j.LoggerFactory;\r\n\r\nimport java.util.Arrays;\r\nimport java.util.Objects;\r\n\r\nimport static com.mongodb.assertions.Assertions.notNull;\r\n\r\n/**\r\n * Created by Hakan on 5/19/2015.\r\n */\r\npublic class MongoConfiguration {\r\n\r\n private final Logger logger = LoggerFactory.getLogger(MongoConfiguration.class);\r\n private final YamlConfiguration config;\r\n private MongoClient client;\r\n\r\n public MongoConfiguration(final YamlConfiguration config) {\r\n this.config = config;\r\n this.prepareClient();\r\n }\r\n\r\n private void prepareClient() {\r\n try {\r\n ServerAddress address = new ServerAddress(config.getMongo().getHost(), config.getMongo().getPort());\r\n MongoClientOptions options = MongoClientOptions.builder()\r\n .serverSelectionTimeout(5000)\r\n .socketKeepAlive(false)\r\n .readPreference(ReadPreference.primaryPreferred())\r\n .sslInvalidHostNameAllowed(true)\r\n .build();\r\n\r\n client = connectToClient(address, options);\r\n } catch (Exception ex) {\r\n logger.error(ex.getMessage(), ex);\r\n System.exit(-1);\r\n }\r\n }\r\n\r\n private MongoClient connectToClient(ServerAddress address, MongoClientOptions options) {\r\n if (Objects.nonNull(config.getMongo().getAuth())) {\r\n\r\n String user = notNull(\"auth.name\", config.getMongo().getAuth().getUser());\r\n String database = config.getMongo().getAuth().getSource();\r\n char[] pwd = config.getMongo().getAuth().getPwd().toCharArray();\r\n String mechanism = config.getMongo().getAuth().getMechanism();\r\n\r\n MongoCredential credential = findMongoCredential(user, database, pwd, mechanism);\r\n return new MongoClient(Arrays.asList(address), Arrays.asList(credential), options);\r\n\r\n } else {\r\n return new MongoClient(Arrays.asList(address), options);\r\n }\r\n }\r\n\r\n private MongoCredential findMongoCredential(String user, String database, char[] pwd, String mechanism) {\r\n MongoCredential credential = null;\r\n switch (mechanism) {\r\n case \"scram-sha-1\":\r\n credential = MongoCredential.createScramSha1Credential(user, database, pwd);\r\n break;\r\n case \"x509\":\r\n credential = MongoCredential.createMongoX509Credential(user);\r\n break;\r\n case \"cr\":\r\n credential = MongoCredential.createMongoCRCredential(user, database, pwd);\r\n break;\r\n case \"plain\":\r\n credential = MongoCredential.createPlainCredential(user, database, pwd);\r\n break;\r\n case \"gssapi\":\r\n credential = MongoCredential.createGSSAPICredential(user);\r\n break;\r\n default:\r\n credential = MongoCredential.createCredential(user, database, pwd);\r\n break;\r\n }\r\n return credential;\r\n }\r\n\r\n public MongoCollection getMongoCollection() {\r\n MongoCollection collection = null;\r\n try {\r\n MongoDatabase database = client.getDatabase(config.getMisc().getDindex().getName());\r\n collection = database.getCollection(config.getMisc().getCtype().getName());\r\n } catch (Exception ex) {\r\n logger.error(ex.getMessage(), ex);\r\n }\r\n return collection;\r\n }\r\n\r\n public void closeConnection() {\r\n if (Objects.nonNull(client))\r\n client.close();\r\n }\r\n\r\n public MongoClient getClient() {\r\n return client;\r\n }\r\n}\r\n" }, { "path": "src/main/java/com/kodcu/config/YamlConfiguration.java", "content": "package com.kodcu.config;\n\nimport com.kodcu.config.structure.Elastic;\nimport com.kodcu.config.structure.Misc;\nimport com.kodcu.config.structure.Mongo;\n\n/**\n * Created by Hakan on 5/19/2015.\n */\npublic class YamlConfiguration {\n\n private Misc misc;\n private Mongo mongo;\n private Elastic elastic;\n\n public Elastic getElastic() {\n return elastic;\n }\n\n public void setElastic(Elastic elastic) {\n this.elastic = elastic;\n }\n\n public Misc getMisc() {\n return misc;\n }\n\n public void setMisc(Misc misc) {\n this.misc = misc;\n }\n\n public Mongo getMongo() {\n return mongo;\n }\n\n public void setMongo(Mongo mongo) {\n this.mongo = mongo;\n }\n\n @Override\n public String toString() {\n return \"{\" +\n \"elastic=\" + elastic +\n \", misc=\" + misc +\n \", mongo=\" + mongo +\n '}';\n }\n}" }, { "path": "src/main/java/com/kodcu/config/structure/Auth.java", "content": "package com.kodcu.config.structure;\n\n/**\n * Created by Hakan on 5/14/2016.\n */\npublic class Auth {\n\n private String user;\n private String pwd;\n private String source;\n private String mechanism;\n\n public String getUser() {\n return user;\n }\n\n public void setUser(String user) {\n this.user = user;\n }\n\n public String getPwd() {\n return pwd;\n }\n\n public void setPwd(String pwd) {\n this.pwd = pwd;\n }\n\n public String getSource() {\n return source;\n }\n\n public void setSource(String source) {\n this.source = source;\n }\n\n public String getMechanism() {\n return mechanism;\n }\n\n public void setMechanism(String mechanism) {\n this.mechanism = mechanism;\n }\n\n @Override\n public String toString() {\n return \"Auth [user=\" + user + \", pwd=\" + pwd + \", source=\" + source + \", mechanism=\" + mechanism + \"]\";\n }\n\n}\n" }, { "path": "src/main/java/com/kodcu/config/structure/Elastic.java", "content": "package com.kodcu.config.structure;\n\n/**\n * Created by Hakan on 1/16/2016.\n */\npublic class Elastic {\n\n private String host;\n private int port;\n private String dateFormat;\n private Boolean longToString = false;\n private String clusterName; // optional\n private Auth auth; // optional\n\n public Auth getAuth() {\n return auth;\n }\n\n public void setAuth(Auth auth) {\n this.auth = auth;\n }\n\n public String getClusterName() {\n return clusterName;\n }\n\n public void setClusterName(String clusterName) {\n this.clusterName = clusterName;\n }\n\n public String getHost() {\n return host;\n }\n\n public void setHost(String host) {\n this.host = host;\n }\n\n public int getPort() {\n return port;\n }\n\n public void setPort(int port) {\n this.port = port;\n }\n\n public String getDateFormat() {\n return dateFormat;\n }\n\n public void setDateFormat(String dateFormat) {\n this.dateFormat = dateFormat;\n }\n\n public Boolean getLongToString() {\n return this.longToString;\n }\n\n public void setLongToString(Boolean longToString) {\n this.longToString = longToString;\n }\n\n @Override\n public String toString() {\n return \"Elastic{\" +\n \"host='\" + host + '\\'' +\n \", port=\" + port +\n \", clusterName=\" + clusterName +\n \", dateFormat=\" + dateFormat +\n \", longToString=\" + longToString +\n \", auth=\" + auth +\n '}';\n }\n\n @Override\n public boolean equals(Object o) {\n if (this == o) return true;\n if (o == null || getClass() != o.getClass()) return false;\n\n Elastic elastic = (Elastic) o;\n\n if (port != elastic.port) return false;\n return host != null ? host.equals(elastic.host) : elastic.host == null;\n\n }\n\n @Override\n public int hashCode() {\n int result = host != null ? host.hashCode() : 0;\n result = 31 * result + port;\n return result;\n }\n}\n" }, { "path": "src/main/java/com/kodcu/config/structure/Misc.java", "content": "package com.kodcu.config.structure;\n\n/**\n * Created by Hakan on 1/16/2016.\n */\npublic class Misc {\n\n private String direction = \"me\";\n private Namespace dindex;\n private Namespace ctype;\n private Boolean dropDataset = true;\n private int batch = 200;\n\n public int getBatch() {\n return batch;\n }\n\n public void setBatch(int batch) {\n this.batch = batch;\n }\n\n public Namespace getCtype() {\n return ctype;\n }\n\n public void setCtype(Namespace ctype) {\n this.ctype = ctype;\n }\n\n public String getDirection() {\n return direction;\n }\n\n public void setDirection(String direction) {\n this.direction = direction;\n }\n\n public Namespace getDindex() {\n return dindex;\n }\n\n public void setDindex(Namespace dindex) {\n this.dindex = dindex;\n }\n\n public Boolean getDropDataset() {\n return this.dropDataset;\n }\n\n public void setDropDataset(Boolean dropDataset) {\n this.dropDataset = dropDataset;\n }\n\n @Override\n public String toString() {\n return \"Misc{\" +\n \"batch=\" + batch +\n \", direction='\" + direction + '\\'' +\n \", dindex=\" + dindex +\n \", ctype=\" + ctype +\n \", dropDataset=\" + dropDataset +\n '}';\n }\n}\n" }, { "path": "src/main/java/com/kodcu/config/structure/Mongo.java", "content": "package com.kodcu.config.structure;\n\n/**\n * Created by Hakan on 1/16/2016.\n */\npublic class Mongo {\n\n private String host;\n private int port;\n private String query = \"{}\";\n private String project;\n private Auth auth;\n\n @Override\n public String toString() {\n return \"Mongo{\" +\n \"host='\" + host + '\\'' +\n \", port=\" + port +\n \", query='\" + query + '\\'' +\n \", project='\" + project + '\\'' +\n \", auth=\" + auth +\n '}';\n }\n\n public String getProject() {\n return project;\n }\n\n public void setProject(String project) {\n this.project = project;\n }\n\n public Auth getAuth() {\n return auth;\n }\n\n public void setAuth(Auth auth) {\n this.auth = auth;\n }\n\n public String getQuery() {\n return query;\n }\n\n public void setQuery(String query) {\n this.query = query;\n }\n\n public int getPort() {\n return port;\n }\n\n public void setPort(int port) {\n this.port = port;\n }\n\n public String getHost() {\n return host;\n }\n\n public void setHost(String host) {\n this.host = host;\n }\n}\n" }, { "path": "src/main/java/com/kodcu/config/structure/Namespace.java", "content": "package com.kodcu.config.structure;\n\n/**\n * Created by Hakan on 1/16/2016.\n */\npublic class Namespace {\n private String name;\n private String as;\n\n public String getAs() {\n return as;\n }\n\n public void setAs(String as) {\n this.as = as;\n }\n\n public String getName() {\n return name;\n }\n\n public void setName(String name) {\n this.name = name;\n }\n\n @Override\n public String toString() {\n return \"Namespace{\" +\n \"as='\" + as + '\\'' +\n \", name='\" + name + '\\'' +\n '}';\n }\n}\n" }, { "path": "src/main/java/com/kodcu/listener/BulkProcessorListener.java", "content": "package com.kodcu.listener;\n\nimport org.elasticsearch.action.bulk.BulkProcessor;\nimport org.elasticsearch.action.bulk.BulkRequest;\nimport org.elasticsearch.action.bulk.BulkResponse;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\n/**\n * Created by Hakan on 5/21/2015.\n */\npublic class BulkProcessorListener implements BulkProcessor.Listener {\n\n private final Logger logger = LoggerFactory.getLogger(BulkProcessorListener.class);\n\n @Override\n public void beforeBulk(long executionId, BulkRequest request) {\n }\n\n @Override\n public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {\n if (response.hasFailures()) {\n logger.error(response.buildFailureMessage());\n } else {\n logger.info(String.format(\"Data transfer successfully terminated.(%d)\", response.getItems().length));\n }\n\n }\n\n @Override\n public void afterBulk(long executionId, BulkRequest request, Throwable failure) {\n logger.error(\"Transfer failed.\");\n logger.error(failure.getMessage(), failure.fillInStackTrace());\n }\n}\n" }, { "path": "src/main/java/com/kodcu/main/Mongolastic.java", "content": "package com.kodcu.main;\n\nimport com.kodcu.config.ElasticConfiguration;\nimport com.kodcu.config.FileConfiguration;\nimport com.kodcu.config.MongoConfiguration;\nimport com.kodcu.config.YamlConfiguration;\nimport com.kodcu.provider.ElasticToMongoProvider;\nimport com.kodcu.provider.MongoToElasticProvider;\nimport com.kodcu.provider.Provider;\nimport com.kodcu.service.BulkService;\nimport com.kodcu.service.ElasticBulkService;\nimport com.kodcu.service.MongoBulkService;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.util.Optional;\n\n/**\n * Created by Hakan on 5/19/2015.\n */\npublic class Mongolastic {\n\n private static final Logger logger = LoggerFactory.getLogger(Mongolastic.class);\n private final String parameter;\n\n public Mongolastic(String parameter) {\n this.parameter = parameter;\n }\n\n public static void main(String[] args) throws Exception {\n configAssertion(args);\n\n String parameter = args[1];\n Mongolastic app = new Mongolastic(parameter);\n app.start();\n }\n\n private static void configAssertion(String[] args) {\n if (args.length == 0) {\n logger.error(\"Incorrect syntax. Should be mongolastic.jar -f /path/file\");\n System.exit(0);\n }\n if (!args[0].equals(\"-f\")) {\n logger.error(\"Please specify the -f parameter with a correct yaml or json file\");\n System.exit(0);\n }\n if (args.length != 2) {\n logger.error(\"Incorrect syntax. Pass max 2 parameters\");\n System.exit(0);\n }\n }\n\n public void start() {\n FileConfiguration fConfig = new FileConfiguration(parameter);\n Optional yamlConfig = Optional.ofNullable(fConfig.getFileContent());\n long begin = System.currentTimeMillis();\n try {\n yamlConfig.ifPresent(this::proceedService);\n } finally {\n logger.info(\"Load duration: \" + (System.currentTimeMillis() - begin) + \"ms\");\n }\n }\n\n public void proceedService(YamlConfiguration config) {\n ElasticConfiguration elastic = new ElasticConfiguration(config);\n MongoConfiguration mongo = new MongoConfiguration(config);\n BulkService bulkService = this.initializeBulkService(config, mongo, elastic);\n Provider provider = this.initializeProvider(config, mongo, elastic);\n provider.transfer(bulkService, config, () -> {\n bulkService.close();\n mongo.closeConnection();\n elastic.closeNode();\n });\n }\n\n private Provider initializeProvider(YamlConfiguration config, MongoConfiguration mongo, ElasticConfiguration elastic) {\n if (config.getMisc().getDirection().equals(\"em\")) {\n return new ElasticToMongoProvider(elastic, config);\n }\n return new MongoToElasticProvider(mongo.getMongoCollection(), config);\n }\n\n private BulkService initializeBulkService(YamlConfiguration config, MongoConfiguration mongo, ElasticConfiguration elastic) {\n if (config.getMisc().getDirection().equals(\"em\")) {\n return new MongoBulkService(mongo.getClient(), config);\n }\n return new ElasticBulkService(config, elastic);\n }\n}\n" }, { "path": "src/main/java/com/kodcu/provider/ElasticToMongoProvider.java", "content": "package com.kodcu.provider;\n\nimport com.kodcu.config.ElasticConfiguration;\nimport com.kodcu.config.YamlConfiguration;\nimport org.bson.Document;\nimport org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsRequestBuilder;\nimport org.elasticsearch.action.search.SearchResponse;\nimport org.elasticsearch.action.search.SearchType;\nimport org.elasticsearch.client.IndicesAdminClient;\nimport org.elasticsearch.common.unit.TimeValue;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.util.Arrays;\nimport java.util.List;\nimport java.util.Objects;\nimport java.util.stream.Collectors;\n\n/**\n * Created by Hakan on 6/29/2015.\n */\npublic class ElasticToMongoProvider implements Provider {\n\n private final Logger logger = LoggerFactory.getLogger(ElasticToMongoProvider.class);\n private final ElasticConfiguration elastic;\n private final YamlConfiguration config;\n private SearchResponse response;\n\n public ElasticToMongoProvider(final ElasticConfiguration elastic, final YamlConfiguration config) {\n this.elastic = elastic;\n this.config = config;\n }\n\n @Override\n public long getCount() {\n long count = 0;\n IndicesAdminClient admin = elastic.getClient().admin().indices();\n IndicesExistsRequestBuilder builder = admin.prepareExists(config.getMisc().getDindex().getName());\n if (builder.execute().actionGet().isExists()) {\n SearchResponse countResponse = elastic.getClient().prepareSearch(config.getMisc().getDindex().getName())\n .setTypes(config.getMisc().getCtype().getName())\n .setSearchType(SearchType.QUERY_THEN_FETCH)\n .setSize(0)\n .execute().actionGet();\n count = countResponse.getHits().getTotalHits();\n } else {\n logger.info(\"Index/Type does not exist or does not contain the record\");\n System.exit(-1);\n }\n\n logger.info(\"Elastic Index/Type count: \" + count);\n return count;\n }\n\n @Override\n public List buildJSONContent(int skip, int limit) {\n\n if (Objects.isNull(response)) {\n response = elastic.getClient().prepareSearch(config.getMisc().getDindex().getName())\n .setTypes(config.getMisc().getCtype().getName())\n .setSearchType(SearchType.QUERY_THEN_FETCH)\n .setScroll(new TimeValue(60000))\n .setSize(limit)\n .execute().actionGet();\n }\n else {\n response = elastic.getClient()\n .prepareSearchScroll(response.getScrollId())\n .setScroll(new TimeValue(60000))\n .execute().actionGet();\n }\n\n return Arrays.stream(response.getHits().getHits())\n .map(hit -> new Document(hit.getSourceAsMap()))\n .collect(Collectors.toList());\n }\n}\n" }, { "path": "src/main/java/com/kodcu/provider/MongoToElasticProvider.java", "content": "package com.kodcu.provider;\n\nimport com.kodcu.config.YamlConfiguration;\nimport com.mongodb.client.AggregateIterable;\nimport com.mongodb.client.MongoCollection;\nimport com.mongodb.client.MongoCursor;\nimport org.bson.Document;\nimport org.bson.conversions.Bson;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Optional;\n\nimport static com.mongodb.client.model.Aggregates.*;\n\n/**\n * Created by Hakan on 5/18/2015.\n */\npublic class MongoToElasticProvider implements Provider {\n\n private final Logger logger = LoggerFactory.getLogger(MongoToElasticProvider.class);\n private final MongoCollection collection;\n private final YamlConfiguration config;\n private MongoCursor cursor;\n private long cursorId = 0;\n\n public MongoToElasticProvider(final MongoCollection collection, final YamlConfiguration config) {\n this.collection = collection;\n this.config = config;\n }\n\n @Override\n public long getCount() {\n long count = collection.count(Document.parse(config.getMongo().getQuery()));\n logger.info(\"Mongo collection count: \" + count);\n if (count == 0) {\n logger.error(\"Database/Collection does not exist or does not contain the record\");\n System.exit(-1);\n }\n return count;\n }\n\n @Override\n public List buildJSONContent(int skip, int limit) {\n ArrayList result = new ArrayList<>(limit);\n result.ensureCapacity(limit);\n\n MongoCursor cursor = getCursor(skip);\n while (cursor.hasNext() && result.size() < limit) {\n result.add(cursor.next());\n }\n return result;\n }\n\n /**\n * Get the MongoDB cursor.\n */\n private MongoCursor getCursor(int skip) {\n if (cursor == null && cursorId == 0) {\n Document query = Document.parse(config.getMongo().getQuery());\n List pipes = new ArrayList<>(3);\n pipes.add(match(query));\n pipes.add(skip(skip));\n\n Optional.ofNullable(config.getMongo().getProject()).ifPresent(p -> pipes.add(project(Document.parse(p))));\n\n AggregateIterable aggregate = collection.aggregate(pipes)\n .allowDiskUse(true)\n .useCursor(true);\n\n cursor = aggregate.iterator();\n\n // TODO: Persist cursor ID somewhere to allow restarts.\n Optional.ofNullable(cursor.getServerCursor()).ifPresent(serverCursor -> cursorId = serverCursor.getId());\n } else if (cursor == null && cursorId != 0) {\n // TODO: Lookup cursor ID for resume.\n // Open existing cursor in case of restart??\n }\n\n return cursor;\n }\n}\n" }, { "path": "src/main/java/com/kodcu/provider/Provider.java", "content": "package com.kodcu.provider;\n\n\nimport com.kodcu.config.YamlConfiguration;\nimport com.kodcu.service.BulkService;\nimport org.bson.Document;\n\nimport java.util.List;\n\n/**\n * Created by Hakan on 6/30/2015.\n */\npublic interface Provider {\n\n default void transfer(final BulkService bulkService, final YamlConfiguration config, final Runnable closeConnections) {\n long count = this.getCount();\n final int limit = config.getMisc().getBatch();\n int skip = 0;\n\n if (count != 0 && config.getMisc().getDropDataset())\n bulkService.dropDataSet();\n\n while (count >= limit) {\n List content = this.buildJSONContent(skip, limit);\n bulkService.proceed(content);\n count -= limit;\n skip += limit;\n }\n\n if (count > 0) {\n List content = this.buildJSONContent(skip, (int) count);\n bulkService.proceed(content);\n }\n\n closeConnections.run();\n }\n\n long getCount();\n\n List buildJSONContent(int skip, int limit);\n}\n" }, { "path": "src/main/java/com/kodcu/service/BulkService.java", "content": "package com.kodcu.service;\n\nimport java.util.List;\n\n/**\n * Created by Hakan on 6/30/2015.\n */\npublic interface BulkService {\n\n void proceed(List content);\n\n void dropDataSet();\n\n void close();\n}\n" }, { "path": "src/main/java/com/kodcu/service/ElasticBulkService.java", "content": "package com.kodcu.service;\n\nimport com.kodcu.config.ElasticConfiguration;\nimport com.kodcu.config.YamlConfiguration;\nimport com.kodcu.listener.BulkProcessorListener;\nimport com.kodcu.util.codecs.CustomDateCodec;\nimport com.kodcu.util.codecs.CustomLongCodec;\nimport com.mongodb.MongoClient;\nimport org.bson.Document;\nimport org.bson.codecs.BsonTypeClassMap;\nimport org.bson.codecs.Codec;\nimport org.bson.codecs.DocumentCodec;\nimport org.bson.codecs.Encoder;\nimport org.bson.codecs.configuration.CodecRegistries;\nimport org.bson.codecs.configuration.CodecRegistry;\nimport org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;\nimport org.elasticsearch.action.admin.indices.delete.DeleteIndexResponse;\nimport org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsRequestBuilder;\nimport org.elasticsearch.action.bulk.BulkProcessor;\nimport org.elasticsearch.action.index.IndexRequest;\nimport org.elasticsearch.client.IndicesAdminClient;\nimport org.elasticsearch.common.unit.ByteSizeUnit;\nimport org.elasticsearch.common.unit.ByteSizeValue;\nimport org.elasticsearch.common.unit.TimeValue;\nimport org.elasticsearch.common.xcontent.XContentType;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.concurrent.TimeUnit;\n\n/**\n * Created by Hakan on 5/21/2015.\n */\npublic class ElasticBulkService implements BulkService {\n\n private final Logger logger = LoggerFactory.getLogger(ElasticBulkService.class);\n private final YamlConfiguration config;\n private final ElasticConfiguration client;\n private final BulkProcessor bulkProcessor;\n private final Encoder encoder;\n\n public ElasticBulkService(final YamlConfiguration config, final ElasticConfiguration client) {\n this.config = config;\n this.client = client;\n\n this.bulkProcessor = BulkProcessor.builder(client.getClient(), new BulkProcessorListener())\n .setBulkActions(config.getMisc().getBatch())\n .setFlushInterval(TimeValue.timeValueSeconds(5))\n .setBulkSize(new ByteSizeValue(1, ByteSizeUnit.GB))\n .build();\n\n encoder = getEncoder();\n }\n\n @Override\n public void proceed(List content) {\n try {\n logger.info(\"Transferring data began to elasticsearch.\");\n final String indexName = config.getMisc().getDindex().getAs();\n final String typeName = config.getMisc().getCtype().getAs();\n for (Object o : content) {\n Document doc = (Document) o;\n Object id = doc.get(\"_id\");\n IndexRequest indexRequest = new IndexRequest(indexName, typeName, String.valueOf(id));\n doc.remove(\"_id\");\n indexRequest.source(doc.toJson(encoder), XContentType.JSON);\n bulkProcessor.add(indexRequest);\n }\n } catch (Exception ex) {\n logger.debug(ex.getMessage(), ex);\n }\n }\n\n @Override\n public void close() {\n try {\n bulkProcessor.awaitClose(10, TimeUnit.MINUTES);\n } catch (InterruptedException ex) {\n logger.error(ex.getMessage(), ex);\n }\n }\n\n @Override\n public void dropDataSet() {\n final String indexName = config.getMisc().getDindex().getAs();\n IndicesAdminClient admin = client.getClient().admin().indices();\n IndicesExistsRequestBuilder builder = admin.prepareExists(indexName);\n if (builder.execute().actionGet().isExists()) {\n DeleteIndexResponse delete = admin.delete(new DeleteIndexRequest(indexName)).actionGet();\n if (delete.isAcknowledged())\n logger.info(String.format(\"The current index %s was deleted.\", indexName));\n else\n logger.info(String.format(\"The current index %s was not deleted.\", indexName));\n }\n }\n\n /**\n * Customizations for the document.toJson output.\n *

\n * http://mongodb.github.io/mongo-java-driver/3.0/bson/codecs/\n *\n * @return the toJson encoder.\n */\n private Encoder getEncoder() {\n ArrayList> codecs = new ArrayList<>();\n\n if (config.getElastic().getDateFormat() != null) {\n // Replace default DateCodec class to use the custom date formatter.\n codecs.add(new CustomDateCodec(config.getElastic().getDateFormat()));\n }\n\n if (config.getElastic().getLongToString()) {\n // Replace default LongCodec class\n codecs.add(new CustomLongCodec());\n }\n\n if (codecs.size() > 0) {\n BsonTypeClassMap bsonTypeClassMap = new BsonTypeClassMap();\n\n CodecRegistry codecRegistry = CodecRegistries.fromRegistries(\n CodecRegistries.fromCodecs(codecs),\n MongoClient.getDefaultCodecRegistry());\n\n return new DocumentCodec(codecRegistry, bsonTypeClassMap);\n } else {\n return new DocumentCodec();\n }\n }\n}\n" }, { "path": "src/main/java/com/kodcu/service/MongoBulkService.java", "content": "package com.kodcu.service;\n\nimport com.kodcu.config.YamlConfiguration;\nimport com.mongodb.MongoClient;\nimport com.mongodb.client.MongoCollection;\nimport org.bson.Document;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.util.List;\n\n/**\n * Created by Hakan on 6/29/2015.\n */\npublic class MongoBulkService implements BulkService {\n\n private final Logger logger = LoggerFactory.getLogger(MongoBulkService.class);\n private final MongoCollection collection;\n\n public MongoBulkService(final MongoClient client, final YamlConfiguration config) {\n this.collection = client.getDatabase(config.getMisc().getDindex().getAs()).getCollection(config.getMisc().getCtype().getAs());\n }\n\n @Override\n public void proceed(List content) {\n try {\n logger.info(\"Transferring data began to mongodb.\");\n collection.insertMany((List) content);\n } catch (Exception ex) {\n logger.debug(ex.getMessage(), ex);\n }\n }\n\n @Override\n public void dropDataSet() {\n if (collection.count() != 0) {\n String collectionName = collection.getNamespace().getCollectionName();\n collection.drop();\n logger.info(String.format(\"The current collection called %s was deleted.\", collectionName));\n }\n }\n\n @Override\n public void close() {\n //no-op\n }\n\n}\n" }, { "path": "src/main/java/com/kodcu/util/codecs/CustomDateCodec.java", "content": "package com.kodcu.util.codecs;\n\nimport org.bson.BsonWriter;\nimport org.bson.codecs.DateCodec;\nimport org.bson.codecs.EncoderContext;\n\nimport java.text.SimpleDateFormat;\nimport java.util.Date;\n\n/**\n * @author wwinder\n * Created on: 5/27/16\n */\npublic class CustomDateCodec extends DateCodec {\n private final SimpleDateFormat formatter;\n\n public CustomDateCodec(String format) {\n formatter = new SimpleDateFormat(format);\n }\n\n @Override\n public void encode(final BsonWriter writer, final Date value, final EncoderContext encoderContext) {\n writer.writeString(formatter.format(value));\n }\n}\n" }, { "path": "src/main/java/com/kodcu/util/codecs/CustomLongCodec.java", "content": "package com.kodcu.util.codecs;\n\nimport org.bson.BsonWriter;\nimport org.bson.codecs.EncoderContext;\nimport org.bson.codecs.LongCodec;\n\n/**\n * @author wwinder\n * Created on: 5/27/16\n */\npublic class CustomLongCodec extends LongCodec {\n @Override\n public void encode(final BsonWriter writer, final Long value, final EncoderContext encoderContext) {\n writer.writeString(value.toString());\n }\n}\n" }, { "path": "src/main/resources/log4j2.properties", "content": "name=PropertiesConfig\nproperty.filename = mongolastic\nappenders = console, file\n\nappender.console.type = Console\nappender.console.name = STDOUT\nappender.console.layout.type = PatternLayout\nappender.console.layout.pattern = [%-5level] [%d{yyyy-MM-dd HH:mm:ss}] [%t] [%p]: - %msg%n\nappender.file.type = File\nappender.file.name = LOGFILE\nappender.file.fileName=mongolastic.log\nappender.file.layout.type=PatternLayout\nappender.file.layout.pattern=[%-5level] [%d{yyyy-MM-dd HH:mm:ss}] [%t] [%p]: - %msg%n\n\nloggers=file\nlogger.file.name=guru.springframework.blog.log4j2properties\nlogger.file.level = debug\nlogger.file.appenderRefs = file\nlogger.file.appenderRef.file.ref = LOGFILE\n\nrootLogger.level = info\nrootLogger.appenderRefs = stdout\nrootLogger.appenderRef.stdout.ref = STDOUT" }, { "path": "src/test/java/com/kodcu/test/TestMongoToElastic.java", "content": "package com.kodcu.test;\n\nimport com.kodcu.config.ElasticConfiguration;\nimport com.kodcu.config.FileConfiguration;\nimport com.kodcu.config.MongoConfiguration;\nimport com.kodcu.config.YamlConfiguration;\nimport com.kodcu.provider.ElasticToMongoProvider;\nimport com.kodcu.provider.MongoToElasticProvider;\nimport com.kodcu.provider.Provider;\nimport com.kodcu.service.BulkService;\nimport com.kodcu.service.ElasticBulkService;\nimport com.kodcu.service.MongoBulkService;\nimport org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsRequestBuilder;\nimport org.elasticsearch.action.admin.indices.flush.FlushRequest;\nimport org.elasticsearch.action.search.SearchResponse;\nimport org.elasticsearch.action.search.SearchType;\nimport org.elasticsearch.client.IndicesAdminClient;\nimport org.junit.Test;\nimport org.junit.runner.RunWith;\nimport org.junit.runners.Parameterized;\n\nimport java.util.Arrays;\nimport java.util.Objects;\n\nimport static org.hamcrest.CoreMatchers.*;\nimport static org.hamcrest.core.Is.is;\nimport static org.junit.Assert.assertThat;\n\n/**\n * Created by Hakan on 9/8/2015.\n */\n@RunWith(Parameterized.class)\npublic class TestMongoToElastic {\n\n private final FileConfiguration file;\n\n public TestMongoToElastic(final FileConfiguration file) {\n super();\n this.file = file;\n }\n\n @Parameterized.Parameters(name = \"{index}: ({0})={1}\")\n public static Iterable queries() throws Exception {\n String query = \"misc:\\n\" +\n \" dindex:\\n\" +\n \" name: twitter\\n\" +\n \" ctype:\\n\" +\n \" name: tweets\\n\" +\n \" batch: 500\\n\" +\n \"mongo:\\n\" +\n \" host: mongo\\n\" +\n \" port: 27017\\n\" +\n \"elastic:\\n\" +\n \" host: es\\n\" +\n \" port: 9300\";\n return Arrays.asList(new Object[][]{\n {new FileConfiguration(\"src/test/resources/conf1\")},\n {new FileConfiguration(\"src/test/resources/conf3\")},\n {new FileConfiguration(query)}\n });\n }\n\n @Test\n public void shouldCopyOneQueryToEsFromMongoDB() {\n YamlConfiguration config = file.getFileContent();\n assertThat(config, is(notNullValue()));\n\n if (Objects.isNull(config.getMongo().getAuth()))\n if (Objects.nonNull(System.getenv(\"MONGO_IP\")))\n config.getMongo().setHost(System.getenv(\"MONGO_IP\"));\n else\n config.getMongo().setHost(\"localhost\");\n else {\n if (Objects.nonNull(System.getenv(\"MONGO_AUTH_IP\")))\n config.getMongo().setHost(System.getenv(\"MONGO_AUTH_IP\"));\n else\n return;\n }\n\n if (Objects.isNull(System.getenv(\"ES_IP\")))\n config.getElastic().setHost(\"localhost\");\n else\n config.getElastic().setHost(System.getenv(\"ES_IP\"));\n\n\n ElasticConfiguration elastic = new ElasticConfiguration(config);\n MongoConfiguration mongo = new MongoConfiguration(config);\n\n BulkService bulkService = this.initializeBulkService(config, mongo, elastic);\n assertThat(bulkService, is(instanceOf(ElasticBulkService.class)));\n\n Provider provider = this.initializeProvider(config, mongo, elastic);\n assertThat(provider, is(instanceOf(MongoToElasticProvider.class)));\n\n provider.transfer(bulkService, config, () -> {\n bulkService.close();\n assertThat(provider.getCount(), equalTo(this.getCount(elastic, config)));\n elastic.closeNode();\n mongo.closeConnection();\n });\n }\n\n private Provider initializeProvider(YamlConfiguration config, MongoConfiguration mongo, ElasticConfiguration elastic) {\n if (config.getMisc().getDirection().equals(\"em\")) {\n return new ElasticToMongoProvider(elastic, config);\n }\n return new MongoToElasticProvider(mongo.getMongoCollection(), config);\n }\n\n private BulkService initializeBulkService(YamlConfiguration config, MongoConfiguration mongo, ElasticConfiguration elastic) {\n if (config.getMisc().getDirection().equals(\"em\")) {\n return new MongoBulkService(mongo.getClient(), config);\n }\n return new ElasticBulkService(config, elastic);\n }\n\n public long getCount(ElasticConfiguration elastic, YamlConfiguration config) {\n IndicesAdminClient admin = elastic.getClient().admin().indices();\n IndicesExistsRequestBuilder builder = admin.prepareExists(config.getMisc().getDindex().getAs());\n assertThat(builder.execute().actionGet().isExists(), is(true));\n\n elastic.getClient().admin().indices().flush(new FlushRequest(config.getMisc().getDindex().getAs())).actionGet();\n\n SearchResponse response = elastic.getClient().prepareSearch(config.getMisc().getDindex().getAs())\n .setTypes(config.getMisc().getCtype().getAs())\n .setSearchType(SearchType.QUERY_THEN_FETCH)\n .setSize(0)\n .execute().actionGet();\n long count = response.getHits().getTotalHits();\n return count;\n }\n}\n" }, { "path": "src/test/java/com/kodcu/test/TestMongolasticQueries.java", "content": "package com.kodcu.test;\n\nimport com.kodcu.config.FileConfiguration;\nimport com.kodcu.config.YamlConfiguration;\nimport com.kodcu.config.structure.Elastic;\nimport com.kodcu.config.structure.Misc;\nimport com.kodcu.config.structure.Mongo;\nimport com.kodcu.config.structure.Namespace;\nimport org.junit.Test;\nimport org.junit.runner.RunWith;\nimport org.junit.runners.Parameterized;\n\nimport java.util.Arrays;\n\nimport static org.hamcrest.CoreMatchers.is;\nimport static org.hamcrest.CoreMatchers.notNullValue;\nimport static org.junit.Assert.assertThat;\n\n/**\n * Created by Hakan on 8/24/2015.\n */\n@RunWith(Parameterized.class)\npublic class TestMongolasticQueries {\n\n private final FileConfiguration config;\n private final YamlConfiguration expected;\n\n public TestMongolasticQueries(final FileConfiguration config, final YamlConfiguration expected) {\n super();\n this.config = config;\n this.expected = expected;\n }\n\n @Parameterized.Parameters(name = \"{index}: ({0})={1}\")\n public static Iterable queries() throws Exception {\n return Arrays.asList(new Object[][]{\n {new FileConfiguration(\"src/test/resources/conf1\"), createQueryConfiguration1()},\n {new FileConfiguration(\"src/test/resources/conf2\"), createQueryConfiguration2()}\n });\n }\n\n private static YamlConfiguration createQueryConfiguration1() {\n YamlConfiguration query = new YamlConfiguration();\n\n Misc misc = new Misc();\n Namespace db = new Namespace();\n db.setName(\"twitter\");\n db.setAs(\"kodcu\");\n misc.setDindex(db);\n Namespace c = new Namespace();\n c.setName(\"tweets\");\n c.setAs(\"tweets\");\n misc.setCtype(c);\n misc.setBatch(300);\n query.setMisc(misc);\n\n Mongo mongod = new Mongo();\n mongod.setHost(\"mongo\");\n mongod.setPort(27017);\n mongod.setQuery(\"{}\");\n query.setMongo(mongod);\n\n Elastic es = new Elastic();\n es.setHost(\"es\");\n es.setPort(9300);\n query.setElastic(es);\n return query;\n }\n\n private static YamlConfiguration createQueryConfiguration2() {\n YamlConfiguration query = new YamlConfiguration();\n\n Misc misc = new Misc();\n Namespace db = new Namespace();\n db.setName(\"twitter\");\n db.setAs(\"twitter\");\n misc.setDindex(db);\n Namespace c = new Namespace();\n c.setName(\"tweets\");\n c.setAs(\"posts\");\n misc.setCtype(c);\n misc.setDirection(\"em\");\n misc.setBatch(200);\n query.setMisc(misc);\n\n Mongo mongod = new Mongo();\n mongod.setHost(\"127.0.0.1\");\n mongod.setPort(27017);\n query.setMongo(mongod);\n\n Elastic es = new Elastic();\n es.setHost(\"127.0.0.1\");\n es.setPort(9300);\n query.setElastic(es);\n return query;\n }\n\n @Test\n public void shouldProceedQueries() {\n YamlConfiguration actual = config.getFileContent();\n assertThat(actual, notNullValue());\n assertThat(actual.getMongo().getQuery(), is(expected.getMongo().getQuery()));\n assertThat(actual.getMongo().getHost(), is(expected.getMongo().getHost()));\n assertThat(actual.getMongo().getPort(), is(expected.getMongo().getPort()));\n assertThat(actual.getElastic().getHost(), is(expected.getElastic().getHost()));\n assertThat(actual.getElastic().getPort(), is(expected.getElastic().getPort()));\n assertThat(actual.getMisc().getDindex().getName(), is(expected.getMisc().getDindex().getName()));\n assertThat(actual.getMisc().getDindex().getAs(), is(expected.getMisc().getDindex().getAs()));\n assertThat(actual.getMisc().getCtype().getName(), is(expected.getMisc().getCtype().getName()));\n assertThat(actual.getMisc().getCtype().getAs(), is(expected.getMisc().getCtype().getAs()));\n }\n}\n" }, { "path": "src/test/resources/conf1", "content": "misc:\n dindex:\n name: twitter\n as: kodcu\n ctype:\n name: tweets\n as: tweets\n batch: 300\n# direction: me\nmongo:\n host: mongo\n port: 27017\nelastic:\n host: es\n port: 9300" }, { "path": "src/test/resources/conf2", "content": "{\n\t\"misc\": {\n\t\t\"dindex\": {\n\t\t\t\"name\": \"twitter\"\n\t\t},\n\t\t\"ctype\": {\n\t\t\t\"name\": \"tweets\",\n\t\t\t\"as\": \"posts\"\n\t\t},\n\t\t\"direction\": \"em\"\n\t},\n\t\"mongo\": {\n\t\t\"host\": \"127.0.0.1\",\n\t\t\"port\": 27017\n\t},\n\t\"elastic\": {\n\t\t\"host\": \"127.0.0.1\",\n\t\t\"port\": 9300\n\t}\n}" }, { "path": "src/test/resources/conf3", "content": "misc:\n dindex:\n name: twitter\n as: twt\n ctype:\n name: tweets\n as: posts\n batch: 500\n# direction: me\nmongo:\n host: mongo_auth\n port: 27017\n auth:\n user: hakan\n pwd: \"1234\"\n source: admin\n mechanism: scram-sha-1\nelastic:\n host: es\n port: 9300" } ]