[ { "path": "LICENSE", "content": "pg_nosql_benchmark\n\nCopyright (c) 2013-2014, EnterpriseDB Corporation\n\nRedistribution and use in source and binary forms, with or without modification,\nare permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this\nlist of conditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice,\nthis list of conditions and the following disclaimer in the documentation and/or\nother materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR\nANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\nLOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON\nANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n" }, { "path": "README.md", "content": "pg_nosql_benchmark\n==================\n\nThis is tool for benchmarking Postgres (JSONB) and MongoDB (BSON)\n\nIntroduction\n-------------\n\nThis is a benchmarking tool developed by EnterpriseDB to benchmark MongoDB 2.6 (BSON) and Postgres 9.4 (JSONB) database using JSON data. The current version focuses on data ingestion and simple select operations in single-instance environments - later versions will include a complete range of workloads (including deleting, updating, appending, and complex select operations) and they will also evaluate multi-server configurations.\n\nThis tool performs the following tasks to compare of MongoDB and PostgreSQL:\n* The tool generates a large set of JSON documents (the number of documents is defined by the value of the variable json_rows in pg_nosql_benchmark)\n* The data set is loaded into MongoDB and PostgreSQL using mongoimport and PostgreSQL's COPY command.\n* The same data is loaded into MongoDB and PostgreSQL using the INSERT command.\n* The tool executes 4 SELECT Queries in MongoDB & PostgreSQL.\n\nRequirements\n------------\n\n* pg_nosql_benchmark uses CentOS 6.4 or later, and is designed for PostgreSQL 9.4 beta server and MongoDB 2.6.\n* The configuration requires three servers\n\t* Load generating server\n\t* MongoDB server\n\t* PostgreSQL server\n* The MongoDB server and the PostgreSQL server should be configured identically\n* The script is designed to run from the central load-generating server, which must have access to the MongoDB and PostgreSQL servers.\n* The following environment variables in pg_nosql_benchmark control the execution:\n\n PostgreSQL Variables:\n```\n PGHOME=/usr/pgsql-9.4 # Installation location of PostgreSQL binaries.\n PGHOST=\"172.17.0.2\" # Hostname/IP address of PostgreSQL\n PGPORT=\"5432\" # Port number on which PostgreSQL is running.\n PGUSER=\"postgres\" # PostgreSQL database username.\n PGPASSWORD=\"postgres\" # PostgreSQL database users password.\n PGBIN=/usr/pgsql-9.4/bin # PostgreSQL binary location.\n```\n\n MongoDB Variables:\n\n```\n MONGO=\"/usr/bin/mongo\" # Complete path of mongo Command binary\n MONGOIMPORT=\"/usr/bin/mongoimport\" # complete path of mongoimport binary\n MONGOHOST=\"172.17.0.3\" # Hostname/IP address of MongoDB\n MONGOPORT=\"27017\" # Port number on which MongoDB is running.\n MONGOUSER=\"mongo\" # Mongo database username\n MONGOPASSWORD=\"mongo\" # MongoDB database username's password\n MONGODBNAME=\"benchmark\" # mongoDB database name.\n```\n\n* To create the admin user in MongoDB use the following command on the MongoDB server:\n```\n > db.createUser({ user: \"mongo\",\n pwd: \"mongo\",\n roles:[{ role: \"userAdmin\",\n db: \"benchmark\"\n }]\n })\n```\n\n* To create the super user in PostgreSQL use the following command:\n```\nCREATE USER postgres PASSWORD '' WITH SUPERUSER;\n```\n\nFor more information on CREATE USER command in PostgreSQL, please check:\n http://www.postgresql.org/docs/9.4/static/sql-createuser.html\n\nRecommended modules\n--------------------\n The following packages are needed to run the benchmark tool:\n 1. mongodb-org-2.6.3-1.x86_64\n 2. postgresql94-9.4beta1-1PGDG.rhel6.x86_64\n 3. bc-1.06.95-1.el6.x86_64\n 4. git-1.7.1-3.el6_4.1.x86_64\n\nInstallation\n------------\n\nTo install this tool on the load generating server, use the following command:\n\n1. git clone https://github.com/EnterpriseDB/pg_nosql_benchmark.git\n2. cd pg_mongo_benchmark\n3. chmod +x pg_nosql_benchmark\n" }, { "path": "lib/common_func_lib.sh", "content": "#!/bin/bash\n\n################################################################################\n# Copyright EnterpriseDB Cooperation\n# All rights reserved.\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions are\n# met:\n# * Redistributions of source code must retain the above copyright\n# notice, this list of conditions and the following disclaimer.\n# * Redistributions in binary form must reproduce the above copyright\n# notice, this list of conditions and the following disclaimer in\n# the documentation and/or other materials provided with the\n# distribution.\n# * Neither the name of PostgreSQL nor the names of its contributors\n# may be used to endorse or promote products derived from this\n# software without specific prior written permission.\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n# \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS\n# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE\n# COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,\n# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,\n# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN\n# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n# POSSIBILITY OF SUCH DAMAGE.\n#\n# Author: Vibhor Kumar\n# E-mail ID: vibhor.aim@gmail.com\n################################################################################\n# quit on any error\nset -e\n# verify any undefined shell variables\nset -u\n\n################################################################################\n# function: print messages with process id\n################################################################################\nfunction process_log()\n{\n echo \"PID: $$ [RUNTIME: $(date +'%m-%d-%y %H:%M:%S')] ${BASENAME}: $*\" >&2\n}\n\n################################################################################\n# function: exit_on_error\n################################################################################\nfunction exit_on_error()\n{\n\n process_log \"ERROR: $*\"\n exit 1\n }\n\n\n\n################################################################################\n# function: get_timestamp_in_nanoseconds\n################################################################################\nfunction get_timestamp_nano ()\n{\n echo $(date +\"%F %T.%N\")\n}\n\n\n################################################################################\n# function: get_timestamp_diff_nanoseconds\n################################################################################\nfunction get_timestamp_diff_nano ()\n{\n typeset -r F_TIMESTAMP1=\"$1\"\n typeset -r F_TIMESTAMP2=\"$2\"\n local SECONDS_DIFF\n local NANOSECONDS_DIFF\n local SECONDS_NANO\n\n SECONDS_DIFF=$(echo $(date -d \"${F_TIMESTAMP1}\" +%s) \\\n - $(date -d \"${F_TIMESTAMP2}\" +%s)|bc)\n NANOSECONDS_DIFF=$(echo $(date -d \"${F_TIMESTAMP1}\" +%N) \\\n - $(date -d \"${F_TIMESTAMP2}\" +%N)|bc)\n SECONDS_NANO=$(echo ${SECONDS_DIFF} \\* 1000000000|bc)\n printf \"%d\\n\" $(((${SECONDS_NANO} + ${NANOSECONDS_DIFF})))\n}\n\n################################################################################\n# function: json_seed_data\n################################################################################\nfunction json_seed_data ()\n{\n\n local INDX=\"$1\"\n local SEED_DATA\n\n if [[ ${INDX} -eq 0 ]]\n then\n INDX=1\n fi\n SEED_DATA=\"{ \\\"name\\\" : \\\"AC3$((${RANDOM}/$INDX + $INDX )) Phone\\\", \\\"brand\\\" : \\\"ACME$((${RANDOM}/$INDX + $INDX ))\\\", \\\"type\\\" : \\\"phone\\\", \\\"price\\\" : 200, \\\"warranty_years\\\" : 1, \\\"available\\\" : true, \\\"description\\\": \\\"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.\\\" }\n{ \\\"name\\\" : \\\"AC7$((${RANDOM}/$INDX + $INDX )) Phone\\\", \\\"brand\\\" : \\\"ACME\\\", \\\"type\\\" : \\\"phone\\\", \\\"price\\\" : 320, \\\"warranty_years\\\" : 1, \\\"available\\\" : false, \\\"description\\\":\\\"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.\\\" }\n{ \\\"name\\\" : \\\"AC3$((${RANDOM}/$INDX + $INDX )) Series Charger\\\", \\\"type\\\" : [ \\\"accessory\\\", \\\"charger\\\" ], \\\"price\\\" : 19, \\\"warranty_years\\\" : 0.25, \\\"for\\\" : [ \\\"ac3\\\", \\\"ac7\\\", \\\"ac9\\\" ], \\\"description\\\": \\\"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.\\\" }\n{ \\\"name\\\" : \\\"AC3$((${RANDOM}/$INDX + $INDX )) Case Green\\\", \\\"type\\\" : [ \\\"accessory\\\", \\\"case\\\" ], \\\"color\\\" : \\\"green\\\", \\\"price\\\" : 12, \\\"warranty_years\\\" : 0, \\\"description\\\": \\\"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.\\\" }\n{ \\\"name\\\" : \\\"Phone Extended Warranty\\\", \\\"type\\\" : \\\"warranty\\\", \\\"price\\\" : 38, \\\"warranty_years\\\" : 2, \\\"for\\\" : [ \\\"ac3\\\", \\\"ac7\\\", \\\"ac9\\\", \\\"qp7\\\", \\\"qp8\\\", \\\"qp9\\\" ], \\\"description\\\": \\\"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.\\\" }\n{ \\\"name\\\" : \\\"AC3$((${RANDOM}/$INDX + $INDX )) Case Black\\\", \\\"type\\\" : [ \\\"accessory\\\", \\\"case\\\" ], \\\"color\\\" : \\\"black\\\", \\\"price\\\" : 12.5, \\\"warranty_years\\\" : 0.25, \\\"available\\\" : false, \\\"for\\\" : \\\"ac3\\\", \\\"description\\\":\\\"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.\\\" }\n{ \\\"name\\\" : \\\"AC3 Case Red\\\", \\\"type\\\" : [ \\\"accessory\\\", \\\"case\\\" ], \\\"color\\\" : \\\"red\\\", \\\"price\\\" : 12, \\\"warranty_years\\\" : 0.25, \\\"available\\\" : true, \\\"for\\\" : \\\"ac3\\\" }\n{ \\\"name\\\" : \\\"Phone Service Basic Plan$((${RANDOM}/$INDX + $INDX ))\\\", \\\"type\\\" : \\\"service$((${RANDOM}/$INDX + $INDX ))\\\", \\\"monthly_price\\\" : 40, \\\"limits\\\" : { \\\"voice\\\" : { \\\"units\\\" : \\\"minutes\\\", \\\"n\\\" : 400, \\\"over_rate\\\" : 0.05 }, \\\"data\\\" : { \\\"units\\\" : \\\"gigabytes\\\", \\\"n\\\" : 20, \\\"over_rate\\\" : 1 }, \\\"sms\\\" : { \\\"units\\\" : \\\"texts sent\\\", \\\"n\\\" : 100, \\\"over_rate\\\" : 0.001 } }, \\\"term_years\\\" : 2 }\n{ \\\"name\\\" : \\\"Phone Service Core Plan\\\", \\\"type\\\" : \\\"service$((${RANDOM}/$INDX + $INDX ))\\\", \\\"monthly_price\\\" : 60, \\\"limits\\\" : { \\\"voice\\\" : { \\\"units\\\" : \\\"minutes\\\", \\\"n\\\" : 1000, \\\"over_rate\\\" : 0.05 }, \\\"data\\\" : { \\\"n\\\" : \\\"unlimited\\\", \\\"over_rate\\\" : 0 }, \\\"sms\\\" : { \\\"n\\\" : \\\"unlimited\\\", \\\"over_rate\\\" : 0 } }, \\\"term_years\\\" : 1, \\\"description\\\": \\\"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.\\\"}\n{ \\\"name\\\" : \\\"Phone Service Family Plan\\\", \\\"type\\\" : \\\"service\\\", \\\"monthly_price\\\" : 90, \\\"limits\\\" : { \\\"voice\\\" : { \\\"units\\\" : \\\"minutes\\\", \\\"n\\\" : 1200, \\\"over_rate\\\" : 0.05 }, \\\"data\\\" : { \\\"n\\\" : \\\"unlimited\\\", \\\"over_rate\\\" : 0 }, \\\"sms\\\" : { \\\"n\\\" : \\\"unlimited\\\", \\\"over_rate\\\" : 0 } }, \\\"sales_tax\\\" : true, \\\"term_years\\\" : 2 }\n{ \\\"name\\\" : \\\"Cable TV Basic Service Package\\\", \\\"type\\\" : \\\"tv\\\", \\\"monthly_price\\\" : 50, \\\"term_years\\\" : 2, \\\"cancel_penalty\\\" : 25, \\\"sales_tax\\\" : true, \\\"additional_tarriffs\\\" : [ { \\\"kind\\\" : \\\"federal tarriff\\\", \\\"amount\\\" : { \\\"percent_of_service\\\" : 0.06 } }, { \\\"kind\\\" : \\\"misc tarriff\\\", \\\"amount\\\" : 2.25 } ], \\\"description\\\":\\\"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin eget elit ut nulla tempor viverra vel eu nulla. Sed luctus porttitor urna, ac dapibus velit fringilla et. Donec iaculis, dolor a vehicula dictum, augue neque suscipit augue, nec mollis massa neque in libero. Donec sed dapibus magna. Pellentesque at condimentum dolor. In nunc nibh, dignissim in risus a, blandit tincidunt velit. Vestibulum rutrum tempus sem eget tempus. Mauris sollicitudin purus auctor dolor vestibulum, vitae pulvinar neque suscipit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Phasellus lacus turpis, vulputate at adipiscing viverra, ultricies at lectus. Pellentesque ut porta leo, vel eleifend neque. Nunc sagittis metus at ante pellentesque, ut condimentum libero semper. In hac habitasse platea dictumst. In dapibus posuere posuere. Fusce vulputate augue eget tellus molestie, vitae egestas ante malesuada. Phasellus nunc mi, faucibus at elementum pharetra, aliquet a enim. In purus est, vulputate in nibh quis, faucibus dapibus magna. In accumsan libero velit, eu accumsan sem commodo id. In fringilla tempor augue, et feugiat erat convallis et. Sed aliquet eget ipsum eu vestibulum.Curabitur blandit leo nec condimentum semper. Mauris lectus sapien, rutrum a tincidunt id, euismod ac elit. Mauris suscipit et arcu et auctor. Quisque mollis magna vel mi viverra rutrum. Nulla non pretium magna. Cras sed tortor non tellus rutrum gravida eu at odio. Aliquam cursus fermentum erat, nec ullamcorper sem gravida sit amet. Donec viverra, erat vel ornare pulvinar, est ipsum accumsan massa, eu tristique lorem ante nec tortor. Sed suscipit iaculis faucibus. Maecenas a suscipit ligula, vitae faucibus turpis.Cras sed tellus auctor, tempor leo eu, molestie leo. Suspendisse ipsum tellus, egestas et ultricies eu, tempus a arcu. Cras laoreet, est dapibus consequat varius, nisi nisi placerat leo, et dictum ante tortor vitae est. Duis eu urna ac felis ullamcorper rutrum. Quisque iaculis, enim eget sodales vehicula, magna orci dignissim eros, nec volutpat massa urna in elit. In interdum pellentesque risus, feugiat pulvinar odio eleifend sit amet. Quisque congue libero quis dolor faucibus, a mollis nisl tempus.\\\" }\"\n\n echo \"${SEED_DATA}\"\n}\n\n################################################################################\n# function: generate_json_data\n################################################################################\nfunction generate_json_rows ()\n{\n typeset -r NO_OF_ROWS=\"$1\"\n typeset -r FILENAME=\"$2\"\n\n rm -rf ${FILENAME}\n process_log \"creating json data.\"\n NO_OF_LOOPS=$((${NO_OF_ROWS}/11 + 1 ))\n for ((i=0;i<${NO_OF_LOOPS};i++))\n do\n json_seed_data $i >>${FILENAME}\n done\n}\n\n################################################################################\n# print integer arrays\n################################################################################\nfunction print_result()\n{\n typeset -r F_TAG=\"$1\"\n shift\n typeset -r F_LOCALARRAY=(${@})\n typeset F_ARRAYLENGTH\n\n F_ARRAYLENGTH=${#F_LOCALARRAY[@]}\n printf \"%20s \" \"${F_TAG}\"\n for (( i=0 ; i < ${F_ARRAYLENGTH} ; i++ ))\n do\n printf \"%14d \" ${F_LOCALARRAY[${i}]}\n done\n printf \"\\n\"\n}\n" }, { "path": "lib/mongo_func_lib.sh", "content": "#!/bin/bash\n\n################################################################################\n# Copyright EnterpriseDB Cooperation\n# All rights reserved.\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions are\n# met:\n# * Redistributions of source code must retain the above copyright\n# notice, this list of conditions and the following disclaimer.\n# * Redistributions in binary form must reproduce the above copyright\n# notice, this list of conditions and the following disclaimer in\n# the documentation and/or other materials provided with the\n# distribution.\n# * Neither the name of PostgreSQL nor the names of its contributors\n# may be used to endorse or promote products derived from this\n# software without specific prior written permission.\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n# \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS\n# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE\n# COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,\n# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,\n# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN\n# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n# POSSIBILITY OF SUCH DAMAGE.\n#\n# Author: Vibhor Kumar\n# E-mail ID: vibhor.aim@gmail.com\n\n################################################################################\n# source common lib\n################################################################################\nDIRECTORY=$(dirname $0)\nsource ${DIRECTORY}/lib/common_func_lib.sh\n\n################################################################################\n# mongo_run_command: send command to mongodb\n################################################################################\nfunction run_mongo_command ()\n{\n typeset -r F_MONGOHOST=\"$1\"\n typeset -r F_MONGOPORT=\"$2\"\n typeset -r F_MONGODBNAME=\"$3\"\n typeset -r F_MONGOUSER=\"$4\"\n typeset -r F_MONGOPASSWORD=\"$5\"\n typeset -r F_MONGOCOMMAND=\"$6\"\n\n ${MONGO} ${F_MONGOHOST}:${F_MONGOPORT}/${F_MONGODBNAME} \\\n --username ${F_MONGOUSER} \\\n --password ${F_MONGOPASSWORD} --quiet <<- EOF\n DBQuery.shellBatchSize = 10000000000;\n ${F_MONGOCOMMAND}\nEOF\n}\n\n################################################################################\n# run_mong_file: execute list of commands on mongo\n################################################################################\nfunction run_mongo_file ()\n{\n typeset -r F_MONGOHOST=\"$1\"\n typeset -r F_MONGOPORT=\"$2\"\n typeset -r F_MONGODBNAME=\"$3\"\n typeset -r F_MONGOUSER=\"$4\"\n typeset -r F_MONGOPASSWORD=\"$5\"\n typeset -r F_COMMANDFILE=\"$6\"\n\n ${MONGO} ${F_MONGOHOST}:${F_MONGOPORT}/${F_MONGODBNAME} \\\n --username ${F_MONGOUSER} \\\n --password ${F_MONGOPASSWORD} --quiet < ${F_COMMANDFILE}\n}\n\n################################################################################\n# function: drop_mongocollection drop specific collection in mongo\n################################################################################\nfunction drop_mongocollection ()\n{\n typeset -r F_MONGOHOST=\"$1\"\n typeset -r F_MONGOPORT=\"$2\"\n typeset -r F_MONGODBNAME=\"$3\"\n typeset -r F_MONGOUSER=\"$4\"\n typeset -r F_MONGOPASSWORD=\"$5\"\n typeset -r F_MONGOCOLLECTION=\"$6\"\n typeset -r F_MONGOCOMMAND=\"printjson(db.${F_MONGOCOLLECTION}.drop())\"\n\n process_log \"dropping mongo collection ${F_MONGOCOLLECTION}\"\n run_mongo_command \"${F_MONGOHOST}\" \"${F_MONGOPORT}\" \"${F_MONGODBNAME}\" \\\n \"${F_MONGOUSER}\" \"${F_MONGOPASSWORD}\" \"${F_MONGOCOMMAND}\" >/dev/null\n\n}\n\n################################################################################\n# function: mongo_insert_maker\n################################################################################\nfunction mongo_json_insert_maker ()\n{\n typeset -r COLLECTION_NAME=\"$1\"\n typeset -r NO_OF_ROWS=\"$2\"\n typeset -r JSON_FILENAME=\"$3\"\n\n rm -rf ${JSON_FILENAME}\n process_log \"preparing mongo insert commands.\"\n NO_OF_LOOPS=$((${NO_OF_ROWS}/11 + 1 ))\n for ((i=0;i<${NO_OF_LOOPS};i++))\n do\n json_seed_data $i | sed \"s/^/db.${COLLECTION_NAME}.insert( /\" | \\\n sed \"s/$/ )/\" >>${JSON_FILENAME}\n done\n}\n\n################################################################################\n# function: benchmark mongo-import\n################################################################################\nfunction mongodb_import_benchmark ()\n{\n\n typeset -r F_MONGOHOST=\"$1\"\n typeset -r F_MONGOPORT=\"$2\"\n typeset -r F_MONGODBNAME=\"$3\"\n typeset -r F_MONGOUSER=\"$4\"\n typeset -r F_MONGOPASSWORD=\"$5\"\n typeset -r F_COLLECTION=\"$6\"\n typeset -r F_FILENAME=\"$7\"\n\n process_log \"testing mongoimport.\"\n start_time=$(get_timestamp_nano)\n ${MONGOIMPORT} --host ${F_MONGOHOST} --db ${F_MONGODBNAME} \\\n --username ${F_MONGOUSER} --password ${F_MONGOPASSWORD} \\\n --type json --port ${F_MONGOPORT} \\\n --collection ${F_COLLECTION} < ${F_FILENAME} >/dev/null \\\n 2>>/dev/null\n end_time=$(get_timestamp_nano)\n total_time=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n echo \"${total_time}\"\n}\n\n################################################################################\n# function: benchmark mongo inserts\n################################################################################\nfunction mongodb_inserts_benchmark ()\n{\n\n typeset -r F_MONGOHOST=\"$1\"\n typeset -r F_MONGOPORT=\"$2\"\n typeset -r F_MONGODBNAME=\"$3\"\n typeset -r F_MONGOUSER=\"$4\"\n typeset -r F_MONGOPASSWORD=\"$5\"\n typeset -r F_COLLECTION=\"$6\"\n typeset -r F_JSONINSERTS=\"$7\"\n\n process_log \"testing inserts in mongo\"\n start_time=$(get_timestamp_nano)\n run_mongo_file \"${F_MONGOHOST}\" \"${MONGOPORT}\" \"${F_MONGODBNAME}\" \\\n \"${F_MONGOUSER}\" \"${F_MONGOPASSWORD}\" \\\n \"${F_JSONINSERTS}\" >/dev/null\n end_time=$(get_timestamp_nano)\n total_time=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n echo \"${total_time}\"\n}\n\n################################################################################\n# function: benchmark mongo-select\n################################################################################\nfunction mongodb_select_benchmark ()\n{\n\n F_MONGOHOST=\"$1\"\n F_MONGOPORT=\"$2\"\n F_MONGODBNAME=\"$3\"\n F_MONGOUSER=\"$4\"\n F_MONGOPASSWORD=\"$5\"\n F_COLLECTION=\"$6\"\n F_MONGOSELECT1=\"db.${F_COLLECTION}.find({ brand: 'ACME'})\"\n F_MONGOSELECT2=\"db.${F_COLLECTION}.find({ name: 'Phone Service Basic Plan'})\"\n F_MONGOSELECT3=\"db.${F_COLLECTION}.find({ name: 'AC3 Case Red'})\"\n F_MONGOSELECT4=\"db.${F_COLLECTION}.find({ type: 'service'})\"\n\n process_log \"testing mongo FIRST SELECT.\"\n start_time=$(get_timestamp_nano)\n run_mongo_command \"${F_MONGOHOST}\" \"${MONGOPORT}\" \"${F_MONGODBNAME}\" \\\n \"${F_MONGOUSER}\" \\\n \"${F_MONGOPASSWORD}\" \"${F_MONGOSELECT1}\" >/dev/null\n end_time=$(get_timestamp_nano)\n total_time1=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n process_log \"testing mongo SECOND SELECT.\"\n start_time=$(get_timestamp_nano)\n run_mongo_command \"${F_MONGOHOST}\" \"${MONGOPORT}\" \"${F_MONGODBNAME}\" \\\n \"${F_MONGOUSER}\" \\\n \"${F_MONGOPASSWORD}\" \"${F_MONGOSELECT2}\" >/dev/null\n end_time=$(get_timestamp_nano)\n total_time2=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n process_log \"testing mongo THIRD SELECT.\"\n start_time=$(get_timestamp_nano)\n run_mongo_command \"${F_MONGOHOST}\" \"${MONGOPORT}\" \"${F_MONGODBNAME}\" \\\n \"${F_MONGOUSER}\" \\\n \"${F_MONGOPASSWORD}\" \"${F_MONGOSELECT3}\" >/dev/null\n end_time=$(get_timestamp_nano)\n total_time3=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n process_log \"testing mongo FOURTH SELECT.\"\n start_time=$(get_timestamp_nano)\n run_mongo_command \"${F_MONGOHOST}\" \"${MONGOPORT}\" \"${F_MONGODBNAME}\" \\\n \"${F_MONGOUSER}\" \\\n \"${F_MONGOPASSWORD}\" \"${F_MONGOSELECT4}\" >/dev/null\n end_time=$(get_timestamp_nano)\n total_time4=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n AVG=\"$(( ($total_time1 + $total_time2 + $total_time3 + $total_time4)/4 ))\"\n\n echo \"${AVG}\"\n}\n\n################################################################################\n# function: mongdb collection size\n################################################################################\nfunction mongo_collection_size ()\n{\n typeset -r F_MONGOHOST=\"$1\"\n typeset -r F_MONGOPORT=\"$2\"\n typeset -r F_MONGODBNAME=\"$3\"\n typeset -r F_MONGOUSER=\"$4\"\n typeset -r F_MONGOPASSWORD=\"$5\"\n typeset -r F_COLLECTION=\"$6\"\n typeset -r F_COMMAND=\"printjson(db.${F_COLLECTION}.stats())\"\n\n process_log \"calculating the size of mongo collection.\"\n output=\"$(run_mongo_command \"${F_MONGOHOST}\" \"${F_MONGOPORT}\" \\\n \"${F_MONGODBNAME}\" \"${F_MONGOUSER}\" \\\n \"${F_MONGOPASSWORD}\" \"${F_COMMAND}\")\"\n collectionsize=\"$(echo ${output}|awk -F\",\" '{print $5}'|cut -d\":\" -f2)\"\n\n echo \"${collectionsize}\"\n}\n\n\n################################################################################\n# function: mongdb version\n################################################################################\nfunction mongo_version ()\n{\n typeset -r F_MONGOHOST=\"$1\"\n typeset -r F_MONGOPORT=\"$2\"\n typeset -r F_MONGODBNAME=\"$3\"\n typeset -r F_MONGOUSER=\"$4\"\n typeset -r F_MONGOPASSWORD=\"$5\"\n typeset -r F_COMMAND=\"db.version()\"\n\n output=\"$(run_mongo_command \"${F_MONGOHOST}\" \"${F_MONGOPORT}\" \\\n \"${F_MONGODBNAME}\" \"${F_MONGOUSER}\" \\\n \"${F_MONGOPASSWORD}\" \"${F_COMMAND}\" )\"\n version=$(echo $output|awk '{print $2}')\n\n echo \"${version}\"\n}\n\n################################################################################\n# function: mongodb create_index\n################################################################################\nfunction mongodb_create_index ()\n{\n\n typeset -r F_MONGOHOST=\"$1\"\n typeset -r F_MONGOPORT=\"$2\"\n typeset -r F_MONGODBNAME=\"$3\"\n typeset -r F_MONGOUSER=\"$4\"\n typeset -r F_MONGOPASSWORD=\"$5\"\n typeset -r F_COLLECTION=\"$6\"\n typeset -r F_MONGODBIDX1=\"db.${F_COLLECTION}.ensureIndex( { \\\"name\\\": 1});\"\n typeset -r F_MONGODBIDX2=\"db.${F_COLLECTION}.ensureIndex( { \\\"type\\\": 1});\"\n typeset -r F_MONGODBIDX3=\"db.${F_COLLECTION}.ensureIndex( { \\\"brand\\\": 1});\"\n\n process_log \"creating index in mongodb.\"\n run_mongo_command \"${F_MONGOHOST}\" \"${MONGOPORT}\" \"${F_MONGODBNAME}\" \\\n \"${F_MONGOUSER}\" \\\n \"${F_MONGOPASSWORD}\" \"${F_MONGODBIDX1}\" >/dev/null\n run_mongo_command \"${F_MONGOHOST}\" \"${MONGOPORT}\" \"${F_MONGODBNAME}\" \\\n \"${F_MONGOUSER}\" \\\n \"${F_MONGOPASSWORD}\" \"${F_MONGODBIDX2}\" >/dev/null\n run_mongo_command \"${F_MONGOHOST}\" \"${MONGOPORT}\" \"${F_MONGODBNAME}\" \\\n \"${F_MONGOUSER}\" \\\n \"${F_MONGOPASSWORD}\" \"${F_MONGODBIDX3}\" >/dev/null\n\n}\n" }, { "path": "lib/pg_func_lib.sh", "content": "#!/bin/bash\n\n################################################################################\n# Copyright EnterpriseDB Cooperation\n# All rights reserved.\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions are\n# met:\n# * Redistributions of source code must retain the above copyright\n# notice, this list of conditions and the following disclaimer.\n# * Redistributions in binary form must reproduce the above copyright\n# notice, this list of conditions and the following disclaimer in\n# the documentation and/or other materials provided with the\n# distribution.\n# * Neither the name of PostgreSQL nor the names of its contributors\n# may be used to endorse or promote products derived from this\n# software without specific prior written permission.\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n# \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS\n# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE\n# COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,\n# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,\n# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN\n# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n# POSSIBILITY OF SUCH DAMAGE.\n#\n# Author: Vibhor Kumar\n# E-mail ID: vibhor.aim@gmail.com\n\n################################################################################\n# source common lib\n################################################################################\nsource $DIRECTORY/lib/common_func_lib.sh\n\n################################################################################\n# function: pg_json_insert_maker\n################################################################################\nfunction pg_json_insert_maker ()\n{\n typeset -r COLLECTION_NAME=\"$1\"\n typeset -r NO_OF_ROWS=\"$2\"\n typeset -r JSON_FILENAME=\"$3\"\n\n process_log \"preparing postgresql INSERTs.\"\n rm -rf ${JSON_FILENAME}\n NO_OF_LOOPS=$((${NO_OF_ROWS}/11 + 1 ))\n for ((i=0;i<${NO_OF_LOOPS};i++))\n do\n json_seed_data $i | \\\n sed \"s/^/INSERT INTO ${COLLECTION_NAME} VALUES(\\$JSON\\$/\"| \\\n sed \"s/$/\\$JSON\\$);/\" >>${JSON_FILENAME}\n done\n}\n\n################################################################################\n# run_sql_file: send SQL from a file to database\n################################################################################\nfunction run_sql_file ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_SQLFILE=\"$6\"\n\n export PGPASSWORD=\"${F_PGPASSWORD}\"\n ${PGHOME}/bin/psql -qAt -h ${F_PGHOST} -p ${F_PGPORT} -U ${F_PGUSER} \\\n --single-transaction -d ${F_DBNAME} -f \"${F_SQLFILE}\"\n}\n\n################################################################################\n# run_sql: send SQL to database\n################################################################################\nfunction run_sql ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_SQL=\"$6\"\n\n export PGPASSWORD=\"${F_PGPASSWORD}\"\n ${PGHOME}/bin/psql -qAt -h ${F_PGHOST} -p ${F_PGPORT} -U ${F_PGUSER} \\\n -d ${F_DBNAME} -c \"${F_SQL}\"\n}\n\n################################################################################\n# function: remove_pgdb (remove postgresql database)\n################################################################################\nfunction remove_pg_db ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_SQL=\"DROP DATABASE IF EXISTS ${F_DBNAME};\"\n\n process_log \"droping database ${F_DBNAME} if exists.\"\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"postgres\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \"${F_SQL}\" 2>/dev/null >/dev/null\n}\n\n################################################################################\n# function: create_pgdb (create postgresql database)\n################################################################################\nfunction create_pg_db ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_SQL=\"CREATE DATABASE ${F_DBNAME};\"\n\n process_log \"creating database ${F_DBNAME}.\"\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"postgres\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \"${F_SQL}\"\n}\n\n################################################################################\n# function: relation_size (calculate postgresql relation size)\n################################################################################\nfunction pg_relation_size ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_RELATION=\"$6\"\n typeset -r F_SQL=\"SELECT pg_catalog.pg_relation_size('${F_RELATION}');\"\n\n process_log \"calculating PostgreSQL collection size.\"\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \"${F_SQL}\"\n}\n\n################################################################################\n# function: check if database exists\n################################################################################\nfunction if_dbexists ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_SQL=\"SELECT COUNT(1)\n FROM pg_catalog.pg_database\n WHERE datname='${F_DBNAME}';\"\n\n output=$(run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"postgres\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \\\n \"${F_SQL}\")\n echo ${output}\n}\n\n################################################################################\n# function: mk_pgjson_collection create json table in PG\n################################################################################\nfunction mk_pg_json_collection ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_TABLE=\"$6\"\n typeset -r F_SQL1=\"DROP TABLE IF EXISTS ${F_TABLE} CASCADE;\"\n typeset -r F_SQL2=\"CREATE TABLE ${F_TABLE} (data JSONB);\"\n\n process_log \"creating ${F_TABLE} collection in postgreSQL.\"\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \"${F_SQL1}\" \\\n >/dev/null 2>/dev/null\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \"${F_SQL2}\" \\\n >/dev/null 2>/dev/null\n\n}\n\n################################################################################\n# function: pg_create_index create json table in PG\n################################################################################\nfunction pg_create_index_collection ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_TABLE=\"$6\"\n typeset -r F_SQL=\"CREATE INDEX ${F_TABLE}_idx ON ${F_TABLE} USING gin(data);\"\n\n process_log \"creating index on postgreSQL collections.\"\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \"${F_SQL}\" \\\n >/dev/null\n\n}\n\n################################################################################\n# function: delete_json_data delete json data in PG\n################################################################################\nfunction delete_json_data ()\n{\n\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_COLLECTION=\"$6\"\n\n process_log \"droping json object in postgresql.\"\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \\\n \"TRUNCATE TABLE ${F_COLLECTION};\" >/dev/null\n}\n\n################################################################################\n# function: pg_copy_benchmark\n################################################################################\nfunction pg_copy_benchmark ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_COLLECTION=\"$6\"\n typeset -r F_JSONFILE=\"$7\"\n typeset -r F_COPY=\"COPY ${F_COLLECTION} FROM STDIN;\"\n\n DBEXISTS=$(if_dbexists \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \\\n \"${F_PGUSER}\" \"${F_PGPASSWORD}\")\n process_log \"loading data in postgresql using ${F_JSONFILE}.\"\n start_time=$(get_timestamp_nano)\n cat ${F_JSONFILE}|run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \\\n \"${F_PGUSER}\" \"${F_PGPASSWORD}\" \"${F_COPY}\"\n end_time=$(get_timestamp_nano)\n total_time=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n echo \"${total_time}\"\n\n}\n\n\n################################################################################\n# function: benchmark postgresql inserts\n################################################################################\nfunction pg_inserts_benchmark ()\n{\n\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_COLLECTION=\"$6\"\n typeset -r F_INSERTS=\"$7\"\n\n process_log \"inserting data in postgresql using ${F_INSERTS}.\"\n start_time=$(get_timestamp_nano)\n run_sql_file \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \"${F_INSERTS}\"\n end_time=$(get_timestamp_nano)\n total_time=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n echo \"${total_time}\"\n}\n\n################################################################################\n# function: benchmark postgresql select\n################################################################################\nfunction pg_select_benchmark ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_COLLECTION=\"$6\"\n typeset -r F_SELECT1=\"SELECT data\n FROM ${F_COLLECTION}\n WHERE (data->>'brand') = 'ACME';\"\n typeset -r F_SELECT2=\"SELECT data\n FROM ${F_COLLECTION}\n WHERE (data->>'name') = 'Phone Service Basic Plan';\"\n typeset -r F_SELECT3=\"SELECT data\n FROM ${F_COLLECTION}\n WHERE (data->>'name') = 'AC3 Case Red';\"\n typeset -r F_SELECT4=\"SELECT data\n FROM ${F_COLLECTION}\n WHERE (data->>'type') = 'service';\"\n local START end_time\n\n process_log \"testing FIRST SELECT in postgresql.\"\n start_time=$(get_timestamp_nano)\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \\\n \"${F_SELECT1}\" >/dev/null || exit_on_error \"failed to execute SELECT 1.\"\n end_time=$(get_timestamp_nano)\n total_time1=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n process_log \"testing SECOND SELECT in postgresql.\"\n start_time=$(get_timestamp_nano)\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \\\n \"${F_SELECT2}\" >/dev/null || exit_on_error \"failed to execute SELECT 2.\"\n end_time=$(get_timestamp_nano)\n total_time2=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n process_log \"testing THIRD SELECT in postgresql.\"\n start_time=$(get_timestamp_nano)\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \\\n \"${F_SELECT3}\" >/dev/null || exit_on_error \"failed to execute SELECT 3.\"\n end_time=$(get_timestamp_nano)\n total_time3=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n process_log \"testing FOURTH SELECT in postgresql.\"\n start_time=$(get_timestamp_nano)\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \\\n \"${F_SELECT4}\" >/dev/null || exit_on_error \"failed to execute SELECT 4.\"\n end_time=$(get_timestamp_nano)\n total_time4=\"$(get_timestamp_diff_nano \"${end_time}\" \"${start_time}\")\"\n\n AVG=$(( ($total_time1 + $total_time2 + $total_time3 + $total_time4 )/4 ))\n\n echo \"${AVG}\"\n}\n\n################################################################################\n# function: mk_pgjson_collection create json table in PG\n################################################################################\nfunction analyze_collections ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_TABLE=\"$6\"\n typeset -r F_SQL=\"VACUUM FREEZE ANALYZE ${F_TABLE};\"\n\n process_log \"performing analyze in postgreSQL.\"\n run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \"${F_SQL}\" \\\n >/dev/null 2>/dev/null\n}\n\n################################################################################\n# function: mk_pgjson_collection create json table in PG\n################################################################################\nfunction pg_version ()\n{\n typeset -r F_PGHOST=\"$1\"\n typeset -r F_PGPORT=\"$2\"\n typeset -r F_DBNAME=\"$3\"\n typeset -r F_PGUSER=\"$4\"\n typeset -r F_PGPASSWORD=\"$5\"\n typeset -r F_SQL=\"select split_part(version(),' ',2);\"\n\n version=$(run_sql \"${F_PGHOST}\" \"${F_PGPORT}\" \"${F_DBNAME}\" \"${F_PGUSER}\" \\\n \"${F_PGPASSWORD}\" \"${F_SQL}\")\n echo \"${version}\"\n}\n" }, { "path": "output/sample_expected.out", "content": "PID: 1251 [RUNTIME: 07-21-14 23:21:24] pg_nosql_benchmark: MongoDB Version 2.6.3\nPID: 1251 [RUNTIME: 07-21-14 23:21:24] pg_nosql_benchmark: PostgreSQL Version 9.4beta1\nPID: 1251 [RUNTIME: 07-21-14 23:21:24] pg_nosql_benchmark: creating json data.\nPID: 1251 [RUNTIME: 07-21-14 23:21:25] pg_nosql_benchmark: preparing postgresql INSERTs.\nPID: 1251 [RUNTIME: 07-21-14 23:21:26] pg_nosql_benchmark: preparing mongo insert commands.\nPID: 1251 [RUNTIME: 07-21-14 23:21:27] pg_nosql_benchmark: droping database benchmark if exists.\nPID: 1251 [RUNTIME: 07-21-14 23:21:27] pg_nosql_benchmark: creating database benchmark.\nPID: 1251 [RUNTIME: 07-21-14 23:21:27] pg_nosql_benchmark: dropping mongo collection json_tables\nPID: 1251 [RUNTIME: 07-21-14 23:21:27] pg_nosql_benchmark: creating json_tables collection in postgreSQL.\nPID: 1251 [RUNTIME: 07-21-14 23:21:28] pg_nosql_benchmark: loading data in postgresql using sample.json.\nPID: 1251 [RUNTIME: 07-21-14 23:21:28] pg_nosql_benchmark: droping json object in postgresql.\nPID: 1251 [RUNTIME: 07-21-14 23:21:28] pg_nosql_benchmark: inserting data in postgresql using sample_pg_inserts.json.\nPID: 1251 [RUNTIME: 07-21-14 23:21:28] pg_nosql_benchmark: creating index on postgreSQL collections.\nPID: 1251 [RUNTIME: 07-21-14 23:21:28] pg_nosql_benchmark: testing mongoimport.\nPID: 1251 [RUNTIME: 07-21-14 23:21:29] pg_nosql_benchmark: dropping mongo collection json_tables\nPID: 1251 [RUNTIME: 07-21-14 23:21:29] pg_nosql_benchmark: testing inserts in mongo\nPID: 1251 [RUNTIME: 07-21-14 23:21:30] pg_nosql_benchmark: creating index in mongodb.\nPID: 1251 [RUNTIME: 07-21-14 23:21:31] pg_nosql_benchmark: testing FIRST SELECT in postgresql.\nPID: 1251 [RUNTIME: 07-21-14 23:21:31] pg_nosql_benchmark: testing SECOND SELECT in postgresql.\nPID: 1251 [RUNTIME: 07-21-14 23:21:31] pg_nosql_benchmark: testing THIRD SELECT in postgresql.\nPID: 1251 [RUNTIME: 07-21-14 23:21:31] pg_nosql_benchmark: testing FOURTH SELECT in postgresql.\nPID: 1251 [RUNTIME: 07-21-14 23:21:31] pg_nosql_benchmark: calculating PostgreSQL collection size.\nPID: 1251 [RUNTIME: 07-21-14 23:21:31] pg_nosql_benchmark: testing mongo FIRST SELECT.\nPID: 1251 [RUNTIME: 07-21-14 23:21:31] pg_nosql_benchmark: testing mongo SECOND SELECT.\nPID: 1251 [RUNTIME: 07-21-14 23:21:31] pg_nosql_benchmark: testing mongo THIRD SELECT.\nPID: 1251 [RUNTIME: 07-21-14 23:21:31] pg_nosql_benchmark: testing mongo FOURTH SELECT.\nPID: 1251 [RUNTIME: 07-21-14 23:21:31] pg_nosql_benchmark: calculating the size of mongo collection.\n number of rows 1000 \n PG COPY (ns) 107488390 \n PG INSERT (ns) 514707930 \n PG SELECT (ns) 24235127 \n PG SIZE (bytes) 1490944 \n MONGO IMPORT (ns) 494436900 \n MONGO INSERT (ns) 1666742540 \n MONGO SELECT (ns) 80460677 \n MONGO SIZE (bytes) 5578752 \n" }, { "path": "pg_nosql_benchmark", "content": "#/bin/bash\n\n#################################################################################\n# Copyright (c) 2013-2014, EnterpriseDB Corporation\n# \n# Redistribution and use in source and binary forms, with or without modification,\n# are permitted provided that the following conditions are met:\n#\n# 1. Redistributions of source code must retain the above copyright notice, this\n# list of conditions and the following disclaimer.\n#\n# 2. Redistributions in binary form must reproduce the above copyright notice,\n# this list of conditions and the following disclaimer in the documentation and/or\n# other materials provided with the distribution.\n#\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\n# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\n# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR\n# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON\n# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\n# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n#\n#===============================================================================\n#title : pg_nosql_benchmark.\n#description : This script will help in benchmarking PostgreSQL (JSONB) and\n# : MongoDB (BSON).\n#author : Vibhor Kumar (vibhor.aim@gmail.com).\n#date : July 17, 2014\n#version : 2.0.1\n#usage : bash pg_nosql_benchmark\n#notes : Install Vim and Emacs to use this script.\n#bash_version : GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)\n#===============================================================================\n\n################################################################################\n# set require variables.\n################################################################################\nDIRECTORY=$(dirname $0)\nBASENAME=$(basename $0)\n\nPGHOME=\"/usr/ppas-9.3\"\nPGHOST=\"172.17.0.4\"\nPGPORT=\"5432\"\nPGUSER=\"postgres\"\nPGPASSWORD=\"postgres\"\nPGDATABASE=\"benchmark\"\n\nPGBIN=\"/usr/ppas-9.3/bin\"\n\n################################################################################\n# set mongo variables.\n################################################################################\nMONGO=\"/usr/bin/mongo\"\nMONGOIMPORT=\"/usr/bin/mongoimport\"\nMONGOHOST=\"172.17.0.3\"\nMONGOPORT=\"27017\"\nMONGOUSER=\"mongo\"\nMONGOPASSWORD=\"mongo\"\nMONGODBNAME=\"benchmark\"\n\nCOLLECTION_NAME=\"json_tables\"\nSAMPLEJSON=\"sample.json\"\nPG_INSERTS=\"sample_pg_inserts.json\"\nMONGO_INSERTS=\"sample_mongo_inserts.json\"\n\nexport PATH=$PGBIN:$PATH\n\n################################################################################\n# source library files\n################################################################################\nsource ${DIRECTORY}/lib/pg_func_lib.sh\nsource ${DIRECTORY}/lib/mongo_func_lib.sh\n\n################################################################################\n# declare require arrays\n################################################################################\ndeclare -a json_rows=(10000000)\n\ndeclare -a pg_size_time\ndeclare -a pg_copy_time\ndeclare -a pg_inserts_time\ndeclare -a pg_select_time\n\n# mongo specific arrays\ndeclare -a mongo_size_time\ndeclare -a mongo_copy_time\ndeclare -a mongo_inserts_time\ndeclare -a mongo_select_time\n\n################################################################################\n# main function\n################################################################################\nmongodb_version=$(mongo_version \"${MONGOHOST}\" \\\n \"${MONGOPORT}\" \\\n \"${MONGODBNAME}\" \\\n \"${MONGOUSER}\" \\\n \"${MONGOPASSWORD}\"\n )\n\npg_version=$(pg_version \"${PGHOST}\" \\\n \"${PGPORT}\" \\\n \"${PGDATABASE}\" \\\n \"${PGUSER}\" \\\n \"${PGPASSWORD}\"\n )\n\nprocess_log \"MongoDB Version $mongodb_version\"\nprocess_log \"PostgreSQL Version $pg_version\"\n\n\nfor (( indx=0 ; indx < ${#json_rows[@]} ; indx++ ))\ndo\n generate_json_rows \"${json_rows[${indx}]}\" \\\n \"${SAMPLEJSON}\"\n\n pg_json_insert_maker \"${COLLECTION_NAME}\" \\\n \"${json_rows[${indx}]}\" \\\n \"${PG_INSERTS}\"\n\n mongo_json_insert_maker \"${COLLECTION_NAME}\" \\\n \"${json_rows[${indx}]}\" \\\n \"${MONGO_INSERTS}\"\n\n\n remove_pg_db \"${PGHOST}\" \\\n \"${PGPORT}\" \\\n \"${PGDATABASE}\" \\\n \"${PGUSER}\" \\\n \"${PGPASSWORD}\"\n create_pg_db \"${PGHOST}\" \\\n \"${PGPORT}\" \\\n \"${PGDATABASE}\" \\\n \"${PGUSER}\" \\\n \"${PGPASSWORD}\"\n\n drop_mongocollection \"${MONGOHOST}\" \\\n \"${MONGOPORT}\" \\\n \"${MONGODBNAME}\" \\\n \"${MONGOUSER}\" \\\n \"${MONGOPASSWORD}\" \\\n \"${COLLECTION_NAME}\"\n\n mk_pg_json_collection \"${PGHOST}\" \\\n \"${PGPORT}\" \\\n \"${PGDATABASE}\" \\\n \"${PGUSER}\" \\\n \"${PGPASSWORD}\" \\\n \"${COLLECTION_NAME}\"\n\n pg_copy_time[${indx}]=$(pg_copy_benchmark \"${PGHOST}\" \\\n \"${PGPORT}\" \\\n \"${PGDATABASE}\" \\\n \"${PGUSER}\" \\\n \"${PGPASSWORD}\" \\\n \"${COLLECTION_NAME}\" \\\n \"${SAMPLEJSON}\"\n )\n\n\n pg_create_index_collection \"${PGHOST}\" \\\n \"${PGPORT}\" \\\n \"${PGDATABASE}\" \\\n \"${PGUSER}\" \\\n \"${PGPASSWORD}\" \\\n \"${COLLECTION_NAME}\"\n\n mongo_copy_time[${indx}]=$(mongodb_import_benchmark \"${MONGOHOST}\" \\\n \"${MONGOPORT}\" \\\n \"${MONGODBNAME}\" \\\n \"${MONGOUSER}\" \\\n \"${MONGOPASSWORD}\" \\\n \"${COLLECTION_NAME}\" \\\n \"${SAMPLEJSON}\"\n )\n\n\n mongodb_create_index \"${MONGOHOST}\" \\\n \"${MONGOPORT}\" \\\n \"${MONGODBNAME}\" \\\n \"${MONGOUSER}\" \\\n \"${MONGOPASSWORD}\" \\\n \"${COLLECTION_NAME}\"\n\n pg_select_time[${indx}]=$(pg_select_benchmark \"${PGHOST}\" \\\n \"${PGPORT}\" \\\n \"${PGDATABASE}\" \\\n \"${PGUSER}\" \\\n \"${PGPASSWORD}\" \\\n \"${COLLECTION_NAME}\"\n )\n pg_size_time[${indx}]=$(pg_relation_size \"${PGHOST}\" \\\n \"${PGPORT}\" \\\n \"${PGDATABASE}\" \\\n \"${PGUSER}\" \\\n \"${PGPASSWORD}\" \\\n \"${COLLECTION_NAME}\"\n )\n\n mongo_select_time[${indx}]=$(mongodb_select_benchmark \"${MONGOHOST}\" \\\n \"${MONGOPORT}\" \\\n \"${MONGODBNAME}\" \\\n \"${MONGOUSER}\" \\\n \"${MONGOPASSWORD}\" \\\n \"${COLLECTION_NAME}\"\n )\n\n mongo_size_time[${indx}]=$(mongo_collection_size \"${MONGOHOST}\" \\\n \"${MONGOPORT}\" \\\n \"${MONGODBNAME}\" \\\n \"${MONGOUSER}\" \\\n \"${MONGOPASSWORD}\" \\\n \"${COLLECTION_NAME}\"\n )\n \n drop_mongocollection \"${MONGOHOST}\" \\\n \"${MONGOPORT}\" \\\n \"${MONGODBNAME}\" \\\n \"${MONGOUSER}\" \\\n \"${MONGOPASSWORD}\" \\\n \"${COLLECTION_NAME}\"\n\n mongo_inserts_time[${indx}]=$(mongodb_inserts_benchmark \"${MONGOHOST}\" \\\n \"${MONGOPORT}\" \\\n \"${MONGODBNAME}\" \\\n \"${MONGOUSER}\" \\\n \"${MONGOPASSWORD}\" \\\n \"${COLLECTION_NAME}\" \\\n \"${MONGO_INSERTS}\"\n )\n delete_json_data \"${PGHOST}\" \\\n \"${PGPORT}\" \\\n \"${PGDATABASE}\" \\\n \"${PGUSER}\" \\\n \"${PGPASSWORD}\" \\\n \"${COLLECTION_NAME}\"\n\n pg_inserts_time[${indx}]=$(pg_inserts_benchmark \"${PGHOST}\" \\\n \"${PGPORT}\" \\\n \"${PGDATABASE}\" \\\n \"${PGUSER}\" \\\n \"${PGPASSWORD}\" \\\n \"${COLLECTION_NAME}\" \\\n \"${PG_INSERTS}\"\n )\ndone\n\n\nprint_result \"number of rows\" \"${json_rows[@]}\"\nprint_result \"PG COPY (ns)\" \"${pg_copy_time[@]}\"\nprint_result \"PG INSERT (ns)\" \"${pg_inserts_time[@]}\"\nprint_result \"PG SELECT (ns)\" \"${pg_select_time[@]}\"\nprint_result \"PG SIZE (bytes)\" \"${pg_size_time[@]}\"\nprint_result \"MONGO IMPORT (ns)\" \"${mongo_copy_time[@]}\"\nprint_result \"MONGO INSERT (ns)\" \"${mongo_inserts_time[@]}\"\nprint_result \"MONGO SELECT (ns)\" \"${mongo_select_time[@]}\"\nprint_result \"MONGO SIZE (bytes)\" \"${mongo_size_time[@]}\"\n\nrm -rf ${SAMPLEJSON}*\nrm -rf ${PG_INSERTS}\nrm -rf ${MONGO_INSERTS}\n" } ]