[ { "path": "HA-keepalived.md", "content": "Introduction\n------------\n\nThis document aims at defining a high level architecture for a highly available OpenStack setup using application-native options and keepalived. It will document the overall setup, the architecture limitations and any important item to keep in mind when deploying.\n\nThe document can be used to create a highly available architecture for:\n\n- [RDO Liberty](http://www.rdoproject.org)\n- [Red Hat Enterprise Linux Openstack Platform 8](http://www.redhat.com/openstack), when available\n\n**Note**: Instructions for Liberty are still work in progress.\n\nIf you are looking for the Kilo edition, check out the [Kilo-RDO7 branch](../Kilo-RDO7/HA-keepalived.md).\n\nMost of the time the instructions will apply to both distributions. If there is any difference, it will be specified throughout the document.\n\n### Authors and changelog\n\n- Javier Peña (jpena@redhat.com) commited the initial version.\n\nArchitecture overview\n---------------------\n\n### Scope\n\nThis document will define, at a high level, the placement and high availability strategy for the different OpenStack services, as well as the limitations of the architecture.\n\nService monitoring and recovery are outside the scope for this document. You may choose to monitor and recover the running services health using any generally available monitoring tool, or implement automated service restart in systemd.\n\n### Server roles\n\nA typical OpenStack architecture will consist of servers performing various roles. The following roles may be spread over different servers or collapsed into some of them:\n\n- Controller nodes\n- Load balancer nodes\n- Database nodes\n- Network nodes\n- Storage nodes\n- Compute nodes\n\nHigh availability for storage and compute nodes is out of the scope for this document.\n\n### High availability strategy\n\nThe following diagram shows a very simplified view of the different strategies used to achieve high availability for the OpenStack services:\n\n![](keepalived/Highlevelarch.jpg \"High level architecture\")\n\nDepending on the method used to communicate with the service, the following availability strategies will be followed:\n\n- Keepalived, for the HAProxy instances.\n- Access via an HAProxy virtual IP, for services accessed via a TCP socket than can be load balanced (e.g. httpd).\n- Built-in application clustering, when available from the application (e.g. Galera) .\n- Starting up one instance of the service on several controller nodes, when they can coexist and coordinate by other means (e.g. RPC, in the case of nova-conductor).\n- No high availability, when the service can only work in active/passive mode.\n\nThe detailed high availability strategy for the OpenStack services is defined in the following table.\n\n| Service | Process | Mode | HA stragegy |\n|------------------|----------------------------|:------:|-------------|\n| Support services |MariaDB - Galera | A/A | HAProxy / app cluster |\n| Support services |RabbitMQ | A/A | App cluster / service config |\n| Support services |HAProxy | A/A | Keepalived |\n| Support services |MongoDB | A/A | App cluster |\n| Support services |Memcached | A/A | Service configuration |\n| Support services |Redis | A/A | App cluster (Sentinel)|\n| Keystone |openstack-keystone | A/A | HAProxy |\n| Glance |openstack-glance-api | A/A | HAProxy |\n| Glance |openstack-glance-registry | A/A | HAProxy |\n| Nova |openstack-nova-api | A/A | HAProxy |\n| Nova |openstack-nova-cert | A/A | |\n| Nova |openstack-nova-compute | A/A | |\n| Nova |openstack-nova-scheduler | A/A | |\n| Nova |openstack-nova-conductor | A/A | |\n| Nova |openstack-nova-novncproxy | A/A | HAProxy |\n| Cinder |openstack-cinder-api | A/A | HAProxy |\n| Cinder |openstack-cinder-scheduler | A/A | |\n| Cinder |openstack-cinder-volume | **A/P**| No HA |\n| Cinder |openstack-cinder-backup | A/A | |\n| Neutron |neutron-server | A/A | HAProxy |\n| Neutron |neutron-dhcp-agent | A/A | Multiple DHCP agents |\n| Neutron |neutron-l3-agent | A/A | L3 HA |\n| Neutron |neutron-metadata-agent | A/A | |\n| Neutron |neutron-lbaas-agent | **A/P**| |\n| Neutron |neutron-openvswitch-agent | A/A | |\n| Neutron |neutron-metering-agent | A/A | |\n| Horizon |httpd | A/A | HAProxy |\n| Ceilometer |openstack-ceilometer-api | A/A | HAProxy |\n| Ceilometer |openstack-ceilometer-central| A/A | Workload partitioning: tooz + Redis|\n| Ceilometer |openstack-ceilometer-compute| A/A | |\n| Ceilometer |openstack-ceilometer-alarm-notifier| A/A | |\n| Ceilometer |openstack-ceilometer-evaluator| A/A | |\n| Ceilometer |openstack-ceilometer-notification| A/A | |\n| Heat |openstack-heat-api | A/A | HAProxy |\n| Heat |openstack-heat-cfn | A/A | |\n| Heat |openstack-heat-cloudwatch | A/A | |\n| Heat |openstack-heat-engine | A/A | |\n| Swift |openstack-swift-proxy | A/A | HAProxy |\n| Swift |openstack-swift-account | A/A | HAProxy |\n| Swift |openstack-swift-container | A/A | HAProxy |\n| Swift |openstack-swift-object | A/A | HAProxy |\n| Sahara |openstack-sahara-api | A/A | HAProxy |\n| Sahara |openstack-sahara-engine | A/A | |\n| Trove |openstack-trove-api | A/A | HAProxy |\n| Trove |openstack-trove-engine | A/A | |\n| Trove |openstack-trove-conductor | A/A | |\n\n**Notes:**\n\n1. There are known issues with cinder-volume that recommend setting it as active-passive for now, see \n2. While there will be multiple Neutron LBaaS agents running, each agent will manage a set of load balancers, that cannot be failed over to another node.\n\nArchitecture limitations\n------------------------\n\nThis architecture has some inherent limitations that should be kept in mind during deployment and daily operations. The following sections describe those limitations.\n\n### Keepalived and network partitions\n\nIn case of a network partitioning, there is a chance that two or more nodes running keepalived claim to hold the same VIP, which may lead to an undesired behaviour. Since keepalived uses VRRP over multicast to elect a master (VIP owner), a network partition in which keepalived nodes cannot communicate will result in the VIPs existing on two nodes. When the network partition is resolved, the duplicate VIPs should also be resolved. Note that this network partition problem with VRRP is a known limitation for this architecture.\n\n### Cinder-volume as a single point of failure\n\nThere are currently concerns over the cinder-volume service ability to run as a fully active-active service. During the Mitaka timeframe, this is being worked on, see [1](https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support). Thus, cinder-volume will only be running on one of the controller nodes, even if it will be configured on all nodes. In case of a failure in the node running cinder-volume, it should be started in a surviving controller node.\n\n### Neutron-lbaas-agent as a single point of failure\n\nThe current design of the Neutron LBaaS agent using the HAProxy driver does not allow high availability for the tenant load balancers. The neutron-lbaas-agent service will be enabled and running on all controllers, allowing for load balancers to be distributed across all nodes. However, a controller node failure will stop all load balancers running on that node until the service is recovered or the load balancer is manually removed and created again.\n\n### Service monitoring and recovery required\n\nAn external service monitoring infrastructure is required to check the OpenStack service health, and notify operators in case of any failure. This architecture does not provide any facility for that, so it would be necessary to integrate the OpenStack deployment with any existing monitoring environment.\n\n### Manual recovery after a full cluster restart\n\nSome support services used by RDO / RHEL OSP use their own form of application clustering. Usually, these services maintain a cluster quorum, that may be lost in case of a simultaneous restart of all cluster nodes, e.g. during a power outage. Each service will require its own procedure to regain quorum:\n\n- Galera: [Galera bootstrap instructions](keepalived/galera-bootstrap.md)\n- RabbitMQ: [RabbitMQ cluster restart](keepalived/rabbitmq-restart.md)\n- MongoDB: [MongoDB cluster recovery](keepalived/mongodb-recovery.md)\n\nImplementation\n--------------\n\nThe implementation will be split into two articles:\n\n- [Controller node implementation](keepalived/controller-node.md)\n- [Compute node implementation](keepalived/compute-node.md)\n" }, { "path": "README.md", "content": "Introduction\n------------\n\nThis repository contains the description of two highly available OpenStack architectures using RDO or Red Hat Enterprise Linux OpenStack Platform:\n\n- An architecture based on [Pacemaker](ha-openstack.md).\n- An architecture based on [application-native tools and Keepalived](HA-keepalived.md).\n\nEach architecture includes a description and implementation instructions. Be sure to understand the contrains and limitations of each architecture, and choose the one that better suits your needs.\n\nHow you can help\n----------------\n\nFeedback is encouraged for this project. Feel free to submit patches and report issues.\n" }, { "path": "build-all.sh", "content": "#!/bin/bash\n\nset -e\n\ndeclare -A nodeMap\ndeclare -A variables\ndeclare -A cluster\n\nnodeMap[\"baremetal\"]=\"east-01 east-02 east-03 east-04 east-05 east-06 east-07\"\nnodeMap[\"virt-hosts\"]=\"east-01 east-02 east-03 east-04\"\nnodeMap[\"galera\"]=\"rdo7-db1.vmnet rdo7-db2.vmnet rdo7-db3.vmnet\"\nnodeMap[\"memcached\"]=\"rdo7-memcache1.vmnet rdo7-memcache2.vmnet rdo7-memcache3.vmnet\"\nnodeMap[\"swift-aco\"]=\"rdo7-swift-brick1.vmnet rdo7-swift-brick2.vmnet rdo7-swift-brick3.vmnet\"\nnodeMap[\"compute-nodes\"]=\"east-05 east-06 east-07\"\nnodeMap[\"controller-managed\"]=\"rdo7-node1.vmnet rdo7-node2.vmnet rdo7-node3.vmnet east-05 east-06 east-07\"\nnodeMap[\"vmsnap\"]=\"east-02 east-03 east-04\"\n\ncluster[\"baremetal\"]=0\ncluster[\"gateway\"]=0\ncluster[\"virt-hosts\"]=0\n\nvariables[\"nodes\"]=\"\"\nvariables[\"network_domain\"]=\"lab.bos.redhat.com\"\nvariables[\"deployment\"]=\"collapsed\"\nvariables[\"status\"]=0\nvariables[\"components\"]=\"lb db rabbitmq memcache mongodb keystone glance cinder swift-brick swift neutron-server neutron-agents ceilometer heat\"\nvariables[\"scenarios-segregated\"]=\"gateway virt-hosts vmsnap-hacks hacks vmsnap-lb lb vmsnap-galera galera vmsnap-rabbitmq rabbitmq vmsnap-memcached memcached vmsnap-mongodb mongodb vmsnap-keystone keystone vmsnap-glance glance vmsnap-cinder cinder vmsnap-swift-aco swift-aco vmsnap-swift swift vmsnap-neutron-server neutron-server vmsnap-neutron-agents neutron-agents vmsnap-nova nova vmsnap-ceilometer ceilometer vmsnap-heat heat vmsnap-horizon horizon compute-common vmsnap-compute compute-cluster\"\nvariables[\"scenarios-collapsed\"]=\"gateway virt-hosts vmsnap-hacks hacks vmsnap-basic-cluster basic-cluster vmsnap-lb lb vmsnap-galera galera vmsnap-rabbitmq rabbitmq vmsnap-memcached memcached vmsnap-mongodb mongodb vmsnap-keystone keystone vmsnap-glance glance vmsnap-cinder cinder vmsnap-swift-aco swift-aco vmsnap-swift swift vmsnap-neutron-server neutron-server vmsnap-neutron-agents neutron-agents vmsnap-nova nova vmsnap-ceilometer ceilometer vmsnap-heat heat vmsnap-horizon horizon compute-common vmsnap-compute\"\n\nfunction create_phd_definition() {\n scenario=$1\n definition=$2\n snapshot_name=$3\n rm -f ${definition}\n\n nodes=${variables[\"nodes\"]}\n\n if [ \"x$nodes\" = x ]; then\n\tnodes=${nodeMap[$scenario]}\n fi\n\n if [ \"x$nodes\" = \"x\" ]; then\n\tfor n in `seq 1 3`; do\n\t nodes=\"$nodes rdo7-${scenario}${n}.vmnet\"\n\tdone\n fi\n\n nodelist=\"nodes=\"\n for node in $nodes; do\n\tnodelist=\"${nodelist}${node}.${variables[\"network_domain\"]} \"\n done\n\n echo \"$nodelist\" >> ${definition}\n if [ -n \"${snapshot_name}\" ]; then\n echo \"snapshot_name=${snapshot_name}\" >> ${definition}\n fi\n cat ${definition}\n}\n\ngenerate=0\nscenarios=\"\"\n\nwhile true ; do\n case \"$1\" in\n\t--help|-h|-\\?) \n\t echo \"$0 \"\n\t exit 0;;\n\t-n|--node) variables[\"nodes\"]=\"${variables[\"nodes\"]} $2\"; shift; shift;;\n\t-c|--collapsed) variables[\"deployment\"]=\"collapsed\"; shift;;\n\t-s|--segregated) variables[\"deployment\"]=\"segregated\"; shift;;\n\t-f|--from) fromscenario=$2; shift; shift;;\n\t-t|--to) toscenario=$2; shift; shift;;\n\t-S|--status) variables[\"status\"]=1; shift;;\n\t-g|--generate) generate=1; shift;;\n\t--mrg)\n\t variables[\"network_domain\"]=\"mpc.lab.eng.bos.redhat.com\"\n\t variables[\"config\"]=\"mrg\";\n\t nodeMap[\"baremetal\"]=\"mrg-01 mrg-02 mrg-03 mrg-04 mrg-07 mrg-08 mrg-09\"\n\t nodeMap[\"virt-hosts\"]=\"mrg-01 mrg-02 mrg-03 mrg-04\"\n\t nodeMap[\"vmsnap\"]=\"mrg-02 mrg-03 mrg-04\"\n\t nodeMap[\"compute-nodes\"]=\"mrg-07 mrg-08 mrg-09\"\n\t nodeMap[\"controller-managed\"]=\"rdo7-node1.vmnet rdo7-node2.vmnet rdo7-node3.vmnet mrg-07 mrg-08 mrg-09\"\n\t shift;;\n\t-m|--method) redeploy=$2; shift; shift;;\n\t-i|--instance) instance=$2; shift; shift;;\n\t-x) set -x ; shift;;\n\t--) shift ; break ;;\n\t-*) echo \"unknown option: $1\"; exit 1;;\n\t\"\") break;;\n\t*) scenarios=\"${scenarios} $1\"; shift;;\n esac\ndone\n\nif [ -z ${variables[\"config\"]} ]; then\n variables[\"config\"]=ha-${variables[\"deployment\"]}\nfi\n\nif [ ${variables[\"status\"]} = 1 ]; then\n if [ ${variables[\"deployment\"]} != \"collapsed\" ]; then\n\tscenarios=node\n\n elif [ \"x${scenarios}\" = x ]; then\n\tscenarios=${variables[\"components\"]}\n fi\n\n for scenario in $scenarios; do\n\tssh rdo7-${scenario}1.vmnet.${variables[\"network_domain\"]} -- crm_mon -1\n done\n exit 0\nfi\n\nif [ \"x${scenarios}\" = x ]; then\n deploy=scenarios-${variables[\"deployment\"]}\n scenarios=${variables[${deploy}]}\n if [ -z \"$redeploy\" ]; then\n\tredeploy=full\n fi\n case $redeploy in\n\trollback) scenarios=\"baremetal-rollback $scenarios\";;\n\tfull) scenarios=\"beaker baremetal $scenarios\";;\n\t*) echo \"unknown redeploy method\"; exit 1;;\n esac\n if [ -z \"$instance\" ]; then\n\tinstance=ha\n fi\n case $instance in\n\tha) scenarios=\"$scenarios compute-managed controller-managed\";;\n\tsingle) scenarios=\"$scenarios compute-cluster\";;\n *) echo \"unknown instance value\"; exit 1;;\n esac\nfi\n\nfunction run_phd() {\n \n if [ ${generate} = 1 ]; then\n\tscripts=$(phd_exec -s ./pcmk/${1}.scenario -d ${HOME}/phd.${scenario}.conf -V ./pcmk/${variables[\"config\"]}.variables -p | grep PHD_SCPT | sort | awk -F= '{print $2}')\n\tfor script in $scripts; do\n\t echo \"#### $script\"\n\t more \"$script\"\n\tdone\n else\n\tphd_exec -s ./pcmk/${1}.scenario -d ${HOME}/phd.${scenario}.conf -V ./pcmk/${variables[\"config\"]}.variables\n fi\n}\n\ninscenario=0\n\nfor scenario in $scenarios; do\n\n if [ -n \"$fromscenario\" ]; then\n\tif [ \"$fromscenario\" = \"${scenario}\" ]; then\n\t\tinscenario=1\n\tfi\n\tif [ \"$inscenario\" = 0 ]; then\n\t\tcontinue;\n\tfi\n fi\n\n snapshot_name=\"\"\n if [ ${variables[\"deployment\"]} = \"collapsed\" ]; then\n\tcase $scenario in\n\t vmsnap-rollback-*)\n\t\tsnapshot_name=$(echo $scenario | sed -e 's/vmsnap-rollback-//g')\n\t\tscenario=vmsnap-rollback\n\t\tnodeMap[$scenario]=${nodeMap[vmsnap]}\n\t\t;;\n\t vmsnap-*)\n\t\tsnapshot_name=$(echo $scenario | sed -e 's/vmsnap-//g')\n\t\tscenario=vmsnap\n\t\t;;\n\t compute-*)\n\t\tnodeMap[$scenario]=${nodeMap[compute-nodes]}\n\t\t;;\n\t beaker|baremetal|gateway)\n\t\tnodeMap[$scenario]=${nodeMap[baremetal]}\n\t\t;;\n\t baremetal-rollback)\n\t\t# stupid hack to avoid spending heaps of time waiting for nfs\n\t\t# server timeout\n\t\tlist=\"\"\n\t\tfor x in ${nodeMap[baremetal]}; do\n\t\t\tlist=\"$x $list\"\n\t\tdone\n\t\tnodeMap[$scenario]=\"$list\"\n\t\t;;\n\t virt-hosts|controller-managed)\n\t\t;;\n\t *) \n\t\t# Overwrite the node list to be the nodes of our collapsed cluster\n\t\tnodeMap[$scenario]=\"rdo7-node1.vmnet rdo7-node2.vmnet rdo7-node3.vmnet\"\n\t\t;;\n\tesac\n fi\n\n create_phd_definition ${scenario} ${HOME}/phd.${scenario}.conf ${snapshot_name}\n\n if [ x${cluster[${scenario}]} = x0 ]; then\n\t: no need to bootstrap a cluster\n\n elif [ ${variables[\"deployment\"]} != \"collapsed\" ]; then\n\t: prep a new cluster for ${scenario}\n\techo \"$(date) :: Initializing cluster for scenario $scenario\"\n\trun_phd basic-cluster\n fi\n\n echo \"$(date) :: Beginning scenario $scenario\"\n run_phd ${scenario}\n\n if [ -n \"$toscenario\" ] && [ \"$toscenario\" = ${scenario} ]; then\n\techo \"$(date) :: Reached $scenario. Stop processing\"\n\tbreak;\n fi\n\ndone\n" }, { "path": "full-stack.md", "content": "This is what the result of `pcs status` should look like:\n\n Cluster name: rhos-node\n Last updated: Fri Mar 6 22:06:28 2015\n Last change: Fri Mar 6 22:03:52 2015\n Stack: corosync\n Current DC: rhos6-node2 (2) - partition with quorum\n Version: 1.1.12-a14efad\n 3 Nodes configured\n 121 Resources configured\n \n \n Online: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n \n Full list of resources:\n \n fence1\t(stonith:fence_xvm):\tStarted rhos6-node1 \n fence2\t(stonith:fence_xvm):\tStarted rhos6-node2 \n fence3\t(stonith:fence_xvm):\tStarted rhos6-node3 \n Clone Set: lb-haproxy-clone [lb-haproxy]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n vip-db\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node1 \n vip-qpid\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node2 \n vip-keystone\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node3 \n vip-glance\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node1 \n vip-cinder\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node2 \n vip-swift\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node3 \n vip-neutron\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node1 \n vip-nova\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node2 \n vip-horizon\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node3 \n vip-heat\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node1 \n vip-ceilometer\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node2 \n vip-rabbitmq\t(ocf::heartbeat:IPaddr2):\tStarted rhos6-node3 \n Master/Slave Set: galera-master [galera]\n Masters: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: mongodb-clone [mongodb]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: memcached-clone [memcached]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: rabbitmq-server-clone [rabbitmq-server]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: keystone-clone [keystone]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: glance-fs-clone [glance-fs]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: glance-registry-clone [glance-registry]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: glance-api-clone [glance-api]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: cinder-api-clone [cinder-api]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: cinder-scheduler-clone [cinder-scheduler]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n cinder-volume\t(systemd:openstack-cinder-volume):\tStarted rhos6-node1 \n Clone Set: swift-fs-clone [swift-fs]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: swift-account-clone [swift-account]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: swift-container-clone [swift-container]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: swift-object-clone [swift-object]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: swift-proxy-clone [swift-proxy]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n swift-object-expirer\t(systemd:openstack-swift-object-expirer):\tStarted rhos6-node2 \n Clone Set: neutron-server-clone [neutron-server]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: neutron-scale-clone [neutron-scale] (unique)\n neutron-scale:0\t(ocf::neutron:NeutronScale):\tStarted rhos6-node3 \n neutron-scale:1\t(ocf::neutron:NeutronScale):\tStarted rhos6-node1 \n neutron-scale:2\t(ocf::neutron:NeutronScale):\tStarted rhos6-node2 \n Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: neutron-l3-agent-clone [neutron-l3-agent]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n ceilometer-central\t(systemd:openstack-ceilometer-central):\tStarted rhos6-node3 \n Clone Set: ceilometer-collector-clone [ceilometer-collector]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: ceilometer-api-clone [ceilometer-api]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: ceilometer-delay-clone [ceilometer-delay]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: ceilometer-alarm-evaluator-clone [ceilometer-alarm-evaluator]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: ceilometer-alarm-notifier-clone [ceilometer-alarm-notifier]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: ceilometer-notification-clone [ceilometer-notification]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: heat-api-clone [heat-api]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: heat-api-cfn-clone [heat-api-cfn]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n Clone Set: heat-api-cloudwatch-clone [heat-api-cloudwatch]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n heat-engine\t(systemd:openstack-heat-engine):\tStarted rhos6-node1 \n Clone Set: horizon-clone [horizon]\n Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]\n \n PCSD Status:\n rhos6-node1: Online\n rhos6-node2: Online\n rhos6-node3: Online\n \n Daemon Status:\n corosync: active/enabled\n pacemaker: active/enabled\n pcsd: active/enabled\n" }, { "path": "ha-openstack.md", "content": "# Highly Available Openstack Deployments\n\nThe current target for this document is RDO 10, based on\nthe OpenStack Newton release.\n\nLooking for an edition prior to Newton (RDO10)?\n\nCheck out the [Juno-RDO6](../Juno-RDO6/ha-openstack.md) or\n[Mitaka-RDO9](../Mitaka-RDO9/ha-openstack.md) branches instead.\n\n## Purpose of this Document\n\nThis document aims at defining a high level architecture for a highly\navailable RHEL OSP setup with the [Pacemaker](http://clusterlabs.org)\ncluster manager which provides:\n\n- detection and recovery of machine and application-level failures\n- startup/shutdown ordering between applications\n- preferences for other applications that must/must-not run on the same machine\n- provably correct response to any failure or cluster state\n\nIt is important to understand the following definitions used to\ndescribe the operational mode of services in a cluster:\n\n- Active/active \n\n Traffic intended for the failed node is either passed onto an\n existing node or load balanced across the remaining nodes. This is\n usually only possible when the nodes use a homogeneous software\n configuration.\n\n- Active/passive\n\n Provides a fully redundant instance of each node, which is only\n brought online when its associated primary node fails. This\n configuration typically requires the most extra hardware.\n\nIn this document, all components are currently modelled as\nactive/active with the exception of:\n\n- cinder-volume\n\nImplementation details are contained in scripts linked to from the main document.\nRead them carefully before considering to run them in your own environment. \n\n## Historical Context\n\nIn the previous OpenStack HA architectures used by Red Hat, SuSE and\nothers, Systemd is the entity in charge of starting and stopping most\nOpenStack services. Pacemaker exists as a layer on top, signalling\nwhen this should happen, but Systemd is the part making it happen.\n\nThis is a valuable contribution for active/passive (A/P) services and\nthose that require all their dependancies be available during their\nstartup and shutdown sequences. However as OpenStack has matured, more\nand more components are able to operate in an unconstrained\nactive/active capacity with little regard for the startup/shutdown\norder of their peers or dependancies - making them well suited to be\nmanaged exclusively by Systemd.\n\n## Overall Design\n\nWith Newton, OpenStack has reached the point where it is now a good\nidea to limit Pacemaker’s involvement to core services like Galera and\nRabbit as well as the few remaining OpenStack services, such as\ncinder-volume, that run A/P.\n\nThis will be particularly useful as we look towards a containerised\nfuture. It both allows OpenStack to play nicely with the current\ngeneration of container managers which lack Orchestration\ncapabilities, as well as reducing recovery and down time by allowing\nfor the maximum possible parallelism.\n\nAny objections to this architecture usually fall into one of three\nmain categories:\n\n1. The use of Pacemaker as an alerting mechanism\n1. The idea that Pacemaker provides better monitoring of systemd services\n1. A believe that active/passive installations are suprior\n\nIf these concerns apply to you then, as the founding author of\nPacemaker, I would like to direct your attention to my\n[post](http://blog.clusterlabs.org/blog/2016/next-openstack-ha-arch)\nwhich will attempt to disuade you of their relevance.\n\nThis reference design is based around a single cluster of 3 or more\nnodes on which every component is running.\n \nThis scenario can be visualized as below:\n \n ![Collapsed deployment architecture](Cluster-deployment-collapsed.png)\n\nWith the advent of composable roles however, it is certainly possible\nto dedicate a subset of nodes for one or more components that are\nexpected to be a bottleneck.\n\nIt is also possible that these dedicated nodes run extra copies of\nthose service, in addition to the ones on a fully symmetrical core set\nof nodes.\n\n## Assumptions\n\nIt is required that the clusters contain at least three nodes so that\nwe take advantage of\n[quorum](http://en.wikipedia.org/wiki/Quorum_(Distributed_Systems))\n\nQuorum becomes important when a failure causes the cluster to split in\ntwo or more paritions. In this situation, you want the majority to\nensure the minority are truely dead (through fencing) and continue to\nhost resources. For a two-node cluster, no side has the majority and\nyou can end up in a situations where both sides fence each other, or\nboth sides are running the same services - leading to data corruption.\n\nClusters with an even number of hosts suffer from similar issues - a\nsingle network failure could easily cause a N:N split where neither\nside retains a majority. For this reason, we recommend an odd number\nof cluster members when scaling up.\n\nYou can have up to 16 cluster members (this is currently limited by\ncorosync's ability to scale higher). In extreme cases, 32 and even up\nto 64 nodes could be possible however this is not well tested.\n\nIn some environments, the available IP address range of the public LAN\nis limited. If this applies to you, you will need one additional node\nto set up as a [gateway](pcmk/gateway.scenario) that will provide DNS\nand DHCP for the guests containing the OpenStack services and expose\nthe required nova and horizon APIs to the external network.\n\n## Solution Components\n\n### Cluster Manager\n\nAt its core, a cluster is a distributed finite state machine capable\nof co-ordinating the startup and recovery of inter-related services\nacross a set of machines.\n\nEven a distributed and/or replicated application that is able to\nsurvive failures on one or more machines can benefit from a\ncluster manager:\n\n1. Awareness of other applications in the stack\n \n While SYS-V init replacements like systemd can provide\n deterministic recovery of a complex stack of services, the\n recovery is limited to one machine and lacks the context of what\n is happening on other machines - context that is crucial to\n determine the difference between a local failure, clean startup\n and recovery after a total site failure.\n\n1. Awareness of instances on other machines\n\n Services like RabbitMQ and Galera have complicated boot-up\n sequences that require co-ordination, and often serialization, of\n startup operations across all machines in the cluster. This is\n especially true after site-wide failure or shutdown where we must\n first determine the last machine to be active.\n \n1. A shared implementation and calculation of [quorum](http://en.wikipedia.org/wiki/Quorum_%28Distributed_Systems%29)\n\n It is very important that all members of the system share the same\n view of who their peers are and whether or not they are in the\n majority. Failure to do this leads very quickly to an internal\n [split-brain](https://en.wikipedia.org/wiki/Split-brain_(computing))\n state - where different parts of the system are pulling in\n different and incompatioble directions.\n\n1. Data integrity through fencing (a non-responsive process does not imply it is not doing anything)\n\n A single application does not have sufficient context to know the\n difference between failure of a machine and failure of the\n applcation on a machine. The usual practice is to assume the\n machine is dead and carry on, however this is highly risky - a\n rogue process or machine could still be responding to requests and\n generally causing havoc. The safer approach is to make use of\n remotely accessible power switches and/or network switches and SAN\n controllers to fence (isolate) the machine before continuing.\n\n1. Automated recovery of failed instances\n \n While the application can still run after the failure of several\n instances, it may not have sufficient capacity to serve the\n required volume of requests. A cluster can automatically recover\n failed instances to prevent additional load induced failures.\n\n\nFor this reason, the use of a cluster manager like\n[Pacemaker](http://clusterlabs.org) is highly recommended.\n\n### Proxy server\n\nAlmost all services in this stack are proxied.\nUsing a proxy server provides:\n\n1. Load distribution\n \n Many services can act in an active/active capacity, however they\n usually require an external mechanism for distributing requests to\n one of the available instances. The proxy server can serve this\n role.\n\n1. API isolation\n \n By sending all API access through the proxy, we can clearly\n identify service interdependancies. We can also move them to\n locations other than `localhost` to increase capacity if the need\n arises.\n\n1. Simplified process for adding/removing of nodes\n \n Since all API access is directed to the proxy, adding or removing\n nodes has no impact on the configuration of other services. This\n can be very useful in upgrade scenarios where an entirely new set\n of machines can be configured and tested in isolation before\n telling the proxy to direct traffic there instead.\n\n1. Enhanced failure detection\n\n The proxy can be configured as a secondary mechanism for detecting\n service failures. It can even be configured to look for nodes in\n a degraded state (such as being 'too far' behind in the\n replication) and take them out of circulation.\n\nThe following components are currently unable to benefit from the use\nof a proxy server:\n\n- RabbitMQ\n- memcached\n- mongodb\n\nHowever the reasons vary and are discussed under each component's\nheading.\n\nWe recommend HAProxy as the load balancer, however there are many\nalternatives in the marketplace.\n\nWe use a check interval of 1 second however the timeouts vary by service.\n\nGenerally we use round-robin to distriute load amongst instances of\nactive/active services, however Galera and Qpid use the `stick-table`\noptions to ensure that incoming connections to the virtual IP (VIP)\nshould be directed to only one of the available backends.\n\nIn Galera's case, although it can run active/active, this helps avoid\nlock contention and prevent deadlocks. It is used in combination with\nthe `httpchk` option that ensures only nodes that are in sync with its\npeers are allowed to handle requests.\n\nQpid however operates in a active/passive configuration, no built-in\nclustering, so in it's case the `stick-table` option ensures that all\nrequests go to the active instance.\n\n### Replicated Database\n\nMost OpenStack components require access to a database.\n\nTo avoid the database being a single point of failure, we require that\nit be replicated and the ability to support multiple masters can help\nwhen trying to scale other components.\n\nOne of the most popular database choices is Galera for MySQL, it supports:\n\n- Synchronous replication\n- active/active multi-master topology\n- Automatic node joining\n- True parallel replication, on row level\n- Direct client connections, native MySQL look & feel\n\nand claims:\n\n- No slave lag\n- No lost transactions\n- Both read and write scalability\n- Smaller client latencies\n\nAlthough galera supports active/active configurations, we recommend\nactive/passive (enforced by the load balancer) in order to avoid lock\ncontention.\n\n### Database Cache\n\nMemcached is a general-purpose distributed memory caching system. It\nis used to speed up dynamic database-driven websites by caching data\nand objects in RAM to reduce the number of times an external data\nsource must be read.\n\n__Note__: Access to memcached is not handled by HAproxy because\nreplicated access is currently only in an experimental state. Instead\nconsumers must be supplied with the full list of hosts running\nmemcached.\n\n### Message Bus\n\nAn AMQP (Advanced Message Queuing Protocol) compliant message bus is\nrequired for most OpenStack components in order to co-ordinate the\nexecution of jobs entered into the system.\n\nRabbitMQ and Qpid are common deployment options. Both support:\n\n- reliable message delivery\n- flexible routing options\n- replicated queues\n\nThis guide assumes RabbitMQ is being deployed, however we also\n[document Qpid (TODO)](pcmk/osp-qpid.scenario) for completeness. Pay\nattention to the comments in that guide for how selecting `Qpid` affects\nthe rest of the configuration.\n\n__Note__: Access to RabbitMQ is not handled by HAproxy. Instead\nconsumers must be supplied with the full list of hosts running\nRabbitMQ with `rabbit_hosts` and `rabbit_ha_queues` options.\n\nJock Eck found the [core\nissue](http://people.redhat.com/jeckersb/private/vip-failover-tcp-persist.html)\nand went into some detail regarding the [history and\nsolution](http://john.eckersberg.com/improving-ha-failures-with-tcp-timeouts.html)\non his blog.\n\nIn summary though:\n\n> The source address for the connection from HAProxy back to the\n> client is the VIP address. However the VIP address is no longer\n> present on the host. This means that the network (IP) layer deems\n> the packet unroutable, and informs the transport (TCP) layer. TCP,\n> however, is a reliable transport. It knows how to handle transient\n> errors and will retry. And so it does.\n\nIn this case that is a problem though, because:\n\n> TCP generally holds on to hope for a long time. A ballpark estimate\n> is somewhere on the order of tens of minutes (30 minutes is commonly\n> referenced). During this time it will keep probing and trying to\n> deliver the data.\n>\n> It's important to note that HAProxy has no idea that any of this is\n> happening. As far as its process is concerned, it called write()\n> with the data and the kernel returned success.\n\nThe [resolution](https://review.openstack.org/#/c/146047/) is already\nunderstood and just needs to make its way through review.\n\n## Core OpenStack services\n\nIn contrast to earlier versions of this guide, with the exception of\nCinder Volume, there are no specific instructions with regards to the\ninstallation core OpenStack services beyond:\n\n1. Ensuring services that make use of RabbitMQ list all configured servers\n1. Accessing Galera and all OpenStack peer APIs (keystone, etc) via the HAProxy and the VIPs \n\nIn all other respects, one should follow standard practices for\ninstalling packages and instructing the system to start them at boot\ntime.\n\n### Cinder\n\nCinder provides 'block storage as a service' suitable for performance\nsensitive scenarios such as databases, expandable file systems, or\nproviding a server with access to raw block level storage.\n\nPersistent block storage can survive instance termination and can also\nbe moved across instances like any external storage device. Cinder\nalso has volume snapshots capability for backing up the volumes.\n\nIn theory cinder can be run as active/active however there are\ncurrently sufficient concerns that cause us to recommend running the\nvolume component as active/passive only.\n\nJon Bernard writes:\n\n> Requests are first seen by Cinder in the API service, and we have a\n> fundamental problem there - a standard test-and-set race condition\n> exists for many operations where the volume status is first checked\n> for an expected status and then (in a different operation) updated to\n> a pending status. The pending status indicates to other incoming\n> requests that the volume is undergoing a current operation, however it\n> is possible for two simultaneous requests to race here, which\n> undefined results.\n> \n> Later, the manager/driver will receive the message and carry out the\n> operation. At this stage there is a question of the synchronization\n> techniques employed by the drivers and what guarantees they make.\n> \n> If cinder-volume processes exist as different process, then the\n> 'synchronized' decorator from the lockutils package will not be\n> sufficient. In this case the programmer can pass an argument to\n> synchronized() 'external=True'. If external is enabled, then the\n> locking will take place on a file located on the filesystem. By\n> default, this file is placed in Cinder's 'state directory' in\n> /var/lib/cinder so won't be visible to cinder-volume instances running\n> on different machines.\n> \n> However, the location for file locking is configurable. So an\n> operator could configure the state directory to reside on shared\n> storage. If the shared storage in use implements unix file locking\n> semantics, then this could provide the requisite synchronization\n> needed for an active/active HA configuration.\n> \n> The remaining issue is that not all drivers use the synchronization\n> methods, and even fewer of those use the external file locks.\n> A sub-concern would be whether they use them correctly.\n\nYou can read more about these concerns on the [Red Hat\nBugzilla](https://bugzilla.redhat.com/show_bug.cgi?id=1193229) and\nthere is a [psuedo roadmap](https://etherpad.openstack.org/p/cinder-kilo-stabilisation-work)\nfor addressing them upstream.\n\n# Implementation\n\nThe best way to visualize the result of this architecture is to make\nuse of\n[tripleo-quickstart](https://github.com/openstack/tripleo-quickstart/blob/master/README.rst)\nwhich implements the described architecture.\n\nThis will take a bare metal installation of your favorite OS (surely CentOS 7.2) and:\n\n1. create a 'stack' user\n1. create several VMs representing the undercloud, control plane and computes\n1. deploy the undercloud (TripleO uses a pre-rolled OpenStack image as a means for deploying and updating the user facing installation of OpenStack aka. the overcloud) \n1. deploy the overcloud for you to investigate and compare your existing architecture against\n\nYou should digest the entire [README](https://github.com/openstack/tripleo-quickstart/blob/master/README.rst), however the quick version for the purposes of seeing this HA architecture in action is:\n\n git clone git@github.com:openstack/tripleo-quickstart.git\n cd tripleo-quickstart\n ./quickstart.sh -b -n -w $PWD -c config/general_config/ha.yml -p quickstart-extras.yml -r quickstart-extras-requirements.txt --tags all -R newton -T all ${the_machine_you_wish_to_install_to}\n\nFor those that would prefer not to deal with TripleO, you can see\nroughly what TripleO does by examining the pseudo code for manually:\n\n1. configuring a basic [pacemaker cluster](pcmk/basic-cluster.scenario)\n1. deploying the [load balancer](pcmk/lb.scenario)\n1. deploying [galera](pcmk/galera.scenario)\n1. deploying [memcached](pcmk/memcached.scenario)\n1. deploying [rabbitmq](pcmk/rabbitmq.scenario)\n1. deploying [cinder volume](pcmk/cinder.scenario)\n\nHere is a [list of variables](pcmk/ha-collapsed.variables) used when\nexecuting the referenced scripts. Modify them to your needs.\n\n## Disclaimer \n\n- The referenced scripts contain many comments and warnings - READ THEM CAREFULLY.\n- There are probably 2^8 other ways to deploy this same scenario. This is only one of them.\n- Due to limited number of available physical LAN connections in the test setup, the instance IP traffic overlaps with the internal/management network.\n- Distributed/Shared storage is provided via NFS from the commodity server due to lack of dedicated CEPH servers. Any other kind of storage supported by OpenStack would work just fine.\n- Bare metal could be used in place of any or all guests.\n- Most of the scripts contain shell expansion to automatically fill in some values. Use your common sense when parsing data. Example:\n\n `openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address $(ip addr show dev vmnet0 scope global | grep inet | sed -e 's#.*inet ##g' -e 's#/.*##g')`\n\n means that we want the IP address from vmnet0 as vncserver_proxyclient_address.\n\n# Compute Nodes \n\nWe will usually need more than 16 compute nodes\nwhich is beyond Corosync's ability to manage. So in order monitor the\nhealthiness of compute nodes and the services running on them, we\npreviously had to create single node clusters.\n\nThe current deployment model allows Pacemaker to continue this role,\nbut presents a single coherent view of the entire deployment while\nallowing us to scale beyond corosync's limits. Having this single\nadministrative domain then allows us to do clever things like\nautomated recovery of VMs running on a failed or failing compute node.\n\nThe main difference with the previous deployment mode is that services\non the compute nodes are now managed and driven by the Pacemaker\ncluster on the control plane. The compute nodes do not become full\nmembers of the cluster and they no longer require the full cluster\nstack, instead they run pacemaker_remoted which acts as a conduit.\n\n> Implementation Details:\n>\n> - Pacemaker monitors the connection to pacemaker_remoted to verify\n> that the node is reachable or not. Failure to talk to a node\n> triggers recovery action.\n>\n> - Pacemaker uses pacemaker_remoted to start compute node services in\n> the same sequence as before (neutron-ovs-agent ->\n> ceilometer-compute -> nova-compute).\n>\n> - If a service fails to start, any services that depend on the\n> FAILED service will not be started. This avoids the issue of \n> adding a broken node (back) to the pool.\n> \n> - If a service fails to stop, the node where the service is running\n> will be fenced. This is necessary to guarantee data integrity and\n> a core HA concept (for the purposes of this particular discussion,\n> please take this as a given).\n> \n> - If a service's health check fails, the resource (and anything that\n> depends on it) will be stopped and then restarted. Remember that\n> failure to stop will trigger a fencing action.\n>\n> - A successful restart of all the services can only potentially\n> affect network connectivity of the instances for a short period of\n> time.\n\nWith these capabilities in place, we can exploit Pacemaker's node\nmonitoring and fencing capabilities to drive nova host-evacuate for\nthe failed compute nodes and recover the VMs elsewhere.\n\nWhen a compute node fails, Pacemaker will:\n\n1. Execute 'nova service-disable'\n2. fence (power off) the failed compute node\n3. fence_compute off (waiting for nova to detect compute node is gone)\n4. fence_compute on (a no-op unless the host happens to be up already)\n5. Execute 'nova service-enable' when the compute node returns\n\nTechnically steps 1 and 5 are optional and they are aimed to improve\nuser experience by immediately excluding a failed host from nova\nscheduling.\n\nThe only benefit is a faster scheduling of VMs that happens during a\nfailure (nova does not have to recognize a host is down, timeout and\nsubsequently schedule the VM on another host).\n\nStep 2 will make sure the host is completely powered off and nothing\nis running on the host. Optionally, you can have the failed host\nreboot which would potentially allow it to re-enter the pool.\n\nWe have an implementation for Step 3 but the ideal solution depends on\nextensions to the nova API. Currently fence_compute loops, waiting\nfor nova to recognise that the failed host is down, before we make a\nhost-evacuate call which triggers nova to restart the VMs on another\nhost. The discussed nova API extensions will speed up recovery times\nby allowing fence_compute to proactively push that information into\nnova instead.\n\n\nTo take advantage of the VM recovery features:\n\n- VMs need to be running off a cinder volume or using shared ephemeral\n storage (like RBD or NFS)\n\n- If VM is not running using shared storage, recovery of the instance\n on a new compute node would need to revert to a previously stored\n snapshot/image in Glance (potentially losing state, but in some\n cases that may not matter)\n\n- RHEL7.1+ required for infrastructure nodes (controllers and\n compute). Instance guests can run anything.\n\n- Compute nodes need to have a working fencing mechanism (IPMI,\n hardware watchdog, etc)\n\n## Compute Node Implementation\n\nStart by creating a minimal CentOS __7__ installation on at least one node.\n\nOnce the machine(s) have been installed, [prepare\nthem](pcmk/baremetal.scenario) for hosting OpenStack.\n\nNext, you can configure them as [compute nodes](pcmk/compute-common.scenario).\n\nWe now add them to the cluster as [partial members](pcmk/compute-managed.scenario).\n\nOnce the compute nodes are configured as remote, they can be added\nto the [controller backplane](pcmk/controller-managed.scenario)\n\n> TODO: what if nova-compute fails to restart and there are scheduled\n> instances? Those can still be accessed from outside but cannot be\n> managed by nova. This might warrant a host-evacuate.\n>\n> Traditionally, HA systems would fence the node at this point.\n\n\n\n" }, { "path": "iha-install-10.sh", "content": "#!/bin/bash\n\nset -ex\n#\n#If your deployment includes this use case, include the no_shared_storage=1 option in step 7.\n\n\nhelper=~/.ssh/config\ncat <> $helper\nHost overcloud-*\n BatchMode yes\n User heat-admin\n StrictHostKeyChecking no\nEOF\n\nsource stackrc\n\nnova list | grep ctlplane | awk -F\\| '{ print \"Host \"$3 \"\\n\\tHostName \" $7}' | sed 's/ctlplane=//' >> $helper\nchmod 600 $helper\n\nCOMPUTES=$(nova list | grep novacompute | awk -F\\| '{ print $3}' | tr '\\n' ' ')\nCONTROLLERS=$(nova list | grep controller | awk -F\\| '{ print $3}' | tr '\\n' ' ')\n\nFIRST_COMPUTE=$(echo $COMPUTES | awk '{print $1}')\nFIRST_CONTROLLER=$(echo $CONTROLLERS | awk '{print $1}')\n\n# 15. Add stonith devices for the Compute nodes. fence_ironic is not\n# a recommended approach, but is usable everywhere which makes it\n# ideal for generic scripts like this\n\nif [ 1 = 1 ]; then\n helper=iha-helper-configure-virt-fencing.sh\n cat < $helper\ncp fence_ironic.py /usr/sbin/fence_ironic\nsudo chmod a+x /usr/sbin/fence_ironic\nEOF\n for node in $CONTROLLERS; do scp $helper fence_ironic.py ${node}: ; done\n for node in $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done\n\n hostmap=\"\";\n hostlist=$(nova list | grep ctlplane | awk -F\\| '{print $3}')\n for host in $hostlist; do hostmap=\"$hostmap $host:$(ironic node-list | grep $(nova list | grep $host | awk '{print $2}') | awk '{print $4}' )\"; done\n\n ssh ${FIRST_CONTROLLER} -- sudo pcs stonith create shooter fence_ironic auth-url=${OS_AUTH_URL} login=${OS_USERNAME} passwd=${OS_PASSWORD} tenant-name=${OS_TENANT_NAME} pcmk_host_map=\\\"${hostmap}\\\" op monitor interval=60s timeout=180s\nfi\n\n# 1. Begin by stopping and disabling libvirtd and all OpenStack services on the Compute nodes:\n\nsource overcloudrc\n\nssh ${FIRST_CONTROLLER} -- sudo pcs property set stonith-enabled=false \n\nhelper=iha-helper-stop-services.sh\ncat < $helper\nset -ex\nfor s in openstack-nova-compute neutron-openvswitch-agent libvirtd; do \n systemctl stop \\${s}\n systemctl disable \\${s}\ndone\n\n# Punch a hole for pacemaker-remote\niptables -I INPUT -p tcp --dport 3121 -j ACCEPT\nservice iptables save\n\nEOF\nfor node in $COMPUTES; do scp $helper ${node}: ; done\nfor node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done\n\n# 2. Create an authentication key for use with pacemaker-remote.\n\ndd if=/dev/urandom of=./authkey bs=4096 count=1\n\n# 3. Copy this key to the director node, and then to the remaining Compute and Controller nodes:\n\nhelper=iha-helper-fix-auth.sh\ncat < $helper\nset -ex\n\nmkdir -p --mode=0750 /etc/pacemaker/\nchgrp haclient /etc/pacemaker\nmv authkey /etc/pacemaker/\nchown root:haclient /etc/pacemaker/authkey\nEOF\nfor node in $COMPUTES $CONTROLLERS; do scp ./authkey $helper ${node}: ; done\nfor node in $COMPUTES $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done\n\n# 4. Enable pacemaker-remote on all Compute nodes:\n\nfor compute in $COMPUTES; do ssh ${compute} -- sudo systemctl enable pacemaker_remote; done\nfor compute in $COMPUTES; do ssh ${compute} -- sudo systemctl start pacemaker_remote; done\n\n# 5. Confirm that the required versions of the pacemaker (1.1.13-10.el7_2.2.x86_64), fence-agents (fence-agents-all-4.0.11-27.el7_2.5.x86_64) and resource-agents (3.9.5-54.el7_2.6.x86_64`) packages are installed on the controller and Compute nodes:\nfor compute in $COMPUTES; do ssh ${compute} -- rpm -qa | egrep '(pacemaker|fence-agents|resource-agents)' ; done\n\n# 7. Create a NovaEvacuate active/passive resource using the overcloudrc file to provide the auth_url, username, tenant and password values:\n# 8. Confirm that nova-evacuate is started after the floating IP resources, and the Image Service (glance), OpenStack Networking (neutron), Compute (nova) services:\n\nhelper=iha-helper-create-evacuate.sh\ncat < $helper\nset -ex\n\npcs resource create nova-evacuate ocf:openstack:NovaEvacuate auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME domain=localdomain no_shared_storage=1 op monitor interval=60s timeout=240s\nsudo pcs constraint order start haproxy-clone then nova-evacuate\nsudo pcs constraint order start galera-clone then nova-evacuate\nsudo pcs constraint order start rabbitmq-clone then nova-evacuate\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n# Note: If you are not using shared storage, include the no_shared_storage=1 option in your resource create ... command above. See Exception for shared storage for more information.\n\n# 10. Create a list of the current controllers using cibadmin data :\n\n#controller-1 # controllers=$(sudo cibadmin -Q -o nodes | grep uname | sed s/.*uname..// | awk -F\\\" '{print $1}')\n#controller-1 # echo $controllers\n\n# 11. Use this list to tag these nodes as controllers with the osprole=controller property:\n# 12. Build a list of stonith devices already present in the environment:\n# 13. Tag the control plane services to make sure they only run on the controllers identified above, skipping any stonith devices listed:\n\nhelper=iha-helper-tag-controllers.sh\ncat < $helper\nset -ex\n\nfor controller in ${CONTROLLERS}; do pcs property set --node \\${controller} osprole=controller ; done\nstonithdevs=\\$(pcs stonith | awk '{print \\$1}')\nfor i in \\$(cibadmin -Q --xpath //primitive --node-path | tr ' ' '\\n' | awk -F \"id='\" '{print \\$2}' | awk -F \"'\" '{print \\$1}' | uniq); do\n found=0\n if [ -n \"\\$stonithdevs\" ]; then\n for x in \\$stonithdevs; do\n if [ \\$x = \\$i ]; then\n found=1\n fi\n done\n fi\n if [ \\$found = 0 ]; then\n pcs constraint location \\$i rule resource-discovery=exclusive score=0 osprole eq controller\n fi\ndone\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n# 14. Begin to populate the Compute node resources within pacemaker, starting with neutron-openvswitch-agent:\n\nhelper=fix-compute.sh\ncat < $helper\nset -x\ngrep crudini /usr/lib/ocf/resource.d/openstack/nova-compute-wait\nif [ \\$? != 0 ]; then\n set -e\n patch -p1 /usr/lib/ocf/resource.d/openstack/nova-compute-wait \\${PWD}/bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch\nfi\nEOF\nfor node in $COMPUTES; do scp bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch $helper ${node}: ; done\nfor node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done\n\nsource overcloudrc\n\nhelper=iha-helper-create-compute-resources.sh\ncat < $helper\nset -ex\n\npcs resource create neutron-openvswitch-agent-compute systemd:neutron-openvswitch-agent op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force\npcs constraint location neutron-openvswitch-agent-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute\n\n# Then the Compute libvirtd resource:\n\npcs resource create libvirtd-compute systemd:libvirtd op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force\npcs constraint location libvirtd-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute\npcs constraint order start neutron-openvswitch-agent-compute-clone then libvirtd-compute-clone\npcs constraint colocation add libvirtd-compute-clone with neutron-openvswitch-agent-compute-clone\n\n# Then the nova-compute resource:\n\npcs resource create nova-compute-checkevacuate ocf:openstack:nova-compute-wait auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME domain=localdomain op start timeout=300 --clone interleave=true --disabled --force\n\npcs constraint location nova-compute-checkevacuate-clone rule resource-discovery=exclusive score=0 osprole eq compute\npcs resource create nova-compute systemd:openstack-nova-compute op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force\npcs constraint location nova-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute\npcs constraint order start nova-compute-checkevacuate-clone then nova-compute-clone require-all=true\npcs constraint order start nova-compute-clone then nova-evacuate require-all=false\npcs constraint order start libvirtd-compute-clone then nova-compute-clone\npcs constraint colocation add nova-compute-clone with libvirtd-compute-clone\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n# 16. Create a seperate fence-nova stonith device:\n\nhelper=iha-helper-create-nova-fence.sh\ncat < $helper\nset -ex\n\npcs stonith create fence-nova fence_compute auth-url=$OS_AUTH_URL login=$OS_USERNAME passwd=$OS_PASSWORD tenant-name=$OS_TENANT_NAME domain=localdomain record-only=1 op monitor interval=60s timeout=180s --force \nEOF\nsource stackrc\n\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n# 17. Make certain the Compute nodes are able to recover after fencing:\n\nssh ${FIRST_CONTROLLER} -- sudo pcs property set cluster-recheck-interval=1min\n\n# 18. Create Compute node resources and set the stonith level 1 to include both the nodes's physical fence device and fence-nova:\n\nhelper=iha-helper-create-computes.sh\ncat < $helper\nset -ex\n\nfor node in $COMPUTES; do\n pcs resource create \\${node} ocf:pacemaker:remote reconnect_interval=60 op monitor interval=20\n pcs property set --node \\${node} osprole=compute\n pcs stonith level add 1 \\${node} shooter,fence-nova\ndone\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n\n# 19. Enable the control and Compute plane services:\n\nhelper=iha-helper-enable-services.sh\ncat < $helper\nset -ex\n\npcs resource enable neutron-openvswitch-agent-compute\npcs resource enable libvirtd-compute\npcs resource enable nova-compute-checkevacuate\npcs resource enable nova-compute\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n# 20. Allow some time for the environment to settle before cleaning up any failed resources:\n\nhelper=iha-helper-cleanup.sh\ncat < $helper\nset -ex\n\nsleep 60\npcs resource cleanup\npcs status\necho pcs property set stonith-enabled=true\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n\n#Test High Availability\n\n#Note: These steps deliberately reboot the Compute node without warning.\n\n#1. The following step boots an instance on the overcloud, and then crashes the Compute node:\n\n#stack@director # . overcloudrc\n#stack@director # nova boot --image cirros --flavor 2 test-failover\n#stack@director # nova list --fields name,status,host\n#stack@director # . stackrc\n#stack@director # ssh -lheat-admin compute-n\n#compute-n # sudo su -\n#root@compute-n # echo c > /proc/sysrq-trigger\n#\n#2. A short time later, the instance should be restarted on a working Compute node:\n#\n#stack@director # nova list --fields name,status,host\n#stack@director # nova service-list\n\n# 3 ip6tables -I INPUT -p tcp --dport 3121 -j ACCEPT\n# 4 iptables -I INPUT -p tcp --dport 3121 -j ACCEPT\n# 5 service iptables save\n# 6 service ip6tables save\n" }, { "path": "iha-install-9.sh", "content": "#!/bin/bash\n\nset -ex\n#\n#If your deployment includes this use case, include the no_shared_storage=1 option in step 7.\n\n\nhelper=~/.ssh/config\ncat < $helper\nHost overcloud-*\n BatchMode yes\n User heat-admin\n StrictHostKeyChecking no\nEOF\n\nsource stackrc\n\nnova list | grep ctlplane | awk -F\\| '{ print \"Host \"$3 \"\\n\\tHostName \" $7}' | sed 's/ctlplane=//' >> $helper\nchmod 600 $helper\n\nCOMPUTES=$(nova list | grep novacompute | awk -F\\| '{ print $3}' | tr '\\n' ' ')\nCONTROLLERS=$(nova list | grep controller | awk -F\\| '{ print $3}' | tr '\\n' ' ')\n\nFIRST_COMPUTE=$(echo $COMPUTES | awk '{print $1}')\nFIRST_CONTROLLER=$(echo $CONTROLLERS | awk '{print $1}')\n\n# 15. Add stonith devices for the Compute nodes. fence_ironic is not\n# a recommended approach, but is usable everywhere which makes it\n# ideal for generic scripts like this\n\nif [ 1 = 1 ]; then\n helper=iha-helper-configure-virt-fencing.sh\n cat < $helper\ncp fence_ironic.py /usr/sbin/fence_ironic\nsudo chmod a+x /usr/sbin/fence_ironic\nEOF\n for node in $CONTROLLERS; do scp $helper fence_ironic.py ${node}: ; done\n for node in $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done\n\n hostmap=$(nova list | grep ctlplane | awk -F\\| '{print $3 \":\" $2}' | tr -d ' ' | tr '\\n' ' ')\n\n ssh ${FIRST_CONTROLLER} -- sudo pcs stonith create shooter fence_ironic auth-url=${OS_AUTH_URL} login=${OS_USERNAME} passwd=${OS_PASSWORD} tenant-name=${OS_TENANT_NAME} pcmk_host_map=\\\"${hostmap}\\\" op monitor interval=60s\nfi\n\n# 1. Begin by stopping and disabling libvirtd and all OpenStack services on the Compute nodes:\n\nsource overcloudrc\n\nssh ${FIRST_CONTROLLER} -- sudo pcs property set stonith-enabled=false \n\nhelper=iha-helper-stop-services.sh\ncat < $helper\nset -ex\nopenstack-service stop\nopenstack-service disable\nsystemctl stop libvirtd\nsystemctl disable libvirtd\nEOF\nfor node in $COMPUTES; do scp $helper ${node}: ; done\nfor node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done\n\n# 2. Create an authentication key for use with pacemaker-remote.\n\ndd if=/dev/urandom of=./authkey bs=4096 count=1\n\n# 3. Copy this key to the director node, and then to the remaining Compute and Controller nodes:\n\nhelper=iha-helper-fix-auth.sh\ncat < $helper\nset -ex\n\nmkdir -p --mode=0750 /etc/pacemaker/\nchgrp haclient /etc/pacemaker\nmv authkey /etc/pacemaker/\nchown root:haclient /etc/pacemaker/authkey\nEOF\nfor node in $COMPUTES $CONTROLLERS; do scp ./authkey $helper ${node}: ; done\nfor node in $COMPUTES $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done\n\n# 4. Enable pacemaker-remote on all Compute nodes:\n\nfor compute in $COMPUTES; do ssh ${compute} -- sudo systemctl enable pacemaker_remote; done\nfor compute in $COMPUTES; do ssh ${compute} -- sudo systemctl start pacemaker_remote; done\n\n# 5. Confirm that the required versions of the pacemaker (1.1.13-10.el7_2.2.x86_64), fence-agents (fence-agents-all-4.0.11-27.el7_2.5.x86_64) and resource-agents (3.9.5-54.el7_2.6.x86_64`) packages are installed on the controller and Compute nodes:\nfor compute in $COMPUTES; do ssh ${compute} -- rpm -qa | egrep '(pacemaker|fence-agents|resource-agents)' ; done\n\n# 6.a Apply the following constraint workarounds required for BZ#1257414:\n\n#Note: This issue has been addressed in RHSA-2015:1862 and might not be required for your environment.\n\n#controller-1 # sudo pcs constraint order start openstack-nova-novncproxy-clone then openstack-nova-api-clone\n#controller-1 # sudo pcs constraint order start rabbitmq-clone then openstack-keystone-clone\n#controller-1 # sudo pcs constraint order promote galera-master then openstack-keystone-clone\n#controller-1 # sudo pcs constraint order start haproxy-clone then openstack-keystone-clone\n#controller-1 # sudo pcs constraint order start memcached-clone then openstack-keystone-clone\n#controller-1 # sudo pcs constraint order promote redis-master then start openstack-ceilometer-central-clone require-all=false\n#controller-1 # sudo pcs resource defaults resource-stickiness=INFINITY\n\n#6.b Apply the following constraint workarounds required for BZ#1295835:\n\n#Note: This issue has been addressed in RHBA-2016:0264-1 and might not be required for your environment.\n\n#sudo pcs config | grep systemd | awk '{print $2}' | while read RESOURCE; do sudo pcs resource update $RESOURCE op start timeout=200s op stop timeout=200s; done\"\n\n# 7. Create a NovaEvacuate active/passive resource using the overcloudrc file to provide the auth_url, username, tenant and password values:\n# 8. Confirm that nova-evacuate is started after the floating IP resources, and the Image Service (glance), OpenStack Networking (neutron), Compute (nova) services:\n\nhelper=iha-helper-create-evacuate.sh\ncat < $helper\nset -ex\n\npcs resource create nova-evacuate ocf:openstack:NovaEvacuate auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME no_shared_storage=1 \nfor i in \\$(sudo pcs status | grep IP | awk '{ print \\$1 }'); do sudo pcs constraint order start \\$i then nova-evacuate ; done\nfor i in openstack-glance-api-clone neutron-metadata-agent-clone openstack-nova-conductor-clone; do sudo pcs constraint order start \\$i then nova-evacuate require-all=false ; done\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n# Note: If you are not using shared storage, include the no_shared_storage=1 option in your resource create ... command above. See Exception for shared storage for more information.\n\n# 9. Disable all OpenStack resources across the control plane:\n\n# Depending on the time needed to stop Identity Service (and on the power of your hardware) you can consider increasing the timeout period (--wait).\n# OSP8: ssh ${FIRST_CONTROLLER} -- sudo pcs resource disable openstack-keystone --wait=600s\n# OSP9+:\nssh ${FIRST_CONTROLLER} -- sudo pcs resource disable openstack-core-clone --wait=600s\n\n\n# 10. Create a list of the current controllers using cibadmin data :\n\n#controller-1 # controllers=$(sudo cibadmin -Q -o nodes | grep uname | sed s/.*uname..// | awk -F\\\" '{print $1}')\n#controller-1 # echo $controllers\n\n# 11. Use this list to tag these nodes as controllers with the osprole=controller property:\n# 12. Build a list of stonith devices already present in the environment:\n# 13. Tag the control plane services to make sure they only run on the controllers identified above, skipping any stonith devices listed:\n\nhelper=iha-helper-tag-controllers.sh\ncat < $helper\nset -ex\n\nfor controller in ${CONTROLLERS}; do pcs property set --node \\${controller} osprole=controller ; done\nstonithdevs=\\$(pcs stonith | awk '{print \\$1}')\nfor i in \\$(cibadmin -Q --xpath //primitive --node-path | tr ' ' '\\n' | awk -F \"id='\" '{print \\$2}' | awk -F \"'\" '{print \\$1}' | uniq); do\n found=0\n if [ -n \"\\$stonithdevs\" ]; then\n for x in \\$stonithdevs; do\n if [ \\$x = \\$i ]; then\n found=1\n fi\n done\n fi\n if [ \\$found = 0 ]; then\n pcs constraint location \\$i rule resource-discovery=exclusive score=0 osprole eq controller\n fi\ndone\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n# 14. Begin to populate the Compute node resources within pacemaker, starting with neutron-openvswitch-agent:\n\nhelper=fix-compute.sh\ncat < $helper\nset -x\ngrep crudini /usr/lib/ocf/resource.d/openstack/nova-compute-wait\nif [ \\$? != 0 ]; then\n set -e\n patch -p1 /usr/lib/ocf/resource.d/openstack/nova-compute-wait \\${PWD}/bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch\nfi\nEOF\nfor node in $COMPUTES; do scp bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch $helper ${node}: ; done\nfor node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done\n\nhelper=iha-helper-create-compute-resources.sh\ncat < $helper\nset -ex\n\npcs resource create neutron-openvswitch-agent-compute systemd:neutron-openvswitch-agent op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force\npcs constraint location neutron-openvswitch-agent-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute\npcs constraint order start neutron-server-clone then neutron-openvswitch-agent-compute-clone require-all=false\n\n# Then the Compute libvirtd resource:\n\npcs resource create libvirtd-compute systemd:libvirtd op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force\npcs constraint location libvirtd-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute\npcs constraint order start neutron-openvswitch-agent-compute-clone then libvirtd-compute-clone\npcs constraint colocation add libvirtd-compute-clone with neutron-openvswitch-agent-compute-clone\n\n# Then the openstack-ceilometer-compute resource:\n\npcs resource create ceilometer-compute systemd:openstack-ceilometer-compute op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force\npcs constraint location ceilometer-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute\npcs constraint order start openstack-ceilometer-notification-clone then ceilometer-compute-clone require-all=false\npcs constraint order start libvirtd-compute-clone then ceilometer-compute-clone\npcs constraint colocation add ceilometer-compute-clone with libvirtd-compute-clone\n\n# Then the nova-compute resource:\n\npcs resource create nova-compute-checkevacuate ocf:openstack:nova-compute-wait auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME domain=localdomain op start timeout=300 --clone interleave=true --disabled --force\n\npcs constraint location nova-compute-checkevacuate-clone rule resource-discovery=exclusive score=0 osprole eq compute\npcs constraint order start openstack-nova-conductor-clone then nova-compute-checkevacuate-clone require-all=false\npcs resource create nova-compute systemd:openstack-nova-compute op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force\npcs constraint location nova-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute\npcs constraint order start nova-compute-checkevacuate-clone then nova-compute-clone require-all=true\npcs constraint order start nova-compute-clone then nova-evacuate require-all=false\npcs constraint order start libvirtd-compute-clone then nova-compute-clone\npcs constraint colocation add nova-compute-clone with libvirtd-compute-clone\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n# 16. Create a seperate fence-nova stonith device:\n\n\nhelper=iha-helper-create-nova-fence.sh\ncat < $helper\nset -ex\n\npcs stonith create fence-nova fence_compute auth-url=$OS_AUTH_URL login=$OS_USERNAME passwd=$OS_PASSWORD tenant-name=$OS_TENANT_NAME domain=localdomain record-only=1 --force\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n# 17. Make certain the Compute nodes are able to recover after fencing:\n\nssh ${FIRST_CONTROLLER} -- sudo pcs property set cluster-recheck-interval=1min\n\n# 18. Create Compute node resources and set the stonith level 1 to include both the nodes's physical fence device and fence-nova:\n\nhelper=iha-helper-create-computes.sh\ncat < $helper\nset -ex\n\nfor node in $COMPUTES; do\n pcs resource create \\${node} ocf:pacemaker:remote reconnect_interval=60 op monitor interval=20\n pcs property set --node \\${node} osprole=compute\n pcs stonith level add 1 \\${node} shooter,fence-nova\ndone\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n\n# 19. Enable the control and Compute plane services:\n\nhelper=iha-helper-enable-services.sh\ncat < $helper\nset -ex\n\n#pcs resource enable openstack-keystone\npcs resource enable openstack-core\npcs resource enable neutron-openvswitch-agent-compute\npcs resource enable libvirtd-compute\npcs resource enable ceilometer-compute\npcs resource enable nova-compute-checkevacuate\npcs resource enable nova-compute\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n# 20. Allow some time for the environment to settle before cleaning up any failed resources:\n\nhelper=iha-helper-cleanup.sh\ncat < $helper\nset -ex\n\nsleep 60\npcs resource cleanup\npcs status\necho pcs property set stonith-enabled=true\nEOF\nscp $helper ${FIRST_CONTROLLER}:\nssh ${FIRST_CONTROLLER} -- sudo bash $helper\n\n\n#Test High Availability\n\n#Note: These steps deliberately reboot the Compute node without warning.\n\n#1. The following step boots an instance on the overcloud, and then crashes the Compute node:\n\n#stack@director # . overcloudrc\n#stack@director # nova boot --image cirros --flavor 2 test-failover\n#stack@director # nova list --fields name,status,host\n#stack@director # . stackrc\n#stack@director # ssh -lheat-admin compute-n\n#compute-n # sudo su -\n#root@compute-n # echo c > /proc/sysrq-trigger\n#\n#2. A short time later, the instance should be restarted on a working Compute node:\n#\n#stack@director # nova list --fields name,status,host\n#stack@director # nova service-list\n\n" }, { "path": "iha-uninstall.sh", "content": "#!/bin/bash\n\nset -ex\n\nsource stackrc\n\n# If your machines don't conform to this structure, try calling the script like this:\n# COMPUTE_PATTERN=mycompute ./iha-uninstall.sh upgrade\n: ${COMPUTE_PATTERN=novacompute}\n: ${CONTROLLER_PATTERN=controller}\n\nCOMPUTES=$(nova list | grep ${COMPUTE_PATTERN} | awk -F\\| '{ print $3}' | tr '\\n' ' ')\nCONTROLLERS=$(nova list | grep ${CONTROLLER_PATTERN} | awk -F\\| '{ print $3}' | tr '\\n' ' ')\n\nFIRST_COMPUTE=$(echo $COMPUTES | awk '{print $1}')\nFIRST_CONTROLLER=$(echo $CONTROLLERS | awk '{print $1}')\n\nssh ${FIRST_CONTROLLER} -- sudo pcs property set stonith-enabled=false\n\nif [ $1 = \"upgrade\" ]; then\n\n SERVICES=\"nova-evacuate neutron-openvswitch-agent-compute libvirtd-compute-clone ceilometer-compute-clone nova-compute-checkevacuate-clone nova-compute-clone\"\n\n helper=iha-helper-remove.sh\n cat < $helper\nset -ex\npcs property set maintenance-mode=true\n\nfor resource in $COMPUTES $SERVICES $FUDGE; do\n pcs resource cleanup \\${resource}\n pcs --force resource delete \\${resource} \ndone\n\nfor node in $COMPUTES; do\n cibadmin --delete --xml-text \"\"\n cibadmin --delete --xml-text \"\"\ndone\n\npcs property set maintenance-mode=false --wait\n\nEOF\n\n scp $helper heat-admin@${FIRST_CONTROLLER}:\n ssh heat-admin@${FIRST_CONTROLLER} -- sudo bash $helper\nfi\n\nhelper=iha-helper-reenable.sh\ncat < $helper\nset -ex\nfor service in neutron-openvswitch-agent openstack-ceilometer-compute openstack-nova-compute libvirtd; do\n systemctl enable \\${service}\ndone\nEOF\n\nfor node in $COMPUTES; do scp $helper heat-admin@${node}: ; ssh heat-admin@${node} -- sudo bash $helper ; done\n\n" }, { "path": "keepalived/ceilometer-config.md", "content": "Introduction\n------------\n\nIn terms of high availability, the Ceilometer central agent deserves special attention. This agent had to run in a single node until the Juno release cycle, since there was no way to coordinate multiple agents and ensure they would not duplicate metrics. Now, multiple central agent instances can run in parallel with workload partitioning among these running instances, using the tooz library with a Redis backend for coordination. See [here](http://docs.openstack.org/admin-guide-cloud/telemetry-data-collection.html#support-for-ha-deployment) for additional information.\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/ceilometer.scenario).\n\nInstall software\n----------------\n\n yum install -y openstack-ceilometer-api openstack-ceilometer-central openstack-ceilometer-collector openstack-ceilometer-common openstack-ceilometer-alarm python-ceilometer python-ceilometerclient python-redis\n\n\nConfigure ceilometer\n--------------------\n\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken username ceilometer\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken password ceilometertest\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken project_name services\n openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211\n openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret\n openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://controller-vip.example.com:5000/v2.0 \n openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer\n openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services\n openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest\n openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://hacontroller1,hacontroller2,hacontroller3:27017/ceilometer?replicaSet=ceilometer\n openstack-config --set /etc/ceilometer/ceilometer.conf database max_retries -1\n\n # keep last 5 days data only (value is in secs)\n openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000\n openstack-config --set /etc/ceilometer/ceilometer.conf api host 192.168.1.22X\n\nConfigure coordination URL\n--------------------------\n\n openstack-config --set /etc/ceilometer/ceilometer.conf coordination backend_url 'redis://hacontroller1:26379?sentinel=mymaster&sentinel_fallback=hacontroller2:26379&sentinel_fallback=hacontroller3:26379'\n\nEnable and start Ceilometer services, open firewall ports\n---------------------------------------------------------\n\n systemctl start openstack-ceilometer-central \n systemctl enable openstack-ceilometer-central \n systemctl start openstack-ceilometer-collector\n systemctl enable openstack-ceilometer-collector\n systemctl start openstack-ceilometer-api \n systemctl enable openstack-ceilometer-api \n systemctl start openstack-ceilometer-alarm-evaluator\n systemctl enable openstack-ceilometer-alarm-evaluator \n systemctl start openstack-ceilometer-alarm-notifier\n systemctl enable openstack-ceilometer-alarm-notifier\n systemctl start openstack-ceilometer-notification\n systemctl enable openstack-ceilometer-notification\n firewall-cmd --add-port=8777/tcp\n firewall-cmd --add-port=8777/tcp --permanent\n firewall-cmd --add-port=4952/udp\n firewall-cmd --add-port=4952/udp --permanent\n\nTests\n-----\n\nOn any node:\n\n . /root/keystonerc_admin\n\n for m in storage.objects image network volume instance ; do ceilometer sample-list -m $m | tail -2 ; done\n" }, { "path": "keepalived/cinder-config.md", "content": "Introduction\n------------\n\nCinder will be configured in this example to use the NFS backend driver. Instructions for any other backend driver will only differ in the `volume_driver` config option and any driver-specific options.\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/cinder.scenario).\n\nInstall software\n----------------\n\n yum install -y openstack-cinder openstack-utils openstack-selinux python-memcached\n\nConfigure\n---------\n\n openstack-config --set /etc/cinder/cinder.conf database connection mysql://cinder:cindertest@controller-vip.example.com/cinder\n openstack-config --set /etc/cinder/cinder.conf database max_retries -1\n openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone\n openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/cinder/cinder.conf keystone_authtoken username cinder\n openstack-config --set /etc/cinder/cinder.conf keystone_authtoken password cindertest\n openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_name services\n openstack-config --set /etc/cinder/cinder.conf DEFAULT notification_driver messaging\n openstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder\n openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_host controller-vip.example.com\n openstack-config --set /etc/cinder/cinder.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211\n openstack-config --set /etc/cinder/cinder.conf DEFAULT host rhos7-cinder\n openstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen 192.168.1.22X\n openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/cinder/cinder.conf keymgr encryption_auth_url http://controller-vip.example.com:5000/v3\n\n**Note:** We are setting a single \"host\" entry for all nodes, this is related to the A/P issues with cinder-volume.\n\nConfigure NFS driver\n--------------------\n\n # Choose whatever NFS share is used\n\n cat > /etc/cinder/nfs_exports << EOF\n 192.168.1.4:/volumeUSB1/usbshare/openstack/cinder \n EOF\n\n chown root:cinder /etc/cinder/nfs_exports\n chmod 0640 /etc/cinder/nfs_exports\n openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfs_exports\n openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_sparsed_volumes true\n openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_mount_options v3\n openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.nfs.NfsDriver\n\nManage DB\n---------\n\nOn node 1:\n\n su cinder -s /bin/sh -c \"cinder-manage db sync\"\n\nStart services\n--------------\n\nOn node 1:\n\n systemctl start openstack-cinder-api\n systemctl start openstack-cinder-scheduler\n systemctl start openstack-cinder-volume\n systemctl enable openstack-cinder-api\n systemctl enable openstack-cinder-scheduler\n systemctl enable openstack-cinder-volume\n\n**Note:** If this node crashes, it should be manually started on another node. Refer to [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1193229) for additional information.\n\nOn nodes 2 and 3:\n\n systemctl start openstack-cinder-api\n systemctl start openstack-cinder-scheduler\n systemctl enable openstack-cinder-api\n systemctl enable openstack-cinder-scheduler\n\nOpen firewall ports\n-------------------\n\nOn all nodes:\n\n firewall-cmd --add-port=8776/tcp\n firewall-cmd --add-port=8776/tcp --permanent\n\nTest\n----\n\nOn any node:\n\n . /root/keystonerc_demo\n cinder create --display-name test 1\n cinder extend test 4\n cinder delete test\n" }, { "path": "keepalived/compute-node.md", "content": "Introduction\n------------\n\nThe compute node implementation is relatively straightforward, compared to the controller node. It will only be necessary to configure:\n\n- OpenvSwitch and Neutron OpenvSwitch agent\n- Nova compute\n- Ceilometer compute agent\n\nYou can find a phd scenario file [here](phd-setup/compute.scenario).\n\nEnvironment description\n-----------------------\n\nThe network configuration was previously discussed in the [Controller node implementation](controller-node.md) section:\n\n![](Controller-network.jpg \"Network configuration\")\n\n- The external network is used by the Neutron floating IPs, and for any external access. The hypervisor nodes (hacompute1 and hacompute2) do not need to be connected to this network, but in the demo setup they are connected for testing purposes.\n- The internal network will carry all other traffic: API traffic, tenant networks and storage traffic.\n- The router providing connectivity between the internal and external networks is only needed if Trove and/or Sahara are being deployed.\n\nRemember this is a minimum test setup. Any production setup should separate internal and external API traffic, tenant networks and storage traffic in different network segments.\n\nCompute node configuration\n--------------------------\n\nThe following commands should be executed on each compute node to be added to the installation. There is no configuration required on the controller nodes, meaning compute nodes can be added anytime.\n\n### Install software\n\n yum install -y openstack-nova-compute openstack-utils python-cinder openstack-neutron-openvswitch openstack-ceilometer-compute openstack-neutron openstack-selinux\n\n### Enable OpenvSwitch, start daemon and create integration bridge\n\n systemctl enable openvswitch\n systemctl start openvswitch\n ovs-vsctl add-br br-int\n\n### Configure Nova compute\n\n openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211\n openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.1.22X\n openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 0.0.0.0\n openstack-config --set /etc/nova/nova.conf vnc novncproxy_base_url http://controller-vip.example.com:6080/vnc_auto.html\n openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@controller-vip.example.com/nova\n openstack-config --set /etc/nova/nova.conf database max_retries -1\n openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone\n openstack-config --set /etc/nova/nova.conf glance host controller-vip.example.com\n openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API\n openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver\n openstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver\n openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron\n openstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL\n openstack-config --set /etc/nova/nova.conf conductor use_local false\n openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True\n openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True\n openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest\n openstack-config --set /etc/nova/nova.conf neutron url http://controller-vip.example.com:9696/\n openstack-config --set /etc/nova/nova.conf neutron project_domain_id default\n openstack-config --set /etc/nova/nova.conf neutron project_name services\n openstack-config --set /etc/nova/nova.conf neutron user_domain_id default\n openstack-config --set /etc/nova/nova.conf neutron username neutron\n openstack-config --set /etc/nova/nova.conf neutron password neutrontest\n openstack-config --set /etc/nova/nova.conf neutron auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/nova/nova.conf neutron auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/nova/nova.conf neutron auth_plugin password\n openstack-config --set /etc/nova/nova.conf neutron region_name regionOne\n openstack-config --set /etc/nova/nova.conf libvirt nfs_mount_options v3\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_plugin password\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken username compute\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken password novatest\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken project_name services\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_uri http://controller-vip.example.com:5000/\n\nOnly run the following command if you are creating a test environment where your hypervisors will be virtual machines.\n\n openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu\n\n### Configure Neutron on compute node\n\n openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password neutrontest\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name services\n openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent tunnel_types vxlan\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 4789\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs enable_tunneling True\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_id_ranges 1:1000\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tenant_network_type vxlan\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs integration_bridge br-int\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_bridge br-tun\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs local_ip 192.168.1.22X\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver \n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent l2_population False\n\n\n### Configure Ceilometer on compute node\n\n openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit True\n openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit_period hour\n openstack-config --set /etc/nova/nova.conf DEFAULT notify_on_state_change vm_and_task_state\n openstack-config --set /etc/nova/nova.conf DEFAULT notification_driver nova.openstack.common.notifier.rpc_notifier\n sed -i -e 's/nova.openstack.common.notifier.rpc_notifier/nova.openstack.common.notifier.rpc_notifier\\nnotification_driver = ceilometer.compute.nova_notifier/g' /etc/nova/nova.conf\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken username ceilometer\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken password ceilometertest\n openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken project_name services\n openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211\n openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret\n openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://controller-vip.example.com:5000/v2.0\n openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer\n openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services\n openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest\n openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://hacontroller1,hacontroller2,hacontroller3:27017/ceilometer?replicaSet=ceilometer\n openstack-config --set /etc/ceilometer/ceilometer.conf database connection max_retries -1\n\n # keep last 5 days data only (value is in secs)\n openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000\n\n**Note:** the following SELinux boolean allows QEMU to use NFS for Cinder volumes. A different boolean may apply if using another type of backend storage.\n\n setsebool -P virt_use_nfs 1\n\n### Set kernel TCP keepalive parameters\n\n cat > /etc/sysctl.d/tcpka.conf << EOF\n net.ipv4.tcp_keepalive_intvl = 1\n net.ipv4.tcp_keepalive_probes = 5\n net.ipv4.tcp_keepalive_time = 5\n EOF\n\n sysctl -p /etc/sysctl.d/tcpka.conf\n\n### Enable and start services, open firewall ports\n\n**Note:** we are enabling ports 5900-5999 for VNC access. If the compute node could host more than 100 VMs, we have to extend this range.\n\n systemctl start libvirtd\n systemctl start neutron-openvswitch-agent\n systemctl enable neutron-openvswitch-agent\n systemctl enable neutron-ovs-cleanup\n systemctl start openstack-ceilometer-compute\n systemctl enable openstack-ceilometer-compute\n systemctl start openstack-nova-compute\n systemctl enable openstack-nova-compute\n firewall-cmd --add-port=4789/udp\n firewall-cmd --add-port=4789/udp --permanent\n firewall-cmd --add-port=5900-5999/tcp\n firewall-cmd --add-port=5900-5999/tcp --permanent\n" }, { "path": "keepalived/controller-node.md", "content": "Introduction\n------------\n\nController nodes will run all OpenStack services in this architecture, while compute nodes will only act as hypervisors.\n\nEnvironment description\n-----------------------\n\n### Network setup\n\nThe basic requirements for this environment include 5 nodes, with the network setup described in the following diagram:\n\n![](Controller-network.jpg \"Network setup\")\n\n- The external network is used by the Neutron floating IPs, and for any external access. The hypervisor nodes (hacompute1 and hacompute2) do not need to be connected to this network, but in the demo setup they are connected for testing purposes.\n- The internal network will carry all other traffic: API traffic, tenant networks and storage traffic.\n- A router will provide connectivity for the controller nodes to the floating IP network as required by Sahara for instance management. In this configuration example, it will also allow virtual machines to access the controller nodes, allowing Trove instances access to the RabbitMQ server (**please note this is not recommended for a production setup**, refer to [trove-config.md](the Trove section) for details). If neitner Sahara or Trove are used, the router is not required.\n\nPlease note this is a minimum test setup. Any production setup should separate internal and external API traffic, tenant networks and storage traffic in different network segments, and route traffic between instances and the controller nodes via a firewall.\n\n### Node setup\n\nThe following table provides the system details for the environment used during testing.\n\n\n| Hostname | NIC 1 | NIC 2 | Disk 1 | Disk 2 |\n|--------------------|--------------------------------------|-------------------|------------------|-----------------|\n|hacontroller1 |eth0: no IP, used for provider network|eth1: 192.168.1.221| 60 GB (/dev/vda) | 8 GB (/dev/vdb) |\n|hacontroller2 |eth0: no IP, used for provider network|eth1: 192.168.1.222| 60 GB (/dev/vda) | 8 GB (/dev/vdb) |\n|hacontroller3 |eth0: no IP, used for provider network|eth1: 192.168.1.223| 60 GB (/dev/vda) | 8 GB (/dev/vdb) |\n|controller-vip (virtual hostname)| |eth1: 192.168.1.220| | |\n|hacompute1 |eth0: 10.10.10.224 (-priv) |eth1: 192.168.1.224| 60 GB (/dev/vda) | |\n|hacompute2 |eth0: 10.10.10.225 (-priv) |eth1: 192.168.1.225| 60 GB (/dev/vda) | |\n\nAll nodes have SELinux set to *Enforcing* and an active firewall.\n\nOn the detailed installation notes, remember to substitute any occurrence of 192.168.1.22X with the IP of the node being configured.\n\n#### Single vs multiple VIPs\n\nPlease note that the current document uses a single virtual IP for the whole stack of OpenStack API services. It is also possible to use multiple virtual IPs, one per API service. Using multiple virtual IPs will allow for load distribution between the three HAProxy instances. In contrast, the requirements for available IP addresses increase, and the initial configuration can be slightly more complex if performed manually.\n\n#### Base operating system installation\n\nAll nodes start from a *minimal* CentOS 7 or RHEL 7 installation, then enabling the required software channels.\n\n- For RDO Liberty, follow the steps specified in the [RDO wiki](https://openstack.redhat.com/Repositories)\n\n**Note:** For now, use [this release RPM](http://rdoproject.org/repos/openstack-liberty/rdo-release-liberty.rpm) to setup the required repositories.\n\n- For the [Red Hat Enterprise Linux OpenStack Platform](http://www.redhat.com/openstack), run the following commands to enable the required repositories:\n\n
subscription-manager register --username=${RHSMUSER} --password=${RHSMPASS} \nsubscription-manager attach --auto\nsubscription-manager repos --disable \\* \nsubscription-manager repos --enable rhel-7-server-rpms \nsubscription-manager repos --enable rhel-7-server-rh-common-rpms \nsubscription-manager repos --enable rhel-7-server-openstack-7.0-rpms\nyum -y update \nreboot
\n\nThe systems also had NetworkManager disabled:\n\n systemctl disable NetworkManager\n systemctl enable network\n\nIf the installed system cannot access the gateway, be sure there is a “GATEWAY=xxx” entry in the relevant ifcfg file:\n\n NAME=eth0\n ...\n GATEWAY=192.168.1.1\n\n**Note:** It is very important to make sure that NTP is correctly configured on all nodes. Failure to do so can cause environment instability.\n\nThe following phd scenario files have been created:\n\n- A file with the [VM creation and initial setup](phd-setup/hypervisors.scenario)\n- A file with some [basic system preparation tasks](phd-setup/serverprep.scenario)\n\n#### Configuration steps\n\n**NOTE:** before moving on, please remember that the following instructions have been created for the above-mentioned scenario (3 controller nodes, with a specific network setup). If you are using this guide to implement your own environment, pay close attention to any IP substitution and other changes that may apply to your environment. Also, do not forget to set sensible passwords.\n\nThe configuration steps can be divided into:\n\n- Installing/configuring/enabling all core non-Openstack services\n - [HAProxy](haproxy-config.md)\n - [Galera](galera-config.md)\n - [RabbitMQ](rabbitmq-config.md)\n - [Memcached](memcached-config.md)\n - [Redis](redis-config.md)\n - [MongoDB](mongodb-config.md)\n - [Keepalived](keepalived-config.md)\n- Installing/configuring/enabling all core Openstack services\n - [Keystone](keystone-config.md)\n - [Glance](glance-config.md)\n - [Cinder](cinder-config.md)\n - [Swift](swift-config.md)\n - [Neutron](neutron-config.md)\n - [Nova](nova-config.md)\n - [Ceilometer](ceilometer-config.md)\n - [Heat](heat-config.md)\n - [Horizon](horizon-config.md)\n - [Trove](trove-config.md)\n - [Sahara](sahara-config.md)\n\nOn each section, a link to a [phd-based](https://github.com/davidvossel/phd) scenario file is provided, as a reference.\n" }, { "path": "keepalived/galera-bootstrap.md", "content": "Introduction\n------------\n\nHere is an outline of the steps needed to re-establish/bootstrap Galera quorum.\n\n1. Determine loss of quorum\n2. Determine systems with last activity\n3. Start first DB on first node\n4. Start DB on remaining nodes\n\nDetermine loss of quorum\n------------------------\n\nConfirm in the */var/log/mariadb/mariadb.log* on each system, looking for Errors\n\n 140929 11:25:40 [ERROR] WSREP: Local state seqno (1399488) is greater than group seqno (10068): states diverged. Aborting to avoid potential data loss. Remove '/var/lib/mysql//grastate.dat' file and restart if you wish to continue. (FATAL)\n 140929 11:25:40 [ERROR] Aborting\n [root@ospha2 ~]#\n\nAlso the clustercheck command should so that there are some systems not in sync\n\n [root@ospha2 ~]# clustercheck\n HTTP/1.1 503 Service Unavailable\n Content-Type: text/plain\n Connection: close\n Content-Length: 36\n\n Galera cluster node is not synced.\n [root@ospha2 ~]#\n\nDetermine systems with last activity\n------------------------------------\n\nIn this section we attempt to determine which system or systems has the highest valid sequence number for the for the latest UUID.\n\n### Orderly shutdown\n\nIf the cluster shutdown correctly the `/var/lib/mysql/grastate.dat` file will have positive numbers for the seqno. Note which system or systems have the greatest seqno. However, if any system has a `-1` value, that indicates the shutdown was not clean and another method to determine the seqno is needed.\n\n [root@ospha2 ~]# cat /var/lib/mysql/grastate.dat\n # GALERA saved state\n version: 2.1\n uuid: b048715d-4369-11e4-b7ef-af1999a6c989\n seqno: -1\n cert_index:\n [root@ospha2 ~]#\n\n### Disorderly Shutdown\n\nThe seqno is in the `/var/log/mariadb/mariadb.log` file. Search for lines with \"Found save state\", ignoring any -1 values. The last value on each line is in the form UUID:seqno.\n\n [root@ospha1 ~]# tail -n 1000 /var/log/mariadb/mariadb.log | grep \"Found saved state\" | grep -v \":-1\"\n 140923 17:49:19 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:2229\n 140924 15:37:13 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:2248\n 140929 11:24:26 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:10060\n [root@ospha1 ~]#\n\n [root@ospha2 ~]# tail -n 1000 /var/log/mariadb/mariadb.log | grep \"Found saved state\" | grep -v \":-1\"\n 140926 14:58:16 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:171535\n 140929 11:24:28 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:1399488\n [root@ospha2 ~]#\n\n [root@ospha3 ~]# tail -n 2000 /var/log/mariadb/mariadb.log | grep \"Found saved state\" | grep -v \":-1\"\n 140923 17:36:57 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:36\n 140923 17:43:18 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:785\n [root@ospha3 ~]#\n\nNotice all servers have the same UUID (b048715d-4369-11e4-b7ef-af1999a6c989), but server *ospha2* has the largest seqno (1399488).\n\nStart first DB on first node\n----------------------------\n\nThe following command will initiate the Galera cluster. Since ospha2 had the highest seqno, that is the node to start first.\n\n [root@ospha2 ~]# sudo -u mysql /usr/libexec/mysqld --wsrep-cluster-address='gcomm://' &\n [1] 1910\n [root@ospha2 ~]# 140929 16:31:00 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295\n 140929 16:31:00 [Warning] Could not increase number of max_open_files to more than 1024 (request: 1835)\n /usr/libexec/mysqld: Query cache is disabled (resize or similar command in progress); repeat this command later\n\nVerify that this brought the this node into sync.\n\n [root@ospha2 ~]# clustercheck\n HTTP/1.1 200 OK\n Content-Type: text/plain\n Connection: close\n Content-Length: 32\n\n Galera cluster node is synced.\n [root@ospha2 ~]#\n\nStart DB on remaining nodes\n---------------------------\n\nOn another cluster member, start the database, and then verify this node reports synced.\n\n [root@ospha1 ~]# systemctl start mariadb\n [root@ospha1 ~]# clustercheck\n HTTP/1.1 200 OK\n Content-Type: text/plain\n Connection: close\n Content-Length: 32\n\n Galera cluster node is synced.\n [root@ospha1 ~]#\n\nOnce `clustercheck` returns 200 on all nodes, restart MariaDB on the first node.\n\n kill \n systemctl start mariadb\n\n" }, { "path": "keepalived/galera-config.md", "content": "Introduction\n------------\n\nMariaDB with Galera provides synchronous database replication, in an active-active, multi-master environment. High availability for the data itself is managed internally by Galera, while access availability is managed by HAProxy.\n\n![](Mariadb-haproxy.jpg \"MariaDB HA architecture\")\n\nIt is important to note that, while Galera supports multi-master access, there are known issues in an OpenStack environment due to its use of optimistic locking (see [this post](http://lists.openstack.org/pipermail/openstack-dev/2014-May/035264.html) for details). Due to this limitation, the HAProxy configuration used for Galera sets a single node to be accessed at all times, and a different node will only be used when the original node fails. This effectively turns the Galera configuration into an active-hot standby one.\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/galera.scenario).\n\nInstall software\n----------------\n\n yum install -y mariadb-galera-server xinetd rsync\n\nConfigure cluster check\n-----------------------\n\n cat > /etc/sysconfig/clustercheck << EOF\n MYSQL_USERNAME=\"clustercheck\"\n MYSQL_PASSWORD=\"redhat\"\n MYSQL_HOST=\"localhost\"\n MYSQL_PORT=\"3306\"\n EOF\n\nStart mysqld and create user for cluster check\n----------------------------------------------\n\n systemctl start mysqld\n mysql -e \"CREATE USER 'clustercheck'@'localhost' IDENTIFIED BY 'redhat';\"\n systemctl stop mysqld\n\nCreate Galera configuration\n---------------------------\n\n cat > /etc/my.cnf.d/galera.cnf << EOF\n [mysqld]\n skip-name-resolve=1\n binlog_format=ROW\n default-storage-engine=innodb\n innodb_autoinc_lock_mode=2\n innodb_locks_unsafe_for_binlog=1\n max_connections=8192\n query_cache_size=0\n query_cache_type=0\n bind_address=192.168.1.22X\n wsrep_provider=/usr/lib64/galera/libgalera_smm.so\n wsrep_cluster_name=\"galera_cluster\"\n wsrep_cluster_address=\"gcomm://192.168.1.221,192.168.1.222,192.168.1.223\"\n wsrep_slave_threads=1\n wsrep_certify_nonPK=1\n wsrep_max_ws_rows=131072\n wsrep_max_ws_size=1073741824\n wsrep_debug=0\n wsrep_convert_LOCK_to_trx=0\n wsrep_retry_autocommit=1\n wsrep_auto_increment_control=1\n wsrep_drupal_282555_workaround=0\n wsrep_causal_reads=0\n wsrep_notify_cmd=\n wsrep_sst_method=rsync\n EOF\n\nCreate Galera systemd configuration file, to allow a higher number of files to be opened\n----------------------------------------------------------------------------------------\n\n mkdir -p /etc/systemd/system/mariadb.service.d/\n cat > /etc/systemd/system/mariadb.service.d/limits.conf << EOF\n [Service]\n LimitNOFILE=16384\n EOF\n\nConfigure monitor service (used by HAProxy)\n-------------------------------------------\n\n cat > /etc/xinetd.d/galera-monitor << EOF\n service galera-monitor\n {\n port = 9200\n disable = no\n socket_type = stream\n protocol = tcp\n wait = no\n user = root\n group = root\n groups = yes\n server = /usr/bin/clustercheck\n type = UNLISTED\n per_source = UNLIMITED\n log_on_success = \n log_on_failure = HOST\n flags = REUSE\n }\n EOF\n\nStart services and open firewall ports\n--------------------------------------\n\n systemctl daemon-reload\n systemctl enable xinetd\n systemctl start xinetd\n systemctl enable haproxy\n systemctl start haproxy\n\n firewall-cmd --add-service=mysql\n firewall-cmd --add-port=4444/tcp \n firewall-cmd --add-port=4567/tcp\n firewall-cmd --add-port=4568/tcp\n firewall-cmd --add-port=4568/tcp --permanent\n firewall-cmd --add-service=mysql --permanent\n firewall-cmd --add-port=4567/tcp --permanent\n firewall-cmd --add-port=4444/tcp --permanent\n firewall-cmd --add-port=9300/tcp\n firewall-cmd --add-port=9300/tcp --permanent\n firewall-cmd --add-port=9200/tcp\n firewall-cmd --add-port=9200/tcp --permanent\n\nStart mariadb cluster\n---------------------\n\n**On all nodes:**\n\n systemctl enable mariadb\n\n**On node1:** \n\n sudo -u mysql /usr/libexec/mysqld --wsrep-cluster-address='gcomm://' &\n\n**On node2 and node3:**\n\n systemctl start mariadb\n\n Once clustecheck returns 200 on all nodes, restart galera on node 1:\n\n kill \n systemctl start mariadb\n\nCreate users and databases\n--------------------------\n\n**On node 1:**\n\n mysql\n\n MariaDB [(none)]> use mysql;\n MariaDB [mysql]> drop user ''@'hacontroller1.example.com';\n MariaDB [mysql]> drop user 'root'@'hacontroller1.example.com';\n MariaDB [mysql]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED by 'mysqltest' WITH GRANT OPTION;\n MariaDB [mysql]> CREATE DATABASE keystone;\n MariaDB [mysql]> GRANT ALL ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'keystonetest';\n MariaDB [mysql]> CREATE DATABASE glance;\n MariaDB [mysql]> GRANT ALL ON glance.* TO 'glance'@'%' IDENTIFIED BY 'glancetest';\n MariaDB [mysql]> CREATE DATABASE cinder;\n MariaDB [mysql]> GRANT ALL ON cinder.* TO 'cinder'@'%' IDENTIFIED BY 'cindertest';\n MariaDB [mysql]> CREATE DATABASE neutron;\n MariaDB [mysql]> GRANT ALL ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'neutrontest';\n MariaDB [mysql]> CREATE DATABASE nova;\n MariaDB [mysql]> GRANT ALL ON nova.* TO 'nova'@'%' IDENTIFIED BY 'novatest';\n MariaDB [mysql]> CREATE DATABASE heat;\n MariaDB [mysql]> GRANT ALL ON heat.* TO 'heat'@'%' IDENTIFIED BY 'heattest';\n MariaDB [mysql]> CREATE DATABASE sahara;\n MariaDB [mysql]> GRANT ALL ON sahara.* TO 'sahara'@'%' IDENTIFIED BY 'saharatest';\n MariaDB [mysql]> CREATE DATABASE trove;\n MariaDB [mysql]> GRANT ALL ON trove.* TO 'trove'@'%' IDENTIFIED BY 'trovetest';\n MariaDB [mysql]> FLUSH PRIVILEGES;\n MariaDB [mysql]> quit\n\n mysqladmin flush-hosts\n" }, { "path": "keepalived/glance-config.md", "content": "Introduction\n------------\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/glance.scenario).\n\nInstall software\n----------------\n\n yum install -y openstack-glance openstack-utils openstack-selinux nfs-utils\n\nConfigure glance-api and glance-registry\n----------------------------------------\n\n openstack-config --set /etc/glance/glance-api.conf database connection mysql://glance:glancetest@controller-vip.example.com/glance\n openstack-config --set /etc/glance/glance-api.conf database max_retries -1\n openstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone\n openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/glance/glance-api.conf keystone_authtoken username glance\n openstack-config --set /etc/glance/glance-api.conf keystone_authtoken password glancetest\n openstack-config --set /etc/glance/glance-api.conf keystone_authtoken project_name services\n openstack-config --set /etc/glance/glance-api.conf DEFAULT notification_driver messaging\n openstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host 192.168.1.22X\n openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host controller-vip.example.com\n openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/glance/glance-registry.conf database connection mysql://glance:glancetest@controller-vip.example.com/glance\n openstack-config --set /etc/glance/glance-registry.conf database max_retries -1\n openstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone\n openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken username glance\n openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken password glancetest\n openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken project_name services\n openstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host 192.168.1.22X\n\nManage DB\n---------\n\nOn node 1:\n\n su glance -s /bin/sh -c \"glance-manage db_sync\"\n\nConfigure backend\n-----------------\n\nFor this setup, NFS will be used. Add the NFS mount to `/etc/fstab`, making sure it is mounted on `/var/lib/glance`. Be aware the last two columns in `fstab` need to be \"0 0\" on RHEL/CentOS 7, due to [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1120367). You may also find [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1203820) if using NFS v3 shares.\n\nAlso, note there is currently a known SELinux issue when using an NFS backend for Glance. See [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1219406) for a description and fix.\n\nOn all nodes:\n\n chown glance:nobody /var/lib/glance\n\nStart services and open firewall ports\n--------------------------------------\n\n systemctl start openstack-glance-registry\n systemctl start openstack-glance-api\n systemctl enable openstack-glance-registry\n systemctl enable openstack-glance-api\n firewall-cmd --add-port=9191/tcp\n firewall-cmd --add-port=9191/tcp --permanent\n firewall-cmd --add-port=9292/tcp\n firewall-cmd --add-port=9292/tcp --permanent\n\nTest\n----\n\nOn any node:\n\n . /root/keystonerc_admin\n wget http://download.cirros-cloud.net/0.3.3/cirros-0.3.3-x86_64-disk.img\n glance image-create --name \"cirros\" --disk-format qcow2 --container-format bare --file cirros-0.3.3-x86_64-disk.img --visibility public\n glance image-list\n" }, { "path": "keepalived/haproxy-config.md", "content": "Introduction\n------------\n\nA load-balancing proxy is used to provide scalability and load balancing for OpenStack API services and some of the supporting services. All requests will be distributed using a round-robin algorithm between all available controller nodes, and HAProxy itself will monitor the availability of each service. In case a node or service goes down, HAProxy will remove it from the active pool after a timeout has been reached, tipically a few seconds.\n\nThe following commands will be executed on all controller nodes.\n\nYou can find a phd scenario file [here](phd-setup/lb.scenario).\n\nInstall packages\n----------------\n\n yum install -y haproxy openstack-selinux\n\nAllow binding to non-local IPs\n------------------------------\n\n echo net.ipv4.ip_nonlocal_bind=1 >> /etc/sysctl.d/haproxy.conf\n echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind\n\nConfigure HAProxy\n-----------------\n\n cat > /etc/haproxy/haproxy.cfg << EOF\n global\n daemon\n stats socket /var/lib/haproxy/stats\n defaults\n mode tcp\n maxconn 10000\n timeout connect 5s\n timeout client 30s\n timeout server 30s\n\n listen monitor\n bind 192.168.1.220:9300 \n mode http\n monitor-uri /status\n stats enable\n stats uri /admin\n stats realm Haproxy\\ Statistics\n stats auth root:redhat\n stats refresh 5s\n\n frontend vip-db\n bind 192.168.1.220:3306\n timeout client 90m\n default_backend db-vms-galera\n backend db-vms-galera\n option httpchk\n stick-table type ip size 1000\n stick on dst\n timeout server 90m\n server rhos8-node1 192.168.1.221:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions\n server rhos8-node2 192.168.1.222:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions\n server rhos8-node3 192.168.1.223:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions\n\n # Note the RabbitMQ entry is only needed for CloudForms compatibility\n # and should be removed in the future\n frontend vip-rabbitmq\n option clitcpka\n bind 192.168.1.220:5672\n timeout client 900m\n default_backend rabbitmq-vms\n backend rabbitmq-vms\n option srvtcpka\n balance roundrobin\n timeout server 900m\n server rhos8-node1 192.168.1.221:5672 check inter 1s\n server rhos8-node2 192.168.1.222:5672 check inter 1s\n server rhos8-node3 192.168.1.223:5672 check inter 1s\n\n frontend vip-keystone-admin\n bind 192.168.1.220:35357\n default_backend keystone-admin-vms\n timeout client 600s\n backend keystone-admin-vms\n balance roundrobin\n timeout server 600s\n server rhos8-node1 192.168.1.221:35357 check inter 1s on-marked-down shutdown-sessions\n server rhos8-node2 192.168.1.222:35357 check inter 1s on-marked-down shutdown-sessions\n server rhos8-node3 192.168.1.223:35357 check inter 1s on-marked-down shutdown-sessions\n\n frontend vip-keystone-public\n bind 192.168.1.220:5000\n default_backend keystone-public-vms\n timeout client 600s\n backend keystone-public-vms\n balance roundrobin\n timeout server 600s\n server rhos8-node1 192.168.1.221:5000 check inter 1s on-marked-down shutdown-sessions\n server rhos8-node2 192.168.1.222:5000 check inter 1s on-marked-down shutdown-sessions\n server rhos8-node3 192.168.1.223:5000 check inter 1s on-marked-down shutdown-sessions\n\n frontend vip-glance-api\n bind 192.168.1.220:9191\n default_backend glance-api-vms\n backend glance-api-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:9191 check inter 1s\n server rhos8-node2 192.168.1.222:9191 check inter 1s\n server rhos8-node3 192.168.1.223:9191 check inter 1s\n\n frontend vip-glance-registry\n bind 192.168.1.220:9292\n default_backend glance-registry-vms\n backend glance-registry-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:9292 check inter 1s\n server rhos8-node2 192.168.1.222:9292 check inter 1s\n server rhos8-node3 192.168.1.223:9292 check inter 1s\n\n frontend vip-cinder\n bind 192.168.1.220:8776\n default_backend cinder-vms\n backend cinder-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:8776 check inter 1s\n server rhos8-node2 192.168.1.222:8776 check inter 1s\n server rhos8-node3 192.168.1.223:8776 check inter 1s\n\n frontend vip-swift\n bind 192.168.1.220:8080\n default_backend swift-vms\n backend swift-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:8080 check inter 1s\n server rhos8-node2 192.168.1.222:8080 check inter 1s\n server rhos8-node3 192.168.1.223:8080 check inter 1s\n\n frontend vip-neutron\n bind 192.168.1.220:9696\n default_backend neutron-vms\n backend neutron-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:9696 check inter 1s\n server rhos8-node2 192.168.1.222:9696 check inter 1s\n server rhos8-node3 192.168.1.223:9696 check inter 1s\n\n frontend vip-nova-vnc-novncproxy\n bind 192.168.1.220:6080\n default_backend nova-vnc-novncproxy-vms\n backend nova-vnc-novncproxy-vms\n balance roundrobin\n timeout tunnel 1h\n server rhos8-node1 192.168.1.221:6080 check inter 1s\n server rhos8-node2 192.168.1.222:6080 check inter 1s\n server rhos8-node3 192.168.1.223:6080 check inter 1s\n\n frontend nova-metadata-vms\n bind 192.168.1.220:8775\n default_backend nova-metadata-vms\n backend nova-metadata-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:8775 check inter 1s\n server rhos8-node2 192.168.1.222:8775 check inter 1s\n server rhos8-node3 192.168.1.223:8775 check inter 1s\n\n frontend vip-nova-api\n bind 192.168.1.220:8774\n default_backend nova-api-vms\n backend nova-api-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:8774 check inter 1s\n server rhos8-node2 192.168.1.222:8774 check inter 1s\n server rhos8-node3 192.168.1.223:8774 check inter 1s\n\n frontend vip-horizon\n bind 192.168.1.220:80\n timeout client 180s\n default_backend horizon-vms\n backend horizon-vms\n balance roundrobin\n timeout server 180s\n mode http\n cookie SERVERID insert indirect nocache\n server rhos8-node1 192.168.1.221:80 check inter 1s cookie rhos8-horizon1 on-marked-down shutdown-sessions\n server rhos8-node2 192.168.1.222:80 check inter 1s cookie rhos8-horizon2 on-marked-down shutdown-sessions\n server rhos8-node3 192.168.1.223:80 check inter 1s cookie rhos8-horizon3 on-marked-down shutdown-sessions\n\n frontend vip-heat-cfn\n bind 192.168.1.220:8000\n default_backend heat-cfn-vms\n backend heat-cfn-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:8000 check inter 1s\n server rhos8-node2 192.168.1.222:8000 check inter 1s\n server rhos8-node3 192.168.1.223:8000 check inter 1s\n\n frontend vip-heat-cloudw\n bind 192.168.1.220:8003\n default_backend heat-cloudw-vms\n backend heat-cloudw-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:8003 check inter 1s\n server rhos8-node2 192.168.1.222:8003 check inter 1s\n server rhos8-node3 192.168.1.223:8003 check inter 1s\n\n frontend vip-heat-srv\n bind 192.168.1.220:8004\n default_backend heat-srv-vms\n backend heat-srv-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:8004 check inter 1s\n server rhos8-node2 192.168.1.222:8004 check inter 1s\n server rhos8-node3 192.168.1.223:8004 check inter 1s\n\n frontend vip-ceilometer\n bind 192.168.1.220:8777\n timeout client 90s\n default_backend ceilometer-vms\n backend ceilometer-vms\n balance roundrobin\n timeout server 90s\n server rhos8-node1 192.168.1.221:8777 check inter 1s\n server rhos8-node2 192.168.1.222:8777 check inter 1s\n server rhos8-node3 192.168.1.223:8777 check inter 1s\n\n frontend vip-sahara\n bind 192.168.1.220:8386\n default_backend sahara-vms\n backend sahara-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:8386 check inter 1s\n server rhos8-node2 192.168.1.222:8386 check inter 1s\n server rhos8-node3 192.168.1.223:8386 check inter 1s\n\n frontend vip-trove\n bind 192.168.1.220:8779\n default_backend trove-vms\n backend trove-vms\n balance roundrobin\n server rhos8-node1 192.168.1.221:8779 check inter 1s\n server rhos8-node2 192.168.1.222:8779 check inter 1s\n server rhos8-node3 192.168.1.223:8779 check inter 1s\n EOF\n\nNote we are **not** starting haproxy yet.\n\nOnce HAproxy is started, you can monitor progress of your service configuration by going to [](http://controller-vip.example.com:9300/admin) (root/redhat, remember to set a sensible password). With this you will be able to see which services are running on which nodes, as seen by HAproxy.\n" }, { "path": "keepalived/heat-config.md", "content": "Introduction\n------------\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/heat.scenario).\n\nInstall software\n----------------\n\n yum install -y openstack-heat-engine openstack-heat-api openstack-heat-api-cfn openstack-heat-api-cloudwatch python-heatclient openstack-utils python-glanceclient\n\nConfigure Heat domain\n---------------------\n\nTo allow non-admin users to create Heat stacks, a Keystone domain needs to be created. Run the following commands to create the Heat domain, and configure Heat to use it.\n\nOn node 1:\n\n . /root/keystonerc_admin\n openstack role create heat_stack_user\n openstack token issue\n\nTake note of the token ID issued, then:\n\n openstack --os-token=${TOKEN_ID} --os-url=http://controller-vip.example.com:5000/v3 --os-identity-api-version=3 domain create heat --description \"Owns users and projects created by heat\"\n openstack --os-token=${TOKEN_ID} --os-url=http://controller-vip.example.com:5000/v3 --os-identity-api-version=3 user create --password heattest --domain heat --description \"Manages users and projects created by heat\" heat_domain_admin\n openstack --os-token=${TOKEN_ID} --os-url=http://controller-vip.example.com:5000/v3 --os-identity-api-version=3 role add --user heat_domain_admin --domain heat admin\n\nOn all nodes:\n\n openstack-config --set /etc/heat/heat.conf DEFAULT stack_domain_admin_password heattest\n openstack-config --set /etc/heat/heat.conf DEFAULT stack_domain_admin heat_domain_admin\n openstack-config --set /etc/heat/heat.conf DEFAULT stack_user_domain_name heat\n\nConfigure Heat\n--------------\n\n openstack-config --set /etc/heat/heat.conf database connection mysql://heat:heattest@controller-vip.example.com/heat\n openstack-config --set /etc/heat/heat.conf database max_retries -1\n openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/heat/heat.conf keystone_authtoken username heat\n openstack-config --set /etc/heat/heat.conf keystone_authtoken password heattest\n openstack-config --set /etc/heat/heat.conf keystone_authtoken project_name services\n openstack-config --set /etc/heat/heat.conf keystone_authtoken keystone_ec2_uri http://controller-vip.example.com:35357/v2.0\n openstack-config --set /etc/heat/heat.conf keystone_authtoken identity_uri http://controller-vip.example.com:35357\n openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_tenant_name services\n openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_user heat\n openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_password heattest\n openstack-config --set /etc/heat/heat.conf ec2authtoken auth_uri http://controller-vip.example.com:5000/v2.0\n openstack-config --set /etc/heat/heat.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211\n openstack-config --set /etc/heat/heat.conf heat_api bind_host 192.168.1.22X\n openstack-config --set /etc/heat/heat.conf heat_api_cfn bind_host 192.168.1.22X\n openstack-config --set /etc/heat/heat.conf heat_api_cloudwatch bind_host 192.168.1.22X\n openstack-config --set /etc/heat/heat.conf DEFAULT heat_metadata_server_url controller-vip.example.com:8000\n openstack-config --set /etc/heat/heat.conf DEFAULT heat_waitcondition_server_url controller-vip.example.com:8000/v1/waitcondition\n openstack-config --set /etc/heat/heat.conf DEFAULT heat_watch_server_url controller-vip.example.com:8003\n openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/heat/heat.conf DEFAULT rpc_backend rabbit\n openstack-config --set /etc/heat/heat.conf DEFAULT notification_driver heat.openstack.common.notifier.rpc_notifier\n openstack-config --set /etc/heat/heat.conf DEFAULT enable_cloud_watch_lite false\n\n\nManage DB\n---------\n\nOn node 1:\n\n su heat -s /bin/sh -c \"heat-manage db_sync\"\n\nStart services, open firewall ports\n-----------------------------------\n\nOn all nodes:\n\n systemctl start openstack-heat-api\n systemctl start openstack-heat-api-cfn\n systemctl start openstack-heat-api-cloudwatch\n systemctl start openstack-heat-engine\n systemctl enable openstack-heat-api\n systemctl enable openstack-heat-api-cfn\n systemctl enable openstack-heat-api-cloudwatch\n systemctl enable openstack-heat-engine\n firewall-cmd --add-port=8000/tcp\n firewall-cmd --add-port=8000/tcp --permanent\n firewall-cmd --add-port=8003/tcp\n firewall-cmd --add-port=8003/tcp --permanent\n firewall-cmd --add-port=8004/tcp\n firewall-cmd --add-port=8004/tcp --permanent\n" }, { "path": "keepalived/horizon-config.md", "content": "Introduction\n------------\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/horizon.scenario).\n\nInstall software\n----------------\n\n yum install -y mod_wsgi httpd mod_ssl python-memcached openstack-dashboard\n\nSet secret key\n--------------\n\nOn node 1:\n\n openssl rand -hex 10\n\nTake note of the generated random value, then on all nodes:\n\n sed -i -e \"s#SECRET_KEY.*#SECRET_KEY = 'VALUE'#g#\" /etc/openstack-dashboard/local_settings\n\nConfigure local\\_settings and httpd.conf\n----------------------------------------\n\n sed -i -e \"s#ALLOWED_HOSTS.*#ALLOWED_HOSTS = ['*',]#g\" \\\n -e \"s#^CACHES#SESSION_ENGINE = 'django.contrib.sessions.backends.cache'\\nCACHES#g#\" \\\n -e \"s#locmem.LocMemCache'#memcached.MemcachedCache',\\n\\t'LOCATION' : [ 'hacontroller1:11211', 'hacontroller2:11211', 'hacontroller3:11211', ]#g\" \\\n -e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = \"controller-vip.example.com\"#g' \\\n -e \"s#^LOCAL_PATH.*#LOCAL_PATH = '/var/lib/openstack-dashboard'#g\" \\\n /etc/openstack-dashboard/local_settings\n\nRestart httpd and open firewall port\n------------------------------------\n\n systemctl daemon-reload\n systemctl restart httpd\n firewall-cmd --add-port=80/tcp\n firewall-cmd --add-port=80/tcp --permanent\n" }, { "path": "keepalived/keepalived-config.md", "content": "Introduction\n------------\n\n[Keepalived](http://www.keepalived.org/) provides simple and robust facilities for load balancing and high-availability to Linux system and Linux based infrastructures. In this highly available OpenStack architecture, it is used to provide high availability to the virtual IP(s) used by HAProxy. High-availability is achieved by VRRP protocol, a fundamental brick for router failover.\n\n![](Keepalived-arch.jpg \"Keepalived architecture\")\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/keepalived.scenario).\n\nInstall software\n----------------\n\n yum -y install keepalived psmisc\n\nCreate configuration file\n-------------------------\n\nOn all nodes:\n\n cat > /etc/keepalived/keepalived.conf << EOF\n\n vrrp_script chk_haproxy {\n script \"/usr/bin/killall -0 haproxy\"\n interval 2\n }\n\n vrrp_instance VI_PUBLIC {\n interface eth1\n state BACKUP\n virtual_router_id 52\n priority 101\n virtual_ipaddress {\n 192.168.1.220 dev eth1\n }\n track_script {\n chk_haproxy\n }\n # Avoid failback\n nopreempt\n }\n\n vrrp_sync_group VG1\n group {\n VI_PUBLIC\n }\n EOF\n\n\nOpen firewall rules and start services\n--------------------------------------\n\nOn all nodes:\n\n firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -i eth1 -d 224.0.0.0/8 -j ACCEPT\n firewall-cmd --direct --perm --add-rule ipv4 filter INPUT 0 -i eth1 -d 224.0.0.0/8 -j ACCEPT\n systemctl start keepalived\n systemctl enable keepalived\n" }, { "path": "keepalived/keystone-config.md", "content": "Introduction\n------------\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/keystone.scenario).\n\nInstall software\n----------------\n\n yum install -y openstack-keystone openstack-utils openstack-selinux httpd mod_wsgi python-openstackclient\n\nCreate service token and distribute to the other controllers\n------------------------------------------------------------\n\nOn node 1:\n\n export SERVICE_TOKEN=$(openssl rand -hex 10)\n echo $SERVICE_TOKEN > /root/keystone_service_token\n scp /root/keystone_service_token root@hacontroller2:/root\n scp /root/keystone_service_token root@hacontroller3:/root\n\nConfigure Apache web server for Keystone\n----------------------------------------\n\n**NOTE:** running Keystone under eventlet has been deprecated as of the Kilo release. Support for utilizing eventlet will be removed as of the M-release. Thus, instructions are provided to run Keystone under the Apache web server, as a WSGI process.\n\nOn all nodes:\n\n cp /usr/share/keystone/wsgi-keystone.conf /etc/httpd/conf.d/\n sed -i -e 's/apache2/httpd/g' /etc/httpd/conf.d/wsgi-keystone.conf\n sed -i -e 's/VirtualHost \\*/VirtualHost 192.168.1.22X/g' /etc/httpd/conf.d/wsgi-keystone.conf \n sed -i -e 's/Listen 5000/Listen 192.168.1.22X:5000/g' /etc/httpd/conf.d/wsgi-keystone.conf \n sed -i -e 's/Listen 35357/Listen 192.168.1.22X:35357/g' /etc/httpd/conf.d/wsgi-keystone.conf \n sed -i -e 's/^Listen.*/Listen 192.168.1.22X:80/g' /etc/httpd/conf/httpd.conf \n\nConfigure Keystone\n------------------\n\nOn all nodes:\n\n export SERVICE_TOKEN=$(cat /root/keystone_service_token)\n openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token $SERVICE_TOKEN\n openstack-config --set /etc/keystone/keystone.conf DEFAULT rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/keystone/keystone.conf DEFAULT rabbit_ha_queues true\n openstack-config --set /etc/keystone/keystone.conf eventlet_server admin_endpoint 'http://controller-vip.example.com:%(admin_port)s/'\n openstack-config --set /etc/keystone/keystone.conf eventlet_server public_endpoint 'http://controller-vip.example.com:%(public_port)s/'\n openstack-config --set /etc/keystone/keystone.conf database connection mysql://keystone:keystonetest@controller-vip.example.com/keystone\n openstack-config --set /etc/keystone/keystone.conf database max_retries -1\n openstack-config --set /etc/keystone/keystone.conf DEFAULT public_bind_host 192.168.1.22X\n openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_bind_host 192.168.1.22X\n openstack-config --set /etc/keystone/keystone.conf token driver keystone.token.persistence.backends.sql.Token\n\nCreate and distribute PKI setup, manage DB\n------------------------------------------\n\nOn node 1:\n\n keystone-manage pki_setup --keystone-user keystone --keystone-group keystone\n chown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/\n su keystone -s /bin/sh -c \"keystone-manage db_sync\"\n cd /etc/keystone/ssl\n tar cvfz /tmp/keystone_ssl.tgz *\n scp /tmp/keystone_ssl.tgz hacontroller2:/tmp\n scp /tmp/keystone_ssl.tgz hacontroller3:/tmp\n\nRestore Keystone PKI setup from node 1\n--------------------------------------\n\nOn nodes 2 and 3:\n\n mkdir -p /etc/keystone/ssl\n cd /etc/keystone/ssl\n tar xvfz /tmp/keystone_ssl.tgz \n chown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/\n restorecon -Rv /etc/keystone/ssl\n touch /var/log/keystone/keystone.log\n chown keystone:keystone /var/log/keystone/keystone.log\n\nCreate cron job to flush expired tokens\n---------------------------------------\n\nOn all nodes:\n\n echo \"1 * * * * keystone keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1\" >> /etc/crontab\n\nStart services and open firewall ports\n--------------------------------------\n\nOn all nodes;\n\n firewall-cmd --add-port=5000/tcp\n firewall-cmd --add-port=5000/tcp --permanent\n firewall-cmd --add-port=35357/tcp\n firewall-cmd --add-port=35357/tcp --permanent\n systemctl start httpd\n systemctl enable httpd\n\nCreate endpoints, services and users for all API services\n---------------------------------------------------------\n\nOn node 1:\n\n export OS_TOKEN=$(cat /root/keystone_service_token)\n export OS_URL=http://controller-vip.example.com:35357/v2.0\n export OS_REGION_NAME=regionOne\n openstack service create --name=keystone --description=\"Keystone Identity Service\" identity\n openstack endpoint create --publicurl 'http://controller-vip.example.com:5000/v2.0' --adminurl 'http://controller-vip.example.com:35357/v2.0' --internalurl 'http://controller-vip.example.com:5000/v2.0' --region regionOne keystone\n openstack user create --password keystonetest admin\n openstack role create admin\n openstack project create admin\n openstack role add --project admin --user admin admin\n openstack user create --password redhat demo\n openstack role create _member_\n openstack project create demo\n openstack role add --project demo --user demo _member_\n openstack project create --description \"Services Tenant\" services\n # glance\n openstack user create --password glancetest glance\n openstack role add --project services --user glance admin\n openstack service create --name=glance --description=\"Glance Image Service\" image\n openstack endpoint create --publicurl 'http://controller-vip.example.com:9292' --adminurl 'http://controller-vip.example.com:9292' --internalurl 'http://controller-vip.example.com:9292' --region regionOne glance\n # cinder\n openstack user create --password cindertest cinder\n openstack role add --project services --user cinder admin\n openstack service create --name=cinder --description=\"Cinder Volume Service\" volume\n openstack endpoint create --publicurl \"http://controller-vip.example.com:8776/v1/\\$(tenant_id)s\" --adminurl \"http://controller-vip.example.com:8776/v1/\\$(tenant_id)s\" --internalurl \"http://controller-vip.example.com:8776/v1/\\$(tenant_id)s\" --region regionOne cinder\n openstack service create --name=cinderv2 --description=\"OpenStack Block Storage\" volumev2\n openstack endpoint create --publicurl \"http://controller-vip.example.com:8776/v2/\\$(tenant_id)s\" --adminurl \"http://controller-vip.example.com:8776/v2/\\$(tenant_id)s\" --internalurl \"http://controller-vip.example.com:8776/v2/\\$(tenant_id)s\" --region regionOne cinderv2\n # swift\n openstack user create --password swifttest swift\n openstack role add --project services --user swift admin\n openstack service create --name=swift --description=\"Swift Storage Service\" object-store\n openstack endpoint create --publicurl \"http://controller-vip.example.com:8080/v1/AUTH_\\$(tenant_id)s\" --adminurl \"http://controller-vip.example.com:8080/v1\" --internalurl \"http://controller-vip.example.com:8080/v1/AUTH_\\$(tenant_id)s\" --region regionOne swift\n # neutron\n openstack user create --password neutrontest neutron\n openstack role add --project services --user neutron admin\n openstack service create --name=neutron --description=\"OpenStack Networking Service\" network\n openstack endpoint create --publicurl \"http://controller-vip.example.com:9696\" --adminurl \"http://controller-vip.example.com:9696\" --internalurl \"http://controller-vip.example.com:9696\" --region regionOne neutron\n # nova\n openstack user create --password novatest compute\n openstack role add --project services --user compute admin\n openstack service create --name=compute --description=\"OpenStack Compute Service\" compute\n openstack endpoint create --publicurl \"http://controller-vip.example.com:8774/v2/\\$(tenant_id)s\" --adminurl \"http://controller-vip.example.com:8774/v2/\\$(tenant_id)s\" --internalurl \"http://controller-vip.example.com:8774/v2/\\$(tenant_id)s\" --region regionOne compute\n # heat\n openstack user create --password heattest heat\n openstack role add --project services --user heat admin\n openstack service create --name=heat --description=\"Heat Orchestration Service\" orchestration\n openstack endpoint create --publicurl \"http://controller-vip.example.com:8004/v1/%(tenant_id)s\" --adminurl \"http://controller-vip.example.com:8004/v1/%(tenant_id)s\" --internalurl \"http://controller-vip.example.com:8004/v1/%(tenant_id)s\" --region regionOne heat\n openstack service create --name=heat-cfn --description=\"Heat CloudFormation Service\" cloudformation\n openstack endpoint create --publicurl \"http://controller-vip.example.com:8000/v1\" --adminurl \"http://controller-vip.example.com:8000/v1\" --internalurl \"http://controller-vip.example.com:8000/v1\" --region regionOne heat-cfn\n # ceilometer\n openstack user create --password ceilometertest ceilometer\n openstack role add --project services --user ceilometer admin\n openstack role create ResellerAdmin\n openstack role add --project services --user ceilometer ResellerAdmin\n openstack service create --name=ceilometer --description=\"OpenStack Telemetry Service\" metering\n openstack endpoint create --publicurl \"http://controller-vip.example.com:8777\" --adminurl \"http://controller-vip.example.com:8777\" --internalurl \"http://controller-vip.example.com:8777\" --region regionOne ceilometer\n # sahara\n openstack user create --password saharatest sahara\n openstack role add --project services --user sahara admin\n openstack service create --name=sahara --description=\"Sahara Data Processing\" data-processing\n openstack endpoint create --publicurl \"http://controller-vip.example.com:8386/v1.1/%(tenant_id)s\" --adminurl \"http://controller-vip.example.com:8386/v1.1/%(tenant_id)s\" --internalurl \"http://controller-vip.example.com:8386/v1.1/%(tenant_id)s\" --region regionOne sahara\n # trove\n openstack user create --password trovetest trove\n openstack role add --project services --user trove admin\n openstack service create --name=trove --description=\"OpenStack Database Service\" database\n openstack endpoint create --publicurl \"http://controller-vip.example.com:8779/v1.0/%(tenant_id)s\" --adminurl \"http://controller-vip.example.com:8779/v1.0/%(tenant_id)s\" --internalurl \"http://controller-vip.example.com:8779/v1.0/%(tenant_id)s\" --region regionOne trove\n\nCreate keystonerc files for simplicity\n--------------------------------------\n\nOn all nodes:\n\n cat > /root/keystonerc_admin << EOF\n export OS_USERNAME=admin \n export OS_TENANT_NAME=admin\n export OS_PROJECT_NAME=admin\n export OS_REGION_NAME=regionOne\n export OS_PASSWORD=keystonetest\n export OS_AUTH_URL=http://controller-vip.example.com:35357/v2.0/\n export PS1='[\\u@\\h \\W(keystone_admin)]\\$ '\n EOF\n\n cat > /root/keystonerc_demo << EOF\n export OS_USERNAME=demo\n export OS_TENANT_NAME=demo\n export OS_PROJECT_NAME=demo\n export OS_REGION_NAME=regionOne\n export OS_PASSWORD=redhat\n export OS_AUTH_URL=http://controller-vip.example.com:5000/v2.0/\n export PS1='[\\u@\\h \\W(keystone_user)]\\$ '\n EOF\n" }, { "path": "keepalived/memcached-config.md", "content": "Introduction\n------------\n\nMemcached is a general-purpose distributed memory caching system. It is used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source must be read.\n\n**Note:** Access to memcached is not handled by HAproxy because replicated access is currently only in an experimental state. Instead consumers must be supplied with the full list of hosts running memcached.\n\nThe following commands will be executed on all controller nodes.\n\nYou can find a phd scenario file [here](phd-setup/memcached.scenario).\n\nInstall and enable memcached\n----------------------------\n\n yum install -y memcached\n systemctl start memcached\n systemctl enable memcached\n firewall-cmd --add-port=11211/tcp\n firewall-cmd --add-port=11211/tcp --permanent\n" }, { "path": "keepalived/mongodb-config.md", "content": "Introduction\n------------\n\nMongoDB can provide high availability through the use of replica sets. A replica set in MongoDB is a group of mongod processes that maintain the same data set, where one of the nodes is specified as master and the rest as slaves. Clients are explicitly told to connect to the replica set, by specifying all its members. In case of a node failure, the client should transparently reconnect to a surviving replica.\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/mongodb.scenario).\n\nInstall packages\n----------------\n\n yum install -y mongodb mongodb-server\n\nListen to external connections, and configure replication set\n-------------------------------------------------------------\n\n sed -i -e 's#bind_ip.*#bind_ip = 0.0.0.0#g' /etc/mongod.conf\n echo \"replSet = ceilometer\" >> /etc/mongod.conf \n\nStart services and enable firewall ports\n----------------------------------------\n\n systemctl start mongod\n systemctl enable mongod\n firewall-cmd --add-port=27017/tcp\n firewall-cmd --add-port=27017/tcp --permanent\n\nCreate replica set\n------------------\n\nOn node 1:\n\n mongo\n\n\n > rs.initiate()\n > sleep(10000)\n > rs.add(\"hacontroller2.example.com\");\n > rs.add(\"hacontroller3.example.com\");\n\nAnd verify:\n\n > rs.status()\n\nUntil all nodes show `\"stateStr\" : \"PRIMARY\"` or `\"stateStr\" : \"SECONDARY\"`, then:\n\n > quit()\n" }, { "path": "keepalived/mongodb-recovery.md", "content": "Introduction\n------------\n\nMongoDB usually does a good job at re-forming a replica set after a full cluster reboot. In case of any failure, [its documentation](http://docs.mongodb.org/v2.6/tutorial/troubleshoot-replica-sets/) provides an excellent reference on how to troubleshoot and fix any error.\n" }, { "path": "keepalived/neutron-config.md", "content": "Introduction\n------------\n\nFor this setup, we will configure an external provider network using eth0 (10.10.10.X network). A different setup would apply for bridged networks, although the HA features should be the same.\n\nThere are two services that require special attention for a highly available architecture:\n\n- Neutron DHCP agent availability is obtained by assigning two or more agents to manage each tenant network, setting `dhcp_agents_per_network=2` in `/etc/neutron/neutron.conf`.\n- Neutron L3 agent availability uses the L3 HA functionality using VRRP. Using this functionality, a new type of router, spawned on two or more different agents, is created. One agent will be in charge of the master version of this router, and the remaining L3 agents will be in charge of the slave routers. Refer to [this blog post](http://assafmuller.com/2014/08/16/layer-3-high-availability/) for a detailed description of the feature.\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/neutron.scenario).\n\nInstall software\n----------------\n\n yum install -y openstack-neutron openstack-neutron-openvswitch openstack-neutron-ml2 openstack-utils openstack-selinux\n\nConfigure Neutron server\n------------------------\n\n openstack-config --set /etc/neutron/neutron.conf DEFAULT bind_host 192.168.1.22X\n openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password neutrontest\n openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name services\n openstack-config --set /etc/neutron/neutron.conf database connection mysql://neutron:neutrontest@controller-vip.example.com:3306/neutron\n openstack-config --set /etc/neutron/neutron.conf database max_retries -1\n openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier\n openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/neutron/neutron.conf nova nova_region_name regionOne\n openstack-config --set /etc/neutron/neutron.conf nova project_domain_id default\n openstack-config --set /etc/neutron/neutron.conf nova project_name services\n openstack-config --set /etc/neutron/neutron.conf nova user_domain_id default\n openstack-config --set /etc/neutron/neutron.conf nova password novatest\n openstack-config --set /etc/neutron/neutron.conf nova username compute\n openstack-config --set /etc/neutron/neutron.conf nova auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/neutron/neutron.conf nova auth_plugin password\n openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes True\n openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_data_changes True\n openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin neutron.plugins.ml2.plugin.Ml2Plugin\n openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router\n openstack-config --set /etc/neutron/neutron.conf DEFAULT router_scheduler_driver neutron.scheduler.l3_agent_scheduler.ChanceScheduler\n openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 2\n openstack-config --set /etc/neutron/neutron.conf DEFAULT api_workers 2\n openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_workers 2\n openstack-config --set /etc/neutron/neutron.conf DEFAULT l3_ha True\n openstack-config --set /etc/neutron/neutron.conf DEFAULT min_l3_agents_per_router 2\n openstack-config --set /etc/neutron/neutron.conf DEFAULT max_l3_agents_per_router 2\n\n### ML2 configuration\n\n ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini\n openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 type_drivers local,gre,flat,vxlan,vlan\n openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan\n openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers openvswitch\n openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks \\*\n openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_gre tunnel_id_ranges 10:10000\n openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vni_ranges 10:10000\n openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vxlan_group 224.0.0.1\n openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_security_group True\n openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver \n\n### LBaaS configuration (optional)\n\n yum -y install openstack-neutron-lbaas\n openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router,lbaas\n openstack-config --set /etc/neutron/lbaas_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver\n openstack-config --set /etc/neutron/lbaas_agent.ini DEFAULT device_driver neutron_lbaas.services.loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver\n openstack-config --set /etc/neutron/lbaas_agent.ini haproxy user_group haproxy \n\n### FWaaS configuration (optional)\n\n yum -y install openstack-neutron-fwaas\n openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router,firewall,lbaas\n openstack-config --set /etc/neutron/fwaas_driver.ini fwaas enabled True\n openstack-config --set /etc/neutron/fwaas_driver.ini fwaas driver neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver\n\nManage DB\n---------\n\nOn node 1:\n\n neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head\n\nStart services, open firewall ports\n-----------------------------------\n\nOn all nodes:\n\n systemctl start neutron-server\n systemctl enable neutron-server\n firewall-cmd --add-port=9696/tcp\n firewall-cmd --add-port=9696/tcp --permanent\n\nOpenvSwitch configuration\n-------------------------\n\n systemctl enable openvswitch\n systemctl start openvswitch\n\n**Note:** we have seeing issues when trying to configure an IP on br-eth0 (specially ARP problems), so it is not recommended.\n\nAssuming eth0 is your interface attached to the external network, create two files in _/etc/sysconfig/network-scripts/_ as follows (change MTU if you need):\n \n\t\tcat < /etc/sysconfig/network-scripts/ifcfg-eth0\n\t\tDEVICE=eth0\n\t\tONBOOT=yes\n\t\tDEVICETYPE=ovs\n\t\tTYPE=OVSPort\n\t\tOVS_BRIDGE=br-eth0\n\t\tONBOOT=yes\n\t\tBOOTPROTO=none\n\t\tVLAN=yes\n\t\tMTU=\"9000\"\n\t\tNM_CONTROLLED=no\n\t\tEOF\n\n\t\tcat < /etc/sysconfig/network-scripts/ifcfg-br-eth0\n\t\tDEVICE=br-eth0\n\t\tDEVICETYPE=ovs\n\t\tOVSBOOTPROTO=none\n\t\tTYPE=OVSBridge\n\t\tONBOOT=yes\n\t\tBOOTPROTO=static\n\t\tMTU=\"9000\"\n\t\tNM_CONTROLLED=no\n\t\tEOF\n\nRestart the network for the changes to take effect.\n \n systemctl restart network\n\nOpenvSwitch agent\n-----------------\n\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent tunnel_types vxlan\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 4789\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs local_ip 192.168.1.22X\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs enable_tunneling True\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs integration_bridge br-int\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_bridge br-tun\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs bridge_mappings physnet1:br-eth0\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs network_vlan_ranges physnet1\n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver \n openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent l2_population False\n\nMetadata agent\n--------------\n\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_strategy keystone\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_url http://controller-vip.example.com:35357/v2.0\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_host controller-vip.example.com\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_region regionOne\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_tenant_name services\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_user neutron\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_password neutrontest\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip controller-vip.example.com\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_port 8775\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret metatest\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_workers 4\n openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_backlog 2048\n\nDHCP agent\n----------\n\n openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver\n openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_config_file /etc/neutron/dnsmasq-neutron.conf\n\nThe following will prevent issues from happening when the network card MTU is 1500. If we are using jumbo frames, it should not be required. Be aware that this only helps on certain operating systems with a well-behaving DHCP client. Windows is known to ignore it.\n\n echo \"dhcp-option-force=26,1400\" > /etc/neutron/dnsmasq-neutron.conf\n chown root:neutron /etc/neutron/dnsmasq-neutron.conf\n chmod 644 /etc/neutron/dnsmasq-neutron.conf\n\nL3 agent\n--------\n\n openstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver\n openstack-config --set /etc/neutron/l3_agent.ini DEFAULT handle_internal_only_routers True\n openstack-config --set /etc/neutron/l3_agent.ini DEFAULT send_arp_for_ha 3\n openstack-config --set /etc/neutron/l3_agent.ini DEFAULT metadata_ip controller-vip.example.com\n openstack-config --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge\n\nStart services and open VXLAN firewall port\n-------------------------------------------\n\n systemctl start neutron-openvswitch-agent\n systemctl start neutron-dhcp-agent\n systemctl start neutron-l3-agent\n systemctl start neutron-metadata-agent\n systemctl start neutron-lbaas-agent\n systemctl enable neutron-openvswitch-agent\n systemctl enable neutron-dhcp-agent\n systemctl enable neutron-l3-agent\n systemctl enable neutron-metadata-agent\n systemctl enable neutron-ovs-cleanup\n systemctl enable neutron-lbaas-agent\n\n firewall-cmd --add-port=4789/udp\n firewall-cmd --add-port=4789/udp --permanent\n\n**NOTE:** During a full cluster reboot, since Galera does not start cleanly neutron-server will wait for some time, then fail due to a service startup timeout (see [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1188198) for details). We can fix that by creating a file named `/etc/systemd/system/neutron-server.service.d/restart.conf` with the following contents:\n\n [Service]\n Restart=on-failure\n\nNeutron server will try to restart indefinitely, then eventually succeed as soon as the Galera DB is running.\n\nCreate provider network\n-----------------------\n\nOn node 1:\n\n . /root/keystonerc_admin\n neutron net-create public --provider:network_type flat --provider:physical_network physnet1 --router:external\n neutron subnet-create --gateway 10.10.10.1 --allocation-pool start=10.10.10.100,end=10.10.10.150 --disable-dhcp --name public_subnet public 10.10.10.0/24\n" }, { "path": "keepalived/nova-config.md", "content": "Introduction\n------------\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/nova.scenario).\n\nInstall software\n----------------\n\n yum install -y openstack-nova-console openstack-nova-novncproxy openstack-utils openstack-nova-api openstack-nova-conductor openstack-nova-scheduler python-cinderclient python-memcached\n\nConfigure Nova API\n------------------\n\n openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211\n openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_host 192.168.1.22X\n openstack-config --set /etc/nova/nova.conf vnc novncproxy_base_url http://controller-vip.example.com:6080/vnc_auto.html\n openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.1.22X\n openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 192.168.1.22X\n openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@controller-vip.example.com/nova\n openstack-config --set /etc/nova/nova.conf database max_retries -1\n openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone\n openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen 192.168.1.22X\n openstack-config --set /etc/nova/nova.conf DEFAULT metadata_host 192.168.1.22X\n openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 192.168.1.22X\n openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen_port 8775\n openstack-config --set /etc/nova/nova.conf DEFAULT glance_host controller-vip.example.com\n openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API\n openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver\n openstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver\n openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron\n openstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL\n openstack-config --set /etc/nova/nova.conf conductor use_local false\n openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True\n openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True\n openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest\n openstack-config --set /etc/nova/nova.conf neutron url http://controller-vip.example.com:9696/\n openstack-config --set /etc/nova/nova.conf neutron project_domain_id default\n openstack-config --set /etc/nova/nova.conf neutron project_name services\n openstack-config --set /etc/nova/nova.conf neutron user_domain_id default\n openstack-config --set /etc/nova/nova.conf neutron username neutron\n openstack-config --set /etc/nova/nova.conf neutron password neutrontest\n openstack-config --set /etc/nova/nova.conf neutron auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/nova/nova.conf neutron auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/nova/nova.conf neutron auth_plugin password\n openstack-config --set /etc/nova/nova.conf neutron region_name regionOne\n\n # REQUIRED FOR A/A scheduler\n openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_host_subset_size 30\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_plugin password\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken username compute\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken password novatest\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken project_name services\n openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_uri http://controller-vip.example.com:5000/\n\n\nOnly run the following command if you are creating a test environment where your hypervisors will be virtual machines\n\n openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu\n\nManage DB\n---------\n\nOn node 1:\n\n su nova -s /bin/sh -c \"nova-manage db sync\"\n\nStart services, open firewall ports\n-----------------------------------\n\nOn all nodes:\n\n systemctl start openstack-nova-consoleauth\n systemctl start openstack-nova-novncproxy \n systemctl start openstack-nova-api\n systemctl start openstack-nova-scheduler\n systemctl start openstack-nova-conductor\n systemctl enable openstack-nova-consoleauth\n systemctl enable openstack-nova-novncproxy \n systemctl enable openstack-nova-api\n systemctl enable openstack-nova-scheduler\n systemctl enable openstack-nova-conductor\n\n firewall-cmd --add-port=8773-8775/tcp\n firewall-cmd --add-port=8773-8775/tcp --permanent\n firewall-cmd --add-port=6080/tcp\n firewall-cmd --add-port=6080/tcp --permanent\n" }, { "path": "keepalived/phd-setup/ceilometer.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Install Ceilometer\n# - Configuring Ceilometer\n# - Start services and add firewall rules\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_nic_external\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_hosts_controllers\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_hosts_mongodb\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_neutron_externalgateway\nPHD_VAR_network_neutron_externalnetwork\nPHD_VAR_network_neutron_allocpoolstart\nPHD_VAR_network_neutron_allocpoolend\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\nyum install -y openstack-ceilometer-api openstack-ceilometer-central openstack-ceilometer-collector openstack-ceilometer-common openstack-ceilometer-alarm python-ceilometer python-ceilometerclient\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_user ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_password ceilometertest\nopenstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://${PHD_VAR_network_hosts_vip}:5000/v2.0 \nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest\nopenstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://${PHD_VAR_network_hosts_mongodb}:27017/ceilometer?replicaSet=ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf database max_retries -1\n\n# keep last 5 days data only (value is in secs)\nopenstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000\nopenstack-config --set /etc/ceilometer/ceilometer.conf api host ${myip}\n\nIFS=', ' read -a controller_names <<< \"${PHD_VAR_network_hosts_controllers}\"\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf coordination backend_url \"redis://${controller_names[0]}:26379?sentinel=mymaster&sentinel_fallback=${controller_names[0]}:26379&sentinel_fallback=${controller_names[0]}:26379\"\n\nsystemctl start openstack-ceilometer-central \nsystemctl enable openstack-ceilometer-central \nsystemctl start openstack-ceilometer-collector\nsystemctl enable openstack-ceilometer-collector\nsystemctl start openstack-ceilometer-api \nsystemctl enable openstack-ceilometer-api \nsystemctl start openstack-ceilometer-alarm-evaluator\nsystemctl enable openstack-ceilometer-alarm-evaluator \nsystemctl start openstack-ceilometer-alarm-notifier\nsystemctl enable openstack-ceilometer-alarm-notifier\nsystemctl start openstack-ceilometer-notification\nsystemctl enable openstack-ceilometer-notification\nfirewall-cmd --add-port=8777/tcp\nfirewall-cmd --add-port=8777/tcp --permanent\nfirewall-cmd --add-port=4952/udp\nfirewall-cmd --add-port=4952/udp --permanent\n....\n" }, { "path": "keepalived/phd-setup/cinder.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Install Cinder\n# - Configure Cinder, using a NFS v3 backend\n# - Start services, open firewall rules\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_nfs_cindershare\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\nyum install -y openstack-cinder openstack-utils openstack-selinux python-memcached\n\nopenstack-config --set /etc/cinder/cinder.conf database connection mysql://cinder:cindertest@${PHD_VAR_network_hosts_vip}/cinder\nopenstack-config --set /etc/cinder/cinder.conf database max_retries -1\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone\nopenstack-config --set /etc/cinder/cinder.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_user cinder\nopenstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_password cindertest\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT notification_driver messaging\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT glance_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT host rhos7-cinder\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen ${myip}\nopenstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues true\n\ncat > /etc/cinder/nfs_exports << EOF\n${PHD_VAR_network_nfs_cindershare}\nEOF\n\nchown root:cinder /etc/cinder/nfs_exports\nchmod 0640 /etc/cinder/nfs_exports\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfs_exports\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_sparsed_volumes true\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_mount_options v3\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.nfs.NfsDriver\n\n....\n\ntarget=$PHD_ENV_nodes1\n....\nsu cinder -s /bin/sh -c \"cinder-manage db sync\"\nsystemctl start openstack-cinder-volume\nsystemctl enable openstack-cinder-volume\n....\n\ntarget=all\n....\nsystemctl start openstack-cinder-api\nsystemctl start openstack-cinder-scheduler\nsystemctl enable openstack-cinder-api\nsystemctl enable openstack-cinder-scheduler\nfirewall-cmd --add-port=8776/tcp\nfirewall-cmd --add-port=8776/tcp --permanent\n....\n" }, { "path": "keepalived/phd-setup/compute.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Install all required packages for a compute node\n# - Configure nova-compute, neutron and ceilometer for the compute node\n# - Tweak TCP keepalive parameters\n# - Start services and open firewall ports\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_nic_external\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_hosts_mongodb\n\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\n# Install packages\nyum install -y openstack-nova-compute openstack-utils python-cinder openstack-neutron-openvswitch openstack-ceilometer-compute openstack-neutron\n# Enable OpenvSwitch\nsystemctl enable openvswitch\nsystemctl start openvswitch\novs-vsctl add-br br-int\n# Configure Nova compute\nopenstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address ${myip}\nopenstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 0.0.0.0\nopenstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://${PHD_VAR_network_hosts_vip}:6080/vnc_auto.html\nopenstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@${PHD_VAR_network_hosts_vip}/nova\nopenstack-config --set /etc/nova/nova.conf database max_retries -1\nopenstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone\nopenstack-config --set /etc/nova/nova.conf glance host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API\nopenstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver\nopenstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver\nopenstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron\nopenstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL\nopenstack-config --set /etc/nova/nova.conf conductor use_local false\nopenstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True\nopenstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True\nopenstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest\nopenstack-config --set /etc/nova/nova.conf neutron url http://${PHD_VAR_network_hosts_vip}:9696/\nopenstack-config --set /etc/nova/nova.conf neutron admin_tenant_name services\nopenstack-config --set /etc/nova/nova.conf neutron admin_username neutron\nopenstack-config --set /etc/nova/nova.conf neutron admin_password neutrontest\nopenstack-config --set /etc/nova/nova.conf neutron admin_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/nova/nova.conf neutron region_name regionOne\nopenstack-config --set /etc/nova/nova.conf libvirt nfs_mount_options v3\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_tenant_name services\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_user compute\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_password novatest\n# Only if the hypervisor is a virtual machine\nopenstack-config --set /etc/nova/nova.conf libvirt virt_type qemu\n# Configure Neutron for compute node\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password neutrontest\nopenstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent tunnel_types vxlan\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent vxlan_udp_port 4789\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs enable_tunneling True\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_id_ranges 1:1000\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tenant_network_type vxlan\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs integration_bridge br-int\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_bridge br-tun\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs local_ip ${myip}\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent l2_population False\n# Configure Ceilometer for compute node\nopenstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit True\nopenstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit_period hour\nopenstack-config --set /etc/nova/nova.conf DEFAULT notify_on_state_change vm_and_task_state\nopenstack-config --set /etc/nova/nova.conf DEFAULT notification_driver nova.openstack.common.notifier.rpc_notifier\nsed -i -e 's/nova.openstack.common.notifier.rpc_notifier/nova.openstack.common.notifier.rpc_notifier\\nnotification_driver = ceilometer.compute.nova_notifier/g' /etc/nova/nova.conf\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_port 35357\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_protocol http\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_user ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_password ceilometertest\nopenstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://${PHD_VAR_network_hosts_vip}:5000/v2.0\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest\nopenstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://${PHD_VAR_network_hosts_mongodb}:27017/ceilometer?replicaSet=ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf database connection max_retries -1\n# keep last 5 days data only (value is in secs)\nopenstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000\n# Cinder uses NFS\nsetsebool -P virt_use_nfs 1\n# Kernel TCP keepalive parameters\ncat > /etc/sysctl.d/tcpka.conf << EOF\nnet.ipv4.tcp_keepalive_intvl = 1\nnet.ipv4.tcp_keepalive_probes = 5\nnet.ipv4.tcp_keepalive_time = 5\nEOF\n\nsysctl -p /etc/sysctl.d/tcpka.conf\n# Start services, open firewall ports\nsystemctl start libvirtd\nsystemctl start neutron-openvswitch-agent\nsystemctl enable neutron-openvswitch-agent\nsystemctl enable neutron-ovs-cleanup\nsystemctl start openstack-ceilometer-compute\nsystemctl enable openstack-ceilometer-compute\nsystemctl start openstack-nova-compute\nsystemctl enable openstack-nova-compute\nfirewall-cmd --add-port=4789/udp\nfirewall-cmd --add-port=4789/udp --permanent\nfirewall-cmd --add-port=5900-5999/tcp\nfirewall-cmd --add-port=5900-5999/tcp --permanent\n....\n\n" }, { "path": "keepalived/phd-setup/galera.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Install MariaDB / Galera\n# - Configure Galera\n# - Bootstrap an initial Galera cluster\n# - Create databases for OpenStack services\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y mariadb-galera-server xinetd rsync psmisc\n\ncat > /etc/sysconfig/clustercheck << EOF\nMYSQL_USERNAME=\"clustercheck\"\nMYSQL_PASSWORD=\"redhat\"\nMYSQL_HOST=\"localhost\"\nMYSQL_PORT=\"3306\"\nEOF\n\nsystemctl start mysqld\nmysql -e \"CREATE USER 'clustercheck'@'localhost' IDENTIFIED BY 'redhat';\"\nsystemctl stop mysqld\n\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\ncat > /etc/my.cnf.d/galera.cnf << EOF\n[mysqld]\nskip-name-resolve=1\nbinlog_format=ROW\ndefault-storage-engine=innodb\ninnodb_autoinc_lock_mode=2\ninnodb_locks_unsafe_for_binlog=1\nmax_connections=2048\nquery_cache_size=0\nquery_cache_type=0\nbind_address=${myip}\nwsrep_provider=/usr/lib64/galera/libgalera_smm.so\nwsrep_cluster_name=\"galera_cluster\"\nwsrep_cluster_address=\"gcomm://192.168.1.221,192.168.1.222,192.168.1.223\"\nwsrep_slave_threads=1\nwsrep_certify_nonPK=1\nwsrep_max_ws_rows=131072\nwsrep_max_ws_size=1073741824\nwsrep_debug=0\nwsrep_convert_LOCK_to_trx=0\nwsrep_retry_autocommit=1\nwsrep_auto_increment_control=1\nwsrep_drupal_282555_workaround=0\nwsrep_causal_reads=0\nwsrep_notify_cmd=\nwsrep_sst_method=rsync\nEOF\n\nmkdir -p /etc/systemd/system/mariadb.service.d/\ncat > /etc/systemd/system/mariadb.service.d/limits.conf << EOF\n[Service]\nLimitNOFILE=16384\nEOF\n\ncat > /etc/xinetd.d/galera-monitor << EOF\nservice galera-monitor\n{\n port = 9200\n disable = no\n socket_type = stream\n protocol = tcp\n wait = no\n user = root\n group = root\n groups = yes\n server = /usr/bin/clustercheck\n type = UNLISTED\n per_source = UNLIMITED\n log_on_success = \n log_on_failure = HOST\n flags = REUSE\n}\nEOF\n\nsystemctl daemon-reload\nsystemctl enable xinetd\nsystemctl start xinetd\nsystemctl enable haproxy\nsystemctl start haproxy\n\nfirewall-cmd --add-service=mysql\nfirewall-cmd --add-port=4444/tcp \nfirewall-cmd --add-port=4567/tcp\nfirewall-cmd --add-port=4568/tcp\nfirewall-cmd --add-port=4568/tcp --permanent\nfirewall-cmd --add-service=mysql --permanent\nfirewall-cmd --add-port=4567/tcp --permanent\nfirewall-cmd --add-port=4444/tcp --permanent\nfirewall-cmd --add-port=9300/tcp\nfirewall-cmd --add-port=9300/tcp --permanent\nfirewall-cmd --add-port=9200/tcp\nfirewall-cmd --add-port=9200/tcp --permanent\n\nsystemctl enable mariadb\n....\n\ntarget=$PHD_ENV_nodes1\n....\n# This is required to allow sudo execution without a tty\nsed -i 's/Defaults requiretty/Defaults !requiretty/g' /etc/sudoers\nnohup sudo -u mysql /usr/libexec/mysqld --wsrep-cluster-address='gcomm://' < /dev/null > /dev/null 2>&1 &\nsleep 30\n# A little cleanup\nsed -i 's/Defaults !requiretty/Defaults requiretty/g' /etc/sudoers\n....\n\ntarget=$PHD_ENV_nodes2\n....\n\nsystemctl start mariadb\nsleep 10\n....\n\ntarget=$PHD_ENV_nodes3\n....\n\nsystemctl start mariadb\nsleep 10\n....\n\ntarget=$PHD_ENV_nodes1\n....\n\ncat > /tmp/mysql.sql << EOF\nuse mysql;\nGRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED by 'mysqltest' WITH GRANT OPTION;\nCREATE DATABASE keystone;\nGRANT ALL ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'keystonetest';\nCREATE DATABASE glance;\nGRANT ALL ON glance.* TO 'glance'@'%' IDENTIFIED BY 'glancetest';\nCREATE DATABASE cinder;\nGRANT ALL ON cinder.* TO 'cinder'@'%' IDENTIFIED BY 'cindertest';\nCREATE DATABASE neutron;\nGRANT ALL ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'neutrontest';\nCREATE DATABASE nova;\nGRANT ALL ON nova.* TO 'nova'@'%' IDENTIFIED BY 'novatest';\nCREATE DATABASE heat;\nGRANT ALL ON heat.* TO 'heat'@'%' IDENTIFIED BY 'heattest';\nCREATE DATABASE sahara;\nGRANT ALL ON sahara.* TO 'sahara'@'%' IDENTIFIED BY 'saharatest';\nCREATE DATABASE trove;\nGRANT ALL ON trove.* TO 'trove'@'%' IDENTIFIED BY 'trovetest';\nFLUSH PRIVILEGES;\nEOF\n\nkillall mysqld\n# it takes some time for mysqld to actually stop after you kill it\nsleep 30 \nsystemctl start mariadb\n\nmysql < /tmp/mysql.sql > /tmp/mysql.out\nrm -f /tmp/mysql.sql\nmysqladmin flush-hosts\n\n....\n" }, { "path": "keepalived/phd-setup/glance.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing Glance\n# - Configuring Glance\n# - Starting services and opening firewall rules\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_nfs_glanceshare\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\nyum install -y openstack-glance openstack-utils openstack-selinux nfs-utils\nopenstack-config --set /etc/glance/glance-api.conf database connection mysql://glance:glancetest@${PHD_VAR_network_hosts_vip}/glance\nopenstack-config --set /etc/glance/glance-api.conf database max_retries -1\nopenstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone\nopenstack-config --set /etc/glance/glance-api.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ \nopenstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_user glance\nopenstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_password glancetest\nopenstack-config --set /etc/glance/glance-api.conf DEFAULT notification_driver messaging\nopenstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host ${myip}\nopenstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/glance/glance-registry.conf database connection mysql://glance:glancetest@${PHD_VAR_network_hosts_vip}/glance\nopenstack-config --set /etc/glance/glance-registry.conf database max_retries -1\nopenstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone\nopenstack-config --set /etc/glance/glance-registry.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_user glance\nopenstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_password glancetest\nopenstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host ${myip}\n....\n\ntarget=$PHD_ENV_nodes1\n....\nsu glance -s /bin/sh -c \"glance-manage db_sync\"\n....\n\ntarget=all\n....\necho \"${PHD_VAR_network_nfs_glanceshare} /var/lib/glance nfs vers=3 0 0\" >> /etc/fstab\n# Workaround for bz#1203820\nsystemctl start rpcbind\nsystemctl start nfs-config\nsystemctl start rpc-statd\nmount -a\nchown glance:nobody /var/lib/glance\nsystemctl start openstack-glance-registry\nsystemctl start openstack-glance-api\nsystemctl enable openstack-glance-registry\nsystemctl enable openstack-glance-api\nfirewall-cmd --add-port=9191/tcp\nfirewall-cmd --add-port=9191/tcp --permanent\nfirewall-cmd --add-port=9292/tcp\nfirewall-cmd --add-port=9292/tcp --permanent\n....\n" }, { "path": "keepalived/phd-setup/ha-collapsed.variables", "content": "# Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD\n# Each scenario file will verify that values have been provided for the variables it requires.\n\ndeployment: collapsed\n\nhypervisors:\n controllers: controller1:oslab1,controller2:oslab2,controller3:oslab3\n computenodes:\tcompute1:oslab1,compute2:oslab3\n\nrhn:\n user: jpena@redhat.com\n pass: OSA07novp7her\n\nnetwork:\n ips:\n vip: 192.168.1.220\n controllers: 192.168.1.221,192.168.1.222,192.168.1.223\n computeinternal: 192.168.1.224,192.168.1.225\n computeexternal: 10.10.10.224,10.10.10.225\n gateway: 192.168.1.1\n netmask: 255.255.255.0\n nfs:\n glanceshare: 192.168.1.4:/volumeUSB1/usbshare/openstack/glance\n cindershare: 192.168.1.4:/volumeUSB1/usbshare/openstack/cinder\n nic:\n base: 54:52:00\n internal: eth0\n external: eth1\n hosts:\n vip: controller-vip.example.com\n controllers: controller1.example.com, controller2.example.com, controller3.example.com\n compute: compute1.example.com, compute2.example.com\n mongodb: controller1,controller2,controller3\n memcache: controller1:11211,controller2:11211,controller3:11211\n rabbitmq: controller1,controller2,controller3\n domain: example.com\n neutron:\n externalgateway: 10.10.10.1\n externalnetwork: 10.10.10.0/24\n allocpoolstart: 10.10.10.100\n allocpoolend: 10.10.10.150\n ssh:\n pubkey: ssh-dss AAAAB3NzaC1kc3MAAACBANTRU30h4EC1sv5kb/KJBDYx0D/YKWAVzMJpeWLzFrECF41Tymte8W5s4oz0CzpQQH4r19YFpeS+jingeakkRc749ImwhJE77yRmPsjaEDe3OZXW+Udz4dikE5HBjuVlEL9UN3yoGfBZCXo9Jm+oRJCH+wx36Pl69IHRZwJIAvQfAAAAFQDTdIeHmLP77PvjG5jo6eEYDTedXwAAAIEAvwKlws7V3qy6Ckd26FcAr1noN+myBdVVpb960F7BPWIxXkA9z12o+3y5JEUJRVTplRf8pBz1JwKdsYS7WJj8teSslmwbQz0kQ61uSNkWqBZGOLE/ddQxCJbD3OTXRZ289A/2mRBcil8mIg82JY1AzNr1OaQQ82ZP35nISA/jeQMAAACBAIeuHCLqkYRf7Xuv2ztFOhPzLrJRRSm7Zcpdxp5Ano7R68Yq1RQbMn0lUIG4QMssviQPyklF/xHYGJLUXBIUfNORn7BYqHikbcguUHKiwiC9io9xR9+zlvgIAIU6bbnsNT9MofTdrRT+H7df6cDP8NO43w8dWFXWX4K3NUU0YjbM\n\ncomponents: serverprep lb galera rabbitmq keystone memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer node\n\nosp:\n major: 7\n\nenv:\n password: cluster\n\n# TODO... \npassword:\n cluster: foo\n keystone: bar\n\n" }, { "path": "keepalived/phd-setup/heat.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing Heat\n# - Creating the required Heat domain\n# - Configuring Heat\n# - Starting services and opening firewall rules\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_nic_external\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_neutron_externalgateway\nPHD_VAR_network_neutron_externalnetwork\nPHD_VAR_network_neutron_allocpoolstart\nPHD_VAR_network_neutron_allocpoolend\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-heat-engine openstack-heat-api openstack-heat-api-cfn openstack-heat-api-cloudwatch python-heatclient openstack-utils python-glanceclient\n....\n\ntarget=$PHD_ENV_nodes1\n....\n. /root/keystonerc_admin\nopenstack role create heat_stack_user\nTOKEN_ID=$(openstack token issue --format value --column id)\n\nopenstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 domain create heat --description \"Owns users and projects created by heat\"\nopenstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 user create --password heattest --domain heat --description \"Manages users and projects created by heat\" heat_domain_admin\nopenstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 role add --user heat_domain_admin --domain heat admin\n....\n\ntarget=all\n....\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\n. /root/keystonerc_admin\nTOKEN_ID=$(openstack token issue --format value --column id)\nHEAT_DOMAIN_ID=$(openstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 domain show heat --column id --format value)\n\nopenstack-config --set /etc/heat/heat.conf DEFAULT stack_domain_admin_password heattest\nopenstack-config --set /etc/heat/heat.conf DEFAULT stack_domain_admin heat_domain_admin\nopenstack-config --set /etc/heat/heat.conf DEFAULT stack_user_domain_id ${HEAT_DOMAIN_ID}\n\nopenstack-config --set /etc/heat/heat.conf database connection mysql://heat:heattest@${PHD_VAR_network_hosts_vip}/heat\nopenstack-config --set /etc/heat/heat.conf database max_retries -1\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken admin_user heat\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken admin_password heattest\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken service_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken keystone_ec2_uri http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/heat/heat.conf ec2authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:5000/v2.0\nopenstack-config --set /etc/heat/heat.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/heat/heat.conf heat_api bind_host ${myip}\nopenstack-config --set /etc/heat/heat.conf heat_api_cfn bind_host ${myip}\nopenstack-config --set /etc/heat/heat.conf heat_api_cloudwatch bind_host ${myip}\nopenstack-config --set /etc/heat/heat.conf DEFAULT heat_metadata_server_url http://${PHD_VAR_network_hosts_vip}:8000\nopenstack-config --set /etc/heat/heat.conf DEFAULT heat_waitcondition_server_url http://${PHD_VAR_network_hosts_vip}:8000/v1/waitcondition\nopenstack-config --set /etc/heat/heat.conf DEFAULT heat_watch_server_url http://${PHD_VAR_network_hosts_vip}:8003\nopenstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/heat/heat.conf DEFAULT rpc_backend rabbit\nopenstack-config --set /etc/heat/heat.conf DEFAULT notification_driver heat.openstack.common.notifier.rpc_notifier\nopenstack-config --set /etc/heat/heat.conf DEFAULT enable_cloud_watch_lite false\n....\n\ntarget=$PHD_ENV_nodes1\n....\nsu heat -s /bin/sh -c \"heat-manage db_sync\"\n....\n\ntarget=all\n....\nsystemctl start openstack-heat-api\nsystemctl start openstack-heat-api-cfn\nsystemctl start openstack-heat-api-cloudwatch\nsystemctl start openstack-heat-engine\nsystemctl enable openstack-heat-api\nsystemctl enable openstack-heat-api-cfn\nsystemctl enable openstack-heat-api-cloudwatch\nsystemctl enable openstack-heat-engine\nfirewall-cmd --add-port=8000/tcp\nfirewall-cmd --add-port=8000/tcp --permanent\nfirewall-cmd --add-port=8003/tcp\nfirewall-cmd --add-port=8003/tcp --permanent\nfirewall-cmd --add-port=8004/tcp\nfirewall-cmd --add-port=8004/tcp --permanent\n....\n" }, { "path": "keepalived/phd-setup/horizon.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing Horizon packages\n# - Configuring the OpenStack dashboard\n# - Starting the service and opening firewall rules\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_hosts_vip\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y mod_wsgi httpd mod_ssl python-memcached openstack-dashboard\n....\n\ntarget=$PHD_ENV_nodes1\n....\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\nopenssl rand -hex 10 > /tmp/horizon_secret.key\nscp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p /tmp/horizon_secret.key ${controller_ips[1]}:/tmp\nscp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p /tmp/horizon_secret.key ${controller_ips[2]}:/tmp\n....\n\ntarget=all\n....\nIFS=', ' read -a memcached_hosts <<< \"${PHD_VAR_network_hosts_memcache}\"\n\nSECRET_KEY=$(cat /tmp/horizon_secret.key)\nrm -f /tmp/horizon_secret.key\n\nsed -i -e \"s#SECRET_KEY.*#SECRET_KEY = \\'${SECRET_KEY}\\'#g#\" /etc/openstack-dashboard/local_settings\n\nsed -i -e \"s#ALLOWED_HOSTS.*#ALLOWED_HOSTS = ['*',]#g\" \\\n-e \"s#^CACHES#SESSION_ENGINE = 'django.contrib.sessions.backends.cache'\\nCACHES#g#\" \\\n-e \"s#locmem.LocMemCache'#memcached.MemcachedCache',\\n\\t'LOCATION' : [ '${memcached_hosts[0]}', '${memcached_hosts[1]}', '${memcached_hosts[2]}', ]#g\" \\\n-e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = \"${PHD_VAR_network_hosts_vip}\"#g' \\\n-e \"s#^LOCAL_PATH.*#LOCAL_PATH = '/var/lib/openstack-dashboard'#g\" \\\n/etc/openstack-dashboard/local_settings\n\nsystemctl daemon-reload\nsystemctl restart httpd\nfirewall-cmd --add-port=80/tcp\nfirewall-cmd --add-port=80/tcp --permanent\n....\n\n\n" }, { "path": "keepalived/phd-setup/hypervisors.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Creating the VMs that form the environment (see readme.txt for details),\n# including required network and storage.\n# - Setting basic repos on the VMs\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_hypervisors_controllers\nPHD_VAR_hypervisors_computenodes\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_ips_computeinternal\nPHD_VAR_network_ips_computeexternal\nPHD_VAR_network_ips_gateway\nPHD_VAR_network_ips_netmask\nPHD_VAR_network_domain\nPHD_VAR_rhn_user\nPHD_VAR_rhn_pass\nPHD_VAR_network_ssh_pubkey\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\ncreate_vm()\n{\n vmname=$1\n export cont_ip=$2\n export external_ip=$3\n\n virt-clone --force -o rhel7-base -n rhel7-${vmname} -f /var/lib/libvirt/images/rhel7-${vmname}.qcow2\n virsh start rhel7-${vmname}\n VMMAC=\"\"\n while [ \"x$VMMAC\" == \"x\" ]\n do\n VMMAC=$(virsh dumpxml rhel7-${vmname} | grep \"mac address\" | awk -F\\' '{print $2}')\n if [ \"x$VMMAC\" == \"x\" ]\n then\n echo \"MAC not found yet, sleeping...\"\n sleep 5\n fi\n done\n\n VMIP=\"\"\n while [ \"x$VMIP\" == \"x\" ]\n do\n VMIP=$(arp -an | grep $VMMAC | awk '{print $2}' | tr -d \\( | tr -d \\))\n if [ \"x$VMIP\" == \"x\" ]\n then\n echo \"IP not found yet, sleeping...\"\n sleep 5\n else\n echo \"VM booted with IP \" $VMIP\n fi\n done\n\n SSHOK=0\n while [ ${SSHOK} -eq 0 ]\n do\n ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null root@${VMIP} ls /root > /dev/null\n if [ $? -eq 0 ]\n then\n SSHOK=1\n else\n echo \"Cannot connect via SSH. Waiting\"\n sleep 10\n fi\n done\n\n ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null root@${VMIP} \"/bin/bash \" << EOF\necho \"${vmname}.${PHD_VAR_network_domain}\" > /etc/hostname\n\nsubscription-manager register --username=${PHD_VAR_rhn_user} --password=${PHD_VAR_rhn_pass} \nsubscription-manager attach --pool=8a85f9874011071c01407da00b997cb2\nsubscription-manager repos --disable \\* \nsubscription-manager repos --enable rhel-7-server-rpms \nsubscription-manager repos --enable rhel-7-server-rh-common-rpms \nsubscription-manager repos --enable rhel-7-server-optional-rpms \nsubscription-manager repos --enable rhel-7-server-extras-rpms \nyum -y install wget vim\nwget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm\nyum -y localinstall epel-release-7*.rpm\nyum -y install http://rdoproject.org/repos/openstack-kilo/rdo-release-kilo.rpm\nyum -y update\n\ncat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOF2\nTYPE=\"Ethernet\"\nBOOTPROTO=\"static\"\nIPV4_FAILURE_FATAL=\"no\"\nNAME=\"eth0\"\nDEVICE=eth0\nIPADDR=${cont_ip}\nNETMASK=${PHD_VAR_network_ips_netmask}\nGATEWAY=${PHD_VAR_network_ips_gateway}\nONBOOT=\"yes\"\nEOF2\n\ncat > /etc/sysconfig/network-scripts/ifcfg-eth1 << EOF3\nTYPE=\"Ethernet\"\nBOOTPROTO=\"static\"\nNAME=\"eth1\"\nDEVICE=eth1\nONBOOT=\"yes\"\nEOF3\n\nif [ \"x${external_ip}\" != \"x\" ]\nthen\n echo \"IPADDR=${external_ip}\" >> /etc/sysconfig/network-scripts/ifcfg-eth1\n echo \"NETMASK=255.255.255.0\" >> /etc/sysconfig/network-scripts/ifcfg-eth1\nfi\n\necho \"NETWORKING=yes\" > /etc/sysconfig/network\necho \"nameserver 8.8.8.8\" > /etc/resolv.conf\necho ${PHD_VAR_network_ssh_pubkey} >> /root/.ssh/authorized_keys\n\n\n#This is a big hack (same ssh private key everywher), until I figure out a better way\necho ${PHD_VAR_network_ssh_pubkey} > /root/.ssh/id_dsa.pub\ncat > /root/.ssh/id_dsa << EOF4\n-----BEGIN DSA PRIVATE KEY-----\nMIIBvAIBAAKBgQDU0VN9IeBAtbL+ZG/yiQQ2MdA/2ClgFczCaXli8xaxAheNU8pr\nXvFubOKM9As6UEB+K9fWBaXkvo4p4HmpJEXO+PSJsISRO+8kZj7I2hA3tzmV1vlH\nc+HYpBORwY7lZRC/VDd8qBnwWQl6PSZvqESQh/sMd+j5evSB0WcCSAL0HwIVANN0\nh4eYs/vs++MbmOjp4RgNN51fAoGBAL8CpcLO1d6sugpHduhXAK9Z6DfpsgXVVaW/\netBewT1iMV5APc9dqPt8uSRFCUVU6ZUX/KQc9ScCnbGEu1iY/LXkrJZsG0M9JEOt\nbkjZFqgWRjixP3XUMQiWw9zk10WdvPQP9pkQXIpfJiIPNiWNQMza9TmkEPNmT9+Z\nyEgP43kDAoGBAIeuHCLqkYRf7Xuv2ztFOhPzLrJRRSm7Zcpdxp5Ano7R68Yq1RQb\nMn0lUIG4QMssviQPyklF/xHYGJLUXBIUfNORn7BYqHikbcguUHKiwiC9io9xR9+z\nlvgIAIU6bbnsNT9MofTdrRT+H7df6cDP8NO43w8dWFXWX4K3NUU0YjbMAhQsr5fG\nvYKFMrvzTcdgEdtWeXx9uA==\n-----END DSA PRIVATE KEY-----\nEOF4\nchmod 600 /root/.ssh/id_dsa \n\nsystemctl disable NetworkManager\nsystemctl enable network\npoweroff\nEOF\n sleep 60 # Give time for poweroff\n virsh dumpxml rhel7-${vmname} > /tmp/vm.txt\n sed -i \"s#type='network'#type='bridge'#\" /tmp/vm.txt\n sed -i \"s#network='default'#bridge='br-public'#\" /tmp/vm.txt\n virsh define /tmp/vm.txt\n virsh attach-interface rhel7-${vmname} bridge br-private --persistent --model virtio\n # Create and attach second disk (used by swift)\n qemu-img create -f qcow2 /var/lib/libvirt/images/rhel7-${vmname}-2.qcow2 8g\n virsh attach-disk --persistent --subdriver qcow2 rhel7-${vmname} /var/lib/libvirt/images/rhel7-${vmname}-2.qcow2 vdb\n virsh start rhel7-${vmname}\n}\n\nIFS=', ' read -a pair <<< \"${PHD_VAR_hypervisors_controllers}\"\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\nfor item in \"${!pair[@]}\"\ndo\n vmname=$(echo ${pair[item]} | awk -F: '{print $1}')\n hypervisor=$(echo ${pair[item]} | awk -F: '{print $2}')\n cont_ip=${controller_ips[item]}\n\n if [ \"$hypervisor\" = `hostname -s` ]\n then\n create_vm $vmname $cont_ip\n fi\ndone\n\nIFS=', ' read -a pair <<< \"${PHD_VAR_hypervisors_computenodes}\"\nIFS=', ' read -a internal_ips <<< \"${PHD_VAR_network_ips_computeinternal}\"\nIFS=', ' read -a external_ips <<< \"${PHD_VAR_network_ips_computeexternal}\"\n\nfor item in \"${!pair[@]}\"\ndo\n vmname=$(echo ${pair[item]} | awk -F: '{print $1}')\n hypervisor=$(echo ${pair[item]} | awk -F: '{print $2}')\n int_ip=${internal_ips[item]}\n ext_ip=${external_ips[item]}\n\n if [ \"$hypervisor\" = `hostname -s` ]\n then\n create_vm $vmname $int_ip $ext_ip\n fi\ndone\n....\n" }, { "path": "keepalived/phd-setup/keepalived.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing keepalived\n# - Configuring keepalived\n# - Starting the service and opening the required firewall rule\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_ips_vip\nPHD_VAR_network_nic_internal\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y keepalived psmisc\n\ncat > /etc/keepalived/keepalived.conf << EOF\nvrrp_script chk_haproxy {\n script \"/usr/bin/killall -0 haproxy\"\n interval 2\n}\n\nvrrp_instance VI_PUBLIC {\n interface ${PHD_VAR_network_nic_internal}\n state BACKUP\n virtual_router_id 52\n priority 101\n virtual_ipaddress {\n ${PHD_VAR_network_ips_vip} dev ${PHD_VAR_network_nic_internal}\n }\n track_script {\n chk_haproxy\n }\n}\n\nvrrp_sync_group VG1\n group {\n VI_PUBLIC\n }\nEOF\n\nfirewall-cmd --direct --add-rule ipv4 filter INPUT 0 -i ${PHD_VAR_network_nic_internal} -d 224.0.0.0/8 -j ACCEPT\nfirewall-cmd --direct --perm --add-rule ipv4 filter INPUT 0 -i ${PHD_VAR_network_nic_internal} -d 224.0.0.0/8 -j ACCEPT\nsystemctl start keepalived\nsystemctl enable keepalived\n....\n" }, { "path": "keepalived/phd-setup/keystone.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing Keystone\n# - Configuring Keystone\n# - Starting the service, opening firewall rules\n# - Creating user accounts, roles, projects, services and endpoints for all\n# services\n# - Creating a keystonerc file\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-keystone openstack-utils openstack-selinux httpd mod_wsgi python-openstackclient\n....\n\ntarget=$PHD_ENV_nodes1\n....\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\nexport SERVICE_TOKEN=$(openssl rand -hex 10)\necho $SERVICE_TOKEN > /root/keystone_service_token\nscp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /root/keystone_service_token root@${controller_ips[1]}:/root\nscp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /root/keystone_service_token root@${controller_ips[2]}:/root\n....\n\ntarget=all\n....\nservername=$(hostname -s)\n\n\nmkdir /var/www/cgi-bin/keystone\nchown keystone:keystone /var/www/cgi-bin/keystone\nln /usr/share/keystone/keystone.wsgi /var/www/cgi-bin/keystone/admin\nln /usr/share/keystone/keystone.wsgi /var/www/cgi-bin/keystone/main\nrestorecon -R /var/www/cgi-bin/keystone /etc/httpd \n\ncat > /etc/httpd/conf.d/10-keystone_wsgi_admin.conf << EOF\n\n ServerName ${servername}\n ## Vhost docroot\n DocumentRoot \"/var/www/cgi-bin/keystone\"\n\n ## Directories, there should at least be a declaration for /var/www/cgi-bin/keystone\n\n \n Options Indexes FollowSymLinks MultiViews\n AllowOverride None\n Require all granted\n \n\n ## Logging\n ErrorLog \"/var/log/httpd/keystone_wsgi_admin_error.log\"\n ServerSignature Off\n CustomLog \"/var/log/httpd/keystone_wsgi_admin_access.log\" combined \n WSGIDaemonProcess keystone_admin group=keystone processes=1 threads=1 user=keystone\n WSGIProcessGroup keystone_admin\n WSGIScriptAlias / \"/var/www/cgi-bin/keystone/admin\"\n\nEOF\n\ncat > /etc/httpd/conf.d/10-keystone_wsgi_main.conf << EOF\n\n ServerName ${servername}\n ## Vhost docroot\n DocumentRoot \"/var/www/cgi-bin/keystone\"\n\n ## Directories, there should at least be a declaration for /var/www/cgi-bin/keystone\n\n \n Options Indexes FollowSymLinks MultiViews\n AllowOverride None\n Require all granted\n \n\n ## Logging\n ErrorLog \"/var/log/httpd/keystone_wsgi_main_error.log\"\n ServerSignature Off\n CustomLog \"/var/log/httpd/keystone_wsgi_main_access.log\" combined \n WSGIDaemonProcess keystone_main group=keystone processes=1 threads=1 user=keystone\n WSGIProcessGroup keystone_main\n WSGIScriptAlias / \"/var/www/cgi-bin/keystone/main\"\n\nEOF\n\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\nsed -i -e \"s/^Listen.*/Listen ${myip}:80/g\" /etc/httpd/conf/httpd.conf \necho \"Listen ${myip}:5000\" >> /etc/httpd/conf/httpd.conf\necho \"Listen ${myip}:35357\" >> /etc/httpd/conf/httpd.conf\n\nexport SERVICE_TOKEN=$(cat /root/keystone_service_token)\nopenstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token $SERVICE_TOKEN\nopenstack-config --set /etc/keystone/keystone.conf DEFAULT rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/keystone/keystone.conf DEFAULT rabbit_ha_queues true\nopenstack-config --set /etc/keystone/keystone.conf DEFAULT admin_endpoint \"http://${PHD_VAR_network_hosts_vip}:%(admin_port)s/\"\nopenstack-config --set /etc/keystone/keystone.conf DEFAULT public_endpoint \"http://${PHD_VAR_network_hosts_vip}:%(public_port)s/\"\nopenstack-config --set /etc/keystone/keystone.conf database connection mysql://keystone:keystonetest@${PHD_VAR_network_hosts_vip}/keystone\nopenstack-config --set /etc/keystone/keystone.conf database max_retries -1\nopenstack-config --set /etc/keystone/keystone.conf DEFAULT public_bind_host ${myip}\nopenstack-config --set /etc/keystone/keystone.conf DEFAULT admin_bind_host ${myip}\nopenstack-config --set /etc/keystone/keystone.conf token driver keystone.token.persistence.backends.sql.Token\n....\n\ntarget=$PHD_ENV_nodes1\n....\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\nkeystone-manage pki_setup --keystone-user keystone --keystone-group keystone\nchown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/\nsu keystone -s /bin/sh -c \"keystone-manage db_sync\"\ncd /etc/keystone/ssl\ntar cvfz /tmp/keystone_ssl.tgz *\nscp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /tmp/keystone_ssl.tgz ${controller_ips[1]}:/tmp\nscp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /tmp/keystone_ssl.tgz ${controller_ips[2]}:/tmp\n....\n\ntarget=$PHD_ENV_nodes2\n....\nmkdir -p /etc/keystone/ssl\ncd /etc/keystone/ssl\ntar xvfz /tmp/keystone_ssl.tgz \nchown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/\nrestorecon -Rv /etc/keystone/ssl\ntouch /var/log/keystone/keystone.log\nchown keystone:keystone /var/log/keystone/keystone.log \n....\n\ntarget=$PHD_ENV_nodes3\n....\nmkdir -p /etc/keystone/ssl\ncd /etc/keystone/ssl\ntar xvfz /tmp/keystone_ssl.tgz \nchown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/\nrestorecon -Rv /etc/keystone/ssl\ntouch /var/log/keystone/keystone.log\nchown keystone:keystone /var/log/keystone/keystone.log \n....\n\ntarget=all\n....\necho \"1 * * * * keystone keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1\" >> /etc/crontab\nfirewall-cmd --add-port=5000/tcp\nfirewall-cmd --add-port=5000/tcp --permanent\nfirewall-cmd --add-port=35357/tcp\nfirewall-cmd --add-port=35357/tcp --permanent\nsystemctl start httpd\nsystemctl enable httpd\n....\n\ntarget=$PHD_ENV_nodes1\n....\nexport OS_TOKEN=$(cat /root/keystone_service_token)\nexport OS_URL=http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nexport OS_REGION_NAME=regionOne\nopenstack service create --name=keystone --description=\"Keystone Identity Service\" identity\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:5000/v2.0\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:35357/v2.0\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:5000/v2.0\" --region regionOne keystone\nopenstack user create --password keystonetest admin\nopenstack role create admin\nopenstack project create admin\nopenstack role add --project admin --user admin admin\nopenstack user create --password redhat demo\nopenstack role create _member_\nopenstack project create demo\nopenstack role add --project demo --user demo _member_\nopenstack project create --description \"Services Tenant\" services\n# glance\nopenstack user create --password glancetest glance\nopenstack role add --project services --user glance admin\nopenstack service create --name=glance --description=\"Glance Image Service\" image\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:9292\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:9292\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:9292\" --region regionOne glance\n# cinder\nopenstack user create --password cindertest cinder\nopenstack role add --project services --user cinder admin\nopenstack service create --name=cinder --description=\"Cinder Volume Service\" volume\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:8776/v1/\\$(tenant_id)s\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:8776/v1/\\$(tenant_id)s\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:8776/v1/\\$(tenant_id)s\" --region regionOne cinder\nopenstack service create --name=cinderv2 --description=\"OpenStack Block Storage\" volumev2\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:8776/v2/\\$(tenant_id)s\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:8776/v2/\\$(tenant_id)s\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:8776/v2/\\$(tenant_id)s\" --region regionOne cinderv2\n# swift\nopenstack user create --password swifttest swift\nopenstack role add --project services --user swift admin\nopenstack service create --name=swift --description=\"Swift Storage Service\" object-store\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:8080/v1/AUTH_\\$(tenant_id)s\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:8080/v1\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:8080/v1/AUTH_\\$(tenant_id)s\" --region regionOne swift\n# neutron\nopenstack user create --password neutrontest neutron\nopenstack role add --project services --user neutron admin\nopenstack service create --name=neutron --description=\"OpenStack Networking Service\" network\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:9696\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:9696\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:9696\" --region regionOne neutron\n# nova\nopenstack user create --password novatest compute\nopenstack role add --project services --user compute admin\nopenstack service create --name=compute --description=\"OpenStack Compute Service\" compute\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:8774/v2/\\$(tenant_id)s\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:8774/v2/\\$(tenant_id)s\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:8774/v2/\\$(tenant_id)s\" --region regionOne compute\n# heat\nopenstack user create --password heattest heat\nopenstack role add --project services --user heat admin\nopenstack service create --name=heat --description=\"Heat Orchestration Service\" orchestration\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:8004/v1/%(tenant_id)s\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:8004/v1/%(tenant_id)s\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:8004/v1/%(tenant_id)s\" --region regionOne heat\nopenstack service create --name=heat-cfn --description=\"Heat CloudFormation Service\" cloudformation\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:8000/v1\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:8000/v1\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:8000/v1\" --region regionOne heat-cfn\n# ceilometer\nopenstack user create --password ceilometertest ceilometer\nopenstack role add --project services --user ceilometer admin\nopenstack role create ResellerAdmin\nopenstack role add --project services --user ceilometer ResellerAdmin\nopenstack service create --name=ceilometer --description=\"OpenStack Telemetry Service\" metering\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:8777\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:8777\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:8777\" --region regionOne ceilometer\n# sahara\nopenstack user create --password saharatest sahara\nopenstack role add --project services --user sahara admin\nopenstack service create --name=sahara --description=\"Sahara Data Processing\" data-processing\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:8386/v1.1/%(tenant_id)s\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:8386/v1.1/%(tenant_id)s\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:8386/v1.1/%(tenant_id)s\" --region regionOne sahara\n# trove\nopenstack user create --password trovetest trove\nopenstack role add --project services --user trove admin\nopenstack service create --name=trove --description=\"OpenStack Database Service\" database\nopenstack endpoint create --publicurl \"http://${PHD_VAR_network_hosts_vip}:8779/v1.0/%(tenant_id)s\" --adminurl \"http://${PHD_VAR_network_hosts_vip}:8779/v1.0/%(tenant_id)s\" --internalurl \"http://${PHD_VAR_network_hosts_vip}:8779/v1.0/%(tenant_id)s\" --region regionOne trove\n....\n\ntarget=all\n....\n\ncat > /root/keystonerc_admin << EOF\nexport OS_USERNAME=admin \nexport OS_TENANT_NAME=admin\nexport OS_PROJECT_NAME=admin\nexport OS_REGION_NAME=regionOne\nexport OS_PASSWORD=keystonetest\nexport OS_AUTH_URL=http://${PHD_VAR_network_hosts_vip}:35357/v2.0/\nexport PS1='[\\u@\\h \\W(keystone_admin)]\\$ '\nEOF\n\ncat > /root/keystonerc_demo << EOF\nexport OS_USERNAME=demo\nexport OS_TENANT_NAME=demo\nexport OS_PROJECT_NAME=demo\nexport OS_REGION_NAME=regionOne\nexport OS_PASSWORD=redhat\nexport OS_AUTH_URL=http://${PHD_VAR_network_hosts_vip}:5000/v2.0/\nexport PS1='[\\u@\\h \\W(keystone_user)]\\$ '\nEOF\n....\n" }, { "path": "keepalived/phd-setup/lb.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Tweaking the IP stack to allow nonlocal binding\n# - Configuring haproxy\n\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_ips_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_deployment\nPHD_VAR_components\nPHD_VAR_osp_major\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y haproxy openstack-selinux\necho net.ipv4.ip_nonlocal_bind=1 >> /etc/sysctl.d/haproxy.conf\necho 1 > /proc/sys/net/ipv4/ip_nonlocal_bind\n\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\n# HA Proxy defaults\n\ncat > /etc/haproxy/haproxy.cfg << EOF\nglobal\n daemon\n stats socket /var/lib/haproxy/stats\ndefaults\n mode tcp\n maxconn 10000\n timeout connect 5s\n timeout client 30s\n timeout server 30s\n\nlisten monitor\n bind ${PHD_VAR_network_ips_vip}:9300 \n mode http\n monitor-uri /status\n stats enable\n stats uri /admin\n stats realm Haproxy\\ Statistics\n stats auth root:redhat\n stats refresh 5s\n\nfrontend vip-db\n bind ${PHD_VAR_network_ips_vip}:3306\n timeout client 90m\n default_backend db-vms-galera\nbackend db-vms-galera\n option httpchk\n stick-table type ip size 1000\n stick on dst\n timeout server 90m\n server rhos7-node1 ${controller_ips[0]}:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions\n server rhos7-node2 ${controller_ips[1]}:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions\n server rhos7-node3 ${controller_ips[2]}:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions\n\n# Note the RabbitMQ entry is only needed for CloudForms compatibility\n# and should be removed in the future\nfrontend vip-rabbitmq\n option clitcpka\n bind ${PHD_VAR_network_ips_vip}:5672\n timeout client 900m\n default_backend rabbitmq-vms\nbackend rabbitmq-vms\n option srvtcpka\n balance roundrobin\n timeout server 900m\n server rhos7-node1 ${controller_ips[0]}:5672 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:5672 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:5672 check inter 1s\n\nfrontend vip-keystone-admin\n bind ${PHD_VAR_network_ips_vip}:35357\n default_backend keystone-admin-vms\n timeout client 600s\nbackend keystone-admin-vms\n balance roundrobin\n timeout server 600s\n server rhos7-node1 ${controller_ips[0]}:35357 check inter 1s on-marked-down shutdown-sessions\n server rhos7-node2 ${controller_ips[1]}:35357 check inter 1s on-marked-down shutdown-sessions\n server rhos7-node3 ${controller_ips[2]}:35357 check inter 1s on-marked-down shutdown-sessions\n\nfrontend vip-keystone-public\n bind ${PHD_VAR_network_ips_vip}:5000\n default_backend keystone-public-vms\n timeout client 600s\nbackend keystone-public-vms\n balance roundrobin\n timeout server 600s\n server rhos7-node1 ${controller_ips[0]}:5000 check inter 1s on-marked-down shutdown-sessions\n server rhos7-node2 ${controller_ips[1]}:5000 check inter 1s on-marked-down shutdown-sessions\n server rhos7-node3 ${controller_ips[2]}:5000 check inter 1s on-marked-down shutdown-sessions\n\nfrontend vip-glance-api\n bind ${PHD_VAR_network_ips_vip}:9191\n default_backend glance-api-vms\nbackend glance-api-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:9191 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:9191 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:9191 check inter 1s\n\nfrontend vip-glance-registry\n bind ${PHD_VAR_network_ips_vip}:9292\n default_backend glance-registry-vms\nbackend glance-registry-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:9292 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:9292 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:9292 check inter 1s\n\nfrontend vip-cinder\n bind ${PHD_VAR_network_ips_vip}:8776\n default_backend cinder-vms\nbackend cinder-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:8776 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:8776 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:8776 check inter 1s\n\nfrontend vip-swift\n bind ${PHD_VAR_network_ips_vip}:8080\n default_backend swift-vms\nbackend swift-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:8080 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:8080 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:8080 check inter 1s\n\nfrontend vip-neutron\n bind ${PHD_VAR_network_ips_vip}:9696\n default_backend neutron-vms\nbackend neutron-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:9696 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:9696 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:9696 check inter 1s\n\nfrontend vip-nova-vnc-novncproxy\n bind ${PHD_VAR_network_ips_vip}:6080\n default_backend nova-vnc-novncproxy-vms\nbackend nova-vnc-novncproxy-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:6080 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:6080 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:6080 check inter 1s\n\nfrontend nova-metadata-vms\n bind ${PHD_VAR_network_ips_vip}:8775\n default_backend nova-metadata-vms\nbackend nova-metadata-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:8775 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:8775 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:8775 check inter 1s\n\nfrontend vip-nova-api\n bind ${PHD_VAR_network_ips_vip}:8774\n default_backend nova-api-vms\nbackend nova-api-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:8774 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:8774 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:8774 check inter 1s\n\nfrontend vip-horizon\n bind ${PHD_VAR_network_ips_vip}:80\n timeout client 180s\n default_backend horizon-vms\nbackend horizon-vms\n balance roundrobin\n timeout server 180s\n mode http\n cookie SERVERID insert indirect nocache\n server rhos7-node1 ${controller_ips[0]}:80 check inter 1s cookie rhos7-horizon1 on-marked-down shutdown-sessions\n server rhos7-node2 ${controller_ips[1]}:80 check inter 1s cookie rhos7-horizon2 on-marked-down shutdown-sessions\n server rhos7-node3 ${controller_ips[2]}:80 check inter 1s cookie rhos7-horizon3 on-marked-down shutdown-sessions\n\nfrontend vip-heat-cfn\n bind ${PHD_VAR_network_ips_vip}:8000\n default_backend heat-cfn-vms\nbackend heat-cfn-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:8000 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:8000 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:8000 check inter 1s\n\nfrontend vip-heat-cloudw\n bind ${PHD_VAR_network_ips_vip}:8003\n default_backend heat-cloudw-vms\nbackend heat-cloudw-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:8003 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:8003 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:8003 check inter 1s\n\nfrontend vip-heat-srv\n bind ${PHD_VAR_network_ips_vip}:8004\n default_backend heat-srv-vms\nbackend heat-srv-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:8004 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:8004 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:8004 check inter 1s\n\nfrontend vip-ceilometer\n bind ${PHD_VAR_network_ips_vip}:8777\n timeout client 90s\n default_backend ceilometer-vms\nbackend ceilometer-vms\n balance roundrobin\n timeout server 90s\n server rhos7-node1 ${controller_ips[0]}:8777 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:8777 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:8777 check inter 1s\n\nfrontend vip-sahara\n bind ${PHD_VAR_network_ips_vip}:8386\n default_backend sahara-vms\nbackend sahara-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:8386 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:8386 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:8386 check inter 1s\n\nfrontend vip-trove\n bind ${PHD_VAR_network_ips_vip}:8779\n default_backend trove-vms\nbackend trove-vms\n balance roundrobin\n server rhos7-node1 ${controller_ips[0]}:8779 check inter 1s\n server rhos7-node2 ${controller_ips[1]}:8779 check inter 1s\n server rhos7-node3 ${controller_ips[2]}:8779 check inter 1s\nEOF\n....\n" }, { "path": "keepalived/phd-setup/memcached.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing memcached\n# - Starting memcached, opening firewall rules\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_ips_vip\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\n\nyum install -y memcached\nsystemctl start memcached\nsystemctl enable memcached\nfirewall-cmd --add-port=11211/tcp\nfirewall-cmd --add-port=11211/tcp --permanent\n....\n" }, { "path": "keepalived/phd-setup/mongodb.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing MongoDB\n# - Setting up MongoDB, including a 3-way replica set\n# - Starting MongoDB, opening firewall ports\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_hosts_mongodb\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\n\nyum install -y mongodb mongodb-server\nsed -i -e 's#bind_ip.*#bind_ip = 0.0.0.0#g' /etc/mongod.conf\necho \"replSet = ceilometer\" >> /etc/mongod.conf \nsystemctl start mongod\nsystemctl enable mongod\nfirewall-cmd --add-port=27017/tcp\nfirewall-cmd --add-port=27017/tcp --permanent\n....\n\ntarget=$PHD_ENV_nodes1\n....\nIFS=', ' read -a controller_names <<< \"${PHD_VAR_network_hosts_mongodb}\"\n\ncat > /tmp/mongoinit.js << EOF\nrs.initiate()\nsleep(10000)\nrs.add(\"${controller_names[1]}\");\nrs.add(\"${controller_names[2]}\");\nEOF\n\nmongo /tmp/mongoinit.js\n....\n\n" }, { "path": "keepalived/phd-setup/neutron.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing Neutron\n# - Configuring Neutron server, FWaaS, LBaaS and Neutron agents\n# - Starting Neutron services, opening firewall rules\n# - Creating an external network using a provider network\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_nic_external\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_neutron_externalgateway\nPHD_VAR_network_neutron_externalnetwork\nPHD_VAR_network_neutron_allocpoolstart\nPHD_VAR_network_neutron_allocpoolend\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\nyum install -y openstack-neutron openstack-neutron-openvswitch openstack-neutron-ml2 openstack-utils openstack-selinux\n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT bind_host ${myip}\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password neutrontest\nopenstack-config --set /etc/neutron/neutron.conf database connection mysql://neutron:neutrontest@${PHD_VAR_network_hosts_vip}:3306/neutron\nopenstack-config --set /etc/neutron/neutron.conf database max_retries -1\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT nova_url http://${PHD_VAR_network_hosts_vip}:8774/v2\nopenstack-config --set /etc/neutron/neutron.conf nova nova_region_name regionOne\nopenstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true\n\n. /root/keystonerc_admin\nservice_tenant_id=$(openstack project show services -f value -c id | head -n 1)\n\n# The tenant_id below is the services tenant ID\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_tenant_id ${service_tenant_id}\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_username compute\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_password novatest\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes True\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_data_changes True\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin neutron.plugins.ml2.plugin.Ml2Plugin\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT router_scheduler_driver neutron.scheduler.l3_agent_scheduler.ChanceScheduler\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 2\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT api_workers 2\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_workers 2\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT l3_ha True\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT min_l3_agents_per_router 2\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT max_l3_agents_per_router 2\n\n\nln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 type_drivers local,gre,flat,vxlan,vlan\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers openvswitch\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks \\*\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_gre tunnel_id_ranges 10:10000\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vni_ranges 10:10000\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vxlan_group 224.0.0.1\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_security_group True\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver \n\nyum -y install openstack-neutron-lbaas\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router,lbaas\nopenstack-config --set /etc/neutron/lbaas_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver\nopenstack-config --set /etc/neutron/lbaas_agent.ini DEFAULT device_driver neutron_lbaas.services.loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver\nopenstack-config --set /etc/neutron/lbaas_agent.ini haproxy user_group haproxy \n\nyum -y install openstack-neutron-fwaas\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router,firewall,lbaas\nopenstack-config --set /etc/neutron/fwaas_driver.ini fwaas enabled True\nopenstack-config --set /etc/neutron/fwaas_driver.ini fwaas driver neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver\n\n....\n\ntarget=$PHD_ENV_nodes1\n....\nneutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head\n....\n\ntarget=all\n....\nsystemctl start neutron-server\nsystemctl enable neutron-server\nfirewall-cmd --add-port=9696/tcp\nfirewall-cmd --add-port=9696/tcp --permanent\n\nsystemctl enable openvswitch\nsystemctl start openvswitch\novs-vsctl add-br br-int\novs-vsctl add-br br-${PHD_VAR_network_nic_external}\novs-vsctl add-port br-${PHD_VAR_network_nic_external} ${PHD_VAR_network_nic_external}\n\n# OpenvSwitch agent\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent tunnel_types vxlan\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent vxlan_udp_port 4789\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs local_ip ${myip}\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs enable_tunneling True\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs integration_bridge br-int\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_bridge br-tun\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs bridge_mappings physnet1:br-${PHD_VAR_network_nic_external}\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs network_vlan_ranges physnet1\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver \nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent l2_population False\n\n# Metadata agent\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_strategy keystone\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_region regionOne\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_tenant_name services\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_user neutron\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_password neutrontest\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_port 8775\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret metatest\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_workers 4\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_backlog 2048\n\n# DHCP agent\nopenstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver\nopenstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dhcp_delete_namespaces False\nopenstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_config_file /etc/neutron/dnsmasq-neutron.conf\necho \"dhcp-option-force=26,1400\" > /etc/neutron/dnsmasq-neutron.conf\nchown root:neutron /etc/neutron/dnsmasq-neutron.conf\nchmod 644 /etc/neutron/dnsmasq-neutron.conf\n\n# L3 agent\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT handle_internal_only_routers True\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT send_arp_for_ha 3\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT router_delete_namespaces False\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT metadata_ip ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge\n\n# Start services\n# Workaround for bz#1188198\nmkdir -p /etc/systemd/system/neutron-server.service.d\ncat > /etc/systemd/system/neutron-server.service.d/restart.conf << EOF\n[Service]\nRestart=on-failure\nEOF\nsystemctl daemon-reload\n\nsystemctl start neutron-openvswitch-agent\nsystemctl start neutron-dhcp-agent\nsystemctl start neutron-l3-agent\nsystemctl start neutron-metadata-agent\nsystemctl start neutron-lbaas-agent\nsystemctl enable neutron-openvswitch-agent\nsystemctl enable neutron-dhcp-agent\nsystemctl enable neutron-l3-agent\nsystemctl enable neutron-metadata-agent\nsystemctl enable neutron-ovs-cleanup\nsystemctl enable neutron-lbaas-agent\nfirewall-cmd --add-port=4789/udp\nfirewall-cmd --add-port=4789/udp --permanent\n....\n\ntarget=$PHD_ENV_nodes1\n....\n. /root/keystonerc_admin\nneutron net-create public --provider:network_type flat --provider:physical_network physnet1 --router:external\nneutron subnet-create --gateway ${PHD_VAR_network_neutron_externalgateway} --allocation-pool start=${PHD_VAR_network_neutron_allocpoolstart},end=${PHD_VAR_network_neutron_allocpoolend} --disable-dhcp --name public_subnet public ${PHD_VAR_network_neutron_externalnetwork}\n....\n\n" }, { "path": "keepalived/phd-setup/nova.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing Nova\n# - Configuring Nova\n# - Starting service, opening firewall ports\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_nic_external\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_neutron_externalgateway\nPHD_VAR_network_neutron_externalnetwork\nPHD_VAR_network_neutron_allocpoolstart\nPHD_VAR_network_neutron_allocpoolend\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\nyum install -y openstack-nova-console openstack-nova-novncproxy openstack-utils openstack-nova-api openstack-nova-conductor openstack-nova-scheduler python-cinderclient python-memcached\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address ${myip}\nopenstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen ${myip}\nopenstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_host ${myip}\nopenstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://${PHD_VAR_network_hosts_vip}:6080/vnc_auto.html\nopenstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@${PHD_VAR_network_hosts_vip}/nova\nopenstack-config --set /etc/nova/nova.conf database max_retries -1\nopenstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone\nopenstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen ${myip}\nopenstack-config --set /etc/nova/nova.conf DEFAULT metadata_host ${myip}\nopenstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen ${myip}\nopenstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen_port 8775\nopenstack-config --set /etc/nova/nova.conf DEFAULT glance_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API\nopenstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver\nopenstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver\nopenstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron\nopenstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL\nopenstack-config --set /etc/nova/nova.conf conductor use_local false\nopenstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True\nopenstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True\nopenstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest\nopenstack-config --set /etc/nova/nova.conf neutron url http://${PHD_VAR_network_hosts_vip}:9696/\nopenstack-config --set /etc/nova/nova.conf neutron admin_tenant_name services\nopenstack-config --set /etc/nova/nova.conf neutron admin_username neutron\nopenstack-config --set /etc/nova/nova.conf neutron admin_password neutrontest\nopenstack-config --set /etc/nova/nova.conf neutron admin_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/nova/nova.conf neutron region_name regionOne\n\n# REQUIRED FOR A/A scheduler\nopenstack-config --set /etc/nova/nova.conf DEFAULT scheduler_host_subset_size 30\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_tenant_name services\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_user compute\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_password novatest\n\n# WARNING: Only run the following if your hypervisors are VMs\nopenstack-config --set /etc/nova/nova.conf libvirt virt_type qemu\n....\n\ntarget=$PHD_ENV_nodes1\n....\nsu nova -s /bin/sh -c \"nova-manage db sync\"\n....\n\ntarget=all\n....\nsystemctl start openstack-nova-consoleauth\nsystemctl start openstack-nova-novncproxy \nsystemctl start openstack-nova-api\nsystemctl start openstack-nova-scheduler\nsystemctl start openstack-nova-conductor\nsystemctl enable openstack-nova-consoleauth\nsystemctl enable openstack-nova-novncproxy \nsystemctl enable openstack-nova-api\nsystemctl enable openstack-nova-scheduler\nsystemctl enable openstack-nova-conductor\n\nfirewall-cmd --add-port=8773-8775/tcp\nfirewall-cmd --add-port=8773-8775/tcp --permanent\nfirewall-cmd --add-port=6080/tcp\nfirewall-cmd --add-port=6080/tcp --permanent\n....\n" }, { "path": "keepalived/phd-setup/rabbitmq.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing RabbitMQ\n# - Configuring RabbitMQ, with a 3-node cluster\n# - Setting TCP keepalived kernel parameters\n# - Starting the RabbitMQ cluster, opening firewall ports\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum -y install rabbitmq-server\n....\n\ntarget=$PHD_ENV_nodes1\n....\n\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\ncat > /etc/rabbitmq/rabbitmq-env.conf << EOF\nNODE_IP_ADDRESS=${controller_ips[0]}\nEOF\n\nsystemctl start rabbitmq-server\nsystemctl stop rabbitmq-server\nscp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p /var/lib/rabbitmq/.erlang.cookie ${controller_ips[1]}:/tmp\nscp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p /var/lib/rabbitmq/.erlang.cookie ${controller_ips[2]}:/tmp\n....\n\ntarget=$PHD_ENV_nodes2\n....\n\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\ncat > /etc/rabbitmq/rabbitmq-env.conf << EOF\nNODE_IP_ADDRESS=${controller_ips[1]}\nEOF\n\ncp /tmp/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie\nchown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie\n....\n\ntarget=$PHD_ENV_nodes3\n....\n\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\ncat > /etc/rabbitmq/rabbitmq-env.conf << EOF\nNODE_IP_ADDRESS=${controller_ips[2]}\nEOF\n\ncp /tmp/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie\nchown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie\n....\n\ntarget=all\n....\nIFS=', ' read -a nodes <<< \"${PHD_VAR_network_hosts_rabbitmq}\"\n\ncat > /etc/rabbitmq/rabbitmq.config << EOF\n\n[\n {rabbit, [\n {cluster_nodes, {['rabbit@${nodes[0]}', 'rabbit@${nodes[1]}', 'rabbit@${nodes[2]}'], disc}},\n {cluster_partition_handling, ignore},\n {default_user, <<\"guest\">>},\n {default_pass, <<\"guest\">>},\n {tcp_listen_options, [binary,\n {packet, raw},\n {reuseaddr, true},\n {backlog, 128},\n {nodelay, true},\n {exit_on_close, false},\n {keepalive, true}]}\n ]},\n {kernel, [\n {inet_dist_listen_max, 44001},\n {inet_dist_listen_min, 44001}\n ]}\n].\n\nEOF\n\ncat > /etc/sysctl.d/tcpka.conf << EOF\nnet.ipv4.tcp_keepalive_intvl = 1\nnet.ipv4.tcp_keepalive_probes = 5\nnet.ipv4.tcp_keepalive_time = 5\nEOF\n\nsysctl -p /etc/sysctl.d/tcpka.conf\n\nfirewall-cmd --add-port=5672/tcp \nfirewall-cmd --add-port=4369/tcp\nfirewall-cmd --add-port=5672/tcp --permanent\nfirewall-cmd --add-port=4369/tcp --permanent\nfirewall-cmd --add-port=44001/tcp\nfirewall-cmd --add-port=44001/tcp --permanent\nsystemctl enable rabbitmq-server\nsystemctl start rabbitmq-server\n....\n\ntarget=$PHD_ENV_nodes1\n....\nrabbitmqctl set_policy HA '^(?!amq\\.).*' '{\"ha-mode\": \"all\"}'\n....\n" }, { "path": "keepalived/phd-setup/readme.txt", "content": "Assumptions:\n\n1. We have a basic rhel7-base image. It is just a cloud image where we have manually injected a root ssh key, and allow direct root login (so the account is not locked). About the rest of the parameters:\n\n - NIC: default NAT\n - Disk: 60 GB, qcow2, virtio\n - CPU: 1\n - RAM: 4 GB\n\n2. About SSH keys, it is important that the base image includes SSH keys in /root/.ssh/authorized_keys for:\n\n - The system running phd\n - The hypervisor that will run the VM\n\n" }, { "path": "keepalived/phd-setup/redis.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing Redis\n# - Setting up Redis, including a master and two slaves\n# - Setting up Sentinel for HA\n# - Starting services, opening firewall ports\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y redis\n....\n\ntarget=$PHD_ENV_nodes1\n....\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\nsed --in-place \"s/bind 127.0.0.1/bind 127.0.0.1 ${controller_ips[0]}/\" /etc/redis.conf\n....\n\ntarget=$PHD_ENV_nodes2\n....\n\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\nsed --in-place \"s/bind 127.0.0.1/bind 127.0.0.1 ${controller_ips[1]}/\" /etc/redis.conf\necho slaveof ''${controller_ips[0]}'' 6379 >> /etc/redis.conf \n....\n\ntarget=$PHD_ENV_nodes3\n....\n\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\nsed --in-place \"s/bind 127.0.0.1/bind 127.0.0.1 ${controller_ips[2]}/\" /etc/redis.conf\necho slaveof ''${controller_ips[0]}'' 6379 >> /etc/redis.conf \n....\n\ntarget=all\n....\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\ncat > /etc/redis-sentinel.conf << EOF\n\nsentinel monitor mymaster ${controller_ips[0]} 6379 2\nsentinel down-after-milliseconds mymaster 30000\nsentinel failover-timeout mymaster 180000\nsentinel parallel-syncs mymaster 1\nmin-slaves-to-write 1\nmin-slaves-max-lag 10\nlogfile /var/log/redis/sentinel.log\nEOF\n\nfirewall-cmd --add-port=6379/tcp\nfirewall-cmd --add-port=6379/tcp --permanent\nfirewall-cmd --add-port=26379/tcp\nfirewall-cmd --add-port=26379/tcp --permanent\nsystemctl enable redis\nsystemctl start redis\nsystemctl enable redis-sentinel\nsystemctl start redis-sentinel\n\n....\n\n" }, { "path": "keepalived/phd-setup/sahara.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing Sahara\n# - Configuring Sahara\n# - Starting services, opening firewall ports\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_nic_external\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_neutron_externalgateway\nPHD_VAR_network_neutron_externalnetwork\nPHD_VAR_network_neutron_allocpoolstart\nPHD_VAR_network_neutron_allocpoolend\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\nyum install -y openstack-sahara-api openstack-sahara-engine openstack-sahara-common openstack-sahara python-saharaclient\n\nopenstack-config --set /etc/sahara/sahara.conf DEFAULT host ${myip}\nopenstack-config --set /etc/sahara/sahara.conf DEFAULT use_floating_ips True\nopenstack-config --set /etc/sahara/sahara.conf DEFAULT use_neutron True\nopenstack-config --set /etc/sahara/sahara.conf DEFAULT rpc_backend rabbit\nopenstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_port 5672\nopenstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_use_ssl False\nopenstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_userid guest\nopenstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_password guest\nopenstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_login_method AMQPLAIN\nopenstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/sahara/sahara.conf DEFAULT notification_topics notifications\nopenstack-config --set /etc/sahara/sahara.conf database connection mysql://sahara:saharatest@${PHD_VAR_network_hosts_vip}/sahara\nopenstack-config --set /etc/sahara/sahara.conf keystone_authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:5000/v2.0\nopenstack-config --set /etc/sahara/sahara.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_user sahara\nopenstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_password saharatest\nopenstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/sahara/sahara.conf DEFAULT log_file /var/log/sahara/sahara.log\n....\n\ntarget=$PHD_ENV_nodes1\n....\nsahara-db-manage --config-file /etc/sahara/sahara.conf upgrade head\n....\n\ntarget=all\n....\nfirewall-cmd --add-port=8386/tcp\nfirewall-cmd --add-port=8386/tcp --permanent\nsystemctl enable openstack-sahara-api\nsystemctl enable openstack-sahara-engine\nsystemctl start openstack-sahara-api\nsystemctl start openstack-sahara-engine\n....\n\n" }, { "path": "keepalived/phd-setup/serverprep.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Tweaking the IP stack to allow nonlocal binding and adjusting keepalive timings\n# - Configuring haproxy\n# - Adding the virtual IPs to the cluster\n# - Putting haproxy under the cluster's control\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_ips_computeinternal\nPHD_VAR_network_hosts_controllers\nPHD_VAR_network_hosts_compute\nPHD_VAR_network_ips_vip\nPHD_VAR_network_hosts_vip\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\n\nIFS=', ' read -a controller_names <<< \"${PHD_VAR_network_hosts_controllers}\"\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\nIFS=', ' read -a compute_names <<< \"${PHD_VAR_network_hosts_compute}\"\nIFS=', ' read -a compute_ips <<< \"${PHD_VAR_network_ips_computeinternal}\"\n\naddhosts=\"\"\n\nfor item in \"${controller_names[@]}\"\ndo\n shortname=$(echo ${controller_names[item]} | awk -F. '{print $1}')\n\n addhosts=$(printf \"${addhosts}\\n${controller_ips[item]} ${controller_names[item]} ${shortname}\")\ndone\n\nfor item in \"${controller_names[@]}\"\ndo\n shortname=$(echo ${compute_names[item]} | awk -F. '{print $1}')\n\n addhosts=$(printf \"${addhosts}\\n${compute_ips[item]} ${compute_names[item]} ${shortname}\")\ndone\n\nshortname=$(echo ${PHD_VAR_network_hosts_vip} | awk -F. '{print $1}')\naddhosts=$(printf \"${addhosts}\\n${PHD_VAR_network_ips_vip} ${PHD_VAR_network_hosts_vip} ${shortname}\")\n\necho \"$addhosts\" >> /etc/hosts\n....\n" }, { "path": "keepalived/phd-setup/swift.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Tweaking the IP stack to allow nonlocal binding and adjusting keepalive timings\n# - Configuring haproxy\n# - Adding the virtual IPs to the cluster\n# - Putting haproxy under the cluster's control\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_hosts_memcache\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\nyum install -y openstack-swift-object openstack-swift-container openstack-swift-account openstack-swift-proxy openstack-utils rsync xfsprogs\n\nmkfs.xfs /dev/vdb\nmkdir -p /srv/node/vdb\necho \"/dev/vdb /srv/node/vdb xfs defaults 1 2\" >> /etc/fstab\nmount -a \nchown -R swift:swift /srv/node\nrestorecon -R /srv/node\n\nopenstack-config --set /etc/swift/object-server.conf DEFAULT bind_ip ${myip}\nopenstack-config --set /etc/swift/object-server.conf DEFAULT devices /srv/node\nopenstack-config --set /etc/swift/account-server.conf DEFAULT bind_ip ${myip}\nopenstack-config --set /etc/swift/account-server.conf DEFAULT devices /srv/node\nopenstack-config --set /etc/swift/container-server.conf DEFAULT bind_ip ${myip}\nopenstack-config --set /etc/swift/container-server.conf DEFAULT devices /srv/node\nchown -R root:swift /etc/swift\n\nsystemctl start openstack-swift-account\nsystemctl start openstack-swift-container\nsystemctl start openstack-swift-object\nsystemctl enable openstack-swift-account\nsystemctl enable openstack-swift-container\nsystemctl enable openstack-swift-object\n\nfirewall-cmd --add-port=6200/tcp\nfirewall-cmd --add-port=6200/tcp --permanent\nfirewall-cmd --add-port=6201/tcp\nfirewall-cmd --add-port=6201/tcp --permanent\nfirewall-cmd --add-port=6202/tcp\nfirewall-cmd --add-port=6202/tcp --permanent\n\nopenstack-config --set /etc/swift/proxy-server.conf filter:authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_tenant_name services\nopenstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_user swift\nopenstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_password swifttest\nopenstack-config --set /etc/swift/proxy-server.conf filter:cache memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/swift/proxy-server.conf DEFAULT bind_ip ${myip}\nopenstack-config --set /etc/swift/object-expirer.conf filter:cache memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/swift/object-expirer.conf object-expirer concurrency 100\n....\n\ntarget=$PHD_ENV_nodes1\n....\nIFS=', ' read -a controller_ips <<< \"${PHD_VAR_network_ips_controllers}\"\n\nopenstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_suffix $(openssl rand -hex 10)\n\ncat >> /etc/swift/swift.conf << EOF\n[filter:ceilometer]\nuse = egg:ceilometer#swift\n[pipeline:main]\npipeline = healthcheck cache authtoken keystoneauth proxy-server ceilometer\nEOF\n\nswift-ring-builder /etc/swift/object.builder create 16 3 24\nswift-ring-builder /etc/swift/container.builder create 16 3 24\nswift-ring-builder /etc/swift/account.builder create 16 3 24\nswift-ring-builder /etc/swift/account.builder add z1-${controller_ips[0]}:6202/vdb 10\nswift-ring-builder /etc/swift/container.builder add z1-${controller_ips[0]}:6201/vdb 10\nswift-ring-builder /etc/swift/object.builder add z1-${controller_ips[0]}:6200/vdb 10\nswift-ring-builder /etc/swift/account.builder add z2-${controller_ips[1]}:6202/vdb 10\nswift-ring-builder /etc/swift/container.builder add z2-${controller_ips[1]}:6201/vdb 10\nswift-ring-builder /etc/swift/object.builder add z2-${controller_ips[1]}:6200/vdb 10\nswift-ring-builder /etc/swift/account.builder add z3-${controller_ips[2]}:6202/vdb 10\nswift-ring-builder /etc/swift/container.builder add z3-${controller_ips[2]}:6201/vdb 10\nswift-ring-builder /etc/swift/object.builder add z3-${controller_ips[2]}:6200/vdb 10\nswift-ring-builder /etc/swift/account.builder rebalance\nswift-ring-builder /etc/swift/container.builder rebalance\nswift-ring-builder /etc/swift/object.builder rebalance\n\ncd /etc/swift\ntar cvfz /tmp/swift_configs.tgz swift.conf *.builder *.gz\nscp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /tmp/swift_configs.tgz ${controller_ips[1]}:/tmp\nscp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /tmp/swift_configs.tgz ${controller_ips[2]}:/tmp\nchown -R root:swift /etc/swift\n....\n\ntarget=$PHD_ENV_nodes2\n....\ncd /etc/swift\ntar xvfz /tmp/swift_configs.tgz\nchown -R root:swift /etc/swift\nrestorecon -R /etc/swift\n....\n\ntarget=$PHD_ENV_nodes3\n....\ncd /etc/swift\ntar xvfz /tmp/swift_configs.tgz\nchown -R root:swift /etc/swift\nrestorecon -R /etc/swift\n....\n\ntarget=all\n....\nsystemctl start openstack-swift-proxy\nsystemctl enable openstack-swift-proxy\nsystemctl start openstack-swift-object-expirer\nsystemctl enable openstack-swift-object-expirer\nfirewall-cmd --add-port=8080/tcp\nfirewall-cmd --add-port=8080/tcp --permanent\n....\n" }, { "path": "keepalived/phd-setup/test.sh", "content": "#!/bin/bash\n\nset -e\n\ndeclare -A nodeMap\ndeclare -A variables\ndeclare -A cluster\n\nnodeMap[\"hypervisors\"]=\"oslab1 oslab2 oslab3\"\nnodeMap[\"controllers\"]=\"controller1 controller2 controller3\"\nnodeMap[\"compute\"]=\"compute1 compute2\"\nnodeMap[\"serverprep\"]=\"controller1 controller2 controller3 compute1 compute2\"\n\nvariables[\"nodes\"]=\"\"\nvariables[\"components\"]=\"hypervisors serverprep lb galera rabbitmq memcached redis mongodb keepalived keystone glance cinder swift neutron nova ceilometer heat horizon sahara trove compute\"\n#variables[\"components\"]=\"compute\"\nvariables[\"network_domain\"]=\"example.com\"\nvariables[\"config\"]=\"ha-collapsed\"\n\n\nfunction create_phd_definition() {\n scenario=$1\n definition=$2\n rm -f ${definition}\n\n nodes=${variables[\"nodes\"]}\n\n if [ \"x$nodes\" = x ]; then\n nodes=${nodeMap[$scenario]}\n fi\n\n if [ \"x$nodes\" = \"x\" ]; then\n for n in `seq 1 3`; do\n nodes=\"$nodes controller${n}\"\n done\n fi\n\n nodelist=\"nodes=\"\n for node in $nodes; do\n nodelist=\"${nodelist}${node}.${variables[\"network_domain\"]} \"\n done\n\n echo \"$nodelist\" >> ${definition}\n cat ${definition}\n}\n\n\nfunction run_phd() {\n phd_exec -s ./${1}.scenario -d ./phd.${1}.conf -V ./${variables[\"config\"]}.variables\n}\n\nscenarios=${variables[\"components\"]}\n\n\nfor scenario in $scenarios; do\n create_phd_definition ${scenario} ./phd.${scenario}.conf\n echo \"$(date) :: Beginning scenario $scenario\"\n run_phd ${scenario}\ndone\n" }, { "path": "keepalived/phd-setup/trove.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Installing Trove\n# - Configuring Trove\n# - Creating a VM image for Trove usage\n# - Starting services, opening firewall ports\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_nic_internal\nPHD_VAR_network_nic_external\nPHD_VAR_network_hosts_vip\nPHD_VAR_network_ips_controllers\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_neutron_externalgateway\nPHD_VAR_network_neutron_externalnetwork\nPHD_VAR_network_neutron_allocpoolstart\nPHD_VAR_network_neutron_allocpoolend\nPHD_VAR_network_ssh_pubkey\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nmyip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)\n\nyum install -y openstack-trove python-troveclient\n\nopenstack-config --set /etc/trove/trove.conf DEFAULT bind_host ${myip}\nopenstack-config --set /etc/trove/trove.conf DEFAULT log_dir /var/log/trove\nopenstack-config --set /etc/trove/trove.conf DEFAULT trove_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/trove/trove.conf DEFAULT os_region_name regionOne\nopenstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_password guest\nopenstack-config --set /etc/trove/trove.conf DEFAULT rpc_backend rabbit\nopenstack-config --set /etc/trove/trove.conf database connection mysql://trove:trovetest@${PHD_VAR_network_hosts_vip}/trove\nopenstack-config --set /etc/trove/trove.conf database max_retries -1\nopenstack-config --set /etc/trove/trove.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/trove/trove.conf keystone_authtoken admin_user trove\nopenstack-config --set /etc/trove/trove.conf keystone_authtoken admin_password trovetest\nopenstack-config --set /etc/trove/trove.conf keystone_authtoken service_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/trove/trove.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/trove/trove.conf keystone_authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/trove/trove.conf DEFAULT control_exchange trove\n\nopenstack-config --set /etc/trove/trove-conductor.conf DEFAULT trove_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/trove/trove-conductor.conf DEFAULT os_region_name regionOne\nopenstack-config --set /etc/trove/trove-conductor.conf DEFAULT log_file trove-conductor.log\nopenstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_password guest\nopenstack-config --set /etc/trove/trove-conductor.conf DEFAULT rpc_backend rabbit\nopenstack-config --set /etc/trove/trove-conductor.conf database connection mysql://trove:trovetest@${PHD_VAR_network_hosts_vip}/trove\nopenstack-config --set /etc/trove/trove-conductor.conf database max_retries -1\nopenstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken admin_user trove\nopenstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken admin_password trovetest\nopenstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken service_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/trove/trove-conductor.conf DEFAULT control_exchange trove\n\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT trove_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT os_region_name regionOne\nopenstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_password guest\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT rpc_backend rabbit\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_user trove\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_pass trovetest\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_tenant_name ${SERVICES_TENANT_ID}\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT log_file trove-taskmanager.log\nopenstack-config --set /etc/trove/trove-taskmanager.conf database connection mysql://trove:trovetest@${PHD_VAR_network_hosts_vip}/trove\nopenstack-config --set /etc/trove/trove-taskmanager.conf database max_retries -1\nopenstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken admin_user trove\nopenstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken admin_password trovetest\nopenstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken service_host ${PHD_VAR_network_hosts_vip}\nopenstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT cloudinit_loaction /etc/trove/cloudinit\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT network_driver trove.network.neutron.NeutronDriver\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT control_exchange trove\n# The following is a workaround for https://bugs.launchpad.net/trove/+bug/1402055\nopenstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT exists_notification_transformer\n\nopenstack-config --set /etc/trove/trove.conf DEFAULT default_datastore mysql\nopenstack-config --set /etc/trove/trove.conf DEFAULT add_addresses True\nopenstack-config --set /etc/trove/trove.conf DEFAULT network_label_regex ^private$\n\ncp /usr/share/trove/trove-dist-paste.ini /etc/trove/api-paste.ini\nopenstack-config --set /etc/trove/api-paste.ini filter:authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/trove/api-paste.ini filter:authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/\nopenstack-config --set /etc/trove/api-paste.ini filter:authtoken admin_password trovetest\nopenstack-config --set /etc/trove/api-paste.ini filter:authtoken admin_user trove\nopenstack-config --set /etc/trove/api-paste.ini filter:authtoken admin_tenant_name services\nopenstack-config --set /etc/trove/trove.conf DEFAULT api_paste_config /etc/trove/api-paste.ini\n....\n\ntarget=$PHD_ENV_nodes1\n....\nsu trove -s /bin/sh -c \"trove-manage db_sync\"\ntrove-manage datastore_update mysql ''\n# Create image using CentOS7.\n# This is not ideal, check https://www.rdoproject.org/forum/discussion/1010/creation-of-trove-compatible-images-for-rdo/p1\n. /root/keystonerc_admin\ncd /tmp\nwget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2\nglance image-create --name centos7 --disk-format qcow2 --container-format bare --is-public True --owner trove --file CentOS-7-x86_64-GenericCloud.qcow2\nIMAGE_ID=$(openstack image show centos7 --format value --column id)\ntrove-manage --config-file=/etc/trove/trove.conf datastore_version_update mysql mysql-5.5 mysql ${IMAGE_ID} mysql55 1\ntrove-manage db_load_datastore_config_parameters mysql \"mysql-5.5\" /usr/lib/python2.7/site-packages/trove/templates/mysql/validation-rules.json\n\n....\n\ntarget=all\n....\nmkdir /etc/trove/cloudinit\ncat > /etc/trove/cloudinit/mysql.cloudinit << EOF\n#!/bin/bash\n\nsed -i'.orig' -e's/without-password/yes/' /etc/ssh/sshd_config\necho \"test\" | passwd --stdin centos\necho \"redhat\" | passwd --stdin root\nservice sshd restart\n\nyum -y install wget epel-release \nyum -y install http://rdoproject.org/repos/openstack-kilo/rdo-testing-kilo.rpm\nyum -y install openstack-trove-guestagent mariadb-server openstack-utils python-oslo-messaging python-osprofiler python-oslo-concurrency\n\nsystemctl enable mariadb\nsystemctl start mariadb\n\necho \"trove ALL=(ALL) NOPASSWD: ALL\" >> /etc/sudoers\necho \"Defaults:trove !requiretty\" >> /etc/sudoers\n\nopenstack-config --set /etc/trove/trove-guestagent.conf DEFAULT rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/trove/trove-guestagent.conf DEFAULT rabbit_password guest\nopenstack-config --set /etc/trove/trove-guestagent.conf DEFAULT nova_proxy_admin_user trove\nopenstack-config --set /etc/trove/trove-guestagent.conf DEFAULT nova_proxy_admin_pass trovetest\nopenstack-config --set /etc/trove/trove-guestagent.conf DEFAULT nova_proxy_admin_tenant_name services\nopenstack-config --set /etc/trove/trove-guestagent.conf DEFAULT trove_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0\nopenstack-config --set /etc/trove/trove-guestagent.conf DEFAULT control_exchange trove\nopenstack-config --set /etc/trove/trove-guestagent.conf DEFAULT log_dir /var/log/trove\nopenstack-config --set /etc/trove/trove-guestagent.conf DEFAULT log_file trove-guestagent.log\nopenstack-config --set /etc/trove/trove-guestagent.conf DEFAULT datastore_manager mysql\n\necho \"${PHD_VAR_network_ssh_pubkey}\" >> /root/.ssh/authorized_keys\necho \"${PHD_VAR_network_ssh_pubkey}\" >> /home/centos/.ssh/authorized_keys\n\nsystemctl stop openstack-trove-guestagent\nsystemctl enable openstack-trove-guestagent\nsystemctl start openstack-trove-guestagent\nEOF\n\nsystemctl enable openstack-trove-api\nsystemctl enable openstack-trove-taskmanager\nsystemctl enable openstack-trove-conductor\nsystemctl start openstack-trove-api\nsystemctl start openstack-trove-taskmanager\nsystemctl start openstack-trove-conductor\nfirewall-cmd --add-port=8779/tcp\nfirewall-cmd --add-port=8779/tcp --permanent\n....\n\n" }, { "path": "keepalived/rabbitmq-config.md", "content": "Introduction\n------------\n\nRabbitMQ can create a native cluster, by grouping several nodes and replicating all message queues. Clustered RabbitMQ environments tolerate the failure of individual nodes. Nodes can be started and stopped at will.\n\n![](Rabbitmq_clustering.jpg \"RabbitMQ clustering\")\n\n**Note:** Access to RabbitMQ is not handled by HAproxy, as there are known issues integrating RabbitMQ with HAProxy. Instead consumers must be supplied with the full list of hosts running RabbitMQ with `rabbit_hosts` and `rabbit_ha_queues` options, and connect directly to one or more RabbitMQ servers. Reconnections in case of a node failure are handled automatically.\n\nThe following commands will be executed on all controller nodes, unless stated otherwise.\n\nYou can find a phd scenario file [here](phd-setup/rabbitmq.scenario).\n\nInstall package\n---------------\n\n yum -y install rabbitmq-server\n\nCreate erlang cookie and distribute\n-----------------------------------\n\n**On node 1:**\n\n cat > /etc/rabbitmq/rabbitmq-env.conf << EOF\n NODE_IP_ADDRESS=192.168.1.221\n EOF\n\n systemctl start rabbitmq-server\n systemctl stop rabbitmq-server\n scp -p /var/lib/rabbitmq/.erlang.cookie hacontroller2:/var/lib/rabbitmq\n scp -p /var/lib/rabbitmq/.erlang.cookie hacontroller3:/var/lib/rabbitmq\n\nSet permissions for erlang cookie\n---------------------------------\n\n**On node 2 and node 3:**\n\n\n chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie\n\n cat > /etc/rabbitmq/rabbitmq-env.conf << EOF\n NODE_IP_ADDRESS=192.168.1.22X\n EOF\n\n### Create rabbitmq configuration\n\n**On all nodes:**\n\n cat > /etc/rabbitmq/rabbitmq.config << EOF\n\n [\n {rabbit, [\n {cluster_nodes, {['rabbit@hacontroller1', 'rabbit@hacontroller2', 'rabbit@hacontroller3'], disc}},\n {cluster_partition_handling, ignore},\n {default_user, <<\"guest\">>},\n {default_pass, <<\"guest\">>},\n {tcp_listen_options, [binary,\n {packet, raw},\n {reuseaddr, true},\n {backlog, 128},\n {nodelay, true},\n {exit_on_close, false},\n {keepalive, true}]}\n ]},\n {kernel, [\n {inet_dist_listen_max, 44001},\n {inet_dist_listen_min, 44001}\n ]}\n ].\n\n EOF\n\nSet kernel TCP keepalive parameters\n-----------------------------------\n\n cat > /etc/sysctl.d/tcpka.conf << EOF\n net.ipv4.tcp_keepalive_intvl = 1\n net.ipv4.tcp_keepalive_probes = 5\n net.ipv4.tcp_keepalive_time = 5\n EOF\n\n sysctl -p /etc/sysctl.d/tcpka.conf\n\nStart services and open firewall ports\n--------------------------------------\n\n firewall-cmd --add-port=5672/tcp \n firewall-cmd --add-port=4369/tcp\n firewall-cmd --add-port=5672/tcp --permanent\n firewall-cmd --add-port=4369/tcp --permanent\n firewall-cmd --add-port=44001/tcp\n firewall-cmd --add-port=44001/tcp --permanent\n systemctl enable rabbitmq-server\n systemctl start rabbitmq-server\n\nAnd check everything is going fine by running:\n\n
 # rabbitmqctl cluster_status\n\nCluster status of node rabbit@hacontroller1 ...\n[{nodes,[{disc,[rabbit@hacontroller1,rabbit@hacontroller2,\nrabbit@hacontroller3]}]},\n{running_nodes,[rabbit@hacontroller3,rabbit@hacontroller2,\nrabbit@hacontroller1]},\n{cluster_name,<<\"rabbit@hacontroller1.example.com\">\\>}, {partitions,[]}]\n\n...done.\n\n
\nSet HA mode for all queues\n--------------------------\n\n**On node 1:**\n\n rabbitmqctl set_policy HA '^(?!amq\\.).*' '{\"ha-mode\": \"all\"}'\n" }, { "path": "keepalived/rabbitmq-restart.md", "content": "Introduction\n------------\n\nIn general, RabbitMQ does a good job at restarting the cluster when all nodes are stared at the same time. However, we may find times were this is not the case, and we will have to restart the cluster manually.\n\nAccording to [](http://previous.rabbitmq.com/v3_3_x/clustering.html) *\"the last node to go down must be the first node to be brought online. If this doesn't happen, the nodes will wait 30 seconds for the last disc node to come back online, and fail afterwards.\"*. Thus, it is necessary to find the last node going down and start it. Depending on how the nodes were started, you may see some nodes running and some stopped.\n\nChecking RabbitMQ cluster status\n--------------------------------\n\nRun the following command to verify the current RabbitMQ cluster status:\n\n rabbitmqctl cluster_status\n\n Cluster status of node rabbit@hacontroller3 ...\n [{nodes,[{disc,[rabbit@hacontroller1,rabbit@hacontroller2,\n rabbit@hacontroller3]}]},\n {running_nodes,[rabbit@hacontroller2,rabbit@hacontroller1,\n rabbit@hacontroller3]},\n {cluster_name,<<\"rabbit@hacontroller1.example.com\">>},\n {partitions,[]}]\n\n### Some nodes are running\n\nIf some nodes are running, the most probable reason is that the failed nodes timed out before finding the last node to come back online. In this case, start rabbitmq-server on the failed nodes.\n\n [root@hacontroller1 ~]# systemctl start rabbitmq-server\n\n### None of the nodes are running\n\nIn this case, we need to find which node should be started first.\n\nSelect a node as first node, start rabbitmq-server, then start it on the remaining nodes.\n\n [root@hacontroller2 ~]# systemctl start rabbitmq-server\n [root@hacontroller3 ~]# systemctl start rabbitmq-server\n [root@hacontroller1 ~]# systemctl start rabbitmq-server\n\nCheck that all nodes are running rabbitmq-server. If not, stop any surviving rabbitmq-server and select a different node as first node.\n" }, { "path": "keepalived/redis-config.md", "content": "Introduction\n------------\n\nRedis in a key-value cache and store, used by Ceilometer with the [tooz](https://github.com/openstack/tooz) library. It uses an master-slave architecture for high availability, where a single node is used for writes and a number of slaves replicate data from it. Using [Sentinel](http://redis.io/topics/sentinel), it is possible monitor node health and fail over automatically to another node if needed. By configuring Ceilometer to access the Sentinel processes, high availability from the consumer point of view is transparent.\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/redis.scenario).\n\nInstall redis\n-------------\n\n yum install -y redis\n\nConfigure bind IP, set master and slaves\n----------------------------------------\n\nOn all nodes:\n\n sed --in-place 's/bind 127.0.0.1/bind 127.0.0.1 192.168.1.X/' /etc/redis.conf\n\nOn node2 and 3:\n\n echo slaveof '''' 6379 >> /etc/redis.conf \n\nConfigure Sentinel, used for master failover\n--------------------------------------------\n\n**On all nodes:**\n\n cat > /etc/redis-sentinel.conf << EOF\n\n sentinel monitor mymaster 6379 2\n sentinel down-after-milliseconds mymaster 30000\n sentinel failover-timeout mymaster 180000\n sentinel parallel-syncs mymaster 1\n min-slaves-to-write 1\n min-slaves-max-lag 10\n logfile /var/log/redis/sentinel.log\n EOF\n\nConfigure firewall, start services\n----------------------------------\n\n firewall-cmd --add-port=6379/tcp\n firewall-cmd --add-port=6379/tcp --permanent\n firewall-cmd --add-port=26379/tcp\n firewall-cmd --add-port=26379/tcp --permanent\n systemctl enable redis\n systemctl start redis\n systemctl enable redis-sentinel\n systemctl start redis-sentinel\n" }, { "path": "keepalived/sahara-config.md", "content": "Introduction\n------------\n\n**Important:** this configuration assumes that the controller nodes can access the floating IP network (10.10.10.0/24 in the example configuration). This was not the case on [the controller node configuration](controller-node.md), because the NIC used for the provider network did not have an IP address. You can accomplish this by setting up routes in the default gateway (192.168.1.1), or creating a separate route on the controller nodes.\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/sahara.scenario).\n\nInstall software\n----------------\n\n yum install -y openstack-sahara-api openstack-sahara-engine openstack-sahara-common openstack-sahara python-saharaclient\n\nConfigure Sahara\n----------------\n\n openstack-config --set /etc/sahara/sahara.conf DEFAULT host 192.168.1.22X\n openstack-config --set /etc/sahara/sahara.conf DEFAULT use_floating_ips True\n openstack-config --set /etc/sahara/sahara.conf DEFAULT use_neutron True\n openstack-config --set /etc/sahara/sahara.conf DEFAULT rpc_backend rabbit\n openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_port 5672\n openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_use_ssl False\n openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_userid guest\n openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_password guest\n openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_login_method AMQPLAIN\n openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/sahara/sahara.conf DEFAULT notification_topics notifications\n openstack-config --set /etc/sahara/sahara.conf database connection mysql://sahara:saharatest@controller-vip.example.com/sahara\n openstack-config --set /etc/sahara/sahara.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/sahara/sahara.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/sahara/sahara.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/sahara/sahara.conf keystone_authtoken username sahara\n openstack-config --set /etc/sahara/sahara.conf keystone_authtoken password saharatest\n openstack-config --set /etc/sahara/sahara.conf keystone_authtoken project_name services\n openstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_tenant_name services\n openstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_user sahara\n openstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_password saharatest\n openstack-config --set /etc/sahara/sahara.conf DEFAULT log_file /var/log/sahara/sahara.log\n\nManage DB\n---------\n\nOn node 1:\n\n sahara-db-manage --config-file /etc/sahara/sahara.conf upgrade head\n\n\nStart services, open firewall ports\n-----------------------------------\n firewall-cmd --add-port=8386/tcp\n firewall-cmd --add-port=8386/tcp --permanent\n systemctl enable openstack-sahara-api\n systemctl enable openstack-sahara-engine\n systemctl start openstack-sahara-api\n systemctl start openstack-sahara-engine\n\nTesting\n-------\n\nOn node 1, run the following commands to test the Sahara API:\n\n . /root/keystonerc_admin\n sahara plugin-list\n\nFurther Sahara testing requires creating a specific virtual machine image, which is outside the scope of this document. You can find instructions on [the Sahara wiki](http://docs.openstack.org/developer/sahara/devref/quickstart.html#upload-an-image-to-the-image-service).\n" }, { "path": "keepalived/swift-config.md", "content": "Introduction\n------------\n\nWe need to have an additional disk, `/dev/vdb` in our test available for Swift usage.\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYou can find a phd scenario file [here](phd-setup/swift.scenario).\n\nInstall software\n----------------\n\n yum install -y openstack-swift-object openstack-swift-container openstack-swift-account openstack-swift-proxy openstack-utils rsync xfsprogs\n\nCreate XFS file system for additional disk, and mount it\n--------------------------------------------------------\n\n mkfs.xfs /dev/vdb\n mkdir -p /srv/node/vdb\n echo \"/dev/vdb /srv/node/vdb xfs defaults 1 2\" >> /etc/fstab\n mount -a \n chown -R swift:swift /srv/node\n restorecon -R /srv/node\n\nConfigure account, container and object services\n------------------------------------------------\n\n openstack-config --set /etc/swift/object-server.conf DEFAULT bind_ip 192.168.1.22X\n openstack-config --set /etc/swift/object-server.conf DEFAULT devices /srv/node\n openstack-config --set /etc/swift/account-server.conf DEFAULT bind_ip 192.168.1.22X\n openstack-config --set /etc/swift/account-server.conf DEFAULT devices /srv/node\n openstack-config --set /etc/swift/container-server.conf DEFAULT bind_ip 192.168.1.22X\n openstack-config --set /etc/swift/container-server.conf DEFAULT devices /srv/node\n chown -R root:swift /etc/swift\n\nStart account, container and object services, open firewall ports\n-----------------------------------------------------------------\n\n systemctl start openstack-swift-account\n systemctl start openstack-swift-container\n systemctl start openstack-swift-object\n systemctl enable openstack-swift-account\n systemctl enable openstack-swift-container\n systemctl enable openstack-swift-object\n\n firewall-cmd --add-port=6200/tcp\n firewall-cmd --add-port=6200/tcp --permanent\n firewall-cmd --add-port=6201/tcp\n firewall-cmd --add-port=6201/tcp --permanent\n firewall-cmd --add-port=6202/tcp\n firewall-cmd --add-port=6202/tcp --permanent\n\nConfigure swift proxy and object expirer\n----------------------------------------\n\n openstack-config --set /etc/swift/proxy-server.conf filter:authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/swift/proxy-server.conf filter:authtoken auth_plugin password\n openstack-config --set /etc/swift/proxy-server.conf filter:authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/swift/proxy-server.conf filter:authtoken username swift\n openstack-config --set /etc/swift/proxy-server.conf filter:authtoken password swifttest\n openstack-config --set /etc/swift/proxy-server.conf filter:authtoken project_name services\n openstack-config --set /etc/swift/proxy-server.conf filter:cache memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211\n openstack-config --set /etc/swift/proxy-server.conf DEFAULT bind_ip 192.168.1.22X\n openstack-config --set /etc/swift/object-expirer.conf filter:cache memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211\n openstack-config --set /etc/swift/object-expirer.conf object-expirer concurrency 100\n\nSet Ceilometer hook\n-------------------\n\nOn node 1:\n\n cat >> /etc/swift/swift.conf << EOF\n [filter:ceilometer]\n use = egg:ceilometer#swift\n [pipeline:main]\n pipeline = healthcheck cache authtoken keystoneauth proxy-server ceilometer\n EOF\n\nConfigure hash path suffix\n--------------------------\n\nOn node 1:\n\n openstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_suffix $(openssl rand -hex 10)\n\nCreate rings\n------------\n\nOn node 1:\n\n swift-ring-builder /etc/swift/object.builder create 16 3 24\n swift-ring-builder /etc/swift/container.builder create 16 3 24\n swift-ring-builder /etc/swift/account.builder create 16 3 24\n swift-ring-builder /etc/swift/account.builder add z1-192.168.1.221:6202/vdb 10\n swift-ring-builder /etc/swift/container.builder add z1-192.168.1.221:6201/vdb 10\n swift-ring-builder /etc/swift/object.builder add z1-192.168.1.221:6200/vdb 10\n swift-ring-builder /etc/swift/account.builder add z2-192.168.1.222:6202/vdb 10\n swift-ring-builder /etc/swift/container.builder add z2-192.168.1.222:6201/vdb 10\n swift-ring-builder /etc/swift/object.builder add z2-192.168.1.222:6200/vdb 10\n swift-ring-builder /etc/swift/account.builder add z3-192.168.1.223:6202/vdb 10\n swift-ring-builder /etc/swift/container.builder add z3-192.168.1.223:6201/vdb 10\n swift-ring-builder /etc/swift/object.builder add z3-192.168.1.223:6200/vdb 10\n swift-ring-builder /etc/swift/account.builder rebalance\n swift-ring-builder /etc/swift/container.builder rebalance\n swift-ring-builder /etc/swift/object.builder rebalance\n\n cd /etc/swift\n tar cvfz /tmp/swift_configs.tgz swift.conf *.builder *.gz\n scp /tmp/swift_configs.tgz hacontroller2:/tmp\n scp /tmp/swift_configs.tgz hacontroller3:/tmp\n chown -R root:swift /etc/swift\n\nImport swift configuration from node 1\n--------------------------------------\n\nOn nodes 2 and 3:\n\n cd /etc/swift\n tar xvfz /tmp/swift_configs.tgz\n chown -R root:swift /etc/swift\n restorecon -R /etc/swift\n\nStart services, open firewall ports\n-----------------------------------\n\nOn all nodes:\n\n systemctl start openstack-swift-proxy\n systemctl enable openstack-swift-proxy\n systemctl start openstack-swift-object-expirer\n systemctl enable openstack-swift-object-expirer\n firewall-cmd --add-port=8080/tcp\n firewall-cmd --add-port=8080/tcp --permanent\n\nTest\n----\n\nOn any node:\n\n . /root/keystonerc_admin\n swift list\n swift upload test /tmp/cirros-0.3.3-x86_64-disk.img \n swift list\n swift list test\n swift download test tmp/cirros-0.3.3-x86_64-disk.img\n" }, { "path": "keepalived/trove-config.md", "content": "Introduction\n------------\n\n**WARNING:** this configuration uses the same RabbitMQ instance as the rest of the OpenStack services. This is not recommended for a production environment, because it creates a security risk (see [this e-mail thread](http://lists.openstack.org/pipermail/openstack-dev/2015-April/061759.html) for details). Because of that, this configuration assumes that virtual machine instances can access the controller node management network (192.168.1.0/24 in the example configuration). This requires setting up the required routes and firewall rules to ensure this is possible. Those firewall rules should allow access from the floating IP network (10.10.10.0/24) to the controller node management network.\n\n**Important:** On a production environment, a separate RabbitMQ instance should be deployed for Trove, so you should adjust the rabbit_* configuration lines accordingly.\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nInstall software\n----------------\n\n yum install -y openstack-trove python-troveclient\n\nConfigure Trove\n---------------\n\n openstack-config --set /etc/trove/trove.conf DEFAULT bind_host 192.168.1.22X\n openstack-config --set /etc/trove/trove.conf DEFAULT log_dir /var/log/trove\n openstack-config --set /etc/trove/trove.conf DEFAULT trove_auth_url http://controller-vip.example.com:35357/v2.0\n openstack-config --set /etc/trove/trove.conf DEFAULT os_region_name regionOne\n openstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_password guest\n openstack-config --set /etc/trove/trove.conf DEFAULT rpc_backend rabbit\n openstack-config --set /etc/trove/trove.conf database connection mysql://trove:trovetest@controller-vip.example.com/trove\n openstack-config --set /etc/trove/trove.conf database max_retries -1\n openstack-config --set /etc/trove/trove.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/trove/trove.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/trove/trove.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/trove/trove.conf keystone_authtoken username trove\n openstack-config --set /etc/trove/trove.conf keystone_authtoken password trovetest\n openstack-config --set /etc/trove/trove.conf keystone_authtoken project_name services\n\n openstack-config --set /etc/trove/trove-conductor.conf DEFAULT trove_auth_url http://controller-vip.example.com:35357/v2.0\n openstack-config --set /etc/trove/trove-conductor.conf DEFAULT os_region_name regionOne\n openstack-config --set /etc/trove/trove-conductor.conf DEFAULT log_file trove-conductor.log\n openstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_password guest\n openstack-config --set /etc/trove/trove-conductor.conf DEFAULT rpc_backend rabbit\n openstack-config --set /etc/trove/trove-conductor.conf database connection mysql://trove:trovetest@controller-vip.example.com/trove\n openstack-config --set /etc/trove/trove-conductor.conf database max_retries -1\n openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken username trove\n openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken password trovetest\n openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken project_name services\n\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT trove_auth_url http://controller-vip.example.com:35357/v2.0\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT os_region_name regionOne\n openstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3\n openstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_ha_queues true\n openstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_password guest\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT rpc_backend rabbit\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_user trove\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_pass trovetest\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_tenant_name ${SERVICES_TENANT_ID}\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT log_file trove-taskmanager.log\n openstack-config --set /etc/trove/trove-taskmanager.conf database connection mysql://trove:trovetest@controller-vip.example.com/trove\n openstack-config --set /etc/trove/trove-taskmanager.conf database max_retries -1\n openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/\n openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken auth_plugin password\n openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken username trove\n openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken password trovetest\n openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken project_name services\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT cloudinit_loaction /etc/trove/cloudinit\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT network_driver trove.network.neutron.NeutronDriver\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT control_exchange trove\n # The following is a workaround for https://bugs.launchpad.net/trove/+bug/1402055\n openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT exists_notification_transformer\n\n openstack-config --set /etc/trove/trove.conf DEFAULT default_datastore mysql\n openstack-config --set /etc/trove/trove.conf DEFAULT add_addresses True\n openstack-config --set /etc/trove/trove.conf DEFAULT network_label_regex ^private$\n\n cp /usr/share/trove/trove-dist-paste.ini /etc/trove/api-paste.ini\n openstack-config --set /etc/trove/api-paste.ini filter:authtoken auth_uri http://controller-vip.example.com:35357/\n openstack-config --set /etc/trove/api-paste.ini filter:authtoken auth_url http://controller-vip.example.com:35357/\n openstack-config --set /etc/trove/api-paste.ini filter:authtoken auth_plugin password\n openstack-config --set /etc/trove/api-paste.ini filter:authtoken password trovetest\n openstack-config --set /etc/trove/api-paste.ini filter:authtoken username trove\n openstack-config --set /etc/trove/api-paste.ini filter:authtoken project_name services\n openstack-config --set /etc/trove/trove.conf DEFAULT api_paste_config /etc/trove/api-paste.ini\n\n\nManage DB\n---------\n\nOn node 1:\n\n su trove -s /bin/sh -c \"trove-manage db_sync\"\n trove-manage datastore_update mysql ''\n\nCreate and upload image\n-----------------------\n\nTrove instances will require a specially crafted virtual machine image with the required database software. The following instructions will create a simple image based on CentOS 7 with MariaDB 5.5. [This article](https://www.rdoproject.org/forum/discussion/1010/creation-of-trove-compatible-images-for-rdo/p1) provides detailed instructions on how to create a Trove-compatible image using trove-image-elements.\n\nPlease note that only limited testing has been performed with the image. These instructions can be updated over time, please feel free to provide feedback if you have a chance to test them.\n\nOn node 1:\n\n yum -y install git\n git clone https://github.com/vkmc/trove-image-elements\n cd trove-image-elements\n ./create_trove_image.sh -d centos -s mysql\n . /root/keystonerc_admin\n glance image-create --name centos-mysql-guest-image --disk-format qcow2 --container-format bare --visibility public --file centos-mysql-guest-image.qcow2\n glance image-update --owner trove\n\nTake note of the image id, then update the Trove database with a reference to the newly uploaded image:\n \n trove-manage --config-file=/etc/trove/trove.conf datastore_version_update mysql mysql-5.5 mysql 9b412ead-5a5c-40ba-ac8f-98f70cc4f682 mysql55 1\n trove-manage db_load_datastore_config_parameters mysql \"mysql-5.5\" /usr/lib/python2.7/site-packages/trove/templates/mysql/validation-rules.json\n trove-manage datastore_update mysql mysql-5.5\n\nStart services, open firewall ports\n-----------------------------------\nOn all nodes:\n\n systemctl enable openstack-trove-api\n systemctl enable openstack-trove-taskmanager\n systemctl enable openstack-trove-conductor\n systemctl start openstack-trove-api\n systemctl start openstack-trove-taskmanager\n systemctl start openstack-trove-conductor\n firewall-cmd --add-port=8779/tcp\n firewall-cmd --add-port=8779/tcp --permanent\n" }, { "path": "make-vm", "content": "#!/bin/bash\n\nonlyone=$1\n\nexport PHD_VAR_network_nic_base=\"54:52:00\"\nexport PHD_VAR_vm_base=\"/srv/rhos6-rhel7-vms/rhos6-rhel7-base.img\"\nexport PHD_VAR_vm_vcpu=\"1\"\nexport PHD_VAR_vm_ram=\"2048\"\nexport PHD_ENV_nodes1=east-01.lab.bos.redhat.com\n\ncat<<-EOF > /localvms/template.xml\n\n VM_NAME\n ${PHD_VAR_vm_ram}000\n ${PHD_VAR_vm_ram}000\n ${PHD_VAR_vm_cpus}\n \n hvm\n \n \n \n \n \n \n \n \n destroy\n restart\n restart\n \n /usr/libexec/qemu-kvm\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\nEOF\nsequence=16\nlastoct=\"$(hostname -s | sed -e 's#^[a-z]*-##g' -e 's#^0*##g')\"\noffset=\"$(echo ${PHD_ENV_nodes1} | sed -e 's#^[a-z]*-##g' -e 's#^0*##g' -e 's#\\..*##')\"\nfor section in lb db rabbitmq memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer qpid node keystone; do\n\n dobuild=0\n if [ -z $onlyone ]; then\n dobuild=1\n elif [ $onlyone = $section ]; then\n dobuild=1\n fi\n \n if [ $dobuild = 1 ]; then\n cd /localvms/\n target=rhos6-${section}$(( ${lastoct} - ${offset} ))\n virsh destroy $target > /dev/null 2>&1\n virsh undefine $target > /dev/null 2>&1\n cp template.xml ${target}.xml\n sed -i.sed s#VM_NAME#${target}#g ${target}.xml\n sed -i.sed s#EXTERNAL_MAC#${PHD_VAR_network_nic_base}:0${lastoct}:00:${sequence}#g ${target}.xml\n sed -i.sed s#INTERNAL_MAC#${PHD_VAR_network_nic_base}:0${lastoct}:01:${sequence}#g ${target}.xml\n sed -i.sed s:source\\ file.*\\/:source\\ file=\\'/localvms/${target}.cow\\'\\/:g ${target}.xml\n diff -u template.xml ${target}.xml\n rm -f /localvms/${target}.cow\n qemu-img create -b /localvms/$(basename ${PHD_VAR_vm_base}) -f qcow2 /localvms/${target}.cow\n virsh define ${target}.xml\n if [ $? != 0 ]; then exit 1; fi\n virsh start ${target}\n if [ $? != 0 ]; then exit 1; fi\n rm ${target}.xml.sed ${target}.xml\n fi\n \n sequence=$((sequence + 1))\ndone\n" }, { "path": "pcmk/NovaCompute", "content": "#!/bin/sh\n#\n#\n# NovaCompute agent manages compute daemons.\n#\n# Copyright (c) 2015\n#\n# This program is free software; you can redistribute it and/or modify\n# it under the terms of version 2 of the GNU General Public License as\n# published by the Free Software Foundation.\n#\n# This program is distributed in the hope that it would be useful, but\n# WITHOUT ANY WARRANTY; without even the implied warranty of\n# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n#\n# Further, this software is distributed without any warranty that it is\n# free of the rightful claim of any third person regarding infringement\n# or the like. Any license provided herein, whether implied or\n# otherwise, applies only to this software file. Patent licenses, if\n# any, provided herein do not apply to combinations of this program with\n# other software, or any other product whatsoever.\n#\n# You should have received a copy of the GNU General Public License\n# along with this program; if not, write the Free Software Foundation,\n# Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.\n#\n\n#######################################################################\n# Initialization:\n\n###\n: ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat}\n. ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs\n###\n\n: ${__OCF_ACTION=$1}\n\n#######################################################################\n\nmeta_data() {\n\tcat <\n\n\n1.0\n\n\nOpenStack Nova Compute Server.\n\nOpenStack Nova Compute Server\n\n\n\n\n\nAuthorization URL for connecting to keystone in admin context\n\nAuthorization URL\n\n\n\n\n\nUsername for connecting to keystone in admin context\n\nUsername\n\n\n\n\nPassword for connecting to keystone in admin context\n\nPassword\n\n\n\n\n\nTenant name for connecting to keystone in admin context.\nNote that with Keystone V3 tenant names are only unique within a domain.\n\nTenant name\n\n\n\n\n\nDNS domain in which hosts live, useful when the cluster uses short names and nova uses FQDN\n\nDNS domain\n\n\n\n\n\nNova API location (internal, public or admin URL)\n\nNova API location (internal, public or admin URL)\n\n\n\n\n\nDisable shared storage recovery for instances. Use at your own risk!\n\nDisable shared storage recovery for instances\n\n\n\n\n\nHow long to wait for nova to finish evacuating instances elsewhere\nbefore starting nova-compute. Only used when the agent detects\nevacuations might be in progress.\n\nYou may need to increase the start timeout when increasing this value.\n\nDelay to allow evacuations time to complete\n\n\n\n\n\n\n\n\n\n\n\n\n\nEND\n}\n\n#######################################################################\n\n# don't exit on TERM, to test that lrmd makes sure that we do exit\ntrap sigterm_handler TERM\nsigterm_handler() {\n\tocf_log info \"They use TERM to bring us down. No such luck.\"\n\treturn\n}\n\nnova_usage() {\n\tcat </dev/null)\n if [ $? = 1 ]; then\n\tif [ \"x${OCF_RESKEY_domain}\" != x ]; then\n\t NOVA_HOST=$(uname -n | awk -F. '{print $1}')\n\telse\n\t NOVA_HOST=$(uname -n)\n\tfi\n fi\n\n # We only need to check a configured value, calculated ones are fine\n openstack-config --get /etc/nova/nova.conf DEFAULT host 2>/dev/null\n if [ $? = 0 ]; then\n\tif [ \"x${OCF_RESKEY_domain}\" != x ]; then\n\t short_host=$(uname -n | awk -F. '{print $1}')\n\t if [ \"x$NOVA_HOST\" != \"x${short_host}\" ]; then\n\t\tocf_exit_reason \"Invalid Nova host name, must be ${short_host} in order for instance recovery to function\"\n\t\trc=$OCF_ERR_CONFIGURED\n\t fi\n\n\telif [ \"x$NOVA_HOST\" != \"x$(uname -n)\" ]; then\n ocf_exit_reason \"Invalid Nova host name, must be $(uname -n) in order for instance recovery to function\"\n\t rc=$OCF_ERR_CONFIGURED\n\tfi\n fi\n\n if [ $rc != $OCF_SUCCESS ]; then\n\texit $rc\n fi\n return $rc\n}\n\n: ${OCF_RESKEY_evacuation_delay=120}\ncase $__OCF_ACTION in\nmeta-data)\tmeta_data\n\t\texit $OCF_SUCCESS\n\t\t;;\nusage|help)\tnova_usage\n\t\texit $OCF_SUCCESS\n\t\t;;\nesac\n\ncase $__OCF_ACTION in\nstart)\t\tnova_validate; nova_start;;\nstop)\t\tnova_stop;;\nmonitor)\tnova_validate; nova_monitor;;\nnotify)\t\tnova_notify;;\nvalidate-all)\texit $OCF_SUCCESS;;\n*)\t\tnova_usage\n\t\texit $OCF_ERR_UNIMPLEMENTED\n\t\t;;\nesac\nrc=$?\nocf_log debug \"${OCF_RESOURCE_INSTANCE} $__OCF_ACTION : $rc\"\nexit $rc\n" }, { "path": "pcmk/NovaEvacuate", "content": "#!/bin/sh\n#\n#\n# NovaCompute agent manages compute daemons.\n#\n# Copyright (c) 2015\n#\n# This program is free software; you can redistribute it and/or modify\n# it under the terms of version 2 of the GNU General Public License as\n# published by the Free Software Foundation.\n#\n# This program is distributed in the hope that it would be useful, but\n# WITHOUT ANY WARRANTY; without even the implied warranty of\n# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n#\n# Further, this software is distributed without any warranty that it is\n# free of the rightful claim of any third person regarding infringement\n# or the like. Any license provided herein, whether implied or\n# otherwise, applies only to this software file. Patent licenses, if\n# any, provided herein do not apply to combinations of this program with\n# other software, or any other product whatsoever.\n#\n# You should have received a copy of the GNU General Public License\n# along with this program; if not, write the Free Software Foundation,\n# Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.\n#\n\n#######################################################################\n# Initialization:\n\n###\n: ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat}\n. ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs\n###\n\n: ${__OCF_ACTION=$1}\n\n#######################################################################\n\nmeta_data() {\n\tcat <\n\n\n1.0\n\n\nFacility for tacking a list of compute nodes and reliably evacuating the ones that fence_evacuate has flagged.\n\nEvacuator for OpenStack Nova Compute Server\n\n\n\n\n\nAuthorization URL for connecting to keystone in admin context\n\nAuthorization URL\n\n\n\n\n\nUsername for connecting to keystone in admin context\n\nUsername\n\n\n\n\nPassword for connecting to keystone in admin context\n\nPassword\n\n\n\n\n\nTenant name for connecting to keystone in admin context.\nNote that with Keystone V3 tenant names are only unique within a domain.\n\nTenant name\n\n\n\n\n\nNova API location (internal, public or admin URL)\n\nNova API location (internal, public or admin URL)\n\n\n\n\n\nDisable shared storage recovery for instances. Use at your own risk!\n\nDisable shared storage recovery for instances\n\n\n\n\n\n\n\n\n\n\n\n\n\nEND\n}\n\n#######################################################################\n\n# don't exit on TERM, to test that lrmd makes sure that we do exit\ntrap sigterm_handler TERM\nsigterm_handler() {\n\tocf_log info \"They use TERM to bring us down. No such luck.\"\n\treturn\n}\n\nevacuate_usage() {\n\tcat < /dev/null 2>&1\" \"$node\"\n phd_wait_connection 2400 $node || exit 1\ndone\n....\n\ntarget=all\n....\n# wait for the old snapshot to be merged/deleted\nloop=0\nwhile ! lvcreate -s -n baremetal_snap -l100%FREE /dev/mapper/*root; do\n sleep 1\n if [ \"$loop\" = 240 ]; then\n echo \"Unknown error waiting for old snap to be deleted/merged\"\n exit 1\n fi\n loop=$((loop + 1))\ndone\n....\n\n" }, { "path": "pcmk/baremetal.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# We start with 3 (or more, up to 16) nodes running a minimal CentOS 6\n#\n# Tasks to be performed include:\n# - setting up the required repositories from which to download Openstack and the HA-Addon\n# - disabling firewalls and SElinux. This is a necessary evil until the proper policies can be written.\n# - creating network bridges for use by VMs hosting OpenStack services\n# - normalizing network interface names\n# - fixing multicast\n# - removing /home and making the root partition as large as possible to maximumize the amount of space available to openstack\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_domain\nPHD_VAR_network_internal\nPHD_VAR_network_nic_external\nPHD_VAR_network_nic_internal\nPHD_VAR_network_named_forwarders\nPHD_VAR_rpm_download\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 9\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\n\nyum install -y http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm wget\n\nrhos-release 7\n\nwget -O /etc/yum.repos.d/test.repo http://www.kronosnet.org/testrepo/test.repo\n\nyum clean all\nyum update -y\n\n# ntpd conflicts with chrony\nyum erase -y chrony\nrm -f /etc/chrony*\n\nyum install -y pacemaker fence-agents resource-agents pcs libvirt qemu-kvm bind-utils net-tools tcpdump ntp ntpdate sos nfs-utils\n\n# The cluster shouldn't need NTP configured, but without it the\n# network goes bye-bye when using DHCP\n#\n# Must point to clock.redhat.com to work internally\nsed -i s/^server.*// /etc/ntp.conf\necho \"server $PHD_VAR_network_clock iburst\" >> /etc/ntp.conf\necho $PHD_VAR_network_clock > /etc/ntp/step-tickers\n\n#sync_to_hardware clock\necho \"SYNC_HWCLOCK=yes\" >> /etc/sysconfig/ntpdate\n\nsystemctl enable ntpdate\n\nsystemctl enable ntpd\n\nsed -i -e 's/=enforcing/=disabled/g' /etc/sysconfig/selinux\nsed -i -e 's/=enforcing/=disabled/g' /etc/selinux/config\n\nsystemctl disable firewalld\n\nsystemctl enable libvirtd\nsystemctl start libvirtd\nvirsh net-destroy default\nvirsh net-undefine default\n\nlastoct=\"$(hostname -s | sed -e 's#^[a-z]*-##g' -e 's#^0*##g')\"\n\ncat > /etc/sysconfig/network-scripts/ifcfg-ext0 << EOF\nDEVICE=ext0\nNAME=ext0\nTYPE=Bridge\nBOOTPROTO=dhcp\nONBOOT=yes\nIPV6INIT=yes\nIPV6_AUTOCONF=yes\nEOF\n\ncat > /etc/sysconfig/network-scripts/ifcfg-vmnet0 << EOF\nDEVICE=vmnet0\nNAME=vmnet0\nTYPE=Bridge\nBOOTPROTO=static\nONBOOT=yes\nIPV6INIT=yes\nIPV6_AUTOCONF=yes\nIPADDR=${PHD_VAR_network_internal}.$lastoct\nNETMASK=255.255.255.0\nNETWORK=${PHD_VAR_network_internal}.0\nEOF\n\nfor device in `ls -1 /sys/class/net`; do\n case ${PHD_VAR_network_nic_external} in\n \t *${device}*) \n\t cat > /etc/sysconfig/network-scripts/ifcfg-${device} << EOF\nDEVICE=${device}\nBOOTPROTO=none\nONBOOT=yes\nBRIDGE=ext0\nNAME=${device}\nEOF\n\t;;\n esac\n case ${PHD_VAR_network_nic_internal} in\n \t *${device}*) \n\t cat > /etc/sysconfig/network-scripts/ifcfg-${device} << EOF\nDEVICE=${device}\nBOOTPROTO=none\nONBOOT=yes\nBRIDGE=vmnet0\nNAME=${device}\nEOF\n\t;;\n esac\ndone\n\nif grep -q ip_forward /etc/sysctl.conf; then\n sed -i -e 's#ip_forward.*#ip_forward = 1#g' /etc/sysctl.conf\nelse\n echo \"net.ipv4.ip_forward = 1\" >> /etc/sysctl.conf\nfi\n\necho \"echo 1 > /sys/class/net/ext0/bridge/multicast_querier\" >> /etc/rc.d/rc.local\necho \"echo 1 > /sys/class/net/vmnet0/bridge/multicast_querier\" >> /etc/rc.d/rc.local\nchmod +x /etc/rc.d/rc.local\n\n# Turn off auto-generation of resolv.conf so we can override it \n# make a backup that we will need for gateway scenario otherwise\n# we have a catch 22\nrm -f /etc/resolv.conf.backup\ncp /etc/resolv.conf /etc/resolv.conf.backup\necho PEERDNS=no >> /etc/sysconfig/network-scripts/ifcfg-ext0\necho search vmnet.${PHD_VAR_network_domain} ${PHD_VAR_network_domain} > /etc/resolv.conf\necho nameserver ${PHD_VAR_network_internal}.1 >> /etc/resolv.conf\n\n# get rid of /home from beaker\nsed -i -e 's#.*home.*##g' /etc/fstab\numount /home\n\n# remove home lv\nlvremove -f /dev/mapper/*home\n\n# expand root lv to 50% vg\nlvresize -f -l50%VG /dev/mapper/*root\n\n# expand root fs\nxfs_growfs /dev/mapper/*-root\n\n# create the snapshot\nlvcreate -s -n baremetal_snap -l100%FREE /dev/mapper/*root\n\n# regenerate initramfs to include dm-snapshot modules/utils\nfor i in $(ls /boot/vmlinuz-*x86*); do\n\tver=$(basename $i | sed -e 's#vmlinuz-##g')\n\tdracut -f --kver $ver\ndone\n\n....\n\n# Implied by the reboot below\n#target=all\n#....\n#service network restart\n#/etc/rc.local\n#....\n\ntarget=local\n....\n# Reboot each node and wait for it to return\n\n# disable set -e when calling phd_cmd_* because\n# phd doesn't manage all return codes properly\nset +e\nfor node in $(echo $PHD_ENV_nodes); do\n phd_cmd_exec \"reboot > /dev/null 2>&1\" \"$node\"\n phd_wait_connection 2400 $node || exit 1\ndone\n....\n" }, { "path": "pcmk/basic-cluster.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - installing the cluster software\n# - enabling the pcs daemon to allow remote management\n# - setting a password for the hacluster user for use with pcs\n# - authenticating to pcs on the other hosts with the hacluster user and password\n# - creating and starting the cluster\n# - configuring fencing using the multicast addresses specified for fence_virt on the bare metal hosts \n\n#################################\n# Scenario Requirements Section #\n#################################\n\n= VARIABLES =\n\nPHD_VAR_env_password\nPHD_VAR_network_clock\nPHD_VAR_network_internal\nPHD_VAR_secrets_fence_xvm\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\n\nyum install -y http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm\n\nrhos-release 7\n\nwget -O /etc/yum.repos.d/test.repo http://www.kronosnet.org/testrepo/test.repo\n\nyum update -y\n\n# install the packages\nyum install -y pcs pacemaker corosync fence-agents-all resource-agents\n\n# enable pcsd\nsystemctl enable pcsd\nsystemctl start pcsd\n\nsystemctl disable firewalld\nsystemctl stop firewalld\n\n# The cluster shouldn't need NTP configured, but without it the\n# network goes bye-bye when using DHCP\n#\n# Must point to clock.redhat.com to work internally\nsed -i s/^server.*// /etc/ntp.conf\necho \"server $PHD_VAR_network_clock iburst\" >> /etc/ntp.conf\necho $PHD_VAR_network_clock > /etc/ntp/step-tickers\n\n#sync_to_hardware clock\necho \"SYNC_HWCLOCK=yes\" >> /etc/sysconfig/ntpdate\n\nsystemctl enable ntpdate\nsystemctl start ntpdate\n\nsystemctl enable ntpd\nsystemctl start ntpd\n\n# set a password for hacluster user. password should be the same on all nodes\necho ${PHD_VAR_env_password} | passwd --stdin hacluster\n\n# Now mount /srv so that we can use $PHD_VAR_osp_configdir further down\n\nif grep -q srv /etc/fstab; then \n echo /srv is already mounted; \nelse\n mkdir -p /srv\n echo \"${PHD_VAR_network_internal}.1:/srv /srv nfs defaults,v3 0 0\" >> /etc/fstab\n mount /srv\nfi\n\n# Set up the authkey\nmkdir -p /etc/cluster\necho ${PHD_VAR_secrets_fence_xvm} > /etc/cluster/fence_xvm.key\n\n....\n\ntarget=$PHD_ENV_nodes1\n....\nshort_nodes=\"\"\nfor node in $PHD_ENV_nodes; do\n short_nodes=\"${short_nodes} $(echo ${node} | sed s/\\\\..*//g)\"\ndone\n\n# autheticate nodes, requires all nodes to have pcsd up and running \n# the -p option is used to give the password on command line and make it easier to script\npcs cluster auth $short_nodes -u hacluster -p ${PHD_VAR_env_password} --force\n\n# Construct and start the cluster\n# The cluster needs a unique name, base it on the first node's name\npcs cluster setup --force --name $(echo $PHD_ENV_nodes1 | sed -e s/[0-9]//g -e s/\\\\..*//g ) ${short_nodes}\npcs cluster enable --all\npcs cluster start --all\n\n# Give the cluster a moment to come up\nsleep 5\n\n# Assumes bare metal nodes are:\n# - named with a trailing offset\n# - numbered sequentially\n# - the first node is a proxy\n# - total of three nodes\n# - are configured with fence-virtd (see virt-hosts.scenario)\n# \n# If any of the above is not true, change the fencing devices below\n\npcs stonith create fence1 fence_xvm multicast_address=225.0.0.2\npcs stonith create fence2 fence_xvm multicast_address=225.0.0.3\npcs stonith create fence3 fence_xvm multicast_address=225.0.0.4\n\n# For clones this is not so important\n#\n# However we really don't wan't VIPs moving around as it can cause\n# fencing to fail (eg. if vip-db is stopped, then keystone won't\n# function and we can't confirm if nova recognized the compute node is\n# down)\npcs resource defaults resource-stickiness=INFINITY\n\n....\n" }, { "path": "pcmk/beaker.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# We start with 3 (or more, up to 16) nodes running a minimal CentOS 6\n#\n# Tasks to be performed include:\n# - setting up the required repositories from which to download Openstack and the HA-Addon\n# - disabling firewalls and SElinux. This is a necessary evil until the proper policies can be written.\n# - creating network bridges for use by VMs hosting OpenStack services\n# - normalizing network interface names\n# - fixing multicast\n# - removing /home and making the root partition as large as possible to maximumize the amount of space available to openstack\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_domain\nPHD_VAR_beaker_disttree\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 7\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=local\n....\n# Reboot each node and wait for it to return\n# REQUIRES kinit and we don't check for completition yet\nfor node in $(echo $PHD_ENV_nodes); do\n loop=0\n bkr system-provision --distro-tree ${PHD_VAR_beaker_disttree} $node\n while [ \"$(ping $node -c 3 -q | grep 'packets transmitted' | sed -e 's#.*transmitted, ##g' -e 's# received.*##g')\" != 0 ] && \\\n\t [ \"$loop\" -lt 180 ]; do\n\tloop=$((loop + 1))\n\techo \"Waiting for beaker to kick in ($loop)\"\n\tsleep 1\n done\ndone\n\n# disable set -e when calling phd_cmd_* because\n# phd doesn't manage all return codes properly\nset +e\nfor node in $(echo $PHD_ENV_nodes); do\n phd_wait_connection 2400 $node || exit 1\ndone\n....\n" }, { "path": "pcmk/ceilometer-test.sh", "content": ". ${PHD_VAR_env_configdir}/keystonerc_admin\nfor m in storage.objects image network volume instance ; do ceilometer sample-list -m $m | tail -2 ; done\n\n# https://bugzilla.redhat.com/show_bug.cgi?id=1127526#c2 <- for A/A testing\n" }, { "path": "pcmk/ceilometer.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line.\n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_hosts_mongodb\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_internal\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-ceilometer-api openstack-ceilometer-central openstack-ceilometer-collector openstack-ceilometer-common openstack-ceilometer-alarm python-ceilometer python-ceilometerclient redis\n\n# have redis listen on all IPs\nsed -i \"s/\\s*bind \\(.*\\)$/#bind \\1/\" /etc/redis.conf\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_host vip-keystone\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_port 35357\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_protocol http\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_user ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_password ceilometertest\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf coordination backend_url 'redis://vip-redis:6379'\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://vip-keystone:5000/v2.0\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://${PHD_VAR_network_hosts_mongodb}:27017/ceilometer?replicaSet=ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf database max_retries -1\n\n# keep last 5 days data only (value is in secs). Don't set to retain all data indefinetely.\nopenstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf api host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\n....\n\ntarget=$PHD_ENV_nodes1\n....\n\npcs resource create redis redis wait_last_known_master=true --master meta notify=true ordered=true interleave=true\npcs resource create vip-redis IPaddr2 ip=${PHD_VAR_network_internal}.217\npcs resource create openstack-ceilometer-central systemd:openstack-ceilometer-central --clone interleave=true\npcs resource create openstack-ceilometer-collector systemd:openstack-ceilometer-collector --clone interleave=true\npcs resource create openstack-ceilometer-api systemd:openstack-ceilometer-api --clone interleave=true\npcs resource create delay Delay startdelay=10 --clone interleave=true\npcs resource create openstack-ceilometer-alarm-evaluator systemd:openstack-ceilometer-alarm-evaluator --clone interleave=true\npcs resource create openstack-ceilometer-alarm-notifier systemd:openstack-ceilometer-alarm-notifier --clone interleave=true\npcs resource create openstack-ceilometer-notification systemd:openstack-ceilometer-notification --clone interleave=true\n\npcs constraint order promote redis-master then start vip-redis\npcs constraint colocation add vip-redis with master redis-master\npcs constraint order start vip-redis then openstack-ceilometer-central-clone kind=Optional\npcs constraint order start openstack-ceilometer-central-clone then openstack-ceilometer-collector-clone\npcs constraint order start openstack-ceilometer-collector-clone then openstack-ceilometer-api-clone\npcs constraint colocation add openstack-ceilometer-api-clone with openstack-ceilometer-collector-clone\npcs constraint order start openstack-ceilometer-api-clone then delay-clone\npcs constraint colocation add openstack-ceilometer-delay-clone with openstack-ceilometer-api-clone\npcs constraint order start openstack-ceilometer-delay-clone then openstack-ceilometer-alarm-evaluator-clone\npcs constraint colocation add openstack-ceilometer-alarm-evaluator-clone with openstack-ceilometer-delay-clone\npcs constraint order start openstack-ceilometer-alarm-evaluator-clone then openstack-ceilometer-alarm-notifier-clone\npcs constraint colocation add openstack-ceilometer-alarm-notifier-clone with openstack-ceilometer-alarm-evaluator-clone\npcs constraint order start openstack-ceilometer-alarm-notifier-clone then openstack-ceilometer-notification-clone\npcs constraint colocation add openstack-ceilometer-notification-clone with openstack-ceilometer-alarm-notifier-clone\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n pcs constraint order start mongod-clone then openstack-ceilometer-central-clone\n pcs constraint order start openstack-keystone-clone then openstack-ceilometer-central-clone\nfi\n....\n" }, { "path": "pcmk/cinder-test.sh", "content": ". ${PHD_VAR_env_configdir}/keystonerc_admin\n\nopenstack volume list\nopenstack volume create --size 10 test-volume\nopenstack volume list\nopenstack volume delete test-volume\nopenstack volume list\n" }, { "path": "pcmk/cinder.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_osp_major\nPHD_VAR_osp_configdir\nPHD_VAR_network_internal\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_hosts_rabbitmq\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-cinder openstack-utils python-memcached python-keystonemiddleware python-openstackclient\n\n# Pending ack from cinder team\n#openstack-config --set /etc/cinder/cinder.conf DEFAULT enable_v1_api false\n#openstack-config --set /etc/cinder/cinder.conf DEFAULT enable_v2_api true\n\nopenstack-config --set /etc/cinder/cinder.conf database connection mysql://cinder:cindertest@vip-db/cinder\nopenstack-config --set /etc/cinder/cinder.conf database max_retries -1\n\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone\nopenstack-config --set /etc/cinder/cinder.conf keystone_authtoken identity_uri http://vip-keystone:35357/\nopenstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://vip-keystone:5000/\nopenstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_user cinder\nopenstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_password cindertest\n\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT notification_driver messaging\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder\n\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT glance_host vip-glance\n\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}\n\nopenstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\n\n# rdo${PHD_VAR_osp_major}-cinder isn't the name of a real host or an IP\n# Its the name which we should advertise ourselves as and for A/P it should be the same everywhere\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT host rdo${PHD_VAR_osp_major}-cinder\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfs_exports\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_sparsed_volumes true\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_mount_options v3\n\nopenstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.nfs.NfsDriver\n\n# NOTE: this config section is to enable and configure the NFS cinder driver.\n\n# Create the directory on the server\nmkdir -p $PHD_VAR_osp_configdir/cinder\n\nchown -R cinder:cinder $PHD_VAR_osp_configdir/cinder\n\ncat > /etc/cinder/nfs_exports << EOF\n${PHD_VAR_network_internal}.1:$PHD_VAR_osp_configdir/cinder\nEOF\n\nchown root:cinder /etc/cinder/nfs_exports\nchmod 0640 /etc/cinder/nfs_exports\n\n....\n\n\ntarget=$PHD_ENV_nodes1\n....\nsu cinder -s /bin/sh -c \"cinder-manage db sync\"\n\n# create services in pacemaker\npcs resource create openstack-cinder-api systemd:openstack-cinder-api --clone interleave=true\npcs resource create openstack-cinder-scheduler systemd:openstack-cinder-scheduler --clone interleave=true\n\n# Volume must be A/P for now. See https://bugzilla.redhat.com/show_bug.cgi?id=1193229\npcs resource create openstack-cinder-volume systemd:openstack-cinder-volume\n\npcs constraint order start openstack-cinder-api-clone then openstack-cinder-scheduler-clone\npcs constraint colocation add openstack-cinder-scheduler-clone with openstack-cinder-api-clone\npcs constraint order start openstack-cinder-scheduler-clone then openstack-cinder-volume\npcs constraint colocation add openstack-cinder-volume with openstack-cinder-scheduler-clone\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n pcs constraint order start openstack-keystone-clone then openstack-cinder-api-clone\nfi\n....\n\n" }, { "path": "pcmk/compute-cluster.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n= VARIABLES =\n\nPHD_VAR_env_password\nPHD_VAR_network_internal\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\n# make sure we are not running on a controller node\npcs resource show swift-fs > /dev/null 2>&1\nif [ $? != 0 ]; then\n\n pcs resource show computenode > /dev/null 2>&1\n if [ $? = 0 ]; then\n\texit 0\n fi\n\n # We must be either the first node to run, or configuring\n # single-node clusters for a segregated deployment\n\n pcs cluster auth $(hostname -s) -u hacluster -p ${PHD_VAR_env_password} --force\n\n pcs cluster setup --name $(hostname -s)-compute $(hostname -s)\n pcs cluster enable --all\n pcs cluster start --all\n\n sleep 30\n\n # hack!! IPMI does not work from localhost to localhost on those test nodes!. Use \n # proper fencing settings!\n # TODO: Add SBD support\n pcs property set stonith-enabled=false\n\n pcs resource create nova-compute-fs Filesystem device=\"${PHD_VAR_network_internal}.1:$PHD_VAR_osp_configdir/instances\" directory=\"/var/lib/nova/instances\" fstype=\"nfs\" options=\"v3\" op start timeout=240 --group computenode\n pcs resource create neutron-openvswitch-agent systemd:neutron-openvswitch-agent --group computenode\n pcs resource create libvirtd systemd:libvirtd --group computenode\n pcs resource create ceilometer-compute systemd:openstack-ceilometer-compute --group computenode\n pcs resource create nova-compute systemd:openstack-nova-compute --group computenode\nfi\n\n....\n" }, { "path": "pcmk/compute-common.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n= VARIABLES =\n\nPHD_VAR_env_password\nPHD_VAR_network_domain\nPHD_VAR_network_hosts_gateway\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_hosts_mongodb\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_internal\nPHD_VAR_osp_configdir\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-nova-compute openstack-utils python-cinder openstack-neutron-openvswitch openstack-ceilometer-compute python-memcached openstack-neutron\n\n# we will use this one as instance shared storage\nmkdir -p ${PHD_VAR_osp_configdir}/instances\nchown nova:nova ${PHD_VAR_osp_configdir}/instances\n\nsystemctl disable firewalld\nsystemctl stop firewalld\n \nsystemctl enable pcsd\nsystemctl start pcsd\n \necho ${PHD_VAR_env_password} | passwd --stdin hacluster\n\nsystemctl enable openvswitch\nsystemctl start openvswitch\n\novs-vsctl add-br br-int\n\nsystemctl stop libvirtd\nsystemctl disable libvirtd\n\n# NOTE: vmnet is the interface connected to the internal network.\nopenstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address $(ip addr show dev vmnet0 scope global | grep inet | sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 0.0.0.0\n\n# NOTE: same consideration as nova configuration applies here. They need to match.\nopenstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://${PHD_VAR_network_hosts_gateway}.${PHD_VAR_network_domain}:6080/vnc_auto.html\n\nopenstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@vip-db/nova\nopenstack-config --set /etc/nova/nova.conf database max_retries -1\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\n\n# FIX ME: nova doesn't like hostnames anymore?\n# openstack-config --set /etc/nova/nova.conf DEFAULT metadata_host vip-nova\nopenstack-config --set /etc/nova/nova.conf DEFAULT metadata_host ${PHD_VAR_network_internal}.210\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 0.0.0.0\nopenstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen_port 8775\nopenstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True\nopenstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest\n\nopenstack-config --set /etc/nova/nova.conf glance host vip-glance\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API\nopenstack-config --set /etc/nova/nova.conf neutron url http://vip-neutron:9696/\nopenstack-config --set /etc/nova/nova.conf neutron admin_tenant_name services\nopenstack-config --set /etc/nova/nova.conf neutron admin_username neutron\nopenstack-config --set /etc/nova/nova.conf neutron admin_password neutrontest\nopenstack-config --set /etc/nova/nova.conf neutron admin_auth_url http://vip-keystone:35357/v2.0\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT libvirt_vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron\n\nopenstack-config --set /etc/nova/nova.conf conductor use_local false\n\n# REQUIRED FOR A/A scheduler\nopenstack-config --set /etc/nova/nova.conf DEFAULT scheduler_host_subset_size 30\n\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_host vip-keystone\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_tenant_name services\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_user compute\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_password novatest\n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password neutrontest\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://vip-keystone:35357/v2.0/\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken identity_uri http://vip-keystone:5000/\n\nopenstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier\n\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent tunnel_types vxlan\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent vxlan_udp_port 4789\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs enable_tunneling True\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_id_ranges 1:1000\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tenant_network_type vxlan\n\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs integration_bridge br-int\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_bridge br-tun\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs local_ip $(ip addr show dev vmnet0 scope global | grep inet | sed -e 's#.*inet ##g' -e 's#/.*##g')\n\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver\n\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent l2_population False\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit True\nopenstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit_period hour\nopenstack-config --set /etc/nova/nova.conf DEFAULT notify_on_state_change vm_and_task_state\nopenstack-config --set /etc/nova/nova.conf DEFAULT notification_driver nova.openstack.common.notifier.rpc_notifier\nsed -i -e 's#nova.openstack.common.notifier.rpc_notifier#nova.openstack.common.notifier.rpc_notifier\\nnotification_driver = ceilometer.compute.nova_notifier#g' /etc/nova/nova.conf\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_host vip-keystone\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_port 35357\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_protocol http\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_user ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_password ceilometertest\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://vip-keystone:5000/v2.0\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services\nopenstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest\n\nopenstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://${PHD_VAR_network_hosts_mongodb}:27017/ceilometer?replicaSet=ceilometer\nopenstack-config --set /etc/ceilometer/ceilometer.conf database connection max_retries -1\n\n# keep last 5 days data only (value is in secs)\nopenstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000\n....\n" }, { "path": "pcmk/compute-managed.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_osp_configdir\nPHD_VAR_network_domain\nPHD_VAR_network_internal\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\n\nif [ $PHD_VAR_deployment = segregated ]; then\n echo \"We don't document managed compute nodes in a segregated environment yet\"\n\n # Certainly none of the location constraints would work and the\n # resource-discovery options are mostly redundant\n exit 1\nfi\n\nyum install -y pacemaker-remote resource-agents pcs\n\nif [ ! -e $PHD_VAR_osp_configdir/pcmk-authkey ]; then\n dd if=/dev/urandom of=$PHD_VAR_osp_configdir/pcmk-authkey bs=4096 count=1\nfi\n\nmkdir -p /etc/pacemaker\ncp $PHD_VAR_osp_configdir/pcmk-authkey /etc/pacemaker/authkey\n\nif [ -z \"$(pidof pacemakerd)\" ]; then\n chkconfig pacemaker_remote on\n service pacemaker_remote start\nfi\n....\n" }, { "path": "pcmk/controller-managed.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_osp_configdir\nPHD_VAR_network_domain\nPHD_VAR_network_internal\nPHD_VAR_network_hosts_gateway\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nif [ $PHD_VAR_deployment = segregated ]; then\n echo \"We don't document managed compute nodes in a segregated environment yet\"\n\n # Certainly none of the location constraints would work and the\n # resource-discovery options are mostly redundant\n\n exit 1\nfi\n\nmkdir -p /etc/pacemaker\ncp $PHD_VAR_osp_configdir/pcmk-authkey /etc/pacemaker/authkey\n....\n\ntarget=$PHD_ENV_nodes1\n....\n\n# add NovaEvacuate. It must be A/P and that is perfectly acceptable\n# to avoid need of a cluster wide locking\npcs resource create nova-evacuate ocf:openstack:NovaEvacuate auth_url=http://vip-keystone:35357/v2.0/ username=admin password=keystonetest tenant_name=admin\n\n# without any of those services, nova-evacuate is useless\n# later we also add a order start on nova-compute (after -compute is defined)\nfor i in vip-glance vip-cinder vip-neutron vip-nova vip-db vip-rabbitmq vip-keystone cinder-volume; do\n pcs constraint order start $i then nova-evacuate\ndone\n\nfor i in glance-api-clone neutron-metadata-agent-clone nova-conductor-clone; do\n pcs constraint order start $i then nova-evacuate require-all=false\ndone\n\n# Take down the ODP control plane\npcs resource disable openstack-keystone --wait=240\n\n# Take advantage of the fact that control nodes will already be part of the cluster\n# At this step, we need to teach the cluster about the compute nodes\n#\n# This requires running commands on the cluster based on the names of the compute nodes\n\ncontrollers=$(cibadmin -Q -o nodes | grep uname | sed s/.*uname..// | awk -F\\\" '{print $1}')\n\nfor controller in ${controllers}; do\n pcs property set --node ${controller} osprole=controller\ndone\n\n# Force services to run only on nodes with osprole = controller\n#\n# Importantly it also tells Pacemaker not to even look for the services on other\n# nodes. This helps reduce noise and collisions with services that fill the same role\n# on compute nodes.\n\nstonithdevs=$(pcs stonith | awk '{print $1}')\n\nfor i in $(cibadmin -Q --xpath //primitive --node-path | tr ' ' '\\n' | awk -F \"id='\" '{print $2}' | awk -F \"'\" '{print $1}' | uniq); do\n found=0\n if [ -n \"$stonithdevs\" ]; then\n\tfor x in $stonithdevs; do\n\t if [ $x = $i ]; then\n\t\tfound=1\n\t fi\n\tdone\n fi\n if [ $found = 0 ]; then\n\tpcs constraint location $i rule resource-discovery=exclusive score=0 osprole eq controller\n fi\ndone\n\n# Now (because the compute nodes have roles assigned to them and keystone is\n# stopped) it is safe to define the services that will run on the compute nodes\n\n# neutron-openvswitch-agent\npcs resource create neutron-openvswitch-agent-compute systemd:neutron-openvswitch-agent --clone interleave=true --disabled --force\npcs constraint location neutron-openvswitch-agent-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute \n\npcs constraint order start neutron-server-clone then neutron-openvswitch-agent-compute-clone require-all=false\n\n# libvirtd\npcs resource create libvirtd-compute systemd:libvirtd --clone interleave=true --disabled --force\npcs constraint location libvirtd-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute\n\npcs constraint order start neutron-openvswitch-agent-compute-clone then libvirtd-compute-clone\npcs constraint colocation add libvirtd-compute-clone with neutron-openvswitch-agent-compute-clone\n\n# openstack-ceilometer-compute\npcs resource create ceilometer-compute systemd:openstack-ceilometer-compute --clone interleave=true --disabled --force\npcs constraint location ceilometer-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute\n\npcs constraint order start ceilometer-notification-clone then ceilometer-compute-clone require-all=false\n\npcs constraint order start libvirtd-compute-clone then ceilometer-compute-clone\npcs constraint colocation add ceilometer-compute-clone with libvirtd-compute-clone\n\n# nfs mount for nova-compute shared storage\npcs resource create nova-compute-fs Filesystem device=\"${PHD_VAR_network_internal}.1:$PHD_VAR_osp_configdir/instances\" directory=\"/var/lib/nova/instances\" fstype=\"nfs\" options=\"v3\" op start timeout=240 --clone interleave=true --disabled --force\npcs constraint location nova-compute-fs-clone rule resource-discovery=exclusive score=0 osprole eq compute\n\npcs constraint order start ceilometer-compute-clone then nova-compute-fs-clone\npcs constraint colocation add nova-compute-fs-clone with ceilometer-compute-clone\n\n# nova-compute\npcs resource create nova-compute ocf:openstack:NovaCompute auth_url=http://vip-keystone:35357/v2.0/ username=admin password=keystonetest tenant_name=admin domain=${PHD_VAR_network_domain} op start timeout=300 --clone interleave=true --disabled --force\npcs constraint location nova-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute\n\npcs constraint order start nova-conductor-clone then nova-compute-clone require-all=false\n\npcs constraint order start nova-compute-fs-clone then nova-compute-clone require-all=false\npcs constraint colocation add nova-compute-clone with nova-compute-fs-clone\n\npcs constraint order start nova-compute-clone then nova-evacuate require-all=false\n\ncase ${PHD_VAR_network_hosts_gateway} in\n east-*)\n\tpcs stonith create fence-compute fence_apc ipaddr=east-apc login=apc passwd=apc pcmk_host_map=\"east-01:2;east-02:3;east-03:4;east-04:5;east-05:6;east-06:7;east-07:9;east-08:10;east-09:11;east-10:12;east-11:13;east-12:14;east-13:15;east-14:18;east-15:17;east-16:19;\" --force\n ;;\n mrg-*)\n\tpcs stonith create fence-compute fence_apc_snmp ipaddr=apc-ap7941-l2h3.mgmt.lab.eng.bos.redhat.com power_wait=10 pcmk_host_map=\"mrg-07:10;mrg-08:12;mrg-09:14\"\n ;;\nesac\n\npcs stonith create fence-nova fence_compute auth-url=http://vip-keystone:35357/v2.0/ login=admin passwd=keystonetest tenant-name=admin domain=${PHD_VAR_network_domain} record-only=1 action=off --force\n\n# while this is set in basic.cluster, it looks like OSPd doesn't set it.\npcs resource defaults resource-stickiness=INFINITY\n\n# allow compute nodes to rejoin the cluster automatically\n# 1m might be a bit aggressive tho\npcs property set cluster-recheck-interval=1min\n\nfor node in ${PHD_ENV_nodes}; do\n found=0\n short_node=$(echo ${node} | sed s/\\\\..*//g)\n\n for controller in ${controllers}; do\n\tif [ ${short_node} = ${controller} ]; then\n\t found=1\n\tfi\n done\n\n if [ $found = 0 ]; then\n\t# We only want to execute the following _for_ the compute nodes, not _on_ the compute nodes\n\t# Rather annoying\n\n\tpcs resource create ${short_node} ocf:pacemaker:remote reconnect_interval=60 op monitor interval=20\n\tpcs property set --node ${short_node} osprole=compute\n\tpcs stonith level add 1 ${short_node} fence-compute,fence-nova\n fi\ndone\n\npcs resource enable openstack-keystone\npcs resource enable neutron-openvswitch-agent-compute\npcs resource enable libvirtd-compute\npcs resource enable openstack-ceilometer-compute\npcs resource enable nova-compute-fs\npcs resource enable nova-compute\n\n# cleanup after us\nsleep 60\npcs resource cleanup\n\n....\n" }, { "path": "pcmk/galera-test.sh", "content": "clustercheck\n\n# verify sync is done\nmysql\nSHOW STATUS LIKE 'wsrep%';\nquit\n" }, { "path": "pcmk/galera.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_env_password\nPHD_VAR_network_domain\nPHD_VAR_deployment\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y mariadb-galera-server xinetd rsync\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n # Allowing the proxy to perform health checks on galera while we're initializing it is... problematic. \n pcs resource disable haproxy\nfi\n\ncat > /etc/sysconfig/clustercheck << EOF\nMYSQL_USERNAME=\"clustercheck\"\nMYSQL_PASSWORD=\"${PHD_VAR_env_password}\"\nMYSQL_HOST=\"localhost\"\nMYSQL_PORT=\"3306\"\nEOF\n\n# workaround some old buggy mariadb packages\n# that created log files as root:root and newer\n# packages would fail to start....\nchown mysql:mysql /var/log/mariadb -R\n\nsystemctl start mysqld\n\n# required for clustercheck to work\nmysql -e \"CREATE USER 'clustercheck'@'localhost' IDENTIFIED BY '${PHD_VAR_env_password}';\"\nsystemctl stop mysqld\n\n# Configure galera cluster\n# NOTE: wsrep ssl encryption is strongly recommended and should be enabled\n# on all production deployments. This how-to does NOT display how to \n# configure ssl. The shell expansion points to the internal IP address of the \n# node.\n\ncat > /etc/my.cnf.d/galera.cnf << EOF\n[mysqld]\nskip-name-resolve=1\nbinlog_format=ROW\ndefault-storage-engine=innodb\ninnodb_autoinc_lock_mode=2\ninnodb_locks_unsafe_for_binlog=1\nquery_cache_size=0\nquery_cache_type=0\nbind_address=$(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\n\nwsrep_provider=/usr/lib64/galera/libgalera_smm.so\nwsrep_cluster_name=\"galera_cluster\"\nwsrep_slave_threads=1\nwsrep_certify_nonPK=1\nwsrep_max_ws_rows=131072\nwsrep_max_ws_size=1073741824\nwsrep_debug=0\nwsrep_convert_LOCK_to_trx=0\nwsrep_retry_autocommit=1\nwsrep_auto_increment_control=1\nwsrep_drupal_282555_workaround=0\nwsrep_causal_reads=0\nwsrep_notify_cmd=\nwsrep_sst_method=rsync\nEOF\n\ncat > /etc/xinetd.d/galera-monitor << EOF\nservice galera-monitor\n{\n port = 9200\n disable = no\n socket_type = stream\n protocol = tcp\n wait = no\n user = root\n group = root\n groups = yes\n server = /usr/bin/clustercheck\n type = UNLISTED\n per_source = UNLIMITED\n log_on_success = \n log_on_failure = HOST\n flags = REUSE\n}\nEOF\n\nsystemctl enable xinetd\nsystemctl start xinetd\n\n....\n\ntarget=$PHD_ENV_nodes1\n....\n# node_list must be of the form node1,node2,node3\n#\n# node names must be in the form that the cluster knows them as\n# (ie. no domains) and there can't be a trailing comma (hence the\n# extra weird sed command)\nnode_list=$(echo $PHD_ENV_nodes | sed -e s/.vmnet.${PHD_VAR_network_domain}\\ /,/g -e s/.vmnet.${PHD_VAR_network_domain}//)\npcs resource create galera galera enable_creation=true wsrep_cluster_address=\"gcomm://${node_list}\" additional_parameters='--open-files-limit=16384' meta master-max=3 ordered=true op promote timeout=300s on-fail=block --master\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n # Now we can re-enable the proxy\n pcs resource enable haproxy\nfi\n\n# wait for galera to start and become promoted\nloop=0; while ! clustercheck > /dev/null 2>&1 && [ \"$loop\" -lt 60 ]; do\n\techo waiting galera to be promoted\n\tloop=$((loop + 1))\n\tsleep 5\ndone\n\n# this one can fail depending on who bootstrapped the cluster\nfor node in $PHD_ENV_nodes; do\n\tmysql -e \"DROP USER ''@'${node}';\" || true\n\tmysql -e \"DROP USER 'root'@'${node}';\" || true\ndone\n\ngalera_script=galera.setup\necho \"\" > $galera_script\n\necho \"GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED by 'mysqltest' WITH GRANT OPTION;\" >> $galera_script\n\nfor db in keystone glance cinder neutron nova heat; do\n cat<> $galera_script\nCREATE DATABASE ${db};\nGRANT ALL ON ${db}.* TO '${db}'@'%' IDENTIFIED BY '${db}test';\nEOF\ndone\n\necho \"FLUSH PRIVILEGES;\" >> $galera_script\n#echo \"quit\" >> $galera_script\n\nif [ \"$loop\" -ge 60 ]; then\n\techo Timeout waiting for galera\nelse\n\tmysql mysql < $galera_script\n\tmysqladmin flush-hosts\nfi\n\n....\n" }, { "path": "pcmk/gateway.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed include:\n# - Setting up haproxy to expose nova and horizon\n# - Installing and configuring bind for DNS\n# - Installing and configuring DHCP. For your own sanity, we define rules that allow you to predictably calculate MAC addresses for the guests. \n# - Turning off auto-generation of resolv.conf so we can point to our local DNS server\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_osp_major\nPHD_VAR_components\nPHD_VAR_network_domain\nPHD_VAR_network_internal\nPHD_VAR_network_nic_base\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=$PHD_ENV_nodes1\n....\n# temporary restore original resolv.conf\nif [ -e /etc/resolv.conf.backup ]; then\n rm -f /etc/resolv.conf.new\n cp /etc/resolv.conf /etc/resolv.conf.new\n rm -f /etc/resolv.conf\n cp /etc/resolv.conf.backup /etc/resolv.conf\nfi\n\necho \"/srv ${PHD_VAR_network_internal}.0/255.255.255.0(rw,sync,no_root_squash,subtree_check,fsid=10)\" > /etc/exports\nsystemctl enable nfs-server\nsystemctl start nfs-server\n\next_ip=$(host $PHD_ENV_nodes1 | awk '{print $4}')\necho \"ext: $ext_ip\"\niptables -t nat -A POSTROUTING -s ${PHD_VAR_network_internal}.0/24 -o ext0 -j SNAT --to-source ${ext_ip}\necho \"iptables -t nat -A POSTROUTING -s ${PHD_VAR_network_internal}.0/24 -o ext0 -j SNAT --to-source ${ext_ip}\" >> /etc/rc.local\n\nyum install -y haproxy\n\next_ip=$(host $PHD_ENV_nodes1 | awk '{print $4}')\necho \"ext: $ext_ip\"\n\ncat <<-EOF > /etc/haproxy/haproxy.cfg \nglobal\n daemon\ndefaults\n mode tcp\n maxconn 10000\n timeout connect 180s\n timeout client 180s\n timeout server 180s\nfrontend vip-horizon-ext\n bind ${ext_ip}:80\n default_backend vip-horizon-int\nbackend vip-horizon-int\n balance roundrobin\n server vip-horizon ${PHD_VAR_network_internal}.211:80 check inter 10s\nfrontend vip-nova-ext\n bind ${ext_ip}:6080\n default_backend vip-nova-int\nbackend vip-nova-int\n balance roundrobin\n server vip-nova ${PHD_VAR_network_internal}.210:6080 check inter 10s\nEOF\n\nsystemctl enable haproxy\nsystemctl start haproxy\n\nyum install -y bind\nsed -i -e 's#127.0.0.1#any#g' -e 's#::1#any#g' -e 's#allow-query#//allow-query#g' /etc/named.conf \nsed -i -e 's#dnssec-enable yes;#dnssec-enable no;#g' -e 's#dnssec-validation yes;#dnssec-validation no;#g' /etc/named.conf \nif\n ! grep -q forwarders /etc/named.conf\nthen\n sed -i -e 's#recursion yes;#recursion yes;\\n\\n\\tforwarders {\\n\\t\\t10.16.36.29;\\n\\t\\t10.11.5.19;\\n\\t\\t10.5.30.160;\\n\\t};\\n#g' /etc/named.conf\n echo 'include \"/etc/named/lab.zones\";' >> /etc/named.conf\nfi\n\nreverse_dns_net=$(echo ${PHD_VAR_network_internal} | awk -F. '{print $3}').$(echo ${PHD_VAR_network_internal} | awk -F. '{print $2}').$(echo ${PHD_VAR_network_internal} | awk -F. '{print $1}')\n\ncat <<-EOF > /etc/named/lab.zones\nzone \"vmnet.${PHD_VAR_network_domain}\" {\n type master;\n file \"/etc/named/vmnet.${PHD_VAR_network_domain}\";\n};\nzone \"${reverse_dns_net}.in-addr.arpa\" {\n type master;\n file \"/etc/named/${reverse_dns_net}.arpa\";\n};\nEOF\n\ncat <<-EOF > /etc/named/${reverse_dns_net}.arpa \n\\$TTL 86400\n@ IN SOA mrg-01. fdinitto.redhat.com. (\n 2014103001 ; Serial\n 21600 ; Refresh\n 10800 ; Retry\n 1209600 ; Expire\n 3600 ) ; Minimum\n IN NS mrg-01.\nEOF\n\ncat <<-EOF > /etc/named/vmnet.${PHD_VAR_network_domain}\n\\$TTL 86400\n@ IN SOA mrg-01. fdinitto.redhat.com. (\n 2014103001 ; serial\n 21600 ; refresh (24 hours)\n 10800 ; retry (2 hours)\n 1209600 ; expire (1000 hours)\n 3600 ; minimum (2 days)\n )\n IN NS mrg-01.\nipv6-localhost IN A 127.0.0.1\n IN AAAA ::1\nlocalhost IN A 127.0.0.1\n IN AAAA ::1\nEOF\n\nlpc=0\n# definition_nodes evaluates to all the bare metal nodes\nfor h in ${definition_nodes}; do\n lpc=$((lpc + 1))\n echo \"$lpc IN PTR ${h}.vmnet.${PHD_VAR_network_domain}.\" >> /etc/named/${reverse_dns_net}.arpa\n echo \"$h IN A ${PHD_VAR_network_internal}.${lpc}\" >> /etc/named/vmnet.${PHD_VAR_network_domain}\ndone\n\ncat <<-EOF >> /etc/named/${reverse_dns_net}.arpa \n53 IN PTR rdo${PHD_VAR_osp_major}-rhel7-base.vmnet.${PHD_VAR_network_domain}.\n54 IN PTR rdo${PHD_VAR_osp_major}-rhel6-base.vmnet.${PHD_VAR_network_domain}.\nEOF\n\ncat <<-EOF >> /etc/named/vmnet.${PHD_VAR_network_domain}\nrdo${PHD_VAR_osp_major}-rhel7-base IN A ${PHD_VAR_network_internal}.53\nrdo${PHD_VAR_osp_major}-rhel6-base IN A ${PHD_VAR_network_internal}.54\nEOF\n\noct=55\nfor section in ${PHD_VAR_components}; do\n for count in 1 2 3; do\n \techo \"${oct} IN PTR rdo${PHD_VAR_osp_major}-${section}${count}.vmnet.${PHD_VAR_network_domain}.\" >> /etc/named/${reverse_dns_net}.arpa\n\techo \"rdo${PHD_VAR_osp_major}-${section}${count} IN A ${PHD_VAR_network_internal}.${oct}\" >> /etc/named/vmnet.${PHD_VAR_network_domain}\n\toct=$((oct + 1))\n done\ndone\n\noct=200\nfor section in ${PHD_VAR_components}; do\n case $section in \n\tlb|memcache|swift-brick|mongodb)\n\t : No VIP needed for $section hence no DNS also\n\t ;;\n\t*)\n\t echo \"${oct} IN PTR vip-${section}.vmnet.${PHD_VAR_network_domain}.\" >> /etc/named/${reverse_dns_net}.arpa\n\t echo \"vip-${section} IN A ${PHD_VAR_network_internal}.${oct}\" >> /etc/named/vmnet.${PHD_VAR_network_domain}\n\t ;;\n esac\n oct=$((oct + 1))\ndone\n\n# add vip-redis outside of PHD_VAR_components because it should not be load balanced\necho \"${oct} IN PTR vip-redis.vmnet.${PHD_VAR_network_domain}.\" >> /etc/named/${reverse_dns_net}.arpa\necho \"vip-redis IN A ${PHD_VAR_network_internal}.${oct}\" >> /etc/named/vmnet.${PHD_VAR_network_domain}\n\noct=240\nwhile [ $oct -lt 254 ]; do\n echo \"${oct} IN PTR dhcp${oct}.vmnet.${PHD_VAR_network_domain}.\" >> /etc/named/${reverse_dns_net}.arpa\n echo \"dhcp${oct} IN A ${PHD_VAR_network_internal}.${oct}\" >> /etc/named/vmnet.${PHD_VAR_network_domain}\n oct=$((oct + 1))\ndone\n\nchown root:named /etc/named -R\nchmod 750 /etc/named\nchmod 640 /etc/named/*\nsystemctl enable named\nsystemctl start named\n\nyum install -y dhcp\n\necho \"DHCDPARGS=vmnet0\" > /etc/sysconfig/dhcpd\ncat <<-EOF > /etc/dhcp/dhcpd.conf \ndefault-lease-time 600;\nmax-lease-time 7200;\nauthoritative;\nlog-facility local7;\nsubnet ${PHD_VAR_network_internal}.0 netmask 255.255.255.0 {\n range ${PHD_VAR_network_internal}.240 ${PHD_VAR_network_internal}.253;\n option domain-name \"vmnet.${PHD_VAR_network_domain}\";\n option domain-search \"vmnet.${PHD_VAR_network_domain}\", \"${PHD_VAR_network_domain}\";\n option domain-name-servers ${PHD_VAR_network_internal}.1;\n option routers ${PHD_VAR_network_internal}.1;\n}\n# 54:52:00: <- BASE\n# 00 <- base 01..09 <- run on mrg-XX\n# 00 eth0 (ext0), 01 eth1 (vmnet0), 02 for nodeX ??\n# 01 <- sequence num for the VM on that host\n\nhost rdo${PHD_VAR_osp_major}-rhel7-base {\n hardware ethernet ${PHD_VAR_network_nic_base}:00:01:02;\n fixed-address rdo${PHD_VAR_osp_major}-rhel7-base.vmnet.${PHD_VAR_network_domain};\n}\nEOF\n\nsequence=16\nfor section in ${PHD_VAR_components}; do\n for count in 1 2 3; do\n offset=$(($count + 1))\n\n cat <<-EOF >> /etc/dhcp/dhcpd.conf \nhost rdo${PHD_VAR_osp_major}-${section}${count} {\n hardware ethernet ${PHD_VAR_network_nic_base}:0${offset}:01:${sequence};\n fixed-address rdo${PHD_VAR_osp_major}-${section}${count}.vmnet.${PHD_VAR_network_domain};\n}\nEOF\n done\n sequence=$((sequence + 1))\ndone\n\nsystemctl enable dhcpd\nsystemctl start dhcpd\n\n# restore resolve.conf now that we can use it\nif [ -e /etc/resolv.conf.new ]; then\n rm -f /etc/resolv.conf\n cp /etc/resolv.conf.new /etc/resolv.conf\n rm -f /etc/resolv.conf.new\nfi\n....\n\ntarget=local\n....\n# Reboot each node and wait for it to return\n\n# disable set -e when calling phd_cmd_* because\n# phd doesn't manage all return codes properly\nset +e\nfor node in $(echo $PHD_ENV_nodes); do\n phd_cmd_exec \"reboot > /dev/null 2>&1\" \"$node\"\n phd_wait_connection 2400 $node || exit 1\ndone\n....\n\ntarget=all\n....\n\nif grep -q srv /etc/exports; then\n echo \"We are on the nfs-server\"\n exit 0\nfi\n\nif grep -q srv /etc/fstab; then\n echo /srv is already mounted;\nelse\n mkdir -p /srv\n echo \"${PHD_VAR_network_internal}.1:/srv /srv nfs defaults,v3 0 0\" >> /etc/fstab\n mount /srv\nfi\n" }, { "path": "pcmk/glance-test.sh", "content": ". ${PHD_VAR_env_configdir}/keystonerc_admin\n\nif [ ! -f ${PHD_VAR_env_configdir}/cirros-0.3.2-x86_64-disk.img ]; then\n\twget -O ${PHD_VAR_env_configdir}/cirros-0.3.2-x86_64-disk.img http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img\nfi\n\nopenstack image create --container-format bare --disk-format qcow2 --public --file ${PHD_VAR_env_configdir}/cirros-0.3.2-x86_64-disk.img cirros\n\nopenstack image list\n" }, { "path": "pcmk/glance.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_osp_configdir\nPHD_VAR_deployment\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\n\ntarget=all\n....\nyum install -y openstack-glance openstack-utils python-openstackclient\n\n# Configure the API service\n\nopenstack-config --set /etc/glance/glance-api.conf database connection mysql://glance:glancetest@vip-db/glance\nopenstack-config --set /etc/glance/glance-api.conf database max_retries -1\n\nopenstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone\n\nopenstack-config --set /etc/glance/glance-api.conf keystone_authtoken identity_uri http://vip-keystone:35357/\nopenstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_uri http://vip-keystone:5000/\nopenstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_user glance\nopenstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_password glancetest\n\nopenstack-config --set /etc/glance/glance-api.conf DEFAULT notification_driver messaging\n\nopenstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\n\nopenstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host vip-glance\nopenstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\n\n# Configure the registry service\n\nopenstack-config --set /etc/glance/glance-registry.conf database connection mysql://glance:glancetest@vip-db/glance\nopenstack-config --set /etc/glance/glance-registry.conf database max_retries -1\n\nopenstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone\n\nopenstack-config --set /etc/glance/glance-registry.conf keystone_authtoken identity_uri http://vip-keystone:35357/\nopenstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_uri http://vip-keystone:5000/\nopenstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_user glance\nopenstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_password glancetest\n\nopenstack-config --set /etc/glance/glance-registry.conf DEFAULT notification_driver messaging\n\nopenstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\n\nopenstack-config --set /etc/glance/glance-registry.conf DEFAULT registry_host vip-glance\nopenstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\n\n# create the NFS share mountpoint on the nfs server\nmkdir -p $PHD_VAR_osp_configdir/glance\n....\n\ntarget=$PHD_ENV_nodes1\n....\n# Now have pacemaker mount the NFS share as service.\npcs resource create glance-fs Filesystem device=\"${PHD_VAR_network_internal}.1:$PHD_VAR_osp_configdir/glance\" directory=\"/var/lib/glance\" fstype=\"nfs\" options=\"v3\" --clone\n\n# wait for glance-fs to be started and running\nsleep 5\n\n# Make sure it's writable\nchown glance:nobody /var/lib/glance\n\n# Now populate the database\nsu glance -s /bin/sh -c \"glance-manage db_sync\"\n\npcs resource create openstack-glance-registry systemd:openstack-glance-registry --clone interleave=true\npcs resource create openstack-glance-api systemd:openstack-glance-api --clone interleave=true\n\npcs constraint order start openstack-glance-fs-clone then openstack-glance-registry-clone\npcs constraint colocation add openstack-glance-registry-clone with openstack-glance-fs-clone\npcs constraint order start openstack-glance-registry-clone then openstack-glance-api-clone\npcs constraint colocation add openstack-glance-api-clone with openstack-glance-registry-clone\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n pcs constraint order start openstack-keystone-clone then openstack-glance-registry-clone\nfi\n....\n\n" }, { "path": "pcmk/ha-collapsed.variables", "content": "# Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD\n# Each scenario file will verify that values have been provided for the variables it requires.\n\n# Deployment types: collapsed | segregated\ndeployment: collapsed\n\nnetwork:\n clock: clock.redhat.com\n domain: lab.bos.redhat.com\n internal: 192.168.124\n named:\n forwarders: 10.16.36.29 10.11.5.19 10.5.30.160\n nic:\n base: 54:52:00\n external: enp1s0 eth0\n internal: enp2s0 eth1\n hosts:\n gateway: east-01\n\n # These services are not able to live behind a proxy, so we must list the hosts explicitly\n # In the segregated case, it is the per-service guests we will create\n # In the collapsed case, it is the members of the single cluster we will create \n mongodb: rdo7-node1,rdo7-node2,rdo7-node3\n memcache: rdo7-node1:11211,rdo7-node2:11211,rdo7-node3:11211\n rabbitmq: rdo7-node1,rdo7-node2,rdo7-node3\n\n# We use a centrally defined list so that segregated guests, DNS, and DHCP are all created consistently\n# Changing this list in any way will probably break the haproxy configuration \n# - ONLY add entries \n# - ALWAYS do so to the end of the list\n# - LOOK for PHD_VAR_components to see places that may be affected\ncomponents: lb db rabbitmq keystone memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer qpid node\n\n# this is RHEL7.1 GA\n# https://beaker.engineering.redhat.com/distros/\n# Use simple search to find release ex: RHEL-7.1\n# click on the GA -> and take the ID from Server install\nbeaker:\n disttree: 69383\n\nrpm:\n download: download.devel.redhat.com\n major: 7\n minor: 1\n #base: rel-eng/latest-RHEL-7/compose/Server/x86_64/os/\n #cluster: rel-eng/latest-RHEL-7/compose/Server/x86_64/os/addons/HighAvailability/\n base: released/RHEL-7/7.1/Server/x86_64/os/\n cluster: released/RHEL-7/7.1/Server/x86_64/os/addons/HighAvailability/\n updates: brewroot/repos/rhel-7.1-z-build/latest/x86_64/\n\nosp:\n configdir: /srv/RDO-7/configs\n major: 7\n\n# generate with openssl rand -hex 10\nsecrets:\n fence_xvm: bcf4e54dcfbfecb9e62c\n keystone_secret: 114a23ae49996a26d916\n swift_prefix: 8f2d8a3e326a078c5edf\n swift_suffix: 8d33d13b18e3eef6d3ca\n horizon_secret: 357a57d6a9a2353ccde5\n\n# Dedicate all cores to the sole VM that we'll create per host\nvm:\n cpus: 4\n ram: 8048\n disk: 25G\n base: rdo7-rhel7-base.img\n key: AAAAB3NzaC1yc2EAAAADAQABAAABAQDHs2qRMxtqEpr7gJygHAn2rSWKUS/FlJ9oLG7cRtzLyhIl+oSrs30KrdzkgsGTZqSEwfKM8f2LGF08x5HbN2cIDc9YhnwHQNnb8qDIXY2UqzpyLUzckctOMSiRSz/qYxeutDYGg/p1lPzPdWQPympFVIoAzCRDhogX26kXQTpKs7uUzEvZCnnzSn2I9ynchKGP3TlOzTaZHqJM4bj5+KqvUTH2ifvX3EgolP/XtIWjW54zhQnlDuS2UsDd8vvB8ZRrgtaFEXhCSivvazE8zMVAOxCFNYjnh+SvV96VB+hEjqQQeDSdhkgC2huHwsAB3Y9XCkyFe6DEfKuQZwLJjlTZ\n\n# I set the password to 'cluster', USE A SAFER ONE\nenv:\n password: cluster\n\n# TODO... \npassword:\n cluster: foo\n keystone: bar\n" }, { "path": "pcmk/ha-segregated.variables", "content": "# Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD\n# Each scenario file will verify that values have been provided for the variables it requires.\n\n# Deployment types: collapsed | segregated\ndeployment: segregated\n\nnetwork:\n domain: lab.bos.redhat.com\n internal: 192.168.124\n named:\n forwarders: 10.16.36.29 10.11.5.19 10.5.30.160\n nic:\n base: 54:52:00\n external: enp1s0 eth0\n internal: enp2s0 eth1\n hosts:\n gateway: east-01\n\n # These services are not able to live behind a proxy, so we must list the hosts explicitly\n # In the segregated case, it is the per-service guests we will create\n # In the collapsed case, it is the members of the single cluster we will create \n mongodb: rhos6-mongodb1,rhos6-mongodb2,rhos6-mongodb3\n memcache: rhos6-memcache1:11211,rhos6-memcache2:11211,rhos6-memcache3:11211\n rabbitmq: rhos6-rabbitmq1,rhos6-rabbitmq2,rhos6-rabbitmq3\n\n# We use a centrally defined list so that segregated guests, DNS, and DHCP are all created consistently\n# Changing this list in any way will probably break the haproxy configuration \n# - ONLY add entries \n# - ALWAYS do so to the end of the list\n# - LOOK for PHD_VAR_components to see places that may be affected\ncomponents: lb db rabbitmq keystone memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer qpid node\n\nrpm:\n download: download.devel.redhat.com\n rhel: 7.1\n osp: 6.0\n# Optional\n# beta: -Beta\n\nvm:\n cpus: 1\n ram: 2048\n disk: 25G\n base: rhos6-rhel7-base.img\n key: AAAAB3NzaC1yc2EAAAADAQABAAABAQDHs2qRMxtqEpr7gJygHAn2rSWKUS/FlJ9oLG7cRtzLyhIl+oSrs30KrdzkgsGTZqSEwfKM8f2LGF08x5HbN2cIDc9YhnwHQNnb8qDIXY2UqzpyLUzckctOMSiRSz/qYxeutDYGg/p1lPzPdWQPympFVIoAzCRDhogX26kXQTpKs7uUzEvZCnnzSn2I9ynchKGP3TlOzTaZHqJM4bj5+KqvUTH2ifvX3EgolP/XtIWjW54zhQnlDuS2UsDd8vvB8ZRrgtaFEXhCSivvazE8zMVAOxCFNYjnh+SvV96VB+hEjqQQeDSdhkgC2huHwsAB3Y9XCkyFe6DEfKuQZwLJjlTZ\n\n# I set the password to 'cluster', USE A SAFER ONE\nenv:\n password: cluster\n configdir: /srv/RDO-6.0/configs\n" }, { "path": "pcmk/hacks.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - installing the cluster software\n# - enabling the pcs daemon to allow remote management\n# - setting a password for the hacluster user for use with pcs\n# - authenticating to pcs on the other hosts with the hacluster user and password\n# - creating and starting the cluster\n# - configuring fencing using the multicast addresses specified for fence_virt on the bare metal hosts \n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\n# hack to set hostnames till we figure out why it doesn't work\n# from kickstart anymore\ntarget=local\n....\nfor node in $(echo $PHD_ENV_nodes); do\n phd_cmd_exec \"hostnamectl set-hostname $node\" \"$node\"\n phd_cmd_exec \"hostname\" \"$node\"\ndone\n....\n" }, { "path": "pcmk/heat-test.sh", "content": "# TEST:\n# Requires a compute node!\n\n. ${PHD_VAR_env_configdir}/keystonerc_admin\n\nnova keypair-add --pub_key ~/.ssh/authorized_keys heat-userkey-test\n\ncat > /root/ha_test.yaml << EOF\nheat_template_version: 2013-05-23\n\ndescription: >\n HA test.\n\nparameters:\n key_name:\n type: string\n description: Name of keypair to assign to servers\n image:\n type: string\n description: Name of image to use for servers\n flavor:\n type: string\n description: Flavor to use for servers\n private_net_id:\n type: string\n description: ID of private network into which servers get deployed\n private_subnet_id:\n type: string\n description: ID of private sub network into which servers get deployed\n\nresources:\n server1:\n type: OS::Nova::Server\n properties:\n name: Server1\n image: { get_param: image }\n flavor: { get_param: flavor }\n key_name: { get_param: key_name }\n networks:\n - port: { get_resource: server1_port }\n\n server1_port:\n type: OS::Neutron::Port\n properties:\n network_id: { get_param: private_net_id }\n fixed_ips:\n - subnet_id: { get_param: private_subnet_id }\n\noutputs:\n server1_private_ip:\n description: IP address of server1 in private network\n value: { get_attr: [ server1, first_address ] }\nEOF\n\nprivatenetid=$(neutron net-list |grep internal_lan | awk '{print $2}')\nprivatesubnetid=$(neutron subnet-list |grep internal_subnet|awk '{print $2}')\n\nheat stack-create testtest --template-file=/root/ha_test.yaml --parameters=\"key_name=heat-userkey-test;image=cirros;flavor=m1.large;private_net_id=$privatenetid;private_subnet_id=$privatesubnetid\"\n\nheat stack-list\n\nheat stack-delete testtest\n\n" }, { "path": "pcmk/heat.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_hosts_rabbitmq\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-heat-engine openstack-heat-api openstack-heat-api-cfn openstack-heat-api-cloudwatch python-heatclient openstack-utils python-glanceclient\n\nopenstack-config --set /etc/heat/heat.conf database connection mysql://heat:heattest@vip-db/heat\nopenstack-config --set /etc/heat/heat.conf database database max_retries -1\n\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken admin_user heat\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken admin_password heattest\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken service_host vip-keystone\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken auth_host vip-keystone\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken auth_uri http://vip-keystone:35357/v2.0\nopenstack-config --set /etc/heat/heat.conf keystone_authtoken keystone_ec2_uri http://vip-keystone:35357/v2.0\nopenstack-config --set /etc/heat/heat.conf ec2authtoken auth_uri http://vip-keystone:5000/v2.0\n\nopenstack-config --set /etc/heat/heat.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\n\n\nopenstack-config --set /etc/heat/heat.conf heat_api bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/heat/heat.conf heat_api_cfn bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/heat/heat.conf heat_api_cloudwatch bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/heat/heat.conf DEFAULT heat_metadata_server_url vip-heat:8000\nopenstack-config --set /etc/heat/heat.conf DEFAULT heat_waitcondition_server_url vip-heat:8000/v1/waitcondition\nopenstack-config --set /etc/heat/heat.conf DEFAULT heat_watch_server_url vip-heat:8003\n\nopenstack-config --set /etc/heat/heat.conf DEFAULT rpc_backend heat.openstack.common.rpc.impl_kombu\n\nopenstack-config --set /etc/heat/heat.conf DEFAULT notification_driver heat.openstack.common.notifier.rpc_notifier\n\n# disable CWLiteAlarm that is incompatible with A/A\nopenstack-config --set /etc/heat/heat.conf DEFAULT enable_cloud_watch_lite false\n\n....\n\ntarget=$PHD_ENV_nodes1\n....\nsu heat -s /bin/sh -c \"heat-manage db_sync\"\n\npcs resource create openstack-heat-api systemd:openstack-heat-api --clone interleave=true\npcs resource create openstack-heat-api-cfn systemd:openstack-heat-api-cfn --clone interleave=true\npcs resource create openstack-heat-api-cloudwatch systemd:openstack-heat-api-cloudwatch --clone interleave=true\npcs resource create openstack-heat-engine systemd:openstack-heat-engine --clone interleave=true\n\npcs constraint order start openstack-heat-api-clone then openstack-heat-api-cfn-clone\npcs constraint colocation add openstack-heat-api-cfn-clone with openstack-heat-api-clone\npcs constraint order start openstack-heat-api-cfn-clone then openstack-heat-api-cloudwatch-clone\npcs constraint colocation add openstack-heat-api-cloudwatch-clone with openstack-heat-api-cfn-clone\npcs constraint order start openstack-heat-api-cloudwatch-clone then openstack-heat-engine-clone\npcs constraint colocation add openstack-heat-engine-clone with openstack-heat-api-cloudwatch-clone\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n pcs constraint order start openstack-ceilometer-notification-clone then openstack-heat-api-clone\nfi\n....\n\n" }, { "path": "pcmk/horizon-test.sh", "content": "\nIt should be possible now to login to http://vip-horizon/dashboard with admin account.\nNote that it is still not possible to deploy instances since compute nodes are not attached.\n" }, { "path": "pcmk/horizon.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_secrets_horizon_secret\nPHD_VAR_network_hosts_memcache\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y mod_wsgi httpd mod_ssl openstack-dashboard\n\n# NOTE this is a rather scary sed and replace operation to configure horizon\n# in one shot, scriptable way.\n# Keypoints:\n# set ALLOWED_HOSTS to access the web service.\n# BE AWARE that this command will allow access from everywhere!\n# connection CACHES to memcacehed\n# connect with keystone for authentication\n# fix a LOCAL_PATH to point to the correct location.\n\nhorizonememcachenodes=$(echo ${PHD_VAR_network_hosts_memcache} | sed -e \"s#,#', '#g\" -e \"s#^#[ '#g\" -e \"s#\\$#', ]#g\")\n\nsed -i \\\n\t-e \"s#ALLOWED_HOSTS.*#ALLOWED_HOSTS = ['*',]#g\" \\\n\t-e \"s#^CACHES#SESSION_ENGINE = 'django.contrib.sessions.backends.cache'\\nCACHES#g#\" \\\n\t-e \"s#locmem.LocMemCache'#memcached.MemcachedCache',\\n\\t'LOCATION' : $horizonememcachenodes#g\" \\\n\t-e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = \"vip-keystone\"#g' \\\n\t-e \"s#^LOCAL_PATH.*#LOCAL_PATH = '/var/lib/openstack-dashboard'#g\" \\\n\t-e \"s#SECRET_KEY.*#SECRET_KEY = '${PHD_VAR_secrets_horizon_secret}'#g#\" \\\n\t/etc/openstack-dashboard/local_settings\n\n# workaround buggy packages\necho \"COMPRESS_OFFLINE = True\" >> /etc/openstack-dashboard/local_settings \npython /usr/share/openstack-dashboard/manage.py compress\n\n# NOTE: fix apache config to listen only on a given interface (internal)\nsed -i -e 's/^Listen.*/Listen '$(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')':80/g' /etc/httpd/conf/httpd.conf \n\n# NOTE: enable server-status. this is required by pacemaker to verify apache is \n# responding. Only allow from localhost.\ncat > /etc/httpd/conf.d/server-status.conf << EOF\n\n SetHandler server-status\n Order deny,allow\n Deny from all\n Allow from localhost\n\nEOF\n\n....\n\n\ntarget=$PHD_ENV_nodes1\n....\npcs resource create httpd apache --clone interleave=true\n....\n" }, { "path": "pcmk/keystone-test.sh", "content": "# TEST (might require logout/login to reset the environmet that was set before \n# during initial bootstrap)\n\nunset SERVICE_TOKEN\nunset SERVICE_ENDPOINT\n. ${PHD_VAR_env_configdir}/keystonerc_user\nopenstack user show demo\n. ${PHD_VAR_env_configdir}/keystonerc_admin\nopenstack user list\n" }, { "path": "pcmk/keystone.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_osp_configdir\nPHD_VAR_secrets_keystone_secret\nPHD_VAR_network_internal\nPHD_VAR_network_hosts_rabbitmq\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-keystone openstack-utils python-openstackclient\n\nmkdir -p ${PHD_VAR_osp_configdir}\n\nopenstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token ${PHD_VAR_secrets_keystone_secret}\n\nopenstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\n\n# Define the API endpoints. Be careful with replacing vip-keystone and shell escapes.\n\nopenstack-config --set /etc/keystone/keystone.conf DEFAULT admin_endpoint 'http://vip-keystone:%(admin_port)s/'\nopenstack-config --set /etc/keystone/keystone.conf DEFAULT public_endpoint 'http://vip-keystone:%(public_port)s/'\n\n# Configure access to galera. Note that several entries in here are dependent on\n# what has been configured before. 'keystone' user, 'keystonetest' password, \n# vip-db.\n\nopenstack-config --set /etc/keystone/keystone.conf database connection mysql://keystone:keystonetest@vip-db/keystone\n\n# Mare sure to retry connection to the DB if the DB is not available immediately at \n# service startup.\n\nopenstack-config --set /etc/keystone/keystone.conf database max_retries -1\n\n# Make sure the API service is listening on the internal IP addresses only.\n# Once again those shell expansions only work for my specific environment.\n\n# phase3: those are obsoleted by mod_wsgi and apache\nopenstack-config --set /etc/keystone/keystone.conf eventlet_server public_bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/keystone/keystone.conf eventlet_server admin_bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\n\n# workaround for buggy packaging (ayoung is informed)\nopenstack-config --set /etc/keystone/keystone.conf token driver keystone.token.persistence.backends.sql.Token\n\nif [ ! -e ${PHD_VAR_osp_configdir}/keystone_ssl.tar ]; then\n keystone-manage pki_setup --keystone-user keystone --keystone-group keystone\n cd /etc/keystone/ssl\n tar cvp -f ${PHD_VAR_osp_configdir}/keystone_ssl.tar *\nfi\n\nmkdir -p /etc/keystone/ssl\ncd /etc/keystone/ssl\ntar xvp -f ${PHD_VAR_osp_configdir}/keystone_ssl.tar\nchown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/\n....\n\n\ntarget=$PHD_ENV_nodes1\n....\nsu keystone -s /bin/sh -c \"keystone-manage -v -d db_sync\"\n\npcs resource create openstack-keystone systemd:openstack-keystone --clone interleave=true\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n\n # In a collapsed environment, we can instruct the cluster to start\n # things in a particular order and require services to be active\n # on the same hosts. We do this with constraints.\n\n pcs constraint order start haproxy-clone then openstack-keystone-clone\n pcs constraint order promote galera-master then openstack-keystone-clone\n pcs constraint order start rabbitmq-clone then openstack-keystone-clone\n pcs constraint order start memcached-clone then openstack-keystone-clone\nfi\n\nexport OS_TOKEN=${PHD_VAR_secrets_keystone_secret}\nexport OS_URL=\"http://vip-keystone:35357/v2.0\"\nexport OS_REGION_NAME=regionOne\n\nwhile ! openstack service list; do\n echo \"Waiting for keystone to be active\"\n sleep 1\ndone\n\nopenstack service create \\\n\t--name=keystone \\\n\t--description=\"Keystone Identity Service\" \\\n\tidentity\n\nopenstack endpoint create \\\n\t--publicurl 'http://vip-keystone:5000/v2.0' \\\n\t--adminurl 'http://vip-keystone:35357/v2.0' \\\n\t--internalurl 'http://vip-keystone:5000/v2.0' \\\n\t--region regionOne \\\n\tkeystone\n\nopenstack user create --password keystonetest admin\nopenstack role create admin\nopenstack project create admin\nopenstack role add --project admin --user admin admin\n\n# Save admin credential in a file. This will be useful many times over the how-to!\n\ncat > ${PHD_VAR_osp_configdir}/keystonerc_admin << EOF\nexport OS_USERNAME=admin \nexport OS_TENANT_NAME=admin\nexport OS_PROJECT_NAME=admin\nexport OS_REGION_NAME=regionOne\nexport OS_PASSWORD=keystonetest\nexport OS_AUTH_URL=http://vip-keystone:35357/v2.0/\nexport PS1='[\\u@\\h \\W(keystone_admin)]\\$ '\nEOF\n\nopenstack user create --password redhat demo\nopenstack role create _member_\nopenstack project create demo\nopenstack role add --project demo --user demo _member_\n\n# Save user credential in a file for testing purposes.\n\ncat > ${PHD_VAR_osp_configdir}/keystonerc_user << EOF\nexport OS_USERNAME=demo\nexport OS_TENANT_NAME=demo\nexport OS_PROJECT_NAME=demo\nexport OS_REGION_NAME=regionOne\nexport OS_PASSWORD=redhat\nexport OS_AUTH_URL=http://vip-keystone:5000/v2.0/\nexport PS1='[\\u@\\h \\W(keystone_user)]\\$ '\nEOF\n\n# create service tenant/project\nopenstack project create --description \"Services Tenant\" services\n\n# glance\nopenstack user create --password glancetest glance\nopenstack role add --project services --user glance admin\nopenstack service create --name=glance --description=\"Glance Image Service\" image\nopenstack endpoint create \\\n\t--publicurl \"http://vip-glance:9292\" \\\n\t--adminurl \"http://vip-glance:9292\" \\\n\t--internalurl \"http://vip-glance:9292\" \\\n\t--region regionOne \\\n\tglance\n\n# cinder\nopenstack user create --password cindertest cinder\nopenstack role add --project services --user cinder admin\nopenstack service create --name=cinder --description=\"Cinder Volume Service\" volume\nopenstack endpoint create \\\n\t--publicurl \"http://vip-cinder:8776/v1/\\$(tenant_id)s\" \\\n\t--adminurl \"http://vip-cinder:8776/v1/\\$(tenant_id)s\" \\\n\t--internalurl \"http://vip-cinder:8776/v1/\\$(tenant_id)s\" \\\n\t--region regionOne \\\n\tcinder\n\nopenstack service create --name=cinderv2 --description=\"OpenStack Block Storage\" volumev2\nopenstack endpoint create \\\n\t--publicurl \"http://vip-cinder:8776/v2/\\$(tenant_id)s\" \\\n\t--adminurl \"http://vip-cinder:8776/v2/\\$(tenant_id)s\" \\\n\t--internalurl \"http://vip-cinder:8776/v2/\\$(tenant_id)s\" \\\n\t--region regionOne \\\n\tcinderv2\n\n# swift\nopenstack user create --password swifttest swift\nopenstack role add --project services --user swift admin\nopenstack service create --name=swift --description=\"Swift Storage Service\" object-store\nopenstack endpoint create \\\n\t--publicurl \"http://vip-swift:8080/v1/AUTH_\\$(tenant_id)s\" \\\n\t--adminurl \"http://vip-swift:8080/v1\" \\\n\t--internalurl \"http://vip-swift:8080/v1/AUTH_\\$(tenant_id)s\" \\\n\t--region regionOne \\\n\tswift\n\n# neutron\nopenstack user create --password neutrontest neutron\nopenstack role add --project services --user neutron admin\nopenstack service create --name=neutron --description=\"OpenStack Networking Service\" network\nopenstack endpoint create \\\n\t--publicurl \"http://vip-neutron:9696\" \\\n\t--adminurl \"http://vip-neutron:9696\" \\\n\t--internalurl \"http://vip-neutron:9696\" \\\n\t--region regionOne \\\n\tneutron\n\n# nova\nopenstack user create --password novatest compute\nopenstack role add --project services --user compute admin\nopenstack service create --name=compute --description=\"OpenStack Compute Service\" compute\nopenstack endpoint create \\\n\t--publicurl \"http://vip-nova:8774/v2/\\$(tenant_id)s\" \\\n\t--adminurl \"http://vip-nova:8774/v2/\\$(tenant_id)s\" \\\n\t--internalurl \"http://vip-nova:8774/v2/\\$(tenant_id)s\" \\\n\t--region regionOne \\\n\tcompute\n\n# heat\nopenstack user create --password heattest heat\nopenstack role add --project services --user heat admin\nopenstack service create --name=heat --description=\"Heat Orchestration Service\" orchestration\nopenstack endpoint create \\\n\t--publicurl \"http://vip-heat:8004/v1/%(tenant_id)s\" \\\n\t--adminurl \"http://vip-heat:8004/v1/%(tenant_id)s\" \\\n\t--internalurl \"http://vip-heat:8004/v1/%(tenant_id)s\" \\\n\t--region regionOne \\\n\theat\nopenstack service create --name=heat-cfn --description=\"Heat CloudFormation Service\" cloudformation\nopenstack endpoint create \\\n\t--publicurl \"http://vip-heat:8000/v1\" \\\n\t--adminurl \"http://vip-heat:8000/v1\" \\\n\t--internalurl \"http://vip-heat:8000/v1\" \\\n\t--region regionOne \\\n\theat-cfn\n\n# ceilometer\nopenstack user create --password ceilometertest ceilometer\nopenstack role add --project services --user ceilometer admin\nopenstack role create ResellerAdmin\nopenstack role add --project services --user ceilometer ResellerAdmin\nopenstack service create --name=ceilometer --description=\"OpenStack Telemetry Service\" metering\nopenstack endpoint create \\\n\t--publicurl \"http://vip-ceilometer:8777\" \\\n\t--adminurl \"http://vip-ceilometer:8777\" \\\n\t--internalurl \"http://vip-ceilometer:8777\" \\\n\t--region regionOne \\\n\tceilometer\n....\n" }, { "path": "pcmk/lb.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n# - Tweaking the IP stack to allow nonlocal binding and adjusting keepalive timings\n# - Configuring haproxy\n# - Adding the virtual IPs to the cluster\n# - Putting haproxy under the cluster's control\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_network_internal\nPHD_VAR_deployment\nPHD_VAR_components\nPHD_VAR_osp_major\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y haproxy\necho net.ipv4.ip_nonlocal_bind=1 >> /etc/sysctl.d/haproxy.conf\necho 1 > /proc/sys/net/ipv4/ip_nonlocal_bind\n# the keepalive settings must be set in *ALL* hosts interacting with rabbitmq.\ncat >/etc/sysctl.d/tcp_keepalive.conf << EOF\nnet.ipv4.tcp_keepalive_intvl = 1\nnet.ipv4.tcp_keepalive_probes = 5\nnet.ipv4.tcp_keepalive_time = 5\nEOF\nsysctl net.ipv4.tcp_keepalive_intvl=1\nsysctl net.ipv4.tcp_keepalive_probes=5\nsysctl net.ipv4.tcp_keepalive_time=5\n\n# HA Proxy defaults\n\ncat > /etc/haproxy/haproxy.cfg << EOF\nglobal\n daemon\ndefaults\n mode tcp\n maxconn 10000\n timeout connect 10s\n timeout client 1m\n timeout server 1m\n timeout check 10s\nEOF\n\n# Special case front-ends\n\ncat >> /etc/haproxy/haproxy.cfg << EOF\nfrontend vip-db\n bind ${PHD_VAR_network_internal}.201:3306\n timeout client 90m\n default_backend db-vms-galera\n\nfrontend vip-qpid\n bind ${PHD_VAR_network_internal}.215:5672\n timeout client 120s\n default_backend qpid-vms\n\nfrontend vip-horizon\n bind ${PHD_VAR_network_internal}.211:80\n timeout client 180s\n cookie SERVERID insert indirect nocache\n default_backend horizon-vms\n\nfrontend vip-ceilometer\n bind ${PHD_VAR_network_internal}.214:8777\n timeout client 90s\n default_backend ceilometer-vms\n\nfrontend vip-rabbitmq\n option clitcpka\n bind ${PHD_VAR_network_internal}.202:5672\n timeout client 900m\n default_backend rabbitmq-vms\nEOF\n\n# nova-metadata needs \"balance roundrobin\" for frontend?\n#db-vms-mariadb:58:3306:90s\n\nmappings=\"\nkeystone-admin:203:64:35357\nkeystone-public:203:64:5000\nglance-api:205:70:9191\nglance-registry:205:70:9292\ncinder:206:73:8776\nswift:208:79:8080\nneutron:209:82:9696\nnova-vnc-novncproxy:210:85:6080\nnova-vnc-xvpvncproxy:210:85:6081\nnova-metadata:210:85:8775\nnova-api:210:85:8774\nhorizon:x:85:80:108s\nheat-cfn:212:91:8000\nheat-cloudw:212:91:8004\nheat-srv:212:91:8004\nceilometer:x:97:8777\n\"\n\nfor mapping in $mappings; do \n server=$(echo $mapping | awk -F: '{print $1}' | awk -F- '{print $1}')\n service=$(echo $mapping | awk -F: '{print $1}')\n src=$(echo $mapping | awk -F: '{print $2}')\n target=$(echo $mapping | awk -F: '{print $3}')\n port=$(echo $mapping | awk -F: '{print $4}')\n timeout=$(echo $mapping | awk -F: '{print $5}')\n\n echo \"Creating mapping for ${server} ${service}\"\n\n if [ ${src} != x ]; then\n\techo \"frontend vip-${service}\" >> /etc/haproxy/haproxy.cfg\n\techo \" bind ${PHD_VAR_network_internal}.${src}:${port}\" >> /etc/haproxy/haproxy.cfg\n\techo \" default_backend ${service}-vms\" >> /etc/haproxy/haproxy.cfg\n fi\n\n echo \"backend ${service}-vms\" >> /etc/haproxy/haproxy.cfg\n echo \" balance roundrobin\" >> /etc/haproxy/haproxy.cfg\n if [ ! -z $timeout ]; then\n\techo \" timeout server ${timeout}\" >> /etc/haproxy/haproxy.cfg\n fi\n for count in 1 2 3; do\n\techo \" server rdo${PHD_VAR_osp_major}-${server}${count} ${PHD_VAR_network_internal}.$(( ${target} + ${count} - 1)):${port} check inter 1s\" >> /etc/haproxy/haproxy.cfg\n done\ndone\n\n# Special case back-ends\n\ncat >> /etc/haproxy/haproxy.cfg << EOF\nbackend qpid-vms\n# comment out 'stick-table' and add 'balance roundrobin' for A/A cluster mode in qpid\n stick-table type ip size 2\n stick on dst\n timeout server 120s\n server rdo${PHD_VAR_osp_major}-qpid1 ${PHD_VAR_network_internal}.103:5672 check inter 1s\n server rdo${PHD_VAR_osp_major}-qpid2 ${PHD_VAR_network_internal}.104:5672 check inter 1s\n server rdo${PHD_VAR_osp_major}-qpid3 ${PHD_VAR_network_internal}.105:5672 check inter 1s\n\nbackend db-vms-galera\n option httpchk\n option tcpka\n stick-table type ip size 1000\n stick on dst\n timeout server 90m\n server rdo${PHD_VAR_osp_major}-db1 ${PHD_VAR_network_internal}.58:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions\n server rdo${PHD_VAR_osp_major}-db2 ${PHD_VAR_network_internal}.59:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions\n server rdo${PHD_VAR_osp_major}-db3 ${PHD_VAR_network_internal}.60:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions\n\nbackend rabbitmq-vms\n option srvtcpka\n balance roundrobin\n timeout server 900m\n server rdo${PHD_VAR_osp_major}-rabbitmq1 ${PHD_VAR_network_internal}.61:5672 check inter 1s\n server rdo${PHD_VAR_osp_major}-rabbitmq2 ${PHD_VAR_network_internal}.62:5672 check inter 1s\n server rdo${PHD_VAR_osp_major}-rabbitmq3 ${PHD_VAR_network_internal}.63:5672 check inter 1s\nEOF\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n # In a collapsed environment everything is installed on rdo${PHD_VAR_osp_major}-node{1,2,3}\n # So rewite the proxy config to talk to those hosts instead\n for section in ${PHD_VAR_components}; do\n \tsed -i s/rdo${PHD_VAR_osp_major}-${section}1\\ ${PHD_VAR_network_internal}.[0-9]*/rdo${PHD_VAR_osp_major}-node1\\ ${PHD_VAR_network_internal}\\.103/g /etc/haproxy/haproxy.cfg\n \tsed -i s/rdo${PHD_VAR_osp_major}-${section}2\\ ${PHD_VAR_network_internal}.[0-9]*/rdo${PHD_VAR_osp_major}-node2\\ ${PHD_VAR_network_internal}\\.104/g /etc/haproxy/haproxy.cfg\n \tsed -i s/rdo${PHD_VAR_osp_major}-${section}3\\ ${PHD_VAR_network_internal}.[0-9]*/rdo${PHD_VAR_osp_major}-node3\\ ${PHD_VAR_network_internal}\\.105/g /etc/haproxy/haproxy.cfg\n done\nfi\n....\n\ntarget=$PHD_ENV_nodes1\n....\npcs resource create haproxy systemd:haproxy --clone\n\n\n# The VIPs changed recently \n####\n# db was 192.168.16.200 -> 201\n# rabbitmq was 192.168.16.213 -> 202\n# qpid was 192.168.16.201 -> 215\n# keystone was 192.168.16.202 -> 203\n# glance was 192.168.16.203 -> 205\n# cinder was 192.168.16.204 -> 206\n# swift was 192.168.16.205 -> 208\n# neutron was 192.168.16.206 -> 209\n# nova was 192.168.16.207 -> 210\n# horizon was 192.168.16.208 -> 211\n# heat was 192.168.16.209 -> 212\n# ceilometer was 192.168.16.211 -> 214\n####\n\noffset=200\nfor section in ${PHD_VAR_components}; do\n case $section in \n\tlb|memcache|swift-brick|mongodb)\n\t : No VIP needed for $section\n\t ;;\n\t*)\n\t pcs resource create vip-${section} IPaddr2 ip=${PHD_VAR_network_internal}.${offset} nic=eth1\n\t ;;\n esac\n offset=$(( $offset + 1 ))\ndone\n\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n\n # In a collapsed environment, we can instruct the cluster to start\n # things in a particular order and require services to be active\n # on the same hosts. We do this with constraints.\n\n for section in ${PHD_VAR_components}; do\n\tcase $section in \n\t lb|memcache|swift-brick|mongodb)\n\t\t: No VIP constraints needed for $section\n\t\t;;\n\t *)\n\t\tpcs constraint order start vip-${section} then haproxy-clone kind=Optional\n\t\tpcs constraint colocation add vip-${section} with haproxy-clone\n\t\t;;\n\tesac\n done\nfi\n....\n" }, { "path": "pcmk/memcached.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y memcached\n....\n\ntarget=$PHD_ENV_nodes1\n....\npcs resource create memcached systemd:memcached --clone interleave=true\n....\n" }, { "path": "pcmk/mongodb.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y mongodb mongodb-server\n\n# set binding IP address and replica set\n# also use smallfiles = true to stall installation while allocating N GB of journals\nsed -i \\\n\t-e 's#.*bind_ip.*#bind_ip = 0.0.0.0#g' \\\n\t-e 's/.*replSet.*/replSet = ceilometer/g' \\\n\t-e 's/.*smallfiles.*/smallfiles = true/g' \\\n\t/etc/mongod.conf\n\n# required to bootstrap mongodb\nsystemctl start mongod\nsystemctl stop mongod\n....\n\ntarget=$PHD_ENV_nodes1\n....\npcs resource create mongod systemd:mongod op start timeout=300s --clone\n\n# Setup replica (need to wait for mongodb to settle down first!)\nsleep 20\n\n# Careful with the node names here, must match FQDN\nrm -f /root/mongo_replica_setup.js\ncat > /root/mongo_replica_setup.js << EOF\nrs.initiate()\nsleep(10000)\nEOF\n\nfor node in $PHD_ENV_nodes; do\ncat >> /root/mongo_replica_setup.js << EOF\n rs.add(\"$node\");\nEOF\ndone\n\nmongo /root/mongo_replica_setup.js\nrm -f /root/mongo_replica_setup.js\n....\n" }, { "path": "pcmk/mrg.variables", "content": "# Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD\n# Each scenario file will verify that values have been provided for the variables it requires.\n\n# Deployment types: collapsed | segregated\ndeployment: collapsed\n\nnetwork:\n clock: clock.redhat.com\n domain: mpc.lab.eng.bos.redhat.com\n internal: 192.168.16\n named:\n forwarders: 10.16.36.29 10.11.5.19 10.5.30.160\n nic:\n base: 54:52:00\n external: em1 eno1 enp1s0 eth0\n internal: em2 eno2 enp2s0 eth1\n hosts:\n gateway: mrg-01\n\n # These services are not able to live behind a proxy, so we must list the hosts explicitly\n # In the segregated case, it is the per-service guests we will create\n # In the collapsed case, it is the members of the single cluster we will create \n mongodb: rdo7-node1,rdo7-node2,rdo7-node3\n memcache: rdo7-node1:11211,rdo7-node2:11211,rdo7-node3:11211\n rabbitmq: rdo7-node1,rdo7-node2,rdo7-node3\n\n# We use a centrally defined list so that segregated guests, DNS, and DHCP are all created consistently\n# Changing this list in any way will probably break the haproxy configuration \n# - ONLY add entries \n# - ALWAYS do so to the end of the list\n# - LOOK for PHD_VAR_components to see places that may be affected\ncomponents: lb db rabbitmq keystone memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer qpid node\n\n# this is RHEL7.1 GA\n# https://beaker.engineering.redhat.com/distros/\n# Use simple search to find release ex: RHEL-7.1\n# click on the GA -> and take the ID from Server install\nbeaker:\n disttree: 69383\n\nrpm:\n download: download.devel.redhat.com\n major: 7\n minor: 1\n base: released/RHEL-7/7.1/Server/x86_64/os/\n cluster: released/RHEL-7/7.1/Server/x86_64/os/addons/HighAvailability/\n updates: brewroot/repos/rhel-7.1-z-build/latest/x86_64/\n\nosp:\n configdir: /srv/RDO-7/configs\n major: 7\n\n # generate with openssl rand -hex 10\nsecrets:\n fence_xvm: bcf4e54dcfbfecb9e62c\n keystone_secret: 114a23ae49996a26d916\n swift_prefix: 8f2d8a3e326a078c5edf\n swift_suffix: 8d33d13b18e3eef6d3ca\n horizon_secret: 357a57d6a9a2353ccde5\n\n# Dedicate all cores to the sole VM that we'll create per host\nvm:\n cpus: 4\n ram: 8048\n disk: 25G\n base: rdo7-rhel7-base.img\n key: AAAAB3NzaC1yc2EAAAADAQABAAABAQDHs2qRMxtqEpr7gJygHAn2rSWKUS/FlJ9oLG7cRtzLyhIl+oSrs30KrdzkgsGTZqSEwfKM8f2LGF08x5HbN2cIDc9YhnwHQNnb8qDIXY2UqzpyLUzckctOMSiRSz/qYxeutDYGg/p1lPzPdWQPympFVIoAzCRDhogX26kXQTpKs7uUzEvZCnnzSn2I9ynchKGP3TlOzTaZHqJM4bj5+KqvUTH2ifvX3EgolP/XtIWjW54zhQnlDuS2UsDd8vvB8ZRrgtaFEXhCSivvazE8zMVAOxCFNYjnh+SvV96VB+hEjqQQeDSdhkgC2huHwsAB3Y9XCkyFe6DEfKuQZwLJjlTZ\n\n# I set the password to 'cluster', USE A SAFER ONE\nenv:\n password: cluster\n\n# TODO... \npassword:\n cluster: foo\n keystone: bar\n" }, { "path": "pcmk/neutron-agents.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_deployment\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-neutron openstack-neutron-openvswitch openvswitch\n\nsystemctl enable openvswitch\nsystemctl start openvswitch\n\novs-vsctl add-br br-int\novs-vsctl add-br br-ex\n\n# NOTE: this is the ethernet connected to the external LAN of the controller nodes!\novs-vsctl add-port br-ex eth0\n\n# workaround for keepalived DNS resolution issue within the \n# ha-routers at config reload\ndig A $(hostname) | grep -A1 \"ANSWER SEC\" | tail -n 1 | awk '{print $NF \" \" $1}' | sed -e 's/.$//g' >>/etc/hosts\ngrep -q $(hostname) /etc/hosts || echo \"Failure to setup hostname entry\"\n\n# openvswitch plugin (used as mechanism within ml2)\n\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent tunnel_types vxlan\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent vxlan_udp_port 4789\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs local_ip $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs enable_tunneling True\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs integration_bridge br-int\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_bridge br-tun\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs bridge_mappings physnet1:br-ex\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver \n\n# ovs l2 population\n\nopenstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent l2_population False\n\n# metadata agent\n\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_strategy keystone\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_url http://vip-keystone:35357/v2.0\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_host vip-keystone\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_region regionOne\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_tenant_name services\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_user neutron\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_password neutrontest\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip vip-nova\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_port 8775\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret metatest\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_workers 4\nopenstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_backlog 2048\n\n# dhcp agent\n\nopenstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver\nopenstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dhcp_delete_namespaces False\n\n# current deployment has a routing problem via qrouter->192.168.16.1 which is the \n# system default nameserver, probably the qrouter has no leg on that net, so... to \n# make that work specify a comma separated\n# list of DNS servers to be available (Forwarded to the instances)\nopenstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_dns_servers 10.35.255.14\n\n# L3 agent\n\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT handle_internal_only_routers True\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT send_arp_for_ha 3\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT router_delete_namespaces False\nopenstack-config --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge br-ex\n....\n\ntarget=$PHD_ENV_nodes1\n....\n# For A/P, set clone-max=1\npcs resource create neutron-scale ocf:neutron:NeutronScale --clone globally-unique=true clone-max=3 interleave=true\n\npcs resource create neutron-ovs-cleanup ocf:neutron:OVSCleanup --clone interleave=true\npcs resource create neutron-netns-cleanup ocf:neutron:NetnsCleanup --clone interleave=true\npcs resource create neutron-openvswitch-agent systemd:neutron-openvswitch-agent --clone interleave=true\npcs resource create neutron-dhcp-agent systemd:neutron-dhcp-agent --clone interleave=true\npcs resource create neutron-l3-agent systemd:neutron-l3-agent --clone interleave=true\npcs resource create neutron-metadata-agent systemd:neutron-metadata-agent --clone interleave=true\n\npcs constraint order start neutron-scale-clone then neutron-ovs-cleanup-clone\npcs constraint colocation add neutron-ovs-cleanup-clone with neutron-scale-clone\npcs constraint order start neutron-ovs-cleanup-clone then neutron-netns-cleanup-clone\npcs constraint colocation add neutron-netns-cleanup-clone with neutron-ovs-cleanup-clone\npcs constraint order start neutron-netns-cleanup-clone then neutron-openvswitch-agent-clone\npcs constraint colocation add neutron-openvswitch-agent-clone with neutron-netns-cleanup-clone\npcs constraint order start neutron-openvswitch-agent-clone then neutron-dhcp-agent-clone\npcs constraint colocation add neutron-dhcp-agent-clone with neutron-openvswitch-agent-clone\npcs constraint order start neutron-dhcp-agent-clone then neutron-l3-agent-clone\npcs constraint colocation add neutron-l3-agent-clone with neutron-dhcp-agent-clone\npcs constraint order start neutron-l3-agent-clone then neutron-metadata-agent-clone\npcs constraint colocation add neutron-metadata-agent-clone with neutron-l3-agent-clone\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n pcs constraint order start neutron-server-clone then neutron-scale-clone\nfi\n....\n" }, { "path": "pcmk/neutron-server.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_osp_configdir\nPHD_VAR_network_hosts_rabbitmq\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\n# NOTE: once again, careful with vip-* / user / password substitutions and local\n# ip addresses via shell expansion\n# While neutron can be replaced with nova-network. This how-to does NOT\n# contain information on how to deploy nova-network in HA fashion.\n# This version of the how-to uses the ML2 plugin. Other supported\n# plugins can be used. Please consult the OSP documentation on how\n# to configure/deploy other plugins.\n# neutron-server requires neutron-agents and viceversa\n\nyum install -y openstack-neutron openstack-neutron-openvswitch openstack-neutron-ml2 \n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password neutrontest\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://vip-keystone:35357/v2.0/\nopenstack-config --set /etc/neutron/neutron.conf keystone_authtoken identity_uri http://vip-keystone:5000/\n\nopenstack-config --set /etc/neutron/neutron.conf database connection mysql://neutron:neutrontest@vip-db:3306/neutron\nopenstack-config --set /etc/neutron/neutron.conf database max_retries -1\n\nopenstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier\n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT nova_url http://vip-nova:8774/v2\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes True\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_data_changes True\n\nopenstack-config --set /etc/neutron/neutron.conf nova auth_url http://vip-keystone:35357/\nopenstack-config --set /etc/neutron/neutron.conf nova auth_plugin password\nopenstack-config --set /etc/neutron/neutron.conf nova project_domain_id default\nopenstack-config --set /etc/neutron/neutron.conf nova user_domain_id default\nopenstack-config --set /etc/neutron/neutron.conf nova region_name regionOne\nopenstack-config --set /etc/neutron/neutron.conf nova project_name services\nopenstack-config --set /etc/neutron/neutron.conf nova username compute\nopenstack-config --set /etc/neutron/neutron.conf nova password novatest\n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin neutron.plugins.ml2.plugin.Ml2Plugin\n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins neutron.services.l3_router.l3_router_plugin.L3RouterPlugin\n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT router_scheduler_driver neutron.scheduler.l3_agent_scheduler.ChanceScheduler\n\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini type_drivers local,gre,flat,vxlan,vlan\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers openvswitch\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks \"*\"\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_gre tunnel_id_ranges 10:10000\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vni_ranges 10:10000\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vxlan_group 224.0.0.1\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_security_group True\n# is this still required?\nopenstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup firewall_driver True\n\nln -sf /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini\n\n# There are a number of different approaches to make neutron highly\n# available, we cover only A/A and A/P here but more details\n# (internal-only) are available in Bugzilla:\n# https://bugzilla.redhat.com/show_bug.cgi?id=1170113#c18\n# \n# 1) Fully neutron A/A, considering nodes A,B,C\n# \n# all nodes would have l3_ha=True , max_l3_agents_per_router=3, min=2\n# A: host=neutron-n-0 B: host=neutron-n-1 C: host=neutron-n-2\n# \n# (this way we cover the upgrade path from OSP5->OSP6, by keeping at least one host withb the old neutron-n-0 ID)\n# \n# 3) A/P, with 1 active node\n# \n# all nodes would have l3_ha=False\n# \n# a) A + B + C have host=neutron-n-0\n# b) like case 2, but: \n# A: host=neutron-n-0 B: (passive not set) C: (passive, not set)\n# and neutron scale does the host= change during failover.\n\n# A/P does NOT require extra settings. Defaults are fine\n# Fully A/A requires extra neutron-server configuration:\n\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT l3_ha True\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT max_l3_agents_per_router 0\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT min_l3_agents_per_router 2\n\n# This value _MUST_ follow the number of nodes in the pacemaker cluster\nopenstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 3\n\n....\n\ntarget=$PHD_ENV_nodes1\n....\n. ${PHD_VAR_osp_configdir}/keystonerc_admin\n\n# required when installing the DB manually\nneutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade kilo\n\nsystemctl start neutron-server\nsystemctl stop neutron-server\n\n# add the service to pacemaker\npcs resource create neutron-server systemd:neutron-server op start timeout=90 --clone interleave=true\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n pcs constraint order start openstack-keystone-clone then neutron-server-clone\nfi\n....\n" }, { "path": "pcmk/neutron-test.sh", "content": "# TEST/create your first network\n# It is not possible to test neutron completely until the full deployment is complete as it is required to run instances to verify network connectivity. \n\n. ${PHD_VAR_env_configdir}/keystonerc_admin\n\n# WARNING: openstack client is NOT ready to manage neutron!\n\nneutron net-create internal_lan\nneutron subnet-create --ip_version 4 --gateway 192.168.100.1 --name \"internal_subnet\" internal_lan 192.168.100.0/24\nneutron net-create public_lan --router:external\nneutron subnet-create --gateway 10.16.151.254 --allocation-pool start=10.16.144.76,end=10.16.144.83 --disable-dhcp --name public_subnet public_lan 10.16.144.0/21\nneutron router-create router\nneutron router-gateway-set router public_lan\nneutron router-interface-add router internal_subnet\n" }, { "path": "pcmk/nova-test.sh", "content": ". ${PHD_VAR_env_configdir}/keystonerc_admin\nnova usage\nnova usage-list" }, { "path": "pcmk/nova.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_network_domain\nPHD_VAR_network_hosts_gateway\nPHD_VAR_network_hosts_memcache\nPHD_VAR_network_hosts_rabbitmq\nPHD_VAR_network_internal\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-nova-console openstack-nova-novncproxy openstack-utils openstack-nova-api openstack-nova-conductor openstack-nova-scheduler python-cinderclient python-memcached\n\nopenstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}\nopenstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues true\nopenstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60\nopenstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers ${PHD_VAR_network_hosts_memcache}\n\n\n\n# Particularly in the collapsed case, we get a lot of conflicts with the haproxy server\nopenstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address $(host $(hostname -s) | awk '{print $4}')\nopenstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nopenstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://${PHD_VAR_network_hosts_gateway}.${PHD_VAR_network_domain}:6080/vnc_auto.html\n\nopenstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone\n# FIX ME: nova doesn't like hostnames anymore?\n# openstack-config --set /etc/nova/nova.conf DEFAULT metadata_host vip-nova\nopenstack-config --set /etc/nova/nova.conf DEFAULT metadata_host ${PHD_VAR_network_internal}.210\nopenstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen_port 8775\n\nopenstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True\nopenstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest\nopenstack-config --set /etc/nova/nova.conf glance host vip-glance\nopenstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API\nopenstack-config --set /etc/nova/nova.conf neutron url http://vip-neutron:9696/\nopenstack-config --set /etc/nova/nova.conf neutron admin_tenant_name services\nopenstack-config --set /etc/nova/nova.conf neutron admin_username neutron\nopenstack-config --set /etc/nova/nova.conf neutron admin_password neutrontest\nopenstack-config --set /etc/nova/nova.conf neutron admin_auth_url http://vip-keystone:35357/v2.0\nopenstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver\nopenstack-config --set /etc/nova/nova.conf DEFAULT libvirt_vif_driver nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver\n\n\n# those entries were workaround for https://bugs.launchpad.net/neutron/+bug/1464178\n# that breaks neutron and evacuation\n# openstack-config --set /etc/nova/nova.conf DEFAULT notify_api_faults False\n# openstack-config --set /etc/nova/nova.conf DEFAULT vif_plugging_is_fatal True\n# openstack-config --set /etc/nova/nova.conf DEFAULT vif_plugging_timeout 300\n\nopenstack-config --set /etc/nova/nova.conf conductor use_local false\n\nopenstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@vip-db/nova\nopenstack-config --set /etc/nova/nova.conf database max_retries -1\n\n# REQUIRED FOR A/A scheduler\nopenstack-config --set /etc/nova/nova.conf DEFAULT scheduler_host_subset_size 30\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_host vip-keystone\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_tenant_name services\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_user compute\nopenstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_password novatest\n....\n\ntarget=$PHD_ENV_nodes1\n....\nsu nova -s /bin/sh -c \"nova-manage db sync\"\n\npcs resource create openstack-nova-consoleauth systemd:openstack-nova-consoleauth --clone interleave=true\npcs resource create openstack-nova-novncproxy systemd:openstack-nova-novncproxy --clone interleave=true\npcs resource create openstack-nova-api systemd:openstack-nova-api --clone interleave=true\npcs resource create openstack-nova-scheduler systemd:openstack-nova-scheduler --clone interleave=true\npcs resource create openstack-nova-conductor systemd:openstack-nova-conductor --clone interleave=true\n\npcs constraint order start openstack-nova-consoleauth-clone then openstack-nova-novncproxy-clone\npcs constraint colocation add openstack-nova-novncproxy-clone with openstack-nova-consoleauth-clone\n\npcs constraint order start openstack-nova-novncproxy-clone then openstack-nova-api-clone\npcs constraint colocation add openstack-nova-api-clone with openstack-nova-novncproxy-clone\n\npcs constraint order start openstack-nova-api-clone then openstack-nova-scheduler-clone\npcs constraint colocation add openstack-nova-scheduler-clone with openstack-nova-api-clone\n\npcs constraint order start openstack-nova-scheduler-clone then openstack-nova-conductor-clone\npcs constraint colocation add openstack-nova-conductor-clone with openstack-nova-scheduler-clone\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n pcs constraint order start openstack-keystone-clone then openstack-nova-consoleauth-clone\nfi\n....\n" }, { "path": "pcmk/nova_client.py", "content": "#!/usr/bin/python -tt\n\nimport argparse\nimport collections\nimport inspect\nimport logging\nimport sys\n\ntry:\n from novaclient import client as nova_client\nexcept ImportError:\n logging.error(\"nova not found or not accessible\")\n sys.exit(1)\n\n\n# NOTE(sbauza): Necessary as the contrib path is not imported directly\ntry:\n from novaclient import extension\n from novaclient.v2.contrib import migrations\nexcept ImportError:\n logging.warning(\"migrations can't be done since the module is not there\")\n migrations = None\n\n\ndef register_named(cls):\n cls.methods_map = {}\n cls.methods_pos_map = {}\n cls.methods_opt_map = {}\n for methodname in dir(cls):\n method = getattr(cls, methodname)\n if hasattr(method, '_named'):\n cls.methods_map.update({method._named: methodname})\n if hasattr(method, '_pos'):\n cls.methods_pos_map.update({method._named: method._pos})\n if hasattr(method, '_opts'):\n cls.methods_opt_map.update({method._named: method._opts})\n return cls\n\n\ndef named(method_name, positionals=[], opts_with_val=[], opts_without_val=[]):\n \"\"\"Decorator for providing the shell name for the method and the optional\n arguments.\n\n :method_name: Shell name for the method (like \"service-enable\")\n :positionals: List of strings corresponding to a positional\n :opts_with_val: Optionals that accept a value (like --foo bar)\n :opts_without_val: Optionals that are boolean (like --foo)\n \"\"\"\n def wrapped(func):\n func._named = method_name\n func._pos = positionals\n func._opts = dict(map(lambda x: (x, True), opts_with_val) + map(\n lambda x: (x, False), opts_without_val))\n return func\n return wrapped\n\n\ndef shell_fields(fields):\n \"\"\"Decorator for providing the list of fields to show up in CLI.\"\"\"\n def wrapped(func):\n func.fields = fields\n return func\n return wrapped\n\n\n@register_named\nclass NovaClientWrapper(object):\n \"\"\"Wrapper for accessing a subset of novaclient API.\"\"\"\n\n def __init__(self, version, username, password, tenant_name, auth_url):\n if migrations:\n extensions = [extension.Extension('migrations', migrations)]\n else:\n extensions = None\n self.nova = nova_client.Client(version, username, password,\n tenant_name, auth_url,\n extensions=extensions)\n\n @shell_fields([\"Id\", \"Binary\", \"Host\", \"Zone\", \"Status\", \"State\",\n \"Updated_at\"])\n @named('service-list', opts_with_val=['host', 'binary'])\n def service_list(self, host=None, binary=None):\n services = self.nova.services.list(host=host, binary=binary)\n return services\n\n @shell_fields(['Host', 'Binary', 'Status'])\n @named('service-enable', positionals=['host', 'binary'])\n def service_enable(self, host, binary):\n return self.nova.services.enable(host=host, binary=binary)\n\n @shell_fields(['Source Node', 'Dest Node', 'Source Compute',\n 'Dest Compute', 'Dest Host', 'Status', 'Instance UUID',\n 'Old Flavor', 'New Flavor', 'Created At', 'Updated At'])\n @named('migration-list', opts_with_val=['host', 'status', 'cell_name'])\n def migration_list(self, host=None, status=None, cell_name=None):\n if hasattr(self.nova, 'migrations'):\n return self.nova.migrations.list(host=host, status=status,\n cell_name=cell_name)\n\n # NOTE(sbauza); We mimic the host-servers-migrate module\n def _server_migrate(self, server):\n success = True\n error_message = \"\"\n try:\n self.nova.servers.migrate(server=server['uuid'])\n except Exception as e:\n success = False\n error_message = \"Error while migrating instance: %s\" % e\n response = collections.namedtuple(\n 'HostServersMigrateResponse',\n ['server_uuid', 'migration_accepted', 'error_message'])\n response._make([server['uuid'], success, error_message])\n return response\n\n @shell_fields([\"Server UUID\", \"Migration Accepted\", \"Error Message\"])\n @named('host-servers-migrate', positionals=['host'])\n def host_servers_migrate(self, host):\n hypervisors = self.nova.hypervisors.search(host, servers=True)\n response = []\n for hyper in hypervisors:\n if hasattr(hyper, 'servers'):\n for server in hyper.servers:\n response.append(self._server_migrate(server))\n return response\n\n def handle_method_and_args(self, args_list):\n (method, extra) = (None, None)\n kwargs = {}\n for meth in self.methods_map:\n try:\n pos = args_list.index(meth)\n except ValueError:\n # Method not found\n continue\n method_name = args_list[pos]\n method = getattr(self, self.methods_map[method_name])\n extra = args_list[:pos] if pos > 0 else None\n args = args_list[pos+1:]\n spec = inspect.getargspec(method)\n for arg in args:\n opt = arg.rsplit(\"-\")[-1]\n if opt in spec.args:\n if opt in self.methods_opt_map[method_name]:\n if self.methods_opt_map[method_name][opt] is True:\n # This is an opt followed by its value\n try:\n val = args[args.index(arg)+1]\n except IndexError:\n # The opt was awaiting a value, invalidating it\n args.remove(arg)\n continue\n kwargs[opt] = val\n args.remove(arg)\n args.remove(val)\n else:\n # This is an opt True or False\n kwargs[opt] = True\n args.remove(arg)\n # Let's zip the remainings arguments with the awaiting positionals\n positionals = dict(zip(self.methods_pos_map[method_name], args))\n kwargs.update(positionals)\n break\n return (extra, method, kwargs)\n\n\ndef print_list(objs, fields):\n # find max column width\n columnWidth = 0\n for obj in objs:\n for field in fields:\n field_name = field.lower().replace(' ', '_')\n width = len(str(getattr(obj, field_name, '')))\n if width > columnWidth:\n columnWidth = width\n if len(field) > columnWidth:\n columnWidth = len(field)\n\n outputStr = '+' + ('-'*(columnWidth + 2) + '+')*len(objs) + '\\n'\n outputStr += '| ' + \" | \".join([field.ljust(columnWidth)\n for field in fields]) + \" |\\n\"\n outputStr += '+' + ('-'*(columnWidth + 2) + '+')*len(objs) + '\\n'\n for obj in objs:\n rowList = []\n for field in fields:\n field_name = field.lower().replace(' ', '_')\n rowList.append(str(getattr(obj, field_name, '')\n ).ljust(columnWidth))\n outputStr += '| ' + ' | '.join(rowList) + \" |\\n\"\n outputStr += '+' + ('-'*(columnWidth + 2) + '+')*len(objs) + '\\n'\n return outputStr\n\n\ndef main():\n logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stderr))\n\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--os-auth-url\", required=True, dest='auth_url')\n parser.add_argument(\"--os-username\", required=True, dest='username')\n parser.add_argument(\"--os-password\", required=True, dest='password')\n parser.add_argument(\"--os-tenant-name\", required=True, dest='tenant_name')\n parser.add_argument(\"remainder\", nargs=argparse.REMAINDER,\n help=\"nova command followed by its args\")\n\n args = parser.parse_args()\n method_and_args = args.remainder\n nova = NovaClientWrapper('2', args.username, args.password,\n args.tenant_name, args.auth_url)\n (extra, method, kwargs) = nova.handle_method_and_args(method_and_args)\n if method is None:\n logging.error(\"Method not mapped in %s\", method_and_args)\n return 1\n result = method(**kwargs)\n print_list([result] if not isinstance(result, list) else result,\n method.fields)\n\nif __name__ == \"__main__\":\n sys.exit(main())\n" }, { "path": "pcmk/rabbitmq-test.sh", "content": "rabbitmqctl cluster_status\nrabbitmqctl list_policies\n" }, { "path": "pcmk/rabbitmq.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_osp_configdir\nPHD_VAR_network_domain\nPHD_VAR_network_internal\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y rabbitmq-server \n\n# NOTE: we need to bind the service to the internal IP address\n\ncat > /etc/rabbitmq/rabbitmq-env.conf << EOF\nNODE_IP_ADDRESS=$(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\nEOF\n\n# required to generate the cookies\nsystemctl start rabbitmq-server\nsystemctl stop rabbitmq-server\n\nmkdir -p $PHD_VAR_osp_configdir\nif [ ! -e $PHD_VAR_osp_configdir/rabbitmq_erlang_cookie ]; then\n cp /var/lib/rabbitmq/.erlang.cookie $PHD_VAR_osp_configdir/rabbitmq_erlang_cookie\nfi\n\n# the cookie has to be the same across all nodes. Copy around as preferred, I am \n# using my NFS commodity storage. Also check for file permission/ownership. I \n# workaround that step by using 'cat' vs cp.\ncat $PHD_VAR_osp_configdir/rabbitmq_erlang_cookie > /var/lib/rabbitmq/.erlang.cookie\n....\n\n\ntarget=$PHD_ENV_nodes1\n....\npcs resource create rabbitmq rabbitmq-cluster set_policy='ha-all ^(?!amq\\.).* {\"ha-mode\":\"all\"}' meta notify=true --clone ordered=true interleave=true\n....\n" }, { "path": "pcmk/swift-aco.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 3\n\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_secrets_swift_prefix\nPHD_VAR_secrets_swift_suffix\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-swift-object openstack-swift-container openstack-swift-account openstack-utils rsync xfsprogs\n\n# If you have a dedicated device, format it here and have the cluster mount it as per 'swift-fs' below\n# We don't, but swift wants a different partition than /, so we'll create a loopback file and mount that for 'swift-fs'\n\nmkdir -p /local/swiftstorage/target\ntruncate --size=1G /local/swift.img\nlosetup /dev/loop0 /local/swift.img\nmkfs.xfs /dev/loop0\nmount /dev/loop0 /local/swiftstorage/target\nchown -R swift:swift /local\numount /dev/loop0\n\n# Some extra magic to set up the loopback device after a reboot\necho \"losetup /dev/loop0 /local/swift.img\" >> /etc/rc.d/rc.local \nchmod a+x /etc/rc.d/rc.local \n\nopenstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_prefix ${PHD_VAR_secrets_swift_prefix}\nopenstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_suffix ${PHD_VAR_secrets_swift_suffix}\nopenstack-config --set /etc/swift/swift.conf filter:ceilometer use egg:ceilometer#swift\nopenstack-config --set /etc/swift/swift.conf pipeline:main pipeline \"healthcheck cache authtoken keystoneauth proxy-server ceilometer\"\n\nopenstack-config --set /etc/swift/object-server.conf DEFAULT bind_ip 0.0.0.0\nopenstack-config --set /etc/swift/object-server.conf DEFAULT devices /local/swiftstorage\nopenstack-config --set /etc/swift/object-server.conf DEFAULT mount_check false\nopenstack-config --set /etc/swift/account-server.conf DEFAULT bind_ip 0.0.0.0\nopenstack-config --set /etc/swift/account-server.conf DEFAULT devices /local/swiftstorage\nopenstack-config --set /etc/swift/account-server.conf DEFAULT mount_check false\nopenstack-config --set /etc/swift/container-server.conf DEFAULT bind_ip 0.0.0.0\nopenstack-config --set /etc/swift/container-server.conf DEFAULT devices /local/swiftstorage\nopenstack-config --set /etc/swift/container-server.conf DEFAULT mount_check false\n\nopenstack-config --set /etc/swift/object-server.conf DEFAULT mount_check false\nopenstack-config --set /etc/swift/account-server.conf DEFAULT mount_check false\nopenstack-config --set /etc/swift/container-server.conf DEFAULT mount_check false\n\nchown -R root:swift /etc/swift\n....\n\ntarget=all\n....\nif ! pcs resource show swift-fs > /dev/null 2>&1; then\n\n # We must be either the first node to run, or configuring\n # single-node clusters for a segregated deployment\n\n pcs resource create swift-fs Filesystem device=\"/dev/loop0\" directory=\"/local/swiftstorage/target\" fstype=\"xfs\" force_clones=\"yes\" --clone interleave=true\n\n pcs resource create swift-account systemd:openstack-swift-account --clone interleave=true\n pcs constraint colocation add swift-account-clone with swift-fs-clone\n pcs constraint order start swift-fs-clone then swift-account-clone\n\n pcs resource create swift-container systemd:openstack-swift-container --clone interleave=true\n pcs constraint colocation add swift-container-clone with swift-account-clone\n pcs constraint order start swift-account-clone then swift-container-clone\n\n pcs resource create swift-object systemd:openstack-swift-object --clone interleave=true\n pcs constraint colocation add swift-object-clone with swift-container-clone\n pcs constraint order start swift-container-clone then swift-object-clone\n if [ $PHD_VAR_deployment = collapsed ]; then\n\tpcs constraint order start openstack-keystone-clone then swift-account-clone\n fi\nfi\n....\n" }, { "path": "pcmk/swift-test.sh", "content": ". ${PHD_VAR_env_configdir}/keystonerc_admin\n\nopenstack container list\nopenstack container create test\nopenstack container list\n\nopenstack object list test\ntruncate --size=1M /tmp/foobar\nopenstack object create test /tmp/foobar\nopenstack object list test\nopenstack object delete test /tmp/foobar\nopenstack object list test\n\nopenstack container delete test\nopenstack container list\n" }, { "path": "pcmk/swift.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed at this step include:\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_network_internal\nPHD_VAR_network_hosts_memcache\nPHD_VAR_osp_configdir\nPHD_VAR_osp_major\nPHD_VAR_secrets_swift_prefix\nPHD_VAR_secrets_swift_suffix\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 1\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\nyum install -y openstack-swift-proxy openstack-utils python-swiftclient python-openstackclient\n....\n\ntarget=$PHD_ENV_nodes1\n....\n# NOTE: you MUST refer to the swift-ring-builder documentation in order to \n# configure prope data redundancy and set those values properly.\n# This is just a generic example that will store 3 copies of the same data for\n# proof-of-concept purposes.\n\nswift-ring-builder /etc/swift/object.builder create 16 3 24\nswift-ring-builder /etc/swift/container.builder create 16 3 24\nswift-ring-builder /etc/swift/account.builder create 16 3 24\n\n# .76,.77 and .78 are the addresses of rdo${PHD_VAR_osp_major}-swift-brick{1,2,3}\n# 'target' here comes from the swift-fs resource created on the proxy\n# pcs resource create swift-fs Filesystem device=\"/local/swiftsource\" directory=\"/local/swiftstorage/target\" fstype=\"none\" options=\"bind\"\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n\tbaseip=102\nelse\n\tbaseip=75\nfi\n\nfor i in 1 2 3; do\n\ttarget=$((baseip + $i))\n\tswift-ring-builder /etc/swift/account.builder add z$i-${PHD_VAR_network_internal}.${target}:6202/target 10\n\tswift-ring-builder /etc/swift/container.builder add z$i-${PHD_VAR_network_internal}.${target}:6201/target 10\n\tswift-ring-builder /etc/swift/object.builder add z$i-${PHD_VAR_network_internal}.${target}:6200/target 10\ndone\n\nswift-ring-builder /etc/swift/account.builder rebalance\nswift-ring-builder /etc/swift/container.builder rebalance\nswift-ring-builder /etc/swift/object.builder rebalance\n\nmkdir -p $PHD_VAR_osp_configdir/swift\ncp /etc/swift/*.builder /etc/swift/*.gz $PHD_VAR_osp_configdir/swift/\n\n....\n\ntarget=all\n....\n\nopenstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_prefix ${PHD_VAR_secrets_swift_prefix}\nopenstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_suffix ${PHD_VAR_secrets_swift_suffix}\nopenstack-config --set /etc/swift/swift.conf filter:ceilometer use egg:ceilometer#swift\nopenstack-config --set /etc/swift/swift.conf pipeline:main pipeline \"healthcheck cache authtoken keystoneauth proxy-server ceilometer\"\n\nopenstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_tenant_name services\nopenstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_user swift\nopenstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_password swifttest\nopenstack-config --set /etc/swift/proxy-server.conf filter:authtoken identity_uri http://vip-keystone:35357/\nopenstack-config --set /etc/swift/proxy-server.conf filter:authtoken auth_uri http://vip-keystone:5000/\nopenstack-config --set /etc/swift/proxy-server.conf DEFAULT bind_ip $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')\n\nopenstack-config --set /etc/swift/object-expirer.conf object-expirer concurrency 100\n\nopenstack-config --set /etc/swift/proxy-server.conf filter:cache memcache_servers ${PHD_VAR_network_hosts_memcache}\nopenstack-config --set /etc/swift/object-expirer.conf filter:cache memcache_servers ${PHD_VAR_network_hosts_memcache}\n\ncp $PHD_VAR_osp_configdir/swift/*.builder $PHD_VAR_osp_configdir/swift/*.gz /etc/swift/\nchown -R root:swift /etc/swift\n\n....\n\ntarget=$PHD_ENV_nodes1\n....\npcs resource create swift-proxy systemd:openstack-swift-proxy --clone interleave=true\npcs resource create swift-object-expirer systemd:openstack-swift-object-expirer\n\npcs constraint order start swift-proxy-clone then swift-object-expirer\n\nif [ $PHD_VAR_deployment = collapsed ]; then\n pcs constraint order start swift-account-clone then swift-proxy-clone\nfi\n....\n" }, { "path": "pcmk/virt-hosts.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# Tasks to be performed for the hosts include:\n# - Install fence_virt so that the guests can be reset by other members of the virtualized clusters\n# - Turning off auto-generation of resolv.conf so we can point to our gateway DNS server\n# - Obtain the golden image from the first host\n# - Clone the golden image for each service, inserting MAC addresses using the same rules as in the \n#\n# Tasks to be performed when creating the image include:\n# - Installing CentOS 7 \n# - Pointing to the OpenStack, HA-Addon and Updates repositories\n# - Ensuring cloned guests can obtain a DHCP lease\n# - Ensuring consistent network interface names\n# - Adding SSH keys if necessary\n# - Configuring guests to obtain their hostname from DHCP\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\n\nPHD_VAR_deployment\nPHD_VAR_network_internal\nPHD_VAR_network_nic_base\nPHD_VAR_rpm_download\nPHD_VAR_secrets_fence_xvm\nPHD_VAR_osp_configdir\nPHD_VAR_osp_major\nPHD_VAR_rpm_major\nPHD_VAR_rpm_minor\nPHD_VAR_vm_base\nPHD_VAR_vm_cpus\nPHD_VAR_vm_disk\nPHD_VAR_vm_key\nPHD_VAR_vm_ram\nPHD_VAR_components\nPHD_VAR_rpm_base\nPHD_VAR_rpm_cluster\nPHD_VAR_rpm_download\nPHD_VAR_rpm_updates\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 4\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\n\ntarget=all\n....\n\n# Not required on the gateway, but wont hurt either\n\nyum install -y fence-virtd fence-virtd-multicast fence-virtd-libvirt\nlastoct=\"$(hostname -s | sed -e 's#^[a-z]*-##g' -e 's#^0*##g')\"\n\ncat > /etc/fence_virt.conf << EOF\nfence_virtd {\n listener = \"multicast\";\n backend = \"libvirt\";\n}\nlisteners {\n multicast {\n # key_file = \"/etc/cluster/fence_xvm.key\";\n address = \"225.0.0.$lastoct\";\n # Needed on Fedora systems\n interface = \"vmnet0\";\n }\n}\nbackends {\n libvirt { \n uri = \"qemu:///system\";\n }\n}\nEOF\n\nmkdir -p /etc/cluster/\necho ${PHD_VAR_secrets_fence_xvm} > /etc/cluster/fence_xvm.key\n\nsystemctl enable fence_virtd\nsystemctl start fence_virtd\n....\n\n\ntarget=$PHD_ENV_nodes1\n....\n\nif [ -e ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} ]; then\n : skip this step if an image already exists\n exit 0\nfi\n\ndistro=${PHD_VAR_rpm_major}.${PHD_VAR_rpm_minor}\n\n# headless - do not bork if we specify --noautoconsole\n# reboot\n\n# Do not specify --hostname= for network\n# If prevents assinging by DHCP \n\ntargettz=\"$(timedatectl | grep Timezone | awk '{print $2}')\"\n\nif [ -z \"$targettz\" ]; then\n targettz=Australia/Melbourne\nfi\n\ncat << EOF > virt-base.ks\ninstall\ntext\nreboot\nrootpw redhat\nlang en_US.UTF-8\nkeyboard us\nnetwork --bootproto dhcp\nfirewall --enabled --ssh\nselinux --permissive\ntimezone --utc $targettz\nbootloader --location=mbr --append=\"console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH serial text headless\"\nzerombr\nclearpart --all --initlabel\nautopart --type=lvm\nskipx\n\n%packages\n@core\ngrep\ngawk\nbind-utils\nvi\nnet-tools\ntcpdump\nwget\nsos\nnfs-utils\nntp\nntpdate\n%end\n\n%post\n\n# ethX will do nicely thankyou\nrm -f /lib/udev/rules.d/75-persistent-net-generator.rules\nrm -f /etc/udev/rules.d/70-persistent-net.rules\n\n# Need PEERDNS=no on the external interface to make sure all lookups\n# go to our gateway (otherwise setting setting hostname via dhcp wont\n# work)\n#\n# MUST add 'DEVICE=ethX' when deleting 'HWADDR', otherwise you end up\n# with crazy things like:\n#\n# [root@localhost ~]# ifdown eth0\n# Device 'eth1' successfully disconnected.\n#\n# Let eth0 come up, but don't let it connect to avoid conflicting dhcp\n# and naming issues\n#\n# /usr/share/doc/initscripts/sysconfig.txt\n\ncat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOT\nNAME=\"eth0\"\nDEVICE=\"eth0\"\nONBOOT=yes\nBOOTPROTO=none\nTYPE=Ethernet\nIPV4_FAILURE_FATAL=no\nEOT\n\ncat > /etc/sysconfig/network-scripts/ifcfg-eth1 << EOT\nNAME=\"eth1\"\nDEVICE=\"eth1\"\nONBOOT=yes\nBOOTPROTO=dhcp\nTYPE=Ethernet\nEOT\n\nmkdir /root/.ssh\ncat <<__EOT__ >> /root/.ssh/authorized_keys\nssh-rsa ${PHD_VAR_vm_key} admin\n__EOT__\n\nchmod 600 /root/.ssh/authorized_keys\n\n# Prevent DHCPNAK by removing stale leases\n# Occurs if we clone the guest and change NICs\nrm -f /var/lib/NetworkManager/dhclient-*.lease\n\n# If hostname is anything other than 'localhost.localdomain', then\n# NetworkManager wont bother looking up DNS/DHCP for the host name\n# when the node boots:\n#\n# May 5 11:04:15 localhost NetworkManager[676]: Setting system hostname to 'rdo7-node1.vmnet.mpc.lab.eng.bos.redhat.com' (from address lookup)\nhostnamectl set-hostname localhost.localdomain\n\ncat > /etc/yum.repos.d/redhat-${distro}.repo << EOT\n[redhat-${distro}]\nname=redhat-${distro}\nbaseurl=http://${PHD_VAR_rpm_download}/${PHD_VAR_rpm_base}\ngpgcheck=0\nenabled=1\n[redhat-ha]\nname=redhat-${distro}-ha\nbaseurl=http://${PHD_VAR_rpm_download}/${PHD_VAR_rpm_cluster}\ngpgcheck=0\nenabled=1\n[redhat-z]\nname=redhat-${distro}-z\nbaseurl=http://${PHD_VAR_rpm_download}/${PHD_VAR_rpm_updates}\ngpgcheck=0\nenabled=1\nEOT\n\n%end\nEOF\n\n\nyum install -y virt-install \nvirsh destroy rdo${PHD_VAR_osp_major}-rhel${PHD_VAR_rpm_major}-base &>/dev/null || true\nvirsh undefine rdo${PHD_VAR_osp_major}-rhel${PHD_VAR_rpm_major}-base &>/dev/null || true\n\n#Create the qcow2 disk image with preallocation and 'facllocate'(which pre-allocates all the blocks to a file) it for max. performance\necho \"Creating qcow2 disk image..\"\nrm -f ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base}\nmkdir -p ${PHD_VAR_osp_configdir}\nqemu-img create -f qcow2 -o preallocation=metadata ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} ${PHD_VAR_vm_disk}\nls -lash ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base}\n#fallocate -l $(ls -al ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} | awk '{print $5}') ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base}\n#ls -lash ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base}\n\n# Turn off install logs\nopts=\"--noautoconsole\"\nopts=\"\"\n\n# --console pty,target_type=virtio \n#Create the regular-guest\nvirt-install --connect=qemu:///system \\\n --network=bridge:ext0,mac=${PHD_VAR_network_nic_base}:00:00:02 \\\n --initrd-inject=./virt-base.ks \\\n --extra-args=\"ks=file:/virt-base.ks console=tty0 console=ttyS0,115200 serial rd_NO_PLYMOUTH\" \\\n --name=rdo${PHD_VAR_osp_major}-rhel${PHD_VAR_rpm_major}-base \\\n --disk path=${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base},format=qcow2,cache=none \\\n --ram ${PHD_VAR_vm_ram} \\\n --vcpus=${PHD_VAR_vm_cpus} \\\n --check-cpu \\\n --accelerate \\\n --os-type linux \\\n --os-variant rhel${PHD_VAR_rpm_major} \\\n --cpuset auto \\\n --hvm \\\n --location=http://${PHD_VAR_rpm_download}/${PHD_VAR_rpm_base} \\\n --nographics $opts\n\n# Needs 15-20 minutes apparently\nsleep 1200\n....\n\n\n#target=$PHD_ENV_nodes1\n#....\n#virsh define /srv/vms/rhos4-base.xml\n#virsh define /srv/rdo${PHD_VAR_osp_major}-rhel7-vms/rdo${PHD_VAR_osp_major}-rhel7-base.xml\n#....\n\ntarget=all\n....\n\nwhich rsync >/dev/null 2>&1 || yum install -y rsync\n\nif [ $(hostname -s | awk -F. '{print $1}') = $(echo $PHD_ENV_nodes1 | awk -F. '{print $1}') ]; then\n : Everyone except the first node needs to run the commands below\n exit 0\nfi\n\nmkdir -p /localvms\n\n# cp /srv/vms/rhos4-base.img /localvms/\nif [ ! -e /localvms/${PHD_VAR_vm_base} ]; then\n # This takes a loooong time, skip it if we already have an image\n rsync -avz --progress ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} /localvms/\nfi\n\n# The internal network must go first so that hostname can be set by\n# DHCP and it will generally be the preferred network\n#\n# Random factoid, setting ONBOOT=no for eth0 will prevent eth1 from\n# being started, even if eth1 has ONBOOT=yes\n\ncat<<-EOF > /localvms/template.xml\n\n VM_NAME\n ${PHD_VAR_vm_ram}000\n ${PHD_VAR_vm_ram}000\n ${PHD_VAR_vm_cpus}\n \n hvm\n \n \n \n \n \n \n \n \n destroy\n restart\n restart\n \n /usr/libexec/qemu-kvm\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\nEOF\n\nsequence=16\nlastoct=\"$(hostname -s | sed -e 's#^[a-z]*-##g' -e 's#^0*##g')\"\noffset=\"$(echo ${PHD_ENV_nodes1} | sed -e 's#\\..*##g' -e 's#^[a-z]*-##g' -e 's#^0*##g')\"\nfor section in ${PHD_VAR_components}; do\n cd /localvms/\n\n target=rdo${PHD_VAR_osp_major}-${section}$(( ${lastoct} - ${offset} ))\n octet=${sequence}\n\n sequence=$((sequence + 1))\n\n virsh destroy $target > /dev/null 2>&1 || true\n virsh undefine $target --snapshots-metadata > /dev/null 2>&1 || true\n\n if [ $PHD_VAR_deployment = collapsed ]; then\n case $section in \n \t node) ;;\n *) \n\t # We don't need any of the other instances\n\t\t # Its good to get rid of any old copies that might exist though\n\t continue;;\n esac\n fi\n\n cp template.xml ${target}.xml\n\n sed -i.sed s#VM_NAME#${target}#g ${target}.xml\n sed -i.sed s#EXTERNAL_MAC#${PHD_VAR_network_nic_base}:0${lastoct}:00:${octet}#g ${target}.xml\n sed -i.sed s#INTERNAL_MAC#${PHD_VAR_network_nic_base}:0${lastoct}:01:${octet}#g ${target}.xml\n sed -i.sed s:source\\ file.*\\/:source\\ file=\\'/localvms/${target}.cow\\'\\/:g ${target}.xml\n diff -u template.xml ${target}.xml || true\n\n rm -f /localvms/${target}.cow /localvms/${target}-*.cow\n qemu-img create -b /localvms/${PHD_VAR_vm_base} -f qcow2 /localvms/${target}.cow\n\n virsh define ${target}.xml\n virsh start ${target}\n\n rm -f ${target}.xml.sed ${target}.xml\ndone\n\n....\n" }, { "path": "pcmk/vmsnap-rollback.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# We start with 3 (or more, up to 16) nodes running a minimal CentOS 6\n#\n# Tasks to be performed include:\n# - setting up the required repositories from which to download Openstack and the HA-Addon\n# - disabling firewalls and SElinux. This is a necessary evil until the proper policies can be written.\n# - creating network bridges for use by VMs hosting OpenStack services\n# - normalizing network interface names\n# - fixing multicast\n# - removing /home and making the root partition as large as possible to maximumize the amount of space available to openstack\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\nPHD_ENV_snapshot_name\nPHD_VAR_osp_configdir\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 3\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\ntarget=all\n....\n# get list of all VMs configured on a given host\nvmlist=$(virsh list --all --name)\n\nfor vm in $vmlist; do\n\t# if the requested snapshot doesn't exist, don't continue with\n\t# disruptive actions\n\tif [ ! -f /localvms/${vm}-${PHD_ENV_snapshot_name}.cow ]; then\n\t\techo \"ERROR: Snapshot ${PHD_ENV_snapshot_name} does not exists\"\n\t\texit 1\n\tfi\n\n\tcurrent=$(virsh snapshot-current --name $vm)\n\tif [ \"$current\" = ${PHD_ENV_snapshot_name} ]; then\n\t\techo \"ERROR: We are already at snapshot: ${PHD_ENV_snapshot_name}\"\n\t\texit 1\n\tfi\n\n\t# make sure that the we are rolling back to a snap\n\t# that is parent of the current\n\tsnaplist=$(virsh snapshot-list --tree $vm | sed -e 's#.*|.*##g' -e 's#.*+- ##g')\n\tfor snap in $snaplist; do\n\t\tif [ \"$snap\" = \"${PHD_ENV_snapshot_name}\" ]; then\n\t\t\t# we found requested snap before current and we are good\n\t\t\techo \"found snap ${PHD_ENV_snapshot_name}\"\n\t\t\tbreak;\n\t\tfi\n\t\tif [ \"$snap\" = \"$current\" ]; then\n\t\t\t# current is before the requested snap, error\n\t\t\techo \"ERROR: Requested snap ${PHD_ENV_snapshot_name} is not a parent of current $current snap\"\n\t\t\texit 1\n\t\tfi\n\tdone\ndone\n\n....\n\ntarget=all\n....\n# to roll back we need to kill hard the VMs but we want to make\n# sure that the other nodes won't fence before we get to\n# revert properly\nsystemctl stop fence_virtd\n\nvmlist=$(virsh list --all --name)\nfor vm in $vmlist; do\n\trollback=\"\"\n\n\tvirsh destroy $vm\n\n\tsnaplist=$(virsh snapshot-list --tree $vm | sed -e 's#.*|.*##g' -e 's#.*+- ##g')\n\n\t# we need to proceed from the bottom of the list\n\tfor snap in $snaplist; do\n\t\trollback=\"$snap $rollback\"\n\tdone\n\n\t# kill all disks\n\tfor snap in $rollback; do\n\t\tif [ \"$snap\" = \"${PHD_ENV_snapshot_name}\" ]; then\n\t\t\tbreak;\n\t\tfi\n\t\trm -f /localvms/${vm}-${snap}.cow\n\t\t# needs extra cleaning\n\t\tcase $snap in\n\t\t\tcinder)\n\t\t\t\trm -rf $PHD_VAR_osp_configdir/cinder\n\t\t\t;;\n\t\t\tglance)\n\t\t\t\trm -rf $PHD_VAR_osp_configdir/glance\n\t\t\t;;\n\t\t\tcompute)\n\t\t\t\trm -rf $PHD_VAR_osp_configdir/instances/*\n\t\t\t;;\n\t\tesac\n\tdone\n\n\t# drop all childs\n\tvirsh snapshot-delete --domain $vm ${PHD_ENV_snapshot_name} --metadata --children-only\n\n\tvirsh detach-disk $vm vda --config\n\tvirsh attach-disk $vm /localvms/$vm-${PHD_ENV_snapshot_name}.cow vda \\\n\t\t--sourcetype file --type disk \\\n\t\t--driver qemu --subdriver qcow2 \\\n\t\t--config\n\ndone\n\n....\n\ntarget=all\n....\n\nsystemctl start fence_virtd\n# get list of all VMs configured on a given host\nvmlist=$(virsh list --all --name)\n\nfor vm in $vmlist; do\n\tvirsh start $vm\ndone\n....\n" }, { "path": "pcmk/vmsnap.scenario", "content": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only requirement is a list\n# of nodes you'd like it to modify.\n#\n# The scope of each command-block is controlled by the preceeding\n# 'target' line. \n#\n# - target=all\n# The commands are executed on evey node provided\n#\n# - target=local\n# The commands are executed from the node hosting phd. When not\n# using phd, they should be run from some other independant host\n# (such as the puppet master)\n#\n# - target=$PHD_ENV_nodes{N}\n# The commands are executed on the Nth node provided.\n# For example, to run on only the first node would be target=$PHD_ENV_nodes1\n#\n# We start with 3 (or more, up to 16) nodes running a minimal CentOS 6\n#\n# Tasks to be performed include:\n# - setting up the required repositories from which to download Openstack and the HA-Addon\n# - disabling firewalls and SElinux. This is a necessary evil until the proper policies can be written.\n# - creating network bridges for use by VMs hosting OpenStack services\n# - normalizing network interface names\n# - fixing multicast\n# - removing /home and making the root partition as large as possible to maximumize the amount of space available to openstack\n\n#################################\n# Scenario Requirements Section #\n#################################\n= VARIABLES =\nPHD_ENV_snapshot_name\n\n#################################\n# Scenario Requirements Section #\n#################################\n= REQUIREMENTS =\nnodes: 3\n\n######################\n# Deployment Scripts #\n######################\n= SCRIPTS =\n\n# links to snapshot management:\n# https://kashyapc.fedorapeople.org/virt/infra.next-2015/Advanced-Snapshots-with-libvirt-and-QEMU.pdf\n# http://lists.openstack.org/pipermail/openstack-dev/2013-June/010212.html\n# https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Deployment_and_Administration_Guide/index.html#warning-live-snapshots\n\ntarget=all\n....\n# get list of all VMs configured on a given host\nvmlist=$(virsh list --all --name)\n\n# check if snapshot already exists, otherwise take the snapshot\nfor vm in $vmlist; do\n\tif [ -f /localvms/${vm}-${PHD_ENV_snapshot_name}.cow ]; then\n\t\techo \"Snapshot ${PHD_ENV_snapshot_name} already exists\"\n\t\texit 0\n\tfi\n\tvirsh snapshot-create-as --domain $vm ${PHD_ENV_snapshot_name} --diskspec vda,file=/localvms/${vm}-${PHD_ENV_snapshot_name}.cow,snapshot=external --disk-only --atomic\ndone\n....\n" } ]