Repository: beekhof/osp-ha-deploy Branch: master Commit: 2373fbfd39fc Files: 102 Total size: 491.7 KB Directory structure: gitextract_4wkbeqjk/ ├── HA-keepalived.md ├── README.md ├── build-all.sh ├── full-stack.md ├── ha-openstack.md ├── iha-install-10.sh ├── iha-install-9.sh ├── iha-uninstall.sh ├── keepalived/ │ ├── ceilometer-config.md │ ├── cinder-config.md │ ├── compute-node.md │ ├── controller-node.md │ ├── galera-bootstrap.md │ ├── galera-config.md │ ├── glance-config.md │ ├── haproxy-config.md │ ├── heat-config.md │ ├── horizon-config.md │ ├── keepalived-config.md │ ├── keystone-config.md │ ├── memcached-config.md │ ├── mongodb-config.md │ ├── mongodb-recovery.md │ ├── neutron-config.md │ ├── nova-config.md │ ├── phd-setup/ │ │ ├── ceilometer.scenario │ │ ├── cinder.scenario │ │ ├── compute.scenario │ │ ├── galera.scenario │ │ ├── glance.scenario │ │ ├── ha-collapsed.variables │ │ ├── heat.scenario │ │ ├── horizon.scenario │ │ ├── hypervisors.scenario │ │ ├── keepalived.scenario │ │ ├── keystone.scenario │ │ ├── lb.scenario │ │ ├── memcached.scenario │ │ ├── mongodb.scenario │ │ ├── neutron.scenario │ │ ├── nova.scenario │ │ ├── rabbitmq.scenario │ │ ├── readme.txt │ │ ├── redis.scenario │ │ ├── sahara.scenario │ │ ├── serverprep.scenario │ │ ├── swift.scenario │ │ ├── test.sh │ │ └── trove.scenario │ ├── rabbitmq-config.md │ ├── rabbitmq-restart.md │ ├── redis-config.md │ ├── sahara-config.md │ ├── swift-config.md │ └── trove-config.md ├── make-vm └── pcmk/ ├── NovaCompute ├── NovaEvacuate ├── baremetal-rollback.scenario ├── baremetal.scenario ├── basic-cluster.scenario ├── beaker.scenario ├── ceilometer-test.sh ├── ceilometer.scenario ├── cinder-test.sh ├── cinder.scenario ├── compute-cluster.scenario ├── compute-common.scenario ├── compute-managed.scenario ├── controller-managed.scenario ├── galera-test.sh ├── galera.scenario ├── gateway.scenario ├── glance-test.sh ├── glance.scenario ├── ha-collapsed.variables ├── ha-segregated.variables ├── hacks.scenario ├── heat-test.sh ├── heat.scenario ├── horizon-test.sh ├── horizon.scenario ├── keystone-test.sh ├── keystone.scenario ├── lb.scenario ├── memcached.scenario ├── mongodb.scenario ├── mrg.variables ├── neutron-agents.scenario ├── neutron-server.scenario ├── neutron-test.sh ├── nova-test.sh ├── nova.scenario ├── nova_client.py ├── rabbitmq-test.sh ├── rabbitmq.scenario ├── swift-aco.scenario ├── swift-test.sh ├── swift.scenario ├── virt-hosts.scenario ├── vmsnap-rollback.scenario └── vmsnap.scenario ================================================ FILE CONTENTS ================================================ ================================================ FILE: HA-keepalived.md ================================================ Introduction ------------ This document aims at defining a high level architecture for a highly available OpenStack setup using application-native options and keepalived. It will document the overall setup, the architecture limitations and any important item to keep in mind when deploying. The document can be used to create a highly available architecture for: - [RDO Liberty](http://www.rdoproject.org) - [Red Hat Enterprise Linux Openstack Platform 8](http://www.redhat.com/openstack), when available **Note**: Instructions for Liberty are still work in progress. If you are looking for the Kilo edition, check out the [Kilo-RDO7 branch](../Kilo-RDO7/HA-keepalived.md). Most of the time the instructions will apply to both distributions. If there is any difference, it will be specified throughout the document. ### Authors and changelog - Javier Peña (jpena@redhat.com) commited the initial version. Architecture overview --------------------- ### Scope This document will define, at a high level, the placement and high availability strategy for the different OpenStack services, as well as the limitations of the architecture. Service monitoring and recovery are outside the scope for this document. You may choose to monitor and recover the running services health using any generally available monitoring tool, or implement automated service restart in systemd. ### Server roles A typical OpenStack architecture will consist of servers performing various roles. The following roles may be spread over different servers or collapsed into some of them: - Controller nodes - Load balancer nodes - Database nodes - Network nodes - Storage nodes - Compute nodes High availability for storage and compute nodes is out of the scope for this document. ### High availability strategy The following diagram shows a very simplified view of the different strategies used to achieve high availability for the OpenStack services: ![](keepalived/Highlevelarch.jpg "High level architecture") Depending on the method used to communicate with the service, the following availability strategies will be followed: - Keepalived, for the HAProxy instances. - Access via an HAProxy virtual IP, for services accessed via a TCP socket than can be load balanced (e.g. httpd). - Built-in application clustering, when available from the application (e.g. Galera) . - Starting up one instance of the service on several controller nodes, when they can coexist and coordinate by other means (e.g. RPC, in the case of nova-conductor). - No high availability, when the service can only work in active/passive mode. The detailed high availability strategy for the OpenStack services is defined in the following table. | Service | Process | Mode | HA stragegy | |------------------|----------------------------|:------:|-------------| | Support services |MariaDB - Galera | A/A | HAProxy / app cluster | | Support services |RabbitMQ | A/A | App cluster / service config | | Support services |HAProxy | A/A | Keepalived | | Support services |MongoDB | A/A | App cluster | | Support services |Memcached | A/A | Service configuration | | Support services |Redis | A/A | App cluster (Sentinel)| | Keystone |openstack-keystone | A/A | HAProxy | | Glance |openstack-glance-api | A/A | HAProxy | | Glance |openstack-glance-registry | A/A | HAProxy | | Nova |openstack-nova-api | A/A | HAProxy | | Nova |openstack-nova-cert | A/A | | | Nova |openstack-nova-compute | A/A | | | Nova |openstack-nova-scheduler | A/A | | | Nova |openstack-nova-conductor | A/A | | | Nova |openstack-nova-novncproxy | A/A | HAProxy | | Cinder |openstack-cinder-api | A/A | HAProxy | | Cinder |openstack-cinder-scheduler | A/A | | | Cinder |openstack-cinder-volume | **A/P**| No HA | | Cinder |openstack-cinder-backup | A/A | | | Neutron |neutron-server | A/A | HAProxy | | Neutron |neutron-dhcp-agent | A/A | Multiple DHCP agents | | Neutron |neutron-l3-agent | A/A | L3 HA | | Neutron |neutron-metadata-agent | A/A | | | Neutron |neutron-lbaas-agent | **A/P**| | | Neutron |neutron-openvswitch-agent | A/A | | | Neutron |neutron-metering-agent | A/A | | | Horizon |httpd | A/A | HAProxy | | Ceilometer |openstack-ceilometer-api | A/A | HAProxy | | Ceilometer |openstack-ceilometer-central| A/A | Workload partitioning: tooz + Redis| | Ceilometer |openstack-ceilometer-compute| A/A | | | Ceilometer |openstack-ceilometer-alarm-notifier| A/A | | | Ceilometer |openstack-ceilometer-evaluator| A/A | | | Ceilometer |openstack-ceilometer-notification| A/A | | | Heat |openstack-heat-api | A/A | HAProxy | | Heat |openstack-heat-cfn | A/A | | | Heat |openstack-heat-cloudwatch | A/A | | | Heat |openstack-heat-engine | A/A | | | Swift |openstack-swift-proxy | A/A | HAProxy | | Swift |openstack-swift-account | A/A | HAProxy | | Swift |openstack-swift-container | A/A | HAProxy | | Swift |openstack-swift-object | A/A | HAProxy | | Sahara |openstack-sahara-api | A/A | HAProxy | | Sahara |openstack-sahara-engine | A/A | | | Trove |openstack-trove-api | A/A | HAProxy | | Trove |openstack-trove-engine | A/A | | | Trove |openstack-trove-conductor | A/A | | **Notes:** 1. There are known issues with cinder-volume that recommend setting it as active-passive for now, see 2. While there will be multiple Neutron LBaaS agents running, each agent will manage a set of load balancers, that cannot be failed over to another node. Architecture limitations ------------------------ This architecture has some inherent limitations that should be kept in mind during deployment and daily operations. The following sections describe those limitations. ### Keepalived and network partitions In case of a network partitioning, there is a chance that two or more nodes running keepalived claim to hold the same VIP, which may lead to an undesired behaviour. Since keepalived uses VRRP over multicast to elect a master (VIP owner), a network partition in which keepalived nodes cannot communicate will result in the VIPs existing on two nodes. When the network partition is resolved, the duplicate VIPs should also be resolved. Note that this network partition problem with VRRP is a known limitation for this architecture. ### Cinder-volume as a single point of failure There are currently concerns over the cinder-volume service ability to run as a fully active-active service. During the Mitaka timeframe, this is being worked on, see [1](https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support). Thus, cinder-volume will only be running on one of the controller nodes, even if it will be configured on all nodes. In case of a failure in the node running cinder-volume, it should be started in a surviving controller node. ### Neutron-lbaas-agent as a single point of failure The current design of the Neutron LBaaS agent using the HAProxy driver does not allow high availability for the tenant load balancers. The neutron-lbaas-agent service will be enabled and running on all controllers, allowing for load balancers to be distributed across all nodes. However, a controller node failure will stop all load balancers running on that node until the service is recovered or the load balancer is manually removed and created again. ### Service monitoring and recovery required An external service monitoring infrastructure is required to check the OpenStack service health, and notify operators in case of any failure. This architecture does not provide any facility for that, so it would be necessary to integrate the OpenStack deployment with any existing monitoring environment. ### Manual recovery after a full cluster restart Some support services used by RDO / RHEL OSP use their own form of application clustering. Usually, these services maintain a cluster quorum, that may be lost in case of a simultaneous restart of all cluster nodes, e.g. during a power outage. Each service will require its own procedure to regain quorum: - Galera: [Galera bootstrap instructions](keepalived/galera-bootstrap.md) - RabbitMQ: [RabbitMQ cluster restart](keepalived/rabbitmq-restart.md) - MongoDB: [MongoDB cluster recovery](keepalived/mongodb-recovery.md) Implementation -------------- The implementation will be split into two articles: - [Controller node implementation](keepalived/controller-node.md) - [Compute node implementation](keepalived/compute-node.md) ================================================ FILE: README.md ================================================ Introduction ------------ This repository contains the description of two highly available OpenStack architectures using RDO or Red Hat Enterprise Linux OpenStack Platform: - An architecture based on [Pacemaker](ha-openstack.md). - An architecture based on [application-native tools and Keepalived](HA-keepalived.md). Each architecture includes a description and implementation instructions. Be sure to understand the contrains and limitations of each architecture, and choose the one that better suits your needs. How you can help ---------------- Feedback is encouraged for this project. Feel free to submit patches and report issues. ================================================ FILE: build-all.sh ================================================ #!/bin/bash set -e declare -A nodeMap declare -A variables declare -A cluster nodeMap["baremetal"]="east-01 east-02 east-03 east-04 east-05 east-06 east-07" nodeMap["virt-hosts"]="east-01 east-02 east-03 east-04" nodeMap["galera"]="rdo7-db1.vmnet rdo7-db2.vmnet rdo7-db3.vmnet" nodeMap["memcached"]="rdo7-memcache1.vmnet rdo7-memcache2.vmnet rdo7-memcache3.vmnet" nodeMap["swift-aco"]="rdo7-swift-brick1.vmnet rdo7-swift-brick2.vmnet rdo7-swift-brick3.vmnet" nodeMap["compute-nodes"]="east-05 east-06 east-07" nodeMap["controller-managed"]="rdo7-node1.vmnet rdo7-node2.vmnet rdo7-node3.vmnet east-05 east-06 east-07" nodeMap["vmsnap"]="east-02 east-03 east-04" cluster["baremetal"]=0 cluster["gateway"]=0 cluster["virt-hosts"]=0 variables["nodes"]="" variables["network_domain"]="lab.bos.redhat.com" variables["deployment"]="collapsed" variables["status"]=0 variables["components"]="lb db rabbitmq memcache mongodb keystone glance cinder swift-brick swift neutron-server neutron-agents ceilometer heat" variables["scenarios-segregated"]="gateway virt-hosts vmsnap-hacks hacks vmsnap-lb lb vmsnap-galera galera vmsnap-rabbitmq rabbitmq vmsnap-memcached memcached vmsnap-mongodb mongodb vmsnap-keystone keystone vmsnap-glance glance vmsnap-cinder cinder vmsnap-swift-aco swift-aco vmsnap-swift swift vmsnap-neutron-server neutron-server vmsnap-neutron-agents neutron-agents vmsnap-nova nova vmsnap-ceilometer ceilometer vmsnap-heat heat vmsnap-horizon horizon compute-common vmsnap-compute compute-cluster" variables["scenarios-collapsed"]="gateway virt-hosts vmsnap-hacks hacks vmsnap-basic-cluster basic-cluster vmsnap-lb lb vmsnap-galera galera vmsnap-rabbitmq rabbitmq vmsnap-memcached memcached vmsnap-mongodb mongodb vmsnap-keystone keystone vmsnap-glance glance vmsnap-cinder cinder vmsnap-swift-aco swift-aco vmsnap-swift swift vmsnap-neutron-server neutron-server vmsnap-neutron-agents neutron-agents vmsnap-nova nova vmsnap-ceilometer ceilometer vmsnap-heat heat vmsnap-horizon horizon compute-common vmsnap-compute" function create_phd_definition() { scenario=$1 definition=$2 snapshot_name=$3 rm -f ${definition} nodes=${variables["nodes"]} if [ "x$nodes" = x ]; then nodes=${nodeMap[$scenario]} fi if [ "x$nodes" = "x" ]; then for n in `seq 1 3`; do nodes="$nodes rdo7-${scenario}${n}.vmnet" done fi nodelist="nodes=" for node in $nodes; do nodelist="${nodelist}${node}.${variables["network_domain"]} " done echo "$nodelist" >> ${definition} if [ -n "${snapshot_name}" ]; then echo "snapshot_name=${snapshot_name}" >> ${definition} fi cat ${definition} } generate=0 scenarios="" while true ; do case "$1" in --help|-h|-\?) echo "$0 " exit 0;; -n|--node) variables["nodes"]="${variables["nodes"]} $2"; shift; shift;; -c|--collapsed) variables["deployment"]="collapsed"; shift;; -s|--segregated) variables["deployment"]="segregated"; shift;; -f|--from) fromscenario=$2; shift; shift;; -t|--to) toscenario=$2; shift; shift;; -S|--status) variables["status"]=1; shift;; -g|--generate) generate=1; shift;; --mrg) variables["network_domain"]="mpc.lab.eng.bos.redhat.com" variables["config"]="mrg"; nodeMap["baremetal"]="mrg-01 mrg-02 mrg-03 mrg-04 mrg-07 mrg-08 mrg-09" nodeMap["virt-hosts"]="mrg-01 mrg-02 mrg-03 mrg-04" nodeMap["vmsnap"]="mrg-02 mrg-03 mrg-04" nodeMap["compute-nodes"]="mrg-07 mrg-08 mrg-09" nodeMap["controller-managed"]="rdo7-node1.vmnet rdo7-node2.vmnet rdo7-node3.vmnet mrg-07 mrg-08 mrg-09" shift;; -m|--method) redeploy=$2; shift; shift;; -i|--instance) instance=$2; shift; shift;; -x) set -x ; shift;; --) shift ; break ;; -*) echo "unknown option: $1"; exit 1;; "") break;; *) scenarios="${scenarios} $1"; shift;; esac done if [ -z ${variables["config"]} ]; then variables["config"]=ha-${variables["deployment"]} fi if [ ${variables["status"]} = 1 ]; then if [ ${variables["deployment"]} != "collapsed" ]; then scenarios=node elif [ "x${scenarios}" = x ]; then scenarios=${variables["components"]} fi for scenario in $scenarios; do ssh rdo7-${scenario}1.vmnet.${variables["network_domain"]} -- crm_mon -1 done exit 0 fi if [ "x${scenarios}" = x ]; then deploy=scenarios-${variables["deployment"]} scenarios=${variables[${deploy}]} if [ -z "$redeploy" ]; then redeploy=full fi case $redeploy in rollback) scenarios="baremetal-rollback $scenarios";; full) scenarios="beaker baremetal $scenarios";; *) echo "unknown redeploy method"; exit 1;; esac if [ -z "$instance" ]; then instance=ha fi case $instance in ha) scenarios="$scenarios compute-managed controller-managed";; single) scenarios="$scenarios compute-cluster";; *) echo "unknown instance value"; exit 1;; esac fi function run_phd() { if [ ${generate} = 1 ]; then scripts=$(phd_exec -s ./pcmk/${1}.scenario -d ${HOME}/phd.${scenario}.conf -V ./pcmk/${variables["config"]}.variables -p | grep PHD_SCPT | sort | awk -F= '{print $2}') for script in $scripts; do echo "#### $script" more "$script" done else phd_exec -s ./pcmk/${1}.scenario -d ${HOME}/phd.${scenario}.conf -V ./pcmk/${variables["config"]}.variables fi } inscenario=0 for scenario in $scenarios; do if [ -n "$fromscenario" ]; then if [ "$fromscenario" = "${scenario}" ]; then inscenario=1 fi if [ "$inscenario" = 0 ]; then continue; fi fi snapshot_name="" if [ ${variables["deployment"]} = "collapsed" ]; then case $scenario in vmsnap-rollback-*) snapshot_name=$(echo $scenario | sed -e 's/vmsnap-rollback-//g') scenario=vmsnap-rollback nodeMap[$scenario]=${nodeMap[vmsnap]} ;; vmsnap-*) snapshot_name=$(echo $scenario | sed -e 's/vmsnap-//g') scenario=vmsnap ;; compute-*) nodeMap[$scenario]=${nodeMap[compute-nodes]} ;; beaker|baremetal|gateway) nodeMap[$scenario]=${nodeMap[baremetal]} ;; baremetal-rollback) # stupid hack to avoid spending heaps of time waiting for nfs # server timeout list="" for x in ${nodeMap[baremetal]}; do list="$x $list" done nodeMap[$scenario]="$list" ;; virt-hosts|controller-managed) ;; *) # Overwrite the node list to be the nodes of our collapsed cluster nodeMap[$scenario]="rdo7-node1.vmnet rdo7-node2.vmnet rdo7-node3.vmnet" ;; esac fi create_phd_definition ${scenario} ${HOME}/phd.${scenario}.conf ${snapshot_name} if [ x${cluster[${scenario}]} = x0 ]; then : no need to bootstrap a cluster elif [ ${variables["deployment"]} != "collapsed" ]; then : prep a new cluster for ${scenario} echo "$(date) :: Initializing cluster for scenario $scenario" run_phd basic-cluster fi echo "$(date) :: Beginning scenario $scenario" run_phd ${scenario} if [ -n "$toscenario" ] && [ "$toscenario" = ${scenario} ]; then echo "$(date) :: Reached $scenario. Stop processing" break; fi done ================================================ FILE: full-stack.md ================================================ This is what the result of `pcs status` should look like: Cluster name: rhos-node Last updated: Fri Mar 6 22:06:28 2015 Last change: Fri Mar 6 22:03:52 2015 Stack: corosync Current DC: rhos6-node2 (2) - partition with quorum Version: 1.1.12-a14efad 3 Nodes configured 121 Resources configured Online: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Full list of resources: fence1 (stonith:fence_xvm): Started rhos6-node1 fence2 (stonith:fence_xvm): Started rhos6-node2 fence3 (stonith:fence_xvm): Started rhos6-node3 Clone Set: lb-haproxy-clone [lb-haproxy] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] vip-db (ocf::heartbeat:IPaddr2): Started rhos6-node1 vip-qpid (ocf::heartbeat:IPaddr2): Started rhos6-node2 vip-keystone (ocf::heartbeat:IPaddr2): Started rhos6-node3 vip-glance (ocf::heartbeat:IPaddr2): Started rhos6-node1 vip-cinder (ocf::heartbeat:IPaddr2): Started rhos6-node2 vip-swift (ocf::heartbeat:IPaddr2): Started rhos6-node3 vip-neutron (ocf::heartbeat:IPaddr2): Started rhos6-node1 vip-nova (ocf::heartbeat:IPaddr2): Started rhos6-node2 vip-horizon (ocf::heartbeat:IPaddr2): Started rhos6-node3 vip-heat (ocf::heartbeat:IPaddr2): Started rhos6-node1 vip-ceilometer (ocf::heartbeat:IPaddr2): Started rhos6-node2 vip-rabbitmq (ocf::heartbeat:IPaddr2): Started rhos6-node3 Master/Slave Set: galera-master [galera] Masters: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: mongodb-clone [mongodb] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: memcached-clone [memcached] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: rabbitmq-server-clone [rabbitmq-server] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: keystone-clone [keystone] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: glance-fs-clone [glance-fs] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: glance-registry-clone [glance-registry] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: glance-api-clone [glance-api] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: cinder-api-clone [cinder-api] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: cinder-scheduler-clone [cinder-scheduler] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] cinder-volume (systemd:openstack-cinder-volume): Started rhos6-node1 Clone Set: swift-fs-clone [swift-fs] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: swift-account-clone [swift-account] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: swift-container-clone [swift-container] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: swift-object-clone [swift-object] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: swift-proxy-clone [swift-proxy] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] swift-object-expirer (systemd:openstack-swift-object-expirer): Started rhos6-node2 Clone Set: neutron-server-clone [neutron-server] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: neutron-scale-clone [neutron-scale] (unique) neutron-scale:0 (ocf::neutron:NeutronScale): Started rhos6-node3 neutron-scale:1 (ocf::neutron:NeutronScale): Started rhos6-node1 neutron-scale:2 (ocf::neutron:NeutronScale): Started rhos6-node2 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: neutron-l3-agent-clone [neutron-l3-agent] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] ceilometer-central (systemd:openstack-ceilometer-central): Started rhos6-node3 Clone Set: ceilometer-collector-clone [ceilometer-collector] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: ceilometer-api-clone [ceilometer-api] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: ceilometer-delay-clone [ceilometer-delay] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: ceilometer-alarm-evaluator-clone [ceilometer-alarm-evaluator] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: ceilometer-alarm-notifier-clone [ceilometer-alarm-notifier] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: ceilometer-notification-clone [ceilometer-notification] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: heat-api-clone [heat-api] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: heat-api-cfn-clone [heat-api-cfn] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] Clone Set: heat-api-cloudwatch-clone [heat-api-cloudwatch] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] heat-engine (systemd:openstack-heat-engine): Started rhos6-node1 Clone Set: horizon-clone [horizon] Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ] PCSD Status: rhos6-node1: Online rhos6-node2: Online rhos6-node3: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled ================================================ FILE: ha-openstack.md ================================================ # Highly Available Openstack Deployments The current target for this document is RDO 10, based on the OpenStack Newton release. Looking for an edition prior to Newton (RDO10)? Check out the [Juno-RDO6](../Juno-RDO6/ha-openstack.md) or [Mitaka-RDO9](../Mitaka-RDO9/ha-openstack.md) branches instead. ## Purpose of this Document This document aims at defining a high level architecture for a highly available RHEL OSP setup with the [Pacemaker](http://clusterlabs.org) cluster manager which provides: - detection and recovery of machine and application-level failures - startup/shutdown ordering between applications - preferences for other applications that must/must-not run on the same machine - provably correct response to any failure or cluster state It is important to understand the following definitions used to describe the operational mode of services in a cluster: - Active/active Traffic intended for the failed node is either passed onto an existing node or load balanced across the remaining nodes. This is usually only possible when the nodes use a homogeneous software configuration. - Active/passive Provides a fully redundant instance of each node, which is only brought online when its associated primary node fails. This configuration typically requires the most extra hardware. In this document, all components are currently modelled as active/active with the exception of: - cinder-volume Implementation details are contained in scripts linked to from the main document. Read them carefully before considering to run them in your own environment. ## Historical Context In the previous OpenStack HA architectures used by Red Hat, SuSE and others, Systemd is the entity in charge of starting and stopping most OpenStack services. Pacemaker exists as a layer on top, signalling when this should happen, but Systemd is the part making it happen. This is a valuable contribution for active/passive (A/P) services and those that require all their dependancies be available during their startup and shutdown sequences. However as OpenStack has matured, more and more components are able to operate in an unconstrained active/active capacity with little regard for the startup/shutdown order of their peers or dependancies - making them well suited to be managed exclusively by Systemd. ## Overall Design With Newton, OpenStack has reached the point where it is now a good idea to limit Pacemaker’s involvement to core services like Galera and Rabbit as well as the few remaining OpenStack services, such as cinder-volume, that run A/P. This will be particularly useful as we look towards a containerised future. It both allows OpenStack to play nicely with the current generation of container managers which lack Orchestration capabilities, as well as reducing recovery and down time by allowing for the maximum possible parallelism. Any objections to this architecture usually fall into one of three main categories: 1. The use of Pacemaker as an alerting mechanism 1. The idea that Pacemaker provides better monitoring of systemd services 1. A believe that active/passive installations are suprior If these concerns apply to you then, as the founding author of Pacemaker, I would like to direct your attention to my [post](http://blog.clusterlabs.org/blog/2016/next-openstack-ha-arch) which will attempt to disuade you of their relevance. This reference design is based around a single cluster of 3 or more nodes on which every component is running. This scenario can be visualized as below: ![Collapsed deployment architecture](Cluster-deployment-collapsed.png) With the advent of composable roles however, it is certainly possible to dedicate a subset of nodes for one or more components that are expected to be a bottleneck. It is also possible that these dedicated nodes run extra copies of those service, in addition to the ones on a fully symmetrical core set of nodes. ## Assumptions It is required that the clusters contain at least three nodes so that we take advantage of [quorum](http://en.wikipedia.org/wiki/Quorum_(Distributed_Systems)) Quorum becomes important when a failure causes the cluster to split in two or more paritions. In this situation, you want the majority to ensure the minority are truely dead (through fencing) and continue to host resources. For a two-node cluster, no side has the majority and you can end up in a situations where both sides fence each other, or both sides are running the same services - leading to data corruption. Clusters with an even number of hosts suffer from similar issues - a single network failure could easily cause a N:N split where neither side retains a majority. For this reason, we recommend an odd number of cluster members when scaling up. You can have up to 16 cluster members (this is currently limited by corosync's ability to scale higher). In extreme cases, 32 and even up to 64 nodes could be possible however this is not well tested. In some environments, the available IP address range of the public LAN is limited. If this applies to you, you will need one additional node to set up as a [gateway](pcmk/gateway.scenario) that will provide DNS and DHCP for the guests containing the OpenStack services and expose the required nova and horizon APIs to the external network. ## Solution Components ### Cluster Manager At its core, a cluster is a distributed finite state machine capable of co-ordinating the startup and recovery of inter-related services across a set of machines. Even a distributed and/or replicated application that is able to survive failures on one or more machines can benefit from a cluster manager: 1. Awareness of other applications in the stack While SYS-V init replacements like systemd can provide deterministic recovery of a complex stack of services, the recovery is limited to one machine and lacks the context of what is happening on other machines - context that is crucial to determine the difference between a local failure, clean startup and recovery after a total site failure. 1. Awareness of instances on other machines Services like RabbitMQ and Galera have complicated boot-up sequences that require co-ordination, and often serialization, of startup operations across all machines in the cluster. This is especially true after site-wide failure or shutdown where we must first determine the last machine to be active. 1. A shared implementation and calculation of [quorum](http://en.wikipedia.org/wiki/Quorum_%28Distributed_Systems%29) It is very important that all members of the system share the same view of who their peers are and whether or not they are in the majority. Failure to do this leads very quickly to an internal [split-brain](https://en.wikipedia.org/wiki/Split-brain_(computing)) state - where different parts of the system are pulling in different and incompatioble directions. 1. Data integrity through fencing (a non-responsive process does not imply it is not doing anything) A single application does not have sufficient context to know the difference between failure of a machine and failure of the applcation on a machine. The usual practice is to assume the machine is dead and carry on, however this is highly risky - a rogue process or machine could still be responding to requests and generally causing havoc. The safer approach is to make use of remotely accessible power switches and/or network switches and SAN controllers to fence (isolate) the machine before continuing. 1. Automated recovery of failed instances While the application can still run after the failure of several instances, it may not have sufficient capacity to serve the required volume of requests. A cluster can automatically recover failed instances to prevent additional load induced failures. For this reason, the use of a cluster manager like [Pacemaker](http://clusterlabs.org) is highly recommended. ### Proxy server Almost all services in this stack are proxied. Using a proxy server provides: 1. Load distribution Many services can act in an active/active capacity, however they usually require an external mechanism for distributing requests to one of the available instances. The proxy server can serve this role. 1. API isolation By sending all API access through the proxy, we can clearly identify service interdependancies. We can also move them to locations other than `localhost` to increase capacity if the need arises. 1. Simplified process for adding/removing of nodes Since all API access is directed to the proxy, adding or removing nodes has no impact on the configuration of other services. This can be very useful in upgrade scenarios where an entirely new set of machines can be configured and tested in isolation before telling the proxy to direct traffic there instead. 1. Enhanced failure detection The proxy can be configured as a secondary mechanism for detecting service failures. It can even be configured to look for nodes in a degraded state (such as being 'too far' behind in the replication) and take them out of circulation. The following components are currently unable to benefit from the use of a proxy server: - RabbitMQ - memcached - mongodb However the reasons vary and are discussed under each component's heading. We recommend HAProxy as the load balancer, however there are many alternatives in the marketplace. We use a check interval of 1 second however the timeouts vary by service. Generally we use round-robin to distriute load amongst instances of active/active services, however Galera and Qpid use the `stick-table` options to ensure that incoming connections to the virtual IP (VIP) should be directed to only one of the available backends. In Galera's case, although it can run active/active, this helps avoid lock contention and prevent deadlocks. It is used in combination with the `httpchk` option that ensures only nodes that are in sync with its peers are allowed to handle requests. Qpid however operates in a active/passive configuration, no built-in clustering, so in it's case the `stick-table` option ensures that all requests go to the active instance. ### Replicated Database Most OpenStack components require access to a database. To avoid the database being a single point of failure, we require that it be replicated and the ability to support multiple masters can help when trying to scale other components. One of the most popular database choices is Galera for MySQL, it supports: - Synchronous replication - active/active multi-master topology - Automatic node joining - True parallel replication, on row level - Direct client connections, native MySQL look & feel and claims: - No slave lag - No lost transactions - Both read and write scalability - Smaller client latencies Although galera supports active/active configurations, we recommend active/passive (enforced by the load balancer) in order to avoid lock contention. ### Database Cache Memcached is a general-purpose distributed memory caching system. It is used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source must be read. __Note__: Access to memcached is not handled by HAproxy because replicated access is currently only in an experimental state. Instead consumers must be supplied with the full list of hosts running memcached. ### Message Bus An AMQP (Advanced Message Queuing Protocol) compliant message bus is required for most OpenStack components in order to co-ordinate the execution of jobs entered into the system. RabbitMQ and Qpid are common deployment options. Both support: - reliable message delivery - flexible routing options - replicated queues This guide assumes RabbitMQ is being deployed, however we also [document Qpid (TODO)](pcmk/osp-qpid.scenario) for completeness. Pay attention to the comments in that guide for how selecting `Qpid` affects the rest of the configuration. __Note__: Access to RabbitMQ is not handled by HAproxy. Instead consumers must be supplied with the full list of hosts running RabbitMQ with `rabbit_hosts` and `rabbit_ha_queues` options. Jock Eck found the [core issue](http://people.redhat.com/jeckersb/private/vip-failover-tcp-persist.html) and went into some detail regarding the [history and solution](http://john.eckersberg.com/improving-ha-failures-with-tcp-timeouts.html) on his blog. In summary though: > The source address for the connection from HAProxy back to the > client is the VIP address. However the VIP address is no longer > present on the host. This means that the network (IP) layer deems > the packet unroutable, and informs the transport (TCP) layer. TCP, > however, is a reliable transport. It knows how to handle transient > errors and will retry. And so it does. In this case that is a problem though, because: > TCP generally holds on to hope for a long time. A ballpark estimate > is somewhere on the order of tens of minutes (30 minutes is commonly > referenced). During this time it will keep probing and trying to > deliver the data. > > It's important to note that HAProxy has no idea that any of this is > happening. As far as its process is concerned, it called write() > with the data and the kernel returned success. The [resolution](https://review.openstack.org/#/c/146047/) is already understood and just needs to make its way through review. ## Core OpenStack services In contrast to earlier versions of this guide, with the exception of Cinder Volume, there are no specific instructions with regards to the installation core OpenStack services beyond: 1. Ensuring services that make use of RabbitMQ list all configured servers 1. Accessing Galera and all OpenStack peer APIs (keystone, etc) via the HAProxy and the VIPs In all other respects, one should follow standard practices for installing packages and instructing the system to start them at boot time. ### Cinder Cinder provides 'block storage as a service' suitable for performance sensitive scenarios such as databases, expandable file systems, or providing a server with access to raw block level storage. Persistent block storage can survive instance termination and can also be moved across instances like any external storage device. Cinder also has volume snapshots capability for backing up the volumes. In theory cinder can be run as active/active however there are currently sufficient concerns that cause us to recommend running the volume component as active/passive only. Jon Bernard writes: > Requests are first seen by Cinder in the API service, and we have a > fundamental problem there - a standard test-and-set race condition > exists for many operations where the volume status is first checked > for an expected status and then (in a different operation) updated to > a pending status. The pending status indicates to other incoming > requests that the volume is undergoing a current operation, however it > is possible for two simultaneous requests to race here, which > undefined results. > > Later, the manager/driver will receive the message and carry out the > operation. At this stage there is a question of the synchronization > techniques employed by the drivers and what guarantees they make. > > If cinder-volume processes exist as different process, then the > 'synchronized' decorator from the lockutils package will not be > sufficient. In this case the programmer can pass an argument to > synchronized() 'external=True'. If external is enabled, then the > locking will take place on a file located on the filesystem. By > default, this file is placed in Cinder's 'state directory' in > /var/lib/cinder so won't be visible to cinder-volume instances running > on different machines. > > However, the location for file locking is configurable. So an > operator could configure the state directory to reside on shared > storage. If the shared storage in use implements unix file locking > semantics, then this could provide the requisite synchronization > needed for an active/active HA configuration. > > The remaining issue is that not all drivers use the synchronization > methods, and even fewer of those use the external file locks. > A sub-concern would be whether they use them correctly. You can read more about these concerns on the [Red Hat Bugzilla](https://bugzilla.redhat.com/show_bug.cgi?id=1193229) and there is a [psuedo roadmap](https://etherpad.openstack.org/p/cinder-kilo-stabilisation-work) for addressing them upstream. # Implementation The best way to visualize the result of this architecture is to make use of [tripleo-quickstart](https://github.com/openstack/tripleo-quickstart/blob/master/README.rst) which implements the described architecture. This will take a bare metal installation of your favorite OS (surely CentOS 7.2) and: 1. create a 'stack' user 1. create several VMs representing the undercloud, control plane and computes 1. deploy the undercloud (TripleO uses a pre-rolled OpenStack image as a means for deploying and updating the user facing installation of OpenStack aka. the overcloud) 1. deploy the overcloud for you to investigate and compare your existing architecture against You should digest the entire [README](https://github.com/openstack/tripleo-quickstart/blob/master/README.rst), however the quick version for the purposes of seeing this HA architecture in action is: git clone git@github.com:openstack/tripleo-quickstart.git cd tripleo-quickstart ./quickstart.sh -b -n -w $PWD -c config/general_config/ha.yml -p quickstart-extras.yml -r quickstart-extras-requirements.txt --tags all -R newton -T all ${the_machine_you_wish_to_install_to} For those that would prefer not to deal with TripleO, you can see roughly what TripleO does by examining the pseudo code for manually: 1. configuring a basic [pacemaker cluster](pcmk/basic-cluster.scenario) 1. deploying the [load balancer](pcmk/lb.scenario) 1. deploying [galera](pcmk/galera.scenario) 1. deploying [memcached](pcmk/memcached.scenario) 1. deploying [rabbitmq](pcmk/rabbitmq.scenario) 1. deploying [cinder volume](pcmk/cinder.scenario) Here is a [list of variables](pcmk/ha-collapsed.variables) used when executing the referenced scripts. Modify them to your needs. ## Disclaimer - The referenced scripts contain many comments and warnings - READ THEM CAREFULLY. - There are probably 2^8 other ways to deploy this same scenario. This is only one of them. - Due to limited number of available physical LAN connections in the test setup, the instance IP traffic overlaps with the internal/management network. - Distributed/Shared storage is provided via NFS from the commodity server due to lack of dedicated CEPH servers. Any other kind of storage supported by OpenStack would work just fine. - Bare metal could be used in place of any or all guests. - Most of the scripts contain shell expansion to automatically fill in some values. Use your common sense when parsing data. Example: `openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address $(ip addr show dev vmnet0 scope global | grep inet | sed -e 's#.*inet ##g' -e 's#/.*##g')` means that we want the IP address from vmnet0 as vncserver_proxyclient_address. # Compute Nodes We will usually need more than 16 compute nodes which is beyond Corosync's ability to manage. So in order monitor the healthiness of compute nodes and the services running on them, we previously had to create single node clusters. The current deployment model allows Pacemaker to continue this role, but presents a single coherent view of the entire deployment while allowing us to scale beyond corosync's limits. Having this single administrative domain then allows us to do clever things like automated recovery of VMs running on a failed or failing compute node. The main difference with the previous deployment mode is that services on the compute nodes are now managed and driven by the Pacemaker cluster on the control plane. The compute nodes do not become full members of the cluster and they no longer require the full cluster stack, instead they run pacemaker_remoted which acts as a conduit. > Implementation Details: > > - Pacemaker monitors the connection to pacemaker_remoted to verify > that the node is reachable or not. Failure to talk to a node > triggers recovery action. > > - Pacemaker uses pacemaker_remoted to start compute node services in > the same sequence as before (neutron-ovs-agent -> > ceilometer-compute -> nova-compute). > > - If a service fails to start, any services that depend on the > FAILED service will not be started. This avoids the issue of > adding a broken node (back) to the pool. > > - If a service fails to stop, the node where the service is running > will be fenced. This is necessary to guarantee data integrity and > a core HA concept (for the purposes of this particular discussion, > please take this as a given). > > - If a service's health check fails, the resource (and anything that > depends on it) will be stopped and then restarted. Remember that > failure to stop will trigger a fencing action. > > - A successful restart of all the services can only potentially > affect network connectivity of the instances for a short period of > time. With these capabilities in place, we can exploit Pacemaker's node monitoring and fencing capabilities to drive nova host-evacuate for the failed compute nodes and recover the VMs elsewhere. When a compute node fails, Pacemaker will: 1. Execute 'nova service-disable' 2. fence (power off) the failed compute node 3. fence_compute off (waiting for nova to detect compute node is gone) 4. fence_compute on (a no-op unless the host happens to be up already) 5. Execute 'nova service-enable' when the compute node returns Technically steps 1 and 5 are optional and they are aimed to improve user experience by immediately excluding a failed host from nova scheduling. The only benefit is a faster scheduling of VMs that happens during a failure (nova does not have to recognize a host is down, timeout and subsequently schedule the VM on another host). Step 2 will make sure the host is completely powered off and nothing is running on the host. Optionally, you can have the failed host reboot which would potentially allow it to re-enter the pool. We have an implementation for Step 3 but the ideal solution depends on extensions to the nova API. Currently fence_compute loops, waiting for nova to recognise that the failed host is down, before we make a host-evacuate call which triggers nova to restart the VMs on another host. The discussed nova API extensions will speed up recovery times by allowing fence_compute to proactively push that information into nova instead. To take advantage of the VM recovery features: - VMs need to be running off a cinder volume or using shared ephemeral storage (like RBD or NFS) - If VM is not running using shared storage, recovery of the instance on a new compute node would need to revert to a previously stored snapshot/image in Glance (potentially losing state, but in some cases that may not matter) - RHEL7.1+ required for infrastructure nodes (controllers and compute). Instance guests can run anything. - Compute nodes need to have a working fencing mechanism (IPMI, hardware watchdog, etc) ## Compute Node Implementation Start by creating a minimal CentOS __7__ installation on at least one node. Once the machine(s) have been installed, [prepare them](pcmk/baremetal.scenario) for hosting OpenStack. Next, you can configure them as [compute nodes](pcmk/compute-common.scenario). We now add them to the cluster as [partial members](pcmk/compute-managed.scenario). Once the compute nodes are configured as remote, they can be added to the [controller backplane](pcmk/controller-managed.scenario) > TODO: what if nova-compute fails to restart and there are scheduled > instances? Those can still be accessed from outside but cannot be > managed by nova. This might warrant a host-evacuate. > > Traditionally, HA systems would fence the node at this point. ================================================ FILE: iha-install-10.sh ================================================ #!/bin/bash set -ex # #If your deployment includes this use case, include the no_shared_storage=1 option in step 7. helper=~/.ssh/config cat <> $helper Host overcloud-* BatchMode yes User heat-admin StrictHostKeyChecking no EOF source stackrc nova list | grep ctlplane | awk -F\| '{ print "Host "$3 "\n\tHostName " $7}' | sed 's/ctlplane=//' >> $helper chmod 600 $helper COMPUTES=$(nova list | grep novacompute | awk -F\| '{ print $3}' | tr '\n' ' ') CONTROLLERS=$(nova list | grep controller | awk -F\| '{ print $3}' | tr '\n' ' ') FIRST_COMPUTE=$(echo $COMPUTES | awk '{print $1}') FIRST_CONTROLLER=$(echo $CONTROLLERS | awk '{print $1}') # 15. Add stonith devices for the Compute nodes. fence_ironic is not # a recommended approach, but is usable everywhere which makes it # ideal for generic scripts like this if [ 1 = 1 ]; then helper=iha-helper-configure-virt-fencing.sh cat < $helper cp fence_ironic.py /usr/sbin/fence_ironic sudo chmod a+x /usr/sbin/fence_ironic EOF for node in $CONTROLLERS; do scp $helper fence_ironic.py ${node}: ; done for node in $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done hostmap=""; hostlist=$(nova list | grep ctlplane | awk -F\| '{print $3}') for host in $hostlist; do hostmap="$hostmap $host:$(ironic node-list | grep $(nova list | grep $host | awk '{print $2}') | awk '{print $4}' )"; done ssh ${FIRST_CONTROLLER} -- sudo pcs stonith create shooter fence_ironic auth-url=${OS_AUTH_URL} login=${OS_USERNAME} passwd=${OS_PASSWORD} tenant-name=${OS_TENANT_NAME} pcmk_host_map=\"${hostmap}\" op monitor interval=60s timeout=180s fi # 1. Begin by stopping and disabling libvirtd and all OpenStack services on the Compute nodes: source overcloudrc ssh ${FIRST_CONTROLLER} -- sudo pcs property set stonith-enabled=false helper=iha-helper-stop-services.sh cat < $helper set -ex for s in openstack-nova-compute neutron-openvswitch-agent libvirtd; do systemctl stop \${s} systemctl disable \${s} done # Punch a hole for pacemaker-remote iptables -I INPUT -p tcp --dport 3121 -j ACCEPT service iptables save EOF for node in $COMPUTES; do scp $helper ${node}: ; done for node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done # 2. Create an authentication key for use with pacemaker-remote. dd if=/dev/urandom of=./authkey bs=4096 count=1 # 3. Copy this key to the director node, and then to the remaining Compute and Controller nodes: helper=iha-helper-fix-auth.sh cat < $helper set -ex mkdir -p --mode=0750 /etc/pacemaker/ chgrp haclient /etc/pacemaker mv authkey /etc/pacemaker/ chown root:haclient /etc/pacemaker/authkey EOF for node in $COMPUTES $CONTROLLERS; do scp ./authkey $helper ${node}: ; done for node in $COMPUTES $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done # 4. Enable pacemaker-remote on all Compute nodes: for compute in $COMPUTES; do ssh ${compute} -- sudo systemctl enable pacemaker_remote; done for compute in $COMPUTES; do ssh ${compute} -- sudo systemctl start pacemaker_remote; done # 5. Confirm that the required versions of the pacemaker (1.1.13-10.el7_2.2.x86_64), fence-agents (fence-agents-all-4.0.11-27.el7_2.5.x86_64) and resource-agents (3.9.5-54.el7_2.6.x86_64`) packages are installed on the controller and Compute nodes: for compute in $COMPUTES; do ssh ${compute} -- rpm -qa | egrep '(pacemaker|fence-agents|resource-agents)' ; done # 7. Create a NovaEvacuate active/passive resource using the overcloudrc file to provide the auth_url, username, tenant and password values: # 8. Confirm that nova-evacuate is started after the floating IP resources, and the Image Service (glance), OpenStack Networking (neutron), Compute (nova) services: helper=iha-helper-create-evacuate.sh cat < $helper set -ex pcs resource create nova-evacuate ocf:openstack:NovaEvacuate auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME domain=localdomain no_shared_storage=1 op monitor interval=60s timeout=240s sudo pcs constraint order start haproxy-clone then nova-evacuate sudo pcs constraint order start galera-clone then nova-evacuate sudo pcs constraint order start rabbitmq-clone then nova-evacuate EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # Note: If you are not using shared storage, include the no_shared_storage=1 option in your resource create ... command above. See Exception for shared storage for more information. # 10. Create a list of the current controllers using cibadmin data : #controller-1 # controllers=$(sudo cibadmin -Q -o nodes | grep uname | sed s/.*uname..// | awk -F\" '{print $1}') #controller-1 # echo $controllers # 11. Use this list to tag these nodes as controllers with the osprole=controller property: # 12. Build a list of stonith devices already present in the environment: # 13. Tag the control plane services to make sure they only run on the controllers identified above, skipping any stonith devices listed: helper=iha-helper-tag-controllers.sh cat < $helper set -ex for controller in ${CONTROLLERS}; do pcs property set --node \${controller} osprole=controller ; done stonithdevs=\$(pcs stonith | awk '{print \$1}') for i in \$(cibadmin -Q --xpath //primitive --node-path | tr ' ' '\n' | awk -F "id='" '{print \$2}' | awk -F "'" '{print \$1}' | uniq); do found=0 if [ -n "\$stonithdevs" ]; then for x in \$stonithdevs; do if [ \$x = \$i ]; then found=1 fi done fi if [ \$found = 0 ]; then pcs constraint location \$i rule resource-discovery=exclusive score=0 osprole eq controller fi done EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # 14. Begin to populate the Compute node resources within pacemaker, starting with neutron-openvswitch-agent: helper=fix-compute.sh cat < $helper set -x grep crudini /usr/lib/ocf/resource.d/openstack/nova-compute-wait if [ \$? != 0 ]; then set -e patch -p1 /usr/lib/ocf/resource.d/openstack/nova-compute-wait \${PWD}/bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch fi EOF for node in $COMPUTES; do scp bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch $helper ${node}: ; done for node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done source overcloudrc helper=iha-helper-create-compute-resources.sh cat < $helper set -ex pcs resource create neutron-openvswitch-agent-compute systemd:neutron-openvswitch-agent op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force pcs constraint location neutron-openvswitch-agent-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute # Then the Compute libvirtd resource: pcs resource create libvirtd-compute systemd:libvirtd op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force pcs constraint location libvirtd-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start neutron-openvswitch-agent-compute-clone then libvirtd-compute-clone pcs constraint colocation add libvirtd-compute-clone with neutron-openvswitch-agent-compute-clone # Then the nova-compute resource: pcs resource create nova-compute-checkevacuate ocf:openstack:nova-compute-wait auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME domain=localdomain op start timeout=300 --clone interleave=true --disabled --force pcs constraint location nova-compute-checkevacuate-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs resource create nova-compute systemd:openstack-nova-compute op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force pcs constraint location nova-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start nova-compute-checkevacuate-clone then nova-compute-clone require-all=true pcs constraint order start nova-compute-clone then nova-evacuate require-all=false pcs constraint order start libvirtd-compute-clone then nova-compute-clone pcs constraint colocation add nova-compute-clone with libvirtd-compute-clone EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # 16. Create a seperate fence-nova stonith device: helper=iha-helper-create-nova-fence.sh cat < $helper set -ex pcs stonith create fence-nova fence_compute auth-url=$OS_AUTH_URL login=$OS_USERNAME passwd=$OS_PASSWORD tenant-name=$OS_TENANT_NAME domain=localdomain record-only=1 op monitor interval=60s timeout=180s --force EOF source stackrc scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # 17. Make certain the Compute nodes are able to recover after fencing: ssh ${FIRST_CONTROLLER} -- sudo pcs property set cluster-recheck-interval=1min # 18. Create Compute node resources and set the stonith level 1 to include both the nodes's physical fence device and fence-nova: helper=iha-helper-create-computes.sh cat < $helper set -ex for node in $COMPUTES; do pcs resource create \${node} ocf:pacemaker:remote reconnect_interval=60 op monitor interval=20 pcs property set --node \${node} osprole=compute pcs stonith level add 1 \${node} shooter,fence-nova done EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # 19. Enable the control and Compute plane services: helper=iha-helper-enable-services.sh cat < $helper set -ex pcs resource enable neutron-openvswitch-agent-compute pcs resource enable libvirtd-compute pcs resource enable nova-compute-checkevacuate pcs resource enable nova-compute EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # 20. Allow some time for the environment to settle before cleaning up any failed resources: helper=iha-helper-cleanup.sh cat < $helper set -ex sleep 60 pcs resource cleanup pcs status echo pcs property set stonith-enabled=true EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper #Test High Availability #Note: These steps deliberately reboot the Compute node without warning. #1. The following step boots an instance on the overcloud, and then crashes the Compute node: #stack@director # . overcloudrc #stack@director # nova boot --image cirros --flavor 2 test-failover #stack@director # nova list --fields name,status,host #stack@director # . stackrc #stack@director # ssh -lheat-admin compute-n #compute-n # sudo su - #root@compute-n # echo c > /proc/sysrq-trigger # #2. A short time later, the instance should be restarted on a working Compute node: # #stack@director # nova list --fields name,status,host #stack@director # nova service-list # 3 ip6tables -I INPUT -p tcp --dport 3121 -j ACCEPT # 4 iptables -I INPUT -p tcp --dport 3121 -j ACCEPT # 5 service iptables save # 6 service ip6tables save ================================================ FILE: iha-install-9.sh ================================================ #!/bin/bash set -ex # #If your deployment includes this use case, include the no_shared_storage=1 option in step 7. helper=~/.ssh/config cat < $helper Host overcloud-* BatchMode yes User heat-admin StrictHostKeyChecking no EOF source stackrc nova list | grep ctlplane | awk -F\| '{ print "Host "$3 "\n\tHostName " $7}' | sed 's/ctlplane=//' >> $helper chmod 600 $helper COMPUTES=$(nova list | grep novacompute | awk -F\| '{ print $3}' | tr '\n' ' ') CONTROLLERS=$(nova list | grep controller | awk -F\| '{ print $3}' | tr '\n' ' ') FIRST_COMPUTE=$(echo $COMPUTES | awk '{print $1}') FIRST_CONTROLLER=$(echo $CONTROLLERS | awk '{print $1}') # 15. Add stonith devices for the Compute nodes. fence_ironic is not # a recommended approach, but is usable everywhere which makes it # ideal for generic scripts like this if [ 1 = 1 ]; then helper=iha-helper-configure-virt-fencing.sh cat < $helper cp fence_ironic.py /usr/sbin/fence_ironic sudo chmod a+x /usr/sbin/fence_ironic EOF for node in $CONTROLLERS; do scp $helper fence_ironic.py ${node}: ; done for node in $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done hostmap=$(nova list | grep ctlplane | awk -F\| '{print $3 ":" $2}' | tr -d ' ' | tr '\n' ' ') ssh ${FIRST_CONTROLLER} -- sudo pcs stonith create shooter fence_ironic auth-url=${OS_AUTH_URL} login=${OS_USERNAME} passwd=${OS_PASSWORD} tenant-name=${OS_TENANT_NAME} pcmk_host_map=\"${hostmap}\" op monitor interval=60s fi # 1. Begin by stopping and disabling libvirtd and all OpenStack services on the Compute nodes: source overcloudrc ssh ${FIRST_CONTROLLER} -- sudo pcs property set stonith-enabled=false helper=iha-helper-stop-services.sh cat < $helper set -ex openstack-service stop openstack-service disable systemctl stop libvirtd systemctl disable libvirtd EOF for node in $COMPUTES; do scp $helper ${node}: ; done for node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done # 2. Create an authentication key for use with pacemaker-remote. dd if=/dev/urandom of=./authkey bs=4096 count=1 # 3. Copy this key to the director node, and then to the remaining Compute and Controller nodes: helper=iha-helper-fix-auth.sh cat < $helper set -ex mkdir -p --mode=0750 /etc/pacemaker/ chgrp haclient /etc/pacemaker mv authkey /etc/pacemaker/ chown root:haclient /etc/pacemaker/authkey EOF for node in $COMPUTES $CONTROLLERS; do scp ./authkey $helper ${node}: ; done for node in $COMPUTES $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done # 4. Enable pacemaker-remote on all Compute nodes: for compute in $COMPUTES; do ssh ${compute} -- sudo systemctl enable pacemaker_remote; done for compute in $COMPUTES; do ssh ${compute} -- sudo systemctl start pacemaker_remote; done # 5. Confirm that the required versions of the pacemaker (1.1.13-10.el7_2.2.x86_64), fence-agents (fence-agents-all-4.0.11-27.el7_2.5.x86_64) and resource-agents (3.9.5-54.el7_2.6.x86_64`) packages are installed on the controller and Compute nodes: for compute in $COMPUTES; do ssh ${compute} -- rpm -qa | egrep '(pacemaker|fence-agents|resource-agents)' ; done # 6.a Apply the following constraint workarounds required for BZ#1257414: #Note: This issue has been addressed in RHSA-2015:1862 and might not be required for your environment. #controller-1 # sudo pcs constraint order start openstack-nova-novncproxy-clone then openstack-nova-api-clone #controller-1 # sudo pcs constraint order start rabbitmq-clone then openstack-keystone-clone #controller-1 # sudo pcs constraint order promote galera-master then openstack-keystone-clone #controller-1 # sudo pcs constraint order start haproxy-clone then openstack-keystone-clone #controller-1 # sudo pcs constraint order start memcached-clone then openstack-keystone-clone #controller-1 # sudo pcs constraint order promote redis-master then start openstack-ceilometer-central-clone require-all=false #controller-1 # sudo pcs resource defaults resource-stickiness=INFINITY #6.b Apply the following constraint workarounds required for BZ#1295835: #Note: This issue has been addressed in RHBA-2016:0264-1 and might not be required for your environment. #sudo pcs config | grep systemd | awk '{print $2}' | while read RESOURCE; do sudo pcs resource update $RESOURCE op start timeout=200s op stop timeout=200s; done" # 7. Create a NovaEvacuate active/passive resource using the overcloudrc file to provide the auth_url, username, tenant and password values: # 8. Confirm that nova-evacuate is started after the floating IP resources, and the Image Service (glance), OpenStack Networking (neutron), Compute (nova) services: helper=iha-helper-create-evacuate.sh cat < $helper set -ex pcs resource create nova-evacuate ocf:openstack:NovaEvacuate auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME no_shared_storage=1 for i in \$(sudo pcs status | grep IP | awk '{ print \$1 }'); do sudo pcs constraint order start \$i then nova-evacuate ; done for i in openstack-glance-api-clone neutron-metadata-agent-clone openstack-nova-conductor-clone; do sudo pcs constraint order start \$i then nova-evacuate require-all=false ; done EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # Note: If you are not using shared storage, include the no_shared_storage=1 option in your resource create ... command above. See Exception for shared storage for more information. # 9. Disable all OpenStack resources across the control plane: # Depending on the time needed to stop Identity Service (and on the power of your hardware) you can consider increasing the timeout period (--wait). # OSP8: ssh ${FIRST_CONTROLLER} -- sudo pcs resource disable openstack-keystone --wait=600s # OSP9+: ssh ${FIRST_CONTROLLER} -- sudo pcs resource disable openstack-core-clone --wait=600s # 10. Create a list of the current controllers using cibadmin data : #controller-1 # controllers=$(sudo cibadmin -Q -o nodes | grep uname | sed s/.*uname..// | awk -F\" '{print $1}') #controller-1 # echo $controllers # 11. Use this list to tag these nodes as controllers with the osprole=controller property: # 12. Build a list of stonith devices already present in the environment: # 13. Tag the control plane services to make sure they only run on the controllers identified above, skipping any stonith devices listed: helper=iha-helper-tag-controllers.sh cat < $helper set -ex for controller in ${CONTROLLERS}; do pcs property set --node \${controller} osprole=controller ; done stonithdevs=\$(pcs stonith | awk '{print \$1}') for i in \$(cibadmin -Q --xpath //primitive --node-path | tr ' ' '\n' | awk -F "id='" '{print \$2}' | awk -F "'" '{print \$1}' | uniq); do found=0 if [ -n "\$stonithdevs" ]; then for x in \$stonithdevs; do if [ \$x = \$i ]; then found=1 fi done fi if [ \$found = 0 ]; then pcs constraint location \$i rule resource-discovery=exclusive score=0 osprole eq controller fi done EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # 14. Begin to populate the Compute node resources within pacemaker, starting with neutron-openvswitch-agent: helper=fix-compute.sh cat < $helper set -x grep crudini /usr/lib/ocf/resource.d/openstack/nova-compute-wait if [ \$? != 0 ]; then set -e patch -p1 /usr/lib/ocf/resource.d/openstack/nova-compute-wait \${PWD}/bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch fi EOF for node in $COMPUTES; do scp bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch $helper ${node}: ; done for node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done helper=iha-helper-create-compute-resources.sh cat < $helper set -ex pcs resource create neutron-openvswitch-agent-compute systemd:neutron-openvswitch-agent op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force pcs constraint location neutron-openvswitch-agent-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start neutron-server-clone then neutron-openvswitch-agent-compute-clone require-all=false # Then the Compute libvirtd resource: pcs resource create libvirtd-compute systemd:libvirtd op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force pcs constraint location libvirtd-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start neutron-openvswitch-agent-compute-clone then libvirtd-compute-clone pcs constraint colocation add libvirtd-compute-clone with neutron-openvswitch-agent-compute-clone # Then the openstack-ceilometer-compute resource: pcs resource create ceilometer-compute systemd:openstack-ceilometer-compute op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force pcs constraint location ceilometer-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start openstack-ceilometer-notification-clone then ceilometer-compute-clone require-all=false pcs constraint order start libvirtd-compute-clone then ceilometer-compute-clone pcs constraint colocation add ceilometer-compute-clone with libvirtd-compute-clone # Then the nova-compute resource: pcs resource create nova-compute-checkevacuate ocf:openstack:nova-compute-wait auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME domain=localdomain op start timeout=300 --clone interleave=true --disabled --force pcs constraint location nova-compute-checkevacuate-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start openstack-nova-conductor-clone then nova-compute-checkevacuate-clone require-all=false pcs resource create nova-compute systemd:openstack-nova-compute op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force pcs constraint location nova-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start nova-compute-checkevacuate-clone then nova-compute-clone require-all=true pcs constraint order start nova-compute-clone then nova-evacuate require-all=false pcs constraint order start libvirtd-compute-clone then nova-compute-clone pcs constraint colocation add nova-compute-clone with libvirtd-compute-clone EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # 16. Create a seperate fence-nova stonith device: helper=iha-helper-create-nova-fence.sh cat < $helper set -ex pcs stonith create fence-nova fence_compute auth-url=$OS_AUTH_URL login=$OS_USERNAME passwd=$OS_PASSWORD tenant-name=$OS_TENANT_NAME domain=localdomain record-only=1 --force EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # 17. Make certain the Compute nodes are able to recover after fencing: ssh ${FIRST_CONTROLLER} -- sudo pcs property set cluster-recheck-interval=1min # 18. Create Compute node resources and set the stonith level 1 to include both the nodes's physical fence device and fence-nova: helper=iha-helper-create-computes.sh cat < $helper set -ex for node in $COMPUTES; do pcs resource create \${node} ocf:pacemaker:remote reconnect_interval=60 op monitor interval=20 pcs property set --node \${node} osprole=compute pcs stonith level add 1 \${node} shooter,fence-nova done EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # 19. Enable the control and Compute plane services: helper=iha-helper-enable-services.sh cat < $helper set -ex #pcs resource enable openstack-keystone pcs resource enable openstack-core pcs resource enable neutron-openvswitch-agent-compute pcs resource enable libvirtd-compute pcs resource enable ceilometer-compute pcs resource enable nova-compute-checkevacuate pcs resource enable nova-compute EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper # 20. Allow some time for the environment to settle before cleaning up any failed resources: helper=iha-helper-cleanup.sh cat < $helper set -ex sleep 60 pcs resource cleanup pcs status echo pcs property set stonith-enabled=true EOF scp $helper ${FIRST_CONTROLLER}: ssh ${FIRST_CONTROLLER} -- sudo bash $helper #Test High Availability #Note: These steps deliberately reboot the Compute node without warning. #1. The following step boots an instance on the overcloud, and then crashes the Compute node: #stack@director # . overcloudrc #stack@director # nova boot --image cirros --flavor 2 test-failover #stack@director # nova list --fields name,status,host #stack@director # . stackrc #stack@director # ssh -lheat-admin compute-n #compute-n # sudo su - #root@compute-n # echo c > /proc/sysrq-trigger # #2. A short time later, the instance should be restarted on a working Compute node: # #stack@director # nova list --fields name,status,host #stack@director # nova service-list ================================================ FILE: iha-uninstall.sh ================================================ #!/bin/bash set -ex source stackrc # If your machines don't conform to this structure, try calling the script like this: # COMPUTE_PATTERN=mycompute ./iha-uninstall.sh upgrade : ${COMPUTE_PATTERN=novacompute} : ${CONTROLLER_PATTERN=controller} COMPUTES=$(nova list | grep ${COMPUTE_PATTERN} | awk -F\| '{ print $3}' | tr '\n' ' ') CONTROLLERS=$(nova list | grep ${CONTROLLER_PATTERN} | awk -F\| '{ print $3}' | tr '\n' ' ') FIRST_COMPUTE=$(echo $COMPUTES | awk '{print $1}') FIRST_CONTROLLER=$(echo $CONTROLLERS | awk '{print $1}') ssh ${FIRST_CONTROLLER} -- sudo pcs property set stonith-enabled=false if [ $1 = "upgrade" ]; then SERVICES="nova-evacuate neutron-openvswitch-agent-compute libvirtd-compute-clone ceilometer-compute-clone nova-compute-checkevacuate-clone nova-compute-clone" helper=iha-helper-remove.sh cat < $helper set -ex pcs property set maintenance-mode=true for resource in $COMPUTES $SERVICES $FUDGE; do pcs resource cleanup \${resource} pcs --force resource delete \${resource} done for node in $COMPUTES; do cibadmin --delete --xml-text "" cibadmin --delete --xml-text "" done pcs property set maintenance-mode=false --wait EOF scp $helper heat-admin@${FIRST_CONTROLLER}: ssh heat-admin@${FIRST_CONTROLLER} -- sudo bash $helper fi helper=iha-helper-reenable.sh cat < $helper set -ex for service in neutron-openvswitch-agent openstack-ceilometer-compute openstack-nova-compute libvirtd; do systemctl enable \${service} done EOF for node in $COMPUTES; do scp $helper heat-admin@${node}: ; ssh heat-admin@${node} -- sudo bash $helper ; done ================================================ FILE: keepalived/ceilometer-config.md ================================================ Introduction ------------ In terms of high availability, the Ceilometer central agent deserves special attention. This agent had to run in a single node until the Juno release cycle, since there was no way to coordinate multiple agents and ensure they would not duplicate metrics. Now, multiple central agent instances can run in parallel with workload partitioning among these running instances, using the tooz library with a Redis backend for coordination. See [here](http://docs.openstack.org/admin-guide-cloud/telemetry-data-collection.html#support-for-ha-deployment) for additional information. The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/ceilometer.scenario). Install software ---------------- yum install -y openstack-ceilometer-api openstack-ceilometer-central openstack-ceilometer-collector openstack-ceilometer-common openstack-ceilometer-alarm python-ceilometer python-ceilometerclient python-redis Configure ceilometer -------------------- openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_plugin password openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken username ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken project_name services openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211 openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://controller-vip.example.com:5000/v2.0 openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://hacontroller1,hacontroller2,hacontroller3:27017/ceilometer?replicaSet=ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf database max_retries -1 # keep last 5 days data only (value is in secs) openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000 openstack-config --set /etc/ceilometer/ceilometer.conf api host 192.168.1.22X Configure coordination URL -------------------------- openstack-config --set /etc/ceilometer/ceilometer.conf coordination backend_url 'redis://hacontroller1:26379?sentinel=mymaster&sentinel_fallback=hacontroller2:26379&sentinel_fallback=hacontroller3:26379' Enable and start Ceilometer services, open firewall ports --------------------------------------------------------- systemctl start openstack-ceilometer-central systemctl enable openstack-ceilometer-central systemctl start openstack-ceilometer-collector systemctl enable openstack-ceilometer-collector systemctl start openstack-ceilometer-api systemctl enable openstack-ceilometer-api systemctl start openstack-ceilometer-alarm-evaluator systemctl enable openstack-ceilometer-alarm-evaluator systemctl start openstack-ceilometer-alarm-notifier systemctl enable openstack-ceilometer-alarm-notifier systemctl start openstack-ceilometer-notification systemctl enable openstack-ceilometer-notification firewall-cmd --add-port=8777/tcp firewall-cmd --add-port=8777/tcp --permanent firewall-cmd --add-port=4952/udp firewall-cmd --add-port=4952/udp --permanent Tests ----- On any node: . /root/keystonerc_admin for m in storage.objects image network volume instance ; do ceilometer sample-list -m $m | tail -2 ; done ================================================ FILE: keepalived/cinder-config.md ================================================ Introduction ------------ Cinder will be configured in this example to use the NFS backend driver. Instructions for any other backend driver will only differ in the `volume_driver` config option and any driver-specific options. The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/cinder.scenario). Install software ---------------- yum install -y openstack-cinder openstack-utils openstack-selinux python-memcached Configure --------- openstack-config --set /etc/cinder/cinder.conf database connection mysql://cinder:cindertest@controller-vip.example.com/cinder openstack-config --set /etc/cinder/cinder.conf database max_retries -1 openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_plugin password openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken username cinder openstack-config --set /etc/cinder/cinder.conf keystone_authtoken password cindertest openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_name services openstack-config --set /etc/cinder/cinder.conf DEFAULT notification_driver messaging openstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_host controller-vip.example.com openstack-config --set /etc/cinder/cinder.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211 openstack-config --set /etc/cinder/cinder.conf DEFAULT host rhos7-cinder openstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen 192.168.1.22X openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/cinder/cinder.conf keymgr encryption_auth_url http://controller-vip.example.com:5000/v3 **Note:** We are setting a single "host" entry for all nodes, this is related to the A/P issues with cinder-volume. Configure NFS driver -------------------- # Choose whatever NFS share is used cat > /etc/cinder/nfs_exports << EOF 192.168.1.4:/volumeUSB1/usbshare/openstack/cinder EOF chown root:cinder /etc/cinder/nfs_exports chmod 0640 /etc/cinder/nfs_exports openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfs_exports openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_sparsed_volumes true openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_mount_options v3 openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.nfs.NfsDriver Manage DB --------- On node 1: su cinder -s /bin/sh -c "cinder-manage db sync" Start services -------------- On node 1: systemctl start openstack-cinder-api systemctl start openstack-cinder-scheduler systemctl start openstack-cinder-volume systemctl enable openstack-cinder-api systemctl enable openstack-cinder-scheduler systemctl enable openstack-cinder-volume **Note:** If this node crashes, it should be manually started on another node. Refer to [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1193229) for additional information. On nodes 2 and 3: systemctl start openstack-cinder-api systemctl start openstack-cinder-scheduler systemctl enable openstack-cinder-api systemctl enable openstack-cinder-scheduler Open firewall ports ------------------- On all nodes: firewall-cmd --add-port=8776/tcp firewall-cmd --add-port=8776/tcp --permanent Test ---- On any node: . /root/keystonerc_demo cinder create --display-name test 1 cinder extend test 4 cinder delete test ================================================ FILE: keepalived/compute-node.md ================================================ Introduction ------------ The compute node implementation is relatively straightforward, compared to the controller node. It will only be necessary to configure: - OpenvSwitch and Neutron OpenvSwitch agent - Nova compute - Ceilometer compute agent You can find a phd scenario file [here](phd-setup/compute.scenario). Environment description ----------------------- The network configuration was previously discussed in the [Controller node implementation](controller-node.md) section: ![](Controller-network.jpg "Network configuration") - The external network is used by the Neutron floating IPs, and for any external access. The hypervisor nodes (hacompute1 and hacompute2) do not need to be connected to this network, but in the demo setup they are connected for testing purposes. - The internal network will carry all other traffic: API traffic, tenant networks and storage traffic. - The router providing connectivity between the internal and external networks is only needed if Trove and/or Sahara are being deployed. Remember this is a minimum test setup. Any production setup should separate internal and external API traffic, tenant networks and storage traffic in different network segments. Compute node configuration -------------------------- The following commands should be executed on each compute node to be added to the installation. There is no configuration required on the controller nodes, meaning compute nodes can be added anytime. ### Install software yum install -y openstack-nova-compute openstack-utils python-cinder openstack-neutron-openvswitch openstack-ceilometer-compute openstack-neutron openstack-selinux ### Enable OpenvSwitch, start daemon and create integration bridge systemctl enable openvswitch systemctl start openvswitch ovs-vsctl add-br br-int ### Configure Nova compute openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211 openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.1.22X openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 0.0.0.0 openstack-config --set /etc/nova/nova.conf vnc novncproxy_base_url http://controller-vip.example.com:6080/vnc_auto.html openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@controller-vip.example.com/nova openstack-config --set /etc/nova/nova.conf database max_retries -1 openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone openstack-config --set /etc/nova/nova.conf glance host controller-vip.example.com openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver openstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron openstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL openstack-config --set /etc/nova/nova.conf conductor use_local false openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest openstack-config --set /etc/nova/nova.conf neutron url http://controller-vip.example.com:9696/ openstack-config --set /etc/nova/nova.conf neutron project_domain_id default openstack-config --set /etc/nova/nova.conf neutron project_name services openstack-config --set /etc/nova/nova.conf neutron user_domain_id default openstack-config --set /etc/nova/nova.conf neutron username neutron openstack-config --set /etc/nova/nova.conf neutron password neutrontest openstack-config --set /etc/nova/nova.conf neutron auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/nova/nova.conf neutron auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/nova/nova.conf neutron auth_plugin password openstack-config --set /etc/nova/nova.conf neutron region_name regionOne openstack-config --set /etc/nova/nova.conf libvirt nfs_mount_options v3 openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_plugin password openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/nova/api-paste.ini filter:authtoken username compute openstack-config --set /etc/nova/api-paste.ini filter:authtoken password novatest openstack-config --set /etc/nova/api-paste.ini filter:authtoken project_name services openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_uri http://controller-vip.example.com:5000/ Only run the following command if you are creating a test environment where your hypervisors will be virtual machines. openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu ### Configure Neutron on compute node openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_plugin password openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password neutrontest openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name services openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent tunnel_types vxlan openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 4789 openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs enable_tunneling True openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_id_ranges 1:1000 openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tenant_network_type vxlan openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs integration_bridge br-int openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_bridge br-tun openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs local_ip 192.168.1.22X openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent l2_population False ### Configure Ceilometer on compute node openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit True openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit_period hour openstack-config --set /etc/nova/nova.conf DEFAULT notify_on_state_change vm_and_task_state openstack-config --set /etc/nova/nova.conf DEFAULT notification_driver nova.openstack.common.notifier.rpc_notifier sed -i -e 's/nova.openstack.common.notifier.rpc_notifier/nova.openstack.common.notifier.rpc_notifier\nnotification_driver = ceilometer.compute.nova_notifier/g' /etc/nova/nova.conf openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_plugin password openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken username ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken project_name services openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211 openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://controller-vip.example.com:5000/v2.0 openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://hacontroller1,hacontroller2,hacontroller3:27017/ceilometer?replicaSet=ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf database connection max_retries -1 # keep last 5 days data only (value is in secs) openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000 **Note:** the following SELinux boolean allows QEMU to use NFS for Cinder volumes. A different boolean may apply if using another type of backend storage. setsebool -P virt_use_nfs 1 ### Set kernel TCP keepalive parameters cat > /etc/sysctl.d/tcpka.conf << EOF net.ipv4.tcp_keepalive_intvl = 1 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_keepalive_time = 5 EOF sysctl -p /etc/sysctl.d/tcpka.conf ### Enable and start services, open firewall ports **Note:** we are enabling ports 5900-5999 for VNC access. If the compute node could host more than 100 VMs, we have to extend this range. systemctl start libvirtd systemctl start neutron-openvswitch-agent systemctl enable neutron-openvswitch-agent systemctl enable neutron-ovs-cleanup systemctl start openstack-ceilometer-compute systemctl enable openstack-ceilometer-compute systemctl start openstack-nova-compute systemctl enable openstack-nova-compute firewall-cmd --add-port=4789/udp firewall-cmd --add-port=4789/udp --permanent firewall-cmd --add-port=5900-5999/tcp firewall-cmd --add-port=5900-5999/tcp --permanent ================================================ FILE: keepalived/controller-node.md ================================================ Introduction ------------ Controller nodes will run all OpenStack services in this architecture, while compute nodes will only act as hypervisors. Environment description ----------------------- ### Network setup The basic requirements for this environment include 5 nodes, with the network setup described in the following diagram: ![](Controller-network.jpg "Network setup") - The external network is used by the Neutron floating IPs, and for any external access. The hypervisor nodes (hacompute1 and hacompute2) do not need to be connected to this network, but in the demo setup they are connected for testing purposes. - The internal network will carry all other traffic: API traffic, tenant networks and storage traffic. - A router will provide connectivity for the controller nodes to the floating IP network as required by Sahara for instance management. In this configuration example, it will also allow virtual machines to access the controller nodes, allowing Trove instances access to the RabbitMQ server (**please note this is not recommended for a production setup**, refer to [trove-config.md](the Trove section) for details). If neitner Sahara or Trove are used, the router is not required. Please note this is a minimum test setup. Any production setup should separate internal and external API traffic, tenant networks and storage traffic in different network segments, and route traffic between instances and the controller nodes via a firewall. ### Node setup The following table provides the system details for the environment used during testing. | Hostname | NIC 1 | NIC 2 | Disk 1 | Disk 2 | |--------------------|--------------------------------------|-------------------|------------------|-----------------| |hacontroller1 |eth0: no IP, used for provider network|eth1: 192.168.1.221| 60 GB (/dev/vda) | 8 GB (/dev/vdb) | |hacontroller2 |eth0: no IP, used for provider network|eth1: 192.168.1.222| 60 GB (/dev/vda) | 8 GB (/dev/vdb) | |hacontroller3 |eth0: no IP, used for provider network|eth1: 192.168.1.223| 60 GB (/dev/vda) | 8 GB (/dev/vdb) | |controller-vip (virtual hostname)| |eth1: 192.168.1.220| | | |hacompute1 |eth0: 10.10.10.224 (-priv) |eth1: 192.168.1.224| 60 GB (/dev/vda) | | |hacompute2 |eth0: 10.10.10.225 (-priv) |eth1: 192.168.1.225| 60 GB (/dev/vda) | | All nodes have SELinux set to *Enforcing* and an active firewall. On the detailed installation notes, remember to substitute any occurrence of 192.168.1.22X with the IP of the node being configured. #### Single vs multiple VIPs Please note that the current document uses a single virtual IP for the whole stack of OpenStack API services. It is also possible to use multiple virtual IPs, one per API service. Using multiple virtual IPs will allow for load distribution between the three HAProxy instances. In contrast, the requirements for available IP addresses increase, and the initial configuration can be slightly more complex if performed manually. #### Base operating system installation All nodes start from a *minimal* CentOS 7 or RHEL 7 installation, then enabling the required software channels. - For RDO Liberty, follow the steps specified in the [RDO wiki](https://openstack.redhat.com/Repositories) **Note:** For now, use [this release RPM](http://rdoproject.org/repos/openstack-liberty/rdo-release-liberty.rpm) to setup the required repositories. - For the [Red Hat Enterprise Linux OpenStack Platform](http://www.redhat.com/openstack), run the following commands to enable the required repositories: 
subscription-manager register --username=${RHSMUSER} --password=${RHSMPASS} 
subscription-manager attach --auto
subscription-manager repos --disable \* 
subscription-manager repos --enable rhel-7-server-rpms 
subscription-manager repos --enable rhel-7-server-rh-common-rpms 
subscription-manager repos --enable rhel-7-server-openstack-7.0-rpms
yum -y update 
reboot
The systems also had NetworkManager disabled: systemctl disable NetworkManager systemctl enable network If the installed system cannot access the gateway, be sure there is a “GATEWAY=xxx” entry in the relevant ifcfg file: NAME=eth0 ... GATEWAY=192.168.1.1 **Note:** It is very important to make sure that NTP is correctly configured on all nodes. Failure to do so can cause environment instability. The following phd scenario files have been created: - A file with the [VM creation and initial setup](phd-setup/hypervisors.scenario) - A file with some [basic system preparation tasks](phd-setup/serverprep.scenario) #### Configuration steps **NOTE:** before moving on, please remember that the following instructions have been created for the above-mentioned scenario (3 controller nodes, with a specific network setup). If you are using this guide to implement your own environment, pay close attention to any IP substitution and other changes that may apply to your environment. Also, do not forget to set sensible passwords. The configuration steps can be divided into: - Installing/configuring/enabling all core non-Openstack services - [HAProxy](haproxy-config.md) - [Galera](galera-config.md) - [RabbitMQ](rabbitmq-config.md) - [Memcached](memcached-config.md) - [Redis](redis-config.md) - [MongoDB](mongodb-config.md) - [Keepalived](keepalived-config.md) - Installing/configuring/enabling all core Openstack services - [Keystone](keystone-config.md) - [Glance](glance-config.md) - [Cinder](cinder-config.md) - [Swift](swift-config.md) - [Neutron](neutron-config.md) - [Nova](nova-config.md) - [Ceilometer](ceilometer-config.md) - [Heat](heat-config.md) - [Horizon](horizon-config.md) - [Trove](trove-config.md) - [Sahara](sahara-config.md) On each section, a link to a [phd-based](https://github.com/davidvossel/phd) scenario file is provided, as a reference. ================================================ FILE: keepalived/galera-bootstrap.md ================================================ Introduction ------------ Here is an outline of the steps needed to re-establish/bootstrap Galera quorum. 1. Determine loss of quorum 2. Determine systems with last activity 3. Start first DB on first node 4. Start DB on remaining nodes Determine loss of quorum ------------------------ Confirm in the */var/log/mariadb/mariadb.log* on each system, looking for Errors 140929 11:25:40 [ERROR] WSREP: Local state seqno (1399488) is greater than group seqno (10068): states diverged. Aborting to avoid potential data loss. Remove '/var/lib/mysql//grastate.dat' file and restart if you wish to continue. (FATAL) 140929 11:25:40 [ERROR] Aborting [root@ospha2 ~]# Also the clustercheck command should so that there are some systems not in sync [root@ospha2 ~]# clustercheck HTTP/1.1 503 Service Unavailable Content-Type: text/plain Connection: close Content-Length: 36 Galera cluster node is not synced. [root@ospha2 ~]# Determine systems with last activity ------------------------------------ In this section we attempt to determine which system or systems has the highest valid sequence number for the for the latest UUID. ### Orderly shutdown If the cluster shutdown correctly the `/var/lib/mysql/grastate.dat` file will have positive numbers for the seqno. Note which system or systems have the greatest seqno. However, if any system has a `-1` value, that indicates the shutdown was not clean and another method to determine the seqno is needed. [root@ospha2 ~]# cat /var/lib/mysql/grastate.dat # GALERA saved state version: 2.1 uuid: b048715d-4369-11e4-b7ef-af1999a6c989 seqno: -1 cert_index: [root@ospha2 ~]# ### Disorderly Shutdown The seqno is in the `/var/log/mariadb/mariadb.log` file. Search for lines with "Found save state", ignoring any -1 values. The last value on each line is in the form UUID:seqno. [root@ospha1 ~]# tail -n 1000 /var/log/mariadb/mariadb.log | grep "Found saved state" | grep -v ":-1" 140923 17:49:19 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:2229 140924 15:37:13 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:2248 140929 11:24:26 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:10060 [root@ospha1 ~]# [root@ospha2 ~]# tail -n 1000 /var/log/mariadb/mariadb.log | grep "Found saved state" | grep -v ":-1" 140926 14:58:16 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:171535 140929 11:24:28 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:1399488 [root@ospha2 ~]# [root@ospha3 ~]# tail -n 2000 /var/log/mariadb/mariadb.log | grep "Found saved state" | grep -v ":-1" 140923 17:36:57 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:36 140923 17:43:18 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:785 [root@ospha3 ~]# Notice all servers have the same UUID (b048715d-4369-11e4-b7ef-af1999a6c989), but server *ospha2* has the largest seqno (1399488). Start first DB on first node ---------------------------- The following command will initiate the Galera cluster. Since ospha2 had the highest seqno, that is the node to start first. [root@ospha2 ~]# sudo -u mysql /usr/libexec/mysqld --wsrep-cluster-address='gcomm://' & [1] 1910 [root@ospha2 ~]# 140929 16:31:00 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 140929 16:31:00 [Warning] Could not increase number of max_open_files to more than 1024 (request: 1835) /usr/libexec/mysqld: Query cache is disabled (resize or similar command in progress); repeat this command later Verify that this brought the this node into sync. [root@ospha2 ~]# clustercheck HTTP/1.1 200 OK Content-Type: text/plain Connection: close Content-Length: 32 Galera cluster node is synced. [root@ospha2 ~]# Start DB on remaining nodes --------------------------- On another cluster member, start the database, and then verify this node reports synced. [root@ospha1 ~]# systemctl start mariadb [root@ospha1 ~]# clustercheck HTTP/1.1 200 OK Content-Type: text/plain Connection: close Content-Length: 32 Galera cluster node is synced. [root@ospha1 ~]# Once `clustercheck` returns 200 on all nodes, restart MariaDB on the first node. kill systemctl start mariadb ================================================ FILE: keepalived/galera-config.md ================================================ Introduction ------------ MariaDB with Galera provides synchronous database replication, in an active-active, multi-master environment. High availability for the data itself is managed internally by Galera, while access availability is managed by HAProxy. ![](Mariadb-haproxy.jpg "MariaDB HA architecture") It is important to note that, while Galera supports multi-master access, there are known issues in an OpenStack environment due to its use of optimistic locking (see [this post](http://lists.openstack.org/pipermail/openstack-dev/2014-May/035264.html) for details). Due to this limitation, the HAProxy configuration used for Galera sets a single node to be accessed at all times, and a different node will only be used when the original node fails. This effectively turns the Galera configuration into an active-hot standby one. The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/galera.scenario). Install software ---------------- yum install -y mariadb-galera-server xinetd rsync Configure cluster check ----------------------- cat > /etc/sysconfig/clustercheck << EOF MYSQL_USERNAME="clustercheck" MYSQL_PASSWORD="redhat" MYSQL_HOST="localhost" MYSQL_PORT="3306" EOF Start mysqld and create user for cluster check ---------------------------------------------- systemctl start mysqld mysql -e "CREATE USER 'clustercheck'@'localhost' IDENTIFIED BY 'redhat';" systemctl stop mysqld Create Galera configuration --------------------------- cat > /etc/my.cnf.d/galera.cnf << EOF [mysqld] skip-name-resolve=1 binlog_format=ROW default-storage-engine=innodb innodb_autoinc_lock_mode=2 innodb_locks_unsafe_for_binlog=1 max_connections=8192 query_cache_size=0 query_cache_type=0 bind_address=192.168.1.22X wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_cluster_name="galera_cluster" wsrep_cluster_address="gcomm://192.168.1.221,192.168.1.222,192.168.1.223" wsrep_slave_threads=1 wsrep_certify_nonPK=1 wsrep_max_ws_rows=131072 wsrep_max_ws_size=1073741824 wsrep_debug=0 wsrep_convert_LOCK_to_trx=0 wsrep_retry_autocommit=1 wsrep_auto_increment_control=1 wsrep_drupal_282555_workaround=0 wsrep_causal_reads=0 wsrep_notify_cmd= wsrep_sst_method=rsync EOF Create Galera systemd configuration file, to allow a higher number of files to be opened ---------------------------------------------------------------------------------------- mkdir -p /etc/systemd/system/mariadb.service.d/ cat > /etc/systemd/system/mariadb.service.d/limits.conf << EOF [Service] LimitNOFILE=16384 EOF Configure monitor service (used by HAProxy) ------------------------------------------- cat > /etc/xinetd.d/galera-monitor << EOF service galera-monitor { port = 9200 disable = no socket_type = stream protocol = tcp wait = no user = root group = root groups = yes server = /usr/bin/clustercheck type = UNLISTED per_source = UNLIMITED log_on_success = log_on_failure = HOST flags = REUSE } EOF Start services and open firewall ports -------------------------------------- systemctl daemon-reload systemctl enable xinetd systemctl start xinetd systemctl enable haproxy systemctl start haproxy firewall-cmd --add-service=mysql firewall-cmd --add-port=4444/tcp firewall-cmd --add-port=4567/tcp firewall-cmd --add-port=4568/tcp firewall-cmd --add-port=4568/tcp --permanent firewall-cmd --add-service=mysql --permanent firewall-cmd --add-port=4567/tcp --permanent firewall-cmd --add-port=4444/tcp --permanent firewall-cmd --add-port=9300/tcp firewall-cmd --add-port=9300/tcp --permanent firewall-cmd --add-port=9200/tcp firewall-cmd --add-port=9200/tcp --permanent Start mariadb cluster --------------------- **On all nodes:** systemctl enable mariadb **On node1:** sudo -u mysql /usr/libexec/mysqld --wsrep-cluster-address='gcomm://' & **On node2 and node3:** systemctl start mariadb Once clustecheck returns 200 on all nodes, restart galera on node 1: kill systemctl start mariadb Create users and databases -------------------------- **On node 1:** mysql MariaDB [(none)]> use mysql; MariaDB [mysql]> drop user ''@'hacontroller1.example.com'; MariaDB [mysql]> drop user 'root'@'hacontroller1.example.com'; MariaDB [mysql]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED by 'mysqltest' WITH GRANT OPTION; MariaDB [mysql]> CREATE DATABASE keystone; MariaDB [mysql]> GRANT ALL ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'keystonetest'; MariaDB [mysql]> CREATE DATABASE glance; MariaDB [mysql]> GRANT ALL ON glance.* TO 'glance'@'%' IDENTIFIED BY 'glancetest'; MariaDB [mysql]> CREATE DATABASE cinder; MariaDB [mysql]> GRANT ALL ON cinder.* TO 'cinder'@'%' IDENTIFIED BY 'cindertest'; MariaDB [mysql]> CREATE DATABASE neutron; MariaDB [mysql]> GRANT ALL ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'neutrontest'; MariaDB [mysql]> CREATE DATABASE nova; MariaDB [mysql]> GRANT ALL ON nova.* TO 'nova'@'%' IDENTIFIED BY 'novatest'; MariaDB [mysql]> CREATE DATABASE heat; MariaDB [mysql]> GRANT ALL ON heat.* TO 'heat'@'%' IDENTIFIED BY 'heattest'; MariaDB [mysql]> CREATE DATABASE sahara; MariaDB [mysql]> GRANT ALL ON sahara.* TO 'sahara'@'%' IDENTIFIED BY 'saharatest'; MariaDB [mysql]> CREATE DATABASE trove; MariaDB [mysql]> GRANT ALL ON trove.* TO 'trove'@'%' IDENTIFIED BY 'trovetest'; MariaDB [mysql]> FLUSH PRIVILEGES; MariaDB [mysql]> quit mysqladmin flush-hosts ================================================ FILE: keepalived/glance-config.md ================================================ Introduction ------------ The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/glance.scenario). Install software ---------------- yum install -y openstack-glance openstack-utils openstack-selinux nfs-utils Configure glance-api and glance-registry ---------------------------------------- openstack-config --set /etc/glance/glance-api.conf database connection mysql://glance:glancetest@controller-vip.example.com/glance openstack-config --set /etc/glance/glance-api.conf database max_retries -1 openstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_plugin password openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken username glance openstack-config --set /etc/glance/glance-api.conf keystone_authtoken password glancetest openstack-config --set /etc/glance/glance-api.conf keystone_authtoken project_name services openstack-config --set /etc/glance/glance-api.conf DEFAULT notification_driver messaging openstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host 192.168.1.22X openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host controller-vip.example.com openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/glance/glance-registry.conf database connection mysql://glance:glancetest@controller-vip.example.com/glance openstack-config --set /etc/glance/glance-registry.conf database max_retries -1 openstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_plugin password openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken username glance openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken password glancetest openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken project_name services openstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host 192.168.1.22X Manage DB --------- On node 1: su glance -s /bin/sh -c "glance-manage db_sync" Configure backend ----------------- For this setup, NFS will be used. Add the NFS mount to `/etc/fstab`, making sure it is mounted on `/var/lib/glance`. Be aware the last two columns in `fstab` need to be "0 0" on RHEL/CentOS 7, due to [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1120367). You may also find [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1203820) if using NFS v3 shares. Also, note there is currently a known SELinux issue when using an NFS backend for Glance. See [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1219406) for a description and fix. On all nodes: chown glance:nobody /var/lib/glance Start services and open firewall ports -------------------------------------- systemctl start openstack-glance-registry systemctl start openstack-glance-api systemctl enable openstack-glance-registry systemctl enable openstack-glance-api firewall-cmd --add-port=9191/tcp firewall-cmd --add-port=9191/tcp --permanent firewall-cmd --add-port=9292/tcp firewall-cmd --add-port=9292/tcp --permanent Test ---- On any node: . /root/keystonerc_admin wget http://download.cirros-cloud.net/0.3.3/cirros-0.3.3-x86_64-disk.img glance image-create --name "cirros" --disk-format qcow2 --container-format bare --file cirros-0.3.3-x86_64-disk.img --visibility public glance image-list ================================================ FILE: keepalived/haproxy-config.md ================================================ Introduction ------------ A load-balancing proxy is used to provide scalability and load balancing for OpenStack API services and some of the supporting services. All requests will be distributed using a round-robin algorithm between all available controller nodes, and HAProxy itself will monitor the availability of each service. In case a node or service goes down, HAProxy will remove it from the active pool after a timeout has been reached, tipically a few seconds. The following commands will be executed on all controller nodes. You can find a phd scenario file [here](phd-setup/lb.scenario). Install packages ---------------- yum install -y haproxy openstack-selinux Allow binding to non-local IPs ------------------------------ echo net.ipv4.ip_nonlocal_bind=1 >> /etc/sysctl.d/haproxy.conf echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind Configure HAProxy ----------------- cat > /etc/haproxy/haproxy.cfg << EOF global daemon stats socket /var/lib/haproxy/stats defaults mode tcp maxconn 10000 timeout connect 5s timeout client 30s timeout server 30s listen monitor bind 192.168.1.220:9300 mode http monitor-uri /status stats enable stats uri /admin stats realm Haproxy\ Statistics stats auth root:redhat stats refresh 5s frontend vip-db bind 192.168.1.220:3306 timeout client 90m default_backend db-vms-galera backend db-vms-galera option httpchk stick-table type ip size 1000 stick on dst timeout server 90m server rhos8-node1 192.168.1.221:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions server rhos8-node2 192.168.1.222:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions server rhos8-node3 192.168.1.223:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions # Note the RabbitMQ entry is only needed for CloudForms compatibility # and should be removed in the future frontend vip-rabbitmq option clitcpka bind 192.168.1.220:5672 timeout client 900m default_backend rabbitmq-vms backend rabbitmq-vms option srvtcpka balance roundrobin timeout server 900m server rhos8-node1 192.168.1.221:5672 check inter 1s server rhos8-node2 192.168.1.222:5672 check inter 1s server rhos8-node3 192.168.1.223:5672 check inter 1s frontend vip-keystone-admin bind 192.168.1.220:35357 default_backend keystone-admin-vms timeout client 600s backend keystone-admin-vms balance roundrobin timeout server 600s server rhos8-node1 192.168.1.221:35357 check inter 1s on-marked-down shutdown-sessions server rhos8-node2 192.168.1.222:35357 check inter 1s on-marked-down shutdown-sessions server rhos8-node3 192.168.1.223:35357 check inter 1s on-marked-down shutdown-sessions frontend vip-keystone-public bind 192.168.1.220:5000 default_backend keystone-public-vms timeout client 600s backend keystone-public-vms balance roundrobin timeout server 600s server rhos8-node1 192.168.1.221:5000 check inter 1s on-marked-down shutdown-sessions server rhos8-node2 192.168.1.222:5000 check inter 1s on-marked-down shutdown-sessions server rhos8-node3 192.168.1.223:5000 check inter 1s on-marked-down shutdown-sessions frontend vip-glance-api bind 192.168.1.220:9191 default_backend glance-api-vms backend glance-api-vms balance roundrobin server rhos8-node1 192.168.1.221:9191 check inter 1s server rhos8-node2 192.168.1.222:9191 check inter 1s server rhos8-node3 192.168.1.223:9191 check inter 1s frontend vip-glance-registry bind 192.168.1.220:9292 default_backend glance-registry-vms backend glance-registry-vms balance roundrobin server rhos8-node1 192.168.1.221:9292 check inter 1s server rhos8-node2 192.168.1.222:9292 check inter 1s server rhos8-node3 192.168.1.223:9292 check inter 1s frontend vip-cinder bind 192.168.1.220:8776 default_backend cinder-vms backend cinder-vms balance roundrobin server rhos8-node1 192.168.1.221:8776 check inter 1s server rhos8-node2 192.168.1.222:8776 check inter 1s server rhos8-node3 192.168.1.223:8776 check inter 1s frontend vip-swift bind 192.168.1.220:8080 default_backend swift-vms backend swift-vms balance roundrobin server rhos8-node1 192.168.1.221:8080 check inter 1s server rhos8-node2 192.168.1.222:8080 check inter 1s server rhos8-node3 192.168.1.223:8080 check inter 1s frontend vip-neutron bind 192.168.1.220:9696 default_backend neutron-vms backend neutron-vms balance roundrobin server rhos8-node1 192.168.1.221:9696 check inter 1s server rhos8-node2 192.168.1.222:9696 check inter 1s server rhos8-node3 192.168.1.223:9696 check inter 1s frontend vip-nova-vnc-novncproxy bind 192.168.1.220:6080 default_backend nova-vnc-novncproxy-vms backend nova-vnc-novncproxy-vms balance roundrobin timeout tunnel 1h server rhos8-node1 192.168.1.221:6080 check inter 1s server rhos8-node2 192.168.1.222:6080 check inter 1s server rhos8-node3 192.168.1.223:6080 check inter 1s frontend nova-metadata-vms bind 192.168.1.220:8775 default_backend nova-metadata-vms backend nova-metadata-vms balance roundrobin server rhos8-node1 192.168.1.221:8775 check inter 1s server rhos8-node2 192.168.1.222:8775 check inter 1s server rhos8-node3 192.168.1.223:8775 check inter 1s frontend vip-nova-api bind 192.168.1.220:8774 default_backend nova-api-vms backend nova-api-vms balance roundrobin server rhos8-node1 192.168.1.221:8774 check inter 1s server rhos8-node2 192.168.1.222:8774 check inter 1s server rhos8-node3 192.168.1.223:8774 check inter 1s frontend vip-horizon bind 192.168.1.220:80 timeout client 180s default_backend horizon-vms backend horizon-vms balance roundrobin timeout server 180s mode http cookie SERVERID insert indirect nocache server rhos8-node1 192.168.1.221:80 check inter 1s cookie rhos8-horizon1 on-marked-down shutdown-sessions server rhos8-node2 192.168.1.222:80 check inter 1s cookie rhos8-horizon2 on-marked-down shutdown-sessions server rhos8-node3 192.168.1.223:80 check inter 1s cookie rhos8-horizon3 on-marked-down shutdown-sessions frontend vip-heat-cfn bind 192.168.1.220:8000 default_backend heat-cfn-vms backend heat-cfn-vms balance roundrobin server rhos8-node1 192.168.1.221:8000 check inter 1s server rhos8-node2 192.168.1.222:8000 check inter 1s server rhos8-node3 192.168.1.223:8000 check inter 1s frontend vip-heat-cloudw bind 192.168.1.220:8003 default_backend heat-cloudw-vms backend heat-cloudw-vms balance roundrobin server rhos8-node1 192.168.1.221:8003 check inter 1s server rhos8-node2 192.168.1.222:8003 check inter 1s server rhos8-node3 192.168.1.223:8003 check inter 1s frontend vip-heat-srv bind 192.168.1.220:8004 default_backend heat-srv-vms backend heat-srv-vms balance roundrobin server rhos8-node1 192.168.1.221:8004 check inter 1s server rhos8-node2 192.168.1.222:8004 check inter 1s server rhos8-node3 192.168.1.223:8004 check inter 1s frontend vip-ceilometer bind 192.168.1.220:8777 timeout client 90s default_backend ceilometer-vms backend ceilometer-vms balance roundrobin timeout server 90s server rhos8-node1 192.168.1.221:8777 check inter 1s server rhos8-node2 192.168.1.222:8777 check inter 1s server rhos8-node3 192.168.1.223:8777 check inter 1s frontend vip-sahara bind 192.168.1.220:8386 default_backend sahara-vms backend sahara-vms balance roundrobin server rhos8-node1 192.168.1.221:8386 check inter 1s server rhos8-node2 192.168.1.222:8386 check inter 1s server rhos8-node3 192.168.1.223:8386 check inter 1s frontend vip-trove bind 192.168.1.220:8779 default_backend trove-vms backend trove-vms balance roundrobin server rhos8-node1 192.168.1.221:8779 check inter 1s server rhos8-node2 192.168.1.222:8779 check inter 1s server rhos8-node3 192.168.1.223:8779 check inter 1s EOF Note we are **not** starting haproxy yet. Once HAproxy is started, you can monitor progress of your service configuration by going to [](http://controller-vip.example.com:9300/admin) (root/redhat, remember to set a sensible password). With this you will be able to see which services are running on which nodes, as seen by HAproxy. ================================================ FILE: keepalived/heat-config.md ================================================ Introduction ------------ The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/heat.scenario). Install software ---------------- yum install -y openstack-heat-engine openstack-heat-api openstack-heat-api-cfn openstack-heat-api-cloudwatch python-heatclient openstack-utils python-glanceclient Configure Heat domain --------------------- To allow non-admin users to create Heat stacks, a Keystone domain needs to be created. Run the following commands to create the Heat domain, and configure Heat to use it. On node 1: . /root/keystonerc_admin openstack role create heat_stack_user openstack token issue Take note of the token ID issued, then: openstack --os-token=${TOKEN_ID} --os-url=http://controller-vip.example.com:5000/v3 --os-identity-api-version=3 domain create heat --description "Owns users and projects created by heat" openstack --os-token=${TOKEN_ID} --os-url=http://controller-vip.example.com:5000/v3 --os-identity-api-version=3 user create --password heattest --domain heat --description "Manages users and projects created by heat" heat_domain_admin openstack --os-token=${TOKEN_ID} --os-url=http://controller-vip.example.com:5000/v3 --os-identity-api-version=3 role add --user heat_domain_admin --domain heat admin On all nodes: openstack-config --set /etc/heat/heat.conf DEFAULT stack_domain_admin_password heattest openstack-config --set /etc/heat/heat.conf DEFAULT stack_domain_admin heat_domain_admin openstack-config --set /etc/heat/heat.conf DEFAULT stack_user_domain_name heat Configure Heat -------------- openstack-config --set /etc/heat/heat.conf database connection mysql://heat:heattest@controller-vip.example.com/heat openstack-config --set /etc/heat/heat.conf database max_retries -1 openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_plugin password openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/heat/heat.conf keystone_authtoken username heat openstack-config --set /etc/heat/heat.conf keystone_authtoken password heattest openstack-config --set /etc/heat/heat.conf keystone_authtoken project_name services openstack-config --set /etc/heat/heat.conf keystone_authtoken keystone_ec2_uri http://controller-vip.example.com:35357/v2.0 openstack-config --set /etc/heat/heat.conf keystone_authtoken identity_uri http://controller-vip.example.com:35357 openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_user heat openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_password heattest openstack-config --set /etc/heat/heat.conf ec2authtoken auth_uri http://controller-vip.example.com:5000/v2.0 openstack-config --set /etc/heat/heat.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211 openstack-config --set /etc/heat/heat.conf heat_api bind_host 192.168.1.22X openstack-config --set /etc/heat/heat.conf heat_api_cfn bind_host 192.168.1.22X openstack-config --set /etc/heat/heat.conf heat_api_cloudwatch bind_host 192.168.1.22X openstack-config --set /etc/heat/heat.conf DEFAULT heat_metadata_server_url controller-vip.example.com:8000 openstack-config --set /etc/heat/heat.conf DEFAULT heat_waitcondition_server_url controller-vip.example.com:8000/v1/waitcondition openstack-config --set /etc/heat/heat.conf DEFAULT heat_watch_server_url controller-vip.example.com:8003 openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/heat/heat.conf DEFAULT rpc_backend rabbit openstack-config --set /etc/heat/heat.conf DEFAULT notification_driver heat.openstack.common.notifier.rpc_notifier openstack-config --set /etc/heat/heat.conf DEFAULT enable_cloud_watch_lite false Manage DB --------- On node 1: su heat -s /bin/sh -c "heat-manage db_sync" Start services, open firewall ports ----------------------------------- On all nodes: systemctl start openstack-heat-api systemctl start openstack-heat-api-cfn systemctl start openstack-heat-api-cloudwatch systemctl start openstack-heat-engine systemctl enable openstack-heat-api systemctl enable openstack-heat-api-cfn systemctl enable openstack-heat-api-cloudwatch systemctl enable openstack-heat-engine firewall-cmd --add-port=8000/tcp firewall-cmd --add-port=8000/tcp --permanent firewall-cmd --add-port=8003/tcp firewall-cmd --add-port=8003/tcp --permanent firewall-cmd --add-port=8004/tcp firewall-cmd --add-port=8004/tcp --permanent ================================================ FILE: keepalived/horizon-config.md ================================================ Introduction ------------ The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/horizon.scenario). Install software ---------------- yum install -y mod_wsgi httpd mod_ssl python-memcached openstack-dashboard Set secret key -------------- On node 1: openssl rand -hex 10 Take note of the generated random value, then on all nodes: sed -i -e "s#SECRET_KEY.*#SECRET_KEY = 'VALUE'#g#" /etc/openstack-dashboard/local_settings Configure local\_settings and httpd.conf ---------------------------------------- sed -i -e "s#ALLOWED_HOSTS.*#ALLOWED_HOSTS = ['*',]#g" \ -e "s#^CACHES#SESSION_ENGINE = 'django.contrib.sessions.backends.cache'\nCACHES#g#" \ -e "s#locmem.LocMemCache'#memcached.MemcachedCache',\n\t'LOCATION' : [ 'hacontroller1:11211', 'hacontroller2:11211', 'hacontroller3:11211', ]#g" \ -e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = "controller-vip.example.com"#g' \ -e "s#^LOCAL_PATH.*#LOCAL_PATH = '/var/lib/openstack-dashboard'#g" \ /etc/openstack-dashboard/local_settings Restart httpd and open firewall port ------------------------------------ systemctl daemon-reload systemctl restart httpd firewall-cmd --add-port=80/tcp firewall-cmd --add-port=80/tcp --permanent ================================================ FILE: keepalived/keepalived-config.md ================================================ Introduction ------------ [Keepalived](http://www.keepalived.org/) provides simple and robust facilities for load balancing and high-availability to Linux system and Linux based infrastructures. In this highly available OpenStack architecture, it is used to provide high availability to the virtual IP(s) used by HAProxy. High-availability is achieved by VRRP protocol, a fundamental brick for router failover. ![](Keepalived-arch.jpg "Keepalived architecture") The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/keepalived.scenario). Install software ---------------- yum -y install keepalived psmisc Create configuration file ------------------------- On all nodes: cat > /etc/keepalived/keepalived.conf << EOF vrrp_script chk_haproxy { script "/usr/bin/killall -0 haproxy" interval 2 } vrrp_instance VI_PUBLIC { interface eth1 state BACKUP virtual_router_id 52 priority 101 virtual_ipaddress { 192.168.1.220 dev eth1 } track_script { chk_haproxy } # Avoid failback nopreempt } vrrp_sync_group VG1 group { VI_PUBLIC } EOF Open firewall rules and start services -------------------------------------- On all nodes: firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -i eth1 -d 224.0.0.0/8 -j ACCEPT firewall-cmd --direct --perm --add-rule ipv4 filter INPUT 0 -i eth1 -d 224.0.0.0/8 -j ACCEPT systemctl start keepalived systemctl enable keepalived ================================================ FILE: keepalived/keystone-config.md ================================================ Introduction ------------ The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/keystone.scenario). Install software ---------------- yum install -y openstack-keystone openstack-utils openstack-selinux httpd mod_wsgi python-openstackclient Create service token and distribute to the other controllers ------------------------------------------------------------ On node 1: export SERVICE_TOKEN=$(openssl rand -hex 10) echo $SERVICE_TOKEN > /root/keystone_service_token scp /root/keystone_service_token root@hacontroller2:/root scp /root/keystone_service_token root@hacontroller3:/root Configure Apache web server for Keystone ---------------------------------------- **NOTE:** running Keystone under eventlet has been deprecated as of the Kilo release. Support for utilizing eventlet will be removed as of the M-release. Thus, instructions are provided to run Keystone under the Apache web server, as a WSGI process. On all nodes: cp /usr/share/keystone/wsgi-keystone.conf /etc/httpd/conf.d/ sed -i -e 's/apache2/httpd/g' /etc/httpd/conf.d/wsgi-keystone.conf sed -i -e 's/VirtualHost \*/VirtualHost 192.168.1.22X/g' /etc/httpd/conf.d/wsgi-keystone.conf sed -i -e 's/Listen 5000/Listen 192.168.1.22X:5000/g' /etc/httpd/conf.d/wsgi-keystone.conf sed -i -e 's/Listen 35357/Listen 192.168.1.22X:35357/g' /etc/httpd/conf.d/wsgi-keystone.conf sed -i -e 's/^Listen.*/Listen 192.168.1.22X:80/g' /etc/httpd/conf/httpd.conf Configure Keystone ------------------ On all nodes: export SERVICE_TOKEN=$(cat /root/keystone_service_token) openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token $SERVICE_TOKEN openstack-config --set /etc/keystone/keystone.conf DEFAULT rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/keystone/keystone.conf DEFAULT rabbit_ha_queues true openstack-config --set /etc/keystone/keystone.conf eventlet_server admin_endpoint 'http://controller-vip.example.com:%(admin_port)s/' openstack-config --set /etc/keystone/keystone.conf eventlet_server public_endpoint 'http://controller-vip.example.com:%(public_port)s/' openstack-config --set /etc/keystone/keystone.conf database connection mysql://keystone:keystonetest@controller-vip.example.com/keystone openstack-config --set /etc/keystone/keystone.conf database max_retries -1 openstack-config --set /etc/keystone/keystone.conf DEFAULT public_bind_host 192.168.1.22X openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_bind_host 192.168.1.22X openstack-config --set /etc/keystone/keystone.conf token driver keystone.token.persistence.backends.sql.Token Create and distribute PKI setup, manage DB ------------------------------------------ On node 1: keystone-manage pki_setup --keystone-user keystone --keystone-group keystone chown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/ su keystone -s /bin/sh -c "keystone-manage db_sync" cd /etc/keystone/ssl tar cvfz /tmp/keystone_ssl.tgz * scp /tmp/keystone_ssl.tgz hacontroller2:/tmp scp /tmp/keystone_ssl.tgz hacontroller3:/tmp Restore Keystone PKI setup from node 1 -------------------------------------- On nodes 2 and 3: mkdir -p /etc/keystone/ssl cd /etc/keystone/ssl tar xvfz /tmp/keystone_ssl.tgz chown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/ restorecon -Rv /etc/keystone/ssl touch /var/log/keystone/keystone.log chown keystone:keystone /var/log/keystone/keystone.log Create cron job to flush expired tokens --------------------------------------- On all nodes: echo "1 * * * * keystone keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1" >> /etc/crontab Start services and open firewall ports -------------------------------------- On all nodes; firewall-cmd --add-port=5000/tcp firewall-cmd --add-port=5000/tcp --permanent firewall-cmd --add-port=35357/tcp firewall-cmd --add-port=35357/tcp --permanent systemctl start httpd systemctl enable httpd Create endpoints, services and users for all API services --------------------------------------------------------- On node 1: export OS_TOKEN=$(cat /root/keystone_service_token) export OS_URL=http://controller-vip.example.com:35357/v2.0 export OS_REGION_NAME=regionOne openstack service create --name=keystone --description="Keystone Identity Service" identity openstack endpoint create --publicurl 'http://controller-vip.example.com:5000/v2.0' --adminurl 'http://controller-vip.example.com:35357/v2.0' --internalurl 'http://controller-vip.example.com:5000/v2.0' --region regionOne keystone openstack user create --password keystonetest admin openstack role create admin openstack project create admin openstack role add --project admin --user admin admin openstack user create --password redhat demo openstack role create _member_ openstack project create demo openstack role add --project demo --user demo _member_ openstack project create --description "Services Tenant" services # glance openstack user create --password glancetest glance openstack role add --project services --user glance admin openstack service create --name=glance --description="Glance Image Service" image openstack endpoint create --publicurl 'http://controller-vip.example.com:9292' --adminurl 'http://controller-vip.example.com:9292' --internalurl 'http://controller-vip.example.com:9292' --region regionOne glance # cinder openstack user create --password cindertest cinder openstack role add --project services --user cinder admin openstack service create --name=cinder --description="Cinder Volume Service" volume openstack endpoint create --publicurl "http://controller-vip.example.com:8776/v1/\$(tenant_id)s" --adminurl "http://controller-vip.example.com:8776/v1/\$(tenant_id)s" --internalurl "http://controller-vip.example.com:8776/v1/\$(tenant_id)s" --region regionOne cinder openstack service create --name=cinderv2 --description="OpenStack Block Storage" volumev2 openstack endpoint create --publicurl "http://controller-vip.example.com:8776/v2/\$(tenant_id)s" --adminurl "http://controller-vip.example.com:8776/v2/\$(tenant_id)s" --internalurl "http://controller-vip.example.com:8776/v2/\$(tenant_id)s" --region regionOne cinderv2 # swift openstack user create --password swifttest swift openstack role add --project services --user swift admin openstack service create --name=swift --description="Swift Storage Service" object-store openstack endpoint create --publicurl "http://controller-vip.example.com:8080/v1/AUTH_\$(tenant_id)s" --adminurl "http://controller-vip.example.com:8080/v1" --internalurl "http://controller-vip.example.com:8080/v1/AUTH_\$(tenant_id)s" --region regionOne swift # neutron openstack user create --password neutrontest neutron openstack role add --project services --user neutron admin openstack service create --name=neutron --description="OpenStack Networking Service" network openstack endpoint create --publicurl "http://controller-vip.example.com:9696" --adminurl "http://controller-vip.example.com:9696" --internalurl "http://controller-vip.example.com:9696" --region regionOne neutron # nova openstack user create --password novatest compute openstack role add --project services --user compute admin openstack service create --name=compute --description="OpenStack Compute Service" compute openstack endpoint create --publicurl "http://controller-vip.example.com:8774/v2/\$(tenant_id)s" --adminurl "http://controller-vip.example.com:8774/v2/\$(tenant_id)s" --internalurl "http://controller-vip.example.com:8774/v2/\$(tenant_id)s" --region regionOne compute # heat openstack user create --password heattest heat openstack role add --project services --user heat admin openstack service create --name=heat --description="Heat Orchestration Service" orchestration openstack endpoint create --publicurl "http://controller-vip.example.com:8004/v1/%(tenant_id)s" --adminurl "http://controller-vip.example.com:8004/v1/%(tenant_id)s" --internalurl "http://controller-vip.example.com:8004/v1/%(tenant_id)s" --region regionOne heat openstack service create --name=heat-cfn --description="Heat CloudFormation Service" cloudformation openstack endpoint create --publicurl "http://controller-vip.example.com:8000/v1" --adminurl "http://controller-vip.example.com:8000/v1" --internalurl "http://controller-vip.example.com:8000/v1" --region regionOne heat-cfn # ceilometer openstack user create --password ceilometertest ceilometer openstack role add --project services --user ceilometer admin openstack role create ResellerAdmin openstack role add --project services --user ceilometer ResellerAdmin openstack service create --name=ceilometer --description="OpenStack Telemetry Service" metering openstack endpoint create --publicurl "http://controller-vip.example.com:8777" --adminurl "http://controller-vip.example.com:8777" --internalurl "http://controller-vip.example.com:8777" --region regionOne ceilometer # sahara openstack user create --password saharatest sahara openstack role add --project services --user sahara admin openstack service create --name=sahara --description="Sahara Data Processing" data-processing openstack endpoint create --publicurl "http://controller-vip.example.com:8386/v1.1/%(tenant_id)s" --adminurl "http://controller-vip.example.com:8386/v1.1/%(tenant_id)s" --internalurl "http://controller-vip.example.com:8386/v1.1/%(tenant_id)s" --region regionOne sahara # trove openstack user create --password trovetest trove openstack role add --project services --user trove admin openstack service create --name=trove --description="OpenStack Database Service" database openstack endpoint create --publicurl "http://controller-vip.example.com:8779/v1.0/%(tenant_id)s" --adminurl "http://controller-vip.example.com:8779/v1.0/%(tenant_id)s" --internalurl "http://controller-vip.example.com:8779/v1.0/%(tenant_id)s" --region regionOne trove Create keystonerc files for simplicity -------------------------------------- On all nodes: cat > /root/keystonerc_admin << EOF export OS_USERNAME=admin export OS_TENANT_NAME=admin export OS_PROJECT_NAME=admin export OS_REGION_NAME=regionOne export OS_PASSWORD=keystonetest export OS_AUTH_URL=http://controller-vip.example.com:35357/v2.0/ export PS1='[\u@\h \W(keystone_admin)]\$ ' EOF cat > /root/keystonerc_demo << EOF export OS_USERNAME=demo export OS_TENANT_NAME=demo export OS_PROJECT_NAME=demo export OS_REGION_NAME=regionOne export OS_PASSWORD=redhat export OS_AUTH_URL=http://controller-vip.example.com:5000/v2.0/ export PS1='[\u@\h \W(keystone_user)]\$ ' EOF ================================================ FILE: keepalived/memcached-config.md ================================================ Introduction ------------ Memcached is a general-purpose distributed memory caching system. It is used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source must be read. **Note:** Access to memcached is not handled by HAproxy because replicated access is currently only in an experimental state. Instead consumers must be supplied with the full list of hosts running memcached. The following commands will be executed on all controller nodes. You can find a phd scenario file [here](phd-setup/memcached.scenario). Install and enable memcached ---------------------------- yum install -y memcached systemctl start memcached systemctl enable memcached firewall-cmd --add-port=11211/tcp firewall-cmd --add-port=11211/tcp --permanent ================================================ FILE: keepalived/mongodb-config.md ================================================ Introduction ------------ MongoDB can provide high availability through the use of replica sets. A replica set in MongoDB is a group of mongod processes that maintain the same data set, where one of the nodes is specified as master and the rest as slaves. Clients are explicitly told to connect to the replica set, by specifying all its members. In case of a node failure, the client should transparently reconnect to a surviving replica. The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/mongodb.scenario). Install packages ---------------- yum install -y mongodb mongodb-server Listen to external connections, and configure replication set ------------------------------------------------------------- sed -i -e 's#bind_ip.*#bind_ip = 0.0.0.0#g' /etc/mongod.conf echo "replSet = ceilometer" >> /etc/mongod.conf Start services and enable firewall ports ---------------------------------------- systemctl start mongod systemctl enable mongod firewall-cmd --add-port=27017/tcp firewall-cmd --add-port=27017/tcp --permanent Create replica set ------------------ On node 1: mongo > rs.initiate() > sleep(10000) > rs.add("hacontroller2.example.com"); > rs.add("hacontroller3.example.com"); And verify: > rs.status() Until all nodes show `"stateStr" : "PRIMARY"` or `"stateStr" : "SECONDARY"`, then: > quit() ================================================ FILE: keepalived/mongodb-recovery.md ================================================ Introduction ------------ MongoDB usually does a good job at re-forming a replica set after a full cluster reboot. In case of any failure, [its documentation](http://docs.mongodb.org/v2.6/tutorial/troubleshoot-replica-sets/) provides an excellent reference on how to troubleshoot and fix any error. ================================================ FILE: keepalived/neutron-config.md ================================================ Introduction ------------ For this setup, we will configure an external provider network using eth0 (10.10.10.X network). A different setup would apply for bridged networks, although the HA features should be the same. There are two services that require special attention for a highly available architecture: - Neutron DHCP agent availability is obtained by assigning two or more agents to manage each tenant network, setting `dhcp_agents_per_network=2` in `/etc/neutron/neutron.conf`. - Neutron L3 agent availability uses the L3 HA functionality using VRRP. Using this functionality, a new type of router, spawned on two or more different agents, is created. One agent will be in charge of the master version of this router, and the remaining L3 agents will be in charge of the slave routers. Refer to [this blog post](http://assafmuller.com/2014/08/16/layer-3-high-availability/) for a detailed description of the feature. The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/neutron.scenario). Install software ---------------- yum install -y openstack-neutron openstack-neutron-openvswitch openstack-neutron-ml2 openstack-utils openstack-selinux Configure Neutron server ------------------------ openstack-config --set /etc/neutron/neutron.conf DEFAULT bind_host 192.168.1.22X openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_plugin password openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password neutrontest openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name services openstack-config --set /etc/neutron/neutron.conf database connection mysql://neutron:neutrontest@controller-vip.example.com:3306/neutron openstack-config --set /etc/neutron/neutron.conf database max_retries -1 openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/neutron/neutron.conf nova nova_region_name regionOne openstack-config --set /etc/neutron/neutron.conf nova project_domain_id default openstack-config --set /etc/neutron/neutron.conf nova project_name services openstack-config --set /etc/neutron/neutron.conf nova user_domain_id default openstack-config --set /etc/neutron/neutron.conf nova password novatest openstack-config --set /etc/neutron/neutron.conf nova username compute openstack-config --set /etc/neutron/neutron.conf nova auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/neutron/neutron.conf nova auth_plugin password openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes True openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_data_changes True openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin neutron.plugins.ml2.plugin.Ml2Plugin openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router openstack-config --set /etc/neutron/neutron.conf DEFAULT router_scheduler_driver neutron.scheduler.l3_agent_scheduler.ChanceScheduler openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 2 openstack-config --set /etc/neutron/neutron.conf DEFAULT api_workers 2 openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_workers 2 openstack-config --set /etc/neutron/neutron.conf DEFAULT l3_ha True openstack-config --set /etc/neutron/neutron.conf DEFAULT min_l3_agents_per_router 2 openstack-config --set /etc/neutron/neutron.conf DEFAULT max_l3_agents_per_router 2 ### ML2 configuration ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 type_drivers local,gre,flat,vxlan,vlan openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers openvswitch openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks \* openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_gre tunnel_id_ranges 10:10000 openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vni_ranges 10:10000 openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vxlan_group 224.0.0.1 openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_security_group True openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver ### LBaaS configuration (optional) yum -y install openstack-neutron-lbaas openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router,lbaas openstack-config --set /etc/neutron/lbaas_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver openstack-config --set /etc/neutron/lbaas_agent.ini DEFAULT device_driver neutron_lbaas.services.loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver openstack-config --set /etc/neutron/lbaas_agent.ini haproxy user_group haproxy ### FWaaS configuration (optional) yum -y install openstack-neutron-fwaas openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router,firewall,lbaas openstack-config --set /etc/neutron/fwaas_driver.ini fwaas enabled True openstack-config --set /etc/neutron/fwaas_driver.ini fwaas driver neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver Manage DB --------- On node 1: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head Start services, open firewall ports ----------------------------------- On all nodes: systemctl start neutron-server systemctl enable neutron-server firewall-cmd --add-port=9696/tcp firewall-cmd --add-port=9696/tcp --permanent OpenvSwitch configuration ------------------------- systemctl enable openvswitch systemctl start openvswitch **Note:** we have seeing issues when trying to configure an IP on br-eth0 (specially ARP problems), so it is not recommended. Assuming eth0 is your interface attached to the external network, create two files in _/etc/sysconfig/network-scripts/_ as follows (change MTU if you need): cat < /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-eth0 ONBOOT=yes BOOTPROTO=none VLAN=yes MTU="9000" NM_CONTROLLED=no EOF cat < /etc/sysconfig/network-scripts/ifcfg-br-eth0 DEVICE=br-eth0 DEVICETYPE=ovs OVSBOOTPROTO=none TYPE=OVSBridge ONBOOT=yes BOOTPROTO=static MTU="9000" NM_CONTROLLED=no EOF Restart the network for the changes to take effect. systemctl restart network OpenvSwitch agent ----------------- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent tunnel_types vxlan openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 4789 openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs local_ip 192.168.1.22X openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs enable_tunneling True openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs integration_bridge br-int openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_bridge br-tun openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs bridge_mappings physnet1:br-eth0 openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs network_vlan_ranges physnet1 openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent l2_population False Metadata agent -------------- openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_strategy keystone openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_url http://controller-vip.example.com:35357/v2.0 openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_host controller-vip.example.com openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_region regionOne openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_tenant_name services openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_user neutron openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_password neutrontest openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip controller-vip.example.com openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_port 8775 openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret metatest openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_workers 4 openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_backlog 2048 DHCP agent ---------- openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_config_file /etc/neutron/dnsmasq-neutron.conf The following will prevent issues from happening when the network card MTU is 1500. If we are using jumbo frames, it should not be required. Be aware that this only helps on certain operating systems with a well-behaving DHCP client. Windows is known to ignore it. echo "dhcp-option-force=26,1400" > /etc/neutron/dnsmasq-neutron.conf chown root:neutron /etc/neutron/dnsmasq-neutron.conf chmod 644 /etc/neutron/dnsmasq-neutron.conf L3 agent -------- openstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver openstack-config --set /etc/neutron/l3_agent.ini DEFAULT handle_internal_only_routers True openstack-config --set /etc/neutron/l3_agent.ini DEFAULT send_arp_for_ha 3 openstack-config --set /etc/neutron/l3_agent.ini DEFAULT metadata_ip controller-vip.example.com openstack-config --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge Start services and open VXLAN firewall port ------------------------------------------- systemctl start neutron-openvswitch-agent systemctl start neutron-dhcp-agent systemctl start neutron-l3-agent systemctl start neutron-metadata-agent systemctl start neutron-lbaas-agent systemctl enable neutron-openvswitch-agent systemctl enable neutron-dhcp-agent systemctl enable neutron-l3-agent systemctl enable neutron-metadata-agent systemctl enable neutron-ovs-cleanup systemctl enable neutron-lbaas-agent firewall-cmd --add-port=4789/udp firewall-cmd --add-port=4789/udp --permanent **NOTE:** During a full cluster reboot, since Galera does not start cleanly neutron-server will wait for some time, then fail due to a service startup timeout (see [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1188198) for details). We can fix that by creating a file named `/etc/systemd/system/neutron-server.service.d/restart.conf` with the following contents: [Service] Restart=on-failure Neutron server will try to restart indefinitely, then eventually succeed as soon as the Galera DB is running. Create provider network ----------------------- On node 1: . /root/keystonerc_admin neutron net-create public --provider:network_type flat --provider:physical_network physnet1 --router:external neutron subnet-create --gateway 10.10.10.1 --allocation-pool start=10.10.10.100,end=10.10.10.150 --disable-dhcp --name public_subnet public 10.10.10.0/24 ================================================ FILE: keepalived/nova-config.md ================================================ Introduction ------------ The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/nova.scenario). Install software ---------------- yum install -y openstack-nova-console openstack-nova-novncproxy openstack-utils openstack-nova-api openstack-nova-conductor openstack-nova-scheduler python-cinderclient python-memcached Configure Nova API ------------------ openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211 openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_host 192.168.1.22X openstack-config --set /etc/nova/nova.conf vnc novncproxy_base_url http://controller-vip.example.com:6080/vnc_auto.html openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.1.22X openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 192.168.1.22X openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@controller-vip.example.com/nova openstack-config --set /etc/nova/nova.conf database max_retries -1 openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen 192.168.1.22X openstack-config --set /etc/nova/nova.conf DEFAULT metadata_host 192.168.1.22X openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 192.168.1.22X openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen_port 8775 openstack-config --set /etc/nova/nova.conf DEFAULT glance_host controller-vip.example.com openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver openstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron openstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL openstack-config --set /etc/nova/nova.conf conductor use_local false openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest openstack-config --set /etc/nova/nova.conf neutron url http://controller-vip.example.com:9696/ openstack-config --set /etc/nova/nova.conf neutron project_domain_id default openstack-config --set /etc/nova/nova.conf neutron project_name services openstack-config --set /etc/nova/nova.conf neutron user_domain_id default openstack-config --set /etc/nova/nova.conf neutron username neutron openstack-config --set /etc/nova/nova.conf neutron password neutrontest openstack-config --set /etc/nova/nova.conf neutron auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/nova/nova.conf neutron auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/nova/nova.conf neutron auth_plugin password openstack-config --set /etc/nova/nova.conf neutron region_name regionOne # REQUIRED FOR A/A scheduler openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_host_subset_size 30 openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_plugin password openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/nova/api-paste.ini filter:authtoken username compute openstack-config --set /etc/nova/api-paste.ini filter:authtoken password novatest openstack-config --set /etc/nova/api-paste.ini filter:authtoken project_name services openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_uri http://controller-vip.example.com:5000/ Only run the following command if you are creating a test environment where your hypervisors will be virtual machines openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu Manage DB --------- On node 1: su nova -s /bin/sh -c "nova-manage db sync" Start services, open firewall ports ----------------------------------- On all nodes: systemctl start openstack-nova-consoleauth systemctl start openstack-nova-novncproxy systemctl start openstack-nova-api systemctl start openstack-nova-scheduler systemctl start openstack-nova-conductor systemctl enable openstack-nova-consoleauth systemctl enable openstack-nova-novncproxy systemctl enable openstack-nova-api systemctl enable openstack-nova-scheduler systemctl enable openstack-nova-conductor firewall-cmd --add-port=8773-8775/tcp firewall-cmd --add-port=8773-8775/tcp --permanent firewall-cmd --add-port=6080/tcp firewall-cmd --add-port=6080/tcp --permanent ================================================ FILE: keepalived/phd-setup/ceilometer.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Install Ceilometer # - Configuring Ceilometer # - Start services and add firewall rules ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_nic_external PHD_VAR_network_hosts_vip PHD_VAR_network_hosts_controllers PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_hosts_memcache PHD_VAR_network_hosts_mongodb PHD_VAR_network_ips_controllers PHD_VAR_network_neutron_externalgateway PHD_VAR_network_neutron_externalnetwork PHD_VAR_network_neutron_allocpoolstart PHD_VAR_network_neutron_allocpoolend ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) yum install -y openstack-ceilometer-api openstack-ceilometer-central openstack-ceilometer-collector openstack-ceilometer-common openstack-ceilometer-alarm python-ceilometer python-ceilometerclient openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_user ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://${PHD_VAR_network_hosts_vip}:5000/v2.0 openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://${PHD_VAR_network_hosts_mongodb}:27017/ceilometer?replicaSet=ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf database max_retries -1 # keep last 5 days data only (value is in secs) openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000 openstack-config --set /etc/ceilometer/ceilometer.conf api host ${myip} IFS=', ' read -a controller_names <<< "${PHD_VAR_network_hosts_controllers}" openstack-config --set /etc/ceilometer/ceilometer.conf coordination backend_url "redis://${controller_names[0]}:26379?sentinel=mymaster&sentinel_fallback=${controller_names[0]}:26379&sentinel_fallback=${controller_names[0]}:26379" systemctl start openstack-ceilometer-central systemctl enable openstack-ceilometer-central systemctl start openstack-ceilometer-collector systemctl enable openstack-ceilometer-collector systemctl start openstack-ceilometer-api systemctl enable openstack-ceilometer-api systemctl start openstack-ceilometer-alarm-evaluator systemctl enable openstack-ceilometer-alarm-evaluator systemctl start openstack-ceilometer-alarm-notifier systemctl enable openstack-ceilometer-alarm-notifier systemctl start openstack-ceilometer-notification systemctl enable openstack-ceilometer-notification firewall-cmd --add-port=8777/tcp firewall-cmd --add-port=8777/tcp --permanent firewall-cmd --add-port=4952/udp firewall-cmd --add-port=4952/udp --permanent .... ================================================ FILE: keepalived/phd-setup/cinder.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Install Cinder # - Configure Cinder, using a NFS v3 backend # - Start services, open firewall rules ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_hosts_vip PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_hosts_memcache PHD_VAR_network_nfs_cindershare ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) yum install -y openstack-cinder openstack-utils openstack-selinux python-memcached openstack-config --set /etc/cinder/cinder.conf database connection mysql://cinder:cindertest@${PHD_VAR_network_hosts_vip}/cinder openstack-config --set /etc/cinder/cinder.conf database max_retries -1 openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone openstack-config --set /etc/cinder/cinder.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_user cinder openstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_password cindertest openstack-config --set /etc/cinder/cinder.conf DEFAULT notification_driver messaging openstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/cinder/cinder.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/cinder/cinder.conf DEFAULT host rhos7-cinder openstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen ${myip} openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues true cat > /etc/cinder/nfs_exports << EOF ${PHD_VAR_network_nfs_cindershare} EOF chown root:cinder /etc/cinder/nfs_exports chmod 0640 /etc/cinder/nfs_exports openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfs_exports openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_sparsed_volumes true openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_mount_options v3 openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.nfs.NfsDriver .... target=$PHD_ENV_nodes1 .... su cinder -s /bin/sh -c "cinder-manage db sync" systemctl start openstack-cinder-volume systemctl enable openstack-cinder-volume .... target=all .... systemctl start openstack-cinder-api systemctl start openstack-cinder-scheduler systemctl enable openstack-cinder-api systemctl enable openstack-cinder-scheduler firewall-cmd --add-port=8776/tcp firewall-cmd --add-port=8776/tcp --permanent .... ================================================ FILE: keepalived/phd-setup/compute.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Install all required packages for a compute node # - Configure nova-compute, neutron and ceilometer for the compute node # - Tweak TCP keepalive parameters # - Start services and open firewall ports ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_nic_external PHD_VAR_network_hosts_vip PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_hosts_memcache PHD_VAR_network_hosts_mongodb ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) # Install packages yum install -y openstack-nova-compute openstack-utils python-cinder openstack-neutron-openvswitch openstack-ceilometer-compute openstack-neutron # Enable OpenvSwitch systemctl enable openvswitch systemctl start openvswitch ovs-vsctl add-br br-int # Configure Nova compute openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address ${myip} openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 0.0.0.0 openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://${PHD_VAR_network_hosts_vip}:6080/vnc_auto.html openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@${PHD_VAR_network_hosts_vip}/nova openstack-config --set /etc/nova/nova.conf database max_retries -1 openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone openstack-config --set /etc/nova/nova.conf glance host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver openstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron openstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL openstack-config --set /etc/nova/nova.conf conductor use_local false openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest openstack-config --set /etc/nova/nova.conf neutron url http://${PHD_VAR_network_hosts_vip}:9696/ openstack-config --set /etc/nova/nova.conf neutron admin_tenant_name services openstack-config --set /etc/nova/nova.conf neutron admin_username neutron openstack-config --set /etc/nova/nova.conf neutron admin_password neutrontest openstack-config --set /etc/nova/nova.conf neutron admin_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/nova/nova.conf neutron region_name regionOne openstack-config --set /etc/nova/nova.conf libvirt nfs_mount_options v3 openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_tenant_name services openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_user compute openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_password novatest # Only if the hypervisor is a virtual machine openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu # Configure Neutron for compute node openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password neutrontest openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent tunnel_types vxlan openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent vxlan_udp_port 4789 openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs enable_tunneling True openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_id_ranges 1:1000 openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tenant_network_type vxlan openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs integration_bridge br-int openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_bridge br-tun openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs local_ip ${myip} openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent l2_population False # Configure Ceilometer for compute node openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit True openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit_period hour openstack-config --set /etc/nova/nova.conf DEFAULT notify_on_state_change vm_and_task_state openstack-config --set /etc/nova/nova.conf DEFAULT notification_driver nova.openstack.common.notifier.rpc_notifier sed -i -e 's/nova.openstack.common.notifier.rpc_notifier/nova.openstack.common.notifier.rpc_notifier\nnotification_driver = ceilometer.compute.nova_notifier/g' /etc/nova/nova.conf openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_port 35357 openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_protocol http openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_user ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://${PHD_VAR_network_hosts_vip}:5000/v2.0 openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://${PHD_VAR_network_hosts_mongodb}:27017/ceilometer?replicaSet=ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf database connection max_retries -1 # keep last 5 days data only (value is in secs) openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000 # Cinder uses NFS setsebool -P virt_use_nfs 1 # Kernel TCP keepalive parameters cat > /etc/sysctl.d/tcpka.conf << EOF net.ipv4.tcp_keepalive_intvl = 1 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_keepalive_time = 5 EOF sysctl -p /etc/sysctl.d/tcpka.conf # Start services, open firewall ports systemctl start libvirtd systemctl start neutron-openvswitch-agent systemctl enable neutron-openvswitch-agent systemctl enable neutron-ovs-cleanup systemctl start openstack-ceilometer-compute systemctl enable openstack-ceilometer-compute systemctl start openstack-nova-compute systemctl enable openstack-nova-compute firewall-cmd --add-port=4789/udp firewall-cmd --add-port=4789/udp --permanent firewall-cmd --add-port=5900-5999/tcp firewall-cmd --add-port=5900-5999/tcp --permanent .... ================================================ FILE: keepalived/phd-setup/galera.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Install MariaDB / Galera # - Configure Galera # - Bootstrap an initial Galera cluster # - Create databases for OpenStack services ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y mariadb-galera-server xinetd rsync psmisc cat > /etc/sysconfig/clustercheck << EOF MYSQL_USERNAME="clustercheck" MYSQL_PASSWORD="redhat" MYSQL_HOST="localhost" MYSQL_PORT="3306" EOF systemctl start mysqld mysql -e "CREATE USER 'clustercheck'@'localhost' IDENTIFIED BY 'redhat';" systemctl stop mysqld myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) cat > /etc/my.cnf.d/galera.cnf << EOF [mysqld] skip-name-resolve=1 binlog_format=ROW default-storage-engine=innodb innodb_autoinc_lock_mode=2 innodb_locks_unsafe_for_binlog=1 max_connections=2048 query_cache_size=0 query_cache_type=0 bind_address=${myip} wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_cluster_name="galera_cluster" wsrep_cluster_address="gcomm://192.168.1.221,192.168.1.222,192.168.1.223" wsrep_slave_threads=1 wsrep_certify_nonPK=1 wsrep_max_ws_rows=131072 wsrep_max_ws_size=1073741824 wsrep_debug=0 wsrep_convert_LOCK_to_trx=0 wsrep_retry_autocommit=1 wsrep_auto_increment_control=1 wsrep_drupal_282555_workaround=0 wsrep_causal_reads=0 wsrep_notify_cmd= wsrep_sst_method=rsync EOF mkdir -p /etc/systemd/system/mariadb.service.d/ cat > /etc/systemd/system/mariadb.service.d/limits.conf << EOF [Service] LimitNOFILE=16384 EOF cat > /etc/xinetd.d/galera-monitor << EOF service galera-monitor { port = 9200 disable = no socket_type = stream protocol = tcp wait = no user = root group = root groups = yes server = /usr/bin/clustercheck type = UNLISTED per_source = UNLIMITED log_on_success = log_on_failure = HOST flags = REUSE } EOF systemctl daemon-reload systemctl enable xinetd systemctl start xinetd systemctl enable haproxy systemctl start haproxy firewall-cmd --add-service=mysql firewall-cmd --add-port=4444/tcp firewall-cmd --add-port=4567/tcp firewall-cmd --add-port=4568/tcp firewall-cmd --add-port=4568/tcp --permanent firewall-cmd --add-service=mysql --permanent firewall-cmd --add-port=4567/tcp --permanent firewall-cmd --add-port=4444/tcp --permanent firewall-cmd --add-port=9300/tcp firewall-cmd --add-port=9300/tcp --permanent firewall-cmd --add-port=9200/tcp firewall-cmd --add-port=9200/tcp --permanent systemctl enable mariadb .... target=$PHD_ENV_nodes1 .... # This is required to allow sudo execution without a tty sed -i 's/Defaults requiretty/Defaults !requiretty/g' /etc/sudoers nohup sudo -u mysql /usr/libexec/mysqld --wsrep-cluster-address='gcomm://' < /dev/null > /dev/null 2>&1 & sleep 30 # A little cleanup sed -i 's/Defaults !requiretty/Defaults requiretty/g' /etc/sudoers .... target=$PHD_ENV_nodes2 .... systemctl start mariadb sleep 10 .... target=$PHD_ENV_nodes3 .... systemctl start mariadb sleep 10 .... target=$PHD_ENV_nodes1 .... cat > /tmp/mysql.sql << EOF use mysql; GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED by 'mysqltest' WITH GRANT OPTION; CREATE DATABASE keystone; GRANT ALL ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'keystonetest'; CREATE DATABASE glance; GRANT ALL ON glance.* TO 'glance'@'%' IDENTIFIED BY 'glancetest'; CREATE DATABASE cinder; GRANT ALL ON cinder.* TO 'cinder'@'%' IDENTIFIED BY 'cindertest'; CREATE DATABASE neutron; GRANT ALL ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'neutrontest'; CREATE DATABASE nova; GRANT ALL ON nova.* TO 'nova'@'%' IDENTIFIED BY 'novatest'; CREATE DATABASE heat; GRANT ALL ON heat.* TO 'heat'@'%' IDENTIFIED BY 'heattest'; CREATE DATABASE sahara; GRANT ALL ON sahara.* TO 'sahara'@'%' IDENTIFIED BY 'saharatest'; CREATE DATABASE trove; GRANT ALL ON trove.* TO 'trove'@'%' IDENTIFIED BY 'trovetest'; FLUSH PRIVILEGES; EOF killall mysqld # it takes some time for mysqld to actually stop after you kill it sleep 30 systemctl start mariadb mysql < /tmp/mysql.sql > /tmp/mysql.out rm -f /tmp/mysql.sql mysqladmin flush-hosts .... ================================================ FILE: keepalived/phd-setup/glance.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing Glance # - Configuring Glance # - Starting services and opening firewall rules ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_hosts_vip PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_nfs_glanceshare ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) yum install -y openstack-glance openstack-utils openstack-selinux nfs-utils openstack-config --set /etc/glance/glance-api.conf database connection mysql://glance:glancetest@${PHD_VAR_network_hosts_vip}/glance openstack-config --set /etc/glance/glance-api.conf database max_retries -1 openstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone openstack-config --set /etc/glance/glance-api.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_user glance openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_password glancetest openstack-config --set /etc/glance/glance-api.conf DEFAULT notification_driver messaging openstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host ${myip} openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/glance/glance-registry.conf database connection mysql://glance:glancetest@${PHD_VAR_network_hosts_vip}/glance openstack-config --set /etc/glance/glance-registry.conf database max_retries -1 openstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_user glance openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_password glancetest openstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host ${myip} .... target=$PHD_ENV_nodes1 .... su glance -s /bin/sh -c "glance-manage db_sync" .... target=all .... echo "${PHD_VAR_network_nfs_glanceshare} /var/lib/glance nfs vers=3 0 0" >> /etc/fstab # Workaround for bz#1203820 systemctl start rpcbind systemctl start nfs-config systemctl start rpc-statd mount -a chown glance:nobody /var/lib/glance systemctl start openstack-glance-registry systemctl start openstack-glance-api systemctl enable openstack-glance-registry systemctl enable openstack-glance-api firewall-cmd --add-port=9191/tcp firewall-cmd --add-port=9191/tcp --permanent firewall-cmd --add-port=9292/tcp firewall-cmd --add-port=9292/tcp --permanent .... ================================================ FILE: keepalived/phd-setup/ha-collapsed.variables ================================================ # Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD # Each scenario file will verify that values have been provided for the variables it requires. deployment: collapsed hypervisors: controllers: controller1:oslab1,controller2:oslab2,controller3:oslab3 computenodes: compute1:oslab1,compute2:oslab3 rhn: user: jpena@redhat.com pass: OSA07novp7her network: ips: vip: 192.168.1.220 controllers: 192.168.1.221,192.168.1.222,192.168.1.223 computeinternal: 192.168.1.224,192.168.1.225 computeexternal: 10.10.10.224,10.10.10.225 gateway: 192.168.1.1 netmask: 255.255.255.0 nfs: glanceshare: 192.168.1.4:/volumeUSB1/usbshare/openstack/glance cindershare: 192.168.1.4:/volumeUSB1/usbshare/openstack/cinder nic: base: 54:52:00 internal: eth0 external: eth1 hosts: vip: controller-vip.example.com controllers: controller1.example.com, controller2.example.com, controller3.example.com compute: compute1.example.com, compute2.example.com mongodb: controller1,controller2,controller3 memcache: controller1:11211,controller2:11211,controller3:11211 rabbitmq: controller1,controller2,controller3 domain: example.com neutron: externalgateway: 10.10.10.1 externalnetwork: 10.10.10.0/24 allocpoolstart: 10.10.10.100 allocpoolend: 10.10.10.150 ssh: pubkey: ssh-dss AAAAB3NzaC1kc3MAAACBANTRU30h4EC1sv5kb/KJBDYx0D/YKWAVzMJpeWLzFrECF41Tymte8W5s4oz0CzpQQH4r19YFpeS+jingeakkRc749ImwhJE77yRmPsjaEDe3OZXW+Udz4dikE5HBjuVlEL9UN3yoGfBZCXo9Jm+oRJCH+wx36Pl69IHRZwJIAvQfAAAAFQDTdIeHmLP77PvjG5jo6eEYDTedXwAAAIEAvwKlws7V3qy6Ckd26FcAr1noN+myBdVVpb960F7BPWIxXkA9z12o+3y5JEUJRVTplRf8pBz1JwKdsYS7WJj8teSslmwbQz0kQ61uSNkWqBZGOLE/ddQxCJbD3OTXRZ289A/2mRBcil8mIg82JY1AzNr1OaQQ82ZP35nISA/jeQMAAACBAIeuHCLqkYRf7Xuv2ztFOhPzLrJRRSm7Zcpdxp5Ano7R68Yq1RQbMn0lUIG4QMssviQPyklF/xHYGJLUXBIUfNORn7BYqHikbcguUHKiwiC9io9xR9+zlvgIAIU6bbnsNT9MofTdrRT+H7df6cDP8NO43w8dWFXWX4K3NUU0YjbM components: serverprep lb galera rabbitmq keystone memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer node osp: major: 7 env: password: cluster # TODO... password: cluster: foo keystone: bar ================================================ FILE: keepalived/phd-setup/heat.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing Heat # - Creating the required Heat domain # - Configuring Heat # - Starting services and opening firewall rules ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_nic_external PHD_VAR_network_hosts_vip PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_hosts_memcache PHD_VAR_network_neutron_externalgateway PHD_VAR_network_neutron_externalnetwork PHD_VAR_network_neutron_allocpoolstart PHD_VAR_network_neutron_allocpoolend ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-heat-engine openstack-heat-api openstack-heat-api-cfn openstack-heat-api-cloudwatch python-heatclient openstack-utils python-glanceclient .... target=$PHD_ENV_nodes1 .... . /root/keystonerc_admin openstack role create heat_stack_user TOKEN_ID=$(openstack token issue --format value --column id) openstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 domain create heat --description "Owns users and projects created by heat" openstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 user create --password heattest --domain heat --description "Manages users and projects created by heat" heat_domain_admin openstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 role add --user heat_domain_admin --domain heat admin .... target=all .... myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) . /root/keystonerc_admin TOKEN_ID=$(openstack token issue --format value --column id) HEAT_DOMAIN_ID=$(openstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 domain show heat --column id --format value) openstack-config --set /etc/heat/heat.conf DEFAULT stack_domain_admin_password heattest openstack-config --set /etc/heat/heat.conf DEFAULT stack_domain_admin heat_domain_admin openstack-config --set /etc/heat/heat.conf DEFAULT stack_user_domain_id ${HEAT_DOMAIN_ID} openstack-config --set /etc/heat/heat.conf database connection mysql://heat:heattest@${PHD_VAR_network_hosts_vip}/heat openstack-config --set /etc/heat/heat.conf database max_retries -1 openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_user heat openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_password heattest openstack-config --set /etc/heat/heat.conf keystone_authtoken service_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/heat/heat.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/heat/heat.conf keystone_authtoken keystone_ec2_uri http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/heat/heat.conf ec2authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:5000/v2.0 openstack-config --set /etc/heat/heat.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/heat/heat.conf heat_api bind_host ${myip} openstack-config --set /etc/heat/heat.conf heat_api_cfn bind_host ${myip} openstack-config --set /etc/heat/heat.conf heat_api_cloudwatch bind_host ${myip} openstack-config --set /etc/heat/heat.conf DEFAULT heat_metadata_server_url http://${PHD_VAR_network_hosts_vip}:8000 openstack-config --set /etc/heat/heat.conf DEFAULT heat_waitcondition_server_url http://${PHD_VAR_network_hosts_vip}:8000/v1/waitcondition openstack-config --set /etc/heat/heat.conf DEFAULT heat_watch_server_url http://${PHD_VAR_network_hosts_vip}:8003 openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/heat/heat.conf DEFAULT rpc_backend rabbit openstack-config --set /etc/heat/heat.conf DEFAULT notification_driver heat.openstack.common.notifier.rpc_notifier openstack-config --set /etc/heat/heat.conf DEFAULT enable_cloud_watch_lite false .... target=$PHD_ENV_nodes1 .... su heat -s /bin/sh -c "heat-manage db_sync" .... target=all .... systemctl start openstack-heat-api systemctl start openstack-heat-api-cfn systemctl start openstack-heat-api-cloudwatch systemctl start openstack-heat-engine systemctl enable openstack-heat-api systemctl enable openstack-heat-api-cfn systemctl enable openstack-heat-api-cloudwatch systemctl enable openstack-heat-engine firewall-cmd --add-port=8000/tcp firewall-cmd --add-port=8000/tcp --permanent firewall-cmd --add-port=8003/tcp firewall-cmd --add-port=8003/tcp --permanent firewall-cmd --add-port=8004/tcp firewall-cmd --add-port=8004/tcp --permanent .... ================================================ FILE: keepalived/phd-setup/horizon.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing Horizon packages # - Configuring the OpenStack dashboard # - Starting the service and opening firewall rules ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_memcache PHD_VAR_network_hosts_vip ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y mod_wsgi httpd mod_ssl python-memcached openstack-dashboard .... target=$PHD_ENV_nodes1 .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" openssl rand -hex 10 > /tmp/horizon_secret.key scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p /tmp/horizon_secret.key ${controller_ips[1]}:/tmp scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p /tmp/horizon_secret.key ${controller_ips[2]}:/tmp .... target=all .... IFS=', ' read -a memcached_hosts <<< "${PHD_VAR_network_hosts_memcache}" SECRET_KEY=$(cat /tmp/horizon_secret.key) rm -f /tmp/horizon_secret.key sed -i -e "s#SECRET_KEY.*#SECRET_KEY = \'${SECRET_KEY}\'#g#" /etc/openstack-dashboard/local_settings sed -i -e "s#ALLOWED_HOSTS.*#ALLOWED_HOSTS = ['*',]#g" \ -e "s#^CACHES#SESSION_ENGINE = 'django.contrib.sessions.backends.cache'\nCACHES#g#" \ -e "s#locmem.LocMemCache'#memcached.MemcachedCache',\n\t'LOCATION' : [ '${memcached_hosts[0]}', '${memcached_hosts[1]}', '${memcached_hosts[2]}', ]#g" \ -e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = "${PHD_VAR_network_hosts_vip}"#g' \ -e "s#^LOCAL_PATH.*#LOCAL_PATH = '/var/lib/openstack-dashboard'#g" \ /etc/openstack-dashboard/local_settings systemctl daemon-reload systemctl restart httpd firewall-cmd --add-port=80/tcp firewall-cmd --add-port=80/tcp --permanent .... ================================================ FILE: keepalived/phd-setup/hypervisors.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Creating the VMs that form the environment (see readme.txt for details), # including required network and storage. # - Setting basic repos on the VMs ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_hypervisors_controllers PHD_VAR_hypervisors_computenodes PHD_VAR_network_ips_controllers PHD_VAR_network_ips_computeinternal PHD_VAR_network_ips_computeexternal PHD_VAR_network_ips_gateway PHD_VAR_network_ips_netmask PHD_VAR_network_domain PHD_VAR_rhn_user PHD_VAR_rhn_pass PHD_VAR_network_ssh_pubkey ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... create_vm() { vmname=$1 export cont_ip=$2 export external_ip=$3 virt-clone --force -o rhel7-base -n rhel7-${vmname} -f /var/lib/libvirt/images/rhel7-${vmname}.qcow2 virsh start rhel7-${vmname} VMMAC="" while [ "x$VMMAC" == "x" ] do VMMAC=$(virsh dumpxml rhel7-${vmname} | grep "mac address" | awk -F\' '{print $2}') if [ "x$VMMAC" == "x" ] then echo "MAC not found yet, sleeping..." sleep 5 fi done VMIP="" while [ "x$VMIP" == "x" ] do VMIP=$(arp -an | grep $VMMAC | awk '{print $2}' | tr -d \( | tr -d \)) if [ "x$VMIP" == "x" ] then echo "IP not found yet, sleeping..." sleep 5 else echo "VM booted with IP " $VMIP fi done SSHOK=0 while [ ${SSHOK} -eq 0 ] do ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null root@${VMIP} ls /root > /dev/null if [ $? -eq 0 ] then SSHOK=1 else echo "Cannot connect via SSH. Waiting" sleep 10 fi done ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null root@${VMIP} "/bin/bash " << EOF echo "${vmname}.${PHD_VAR_network_domain}" > /etc/hostname subscription-manager register --username=${PHD_VAR_rhn_user} --password=${PHD_VAR_rhn_pass} subscription-manager attach --pool=8a85f9874011071c01407da00b997cb2 subscription-manager repos --disable \* subscription-manager repos --enable rhel-7-server-rpms subscription-manager repos --enable rhel-7-server-rh-common-rpms subscription-manager repos --enable rhel-7-server-optional-rpms subscription-manager repos --enable rhel-7-server-extras-rpms yum -y install wget vim wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm yum -y localinstall epel-release-7*.rpm yum -y install http://rdoproject.org/repos/openstack-kilo/rdo-release-kilo.rpm yum -y update cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOF2 TYPE="Ethernet" BOOTPROTO="static" IPV4_FAILURE_FATAL="no" NAME="eth0" DEVICE=eth0 IPADDR=${cont_ip} NETMASK=${PHD_VAR_network_ips_netmask} GATEWAY=${PHD_VAR_network_ips_gateway} ONBOOT="yes" EOF2 cat > /etc/sysconfig/network-scripts/ifcfg-eth1 << EOF3 TYPE="Ethernet" BOOTPROTO="static" NAME="eth1" DEVICE=eth1 ONBOOT="yes" EOF3 if [ "x${external_ip}" != "x" ] then echo "IPADDR=${external_ip}" >> /etc/sysconfig/network-scripts/ifcfg-eth1 echo "NETMASK=255.255.255.0" >> /etc/sysconfig/network-scripts/ifcfg-eth1 fi echo "NETWORKING=yes" > /etc/sysconfig/network echo "nameserver 8.8.8.8" > /etc/resolv.conf echo ${PHD_VAR_network_ssh_pubkey} >> /root/.ssh/authorized_keys #This is a big hack (same ssh private key everywher), until I figure out a better way echo ${PHD_VAR_network_ssh_pubkey} > /root/.ssh/id_dsa.pub cat > /root/.ssh/id_dsa << EOF4 -----BEGIN DSA PRIVATE KEY----- MIIBvAIBAAKBgQDU0VN9IeBAtbL+ZG/yiQQ2MdA/2ClgFczCaXli8xaxAheNU8pr XvFubOKM9As6UEB+K9fWBaXkvo4p4HmpJEXO+PSJsISRO+8kZj7I2hA3tzmV1vlH c+HYpBORwY7lZRC/VDd8qBnwWQl6PSZvqESQh/sMd+j5evSB0WcCSAL0HwIVANN0 h4eYs/vs++MbmOjp4RgNN51fAoGBAL8CpcLO1d6sugpHduhXAK9Z6DfpsgXVVaW/ etBewT1iMV5APc9dqPt8uSRFCUVU6ZUX/KQc9ScCnbGEu1iY/LXkrJZsG0M9JEOt bkjZFqgWRjixP3XUMQiWw9zk10WdvPQP9pkQXIpfJiIPNiWNQMza9TmkEPNmT9+Z yEgP43kDAoGBAIeuHCLqkYRf7Xuv2ztFOhPzLrJRRSm7Zcpdxp5Ano7R68Yq1RQb Mn0lUIG4QMssviQPyklF/xHYGJLUXBIUfNORn7BYqHikbcguUHKiwiC9io9xR9+z lvgIAIU6bbnsNT9MofTdrRT+H7df6cDP8NO43w8dWFXWX4K3NUU0YjbMAhQsr5fG vYKFMrvzTcdgEdtWeXx9uA== -----END DSA PRIVATE KEY----- EOF4 chmod 600 /root/.ssh/id_dsa systemctl disable NetworkManager systemctl enable network poweroff EOF sleep 60 # Give time for poweroff virsh dumpxml rhel7-${vmname} > /tmp/vm.txt sed -i "s#type='network'#type='bridge'#" /tmp/vm.txt sed -i "s#network='default'#bridge='br-public'#" /tmp/vm.txt virsh define /tmp/vm.txt virsh attach-interface rhel7-${vmname} bridge br-private --persistent --model virtio # Create and attach second disk (used by swift) qemu-img create -f qcow2 /var/lib/libvirt/images/rhel7-${vmname}-2.qcow2 8g virsh attach-disk --persistent --subdriver qcow2 rhel7-${vmname} /var/lib/libvirt/images/rhel7-${vmname}-2.qcow2 vdb virsh start rhel7-${vmname} } IFS=', ' read -a pair <<< "${PHD_VAR_hypervisors_controllers}" IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" for item in "${!pair[@]}" do vmname=$(echo ${pair[item]} | awk -F: '{print $1}') hypervisor=$(echo ${pair[item]} | awk -F: '{print $2}') cont_ip=${controller_ips[item]} if [ "$hypervisor" = `hostname -s` ] then create_vm $vmname $cont_ip fi done IFS=', ' read -a pair <<< "${PHD_VAR_hypervisors_computenodes}" IFS=', ' read -a internal_ips <<< "${PHD_VAR_network_ips_computeinternal}" IFS=', ' read -a external_ips <<< "${PHD_VAR_network_ips_computeexternal}" for item in "${!pair[@]}" do vmname=$(echo ${pair[item]} | awk -F: '{print $1}') hypervisor=$(echo ${pair[item]} | awk -F: '{print $2}') int_ip=${internal_ips[item]} ext_ip=${external_ips[item]} if [ "$hypervisor" = `hostname -s` ] then create_vm $vmname $int_ip $ext_ip fi done .... ================================================ FILE: keepalived/phd-setup/keepalived.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing keepalived # - Configuring keepalived # - Starting the service and opening the required firewall rule ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_ips_vip PHD_VAR_network_nic_internal ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y keepalived psmisc cat > /etc/keepalived/keepalived.conf << EOF vrrp_script chk_haproxy { script "/usr/bin/killall -0 haproxy" interval 2 } vrrp_instance VI_PUBLIC { interface ${PHD_VAR_network_nic_internal} state BACKUP virtual_router_id 52 priority 101 virtual_ipaddress { ${PHD_VAR_network_ips_vip} dev ${PHD_VAR_network_nic_internal} } track_script { chk_haproxy } } vrrp_sync_group VG1 group { VI_PUBLIC } EOF firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -i ${PHD_VAR_network_nic_internal} -d 224.0.0.0/8 -j ACCEPT firewall-cmd --direct --perm --add-rule ipv4 filter INPUT 0 -i ${PHD_VAR_network_nic_internal} -d 224.0.0.0/8 -j ACCEPT systemctl start keepalived systemctl enable keepalived .... ================================================ FILE: keepalived/phd-setup/keystone.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing Keystone # - Configuring Keystone # - Starting the service, opening firewall rules # - Creating user accounts, roles, projects, services and endpoints for all # services # - Creating a keystonerc file ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_hosts_vip PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-keystone openstack-utils openstack-selinux httpd mod_wsgi python-openstackclient .... target=$PHD_ENV_nodes1 .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" export SERVICE_TOKEN=$(openssl rand -hex 10) echo $SERVICE_TOKEN > /root/keystone_service_token scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /root/keystone_service_token root@${controller_ips[1]}:/root scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /root/keystone_service_token root@${controller_ips[2]}:/root .... target=all .... servername=$(hostname -s) mkdir /var/www/cgi-bin/keystone chown keystone:keystone /var/www/cgi-bin/keystone ln /usr/share/keystone/keystone.wsgi /var/www/cgi-bin/keystone/admin ln /usr/share/keystone/keystone.wsgi /var/www/cgi-bin/keystone/main restorecon -R /var/www/cgi-bin/keystone /etc/httpd cat > /etc/httpd/conf.d/10-keystone_wsgi_admin.conf << EOF ServerName ${servername} ## Vhost docroot DocumentRoot "/var/www/cgi-bin/keystone" ## Directories, there should at least be a declaration for /var/www/cgi-bin/keystone Options Indexes FollowSymLinks MultiViews AllowOverride None Require all granted ## Logging ErrorLog "/var/log/httpd/keystone_wsgi_admin_error.log" ServerSignature Off CustomLog "/var/log/httpd/keystone_wsgi_admin_access.log" combined WSGIDaemonProcess keystone_admin group=keystone processes=1 threads=1 user=keystone WSGIProcessGroup keystone_admin WSGIScriptAlias / "/var/www/cgi-bin/keystone/admin" EOF cat > /etc/httpd/conf.d/10-keystone_wsgi_main.conf << EOF ServerName ${servername} ## Vhost docroot DocumentRoot "/var/www/cgi-bin/keystone" ## Directories, there should at least be a declaration for /var/www/cgi-bin/keystone Options Indexes FollowSymLinks MultiViews AllowOverride None Require all granted ## Logging ErrorLog "/var/log/httpd/keystone_wsgi_main_error.log" ServerSignature Off CustomLog "/var/log/httpd/keystone_wsgi_main_access.log" combined WSGIDaemonProcess keystone_main group=keystone processes=1 threads=1 user=keystone WSGIProcessGroup keystone_main WSGIScriptAlias / "/var/www/cgi-bin/keystone/main" EOF myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) sed -i -e "s/^Listen.*/Listen ${myip}:80/g" /etc/httpd/conf/httpd.conf echo "Listen ${myip}:5000" >> /etc/httpd/conf/httpd.conf echo "Listen ${myip}:35357" >> /etc/httpd/conf/httpd.conf export SERVICE_TOKEN=$(cat /root/keystone_service_token) openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token $SERVICE_TOKEN openstack-config --set /etc/keystone/keystone.conf DEFAULT rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/keystone/keystone.conf DEFAULT rabbit_ha_queues true openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_endpoint "http://${PHD_VAR_network_hosts_vip}:%(admin_port)s/" openstack-config --set /etc/keystone/keystone.conf DEFAULT public_endpoint "http://${PHD_VAR_network_hosts_vip}:%(public_port)s/" openstack-config --set /etc/keystone/keystone.conf database connection mysql://keystone:keystonetest@${PHD_VAR_network_hosts_vip}/keystone openstack-config --set /etc/keystone/keystone.conf database max_retries -1 openstack-config --set /etc/keystone/keystone.conf DEFAULT public_bind_host ${myip} openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_bind_host ${myip} openstack-config --set /etc/keystone/keystone.conf token driver keystone.token.persistence.backends.sql.Token .... target=$PHD_ENV_nodes1 .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" keystone-manage pki_setup --keystone-user keystone --keystone-group keystone chown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/ su keystone -s /bin/sh -c "keystone-manage db_sync" cd /etc/keystone/ssl tar cvfz /tmp/keystone_ssl.tgz * scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /tmp/keystone_ssl.tgz ${controller_ips[1]}:/tmp scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /tmp/keystone_ssl.tgz ${controller_ips[2]}:/tmp .... target=$PHD_ENV_nodes2 .... mkdir -p /etc/keystone/ssl cd /etc/keystone/ssl tar xvfz /tmp/keystone_ssl.tgz chown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/ restorecon -Rv /etc/keystone/ssl touch /var/log/keystone/keystone.log chown keystone:keystone /var/log/keystone/keystone.log .... target=$PHD_ENV_nodes3 .... mkdir -p /etc/keystone/ssl cd /etc/keystone/ssl tar xvfz /tmp/keystone_ssl.tgz chown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/ restorecon -Rv /etc/keystone/ssl touch /var/log/keystone/keystone.log chown keystone:keystone /var/log/keystone/keystone.log .... target=all .... echo "1 * * * * keystone keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1" >> /etc/crontab firewall-cmd --add-port=5000/tcp firewall-cmd --add-port=5000/tcp --permanent firewall-cmd --add-port=35357/tcp firewall-cmd --add-port=35357/tcp --permanent systemctl start httpd systemctl enable httpd .... target=$PHD_ENV_nodes1 .... export OS_TOKEN=$(cat /root/keystone_service_token) export OS_URL=http://${PHD_VAR_network_hosts_vip}:35357/v2.0 export OS_REGION_NAME=regionOne openstack service create --name=keystone --description="Keystone Identity Service" identity openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:5000/v2.0" --adminurl "http://${PHD_VAR_network_hosts_vip}:35357/v2.0" --internalurl "http://${PHD_VAR_network_hosts_vip}:5000/v2.0" --region regionOne keystone openstack user create --password keystonetest admin openstack role create admin openstack project create admin openstack role add --project admin --user admin admin openstack user create --password redhat demo openstack role create _member_ openstack project create demo openstack role add --project demo --user demo _member_ openstack project create --description "Services Tenant" services # glance openstack user create --password glancetest glance openstack role add --project services --user glance admin openstack service create --name=glance --description="Glance Image Service" image openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:9292" --adminurl "http://${PHD_VAR_network_hosts_vip}:9292" --internalurl "http://${PHD_VAR_network_hosts_vip}:9292" --region regionOne glance # cinder openstack user create --password cindertest cinder openstack role add --project services --user cinder admin openstack service create --name=cinder --description="Cinder Volume Service" volume openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:8776/v1/\$(tenant_id)s" --adminurl "http://${PHD_VAR_network_hosts_vip}:8776/v1/\$(tenant_id)s" --internalurl "http://${PHD_VAR_network_hosts_vip}:8776/v1/\$(tenant_id)s" --region regionOne cinder openstack service create --name=cinderv2 --description="OpenStack Block Storage" volumev2 openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:8776/v2/\$(tenant_id)s" --adminurl "http://${PHD_VAR_network_hosts_vip}:8776/v2/\$(tenant_id)s" --internalurl "http://${PHD_VAR_network_hosts_vip}:8776/v2/\$(tenant_id)s" --region regionOne cinderv2 # swift openstack user create --password swifttest swift openstack role add --project services --user swift admin openstack service create --name=swift --description="Swift Storage Service" object-store openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:8080/v1/AUTH_\$(tenant_id)s" --adminurl "http://${PHD_VAR_network_hosts_vip}:8080/v1" --internalurl "http://${PHD_VAR_network_hosts_vip}:8080/v1/AUTH_\$(tenant_id)s" --region regionOne swift # neutron openstack user create --password neutrontest neutron openstack role add --project services --user neutron admin openstack service create --name=neutron --description="OpenStack Networking Service" network openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:9696" --adminurl "http://${PHD_VAR_network_hosts_vip}:9696" --internalurl "http://${PHD_VAR_network_hosts_vip}:9696" --region regionOne neutron # nova openstack user create --password novatest compute openstack role add --project services --user compute admin openstack service create --name=compute --description="OpenStack Compute Service" compute openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:8774/v2/\$(tenant_id)s" --adminurl "http://${PHD_VAR_network_hosts_vip}:8774/v2/\$(tenant_id)s" --internalurl "http://${PHD_VAR_network_hosts_vip}:8774/v2/\$(tenant_id)s" --region regionOne compute # heat openstack user create --password heattest heat openstack role add --project services --user heat admin openstack service create --name=heat --description="Heat Orchestration Service" orchestration openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:8004/v1/%(tenant_id)s" --adminurl "http://${PHD_VAR_network_hosts_vip}:8004/v1/%(tenant_id)s" --internalurl "http://${PHD_VAR_network_hosts_vip}:8004/v1/%(tenant_id)s" --region regionOne heat openstack service create --name=heat-cfn --description="Heat CloudFormation Service" cloudformation openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:8000/v1" --adminurl "http://${PHD_VAR_network_hosts_vip}:8000/v1" --internalurl "http://${PHD_VAR_network_hosts_vip}:8000/v1" --region regionOne heat-cfn # ceilometer openstack user create --password ceilometertest ceilometer openstack role add --project services --user ceilometer admin openstack role create ResellerAdmin openstack role add --project services --user ceilometer ResellerAdmin openstack service create --name=ceilometer --description="OpenStack Telemetry Service" metering openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:8777" --adminurl "http://${PHD_VAR_network_hosts_vip}:8777" --internalurl "http://${PHD_VAR_network_hosts_vip}:8777" --region regionOne ceilometer # sahara openstack user create --password saharatest sahara openstack role add --project services --user sahara admin openstack service create --name=sahara --description="Sahara Data Processing" data-processing openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:8386/v1.1/%(tenant_id)s" --adminurl "http://${PHD_VAR_network_hosts_vip}:8386/v1.1/%(tenant_id)s" --internalurl "http://${PHD_VAR_network_hosts_vip}:8386/v1.1/%(tenant_id)s" --region regionOne sahara # trove openstack user create --password trovetest trove openstack role add --project services --user trove admin openstack service create --name=trove --description="OpenStack Database Service" database openstack endpoint create --publicurl "http://${PHD_VAR_network_hosts_vip}:8779/v1.0/%(tenant_id)s" --adminurl "http://${PHD_VAR_network_hosts_vip}:8779/v1.0/%(tenant_id)s" --internalurl "http://${PHD_VAR_network_hosts_vip}:8779/v1.0/%(tenant_id)s" --region regionOne trove .... target=all .... cat > /root/keystonerc_admin << EOF export OS_USERNAME=admin export OS_TENANT_NAME=admin export OS_PROJECT_NAME=admin export OS_REGION_NAME=regionOne export OS_PASSWORD=keystonetest export OS_AUTH_URL=http://${PHD_VAR_network_hosts_vip}:35357/v2.0/ export PS1='[\u@\h \W(keystone_admin)]\$ ' EOF cat > /root/keystonerc_demo << EOF export OS_USERNAME=demo export OS_TENANT_NAME=demo export OS_PROJECT_NAME=demo export OS_REGION_NAME=regionOne export OS_PASSWORD=redhat export OS_AUTH_URL=http://${PHD_VAR_network_hosts_vip}:5000/v2.0/ export PS1='[\u@\h \W(keystone_user)]\$ ' EOF .... ================================================ FILE: keepalived/phd-setup/lb.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Tweaking the IP stack to allow nonlocal binding # - Configuring haproxy ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_ips_vip PHD_VAR_network_ips_controllers PHD_VAR_deployment PHD_VAR_components PHD_VAR_osp_major ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y haproxy openstack-selinux echo net.ipv4.ip_nonlocal_bind=1 >> /etc/sysctl.d/haproxy.conf echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" # HA Proxy defaults cat > /etc/haproxy/haproxy.cfg << EOF global daemon stats socket /var/lib/haproxy/stats defaults mode tcp maxconn 10000 timeout connect 5s timeout client 30s timeout server 30s listen monitor bind ${PHD_VAR_network_ips_vip}:9300 mode http monitor-uri /status stats enable stats uri /admin stats realm Haproxy\ Statistics stats auth root:redhat stats refresh 5s frontend vip-db bind ${PHD_VAR_network_ips_vip}:3306 timeout client 90m default_backend db-vms-galera backend db-vms-galera option httpchk stick-table type ip size 1000 stick on dst timeout server 90m server rhos7-node1 ${controller_ips[0]}:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions server rhos7-node2 ${controller_ips[1]}:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions server rhos7-node3 ${controller_ips[2]}:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions # Note the RabbitMQ entry is only needed for CloudForms compatibility # and should be removed in the future frontend vip-rabbitmq option clitcpka bind ${PHD_VAR_network_ips_vip}:5672 timeout client 900m default_backend rabbitmq-vms backend rabbitmq-vms option srvtcpka balance roundrobin timeout server 900m server rhos7-node1 ${controller_ips[0]}:5672 check inter 1s server rhos7-node2 ${controller_ips[1]}:5672 check inter 1s server rhos7-node3 ${controller_ips[2]}:5672 check inter 1s frontend vip-keystone-admin bind ${PHD_VAR_network_ips_vip}:35357 default_backend keystone-admin-vms timeout client 600s backend keystone-admin-vms balance roundrobin timeout server 600s server rhos7-node1 ${controller_ips[0]}:35357 check inter 1s on-marked-down shutdown-sessions server rhos7-node2 ${controller_ips[1]}:35357 check inter 1s on-marked-down shutdown-sessions server rhos7-node3 ${controller_ips[2]}:35357 check inter 1s on-marked-down shutdown-sessions frontend vip-keystone-public bind ${PHD_VAR_network_ips_vip}:5000 default_backend keystone-public-vms timeout client 600s backend keystone-public-vms balance roundrobin timeout server 600s server rhos7-node1 ${controller_ips[0]}:5000 check inter 1s on-marked-down shutdown-sessions server rhos7-node2 ${controller_ips[1]}:5000 check inter 1s on-marked-down shutdown-sessions server rhos7-node3 ${controller_ips[2]}:5000 check inter 1s on-marked-down shutdown-sessions frontend vip-glance-api bind ${PHD_VAR_network_ips_vip}:9191 default_backend glance-api-vms backend glance-api-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:9191 check inter 1s server rhos7-node2 ${controller_ips[1]}:9191 check inter 1s server rhos7-node3 ${controller_ips[2]}:9191 check inter 1s frontend vip-glance-registry bind ${PHD_VAR_network_ips_vip}:9292 default_backend glance-registry-vms backend glance-registry-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:9292 check inter 1s server rhos7-node2 ${controller_ips[1]}:9292 check inter 1s server rhos7-node3 ${controller_ips[2]}:9292 check inter 1s frontend vip-cinder bind ${PHD_VAR_network_ips_vip}:8776 default_backend cinder-vms backend cinder-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:8776 check inter 1s server rhos7-node2 ${controller_ips[1]}:8776 check inter 1s server rhos7-node3 ${controller_ips[2]}:8776 check inter 1s frontend vip-swift bind ${PHD_VAR_network_ips_vip}:8080 default_backend swift-vms backend swift-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:8080 check inter 1s server rhos7-node2 ${controller_ips[1]}:8080 check inter 1s server rhos7-node3 ${controller_ips[2]}:8080 check inter 1s frontend vip-neutron bind ${PHD_VAR_network_ips_vip}:9696 default_backend neutron-vms backend neutron-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:9696 check inter 1s server rhos7-node2 ${controller_ips[1]}:9696 check inter 1s server rhos7-node3 ${controller_ips[2]}:9696 check inter 1s frontend vip-nova-vnc-novncproxy bind ${PHD_VAR_network_ips_vip}:6080 default_backend nova-vnc-novncproxy-vms backend nova-vnc-novncproxy-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:6080 check inter 1s server rhos7-node2 ${controller_ips[1]}:6080 check inter 1s server rhos7-node3 ${controller_ips[2]}:6080 check inter 1s frontend nova-metadata-vms bind ${PHD_VAR_network_ips_vip}:8775 default_backend nova-metadata-vms backend nova-metadata-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:8775 check inter 1s server rhos7-node2 ${controller_ips[1]}:8775 check inter 1s server rhos7-node3 ${controller_ips[2]}:8775 check inter 1s frontend vip-nova-api bind ${PHD_VAR_network_ips_vip}:8774 default_backend nova-api-vms backend nova-api-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:8774 check inter 1s server rhos7-node2 ${controller_ips[1]}:8774 check inter 1s server rhos7-node3 ${controller_ips[2]}:8774 check inter 1s frontend vip-horizon bind ${PHD_VAR_network_ips_vip}:80 timeout client 180s default_backend horizon-vms backend horizon-vms balance roundrobin timeout server 180s mode http cookie SERVERID insert indirect nocache server rhos7-node1 ${controller_ips[0]}:80 check inter 1s cookie rhos7-horizon1 on-marked-down shutdown-sessions server rhos7-node2 ${controller_ips[1]}:80 check inter 1s cookie rhos7-horizon2 on-marked-down shutdown-sessions server rhos7-node3 ${controller_ips[2]}:80 check inter 1s cookie rhos7-horizon3 on-marked-down shutdown-sessions frontend vip-heat-cfn bind ${PHD_VAR_network_ips_vip}:8000 default_backend heat-cfn-vms backend heat-cfn-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:8000 check inter 1s server rhos7-node2 ${controller_ips[1]}:8000 check inter 1s server rhos7-node3 ${controller_ips[2]}:8000 check inter 1s frontend vip-heat-cloudw bind ${PHD_VAR_network_ips_vip}:8003 default_backend heat-cloudw-vms backend heat-cloudw-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:8003 check inter 1s server rhos7-node2 ${controller_ips[1]}:8003 check inter 1s server rhos7-node3 ${controller_ips[2]}:8003 check inter 1s frontend vip-heat-srv bind ${PHD_VAR_network_ips_vip}:8004 default_backend heat-srv-vms backend heat-srv-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:8004 check inter 1s server rhos7-node2 ${controller_ips[1]}:8004 check inter 1s server rhos7-node3 ${controller_ips[2]}:8004 check inter 1s frontend vip-ceilometer bind ${PHD_VAR_network_ips_vip}:8777 timeout client 90s default_backend ceilometer-vms backend ceilometer-vms balance roundrobin timeout server 90s server rhos7-node1 ${controller_ips[0]}:8777 check inter 1s server rhos7-node2 ${controller_ips[1]}:8777 check inter 1s server rhos7-node3 ${controller_ips[2]}:8777 check inter 1s frontend vip-sahara bind ${PHD_VAR_network_ips_vip}:8386 default_backend sahara-vms backend sahara-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:8386 check inter 1s server rhos7-node2 ${controller_ips[1]}:8386 check inter 1s server rhos7-node3 ${controller_ips[2]}:8386 check inter 1s frontend vip-trove bind ${PHD_VAR_network_ips_vip}:8779 default_backend trove-vms backend trove-vms balance roundrobin server rhos7-node1 ${controller_ips[0]}:8779 check inter 1s server rhos7-node2 ${controller_ips[1]}:8779 check inter 1s server rhos7-node3 ${controller_ips[2]}:8779 check inter 1s EOF .... ================================================ FILE: keepalived/phd-setup/memcached.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing memcached # - Starting memcached, opening firewall rules ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_ips_vip ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y memcached systemctl start memcached systemctl enable memcached firewall-cmd --add-port=11211/tcp firewall-cmd --add-port=11211/tcp --permanent .... ================================================ FILE: keepalived/phd-setup/mongodb.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing MongoDB # - Setting up MongoDB, including a 3-way replica set # - Starting MongoDB, opening firewall ports ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_hosts_mongodb ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y mongodb mongodb-server sed -i -e 's#bind_ip.*#bind_ip = 0.0.0.0#g' /etc/mongod.conf echo "replSet = ceilometer" >> /etc/mongod.conf systemctl start mongod systemctl enable mongod firewall-cmd --add-port=27017/tcp firewall-cmd --add-port=27017/tcp --permanent .... target=$PHD_ENV_nodes1 .... IFS=', ' read -a controller_names <<< "${PHD_VAR_network_hosts_mongodb}" cat > /tmp/mongoinit.js << EOF rs.initiate() sleep(10000) rs.add("${controller_names[1]}"); rs.add("${controller_names[2]}"); EOF mongo /tmp/mongoinit.js .... ================================================ FILE: keepalived/phd-setup/neutron.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing Neutron # - Configuring Neutron server, FWaaS, LBaaS and Neutron agents # - Starting Neutron services, opening firewall rules # - Creating an external network using a provider network ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_nic_external PHD_VAR_network_hosts_vip PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_hosts_memcache PHD_VAR_network_neutron_externalgateway PHD_VAR_network_neutron_externalnetwork PHD_VAR_network_neutron_allocpoolstart PHD_VAR_network_neutron_allocpoolend ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) yum install -y openstack-neutron openstack-neutron-openvswitch openstack-neutron-ml2 openstack-utils openstack-selinux openstack-config --set /etc/neutron/neutron.conf DEFAULT bind_host ${myip} openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone openstack-config --set /etc/neutron/neutron.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password neutrontest openstack-config --set /etc/neutron/neutron.conf database connection mysql://neutron:neutrontest@${PHD_VAR_network_hosts_vip}:3306/neutron openstack-config --set /etc/neutron/neutron.conf database max_retries -1 openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_url http://${PHD_VAR_network_hosts_vip}:8774/v2 openstack-config --set /etc/neutron/neutron.conf nova nova_region_name regionOne openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true . /root/keystonerc_admin service_tenant_id=$(openstack project show services -f value -c id | head -n 1) # The tenant_id below is the services tenant ID openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_tenant_id ${service_tenant_id} openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_username compute openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_password novatest openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_admin_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes True openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_data_changes True openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin neutron.plugins.ml2.plugin.Ml2Plugin openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router openstack-config --set /etc/neutron/neutron.conf DEFAULT router_scheduler_driver neutron.scheduler.l3_agent_scheduler.ChanceScheduler openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 2 openstack-config --set /etc/neutron/neutron.conf DEFAULT api_workers 2 openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_workers 2 openstack-config --set /etc/neutron/neutron.conf DEFAULT l3_ha True openstack-config --set /etc/neutron/neutron.conf DEFAULT min_l3_agents_per_router 2 openstack-config --set /etc/neutron/neutron.conf DEFAULT max_l3_agents_per_router 2 ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 type_drivers local,gre,flat,vxlan,vlan openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers openvswitch openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks \* openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_gre tunnel_id_ranges 10:10000 openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vni_ranges 10:10000 openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vxlan_group 224.0.0.1 openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_security_group True openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver yum -y install openstack-neutron-lbaas openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router,lbaas openstack-config --set /etc/neutron/lbaas_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver openstack-config --set /etc/neutron/lbaas_agent.ini DEFAULT device_driver neutron_lbaas.services.loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver openstack-config --set /etc/neutron/lbaas_agent.ini haproxy user_group haproxy yum -y install openstack-neutron-fwaas openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router,firewall,lbaas openstack-config --set /etc/neutron/fwaas_driver.ini fwaas enabled True openstack-config --set /etc/neutron/fwaas_driver.ini fwaas driver neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver .... target=$PHD_ENV_nodes1 .... neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head .... target=all .... systemctl start neutron-server systemctl enable neutron-server firewall-cmd --add-port=9696/tcp firewall-cmd --add-port=9696/tcp --permanent systemctl enable openvswitch systemctl start openvswitch ovs-vsctl add-br br-int ovs-vsctl add-br br-${PHD_VAR_network_nic_external} ovs-vsctl add-port br-${PHD_VAR_network_nic_external} ${PHD_VAR_network_nic_external} # OpenvSwitch agent myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent tunnel_types vxlan openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent vxlan_udp_port 4789 openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs local_ip ${myip} openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs enable_tunneling True openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs integration_bridge br-int openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_bridge br-tun openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs bridge_mappings physnet1:br-${PHD_VAR_network_nic_external} openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs network_vlan_ranges physnet1 openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent l2_population False # Metadata agent openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_strategy keystone openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_region regionOne openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_tenant_name services openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_user neutron openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_password neutrontest openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_port 8775 openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret metatest openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_workers 4 openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_backlog 2048 # DHCP agent openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dhcp_delete_namespaces False openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_config_file /etc/neutron/dnsmasq-neutron.conf echo "dhcp-option-force=26,1400" > /etc/neutron/dnsmasq-neutron.conf chown root:neutron /etc/neutron/dnsmasq-neutron.conf chmod 644 /etc/neutron/dnsmasq-neutron.conf # L3 agent openstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver openstack-config --set /etc/neutron/l3_agent.ini DEFAULT handle_internal_only_routers True openstack-config --set /etc/neutron/l3_agent.ini DEFAULT send_arp_for_ha 3 openstack-config --set /etc/neutron/l3_agent.ini DEFAULT router_delete_namespaces False openstack-config --set /etc/neutron/l3_agent.ini DEFAULT metadata_ip ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge # Start services # Workaround for bz#1188198 mkdir -p /etc/systemd/system/neutron-server.service.d cat > /etc/systemd/system/neutron-server.service.d/restart.conf << EOF [Service] Restart=on-failure EOF systemctl daemon-reload systemctl start neutron-openvswitch-agent systemctl start neutron-dhcp-agent systemctl start neutron-l3-agent systemctl start neutron-metadata-agent systemctl start neutron-lbaas-agent systemctl enable neutron-openvswitch-agent systemctl enable neutron-dhcp-agent systemctl enable neutron-l3-agent systemctl enable neutron-metadata-agent systemctl enable neutron-ovs-cleanup systemctl enable neutron-lbaas-agent firewall-cmd --add-port=4789/udp firewall-cmd --add-port=4789/udp --permanent .... target=$PHD_ENV_nodes1 .... . /root/keystonerc_admin neutron net-create public --provider:network_type flat --provider:physical_network physnet1 --router:external neutron subnet-create --gateway ${PHD_VAR_network_neutron_externalgateway} --allocation-pool start=${PHD_VAR_network_neutron_allocpoolstart},end=${PHD_VAR_network_neutron_allocpoolend} --disable-dhcp --name public_subnet public ${PHD_VAR_network_neutron_externalnetwork} .... ================================================ FILE: keepalived/phd-setup/nova.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing Nova # - Configuring Nova # - Starting service, opening firewall ports ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_nic_external PHD_VAR_network_hosts_vip PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_hosts_memcache PHD_VAR_network_neutron_externalgateway PHD_VAR_network_neutron_externalnetwork PHD_VAR_network_neutron_allocpoolstart PHD_VAR_network_neutron_allocpoolend ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) yum install -y openstack-nova-console openstack-nova-novncproxy openstack-utils openstack-nova-api openstack-nova-conductor openstack-nova-scheduler python-cinderclient python-memcached openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address ${myip} openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen ${myip} openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_host ${myip} openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://${PHD_VAR_network_hosts_vip}:6080/vnc_auto.html openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@${PHD_VAR_network_hosts_vip}/nova openstack-config --set /etc/nova/nova.conf database max_retries -1 openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen ${myip} openstack-config --set /etc/nova/nova.conf DEFAULT metadata_host ${myip} openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen ${myip} openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen_port 8775 openstack-config --set /etc/nova/nova.conf DEFAULT glance_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver openstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron openstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL openstack-config --set /etc/nova/nova.conf conductor use_local false openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest openstack-config --set /etc/nova/nova.conf neutron url http://${PHD_VAR_network_hosts_vip}:9696/ openstack-config --set /etc/nova/nova.conf neutron admin_tenant_name services openstack-config --set /etc/nova/nova.conf neutron admin_username neutron openstack-config --set /etc/nova/nova.conf neutron admin_password neutrontest openstack-config --set /etc/nova/nova.conf neutron admin_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/nova/nova.conf neutron region_name regionOne # REQUIRED FOR A/A scheduler openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_host_subset_size 30 openstack-config --set /etc/nova/api-paste.ini filter:authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_tenant_name services openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_user compute openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_password novatest # WARNING: Only run the following if your hypervisors are VMs openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu .... target=$PHD_ENV_nodes1 .... su nova -s /bin/sh -c "nova-manage db sync" .... target=all .... systemctl start openstack-nova-consoleauth systemctl start openstack-nova-novncproxy systemctl start openstack-nova-api systemctl start openstack-nova-scheduler systemctl start openstack-nova-conductor systemctl enable openstack-nova-consoleauth systemctl enable openstack-nova-novncproxy systemctl enable openstack-nova-api systemctl enable openstack-nova-scheduler systemctl enable openstack-nova-conductor firewall-cmd --add-port=8773-8775/tcp firewall-cmd --add-port=8773-8775/tcp --permanent firewall-cmd --add-port=6080/tcp firewall-cmd --add-port=6080/tcp --permanent .... ================================================ FILE: keepalived/phd-setup/rabbitmq.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing RabbitMQ # - Configuring RabbitMQ, with a 3-node cluster # - Setting TCP keepalived kernel parameters # - Starting the RabbitMQ cluster, opening firewall ports ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum -y install rabbitmq-server .... target=$PHD_ENV_nodes1 .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" cat > /etc/rabbitmq/rabbitmq-env.conf << EOF NODE_IP_ADDRESS=${controller_ips[0]} EOF systemctl start rabbitmq-server systemctl stop rabbitmq-server scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p /var/lib/rabbitmq/.erlang.cookie ${controller_ips[1]}:/tmp scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -p /var/lib/rabbitmq/.erlang.cookie ${controller_ips[2]}:/tmp .... target=$PHD_ENV_nodes2 .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" cat > /etc/rabbitmq/rabbitmq-env.conf << EOF NODE_IP_ADDRESS=${controller_ips[1]} EOF cp /tmp/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie .... target=$PHD_ENV_nodes3 .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" cat > /etc/rabbitmq/rabbitmq-env.conf << EOF NODE_IP_ADDRESS=${controller_ips[2]} EOF cp /tmp/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie .... target=all .... IFS=', ' read -a nodes <<< "${PHD_VAR_network_hosts_rabbitmq}" cat > /etc/rabbitmq/rabbitmq.config << EOF [ {rabbit, [ {cluster_nodes, {['rabbit@${nodes[0]}', 'rabbit@${nodes[1]}', 'rabbit@${nodes[2]}'], disc}}, {cluster_partition_handling, ignore}, {default_user, <<"guest">>}, {default_pass, <<"guest">>}, {tcp_listen_options, [binary, {packet, raw}, {reuseaddr, true}, {backlog, 128}, {nodelay, true}, {exit_on_close, false}, {keepalive, true}]} ]}, {kernel, [ {inet_dist_listen_max, 44001}, {inet_dist_listen_min, 44001} ]} ]. EOF cat > /etc/sysctl.d/tcpka.conf << EOF net.ipv4.tcp_keepalive_intvl = 1 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_keepalive_time = 5 EOF sysctl -p /etc/sysctl.d/tcpka.conf firewall-cmd --add-port=5672/tcp firewall-cmd --add-port=4369/tcp firewall-cmd --add-port=5672/tcp --permanent firewall-cmd --add-port=4369/tcp --permanent firewall-cmd --add-port=44001/tcp firewall-cmd --add-port=44001/tcp --permanent systemctl enable rabbitmq-server systemctl start rabbitmq-server .... target=$PHD_ENV_nodes1 .... rabbitmqctl set_policy HA '^(?!amq\.).*' '{"ha-mode": "all"}' .... ================================================ FILE: keepalived/phd-setup/readme.txt ================================================ Assumptions: 1. We have a basic rhel7-base image. It is just a cloud image where we have manually injected a root ssh key, and allow direct root login (so the account is not locked). About the rest of the parameters: - NIC: default NAT - Disk: 60 GB, qcow2, virtio - CPU: 1 - RAM: 4 GB 2. About SSH keys, it is important that the base image includes SSH keys in /root/.ssh/authorized_keys for: - The system running phd - The hypervisor that will run the VM ================================================ FILE: keepalived/phd-setup/redis.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing Redis # - Setting up Redis, including a master and two slaves # - Setting up Sentinel for HA # - Starting services, opening firewall ports ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y redis .... target=$PHD_ENV_nodes1 .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" sed --in-place "s/bind 127.0.0.1/bind 127.0.0.1 ${controller_ips[0]}/" /etc/redis.conf .... target=$PHD_ENV_nodes2 .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" sed --in-place "s/bind 127.0.0.1/bind 127.0.0.1 ${controller_ips[1]}/" /etc/redis.conf echo slaveof ''${controller_ips[0]}'' 6379 >> /etc/redis.conf .... target=$PHD_ENV_nodes3 .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" sed --in-place "s/bind 127.0.0.1/bind 127.0.0.1 ${controller_ips[2]}/" /etc/redis.conf echo slaveof ''${controller_ips[0]}'' 6379 >> /etc/redis.conf .... target=all .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" cat > /etc/redis-sentinel.conf << EOF sentinel monitor mymaster ${controller_ips[0]} 6379 2 sentinel down-after-milliseconds mymaster 30000 sentinel failover-timeout mymaster 180000 sentinel parallel-syncs mymaster 1 min-slaves-to-write 1 min-slaves-max-lag 10 logfile /var/log/redis/sentinel.log EOF firewall-cmd --add-port=6379/tcp firewall-cmd --add-port=6379/tcp --permanent firewall-cmd --add-port=26379/tcp firewall-cmd --add-port=26379/tcp --permanent systemctl enable redis systemctl start redis systemctl enable redis-sentinel systemctl start redis-sentinel .... ================================================ FILE: keepalived/phd-setup/sahara.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing Sahara # - Configuring Sahara # - Starting services, opening firewall ports ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_nic_external PHD_VAR_network_hosts_vip PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_hosts_memcache PHD_VAR_network_neutron_externalgateway PHD_VAR_network_neutron_externalnetwork PHD_VAR_network_neutron_allocpoolstart PHD_VAR_network_neutron_allocpoolend ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) yum install -y openstack-sahara-api openstack-sahara-engine openstack-sahara-common openstack-sahara python-saharaclient openstack-config --set /etc/sahara/sahara.conf DEFAULT host ${myip} openstack-config --set /etc/sahara/sahara.conf DEFAULT use_floating_ips True openstack-config --set /etc/sahara/sahara.conf DEFAULT use_neutron True openstack-config --set /etc/sahara/sahara.conf DEFAULT rpc_backend rabbit openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_port 5672 openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_use_ssl False openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_userid guest openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_password guest openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_login_method AMQPLAIN openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/sahara/sahara.conf DEFAULT notification_topics notifications openstack-config --set /etc/sahara/sahara.conf database connection mysql://sahara:saharatest@${PHD_VAR_network_hosts_vip}/sahara openstack-config --set /etc/sahara/sahara.conf keystone_authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:5000/v2.0 openstack-config --set /etc/sahara/sahara.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_user sahara openstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_password saharatest openstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/sahara/sahara.conf DEFAULT log_file /var/log/sahara/sahara.log .... target=$PHD_ENV_nodes1 .... sahara-db-manage --config-file /etc/sahara/sahara.conf upgrade head .... target=all .... firewall-cmd --add-port=8386/tcp firewall-cmd --add-port=8386/tcp --permanent systemctl enable openstack-sahara-api systemctl enable openstack-sahara-engine systemctl start openstack-sahara-api systemctl start openstack-sahara-engine .... ================================================ FILE: keepalived/phd-setup/serverprep.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Tweaking the IP stack to allow nonlocal binding and adjusting keepalive timings # - Configuring haproxy # - Adding the virtual IPs to the cluster # - Putting haproxy under the cluster's control ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_ips_controllers PHD_VAR_network_ips_computeinternal PHD_VAR_network_hosts_controllers PHD_VAR_network_hosts_compute PHD_VAR_network_ips_vip PHD_VAR_network_hosts_vip ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... IFS=', ' read -a controller_names <<< "${PHD_VAR_network_hosts_controllers}" IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" IFS=', ' read -a compute_names <<< "${PHD_VAR_network_hosts_compute}" IFS=', ' read -a compute_ips <<< "${PHD_VAR_network_ips_computeinternal}" addhosts="" for item in "${controller_names[@]}" do shortname=$(echo ${controller_names[item]} | awk -F. '{print $1}') addhosts=$(printf "${addhosts}\n${controller_ips[item]} ${controller_names[item]} ${shortname}") done for item in "${controller_names[@]}" do shortname=$(echo ${compute_names[item]} | awk -F. '{print $1}') addhosts=$(printf "${addhosts}\n${compute_ips[item]} ${compute_names[item]} ${shortname}") done shortname=$(echo ${PHD_VAR_network_hosts_vip} | awk -F. '{print $1}') addhosts=$(printf "${addhosts}\n${PHD_VAR_network_ips_vip} ${PHD_VAR_network_hosts_vip} ${shortname}") echo "$addhosts" >> /etc/hosts .... ================================================ FILE: keepalived/phd-setup/swift.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Tweaking the IP stack to allow nonlocal binding and adjusting keepalive timings # - Configuring haproxy # - Adding the virtual IPs to the cluster # - Putting haproxy under the cluster's control ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_hosts_vip PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_hosts_memcache ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) yum install -y openstack-swift-object openstack-swift-container openstack-swift-account openstack-swift-proxy openstack-utils rsync xfsprogs mkfs.xfs /dev/vdb mkdir -p /srv/node/vdb echo "/dev/vdb /srv/node/vdb xfs defaults 1 2" >> /etc/fstab mount -a chown -R swift:swift /srv/node restorecon -R /srv/node openstack-config --set /etc/swift/object-server.conf DEFAULT bind_ip ${myip} openstack-config --set /etc/swift/object-server.conf DEFAULT devices /srv/node openstack-config --set /etc/swift/account-server.conf DEFAULT bind_ip ${myip} openstack-config --set /etc/swift/account-server.conf DEFAULT devices /srv/node openstack-config --set /etc/swift/container-server.conf DEFAULT bind_ip ${myip} openstack-config --set /etc/swift/container-server.conf DEFAULT devices /srv/node chown -R root:swift /etc/swift systemctl start openstack-swift-account systemctl start openstack-swift-container systemctl start openstack-swift-object systemctl enable openstack-swift-account systemctl enable openstack-swift-container systemctl enable openstack-swift-object firewall-cmd --add-port=6200/tcp firewall-cmd --add-port=6200/tcp --permanent firewall-cmd --add-port=6201/tcp firewall-cmd --add-port=6201/tcp --permanent firewall-cmd --add-port=6202/tcp firewall-cmd --add-port=6202/tcp --permanent openstack-config --set /etc/swift/proxy-server.conf filter:authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_tenant_name services openstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_user swift openstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_password swifttest openstack-config --set /etc/swift/proxy-server.conf filter:cache memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/swift/proxy-server.conf DEFAULT bind_ip ${myip} openstack-config --set /etc/swift/object-expirer.conf filter:cache memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/swift/object-expirer.conf object-expirer concurrency 100 .... target=$PHD_ENV_nodes1 .... IFS=', ' read -a controller_ips <<< "${PHD_VAR_network_ips_controllers}" openstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_suffix $(openssl rand -hex 10) cat >> /etc/swift/swift.conf << EOF [filter:ceilometer] use = egg:ceilometer#swift [pipeline:main] pipeline = healthcheck cache authtoken keystoneauth proxy-server ceilometer EOF swift-ring-builder /etc/swift/object.builder create 16 3 24 swift-ring-builder /etc/swift/container.builder create 16 3 24 swift-ring-builder /etc/swift/account.builder create 16 3 24 swift-ring-builder /etc/swift/account.builder add z1-${controller_ips[0]}:6202/vdb 10 swift-ring-builder /etc/swift/container.builder add z1-${controller_ips[0]}:6201/vdb 10 swift-ring-builder /etc/swift/object.builder add z1-${controller_ips[0]}:6200/vdb 10 swift-ring-builder /etc/swift/account.builder add z2-${controller_ips[1]}:6202/vdb 10 swift-ring-builder /etc/swift/container.builder add z2-${controller_ips[1]}:6201/vdb 10 swift-ring-builder /etc/swift/object.builder add z2-${controller_ips[1]}:6200/vdb 10 swift-ring-builder /etc/swift/account.builder add z3-${controller_ips[2]}:6202/vdb 10 swift-ring-builder /etc/swift/container.builder add z3-${controller_ips[2]}:6201/vdb 10 swift-ring-builder /etc/swift/object.builder add z3-${controller_ips[2]}:6200/vdb 10 swift-ring-builder /etc/swift/account.builder rebalance swift-ring-builder /etc/swift/container.builder rebalance swift-ring-builder /etc/swift/object.builder rebalance cd /etc/swift tar cvfz /tmp/swift_configs.tgz swift.conf *.builder *.gz scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /tmp/swift_configs.tgz ${controller_ips[1]}:/tmp scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null /tmp/swift_configs.tgz ${controller_ips[2]}:/tmp chown -R root:swift /etc/swift .... target=$PHD_ENV_nodes2 .... cd /etc/swift tar xvfz /tmp/swift_configs.tgz chown -R root:swift /etc/swift restorecon -R /etc/swift .... target=$PHD_ENV_nodes3 .... cd /etc/swift tar xvfz /tmp/swift_configs.tgz chown -R root:swift /etc/swift restorecon -R /etc/swift .... target=all .... systemctl start openstack-swift-proxy systemctl enable openstack-swift-proxy systemctl start openstack-swift-object-expirer systemctl enable openstack-swift-object-expirer firewall-cmd --add-port=8080/tcp firewall-cmd --add-port=8080/tcp --permanent .... ================================================ FILE: keepalived/phd-setup/test.sh ================================================ #!/bin/bash set -e declare -A nodeMap declare -A variables declare -A cluster nodeMap["hypervisors"]="oslab1 oslab2 oslab3" nodeMap["controllers"]="controller1 controller2 controller3" nodeMap["compute"]="compute1 compute2" nodeMap["serverprep"]="controller1 controller2 controller3 compute1 compute2" variables["nodes"]="" variables["components"]="hypervisors serverprep lb galera rabbitmq memcached redis mongodb keepalived keystone glance cinder swift neutron nova ceilometer heat horizon sahara trove compute" #variables["components"]="compute" variables["network_domain"]="example.com" variables["config"]="ha-collapsed" function create_phd_definition() { scenario=$1 definition=$2 rm -f ${definition} nodes=${variables["nodes"]} if [ "x$nodes" = x ]; then nodes=${nodeMap[$scenario]} fi if [ "x$nodes" = "x" ]; then for n in `seq 1 3`; do nodes="$nodes controller${n}" done fi nodelist="nodes=" for node in $nodes; do nodelist="${nodelist}${node}.${variables["network_domain"]} " done echo "$nodelist" >> ${definition} cat ${definition} } function run_phd() { phd_exec -s ./${1}.scenario -d ./phd.${1}.conf -V ./${variables["config"]}.variables } scenarios=${variables["components"]} for scenario in $scenarios; do create_phd_definition ${scenario} ./phd.${scenario}.conf echo "$(date) :: Beginning scenario $scenario" run_phd ${scenario} done ================================================ FILE: keepalived/phd-setup/trove.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Installing Trove # - Configuring Trove # - Creating a VM image for Trove usage # - Starting services, opening firewall ports ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_nic_internal PHD_VAR_network_nic_external PHD_VAR_network_hosts_vip PHD_VAR_network_ips_controllers PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_hosts_memcache PHD_VAR_network_neutron_externalgateway PHD_VAR_network_neutron_externalnetwork PHD_VAR_network_neutron_allocpoolstart PHD_VAR_network_neutron_allocpoolend PHD_VAR_network_ssh_pubkey ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1) yum install -y openstack-trove python-troveclient openstack-config --set /etc/trove/trove.conf DEFAULT bind_host ${myip} openstack-config --set /etc/trove/trove.conf DEFAULT log_dir /var/log/trove openstack-config --set /etc/trove/trove.conf DEFAULT trove_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/trove/trove.conf DEFAULT os_region_name regionOne openstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_password guest openstack-config --set /etc/trove/trove.conf DEFAULT rpc_backend rabbit openstack-config --set /etc/trove/trove.conf database connection mysql://trove:trovetest@${PHD_VAR_network_hosts_vip}/trove openstack-config --set /etc/trove/trove.conf database max_retries -1 openstack-config --set /etc/trove/trove.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/trove/trove.conf keystone_authtoken admin_user trove openstack-config --set /etc/trove/trove.conf keystone_authtoken admin_password trovetest openstack-config --set /etc/trove/trove.conf keystone_authtoken service_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/trove/trove.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/trove/trove.conf keystone_authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/trove/trove.conf DEFAULT control_exchange trove openstack-config --set /etc/trove/trove-conductor.conf DEFAULT trove_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/trove/trove-conductor.conf DEFAULT os_region_name regionOne openstack-config --set /etc/trove/trove-conductor.conf DEFAULT log_file trove-conductor.log openstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_password guest openstack-config --set /etc/trove/trove-conductor.conf DEFAULT rpc_backend rabbit openstack-config --set /etc/trove/trove-conductor.conf database connection mysql://trove:trovetest@${PHD_VAR_network_hosts_vip}/trove openstack-config --set /etc/trove/trove-conductor.conf database max_retries -1 openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken admin_user trove openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken admin_password trovetest openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken service_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/trove/trove-conductor.conf DEFAULT control_exchange trove openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT trove_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT os_region_name regionOne openstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_password guest openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT rpc_backend rabbit openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_user trove openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_pass trovetest openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_tenant_name ${SERVICES_TENANT_ID} openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT log_file trove-taskmanager.log openstack-config --set /etc/trove/trove-taskmanager.conf database connection mysql://trove:trovetest@${PHD_VAR_network_hosts_vip}/trove openstack-config --set /etc/trove/trove-taskmanager.conf database max_retries -1 openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken admin_user trove openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken admin_password trovetest openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken service_host ${PHD_VAR_network_hosts_vip} openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT cloudinit_loaction /etc/trove/cloudinit openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT network_driver trove.network.neutron.NeutronDriver openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT control_exchange trove # The following is a workaround for https://bugs.launchpad.net/trove/+bug/1402055 openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT exists_notification_transformer openstack-config --set /etc/trove/trove.conf DEFAULT default_datastore mysql openstack-config --set /etc/trove/trove.conf DEFAULT add_addresses True openstack-config --set /etc/trove/trove.conf DEFAULT network_label_regex ^private$ cp /usr/share/trove/trove-dist-paste.ini /etc/trove/api-paste.ini openstack-config --set /etc/trove/api-paste.ini filter:authtoken auth_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/trove/api-paste.ini filter:authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/ openstack-config --set /etc/trove/api-paste.ini filter:authtoken admin_password trovetest openstack-config --set /etc/trove/api-paste.ini filter:authtoken admin_user trove openstack-config --set /etc/trove/api-paste.ini filter:authtoken admin_tenant_name services openstack-config --set /etc/trove/trove.conf DEFAULT api_paste_config /etc/trove/api-paste.ini .... target=$PHD_ENV_nodes1 .... su trove -s /bin/sh -c "trove-manage db_sync" trove-manage datastore_update mysql '' # Create image using CentOS7. # This is not ideal, check https://www.rdoproject.org/forum/discussion/1010/creation-of-trove-compatible-images-for-rdo/p1 . /root/keystonerc_admin cd /tmp wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 glance image-create --name centos7 --disk-format qcow2 --container-format bare --is-public True --owner trove --file CentOS-7-x86_64-GenericCloud.qcow2 IMAGE_ID=$(openstack image show centos7 --format value --column id) trove-manage --config-file=/etc/trove/trove.conf datastore_version_update mysql mysql-5.5 mysql ${IMAGE_ID} mysql55 1 trove-manage db_load_datastore_config_parameters mysql "mysql-5.5" /usr/lib/python2.7/site-packages/trove/templates/mysql/validation-rules.json .... target=all .... mkdir /etc/trove/cloudinit cat > /etc/trove/cloudinit/mysql.cloudinit << EOF #!/bin/bash sed -i'.orig' -e's/without-password/yes/' /etc/ssh/sshd_config echo "test" | passwd --stdin centos echo "redhat" | passwd --stdin root service sshd restart yum -y install wget epel-release yum -y install http://rdoproject.org/repos/openstack-kilo/rdo-testing-kilo.rpm yum -y install openstack-trove-guestagent mariadb-server openstack-utils python-oslo-messaging python-osprofiler python-oslo-concurrency systemctl enable mariadb systemctl start mariadb echo "trove ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers echo "Defaults:trove !requiretty" >> /etc/sudoers openstack-config --set /etc/trove/trove-guestagent.conf DEFAULT rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/trove/trove-guestagent.conf DEFAULT rabbit_password guest openstack-config --set /etc/trove/trove-guestagent.conf DEFAULT nova_proxy_admin_user trove openstack-config --set /etc/trove/trove-guestagent.conf DEFAULT nova_proxy_admin_pass trovetest openstack-config --set /etc/trove/trove-guestagent.conf DEFAULT nova_proxy_admin_tenant_name services openstack-config --set /etc/trove/trove-guestagent.conf DEFAULT trove_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0 openstack-config --set /etc/trove/trove-guestagent.conf DEFAULT control_exchange trove openstack-config --set /etc/trove/trove-guestagent.conf DEFAULT log_dir /var/log/trove openstack-config --set /etc/trove/trove-guestagent.conf DEFAULT log_file trove-guestagent.log openstack-config --set /etc/trove/trove-guestagent.conf DEFAULT datastore_manager mysql echo "${PHD_VAR_network_ssh_pubkey}" >> /root/.ssh/authorized_keys echo "${PHD_VAR_network_ssh_pubkey}" >> /home/centos/.ssh/authorized_keys systemctl stop openstack-trove-guestagent systemctl enable openstack-trove-guestagent systemctl start openstack-trove-guestagent EOF systemctl enable openstack-trove-api systemctl enable openstack-trove-taskmanager systemctl enable openstack-trove-conductor systemctl start openstack-trove-api systemctl start openstack-trove-taskmanager systemctl start openstack-trove-conductor firewall-cmd --add-port=8779/tcp firewall-cmd --add-port=8779/tcp --permanent .... ================================================ FILE: keepalived/rabbitmq-config.md ================================================ Introduction ------------ RabbitMQ can create a native cluster, by grouping several nodes and replicating all message queues. Clustered RabbitMQ environments tolerate the failure of individual nodes. Nodes can be started and stopped at will. ![](Rabbitmq_clustering.jpg "RabbitMQ clustering") **Note:** Access to RabbitMQ is not handled by HAproxy, as there are known issues integrating RabbitMQ with HAProxy. Instead consumers must be supplied with the full list of hosts running RabbitMQ with `rabbit_hosts` and `rabbit_ha_queues` options, and connect directly to one or more RabbitMQ servers. Reconnections in case of a node failure are handled automatically. The following commands will be executed on all controller nodes, unless stated otherwise. You can find a phd scenario file [here](phd-setup/rabbitmq.scenario). Install package --------------- yum -y install rabbitmq-server Create erlang cookie and distribute ----------------------------------- **On node 1:** cat > /etc/rabbitmq/rabbitmq-env.conf << EOF NODE_IP_ADDRESS=192.168.1.221 EOF systemctl start rabbitmq-server systemctl stop rabbitmq-server scp -p /var/lib/rabbitmq/.erlang.cookie hacontroller2:/var/lib/rabbitmq scp -p /var/lib/rabbitmq/.erlang.cookie hacontroller3:/var/lib/rabbitmq Set permissions for erlang cookie --------------------------------- **On node 2 and node 3:** chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie cat > /etc/rabbitmq/rabbitmq-env.conf << EOF NODE_IP_ADDRESS=192.168.1.22X EOF ### Create rabbitmq configuration **On all nodes:** cat > /etc/rabbitmq/rabbitmq.config << EOF [ {rabbit, [ {cluster_nodes, {['rabbit@hacontroller1', 'rabbit@hacontroller2', 'rabbit@hacontroller3'], disc}}, {cluster_partition_handling, ignore}, {default_user, <<"guest">>}, {default_pass, <<"guest">>}, {tcp_listen_options, [binary, {packet, raw}, {reuseaddr, true}, {backlog, 128}, {nodelay, true}, {exit_on_close, false}, {keepalive, true}]} ]}, {kernel, [ {inet_dist_listen_max, 44001}, {inet_dist_listen_min, 44001} ]} ]. EOF Set kernel TCP keepalive parameters ----------------------------------- cat > /etc/sysctl.d/tcpka.conf << EOF net.ipv4.tcp_keepalive_intvl = 1 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_keepalive_time = 5 EOF sysctl -p /etc/sysctl.d/tcpka.conf Start services and open firewall ports -------------------------------------- firewall-cmd --add-port=5672/tcp firewall-cmd --add-port=4369/tcp firewall-cmd --add-port=5672/tcp --permanent firewall-cmd --add-port=4369/tcp --permanent firewall-cmd --add-port=44001/tcp firewall-cmd --add-port=44001/tcp --permanent systemctl enable rabbitmq-server systemctl start rabbitmq-server And check everything is going fine by running: 
 # rabbitmqctl cluster_status

Cluster status of node rabbit@hacontroller1 ...
[{nodes,[{disc,[rabbit@hacontroller1,rabbit@hacontroller2,
rabbit@hacontroller3]}]},
{running_nodes,[rabbit@hacontroller3,rabbit@hacontroller2,
rabbit@hacontroller1]},
{cluster_name,<<"rabbit@hacontroller1.example.com">\>}, {partitions,[]}]

...done.
Set HA mode for all queues -------------------------- **On node 1:** rabbitmqctl set_policy HA '^(?!amq\.).*' '{"ha-mode": "all"}' ================================================ FILE: keepalived/rabbitmq-restart.md ================================================ Introduction ------------ In general, RabbitMQ does a good job at restarting the cluster when all nodes are stared at the same time. However, we may find times were this is not the case, and we will have to restart the cluster manually. According to [](http://previous.rabbitmq.com/v3_3_x/clustering.html) *"the last node to go down must be the first node to be brought online. If this doesn't happen, the nodes will wait 30 seconds for the last disc node to come back online, and fail afterwards."*. Thus, it is necessary to find the last node going down and start it. Depending on how the nodes were started, you may see some nodes running and some stopped. Checking RabbitMQ cluster status -------------------------------- Run the following command to verify the current RabbitMQ cluster status: rabbitmqctl cluster_status Cluster status of node rabbit@hacontroller3 ... [{nodes,[{disc,[rabbit@hacontroller1,rabbit@hacontroller2, rabbit@hacontroller3]}]}, {running_nodes,[rabbit@hacontroller2,rabbit@hacontroller1, rabbit@hacontroller3]}, {cluster_name,<<"rabbit@hacontroller1.example.com">>}, {partitions,[]}] ### Some nodes are running If some nodes are running, the most probable reason is that the failed nodes timed out before finding the last node to come back online. In this case, start rabbitmq-server on the failed nodes. [root@hacontroller1 ~]# systemctl start rabbitmq-server ### None of the nodes are running In this case, we need to find which node should be started first. Select a node as first node, start rabbitmq-server, then start it on the remaining nodes. [root@hacontroller2 ~]# systemctl start rabbitmq-server [root@hacontroller3 ~]# systemctl start rabbitmq-server [root@hacontroller1 ~]# systemctl start rabbitmq-server Check that all nodes are running rabbitmq-server. If not, stop any surviving rabbitmq-server and select a different node as first node. ================================================ FILE: keepalived/redis-config.md ================================================ Introduction ------------ Redis in a key-value cache and store, used by Ceilometer with the [tooz](https://github.com/openstack/tooz) library. It uses an master-slave architecture for high availability, where a single node is used for writes and a number of slaves replicate data from it. Using [Sentinel](http://redis.io/topics/sentinel), it is possible monitor node health and fail over automatically to another node if needed. By configuring Ceilometer to access the Sentinel processes, high availability from the consumer point of view is transparent. The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/redis.scenario). Install redis ------------- yum install -y redis Configure bind IP, set master and slaves ---------------------------------------- On all nodes: sed --in-place 's/bind 127.0.0.1/bind 127.0.0.1 192.168.1.X/' /etc/redis.conf On node2 and 3: echo slaveof '''' 6379 >> /etc/redis.conf Configure Sentinel, used for master failover -------------------------------------------- **On all nodes:** cat > /etc/redis-sentinel.conf << EOF sentinel monitor mymaster 6379 2 sentinel down-after-milliseconds mymaster 30000 sentinel failover-timeout mymaster 180000 sentinel parallel-syncs mymaster 1 min-slaves-to-write 1 min-slaves-max-lag 10 logfile /var/log/redis/sentinel.log EOF Configure firewall, start services ---------------------------------- firewall-cmd --add-port=6379/tcp firewall-cmd --add-port=6379/tcp --permanent firewall-cmd --add-port=26379/tcp firewall-cmd --add-port=26379/tcp --permanent systemctl enable redis systemctl start redis systemctl enable redis-sentinel systemctl start redis-sentinel ================================================ FILE: keepalived/sahara-config.md ================================================ Introduction ------------ **Important:** this configuration assumes that the controller nodes can access the floating IP network (10.10.10.0/24 in the example configuration). This was not the case on [the controller node configuration](controller-node.md), because the NIC used for the provider network did not have an IP address. You can accomplish this by setting up routes in the default gateway (192.168.1.1), or creating a separate route on the controller nodes. The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/sahara.scenario). Install software ---------------- yum install -y openstack-sahara-api openstack-sahara-engine openstack-sahara-common openstack-sahara python-saharaclient Configure Sahara ---------------- openstack-config --set /etc/sahara/sahara.conf DEFAULT host 192.168.1.22X openstack-config --set /etc/sahara/sahara.conf DEFAULT use_floating_ips True openstack-config --set /etc/sahara/sahara.conf DEFAULT use_neutron True openstack-config --set /etc/sahara/sahara.conf DEFAULT rpc_backend rabbit openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_port 5672 openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_use_ssl False openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_userid guest openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_password guest openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_login_method AMQPLAIN openstack-config --set /etc/sahara/sahara.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/sahara/sahara.conf DEFAULT notification_topics notifications openstack-config --set /etc/sahara/sahara.conf database connection mysql://sahara:saharatest@controller-vip.example.com/sahara openstack-config --set /etc/sahara/sahara.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/sahara/sahara.conf keystone_authtoken auth_plugin password openstack-config --set /etc/sahara/sahara.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/sahara/sahara.conf keystone_authtoken username sahara openstack-config --set /etc/sahara/sahara.conf keystone_authtoken password saharatest openstack-config --set /etc/sahara/sahara.conf keystone_authtoken project_name services openstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_user sahara openstack-config --set /etc/sahara/sahara.conf keystone_authtoken admin_password saharatest openstack-config --set /etc/sahara/sahara.conf DEFAULT log_file /var/log/sahara/sahara.log Manage DB --------- On node 1: sahara-db-manage --config-file /etc/sahara/sahara.conf upgrade head Start services, open firewall ports ----------------------------------- firewall-cmd --add-port=8386/tcp firewall-cmd --add-port=8386/tcp --permanent systemctl enable openstack-sahara-api systemctl enable openstack-sahara-engine systemctl start openstack-sahara-api systemctl start openstack-sahara-engine Testing ------- On node 1, run the following commands to test the Sahara API: . /root/keystonerc_admin sahara plugin-list Further Sahara testing requires creating a specific virtual machine image, which is outside the scope of this document. You can find instructions on [the Sahara wiki](http://docs.openstack.org/developer/sahara/devref/quickstart.html#upload-an-image-to-the-image-service). ================================================ FILE: keepalived/swift-config.md ================================================ Introduction ------------ We need to have an additional disk, `/dev/vdb` in our test available for Swift usage. The following commands will be executed on all controller nodes, unless otherwise stated. You can find a phd scenario file [here](phd-setup/swift.scenario). Install software ---------------- yum install -y openstack-swift-object openstack-swift-container openstack-swift-account openstack-swift-proxy openstack-utils rsync xfsprogs Create XFS file system for additional disk, and mount it -------------------------------------------------------- mkfs.xfs /dev/vdb mkdir -p /srv/node/vdb echo "/dev/vdb /srv/node/vdb xfs defaults 1 2" >> /etc/fstab mount -a chown -R swift:swift /srv/node restorecon -R /srv/node Configure account, container and object services ------------------------------------------------ openstack-config --set /etc/swift/object-server.conf DEFAULT bind_ip 192.168.1.22X openstack-config --set /etc/swift/object-server.conf DEFAULT devices /srv/node openstack-config --set /etc/swift/account-server.conf DEFAULT bind_ip 192.168.1.22X openstack-config --set /etc/swift/account-server.conf DEFAULT devices /srv/node openstack-config --set /etc/swift/container-server.conf DEFAULT bind_ip 192.168.1.22X openstack-config --set /etc/swift/container-server.conf DEFAULT devices /srv/node chown -R root:swift /etc/swift Start account, container and object services, open firewall ports ----------------------------------------------------------------- systemctl start openstack-swift-account systemctl start openstack-swift-container systemctl start openstack-swift-object systemctl enable openstack-swift-account systemctl enable openstack-swift-container systemctl enable openstack-swift-object firewall-cmd --add-port=6200/tcp firewall-cmd --add-port=6200/tcp --permanent firewall-cmd --add-port=6201/tcp firewall-cmd --add-port=6201/tcp --permanent firewall-cmd --add-port=6202/tcp firewall-cmd --add-port=6202/tcp --permanent Configure swift proxy and object expirer ---------------------------------------- openstack-config --set /etc/swift/proxy-server.conf filter:authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/swift/proxy-server.conf filter:authtoken auth_plugin password openstack-config --set /etc/swift/proxy-server.conf filter:authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/swift/proxy-server.conf filter:authtoken username swift openstack-config --set /etc/swift/proxy-server.conf filter:authtoken password swifttest openstack-config --set /etc/swift/proxy-server.conf filter:authtoken project_name services openstack-config --set /etc/swift/proxy-server.conf filter:cache memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211 openstack-config --set /etc/swift/proxy-server.conf DEFAULT bind_ip 192.168.1.22X openstack-config --set /etc/swift/object-expirer.conf filter:cache memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211 openstack-config --set /etc/swift/object-expirer.conf object-expirer concurrency 100 Set Ceilometer hook ------------------- On node 1: cat >> /etc/swift/swift.conf << EOF [filter:ceilometer] use = egg:ceilometer#swift [pipeline:main] pipeline = healthcheck cache authtoken keystoneauth proxy-server ceilometer EOF Configure hash path suffix -------------------------- On node 1: openstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_suffix $(openssl rand -hex 10) Create rings ------------ On node 1: swift-ring-builder /etc/swift/object.builder create 16 3 24 swift-ring-builder /etc/swift/container.builder create 16 3 24 swift-ring-builder /etc/swift/account.builder create 16 3 24 swift-ring-builder /etc/swift/account.builder add z1-192.168.1.221:6202/vdb 10 swift-ring-builder /etc/swift/container.builder add z1-192.168.1.221:6201/vdb 10 swift-ring-builder /etc/swift/object.builder add z1-192.168.1.221:6200/vdb 10 swift-ring-builder /etc/swift/account.builder add z2-192.168.1.222:6202/vdb 10 swift-ring-builder /etc/swift/container.builder add z2-192.168.1.222:6201/vdb 10 swift-ring-builder /etc/swift/object.builder add z2-192.168.1.222:6200/vdb 10 swift-ring-builder /etc/swift/account.builder add z3-192.168.1.223:6202/vdb 10 swift-ring-builder /etc/swift/container.builder add z3-192.168.1.223:6201/vdb 10 swift-ring-builder /etc/swift/object.builder add z3-192.168.1.223:6200/vdb 10 swift-ring-builder /etc/swift/account.builder rebalance swift-ring-builder /etc/swift/container.builder rebalance swift-ring-builder /etc/swift/object.builder rebalance cd /etc/swift tar cvfz /tmp/swift_configs.tgz swift.conf *.builder *.gz scp /tmp/swift_configs.tgz hacontroller2:/tmp scp /tmp/swift_configs.tgz hacontroller3:/tmp chown -R root:swift /etc/swift Import swift configuration from node 1 -------------------------------------- On nodes 2 and 3: cd /etc/swift tar xvfz /tmp/swift_configs.tgz chown -R root:swift /etc/swift restorecon -R /etc/swift Start services, open firewall ports ----------------------------------- On all nodes: systemctl start openstack-swift-proxy systemctl enable openstack-swift-proxy systemctl start openstack-swift-object-expirer systemctl enable openstack-swift-object-expirer firewall-cmd --add-port=8080/tcp firewall-cmd --add-port=8080/tcp --permanent Test ---- On any node: . /root/keystonerc_admin swift list swift upload test /tmp/cirros-0.3.3-x86_64-disk.img swift list swift list test swift download test tmp/cirros-0.3.3-x86_64-disk.img ================================================ FILE: keepalived/trove-config.md ================================================ Introduction ------------ **WARNING:** this configuration uses the same RabbitMQ instance as the rest of the OpenStack services. This is not recommended for a production environment, because it creates a security risk (see [this e-mail thread](http://lists.openstack.org/pipermail/openstack-dev/2015-April/061759.html) for details). Because of that, this configuration assumes that virtual machine instances can access the controller node management network (192.168.1.0/24 in the example configuration). This requires setting up the required routes and firewall rules to ensure this is possible. Those firewall rules should allow access from the floating IP network (10.10.10.0/24) to the controller node management network. **Important:** On a production environment, a separate RabbitMQ instance should be deployed for Trove, so you should adjust the rabbit_* configuration lines accordingly. The following commands will be executed on all controller nodes, unless otherwise stated. Install software ---------------- yum install -y openstack-trove python-troveclient Configure Trove --------------- openstack-config --set /etc/trove/trove.conf DEFAULT bind_host 192.168.1.22X openstack-config --set /etc/trove/trove.conf DEFAULT log_dir /var/log/trove openstack-config --set /etc/trove/trove.conf DEFAULT trove_auth_url http://controller-vip.example.com:35357/v2.0 openstack-config --set /etc/trove/trove.conf DEFAULT os_region_name regionOne openstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/trove/trove.conf oslo_messaging_rabbit rabbit_password guest openstack-config --set /etc/trove/trove.conf DEFAULT rpc_backend rabbit openstack-config --set /etc/trove/trove.conf database connection mysql://trove:trovetest@controller-vip.example.com/trove openstack-config --set /etc/trove/trove.conf database max_retries -1 openstack-config --set /etc/trove/trove.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/trove/trove.conf keystone_authtoken auth_plugin password openstack-config --set /etc/trove/trove.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/trove/trove.conf keystone_authtoken username trove openstack-config --set /etc/trove/trove.conf keystone_authtoken password trovetest openstack-config --set /etc/trove/trove.conf keystone_authtoken project_name services openstack-config --set /etc/trove/trove-conductor.conf DEFAULT trove_auth_url http://controller-vip.example.com:35357/v2.0 openstack-config --set /etc/trove/trove-conductor.conf DEFAULT os_region_name regionOne openstack-config --set /etc/trove/trove-conductor.conf DEFAULT log_file trove-conductor.log openstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/trove/trove-conductor.conf oslo_messaging_rabbit rabbit_password guest openstack-config --set /etc/trove/trove-conductor.conf DEFAULT rpc_backend rabbit openstack-config --set /etc/trove/trove-conductor.conf database connection mysql://trove:trovetest@controller-vip.example.com/trove openstack-config --set /etc/trove/trove-conductor.conf database max_retries -1 openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken auth_plugin password openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken username trove openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken password trovetest openstack-config --set /etc/trove/trove-conductor.conf keystone_authtoken project_name services openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT trove_auth_url http://controller-vip.example.com:35357/v2.0 openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT os_region_name regionOne openstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3 openstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/trove/trove-taskmanager.conf oslo_messaging_rabbit rabbit_password guest openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT rpc_backend rabbit openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_user trove openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_pass trovetest openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT nova_proxy_admin_tenant_name ${SERVICES_TENANT_ID} openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT log_file trove-taskmanager.log openstack-config --set /etc/trove/trove-taskmanager.conf database connection mysql://trove:trovetest@controller-vip.example.com/trove openstack-config --set /etc/trove/trove-taskmanager.conf database max_retries -1 openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/ openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken auth_plugin password openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken username trove openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken password trovetest openstack-config --set /etc/trove/trove-taskmanager.conf keystone_authtoken project_name services openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT cloudinit_loaction /etc/trove/cloudinit openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT network_driver trove.network.neutron.NeutronDriver openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT control_exchange trove # The following is a workaround for https://bugs.launchpad.net/trove/+bug/1402055 openstack-config --set /etc/trove/trove-taskmanager.conf DEFAULT exists_notification_transformer openstack-config --set /etc/trove/trove.conf DEFAULT default_datastore mysql openstack-config --set /etc/trove/trove.conf DEFAULT add_addresses True openstack-config --set /etc/trove/trove.conf DEFAULT network_label_regex ^private$ cp /usr/share/trove/trove-dist-paste.ini /etc/trove/api-paste.ini openstack-config --set /etc/trove/api-paste.ini filter:authtoken auth_uri http://controller-vip.example.com:35357/ openstack-config --set /etc/trove/api-paste.ini filter:authtoken auth_url http://controller-vip.example.com:35357/ openstack-config --set /etc/trove/api-paste.ini filter:authtoken auth_plugin password openstack-config --set /etc/trove/api-paste.ini filter:authtoken password trovetest openstack-config --set /etc/trove/api-paste.ini filter:authtoken username trove openstack-config --set /etc/trove/api-paste.ini filter:authtoken project_name services openstack-config --set /etc/trove/trove.conf DEFAULT api_paste_config /etc/trove/api-paste.ini Manage DB --------- On node 1: su trove -s /bin/sh -c "trove-manage db_sync" trove-manage datastore_update mysql '' Create and upload image ----------------------- Trove instances will require a specially crafted virtual machine image with the required database software. The following instructions will create a simple image based on CentOS 7 with MariaDB 5.5. [This article](https://www.rdoproject.org/forum/discussion/1010/creation-of-trove-compatible-images-for-rdo/p1) provides detailed instructions on how to create a Trove-compatible image using trove-image-elements. Please note that only limited testing has been performed with the image. These instructions can be updated over time, please feel free to provide feedback if you have a chance to test them. On node 1: yum -y install git git clone https://github.com/vkmc/trove-image-elements cd trove-image-elements ./create_trove_image.sh -d centos -s mysql . /root/keystonerc_admin glance image-create --name centos-mysql-guest-image --disk-format qcow2 --container-format bare --visibility public --file centos-mysql-guest-image.qcow2 glance image-update --owner trove Take note of the image id, then update the Trove database with a reference to the newly uploaded image: trove-manage --config-file=/etc/trove/trove.conf datastore_version_update mysql mysql-5.5 mysql 9b412ead-5a5c-40ba-ac8f-98f70cc4f682 mysql55 1 trove-manage db_load_datastore_config_parameters mysql "mysql-5.5" /usr/lib/python2.7/site-packages/trove/templates/mysql/validation-rules.json trove-manage datastore_update mysql mysql-5.5 Start services, open firewall ports ----------------------------------- On all nodes: systemctl enable openstack-trove-api systemctl enable openstack-trove-taskmanager systemctl enable openstack-trove-conductor systemctl start openstack-trove-api systemctl start openstack-trove-taskmanager systemctl start openstack-trove-conductor firewall-cmd --add-port=8779/tcp firewall-cmd --add-port=8779/tcp --permanent ================================================ FILE: make-vm ================================================ #!/bin/bash onlyone=$1 export PHD_VAR_network_nic_base="54:52:00" export PHD_VAR_vm_base="/srv/rhos6-rhel7-vms/rhos6-rhel7-base.img" export PHD_VAR_vm_vcpu="1" export PHD_VAR_vm_ram="2048" export PHD_ENV_nodes1=east-01.lab.bos.redhat.com cat<<-EOF > /localvms/template.xml VM_NAME ${PHD_VAR_vm_ram}000 ${PHD_VAR_vm_ram}000 ${PHD_VAR_vm_cpus} hvm destroy restart restart /usr/libexec/qemu-kvm EOF sequence=16 lastoct="$(hostname -s | sed -e 's#^[a-z]*-##g' -e 's#^0*##g')" offset="$(echo ${PHD_ENV_nodes1} | sed -e 's#^[a-z]*-##g' -e 's#^0*##g' -e 's#\..*##')" for section in lb db rabbitmq memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer qpid node keystone; do dobuild=0 if [ -z $onlyone ]; then dobuild=1 elif [ $onlyone = $section ]; then dobuild=1 fi if [ $dobuild = 1 ]; then cd /localvms/ target=rhos6-${section}$(( ${lastoct} - ${offset} )) virsh destroy $target > /dev/null 2>&1 virsh undefine $target > /dev/null 2>&1 cp template.xml ${target}.xml sed -i.sed s#VM_NAME#${target}#g ${target}.xml sed -i.sed s#EXTERNAL_MAC#${PHD_VAR_network_nic_base}:0${lastoct}:00:${sequence}#g ${target}.xml sed -i.sed s#INTERNAL_MAC#${PHD_VAR_network_nic_base}:0${lastoct}:01:${sequence}#g ${target}.xml sed -i.sed s:source\ file.*\/:source\ file=\'/localvms/${target}.cow\'\/:g ${target}.xml diff -u template.xml ${target}.xml rm -f /localvms/${target}.cow qemu-img create -b /localvms/$(basename ${PHD_VAR_vm_base}) -f qcow2 /localvms/${target}.cow virsh define ${target}.xml if [ $? != 0 ]; then exit 1; fi virsh start ${target} if [ $? != 0 ]; then exit 1; fi rm ${target}.xml.sed ${target}.xml fi sequence=$((sequence + 1)) done ================================================ FILE: pcmk/NovaCompute ================================================ #!/bin/sh # # # NovaCompute agent manages compute daemons. # # Copyright (c) 2015 # # This program is free software; you can redistribute it and/or modify # it under the terms of version 2 of the GNU General Public License as # published by the Free Software Foundation. # # This program is distributed in the hope that it would be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # # Further, this software is distributed without any warranty that it is # free of the rightful claim of any third person regarding infringement # or the like. Any license provided herein, whether implied or # otherwise, applies only to this software file. Patent licenses, if # any, provided herein do not apply to combinations of this program with # other software, or any other product whatsoever. # # You should have received a copy of the GNU General Public License # along with this program; if not, write the Free Software Foundation, # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. # ####################################################################### # Initialization: ### : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat} . ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs ### : ${__OCF_ACTION=$1} ####################################################################### meta_data() { cat < 1.0 OpenStack Nova Compute Server. OpenStack Nova Compute Server Authorization URL for connecting to keystone in admin context Authorization URL Username for connecting to keystone in admin context Username Password for connecting to keystone in admin context Password Tenant name for connecting to keystone in admin context. Note that with Keystone V3 tenant names are only unique within a domain. Tenant name DNS domain in which hosts live, useful when the cluster uses short names and nova uses FQDN DNS domain Nova API location (internal, public or admin URL) Nova API location (internal, public or admin URL) Disable shared storage recovery for instances. Use at your own risk! Disable shared storage recovery for instances How long to wait for nova to finish evacuating instances elsewhere before starting nova-compute. Only used when the agent detects evacuations might be in progress. You may need to increase the start timeout when increasing this value. Delay to allow evacuations time to complete END } ####################################################################### # don't exit on TERM, to test that lrmd makes sure that we do exit trap sigterm_handler TERM sigterm_handler() { ocf_log info "They use TERM to bring us down. No such luck." return } nova_usage() { cat </dev/null) if [ $? = 1 ]; then if [ "x${OCF_RESKEY_domain}" != x ]; then NOVA_HOST=$(uname -n | awk -F. '{print $1}') else NOVA_HOST=$(uname -n) fi fi # We only need to check a configured value, calculated ones are fine openstack-config --get /etc/nova/nova.conf DEFAULT host 2>/dev/null if [ $? = 0 ]; then if [ "x${OCF_RESKEY_domain}" != x ]; then short_host=$(uname -n | awk -F. '{print $1}') if [ "x$NOVA_HOST" != "x${short_host}" ]; then ocf_exit_reason "Invalid Nova host name, must be ${short_host} in order for instance recovery to function" rc=$OCF_ERR_CONFIGURED fi elif [ "x$NOVA_HOST" != "x$(uname -n)" ]; then ocf_exit_reason "Invalid Nova host name, must be $(uname -n) in order for instance recovery to function" rc=$OCF_ERR_CONFIGURED fi fi if [ $rc != $OCF_SUCCESS ]; then exit $rc fi return $rc } : ${OCF_RESKEY_evacuation_delay=120} case $__OCF_ACTION in meta-data) meta_data exit $OCF_SUCCESS ;; usage|help) nova_usage exit $OCF_SUCCESS ;; esac case $__OCF_ACTION in start) nova_validate; nova_start;; stop) nova_stop;; monitor) nova_validate; nova_monitor;; notify) nova_notify;; validate-all) exit $OCF_SUCCESS;; *) nova_usage exit $OCF_ERR_UNIMPLEMENTED ;; esac rc=$? ocf_log debug "${OCF_RESOURCE_INSTANCE} $__OCF_ACTION : $rc" exit $rc ================================================ FILE: pcmk/NovaEvacuate ================================================ #!/bin/sh # # # NovaCompute agent manages compute daemons. # # Copyright (c) 2015 # # This program is free software; you can redistribute it and/or modify # it under the terms of version 2 of the GNU General Public License as # published by the Free Software Foundation. # # This program is distributed in the hope that it would be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # # Further, this software is distributed without any warranty that it is # free of the rightful claim of any third person regarding infringement # or the like. Any license provided herein, whether implied or # otherwise, applies only to this software file. Patent licenses, if # any, provided herein do not apply to combinations of this program with # other software, or any other product whatsoever. # # You should have received a copy of the GNU General Public License # along with this program; if not, write the Free Software Foundation, # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. # ####################################################################### # Initialization: ### : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat} . ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs ### : ${__OCF_ACTION=$1} ####################################################################### meta_data() { cat < 1.0 Facility for tacking a list of compute nodes and reliably evacuating the ones that fence_evacuate has flagged. Evacuator for OpenStack Nova Compute Server Authorization URL for connecting to keystone in admin context Authorization URL Username for connecting to keystone in admin context Username Password for connecting to keystone in admin context Password Tenant name for connecting to keystone in admin context. Note that with Keystone V3 tenant names are only unique within a domain. Tenant name Nova API location (internal, public or admin URL) Nova API location (internal, public or admin URL) Disable shared storage recovery for instances. Use at your own risk! Disable shared storage recovery for instances END } ####################################################################### # don't exit on TERM, to test that lrmd makes sure that we do exit trap sigterm_handler TERM sigterm_handler() { ocf_log info "They use TERM to bring us down. No such luck." return } evacuate_usage() { cat < /dev/null 2>&1" "$node" phd_wait_connection 2400 $node || exit 1 done .... target=all .... # wait for the old snapshot to be merged/deleted loop=0 while ! lvcreate -s -n baremetal_snap -l100%FREE /dev/mapper/*root; do sleep 1 if [ "$loop" = 240 ]; then echo "Unknown error waiting for old snap to be deleted/merged" exit 1 fi loop=$((loop + 1)) done .... ================================================ FILE: pcmk/baremetal.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # We start with 3 (or more, up to 16) nodes running a minimal CentOS 6 # # Tasks to be performed include: # - setting up the required repositories from which to download Openstack and the HA-Addon # - disabling firewalls and SElinux. This is a necessary evil until the proper policies can be written. # - creating network bridges for use by VMs hosting OpenStack services # - normalizing network interface names # - fixing multicast # - removing /home and making the root partition as large as possible to maximumize the amount of space available to openstack ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_domain PHD_VAR_network_internal PHD_VAR_network_nic_external PHD_VAR_network_nic_internal PHD_VAR_network_named_forwarders PHD_VAR_rpm_download ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 9 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm wget rhos-release 7 wget -O /etc/yum.repos.d/test.repo http://www.kronosnet.org/testrepo/test.repo yum clean all yum update -y # ntpd conflicts with chrony yum erase -y chrony rm -f /etc/chrony* yum install -y pacemaker fence-agents resource-agents pcs libvirt qemu-kvm bind-utils net-tools tcpdump ntp ntpdate sos nfs-utils # The cluster shouldn't need NTP configured, but without it the # network goes bye-bye when using DHCP # # Must point to clock.redhat.com to work internally sed -i s/^server.*// /etc/ntp.conf echo "server $PHD_VAR_network_clock iburst" >> /etc/ntp.conf echo $PHD_VAR_network_clock > /etc/ntp/step-tickers #sync_to_hardware clock echo "SYNC_HWCLOCK=yes" >> /etc/sysconfig/ntpdate systemctl enable ntpdate systemctl enable ntpd sed -i -e 's/=enforcing/=disabled/g' /etc/sysconfig/selinux sed -i -e 's/=enforcing/=disabled/g' /etc/selinux/config systemctl disable firewalld systemctl enable libvirtd systemctl start libvirtd virsh net-destroy default virsh net-undefine default lastoct="$(hostname -s | sed -e 's#^[a-z]*-##g' -e 's#^0*##g')" cat > /etc/sysconfig/network-scripts/ifcfg-ext0 << EOF DEVICE=ext0 NAME=ext0 TYPE=Bridge BOOTPROTO=dhcp ONBOOT=yes IPV6INIT=yes IPV6_AUTOCONF=yes EOF cat > /etc/sysconfig/network-scripts/ifcfg-vmnet0 << EOF DEVICE=vmnet0 NAME=vmnet0 TYPE=Bridge BOOTPROTO=static ONBOOT=yes IPV6INIT=yes IPV6_AUTOCONF=yes IPADDR=${PHD_VAR_network_internal}.$lastoct NETMASK=255.255.255.0 NETWORK=${PHD_VAR_network_internal}.0 EOF for device in `ls -1 /sys/class/net`; do case ${PHD_VAR_network_nic_external} in *${device}*) cat > /etc/sysconfig/network-scripts/ifcfg-${device} << EOF DEVICE=${device} BOOTPROTO=none ONBOOT=yes BRIDGE=ext0 NAME=${device} EOF ;; esac case ${PHD_VAR_network_nic_internal} in *${device}*) cat > /etc/sysconfig/network-scripts/ifcfg-${device} << EOF DEVICE=${device} BOOTPROTO=none ONBOOT=yes BRIDGE=vmnet0 NAME=${device} EOF ;; esac done if grep -q ip_forward /etc/sysctl.conf; then sed -i -e 's#ip_forward.*#ip_forward = 1#g' /etc/sysctl.conf else echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf fi echo "echo 1 > /sys/class/net/ext0/bridge/multicast_querier" >> /etc/rc.d/rc.local echo "echo 1 > /sys/class/net/vmnet0/bridge/multicast_querier" >> /etc/rc.d/rc.local chmod +x /etc/rc.d/rc.local # Turn off auto-generation of resolv.conf so we can override it # make a backup that we will need for gateway scenario otherwise # we have a catch 22 rm -f /etc/resolv.conf.backup cp /etc/resolv.conf /etc/resolv.conf.backup echo PEERDNS=no >> /etc/sysconfig/network-scripts/ifcfg-ext0 echo search vmnet.${PHD_VAR_network_domain} ${PHD_VAR_network_domain} > /etc/resolv.conf echo nameserver ${PHD_VAR_network_internal}.1 >> /etc/resolv.conf # get rid of /home from beaker sed -i -e 's#.*home.*##g' /etc/fstab umount /home # remove home lv lvremove -f /dev/mapper/*home # expand root lv to 50% vg lvresize -f -l50%VG /dev/mapper/*root # expand root fs xfs_growfs /dev/mapper/*-root # create the snapshot lvcreate -s -n baremetal_snap -l100%FREE /dev/mapper/*root # regenerate initramfs to include dm-snapshot modules/utils for i in $(ls /boot/vmlinuz-*x86*); do ver=$(basename $i | sed -e 's#vmlinuz-##g') dracut -f --kver $ver done .... # Implied by the reboot below #target=all #.... #service network restart #/etc/rc.local #.... target=local .... # Reboot each node and wait for it to return # disable set -e when calling phd_cmd_* because # phd doesn't manage all return codes properly set +e for node in $(echo $PHD_ENV_nodes); do phd_cmd_exec "reboot > /dev/null 2>&1" "$node" phd_wait_connection 2400 $node || exit 1 done .... ================================================ FILE: pcmk/basic-cluster.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - installing the cluster software # - enabling the pcs daemon to allow remote management # - setting a password for the hacluster user for use with pcs # - authenticating to pcs on the other hosts with the hacluster user and password # - creating and starting the cluster # - configuring fencing using the multicast addresses specified for fence_virt on the bare metal hosts ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_env_password PHD_VAR_network_clock PHD_VAR_network_internal PHD_VAR_secrets_fence_xvm ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm rhos-release 7 wget -O /etc/yum.repos.d/test.repo http://www.kronosnet.org/testrepo/test.repo yum update -y # install the packages yum install -y pcs pacemaker corosync fence-agents-all resource-agents # enable pcsd systemctl enable pcsd systemctl start pcsd systemctl disable firewalld systemctl stop firewalld # The cluster shouldn't need NTP configured, but without it the # network goes bye-bye when using DHCP # # Must point to clock.redhat.com to work internally sed -i s/^server.*// /etc/ntp.conf echo "server $PHD_VAR_network_clock iburst" >> /etc/ntp.conf echo $PHD_VAR_network_clock > /etc/ntp/step-tickers #sync_to_hardware clock echo "SYNC_HWCLOCK=yes" >> /etc/sysconfig/ntpdate systemctl enable ntpdate systemctl start ntpdate systemctl enable ntpd systemctl start ntpd # set a password for hacluster user. password should be the same on all nodes echo ${PHD_VAR_env_password} | passwd --stdin hacluster # Now mount /srv so that we can use $PHD_VAR_osp_configdir further down if grep -q srv /etc/fstab; then echo /srv is already mounted; else mkdir -p /srv echo "${PHD_VAR_network_internal}.1:/srv /srv nfs defaults,v3 0 0" >> /etc/fstab mount /srv fi # Set up the authkey mkdir -p /etc/cluster echo ${PHD_VAR_secrets_fence_xvm} > /etc/cluster/fence_xvm.key .... target=$PHD_ENV_nodes1 .... short_nodes="" for node in $PHD_ENV_nodes; do short_nodes="${short_nodes} $(echo ${node} | sed s/\\..*//g)" done # autheticate nodes, requires all nodes to have pcsd up and running # the -p option is used to give the password on command line and make it easier to script pcs cluster auth $short_nodes -u hacluster -p ${PHD_VAR_env_password} --force # Construct and start the cluster # The cluster needs a unique name, base it on the first node's name pcs cluster setup --force --name $(echo $PHD_ENV_nodes1 | sed -e s/[0-9]//g -e s/\\..*//g ) ${short_nodes} pcs cluster enable --all pcs cluster start --all # Give the cluster a moment to come up sleep 5 # Assumes bare metal nodes are: # - named with a trailing offset # - numbered sequentially # - the first node is a proxy # - total of three nodes # - are configured with fence-virtd (see virt-hosts.scenario) # # If any of the above is not true, change the fencing devices below pcs stonith create fence1 fence_xvm multicast_address=225.0.0.2 pcs stonith create fence2 fence_xvm multicast_address=225.0.0.3 pcs stonith create fence3 fence_xvm multicast_address=225.0.0.4 # For clones this is not so important # # However we really don't wan't VIPs moving around as it can cause # fencing to fail (eg. if vip-db is stopped, then keystone won't # function and we can't confirm if nova recognized the compute node is # down) pcs resource defaults resource-stickiness=INFINITY .... ================================================ FILE: pcmk/beaker.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # We start with 3 (or more, up to 16) nodes running a minimal CentOS 6 # # Tasks to be performed include: # - setting up the required repositories from which to download Openstack and the HA-Addon # - disabling firewalls and SElinux. This is a necessary evil until the proper policies can be written. # - creating network bridges for use by VMs hosting OpenStack services # - normalizing network interface names # - fixing multicast # - removing /home and making the root partition as large as possible to maximumize the amount of space available to openstack ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_domain PHD_VAR_beaker_disttree ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 7 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=local .... # Reboot each node and wait for it to return # REQUIRES kinit and we don't check for completition yet for node in $(echo $PHD_ENV_nodes); do loop=0 bkr system-provision --distro-tree ${PHD_VAR_beaker_disttree} $node while [ "$(ping $node -c 3 -q | grep 'packets transmitted' | sed -e 's#.*transmitted, ##g' -e 's# received.*##g')" != 0 ] && \ [ "$loop" -lt 180 ]; do loop=$((loop + 1)) echo "Waiting for beaker to kick in ($loop)" sleep 1 done done # disable set -e when calling phd_cmd_* because # phd doesn't manage all return codes properly set +e for node in $(echo $PHD_ENV_nodes); do phd_wait_connection 2400 $node || exit 1 done .... ================================================ FILE: pcmk/ceilometer-test.sh ================================================ . ${PHD_VAR_env_configdir}/keystonerc_admin for m in storage.objects image network volume instance ; do ceilometer sample-list -m $m | tail -2 ; done # https://bugzilla.redhat.com/show_bug.cgi?id=1127526#c2 <- for A/A testing ================================================ FILE: pcmk/ceilometer.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 = VARIABLES = PHD_VAR_deployment PHD_VAR_network_hosts_memcache PHD_VAR_network_hosts_mongodb PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_internal ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-ceilometer-api openstack-ceilometer-central openstack-ceilometer-collector openstack-ceilometer-common openstack-ceilometer-alarm python-ceilometer python-ceilometerclient redis # have redis listen on all IPs sed -i "s/\s*bind \(.*\)$/#bind \1/" /etc/redis.conf openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_host vip-keystone openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_port 35357 openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_protocol http openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_user ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 openstack-config --set /etc/ceilometer/ceilometer.conf coordination backend_url 'redis://vip-redis:6379' openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://vip-keystone:5000/v2.0 openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://${PHD_VAR_network_hosts_mongodb}:27017/ceilometer?replicaSet=ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf database max_retries -1 # keep last 5 days data only (value is in secs). Don't set to retain all data indefinetely. openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000 openstack-config --set /etc/ceilometer/ceilometer.conf api host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') .... target=$PHD_ENV_nodes1 .... pcs resource create redis redis wait_last_known_master=true --master meta notify=true ordered=true interleave=true pcs resource create vip-redis IPaddr2 ip=${PHD_VAR_network_internal}.217 pcs resource create openstack-ceilometer-central systemd:openstack-ceilometer-central --clone interleave=true pcs resource create openstack-ceilometer-collector systemd:openstack-ceilometer-collector --clone interleave=true pcs resource create openstack-ceilometer-api systemd:openstack-ceilometer-api --clone interleave=true pcs resource create delay Delay startdelay=10 --clone interleave=true pcs resource create openstack-ceilometer-alarm-evaluator systemd:openstack-ceilometer-alarm-evaluator --clone interleave=true pcs resource create openstack-ceilometer-alarm-notifier systemd:openstack-ceilometer-alarm-notifier --clone interleave=true pcs resource create openstack-ceilometer-notification systemd:openstack-ceilometer-notification --clone interleave=true pcs constraint order promote redis-master then start vip-redis pcs constraint colocation add vip-redis with master redis-master pcs constraint order start vip-redis then openstack-ceilometer-central-clone kind=Optional pcs constraint order start openstack-ceilometer-central-clone then openstack-ceilometer-collector-clone pcs constraint order start openstack-ceilometer-collector-clone then openstack-ceilometer-api-clone pcs constraint colocation add openstack-ceilometer-api-clone with openstack-ceilometer-collector-clone pcs constraint order start openstack-ceilometer-api-clone then delay-clone pcs constraint colocation add openstack-ceilometer-delay-clone with openstack-ceilometer-api-clone pcs constraint order start openstack-ceilometer-delay-clone then openstack-ceilometer-alarm-evaluator-clone pcs constraint colocation add openstack-ceilometer-alarm-evaluator-clone with openstack-ceilometer-delay-clone pcs constraint order start openstack-ceilometer-alarm-evaluator-clone then openstack-ceilometer-alarm-notifier-clone pcs constraint colocation add openstack-ceilometer-alarm-notifier-clone with openstack-ceilometer-alarm-evaluator-clone pcs constraint order start openstack-ceilometer-alarm-notifier-clone then openstack-ceilometer-notification-clone pcs constraint colocation add openstack-ceilometer-notification-clone with openstack-ceilometer-alarm-notifier-clone if [ $PHD_VAR_deployment = collapsed ]; then pcs constraint order start mongod-clone then openstack-ceilometer-central-clone pcs constraint order start openstack-keystone-clone then openstack-ceilometer-central-clone fi .... ================================================ FILE: pcmk/cinder-test.sh ================================================ . ${PHD_VAR_env_configdir}/keystonerc_admin openstack volume list openstack volume create --size 10 test-volume openstack volume list openstack volume delete test-volume openstack volume list ================================================ FILE: pcmk/cinder.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_deployment PHD_VAR_osp_major PHD_VAR_osp_configdir PHD_VAR_network_internal PHD_VAR_network_hosts_memcache PHD_VAR_network_hosts_rabbitmq ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-cinder openstack-utils python-memcached python-keystonemiddleware python-openstackclient # Pending ack from cinder team #openstack-config --set /etc/cinder/cinder.conf DEFAULT enable_v1_api false #openstack-config --set /etc/cinder/cinder.conf DEFAULT enable_v2_api true openstack-config --set /etc/cinder/cinder.conf database connection mysql://cinder:cindertest@vip-db/cinder openstack-config --set /etc/cinder/cinder.conf database max_retries -1 openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone openstack-config --set /etc/cinder/cinder.conf keystone_authtoken identity_uri http://vip-keystone:35357/ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://vip-keystone:5000/ openstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_user cinder openstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_password cindertest openstack-config --set /etc/cinder/cinder.conf DEFAULT notification_driver messaging openstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_host vip-glance openstack-config --set /etc/cinder/cinder.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 # rdo${PHD_VAR_osp_major}-cinder isn't the name of a real host or an IP # Its the name which we should advertise ourselves as and for A/P it should be the same everywhere openstack-config --set /etc/cinder/cinder.conf DEFAULT host rdo${PHD_VAR_osp_major}-cinder openstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfs_exports openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_sparsed_volumes true openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_mount_options v3 openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.nfs.NfsDriver # NOTE: this config section is to enable and configure the NFS cinder driver. # Create the directory on the server mkdir -p $PHD_VAR_osp_configdir/cinder chown -R cinder:cinder $PHD_VAR_osp_configdir/cinder cat > /etc/cinder/nfs_exports << EOF ${PHD_VAR_network_internal}.1:$PHD_VAR_osp_configdir/cinder EOF chown root:cinder /etc/cinder/nfs_exports chmod 0640 /etc/cinder/nfs_exports .... target=$PHD_ENV_nodes1 .... su cinder -s /bin/sh -c "cinder-manage db sync" # create services in pacemaker pcs resource create openstack-cinder-api systemd:openstack-cinder-api --clone interleave=true pcs resource create openstack-cinder-scheduler systemd:openstack-cinder-scheduler --clone interleave=true # Volume must be A/P for now. See https://bugzilla.redhat.com/show_bug.cgi?id=1193229 pcs resource create openstack-cinder-volume systemd:openstack-cinder-volume pcs constraint order start openstack-cinder-api-clone then openstack-cinder-scheduler-clone pcs constraint colocation add openstack-cinder-scheduler-clone with openstack-cinder-api-clone pcs constraint order start openstack-cinder-scheduler-clone then openstack-cinder-volume pcs constraint colocation add openstack-cinder-volume with openstack-cinder-scheduler-clone if [ $PHD_VAR_deployment = collapsed ]; then pcs constraint order start openstack-keystone-clone then openstack-cinder-api-clone fi .... ================================================ FILE: pcmk/compute-cluster.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 = VARIABLES = PHD_VAR_env_password PHD_VAR_network_internal ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... # make sure we are not running on a controller node pcs resource show swift-fs > /dev/null 2>&1 if [ $? != 0 ]; then pcs resource show computenode > /dev/null 2>&1 if [ $? = 0 ]; then exit 0 fi # We must be either the first node to run, or configuring # single-node clusters for a segregated deployment pcs cluster auth $(hostname -s) -u hacluster -p ${PHD_VAR_env_password} --force pcs cluster setup --name $(hostname -s)-compute $(hostname -s) pcs cluster enable --all pcs cluster start --all sleep 30 # hack!! IPMI does not work from localhost to localhost on those test nodes!. Use # proper fencing settings! # TODO: Add SBD support pcs property set stonith-enabled=false pcs resource create nova-compute-fs Filesystem device="${PHD_VAR_network_internal}.1:$PHD_VAR_osp_configdir/instances" directory="/var/lib/nova/instances" fstype="nfs" options="v3" op start timeout=240 --group computenode pcs resource create neutron-openvswitch-agent systemd:neutron-openvswitch-agent --group computenode pcs resource create libvirtd systemd:libvirtd --group computenode pcs resource create ceilometer-compute systemd:openstack-ceilometer-compute --group computenode pcs resource create nova-compute systemd:openstack-nova-compute --group computenode fi .... ================================================ FILE: pcmk/compute-common.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 = VARIABLES = PHD_VAR_env_password PHD_VAR_network_domain PHD_VAR_network_hosts_gateway PHD_VAR_network_hosts_memcache PHD_VAR_network_hosts_mongodb PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_internal PHD_VAR_osp_configdir ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-nova-compute openstack-utils python-cinder openstack-neutron-openvswitch openstack-ceilometer-compute python-memcached openstack-neutron # we will use this one as instance shared storage mkdir -p ${PHD_VAR_osp_configdir}/instances chown nova:nova ${PHD_VAR_osp_configdir}/instances systemctl disable firewalld systemctl stop firewalld systemctl enable pcsd systemctl start pcsd echo ${PHD_VAR_env_password} | passwd --stdin hacluster systemctl enable openvswitch systemctl start openvswitch ovs-vsctl add-br br-int systemctl stop libvirtd systemctl disable libvirtd # NOTE: vmnet is the interface connected to the internal network. openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address $(ip addr show dev vmnet0 scope global | grep inet | sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 0.0.0.0 # NOTE: same consideration as nova configuration applies here. They need to match. openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://${PHD_VAR_network_hosts_gateway}.${PHD_VAR_network_domain}:6080/vnc_auto.html openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@vip-db/nova openstack-config --set /etc/nova/nova.conf database max_retries -1 openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone openstack-config --set /etc/nova/nova.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 # FIX ME: nova doesn't like hostnames anymore? # openstack-config --set /etc/nova/nova.conf DEFAULT metadata_host vip-nova openstack-config --set /etc/nova/nova.conf DEFAULT metadata_host ${PHD_VAR_network_internal}.210 openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 0.0.0.0 openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen_port 8775 openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest openstack-config --set /etc/nova/nova.conf glance host vip-glance openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API openstack-config --set /etc/nova/nova.conf neutron url http://vip-neutron:9696/ openstack-config --set /etc/nova/nova.conf neutron admin_tenant_name services openstack-config --set /etc/nova/nova.conf neutron admin_username neutron openstack-config --set /etc/nova/nova.conf neutron admin_password neutrontest openstack-config --set /etc/nova/nova.conf neutron admin_auth_url http://vip-keystone:35357/v2.0 openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver openstack-config --set /etc/nova/nova.conf DEFAULT libvirt_vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron openstack-config --set /etc/nova/nova.conf conductor use_local false # REQUIRED FOR A/A scheduler openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_host_subset_size 30 openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_host vip-keystone openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_tenant_name services openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_user compute openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_password novatest openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password neutrontest openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://vip-keystone:35357/v2.0/ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken identity_uri http://vip-keystone:5000/ openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent tunnel_types vxlan openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent vxlan_udp_port 4789 openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs enable_tunneling True openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_id_ranges 1:1000 openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tenant_network_type vxlan openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs integration_bridge br-int openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_bridge br-tun openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs local_ip $(ip addr show dev vmnet0 scope global | grep inet | sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent l2_population False openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit True openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit_period hour openstack-config --set /etc/nova/nova.conf DEFAULT notify_on_state_change vm_and_task_state openstack-config --set /etc/nova/nova.conf DEFAULT notification_driver nova.openstack.common.notifier.rpc_notifier sed -i -e 's#nova.openstack.common.notifier.rpc_notifier#nova.openstack.common.notifier.rpc_notifier\nnotification_driver = ceilometer.compute.nova_notifier#g' /etc/nova/nova.conf openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_host vip-keystone openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_port 35357 openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_protocol http openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_user ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://vip-keystone:5000/v2.0 openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://${PHD_VAR_network_hosts_mongodb}:27017/ceilometer?replicaSet=ceilometer openstack-config --set /etc/ceilometer/ceilometer.conf database connection max_retries -1 # keep last 5 days data only (value is in secs) openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000 .... ================================================ FILE: pcmk/compute-managed.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 = VARIABLES = PHD_VAR_deployment PHD_VAR_osp_configdir PHD_VAR_network_domain PHD_VAR_network_internal ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... if [ $PHD_VAR_deployment = segregated ]; then echo "We don't document managed compute nodes in a segregated environment yet" # Certainly none of the location constraints would work and the # resource-discovery options are mostly redundant exit 1 fi yum install -y pacemaker-remote resource-agents pcs if [ ! -e $PHD_VAR_osp_configdir/pcmk-authkey ]; then dd if=/dev/urandom of=$PHD_VAR_osp_configdir/pcmk-authkey bs=4096 count=1 fi mkdir -p /etc/pacemaker cp $PHD_VAR_osp_configdir/pcmk-authkey /etc/pacemaker/authkey if [ -z "$(pidof pacemakerd)" ]; then chkconfig pacemaker_remote on service pacemaker_remote start fi .... ================================================ FILE: pcmk/controller-managed.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 = VARIABLES = PHD_VAR_deployment PHD_VAR_osp_configdir PHD_VAR_network_domain PHD_VAR_network_internal PHD_VAR_network_hosts_gateway ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... if [ $PHD_VAR_deployment = segregated ]; then echo "We don't document managed compute nodes in a segregated environment yet" # Certainly none of the location constraints would work and the # resource-discovery options are mostly redundant exit 1 fi mkdir -p /etc/pacemaker cp $PHD_VAR_osp_configdir/pcmk-authkey /etc/pacemaker/authkey .... target=$PHD_ENV_nodes1 .... # add NovaEvacuate. It must be A/P and that is perfectly acceptable # to avoid need of a cluster wide locking pcs resource create nova-evacuate ocf:openstack:NovaEvacuate auth_url=http://vip-keystone:35357/v2.0/ username=admin password=keystonetest tenant_name=admin # without any of those services, nova-evacuate is useless # later we also add a order start on nova-compute (after -compute is defined) for i in vip-glance vip-cinder vip-neutron vip-nova vip-db vip-rabbitmq vip-keystone cinder-volume; do pcs constraint order start $i then nova-evacuate done for i in glance-api-clone neutron-metadata-agent-clone nova-conductor-clone; do pcs constraint order start $i then nova-evacuate require-all=false done # Take down the ODP control plane pcs resource disable openstack-keystone --wait=240 # Take advantage of the fact that control nodes will already be part of the cluster # At this step, we need to teach the cluster about the compute nodes # # This requires running commands on the cluster based on the names of the compute nodes controllers=$(cibadmin -Q -o nodes | grep uname | sed s/.*uname..// | awk -F\" '{print $1}') for controller in ${controllers}; do pcs property set --node ${controller} osprole=controller done # Force services to run only on nodes with osprole = controller # # Importantly it also tells Pacemaker not to even look for the services on other # nodes. This helps reduce noise and collisions with services that fill the same role # on compute nodes. stonithdevs=$(pcs stonith | awk '{print $1}') for i in $(cibadmin -Q --xpath //primitive --node-path | tr ' ' '\n' | awk -F "id='" '{print $2}' | awk -F "'" '{print $1}' | uniq); do found=0 if [ -n "$stonithdevs" ]; then for x in $stonithdevs; do if [ $x = $i ]; then found=1 fi done fi if [ $found = 0 ]; then pcs constraint location $i rule resource-discovery=exclusive score=0 osprole eq controller fi done # Now (because the compute nodes have roles assigned to them and keystone is # stopped) it is safe to define the services that will run on the compute nodes # neutron-openvswitch-agent pcs resource create neutron-openvswitch-agent-compute systemd:neutron-openvswitch-agent --clone interleave=true --disabled --force pcs constraint location neutron-openvswitch-agent-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start neutron-server-clone then neutron-openvswitch-agent-compute-clone require-all=false # libvirtd pcs resource create libvirtd-compute systemd:libvirtd --clone interleave=true --disabled --force pcs constraint location libvirtd-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start neutron-openvswitch-agent-compute-clone then libvirtd-compute-clone pcs constraint colocation add libvirtd-compute-clone with neutron-openvswitch-agent-compute-clone # openstack-ceilometer-compute pcs resource create ceilometer-compute systemd:openstack-ceilometer-compute --clone interleave=true --disabled --force pcs constraint location ceilometer-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start ceilometer-notification-clone then ceilometer-compute-clone require-all=false pcs constraint order start libvirtd-compute-clone then ceilometer-compute-clone pcs constraint colocation add ceilometer-compute-clone with libvirtd-compute-clone # nfs mount for nova-compute shared storage pcs resource create nova-compute-fs Filesystem device="${PHD_VAR_network_internal}.1:$PHD_VAR_osp_configdir/instances" directory="/var/lib/nova/instances" fstype="nfs" options="v3" op start timeout=240 --clone interleave=true --disabled --force pcs constraint location nova-compute-fs-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start ceilometer-compute-clone then nova-compute-fs-clone pcs constraint colocation add nova-compute-fs-clone with ceilometer-compute-clone # nova-compute pcs resource create nova-compute ocf:openstack:NovaCompute auth_url=http://vip-keystone:35357/v2.0/ username=admin password=keystonetest tenant_name=admin domain=${PHD_VAR_network_domain} op start timeout=300 --clone interleave=true --disabled --force pcs constraint location nova-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute pcs constraint order start nova-conductor-clone then nova-compute-clone require-all=false pcs constraint order start nova-compute-fs-clone then nova-compute-clone require-all=false pcs constraint colocation add nova-compute-clone with nova-compute-fs-clone pcs constraint order start nova-compute-clone then nova-evacuate require-all=false case ${PHD_VAR_network_hosts_gateway} in east-*) pcs stonith create fence-compute fence_apc ipaddr=east-apc login=apc passwd=apc pcmk_host_map="east-01:2;east-02:3;east-03:4;east-04:5;east-05:6;east-06:7;east-07:9;east-08:10;east-09:11;east-10:12;east-11:13;east-12:14;east-13:15;east-14:18;east-15:17;east-16:19;" --force ;; mrg-*) pcs stonith create fence-compute fence_apc_snmp ipaddr=apc-ap7941-l2h3.mgmt.lab.eng.bos.redhat.com power_wait=10 pcmk_host_map="mrg-07:10;mrg-08:12;mrg-09:14" ;; esac pcs stonith create fence-nova fence_compute auth-url=http://vip-keystone:35357/v2.0/ login=admin passwd=keystonetest tenant-name=admin domain=${PHD_VAR_network_domain} record-only=1 action=off --force # while this is set in basic.cluster, it looks like OSPd doesn't set it. pcs resource defaults resource-stickiness=INFINITY # allow compute nodes to rejoin the cluster automatically # 1m might be a bit aggressive tho pcs property set cluster-recheck-interval=1min for node in ${PHD_ENV_nodes}; do found=0 short_node=$(echo ${node} | sed s/\\..*//g) for controller in ${controllers}; do if [ ${short_node} = ${controller} ]; then found=1 fi done if [ $found = 0 ]; then # We only want to execute the following _for_ the compute nodes, not _on_ the compute nodes # Rather annoying pcs resource create ${short_node} ocf:pacemaker:remote reconnect_interval=60 op monitor interval=20 pcs property set --node ${short_node} osprole=compute pcs stonith level add 1 ${short_node} fence-compute,fence-nova fi done pcs resource enable openstack-keystone pcs resource enable neutron-openvswitch-agent-compute pcs resource enable libvirtd-compute pcs resource enable openstack-ceilometer-compute pcs resource enable nova-compute-fs pcs resource enable nova-compute # cleanup after us sleep 60 pcs resource cleanup .... ================================================ FILE: pcmk/galera-test.sh ================================================ clustercheck # verify sync is done mysql SHOW STATUS LIKE 'wsrep%'; quit ================================================ FILE: pcmk/galera.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_env_password PHD_VAR_network_domain PHD_VAR_deployment ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y mariadb-galera-server xinetd rsync if [ $PHD_VAR_deployment = collapsed ]; then # Allowing the proxy to perform health checks on galera while we're initializing it is... problematic. pcs resource disable haproxy fi cat > /etc/sysconfig/clustercheck << EOF MYSQL_USERNAME="clustercheck" MYSQL_PASSWORD="${PHD_VAR_env_password}" MYSQL_HOST="localhost" MYSQL_PORT="3306" EOF # workaround some old buggy mariadb packages # that created log files as root:root and newer # packages would fail to start.... chown mysql:mysql /var/log/mariadb -R systemctl start mysqld # required for clustercheck to work mysql -e "CREATE USER 'clustercheck'@'localhost' IDENTIFIED BY '${PHD_VAR_env_password}';" systemctl stop mysqld # Configure galera cluster # NOTE: wsrep ssl encryption is strongly recommended and should be enabled # on all production deployments. This how-to does NOT display how to # configure ssl. The shell expansion points to the internal IP address of the # node. cat > /etc/my.cnf.d/galera.cnf << EOF [mysqld] skip-name-resolve=1 binlog_format=ROW default-storage-engine=innodb innodb_autoinc_lock_mode=2 innodb_locks_unsafe_for_binlog=1 query_cache_size=0 query_cache_type=0 bind_address=$(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_cluster_name="galera_cluster" wsrep_slave_threads=1 wsrep_certify_nonPK=1 wsrep_max_ws_rows=131072 wsrep_max_ws_size=1073741824 wsrep_debug=0 wsrep_convert_LOCK_to_trx=0 wsrep_retry_autocommit=1 wsrep_auto_increment_control=1 wsrep_drupal_282555_workaround=0 wsrep_causal_reads=0 wsrep_notify_cmd= wsrep_sst_method=rsync EOF cat > /etc/xinetd.d/galera-monitor << EOF service galera-monitor { port = 9200 disable = no socket_type = stream protocol = tcp wait = no user = root group = root groups = yes server = /usr/bin/clustercheck type = UNLISTED per_source = UNLIMITED log_on_success = log_on_failure = HOST flags = REUSE } EOF systemctl enable xinetd systemctl start xinetd .... target=$PHD_ENV_nodes1 .... # node_list must be of the form node1,node2,node3 # # node names must be in the form that the cluster knows them as # (ie. no domains) and there can't be a trailing comma (hence the # extra weird sed command) node_list=$(echo $PHD_ENV_nodes | sed -e s/.vmnet.${PHD_VAR_network_domain}\ /,/g -e s/.vmnet.${PHD_VAR_network_domain}//) pcs resource create galera galera enable_creation=true wsrep_cluster_address="gcomm://${node_list}" additional_parameters='--open-files-limit=16384' meta master-max=3 ordered=true op promote timeout=300s on-fail=block --master if [ $PHD_VAR_deployment = collapsed ]; then # Now we can re-enable the proxy pcs resource enable haproxy fi # wait for galera to start and become promoted loop=0; while ! clustercheck > /dev/null 2>&1 && [ "$loop" -lt 60 ]; do echo waiting galera to be promoted loop=$((loop + 1)) sleep 5 done # this one can fail depending on who bootstrapped the cluster for node in $PHD_ENV_nodes; do mysql -e "DROP USER ''@'${node}';" || true mysql -e "DROP USER 'root'@'${node}';" || true done galera_script=galera.setup echo "" > $galera_script echo "GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED by 'mysqltest' WITH GRANT OPTION;" >> $galera_script for db in keystone glance cinder neutron nova heat; do cat<> $galera_script CREATE DATABASE ${db}; GRANT ALL ON ${db}.* TO '${db}'@'%' IDENTIFIED BY '${db}test'; EOF done echo "FLUSH PRIVILEGES;" >> $galera_script #echo "quit" >> $galera_script if [ "$loop" -ge 60 ]; then echo Timeout waiting for galera else mysql mysql < $galera_script mysqladmin flush-hosts fi .... ================================================ FILE: pcmk/gateway.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed include: # - Setting up haproxy to expose nova and horizon # - Installing and configuring bind for DNS # - Installing and configuring DHCP. For your own sanity, we define rules that allow you to predictably calculate MAC addresses for the guests. # - Turning off auto-generation of resolv.conf so we can point to our local DNS server ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_osp_major PHD_VAR_components PHD_VAR_network_domain PHD_VAR_network_internal PHD_VAR_network_nic_base ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=$PHD_ENV_nodes1 .... # temporary restore original resolv.conf if [ -e /etc/resolv.conf.backup ]; then rm -f /etc/resolv.conf.new cp /etc/resolv.conf /etc/resolv.conf.new rm -f /etc/resolv.conf cp /etc/resolv.conf.backup /etc/resolv.conf fi echo "/srv ${PHD_VAR_network_internal}.0/255.255.255.0(rw,sync,no_root_squash,subtree_check,fsid=10)" > /etc/exports systemctl enable nfs-server systemctl start nfs-server ext_ip=$(host $PHD_ENV_nodes1 | awk '{print $4}') echo "ext: $ext_ip" iptables -t nat -A POSTROUTING -s ${PHD_VAR_network_internal}.0/24 -o ext0 -j SNAT --to-source ${ext_ip} echo "iptables -t nat -A POSTROUTING -s ${PHD_VAR_network_internal}.0/24 -o ext0 -j SNAT --to-source ${ext_ip}" >> /etc/rc.local yum install -y haproxy ext_ip=$(host $PHD_ENV_nodes1 | awk '{print $4}') echo "ext: $ext_ip" cat <<-EOF > /etc/haproxy/haproxy.cfg global daemon defaults mode tcp maxconn 10000 timeout connect 180s timeout client 180s timeout server 180s frontend vip-horizon-ext bind ${ext_ip}:80 default_backend vip-horizon-int backend vip-horizon-int balance roundrobin server vip-horizon ${PHD_VAR_network_internal}.211:80 check inter 10s frontend vip-nova-ext bind ${ext_ip}:6080 default_backend vip-nova-int backend vip-nova-int balance roundrobin server vip-nova ${PHD_VAR_network_internal}.210:6080 check inter 10s EOF systemctl enable haproxy systemctl start haproxy yum install -y bind sed -i -e 's#127.0.0.1#any#g' -e 's#::1#any#g' -e 's#allow-query#//allow-query#g' /etc/named.conf sed -i -e 's#dnssec-enable yes;#dnssec-enable no;#g' -e 's#dnssec-validation yes;#dnssec-validation no;#g' /etc/named.conf if ! grep -q forwarders /etc/named.conf then sed -i -e 's#recursion yes;#recursion yes;\n\n\tforwarders {\n\t\t10.16.36.29;\n\t\t10.11.5.19;\n\t\t10.5.30.160;\n\t};\n#g' /etc/named.conf echo 'include "/etc/named/lab.zones";' >> /etc/named.conf fi reverse_dns_net=$(echo ${PHD_VAR_network_internal} | awk -F. '{print $3}').$(echo ${PHD_VAR_network_internal} | awk -F. '{print $2}').$(echo ${PHD_VAR_network_internal} | awk -F. '{print $1}') cat <<-EOF > /etc/named/lab.zones zone "vmnet.${PHD_VAR_network_domain}" { type master; file "/etc/named/vmnet.${PHD_VAR_network_domain}"; }; zone "${reverse_dns_net}.in-addr.arpa" { type master; file "/etc/named/${reverse_dns_net}.arpa"; }; EOF cat <<-EOF > /etc/named/${reverse_dns_net}.arpa \$TTL 86400 @ IN SOA mrg-01. fdinitto.redhat.com. ( 2014103001 ; Serial 21600 ; Refresh 10800 ; Retry 1209600 ; Expire 3600 ) ; Minimum IN NS mrg-01. EOF cat <<-EOF > /etc/named/vmnet.${PHD_VAR_network_domain} \$TTL 86400 @ IN SOA mrg-01. fdinitto.redhat.com. ( 2014103001 ; serial 21600 ; refresh (24 hours) 10800 ; retry (2 hours) 1209600 ; expire (1000 hours) 3600 ; minimum (2 days) ) IN NS mrg-01. ipv6-localhost IN A 127.0.0.1 IN AAAA ::1 localhost IN A 127.0.0.1 IN AAAA ::1 EOF lpc=0 # definition_nodes evaluates to all the bare metal nodes for h in ${definition_nodes}; do lpc=$((lpc + 1)) echo "$lpc IN PTR ${h}.vmnet.${PHD_VAR_network_domain}." >> /etc/named/${reverse_dns_net}.arpa echo "$h IN A ${PHD_VAR_network_internal}.${lpc}" >> /etc/named/vmnet.${PHD_VAR_network_domain} done cat <<-EOF >> /etc/named/${reverse_dns_net}.arpa 53 IN PTR rdo${PHD_VAR_osp_major}-rhel7-base.vmnet.${PHD_VAR_network_domain}. 54 IN PTR rdo${PHD_VAR_osp_major}-rhel6-base.vmnet.${PHD_VAR_network_domain}. EOF cat <<-EOF >> /etc/named/vmnet.${PHD_VAR_network_domain} rdo${PHD_VAR_osp_major}-rhel7-base IN A ${PHD_VAR_network_internal}.53 rdo${PHD_VAR_osp_major}-rhel6-base IN A ${PHD_VAR_network_internal}.54 EOF oct=55 for section in ${PHD_VAR_components}; do for count in 1 2 3; do echo "${oct} IN PTR rdo${PHD_VAR_osp_major}-${section}${count}.vmnet.${PHD_VAR_network_domain}." >> /etc/named/${reverse_dns_net}.arpa echo "rdo${PHD_VAR_osp_major}-${section}${count} IN A ${PHD_VAR_network_internal}.${oct}" >> /etc/named/vmnet.${PHD_VAR_network_domain} oct=$((oct + 1)) done done oct=200 for section in ${PHD_VAR_components}; do case $section in lb|memcache|swift-brick|mongodb) : No VIP needed for $section hence no DNS also ;; *) echo "${oct} IN PTR vip-${section}.vmnet.${PHD_VAR_network_domain}." >> /etc/named/${reverse_dns_net}.arpa echo "vip-${section} IN A ${PHD_VAR_network_internal}.${oct}" >> /etc/named/vmnet.${PHD_VAR_network_domain} ;; esac oct=$((oct + 1)) done # add vip-redis outside of PHD_VAR_components because it should not be load balanced echo "${oct} IN PTR vip-redis.vmnet.${PHD_VAR_network_domain}." >> /etc/named/${reverse_dns_net}.arpa echo "vip-redis IN A ${PHD_VAR_network_internal}.${oct}" >> /etc/named/vmnet.${PHD_VAR_network_domain} oct=240 while [ $oct -lt 254 ]; do echo "${oct} IN PTR dhcp${oct}.vmnet.${PHD_VAR_network_domain}." >> /etc/named/${reverse_dns_net}.arpa echo "dhcp${oct} IN A ${PHD_VAR_network_internal}.${oct}" >> /etc/named/vmnet.${PHD_VAR_network_domain} oct=$((oct + 1)) done chown root:named /etc/named -R chmod 750 /etc/named chmod 640 /etc/named/* systemctl enable named systemctl start named yum install -y dhcp echo "DHCDPARGS=vmnet0" > /etc/sysconfig/dhcpd cat <<-EOF > /etc/dhcp/dhcpd.conf default-lease-time 600; max-lease-time 7200; authoritative; log-facility local7; subnet ${PHD_VAR_network_internal}.0 netmask 255.255.255.0 { range ${PHD_VAR_network_internal}.240 ${PHD_VAR_network_internal}.253; option domain-name "vmnet.${PHD_VAR_network_domain}"; option domain-search "vmnet.${PHD_VAR_network_domain}", "${PHD_VAR_network_domain}"; option domain-name-servers ${PHD_VAR_network_internal}.1; option routers ${PHD_VAR_network_internal}.1; } # 54:52:00: <- BASE # 00 <- base 01..09 <- run on mrg-XX # 00 eth0 (ext0), 01 eth1 (vmnet0), 02 for nodeX ?? # 01 <- sequence num for the VM on that host host rdo${PHD_VAR_osp_major}-rhel7-base { hardware ethernet ${PHD_VAR_network_nic_base}:00:01:02; fixed-address rdo${PHD_VAR_osp_major}-rhel7-base.vmnet.${PHD_VAR_network_domain}; } EOF sequence=16 for section in ${PHD_VAR_components}; do for count in 1 2 3; do offset=$(($count + 1)) cat <<-EOF >> /etc/dhcp/dhcpd.conf host rdo${PHD_VAR_osp_major}-${section}${count} { hardware ethernet ${PHD_VAR_network_nic_base}:0${offset}:01:${sequence}; fixed-address rdo${PHD_VAR_osp_major}-${section}${count}.vmnet.${PHD_VAR_network_domain}; } EOF done sequence=$((sequence + 1)) done systemctl enable dhcpd systemctl start dhcpd # restore resolve.conf now that we can use it if [ -e /etc/resolv.conf.new ]; then rm -f /etc/resolv.conf cp /etc/resolv.conf.new /etc/resolv.conf rm -f /etc/resolv.conf.new fi .... target=local .... # Reboot each node and wait for it to return # disable set -e when calling phd_cmd_* because # phd doesn't manage all return codes properly set +e for node in $(echo $PHD_ENV_nodes); do phd_cmd_exec "reboot > /dev/null 2>&1" "$node" phd_wait_connection 2400 $node || exit 1 done .... target=all .... if grep -q srv /etc/exports; then echo "We are on the nfs-server" exit 0 fi if grep -q srv /etc/fstab; then echo /srv is already mounted; else mkdir -p /srv echo "${PHD_VAR_network_internal}.1:/srv /srv nfs defaults,v3 0 0" >> /etc/fstab mount /srv fi ================================================ FILE: pcmk/glance-test.sh ================================================ . ${PHD_VAR_env_configdir}/keystonerc_admin if [ ! -f ${PHD_VAR_env_configdir}/cirros-0.3.2-x86_64-disk.img ]; then wget -O ${PHD_VAR_env_configdir}/cirros-0.3.2-x86_64-disk.img http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img fi openstack image create --container-format bare --disk-format qcow2 --public --file ${PHD_VAR_env_configdir}/cirros-0.3.2-x86_64-disk.img cirros openstack image list ================================================ FILE: pcmk/glance.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_hosts_rabbitmq PHD_VAR_osp_configdir PHD_VAR_deployment ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-glance openstack-utils python-openstackclient # Configure the API service openstack-config --set /etc/glance/glance-api.conf database connection mysql://glance:glancetest@vip-db/glance openstack-config --set /etc/glance/glance-api.conf database max_retries -1 openstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone openstack-config --set /etc/glance/glance-api.conf keystone_authtoken identity_uri http://vip-keystone:35357/ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_uri http://vip-keystone:5000/ openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_user glance openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_password glancetest openstack-config --set /etc/glance/glance-api.conf DEFAULT notification_driver messaging openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host vip-glance openstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') # Configure the registry service openstack-config --set /etc/glance/glance-registry.conf database connection mysql://glance:glancetest@vip-db/glance openstack-config --set /etc/glance/glance-registry.conf database max_retries -1 openstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken identity_uri http://vip-keystone:35357/ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_uri http://vip-keystone:5000/ openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_user glance openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_password glancetest openstack-config --set /etc/glance/glance-registry.conf DEFAULT notification_driver messaging openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 openstack-config --set /etc/glance/glance-registry.conf DEFAULT registry_host vip-glance openstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') # create the NFS share mountpoint on the nfs server mkdir -p $PHD_VAR_osp_configdir/glance .... target=$PHD_ENV_nodes1 .... # Now have pacemaker mount the NFS share as service. pcs resource create glance-fs Filesystem device="${PHD_VAR_network_internal}.1:$PHD_VAR_osp_configdir/glance" directory="/var/lib/glance" fstype="nfs" options="v3" --clone # wait for glance-fs to be started and running sleep 5 # Make sure it's writable chown glance:nobody /var/lib/glance # Now populate the database su glance -s /bin/sh -c "glance-manage db_sync" pcs resource create openstack-glance-registry systemd:openstack-glance-registry --clone interleave=true pcs resource create openstack-glance-api systemd:openstack-glance-api --clone interleave=true pcs constraint order start openstack-glance-fs-clone then openstack-glance-registry-clone pcs constraint colocation add openstack-glance-registry-clone with openstack-glance-fs-clone pcs constraint order start openstack-glance-registry-clone then openstack-glance-api-clone pcs constraint colocation add openstack-glance-api-clone with openstack-glance-registry-clone if [ $PHD_VAR_deployment = collapsed ]; then pcs constraint order start openstack-keystone-clone then openstack-glance-registry-clone fi .... ================================================ FILE: pcmk/ha-collapsed.variables ================================================ # Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD # Each scenario file will verify that values have been provided for the variables it requires. # Deployment types: collapsed | segregated deployment: collapsed network: clock: clock.redhat.com domain: lab.bos.redhat.com internal: 192.168.124 named: forwarders: 10.16.36.29 10.11.5.19 10.5.30.160 nic: base: 54:52:00 external: enp1s0 eth0 internal: enp2s0 eth1 hosts: gateway: east-01 # These services are not able to live behind a proxy, so we must list the hosts explicitly # In the segregated case, it is the per-service guests we will create # In the collapsed case, it is the members of the single cluster we will create mongodb: rdo7-node1,rdo7-node2,rdo7-node3 memcache: rdo7-node1:11211,rdo7-node2:11211,rdo7-node3:11211 rabbitmq: rdo7-node1,rdo7-node2,rdo7-node3 # We use a centrally defined list so that segregated guests, DNS, and DHCP are all created consistently # Changing this list in any way will probably break the haproxy configuration # - ONLY add entries # - ALWAYS do so to the end of the list # - LOOK for PHD_VAR_components to see places that may be affected components: lb db rabbitmq keystone memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer qpid node # this is RHEL7.1 GA # https://beaker.engineering.redhat.com/distros/ # Use simple search to find release ex: RHEL-7.1 # click on the GA -> and take the ID from Server install beaker: disttree: 69383 rpm: download: download.devel.redhat.com major: 7 minor: 1 #base: rel-eng/latest-RHEL-7/compose/Server/x86_64/os/ #cluster: rel-eng/latest-RHEL-7/compose/Server/x86_64/os/addons/HighAvailability/ base: released/RHEL-7/7.1/Server/x86_64/os/ cluster: released/RHEL-7/7.1/Server/x86_64/os/addons/HighAvailability/ updates: brewroot/repos/rhel-7.1-z-build/latest/x86_64/ osp: configdir: /srv/RDO-7/configs major: 7 # generate with openssl rand -hex 10 secrets: fence_xvm: bcf4e54dcfbfecb9e62c keystone_secret: 114a23ae49996a26d916 swift_prefix: 8f2d8a3e326a078c5edf swift_suffix: 8d33d13b18e3eef6d3ca horizon_secret: 357a57d6a9a2353ccde5 # Dedicate all cores to the sole VM that we'll create per host vm: cpus: 4 ram: 8048 disk: 25G base: rdo7-rhel7-base.img key: AAAAB3NzaC1yc2EAAAADAQABAAABAQDHs2qRMxtqEpr7gJygHAn2rSWKUS/FlJ9oLG7cRtzLyhIl+oSrs30KrdzkgsGTZqSEwfKM8f2LGF08x5HbN2cIDc9YhnwHQNnb8qDIXY2UqzpyLUzckctOMSiRSz/qYxeutDYGg/p1lPzPdWQPympFVIoAzCRDhogX26kXQTpKs7uUzEvZCnnzSn2I9ynchKGP3TlOzTaZHqJM4bj5+KqvUTH2ifvX3EgolP/XtIWjW54zhQnlDuS2UsDd8vvB8ZRrgtaFEXhCSivvazE8zMVAOxCFNYjnh+SvV96VB+hEjqQQeDSdhkgC2huHwsAB3Y9XCkyFe6DEfKuQZwLJjlTZ # I set the password to 'cluster', USE A SAFER ONE env: password: cluster # TODO... password: cluster: foo keystone: bar ================================================ FILE: pcmk/ha-segregated.variables ================================================ # Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD # Each scenario file will verify that values have been provided for the variables it requires. # Deployment types: collapsed | segregated deployment: segregated network: domain: lab.bos.redhat.com internal: 192.168.124 named: forwarders: 10.16.36.29 10.11.5.19 10.5.30.160 nic: base: 54:52:00 external: enp1s0 eth0 internal: enp2s0 eth1 hosts: gateway: east-01 # These services are not able to live behind a proxy, so we must list the hosts explicitly # In the segregated case, it is the per-service guests we will create # In the collapsed case, it is the members of the single cluster we will create mongodb: rhos6-mongodb1,rhos6-mongodb2,rhos6-mongodb3 memcache: rhos6-memcache1:11211,rhos6-memcache2:11211,rhos6-memcache3:11211 rabbitmq: rhos6-rabbitmq1,rhos6-rabbitmq2,rhos6-rabbitmq3 # We use a centrally defined list so that segregated guests, DNS, and DHCP are all created consistently # Changing this list in any way will probably break the haproxy configuration # - ONLY add entries # - ALWAYS do so to the end of the list # - LOOK for PHD_VAR_components to see places that may be affected components: lb db rabbitmq keystone memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer qpid node rpm: download: download.devel.redhat.com rhel: 7.1 osp: 6.0 # Optional # beta: -Beta vm: cpus: 1 ram: 2048 disk: 25G base: rhos6-rhel7-base.img key: AAAAB3NzaC1yc2EAAAADAQABAAABAQDHs2qRMxtqEpr7gJygHAn2rSWKUS/FlJ9oLG7cRtzLyhIl+oSrs30KrdzkgsGTZqSEwfKM8f2LGF08x5HbN2cIDc9YhnwHQNnb8qDIXY2UqzpyLUzckctOMSiRSz/qYxeutDYGg/p1lPzPdWQPympFVIoAzCRDhogX26kXQTpKs7uUzEvZCnnzSn2I9ynchKGP3TlOzTaZHqJM4bj5+KqvUTH2ifvX3EgolP/XtIWjW54zhQnlDuS2UsDd8vvB8ZRrgtaFEXhCSivvazE8zMVAOxCFNYjnh+SvV96VB+hEjqQQeDSdhkgC2huHwsAB3Y9XCkyFe6DEfKuQZwLJjlTZ # I set the password to 'cluster', USE A SAFER ONE env: password: cluster configdir: /srv/RDO-6.0/configs ================================================ FILE: pcmk/hacks.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - installing the cluster software # - enabling the pcs daemon to allow remote management # - setting a password for the hacluster user for use with pcs # - authenticating to pcs on the other hosts with the hacluster user and password # - creating and starting the cluster # - configuring fencing using the multicast addresses specified for fence_virt on the bare metal hosts ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = # hack to set hostnames till we figure out why it doesn't work # from kickstart anymore target=local .... for node in $(echo $PHD_ENV_nodes); do phd_cmd_exec "hostnamectl set-hostname $node" "$node" phd_cmd_exec "hostname" "$node" done .... ================================================ FILE: pcmk/heat-test.sh ================================================ # TEST: # Requires a compute node! . ${PHD_VAR_env_configdir}/keystonerc_admin nova keypair-add --pub_key ~/.ssh/authorized_keys heat-userkey-test cat > /root/ha_test.yaml << EOF heat_template_version: 2013-05-23 description: > HA test. parameters: key_name: type: string description: Name of keypair to assign to servers image: type: string description: Name of image to use for servers flavor: type: string description: Flavor to use for servers private_net_id: type: string description: ID of private network into which servers get deployed private_subnet_id: type: string description: ID of private sub network into which servers get deployed resources: server1: type: OS::Nova::Server properties: name: Server1 image: { get_param: image } flavor: { get_param: flavor } key_name: { get_param: key_name } networks: - port: { get_resource: server1_port } server1_port: type: OS::Neutron::Port properties: network_id: { get_param: private_net_id } fixed_ips: - subnet_id: { get_param: private_subnet_id } outputs: server1_private_ip: description: IP address of server1 in private network value: { get_attr: [ server1, first_address ] } EOF privatenetid=$(neutron net-list |grep internal_lan | awk '{print $2}') privatesubnetid=$(neutron subnet-list |grep internal_subnet|awk '{print $2}') heat stack-create testtest --template-file=/root/ha_test.yaml --parameters="key_name=heat-userkey-test;image=cirros;flavor=m1.large;private_net_id=$privatenetid;private_subnet_id=$privatesubnetid" heat stack-list heat stack-delete testtest ================================================ FILE: pcmk/heat.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 = VARIABLES = PHD_VAR_deployment PHD_VAR_network_hosts_memcache PHD_VAR_network_hosts_rabbitmq ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-heat-engine openstack-heat-api openstack-heat-api-cfn openstack-heat-api-cloudwatch python-heatclient openstack-utils python-glanceclient openstack-config --set /etc/heat/heat.conf database connection mysql://heat:heattest@vip-db/heat openstack-config --set /etc/heat/heat.conf database database max_retries -1 openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_user heat openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_password heattest openstack-config --set /etc/heat/heat.conf keystone_authtoken service_host vip-keystone openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_host vip-keystone openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_uri http://vip-keystone:35357/v2.0 openstack-config --set /etc/heat/heat.conf keystone_authtoken keystone_ec2_uri http://vip-keystone:35357/v2.0 openstack-config --set /etc/heat/heat.conf ec2authtoken auth_uri http://vip-keystone:5000/v2.0 openstack-config --set /etc/heat/heat.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 openstack-config --set /etc/heat/heat.conf heat_api bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/heat/heat.conf heat_api_cfn bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/heat/heat.conf heat_api_cloudwatch bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/heat/heat.conf DEFAULT heat_metadata_server_url vip-heat:8000 openstack-config --set /etc/heat/heat.conf DEFAULT heat_waitcondition_server_url vip-heat:8000/v1/waitcondition openstack-config --set /etc/heat/heat.conf DEFAULT heat_watch_server_url vip-heat:8003 openstack-config --set /etc/heat/heat.conf DEFAULT rpc_backend heat.openstack.common.rpc.impl_kombu openstack-config --set /etc/heat/heat.conf DEFAULT notification_driver heat.openstack.common.notifier.rpc_notifier # disable CWLiteAlarm that is incompatible with A/A openstack-config --set /etc/heat/heat.conf DEFAULT enable_cloud_watch_lite false .... target=$PHD_ENV_nodes1 .... su heat -s /bin/sh -c "heat-manage db_sync" pcs resource create openstack-heat-api systemd:openstack-heat-api --clone interleave=true pcs resource create openstack-heat-api-cfn systemd:openstack-heat-api-cfn --clone interleave=true pcs resource create openstack-heat-api-cloudwatch systemd:openstack-heat-api-cloudwatch --clone interleave=true pcs resource create openstack-heat-engine systemd:openstack-heat-engine --clone interleave=true pcs constraint order start openstack-heat-api-clone then openstack-heat-api-cfn-clone pcs constraint colocation add openstack-heat-api-cfn-clone with openstack-heat-api-clone pcs constraint order start openstack-heat-api-cfn-clone then openstack-heat-api-cloudwatch-clone pcs constraint colocation add openstack-heat-api-cloudwatch-clone with openstack-heat-api-cfn-clone pcs constraint order start openstack-heat-api-cloudwatch-clone then openstack-heat-engine-clone pcs constraint colocation add openstack-heat-engine-clone with openstack-heat-api-cloudwatch-clone if [ $PHD_VAR_deployment = collapsed ]; then pcs constraint order start openstack-ceilometer-notification-clone then openstack-heat-api-clone fi .... ================================================ FILE: pcmk/horizon-test.sh ================================================ It should be possible now to login to http://vip-horizon/dashboard with admin account. Note that it is still not possible to deploy instances since compute nodes are not attached. ================================================ FILE: pcmk/horizon.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_secrets_horizon_secret PHD_VAR_network_hosts_memcache ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y mod_wsgi httpd mod_ssl openstack-dashboard # NOTE this is a rather scary sed and replace operation to configure horizon # in one shot, scriptable way. # Keypoints: # set ALLOWED_HOSTS to access the web service. # BE AWARE that this command will allow access from everywhere! # connection CACHES to memcacehed # connect with keystone for authentication # fix a LOCAL_PATH to point to the correct location. horizonememcachenodes=$(echo ${PHD_VAR_network_hosts_memcache} | sed -e "s#,#', '#g" -e "s#^#[ '#g" -e "s#\$#', ]#g") sed -i \ -e "s#ALLOWED_HOSTS.*#ALLOWED_HOSTS = ['*',]#g" \ -e "s#^CACHES#SESSION_ENGINE = 'django.contrib.sessions.backends.cache'\nCACHES#g#" \ -e "s#locmem.LocMemCache'#memcached.MemcachedCache',\n\t'LOCATION' : $horizonememcachenodes#g" \ -e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = "vip-keystone"#g' \ -e "s#^LOCAL_PATH.*#LOCAL_PATH = '/var/lib/openstack-dashboard'#g" \ -e "s#SECRET_KEY.*#SECRET_KEY = '${PHD_VAR_secrets_horizon_secret}'#g#" \ /etc/openstack-dashboard/local_settings # workaround buggy packages echo "COMPRESS_OFFLINE = True" >> /etc/openstack-dashboard/local_settings python /usr/share/openstack-dashboard/manage.py compress # NOTE: fix apache config to listen only on a given interface (internal) sed -i -e 's/^Listen.*/Listen '$(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g')':80/g' /etc/httpd/conf/httpd.conf # NOTE: enable server-status. this is required by pacemaker to verify apache is # responding. Only allow from localhost. cat > /etc/httpd/conf.d/server-status.conf << EOF SetHandler server-status Order deny,allow Deny from all Allow from localhost EOF .... target=$PHD_ENV_nodes1 .... pcs resource create httpd apache --clone interleave=true .... ================================================ FILE: pcmk/keystone-test.sh ================================================ # TEST (might require logout/login to reset the environmet that was set before # during initial bootstrap) unset SERVICE_TOKEN unset SERVICE_ENDPOINT . ${PHD_VAR_env_configdir}/keystonerc_user openstack user show demo . ${PHD_VAR_env_configdir}/keystonerc_admin openstack user list ================================================ FILE: pcmk/keystone.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_deployment PHD_VAR_osp_configdir PHD_VAR_secrets_keystone_secret PHD_VAR_network_internal PHD_VAR_network_hosts_rabbitmq ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-keystone openstack-utils python-openstackclient mkdir -p ${PHD_VAR_osp_configdir} openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token ${PHD_VAR_secrets_keystone_secret} openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 # Define the API endpoints. Be careful with replacing vip-keystone and shell escapes. openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_endpoint 'http://vip-keystone:%(admin_port)s/' openstack-config --set /etc/keystone/keystone.conf DEFAULT public_endpoint 'http://vip-keystone:%(public_port)s/' # Configure access to galera. Note that several entries in here are dependent on # what has been configured before. 'keystone' user, 'keystonetest' password, # vip-db. openstack-config --set /etc/keystone/keystone.conf database connection mysql://keystone:keystonetest@vip-db/keystone # Mare sure to retry connection to the DB if the DB is not available immediately at # service startup. openstack-config --set /etc/keystone/keystone.conf database max_retries -1 # Make sure the API service is listening on the internal IP addresses only. # Once again those shell expansions only work for my specific environment. # phase3: those are obsoleted by mod_wsgi and apache openstack-config --set /etc/keystone/keystone.conf eventlet_server public_bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/keystone/keystone.conf eventlet_server admin_bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') # workaround for buggy packaging (ayoung is informed) openstack-config --set /etc/keystone/keystone.conf token driver keystone.token.persistence.backends.sql.Token if [ ! -e ${PHD_VAR_osp_configdir}/keystone_ssl.tar ]; then keystone-manage pki_setup --keystone-user keystone --keystone-group keystone cd /etc/keystone/ssl tar cvp -f ${PHD_VAR_osp_configdir}/keystone_ssl.tar * fi mkdir -p /etc/keystone/ssl cd /etc/keystone/ssl tar xvp -f ${PHD_VAR_osp_configdir}/keystone_ssl.tar chown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/ .... target=$PHD_ENV_nodes1 .... su keystone -s /bin/sh -c "keystone-manage -v -d db_sync" pcs resource create openstack-keystone systemd:openstack-keystone --clone interleave=true if [ $PHD_VAR_deployment = collapsed ]; then # In a collapsed environment, we can instruct the cluster to start # things in a particular order and require services to be active # on the same hosts. We do this with constraints. pcs constraint order start haproxy-clone then openstack-keystone-clone pcs constraint order promote galera-master then openstack-keystone-clone pcs constraint order start rabbitmq-clone then openstack-keystone-clone pcs constraint order start memcached-clone then openstack-keystone-clone fi export OS_TOKEN=${PHD_VAR_secrets_keystone_secret} export OS_URL="http://vip-keystone:35357/v2.0" export OS_REGION_NAME=regionOne while ! openstack service list; do echo "Waiting for keystone to be active" sleep 1 done openstack service create \ --name=keystone \ --description="Keystone Identity Service" \ identity openstack endpoint create \ --publicurl 'http://vip-keystone:5000/v2.0' \ --adminurl 'http://vip-keystone:35357/v2.0' \ --internalurl 'http://vip-keystone:5000/v2.0' \ --region regionOne \ keystone openstack user create --password keystonetest admin openstack role create admin openstack project create admin openstack role add --project admin --user admin admin # Save admin credential in a file. This will be useful many times over the how-to! cat > ${PHD_VAR_osp_configdir}/keystonerc_admin << EOF export OS_USERNAME=admin export OS_TENANT_NAME=admin export OS_PROJECT_NAME=admin export OS_REGION_NAME=regionOne export OS_PASSWORD=keystonetest export OS_AUTH_URL=http://vip-keystone:35357/v2.0/ export PS1='[\u@\h \W(keystone_admin)]\$ ' EOF openstack user create --password redhat demo openstack role create _member_ openstack project create demo openstack role add --project demo --user demo _member_ # Save user credential in a file for testing purposes. cat > ${PHD_VAR_osp_configdir}/keystonerc_user << EOF export OS_USERNAME=demo export OS_TENANT_NAME=demo export OS_PROJECT_NAME=demo export OS_REGION_NAME=regionOne export OS_PASSWORD=redhat export OS_AUTH_URL=http://vip-keystone:5000/v2.0/ export PS1='[\u@\h \W(keystone_user)]\$ ' EOF # create service tenant/project openstack project create --description "Services Tenant" services # glance openstack user create --password glancetest glance openstack role add --project services --user glance admin openstack service create --name=glance --description="Glance Image Service" image openstack endpoint create \ --publicurl "http://vip-glance:9292" \ --adminurl "http://vip-glance:9292" \ --internalurl "http://vip-glance:9292" \ --region regionOne \ glance # cinder openstack user create --password cindertest cinder openstack role add --project services --user cinder admin openstack service create --name=cinder --description="Cinder Volume Service" volume openstack endpoint create \ --publicurl "http://vip-cinder:8776/v1/\$(tenant_id)s" \ --adminurl "http://vip-cinder:8776/v1/\$(tenant_id)s" \ --internalurl "http://vip-cinder:8776/v1/\$(tenant_id)s" \ --region regionOne \ cinder openstack service create --name=cinderv2 --description="OpenStack Block Storage" volumev2 openstack endpoint create \ --publicurl "http://vip-cinder:8776/v2/\$(tenant_id)s" \ --adminurl "http://vip-cinder:8776/v2/\$(tenant_id)s" \ --internalurl "http://vip-cinder:8776/v2/\$(tenant_id)s" \ --region regionOne \ cinderv2 # swift openstack user create --password swifttest swift openstack role add --project services --user swift admin openstack service create --name=swift --description="Swift Storage Service" object-store openstack endpoint create \ --publicurl "http://vip-swift:8080/v1/AUTH_\$(tenant_id)s" \ --adminurl "http://vip-swift:8080/v1" \ --internalurl "http://vip-swift:8080/v1/AUTH_\$(tenant_id)s" \ --region regionOne \ swift # neutron openstack user create --password neutrontest neutron openstack role add --project services --user neutron admin openstack service create --name=neutron --description="OpenStack Networking Service" network openstack endpoint create \ --publicurl "http://vip-neutron:9696" \ --adminurl "http://vip-neutron:9696" \ --internalurl "http://vip-neutron:9696" \ --region regionOne \ neutron # nova openstack user create --password novatest compute openstack role add --project services --user compute admin openstack service create --name=compute --description="OpenStack Compute Service" compute openstack endpoint create \ --publicurl "http://vip-nova:8774/v2/\$(tenant_id)s" \ --adminurl "http://vip-nova:8774/v2/\$(tenant_id)s" \ --internalurl "http://vip-nova:8774/v2/\$(tenant_id)s" \ --region regionOne \ compute # heat openstack user create --password heattest heat openstack role add --project services --user heat admin openstack service create --name=heat --description="Heat Orchestration Service" orchestration openstack endpoint create \ --publicurl "http://vip-heat:8004/v1/%(tenant_id)s" \ --adminurl "http://vip-heat:8004/v1/%(tenant_id)s" \ --internalurl "http://vip-heat:8004/v1/%(tenant_id)s" \ --region regionOne \ heat openstack service create --name=heat-cfn --description="Heat CloudFormation Service" cloudformation openstack endpoint create \ --publicurl "http://vip-heat:8000/v1" \ --adminurl "http://vip-heat:8000/v1" \ --internalurl "http://vip-heat:8000/v1" \ --region regionOne \ heat-cfn # ceilometer openstack user create --password ceilometertest ceilometer openstack role add --project services --user ceilometer admin openstack role create ResellerAdmin openstack role add --project services --user ceilometer ResellerAdmin openstack service create --name=ceilometer --description="OpenStack Telemetry Service" metering openstack endpoint create \ --publicurl "http://vip-ceilometer:8777" \ --adminurl "http://vip-ceilometer:8777" \ --internalurl "http://vip-ceilometer:8777" \ --region regionOne \ ceilometer .... ================================================ FILE: pcmk/lb.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: # - Tweaking the IP stack to allow nonlocal binding and adjusting keepalive timings # - Configuring haproxy # - Adding the virtual IPs to the cluster # - Putting haproxy under the cluster's control ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_network_internal PHD_VAR_deployment PHD_VAR_components PHD_VAR_osp_major ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y haproxy echo net.ipv4.ip_nonlocal_bind=1 >> /etc/sysctl.d/haproxy.conf echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind # the keepalive settings must be set in *ALL* hosts interacting with rabbitmq. cat >/etc/sysctl.d/tcp_keepalive.conf << EOF net.ipv4.tcp_keepalive_intvl = 1 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_keepalive_time = 5 EOF sysctl net.ipv4.tcp_keepalive_intvl=1 sysctl net.ipv4.tcp_keepalive_probes=5 sysctl net.ipv4.tcp_keepalive_time=5 # HA Proxy defaults cat > /etc/haproxy/haproxy.cfg << EOF global daemon defaults mode tcp maxconn 10000 timeout connect 10s timeout client 1m timeout server 1m timeout check 10s EOF # Special case front-ends cat >> /etc/haproxy/haproxy.cfg << EOF frontend vip-db bind ${PHD_VAR_network_internal}.201:3306 timeout client 90m default_backend db-vms-galera frontend vip-qpid bind ${PHD_VAR_network_internal}.215:5672 timeout client 120s default_backend qpid-vms frontend vip-horizon bind ${PHD_VAR_network_internal}.211:80 timeout client 180s cookie SERVERID insert indirect nocache default_backend horizon-vms frontend vip-ceilometer bind ${PHD_VAR_network_internal}.214:8777 timeout client 90s default_backend ceilometer-vms frontend vip-rabbitmq option clitcpka bind ${PHD_VAR_network_internal}.202:5672 timeout client 900m default_backend rabbitmq-vms EOF # nova-metadata needs "balance roundrobin" for frontend? #db-vms-mariadb:58:3306:90s mappings=" keystone-admin:203:64:35357 keystone-public:203:64:5000 glance-api:205:70:9191 glance-registry:205:70:9292 cinder:206:73:8776 swift:208:79:8080 neutron:209:82:9696 nova-vnc-novncproxy:210:85:6080 nova-vnc-xvpvncproxy:210:85:6081 nova-metadata:210:85:8775 nova-api:210:85:8774 horizon:x:85:80:108s heat-cfn:212:91:8000 heat-cloudw:212:91:8004 heat-srv:212:91:8004 ceilometer:x:97:8777 " for mapping in $mappings; do server=$(echo $mapping | awk -F: '{print $1}' | awk -F- '{print $1}') service=$(echo $mapping | awk -F: '{print $1}') src=$(echo $mapping | awk -F: '{print $2}') target=$(echo $mapping | awk -F: '{print $3}') port=$(echo $mapping | awk -F: '{print $4}') timeout=$(echo $mapping | awk -F: '{print $5}') echo "Creating mapping for ${server} ${service}" if [ ${src} != x ]; then echo "frontend vip-${service}" >> /etc/haproxy/haproxy.cfg echo " bind ${PHD_VAR_network_internal}.${src}:${port}" >> /etc/haproxy/haproxy.cfg echo " default_backend ${service}-vms" >> /etc/haproxy/haproxy.cfg fi echo "backend ${service}-vms" >> /etc/haproxy/haproxy.cfg echo " balance roundrobin" >> /etc/haproxy/haproxy.cfg if [ ! -z $timeout ]; then echo " timeout server ${timeout}" >> /etc/haproxy/haproxy.cfg fi for count in 1 2 3; do echo " server rdo${PHD_VAR_osp_major}-${server}${count} ${PHD_VAR_network_internal}.$(( ${target} + ${count} - 1)):${port} check inter 1s" >> /etc/haproxy/haproxy.cfg done done # Special case back-ends cat >> /etc/haproxy/haproxy.cfg << EOF backend qpid-vms # comment out 'stick-table' and add 'balance roundrobin' for A/A cluster mode in qpid stick-table type ip size 2 stick on dst timeout server 120s server rdo${PHD_VAR_osp_major}-qpid1 ${PHD_VAR_network_internal}.103:5672 check inter 1s server rdo${PHD_VAR_osp_major}-qpid2 ${PHD_VAR_network_internal}.104:5672 check inter 1s server rdo${PHD_VAR_osp_major}-qpid3 ${PHD_VAR_network_internal}.105:5672 check inter 1s backend db-vms-galera option httpchk option tcpka stick-table type ip size 1000 stick on dst timeout server 90m server rdo${PHD_VAR_osp_major}-db1 ${PHD_VAR_network_internal}.58:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions server rdo${PHD_VAR_osp_major}-db2 ${PHD_VAR_network_internal}.59:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions server rdo${PHD_VAR_osp_major}-db3 ${PHD_VAR_network_internal}.60:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions backend rabbitmq-vms option srvtcpka balance roundrobin timeout server 900m server rdo${PHD_VAR_osp_major}-rabbitmq1 ${PHD_VAR_network_internal}.61:5672 check inter 1s server rdo${PHD_VAR_osp_major}-rabbitmq2 ${PHD_VAR_network_internal}.62:5672 check inter 1s server rdo${PHD_VAR_osp_major}-rabbitmq3 ${PHD_VAR_network_internal}.63:5672 check inter 1s EOF if [ $PHD_VAR_deployment = collapsed ]; then # In a collapsed environment everything is installed on rdo${PHD_VAR_osp_major}-node{1,2,3} # So rewite the proxy config to talk to those hosts instead for section in ${PHD_VAR_components}; do sed -i s/rdo${PHD_VAR_osp_major}-${section}1\ ${PHD_VAR_network_internal}.[0-9]*/rdo${PHD_VAR_osp_major}-node1\ ${PHD_VAR_network_internal}\.103/g /etc/haproxy/haproxy.cfg sed -i s/rdo${PHD_VAR_osp_major}-${section}2\ ${PHD_VAR_network_internal}.[0-9]*/rdo${PHD_VAR_osp_major}-node2\ ${PHD_VAR_network_internal}\.104/g /etc/haproxy/haproxy.cfg sed -i s/rdo${PHD_VAR_osp_major}-${section}3\ ${PHD_VAR_network_internal}.[0-9]*/rdo${PHD_VAR_osp_major}-node3\ ${PHD_VAR_network_internal}\.105/g /etc/haproxy/haproxy.cfg done fi .... target=$PHD_ENV_nodes1 .... pcs resource create haproxy systemd:haproxy --clone # The VIPs changed recently #### # db was 192.168.16.200 -> 201 # rabbitmq was 192.168.16.213 -> 202 # qpid was 192.168.16.201 -> 215 # keystone was 192.168.16.202 -> 203 # glance was 192.168.16.203 -> 205 # cinder was 192.168.16.204 -> 206 # swift was 192.168.16.205 -> 208 # neutron was 192.168.16.206 -> 209 # nova was 192.168.16.207 -> 210 # horizon was 192.168.16.208 -> 211 # heat was 192.168.16.209 -> 212 # ceilometer was 192.168.16.211 -> 214 #### offset=200 for section in ${PHD_VAR_components}; do case $section in lb|memcache|swift-brick|mongodb) : No VIP needed for $section ;; *) pcs resource create vip-${section} IPaddr2 ip=${PHD_VAR_network_internal}.${offset} nic=eth1 ;; esac offset=$(( $offset + 1 )) done if [ $PHD_VAR_deployment = collapsed ]; then # In a collapsed environment, we can instruct the cluster to start # things in a particular order and require services to be active # on the same hosts. We do this with constraints. for section in ${PHD_VAR_components}; do case $section in lb|memcache|swift-brick|mongodb) : No VIP constraints needed for $section ;; *) pcs constraint order start vip-${section} then haproxy-clone kind=Optional pcs constraint colocation add vip-${section} with haproxy-clone ;; esac done fi .... ================================================ FILE: pcmk/memcached.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y memcached .... target=$PHD_ENV_nodes1 .... pcs resource create memcached systemd:memcached --clone interleave=true .... ================================================ FILE: pcmk/mongodb.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y mongodb mongodb-server # set binding IP address and replica set # also use smallfiles = true to stall installation while allocating N GB of journals sed -i \ -e 's#.*bind_ip.*#bind_ip = 0.0.0.0#g' \ -e 's/.*replSet.*/replSet = ceilometer/g' \ -e 's/.*smallfiles.*/smallfiles = true/g' \ /etc/mongod.conf # required to bootstrap mongodb systemctl start mongod systemctl stop mongod .... target=$PHD_ENV_nodes1 .... pcs resource create mongod systemd:mongod op start timeout=300s --clone # Setup replica (need to wait for mongodb to settle down first!) sleep 20 # Careful with the node names here, must match FQDN rm -f /root/mongo_replica_setup.js cat > /root/mongo_replica_setup.js << EOF rs.initiate() sleep(10000) EOF for node in $PHD_ENV_nodes; do cat >> /root/mongo_replica_setup.js << EOF rs.add("$node"); EOF done mongo /root/mongo_replica_setup.js rm -f /root/mongo_replica_setup.js .... ================================================ FILE: pcmk/mrg.variables ================================================ # Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD # Each scenario file will verify that values have been provided for the variables it requires. # Deployment types: collapsed | segregated deployment: collapsed network: clock: clock.redhat.com domain: mpc.lab.eng.bos.redhat.com internal: 192.168.16 named: forwarders: 10.16.36.29 10.11.5.19 10.5.30.160 nic: base: 54:52:00 external: em1 eno1 enp1s0 eth0 internal: em2 eno2 enp2s0 eth1 hosts: gateway: mrg-01 # These services are not able to live behind a proxy, so we must list the hosts explicitly # In the segregated case, it is the per-service guests we will create # In the collapsed case, it is the members of the single cluster we will create mongodb: rdo7-node1,rdo7-node2,rdo7-node3 memcache: rdo7-node1:11211,rdo7-node2:11211,rdo7-node3:11211 rabbitmq: rdo7-node1,rdo7-node2,rdo7-node3 # We use a centrally defined list so that segregated guests, DNS, and DHCP are all created consistently # Changing this list in any way will probably break the haproxy configuration # - ONLY add entries # - ALWAYS do so to the end of the list # - LOOK for PHD_VAR_components to see places that may be affected components: lb db rabbitmq keystone memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer qpid node # this is RHEL7.1 GA # https://beaker.engineering.redhat.com/distros/ # Use simple search to find release ex: RHEL-7.1 # click on the GA -> and take the ID from Server install beaker: disttree: 69383 rpm: download: download.devel.redhat.com major: 7 minor: 1 base: released/RHEL-7/7.1/Server/x86_64/os/ cluster: released/RHEL-7/7.1/Server/x86_64/os/addons/HighAvailability/ updates: brewroot/repos/rhel-7.1-z-build/latest/x86_64/ osp: configdir: /srv/RDO-7/configs major: 7 # generate with openssl rand -hex 10 secrets: fence_xvm: bcf4e54dcfbfecb9e62c keystone_secret: 114a23ae49996a26d916 swift_prefix: 8f2d8a3e326a078c5edf swift_suffix: 8d33d13b18e3eef6d3ca horizon_secret: 357a57d6a9a2353ccde5 # Dedicate all cores to the sole VM that we'll create per host vm: cpus: 4 ram: 8048 disk: 25G base: rdo7-rhel7-base.img key: AAAAB3NzaC1yc2EAAAADAQABAAABAQDHs2qRMxtqEpr7gJygHAn2rSWKUS/FlJ9oLG7cRtzLyhIl+oSrs30KrdzkgsGTZqSEwfKM8f2LGF08x5HbN2cIDc9YhnwHQNnb8qDIXY2UqzpyLUzckctOMSiRSz/qYxeutDYGg/p1lPzPdWQPympFVIoAzCRDhogX26kXQTpKs7uUzEvZCnnzSn2I9ynchKGP3TlOzTaZHqJM4bj5+KqvUTH2ifvX3EgolP/XtIWjW54zhQnlDuS2UsDd8vvB8ZRrgtaFEXhCSivvazE8zMVAOxCFNYjnh+SvV96VB+hEjqQQeDSdhkgC2huHwsAB3Y9XCkyFe6DEfKuQZwLJjlTZ # I set the password to 'cluster', USE A SAFER ONE env: password: cluster # TODO... password: cluster: foo keystone: bar ================================================ FILE: pcmk/neutron-agents.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_deployment ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-neutron openstack-neutron-openvswitch openvswitch systemctl enable openvswitch systemctl start openvswitch ovs-vsctl add-br br-int ovs-vsctl add-br br-ex # NOTE: this is the ethernet connected to the external LAN of the controller nodes! ovs-vsctl add-port br-ex eth0 # workaround for keepalived DNS resolution issue within the # ha-routers at config reload dig A $(hostname) | grep -A1 "ANSWER SEC" | tail -n 1 | awk '{print $NF " " $1}' | sed -e 's/.$//g' >>/etc/hosts grep -q $(hostname) /etc/hosts || echo "Failure to setup hostname entry" # openvswitch plugin (used as mechanism within ml2) openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent tunnel_types vxlan openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent vxlan_udp_port 4789 openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs local_ip $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs enable_tunneling True openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs integration_bridge br-int openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_bridge br-tun openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs bridge_mappings physnet1:br-ex openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver # ovs l2 population openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent l2_population False # metadata agent openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_strategy keystone openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_url http://vip-keystone:35357/v2.0 openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_host vip-keystone openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_region regionOne openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_tenant_name services openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_user neutron openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_password neutrontest openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip vip-nova openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_port 8775 openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret metatest openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_workers 4 openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_backlog 2048 # dhcp agent openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dhcp_delete_namespaces False # current deployment has a routing problem via qrouter->192.168.16.1 which is the # system default nameserver, probably the qrouter has no leg on that net, so... to # make that work specify a comma separated # list of DNS servers to be available (Forwarded to the instances) openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_dns_servers 10.35.255.14 # L3 agent openstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver openstack-config --set /etc/neutron/l3_agent.ini DEFAULT handle_internal_only_routers True openstack-config --set /etc/neutron/l3_agent.ini DEFAULT send_arp_for_ha 3 openstack-config --set /etc/neutron/l3_agent.ini DEFAULT router_delete_namespaces False openstack-config --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge br-ex .... target=$PHD_ENV_nodes1 .... # For A/P, set clone-max=1 pcs resource create neutron-scale ocf:neutron:NeutronScale --clone globally-unique=true clone-max=3 interleave=true pcs resource create neutron-ovs-cleanup ocf:neutron:OVSCleanup --clone interleave=true pcs resource create neutron-netns-cleanup ocf:neutron:NetnsCleanup --clone interleave=true pcs resource create neutron-openvswitch-agent systemd:neutron-openvswitch-agent --clone interleave=true pcs resource create neutron-dhcp-agent systemd:neutron-dhcp-agent --clone interleave=true pcs resource create neutron-l3-agent systemd:neutron-l3-agent --clone interleave=true pcs resource create neutron-metadata-agent systemd:neutron-metadata-agent --clone interleave=true pcs constraint order start neutron-scale-clone then neutron-ovs-cleanup-clone pcs constraint colocation add neutron-ovs-cleanup-clone with neutron-scale-clone pcs constraint order start neutron-ovs-cleanup-clone then neutron-netns-cleanup-clone pcs constraint colocation add neutron-netns-cleanup-clone with neutron-ovs-cleanup-clone pcs constraint order start neutron-netns-cleanup-clone then neutron-openvswitch-agent-clone pcs constraint colocation add neutron-openvswitch-agent-clone with neutron-netns-cleanup-clone pcs constraint order start neutron-openvswitch-agent-clone then neutron-dhcp-agent-clone pcs constraint colocation add neutron-dhcp-agent-clone with neutron-openvswitch-agent-clone pcs constraint order start neutron-dhcp-agent-clone then neutron-l3-agent-clone pcs constraint colocation add neutron-l3-agent-clone with neutron-dhcp-agent-clone pcs constraint order start neutron-l3-agent-clone then neutron-metadata-agent-clone pcs constraint colocation add neutron-metadata-agent-clone with neutron-l3-agent-clone if [ $PHD_VAR_deployment = collapsed ]; then pcs constraint order start neutron-server-clone then neutron-scale-clone fi .... ================================================ FILE: pcmk/neutron-server.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_deployment PHD_VAR_osp_configdir PHD_VAR_network_hosts_rabbitmq ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... # NOTE: once again, careful with vip-* / user / password substitutions and local # ip addresses via shell expansion # While neutron can be replaced with nova-network. This how-to does NOT # contain information on how to deploy nova-network in HA fashion. # This version of the how-to uses the ML2 plugin. Other supported # plugins can be used. Please consult the OSP documentation on how # to configure/deploy other plugins. # neutron-server requires neutron-agents and viceversa yum install -y openstack-neutron openstack-neutron-openvswitch openstack-neutron-ml2 openstack-config --set /etc/neutron/neutron.conf DEFAULT bind_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password neutrontest openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://vip-keystone:35357/v2.0/ openstack-config --set /etc/neutron/neutron.conf keystone_authtoken identity_uri http://vip-keystone:5000/ openstack-config --set /etc/neutron/neutron.conf database connection mysql://neutron:neutrontest@vip-db:3306/neutron openstack-config --set /etc/neutron/neutron.conf database max_retries -1 openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier openstack-config --set /etc/neutron/neutron.conf DEFAULT nova_url http://vip-nova:8774/v2 openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes True openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_data_changes True openstack-config --set /etc/neutron/neutron.conf nova auth_url http://vip-keystone:35357/ openstack-config --set /etc/neutron/neutron.conf nova auth_plugin password openstack-config --set /etc/neutron/neutron.conf nova project_domain_id default openstack-config --set /etc/neutron/neutron.conf nova user_domain_id default openstack-config --set /etc/neutron/neutron.conf nova region_name regionOne openstack-config --set /etc/neutron/neutron.conf nova project_name services openstack-config --set /etc/neutron/neutron.conf nova username compute openstack-config --set /etc/neutron/neutron.conf nova password novatest openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin neutron.plugins.ml2.plugin.Ml2Plugin openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins neutron.services.l3_router.l3_router_plugin.L3RouterPlugin openstack-config --set /etc/neutron/neutron.conf DEFAULT router_scheduler_driver neutron.scheduler.l3_agent_scheduler.ChanceScheduler openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini type_drivers local,gre,flat,vxlan,vlan openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers openvswitch openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks "*" openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_gre tunnel_id_ranges 10:10000 openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vni_ranges 10:10000 openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vxlan_group 224.0.0.1 openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_security_group True # is this still required? openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup firewall_driver True ln -sf /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini # There are a number of different approaches to make neutron highly # available, we cover only A/A and A/P here but more details # (internal-only) are available in Bugzilla: # https://bugzilla.redhat.com/show_bug.cgi?id=1170113#c18 # # 1) Fully neutron A/A, considering nodes A,B,C # # all nodes would have l3_ha=True , max_l3_agents_per_router=3, min=2 # A: host=neutron-n-0 B: host=neutron-n-1 C: host=neutron-n-2 # # (this way we cover the upgrade path from OSP5->OSP6, by keeping at least one host withb the old neutron-n-0 ID) # # 3) A/P, with 1 active node # # all nodes would have l3_ha=False # # a) A + B + C have host=neutron-n-0 # b) like case 2, but: # A: host=neutron-n-0 B: (passive not set) C: (passive, not set) # and neutron scale does the host= change during failover. # A/P does NOT require extra settings. Defaults are fine # Fully A/A requires extra neutron-server configuration: openstack-config --set /etc/neutron/neutron.conf DEFAULT l3_ha True openstack-config --set /etc/neutron/neutron.conf DEFAULT max_l3_agents_per_router 0 openstack-config --set /etc/neutron/neutron.conf DEFAULT min_l3_agents_per_router 2 # This value _MUST_ follow the number of nodes in the pacemaker cluster openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 3 .... target=$PHD_ENV_nodes1 .... . ${PHD_VAR_osp_configdir}/keystonerc_admin # required when installing the DB manually neutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade kilo systemctl start neutron-server systemctl stop neutron-server # add the service to pacemaker pcs resource create neutron-server systemd:neutron-server op start timeout=90 --clone interleave=true if [ $PHD_VAR_deployment = collapsed ]; then pcs constraint order start openstack-keystone-clone then neutron-server-clone fi .... ================================================ FILE: pcmk/neutron-test.sh ================================================ # TEST/create your first network # It is not possible to test neutron completely until the full deployment is complete as it is required to run instances to verify network connectivity. . ${PHD_VAR_env_configdir}/keystonerc_admin # WARNING: openstack client is NOT ready to manage neutron! neutron net-create internal_lan neutron subnet-create --ip_version 4 --gateway 192.168.100.1 --name "internal_subnet" internal_lan 192.168.100.0/24 neutron net-create public_lan --router:external neutron subnet-create --gateway 10.16.151.254 --allocation-pool start=10.16.144.76,end=10.16.144.83 --disable-dhcp --name public_subnet public_lan 10.16.144.0/21 neutron router-create router neutron router-gateway-set router public_lan neutron router-interface-add router internal_subnet ================================================ FILE: pcmk/nova-test.sh ================================================ . ${PHD_VAR_env_configdir}/keystonerc_admin nova usage nova usage-list ================================================ FILE: pcmk/nova.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 = VARIABLES = PHD_VAR_deployment PHD_VAR_network_domain PHD_VAR_network_hosts_gateway PHD_VAR_network_hosts_memcache PHD_VAR_network_hosts_rabbitmq PHD_VAR_network_internal ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-nova-console openstack-nova-novncproxy openstack-utils openstack-nova-api openstack-nova-conductor openstack-nova-scheduler python-cinderclient python-memcached openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq} openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues true openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit heartbeat_timeout_threshold 60 openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers ${PHD_VAR_network_hosts_memcache} # Particularly in the collapsed case, we get a lot of conflicts with the haproxy server openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address $(host $(hostname -s) | awk '{print $4}') openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_host $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://${PHD_VAR_network_hosts_gateway}.${PHD_VAR_network_domain}:6080/vnc_auto.html openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone # FIX ME: nova doesn't like hostnames anymore? # openstack-config --set /etc/nova/nova.conf DEFAULT metadata_host vip-nova openstack-config --set /etc/nova/nova.conf DEFAULT metadata_host ${PHD_VAR_network_internal}.210 openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen_port 8775 openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest openstack-config --set /etc/nova/nova.conf glance host vip-glance openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API openstack-config --set /etc/nova/nova.conf neutron url http://vip-neutron:9696/ openstack-config --set /etc/nova/nova.conf neutron admin_tenant_name services openstack-config --set /etc/nova/nova.conf neutron admin_username neutron openstack-config --set /etc/nova/nova.conf neutron admin_password neutrontest openstack-config --set /etc/nova/nova.conf neutron admin_auth_url http://vip-keystone:35357/v2.0 openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver openstack-config --set /etc/nova/nova.conf DEFAULT libvirt_vif_driver nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver # those entries were workaround for https://bugs.launchpad.net/neutron/+bug/1464178 # that breaks neutron and evacuation # openstack-config --set /etc/nova/nova.conf DEFAULT notify_api_faults False # openstack-config --set /etc/nova/nova.conf DEFAULT vif_plugging_is_fatal True # openstack-config --set /etc/nova/nova.conf DEFAULT vif_plugging_timeout 300 openstack-config --set /etc/nova/nova.conf conductor use_local false openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@vip-db/nova openstack-config --set /etc/nova/nova.conf database max_retries -1 # REQUIRED FOR A/A scheduler openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_host_subset_size 30 openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_host vip-keystone openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_tenant_name services openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_user compute openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_password novatest .... target=$PHD_ENV_nodes1 .... su nova -s /bin/sh -c "nova-manage db sync" pcs resource create openstack-nova-consoleauth systemd:openstack-nova-consoleauth --clone interleave=true pcs resource create openstack-nova-novncproxy systemd:openstack-nova-novncproxy --clone interleave=true pcs resource create openstack-nova-api systemd:openstack-nova-api --clone interleave=true pcs resource create openstack-nova-scheduler systemd:openstack-nova-scheduler --clone interleave=true pcs resource create openstack-nova-conductor systemd:openstack-nova-conductor --clone interleave=true pcs constraint order start openstack-nova-consoleauth-clone then openstack-nova-novncproxy-clone pcs constraint colocation add openstack-nova-novncproxy-clone with openstack-nova-consoleauth-clone pcs constraint order start openstack-nova-novncproxy-clone then openstack-nova-api-clone pcs constraint colocation add openstack-nova-api-clone with openstack-nova-novncproxy-clone pcs constraint order start openstack-nova-api-clone then openstack-nova-scheduler-clone pcs constraint colocation add openstack-nova-scheduler-clone with openstack-nova-api-clone pcs constraint order start openstack-nova-scheduler-clone then openstack-nova-conductor-clone pcs constraint colocation add openstack-nova-conductor-clone with openstack-nova-scheduler-clone if [ $PHD_VAR_deployment = collapsed ]; then pcs constraint order start openstack-keystone-clone then openstack-nova-consoleauth-clone fi .... ================================================ FILE: pcmk/nova_client.py ================================================ #!/usr/bin/python -tt import argparse import collections import inspect import logging import sys try: from novaclient import client as nova_client except ImportError: logging.error("nova not found or not accessible") sys.exit(1) # NOTE(sbauza): Necessary as the contrib path is not imported directly try: from novaclient import extension from novaclient.v2.contrib import migrations except ImportError: logging.warning("migrations can't be done since the module is not there") migrations = None def register_named(cls): cls.methods_map = {} cls.methods_pos_map = {} cls.methods_opt_map = {} for methodname in dir(cls): method = getattr(cls, methodname) if hasattr(method, '_named'): cls.methods_map.update({method._named: methodname}) if hasattr(method, '_pos'): cls.methods_pos_map.update({method._named: method._pos}) if hasattr(method, '_opts'): cls.methods_opt_map.update({method._named: method._opts}) return cls def named(method_name, positionals=[], opts_with_val=[], opts_without_val=[]): """Decorator for providing the shell name for the method and the optional arguments. :method_name: Shell name for the method (like "service-enable") :positionals: List of strings corresponding to a positional :opts_with_val: Optionals that accept a value (like --foo bar) :opts_without_val: Optionals that are boolean (like --foo) """ def wrapped(func): func._named = method_name func._pos = positionals func._opts = dict(map(lambda x: (x, True), opts_with_val) + map( lambda x: (x, False), opts_without_val)) return func return wrapped def shell_fields(fields): """Decorator for providing the list of fields to show up in CLI.""" def wrapped(func): func.fields = fields return func return wrapped @register_named class NovaClientWrapper(object): """Wrapper for accessing a subset of novaclient API.""" def __init__(self, version, username, password, tenant_name, auth_url): if migrations: extensions = [extension.Extension('migrations', migrations)] else: extensions = None self.nova = nova_client.Client(version, username, password, tenant_name, auth_url, extensions=extensions) @shell_fields(["Id", "Binary", "Host", "Zone", "Status", "State", "Updated_at"]) @named('service-list', opts_with_val=['host', 'binary']) def service_list(self, host=None, binary=None): services = self.nova.services.list(host=host, binary=binary) return services @shell_fields(['Host', 'Binary', 'Status']) @named('service-enable', positionals=['host', 'binary']) def service_enable(self, host, binary): return self.nova.services.enable(host=host, binary=binary) @shell_fields(['Source Node', 'Dest Node', 'Source Compute', 'Dest Compute', 'Dest Host', 'Status', 'Instance UUID', 'Old Flavor', 'New Flavor', 'Created At', 'Updated At']) @named('migration-list', opts_with_val=['host', 'status', 'cell_name']) def migration_list(self, host=None, status=None, cell_name=None): if hasattr(self.nova, 'migrations'): return self.nova.migrations.list(host=host, status=status, cell_name=cell_name) # NOTE(sbauza); We mimic the host-servers-migrate module def _server_migrate(self, server): success = True error_message = "" try: self.nova.servers.migrate(server=server['uuid']) except Exception as e: success = False error_message = "Error while migrating instance: %s" % e response = collections.namedtuple( 'HostServersMigrateResponse', ['server_uuid', 'migration_accepted', 'error_message']) response._make([server['uuid'], success, error_message]) return response @shell_fields(["Server UUID", "Migration Accepted", "Error Message"]) @named('host-servers-migrate', positionals=['host']) def host_servers_migrate(self, host): hypervisors = self.nova.hypervisors.search(host, servers=True) response = [] for hyper in hypervisors: if hasattr(hyper, 'servers'): for server in hyper.servers: response.append(self._server_migrate(server)) return response def handle_method_and_args(self, args_list): (method, extra) = (None, None) kwargs = {} for meth in self.methods_map: try: pos = args_list.index(meth) except ValueError: # Method not found continue method_name = args_list[pos] method = getattr(self, self.methods_map[method_name]) extra = args_list[:pos] if pos > 0 else None args = args_list[pos+1:] spec = inspect.getargspec(method) for arg in args: opt = arg.rsplit("-")[-1] if opt in spec.args: if opt in self.methods_opt_map[method_name]: if self.methods_opt_map[method_name][opt] is True: # This is an opt followed by its value try: val = args[args.index(arg)+1] except IndexError: # The opt was awaiting a value, invalidating it args.remove(arg) continue kwargs[opt] = val args.remove(arg) args.remove(val) else: # This is an opt True or False kwargs[opt] = True args.remove(arg) # Let's zip the remainings arguments with the awaiting positionals positionals = dict(zip(self.methods_pos_map[method_name], args)) kwargs.update(positionals) break return (extra, method, kwargs) def print_list(objs, fields): # find max column width columnWidth = 0 for obj in objs: for field in fields: field_name = field.lower().replace(' ', '_') width = len(str(getattr(obj, field_name, ''))) if width > columnWidth: columnWidth = width if len(field) > columnWidth: columnWidth = len(field) outputStr = '+' + ('-'*(columnWidth + 2) + '+')*len(objs) + '\n' outputStr += '| ' + " | ".join([field.ljust(columnWidth) for field in fields]) + " |\n" outputStr += '+' + ('-'*(columnWidth + 2) + '+')*len(objs) + '\n' for obj in objs: rowList = [] for field in fields: field_name = field.lower().replace(' ', '_') rowList.append(str(getattr(obj, field_name, '') ).ljust(columnWidth)) outputStr += '| ' + ' | '.join(rowList) + " |\n" outputStr += '+' + ('-'*(columnWidth + 2) + '+')*len(objs) + '\n' return outputStr def main(): logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stderr)) parser = argparse.ArgumentParser() parser.add_argument("--os-auth-url", required=True, dest='auth_url') parser.add_argument("--os-username", required=True, dest='username') parser.add_argument("--os-password", required=True, dest='password') parser.add_argument("--os-tenant-name", required=True, dest='tenant_name') parser.add_argument("remainder", nargs=argparse.REMAINDER, help="nova command followed by its args") args = parser.parse_args() method_and_args = args.remainder nova = NovaClientWrapper('2', args.username, args.password, args.tenant_name, args.auth_url) (extra, method, kwargs) = nova.handle_method_and_args(method_and_args) if method is None: logging.error("Method not mapped in %s", method_and_args) return 1 result = method(**kwargs) print_list([result] if not isinstance(result, list) else result, method.fields) if __name__ == "__main__": sys.exit(main()) ================================================ FILE: pcmk/rabbitmq-test.sh ================================================ rabbitmqctl cluster_status rabbitmqctl list_policies ================================================ FILE: pcmk/rabbitmq.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_deployment PHD_VAR_osp_configdir PHD_VAR_network_domain PHD_VAR_network_internal ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y rabbitmq-server # NOTE: we need to bind the service to the internal IP address cat > /etc/rabbitmq/rabbitmq-env.conf << EOF NODE_IP_ADDRESS=$(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') EOF # required to generate the cookies systemctl start rabbitmq-server systemctl stop rabbitmq-server mkdir -p $PHD_VAR_osp_configdir if [ ! -e $PHD_VAR_osp_configdir/rabbitmq_erlang_cookie ]; then cp /var/lib/rabbitmq/.erlang.cookie $PHD_VAR_osp_configdir/rabbitmq_erlang_cookie fi # the cookie has to be the same across all nodes. Copy around as preferred, I am # using my NFS commodity storage. Also check for file permission/ownership. I # workaround that step by using 'cat' vs cp. cat $PHD_VAR_osp_configdir/rabbitmq_erlang_cookie > /var/lib/rabbitmq/.erlang.cookie .... target=$PHD_ENV_nodes1 .... pcs resource create rabbitmq rabbitmq-cluster set_policy='ha-all ^(?!amq\.).* {"ha-mode":"all"}' meta notify=true --clone ordered=true interleave=true .... ================================================ FILE: pcmk/swift-aco.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 3 = VARIABLES = PHD_VAR_deployment PHD_VAR_secrets_swift_prefix PHD_VAR_secrets_swift_suffix ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-swift-object openstack-swift-container openstack-swift-account openstack-utils rsync xfsprogs # If you have a dedicated device, format it here and have the cluster mount it as per 'swift-fs' below # We don't, but swift wants a different partition than /, so we'll create a loopback file and mount that for 'swift-fs' mkdir -p /local/swiftstorage/target truncate --size=1G /local/swift.img losetup /dev/loop0 /local/swift.img mkfs.xfs /dev/loop0 mount /dev/loop0 /local/swiftstorage/target chown -R swift:swift /local umount /dev/loop0 # Some extra magic to set up the loopback device after a reboot echo "losetup /dev/loop0 /local/swift.img" >> /etc/rc.d/rc.local chmod a+x /etc/rc.d/rc.local openstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_prefix ${PHD_VAR_secrets_swift_prefix} openstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_suffix ${PHD_VAR_secrets_swift_suffix} openstack-config --set /etc/swift/swift.conf filter:ceilometer use egg:ceilometer#swift openstack-config --set /etc/swift/swift.conf pipeline:main pipeline "healthcheck cache authtoken keystoneauth proxy-server ceilometer" openstack-config --set /etc/swift/object-server.conf DEFAULT bind_ip 0.0.0.0 openstack-config --set /etc/swift/object-server.conf DEFAULT devices /local/swiftstorage openstack-config --set /etc/swift/object-server.conf DEFAULT mount_check false openstack-config --set /etc/swift/account-server.conf DEFAULT bind_ip 0.0.0.0 openstack-config --set /etc/swift/account-server.conf DEFAULT devices /local/swiftstorage openstack-config --set /etc/swift/account-server.conf DEFAULT mount_check false openstack-config --set /etc/swift/container-server.conf DEFAULT bind_ip 0.0.0.0 openstack-config --set /etc/swift/container-server.conf DEFAULT devices /local/swiftstorage openstack-config --set /etc/swift/container-server.conf DEFAULT mount_check false openstack-config --set /etc/swift/object-server.conf DEFAULT mount_check false openstack-config --set /etc/swift/account-server.conf DEFAULT mount_check false openstack-config --set /etc/swift/container-server.conf DEFAULT mount_check false chown -R root:swift /etc/swift .... target=all .... if ! pcs resource show swift-fs > /dev/null 2>&1; then # We must be either the first node to run, or configuring # single-node clusters for a segregated deployment pcs resource create swift-fs Filesystem device="/dev/loop0" directory="/local/swiftstorage/target" fstype="xfs" force_clones="yes" --clone interleave=true pcs resource create swift-account systemd:openstack-swift-account --clone interleave=true pcs constraint colocation add swift-account-clone with swift-fs-clone pcs constraint order start swift-fs-clone then swift-account-clone pcs resource create swift-container systemd:openstack-swift-container --clone interleave=true pcs constraint colocation add swift-container-clone with swift-account-clone pcs constraint order start swift-account-clone then swift-container-clone pcs resource create swift-object systemd:openstack-swift-object --clone interleave=true pcs constraint colocation add swift-object-clone with swift-container-clone pcs constraint order start swift-container-clone then swift-object-clone if [ $PHD_VAR_deployment = collapsed ]; then pcs constraint order start openstack-keystone-clone then swift-account-clone fi fi .... ================================================ FILE: pcmk/swift-test.sh ================================================ . ${PHD_VAR_env_configdir}/keystonerc_admin openstack container list openstack container create test openstack container list openstack object list test truncate --size=1M /tmp/foobar openstack object create test /tmp/foobar openstack object list test openstack object delete test /tmp/foobar openstack object list test openstack container delete test openstack container list ================================================ FILE: pcmk/swift.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed at this step include: ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_deployment PHD_VAR_network_internal PHD_VAR_network_hosts_memcache PHD_VAR_osp_configdir PHD_VAR_osp_major PHD_VAR_secrets_swift_prefix PHD_VAR_secrets_swift_suffix ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 1 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... yum install -y openstack-swift-proxy openstack-utils python-swiftclient python-openstackclient .... target=$PHD_ENV_nodes1 .... # NOTE: you MUST refer to the swift-ring-builder documentation in order to # configure prope data redundancy and set those values properly. # This is just a generic example that will store 3 copies of the same data for # proof-of-concept purposes. swift-ring-builder /etc/swift/object.builder create 16 3 24 swift-ring-builder /etc/swift/container.builder create 16 3 24 swift-ring-builder /etc/swift/account.builder create 16 3 24 # .76,.77 and .78 are the addresses of rdo${PHD_VAR_osp_major}-swift-brick{1,2,3} # 'target' here comes from the swift-fs resource created on the proxy # pcs resource create swift-fs Filesystem device="/local/swiftsource" directory="/local/swiftstorage/target" fstype="none" options="bind" if [ $PHD_VAR_deployment = collapsed ]; then baseip=102 else baseip=75 fi for i in 1 2 3; do target=$((baseip + $i)) swift-ring-builder /etc/swift/account.builder add z$i-${PHD_VAR_network_internal}.${target}:6202/target 10 swift-ring-builder /etc/swift/container.builder add z$i-${PHD_VAR_network_internal}.${target}:6201/target 10 swift-ring-builder /etc/swift/object.builder add z$i-${PHD_VAR_network_internal}.${target}:6200/target 10 done swift-ring-builder /etc/swift/account.builder rebalance swift-ring-builder /etc/swift/container.builder rebalance swift-ring-builder /etc/swift/object.builder rebalance mkdir -p $PHD_VAR_osp_configdir/swift cp /etc/swift/*.builder /etc/swift/*.gz $PHD_VAR_osp_configdir/swift/ .... target=all .... openstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_prefix ${PHD_VAR_secrets_swift_prefix} openstack-config --set /etc/swift/swift.conf swift-hash swift_hash_path_suffix ${PHD_VAR_secrets_swift_suffix} openstack-config --set /etc/swift/swift.conf filter:ceilometer use egg:ceilometer#swift openstack-config --set /etc/swift/swift.conf pipeline:main pipeline "healthcheck cache authtoken keystoneauth proxy-server ceilometer" openstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_tenant_name services openstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_user swift openstack-config --set /etc/swift/proxy-server.conf filter:authtoken admin_password swifttest openstack-config --set /etc/swift/proxy-server.conf filter:authtoken identity_uri http://vip-keystone:35357/ openstack-config --set /etc/swift/proxy-server.conf filter:authtoken auth_uri http://vip-keystone:5000/ openstack-config --set /etc/swift/proxy-server.conf DEFAULT bind_ip $(ip addr show dev eth1 scope global | grep dynamic| sed -e 's#.*inet ##g' -e 's#/.*##g') openstack-config --set /etc/swift/object-expirer.conf object-expirer concurrency 100 openstack-config --set /etc/swift/proxy-server.conf filter:cache memcache_servers ${PHD_VAR_network_hosts_memcache} openstack-config --set /etc/swift/object-expirer.conf filter:cache memcache_servers ${PHD_VAR_network_hosts_memcache} cp $PHD_VAR_osp_configdir/swift/*.builder $PHD_VAR_osp_configdir/swift/*.gz /etc/swift/ chown -R root:swift /etc/swift .... target=$PHD_ENV_nodes1 .... pcs resource create swift-proxy systemd:openstack-swift-proxy --clone interleave=true pcs resource create swift-object-expirer systemd:openstack-swift-object-expirer pcs constraint order start swift-proxy-clone then swift-object-expirer if [ $PHD_VAR_deployment = collapsed ]; then pcs constraint order start swift-account-clone then swift-proxy-clone fi .... ================================================ FILE: pcmk/virt-hosts.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # Tasks to be performed for the hosts include: # - Install fence_virt so that the guests can be reset by other members of the virtualized clusters # - Turning off auto-generation of resolv.conf so we can point to our gateway DNS server # - Obtain the golden image from the first host # - Clone the golden image for each service, inserting MAC addresses using the same rules as in the # # Tasks to be performed when creating the image include: # - Installing CentOS 7 # - Pointing to the OpenStack, HA-Addon and Updates repositories # - Ensuring cloned guests can obtain a DHCP lease # - Ensuring consistent network interface names # - Adding SSH keys if necessary # - Configuring guests to obtain their hostname from DHCP ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_VAR_deployment PHD_VAR_network_internal PHD_VAR_network_nic_base PHD_VAR_rpm_download PHD_VAR_secrets_fence_xvm PHD_VAR_osp_configdir PHD_VAR_osp_major PHD_VAR_rpm_major PHD_VAR_rpm_minor PHD_VAR_vm_base PHD_VAR_vm_cpus PHD_VAR_vm_disk PHD_VAR_vm_key PHD_VAR_vm_ram PHD_VAR_components PHD_VAR_rpm_base PHD_VAR_rpm_cluster PHD_VAR_rpm_download PHD_VAR_rpm_updates ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 4 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... # Not required on the gateway, but wont hurt either yum install -y fence-virtd fence-virtd-multicast fence-virtd-libvirt lastoct="$(hostname -s | sed -e 's#^[a-z]*-##g' -e 's#^0*##g')" cat > /etc/fence_virt.conf << EOF fence_virtd { listener = "multicast"; backend = "libvirt"; } listeners { multicast { # key_file = "/etc/cluster/fence_xvm.key"; address = "225.0.0.$lastoct"; # Needed on Fedora systems interface = "vmnet0"; } } backends { libvirt { uri = "qemu:///system"; } } EOF mkdir -p /etc/cluster/ echo ${PHD_VAR_secrets_fence_xvm} > /etc/cluster/fence_xvm.key systemctl enable fence_virtd systemctl start fence_virtd .... target=$PHD_ENV_nodes1 .... if [ -e ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} ]; then : skip this step if an image already exists exit 0 fi distro=${PHD_VAR_rpm_major}.${PHD_VAR_rpm_minor} # headless - do not bork if we specify --noautoconsole # reboot # Do not specify --hostname= for network # If prevents assinging by DHCP targettz="$(timedatectl | grep Timezone | awk '{print $2}')" if [ -z "$targettz" ]; then targettz=Australia/Melbourne fi cat << EOF > virt-base.ks install text reboot rootpw redhat lang en_US.UTF-8 keyboard us network --bootproto dhcp firewall --enabled --ssh selinux --permissive timezone --utc $targettz bootloader --location=mbr --append="console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH serial text headless" zerombr clearpart --all --initlabel autopart --type=lvm skipx %packages @core grep gawk bind-utils vi net-tools tcpdump wget sos nfs-utils ntp ntpdate %end %post # ethX will do nicely thankyou rm -f /lib/udev/rules.d/75-persistent-net-generator.rules rm -f /etc/udev/rules.d/70-persistent-net.rules # Need PEERDNS=no on the external interface to make sure all lookups # go to our gateway (otherwise setting setting hostname via dhcp wont # work) # # MUST add 'DEVICE=ethX' when deleting 'HWADDR', otherwise you end up # with crazy things like: # # [root@localhost ~]# ifdown eth0 # Device 'eth1' successfully disconnected. # # Let eth0 come up, but don't let it connect to avoid conflicting dhcp # and naming issues # # /usr/share/doc/initscripts/sysconfig.txt cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOT NAME="eth0" DEVICE="eth0" ONBOOT=yes BOOTPROTO=none TYPE=Ethernet IPV4_FAILURE_FATAL=no EOT cat > /etc/sysconfig/network-scripts/ifcfg-eth1 << EOT NAME="eth1" DEVICE="eth1" ONBOOT=yes BOOTPROTO=dhcp TYPE=Ethernet EOT mkdir /root/.ssh cat <<__EOT__ >> /root/.ssh/authorized_keys ssh-rsa ${PHD_VAR_vm_key} admin __EOT__ chmod 600 /root/.ssh/authorized_keys # Prevent DHCPNAK by removing stale leases # Occurs if we clone the guest and change NICs rm -f /var/lib/NetworkManager/dhclient-*.lease # If hostname is anything other than 'localhost.localdomain', then # NetworkManager wont bother looking up DNS/DHCP for the host name # when the node boots: # # May 5 11:04:15 localhost NetworkManager[676]: Setting system hostname to 'rdo7-node1.vmnet.mpc.lab.eng.bos.redhat.com' (from address lookup) hostnamectl set-hostname localhost.localdomain cat > /etc/yum.repos.d/redhat-${distro}.repo << EOT [redhat-${distro}] name=redhat-${distro} baseurl=http://${PHD_VAR_rpm_download}/${PHD_VAR_rpm_base} gpgcheck=0 enabled=1 [redhat-ha] name=redhat-${distro}-ha baseurl=http://${PHD_VAR_rpm_download}/${PHD_VAR_rpm_cluster} gpgcheck=0 enabled=1 [redhat-z] name=redhat-${distro}-z baseurl=http://${PHD_VAR_rpm_download}/${PHD_VAR_rpm_updates} gpgcheck=0 enabled=1 EOT %end EOF yum install -y virt-install virsh destroy rdo${PHD_VAR_osp_major}-rhel${PHD_VAR_rpm_major}-base &>/dev/null || true virsh undefine rdo${PHD_VAR_osp_major}-rhel${PHD_VAR_rpm_major}-base &>/dev/null || true #Create the qcow2 disk image with preallocation and 'facllocate'(which pre-allocates all the blocks to a file) it for max. performance echo "Creating qcow2 disk image.." rm -f ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} mkdir -p ${PHD_VAR_osp_configdir} qemu-img create -f qcow2 -o preallocation=metadata ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} ${PHD_VAR_vm_disk} ls -lash ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} #fallocate -l $(ls -al ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} | awk '{print $5}') ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} #ls -lash ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} # Turn off install logs opts="--noautoconsole" opts="" # --console pty,target_type=virtio #Create the regular-guest virt-install --connect=qemu:///system \ --network=bridge:ext0,mac=${PHD_VAR_network_nic_base}:00:00:02 \ --initrd-inject=./virt-base.ks \ --extra-args="ks=file:/virt-base.ks console=tty0 console=ttyS0,115200 serial rd_NO_PLYMOUTH" \ --name=rdo${PHD_VAR_osp_major}-rhel${PHD_VAR_rpm_major}-base \ --disk path=${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base},format=qcow2,cache=none \ --ram ${PHD_VAR_vm_ram} \ --vcpus=${PHD_VAR_vm_cpus} \ --check-cpu \ --accelerate \ --os-type linux \ --os-variant rhel${PHD_VAR_rpm_major} \ --cpuset auto \ --hvm \ --location=http://${PHD_VAR_rpm_download}/${PHD_VAR_rpm_base} \ --nographics $opts # Needs 15-20 minutes apparently sleep 1200 .... #target=$PHD_ENV_nodes1 #.... #virsh define /srv/vms/rhos4-base.xml #virsh define /srv/rdo${PHD_VAR_osp_major}-rhel7-vms/rdo${PHD_VAR_osp_major}-rhel7-base.xml #.... target=all .... which rsync >/dev/null 2>&1 || yum install -y rsync if [ $(hostname -s | awk -F. '{print $1}') = $(echo $PHD_ENV_nodes1 | awk -F. '{print $1}') ]; then : Everyone except the first node needs to run the commands below exit 0 fi mkdir -p /localvms # cp /srv/vms/rhos4-base.img /localvms/ if [ ! -e /localvms/${PHD_VAR_vm_base} ]; then # This takes a loooong time, skip it if we already have an image rsync -avz --progress ${PHD_VAR_osp_configdir}/${PHD_VAR_vm_base} /localvms/ fi # The internal network must go first so that hostname can be set by # DHCP and it will generally be the preferred network # # Random factoid, setting ONBOOT=no for eth0 will prevent eth1 from # being started, even if eth1 has ONBOOT=yes cat<<-EOF > /localvms/template.xml VM_NAME ${PHD_VAR_vm_ram}000 ${PHD_VAR_vm_ram}000 ${PHD_VAR_vm_cpus} hvm destroy restart restart /usr/libexec/qemu-kvm EOF sequence=16 lastoct="$(hostname -s | sed -e 's#^[a-z]*-##g' -e 's#^0*##g')" offset="$(echo ${PHD_ENV_nodes1} | sed -e 's#\..*##g' -e 's#^[a-z]*-##g' -e 's#^0*##g')" for section in ${PHD_VAR_components}; do cd /localvms/ target=rdo${PHD_VAR_osp_major}-${section}$(( ${lastoct} - ${offset} )) octet=${sequence} sequence=$((sequence + 1)) virsh destroy $target > /dev/null 2>&1 || true virsh undefine $target --snapshots-metadata > /dev/null 2>&1 || true if [ $PHD_VAR_deployment = collapsed ]; then case $section in node) ;; *) # We don't need any of the other instances # Its good to get rid of any old copies that might exist though continue;; esac fi cp template.xml ${target}.xml sed -i.sed s#VM_NAME#${target}#g ${target}.xml sed -i.sed s#EXTERNAL_MAC#${PHD_VAR_network_nic_base}:0${lastoct}:00:${octet}#g ${target}.xml sed -i.sed s#INTERNAL_MAC#${PHD_VAR_network_nic_base}:0${lastoct}:01:${octet}#g ${target}.xml sed -i.sed s:source\ file.*\/:source\ file=\'/localvms/${target}.cow\'\/:g ${target}.xml diff -u template.xml ${target}.xml || true rm -f /localvms/${target}.cow /localvms/${target}-*.cow qemu-img create -b /localvms/${PHD_VAR_vm_base} -f qcow2 /localvms/${target}.cow virsh define ${target}.xml virsh start ${target} rm -f ${target}.xml.sed ${target}.xml done .... ================================================ FILE: pcmk/vmsnap-rollback.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # We start with 3 (or more, up to 16) nodes running a minimal CentOS 6 # # Tasks to be performed include: # - setting up the required repositories from which to download Openstack and the HA-Addon # - disabling firewalls and SElinux. This is a necessary evil until the proper policies can be written. # - creating network bridges for use by VMs hosting OpenStack services # - normalizing network interface names # - fixing multicast # - removing /home and making the root partition as large as possible to maximumize the amount of space available to openstack ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_ENV_snapshot_name PHD_VAR_osp_configdir ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 3 ###################### # Deployment Scripts # ###################### = SCRIPTS = target=all .... # get list of all VMs configured on a given host vmlist=$(virsh list --all --name) for vm in $vmlist; do # if the requested snapshot doesn't exist, don't continue with # disruptive actions if [ ! -f /localvms/${vm}-${PHD_ENV_snapshot_name}.cow ]; then echo "ERROR: Snapshot ${PHD_ENV_snapshot_name} does not exists" exit 1 fi current=$(virsh snapshot-current --name $vm) if [ "$current" = ${PHD_ENV_snapshot_name} ]; then echo "ERROR: We are already at snapshot: ${PHD_ENV_snapshot_name}" exit 1 fi # make sure that the we are rolling back to a snap # that is parent of the current snaplist=$(virsh snapshot-list --tree $vm | sed -e 's#.*|.*##g' -e 's#.*+- ##g') for snap in $snaplist; do if [ "$snap" = "${PHD_ENV_snapshot_name}" ]; then # we found requested snap before current and we are good echo "found snap ${PHD_ENV_snapshot_name}" break; fi if [ "$snap" = "$current" ]; then # current is before the requested snap, error echo "ERROR: Requested snap ${PHD_ENV_snapshot_name} is not a parent of current $current snap" exit 1 fi done done .... target=all .... # to roll back we need to kill hard the VMs but we want to make # sure that the other nodes won't fence before we get to # revert properly systemctl stop fence_virtd vmlist=$(virsh list --all --name) for vm in $vmlist; do rollback="" virsh destroy $vm snaplist=$(virsh snapshot-list --tree $vm | sed -e 's#.*|.*##g' -e 's#.*+- ##g') # we need to proceed from the bottom of the list for snap in $snaplist; do rollback="$snap $rollback" done # kill all disks for snap in $rollback; do if [ "$snap" = "${PHD_ENV_snapshot_name}" ]; then break; fi rm -f /localvms/${vm}-${snap}.cow # needs extra cleaning case $snap in cinder) rm -rf $PHD_VAR_osp_configdir/cinder ;; glance) rm -rf $PHD_VAR_osp_configdir/glance ;; compute) rm -rf $PHD_VAR_osp_configdir/instances/* ;; esac done # drop all childs virsh snapshot-delete --domain $vm ${PHD_ENV_snapshot_name} --metadata --children-only virsh detach-disk $vm vda --config virsh attach-disk $vm /localvms/$vm-${PHD_ENV_snapshot_name}.cow vda \ --sourcetype file --type disk \ --driver qemu --subdriver qcow2 \ --config done .... target=all .... systemctl start fence_virtd # get list of all VMs configured on a given host vmlist=$(virsh list --all --name) for vm in $vmlist; do virsh start $vm done .... ================================================ FILE: pcmk/vmsnap.scenario ================================================ # This file can be used directly by 'phd', see 'build-all.sh' in this # directory for how it can be invoked. The only requirement is a list # of nodes you'd like it to modify. # # The scope of each command-block is controlled by the preceeding # 'target' line. # # - target=all # The commands are executed on evey node provided # # - target=local # The commands are executed from the node hosting phd. When not # using phd, they should be run from some other independant host # (such as the puppet master) # # - target=$PHD_ENV_nodes{N} # The commands are executed on the Nth node provided. # For example, to run on only the first node would be target=$PHD_ENV_nodes1 # # We start with 3 (or more, up to 16) nodes running a minimal CentOS 6 # # Tasks to be performed include: # - setting up the required repositories from which to download Openstack and the HA-Addon # - disabling firewalls and SElinux. This is a necessary evil until the proper policies can be written. # - creating network bridges for use by VMs hosting OpenStack services # - normalizing network interface names # - fixing multicast # - removing /home and making the root partition as large as possible to maximumize the amount of space available to openstack ################################# # Scenario Requirements Section # ################################# = VARIABLES = PHD_ENV_snapshot_name ################################# # Scenario Requirements Section # ################################# = REQUIREMENTS = nodes: 3 ###################### # Deployment Scripts # ###################### = SCRIPTS = # links to snapshot management: # https://kashyapc.fedorapeople.org/virt/infra.next-2015/Advanced-Snapshots-with-libvirt-and-QEMU.pdf # http://lists.openstack.org/pipermail/openstack-dev/2013-June/010212.html # https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Virtualization_Deployment_and_Administration_Guide/index.html#warning-live-snapshots target=all .... # get list of all VMs configured on a given host vmlist=$(virsh list --all --name) # check if snapshot already exists, otherwise take the snapshot for vm in $vmlist; do if [ -f /localvms/${vm}-${PHD_ENV_snapshot_name}.cow ]; then echo "Snapshot ${PHD_ENV_snapshot_name} already exists" exit 0 fi virsh snapshot-create-as --domain $vm ${PHD_ENV_snapshot_name} --diskspec vda,file=/localvms/${vm}-${PHD_ENV_snapshot_name}.cow,snapshot=external --disk-only --atomic done ....