Showing preview only (521K chars total). Download the full file or copy to clipboard to get everything.
Repository: beekhof/osp-ha-deploy
Branch: master
Commit: 2373fbfd39fc
Files: 102
Total size: 491.7 KB
Directory structure:
gitextract_4wkbeqjk/
├── HA-keepalived.md
├── README.md
├── build-all.sh
├── full-stack.md
├── ha-openstack.md
├── iha-install-10.sh
├── iha-install-9.sh
├── iha-uninstall.sh
├── keepalived/
│ ├── ceilometer-config.md
│ ├── cinder-config.md
│ ├── compute-node.md
│ ├── controller-node.md
│ ├── galera-bootstrap.md
│ ├── galera-config.md
│ ├── glance-config.md
│ ├── haproxy-config.md
│ ├── heat-config.md
│ ├── horizon-config.md
│ ├── keepalived-config.md
│ ├── keystone-config.md
│ ├── memcached-config.md
│ ├── mongodb-config.md
│ ├── mongodb-recovery.md
│ ├── neutron-config.md
│ ├── nova-config.md
│ ├── phd-setup/
│ │ ├── ceilometer.scenario
│ │ ├── cinder.scenario
│ │ ├── compute.scenario
│ │ ├── galera.scenario
│ │ ├── glance.scenario
│ │ ├── ha-collapsed.variables
│ │ ├── heat.scenario
│ │ ├── horizon.scenario
│ │ ├── hypervisors.scenario
│ │ ├── keepalived.scenario
│ │ ├── keystone.scenario
│ │ ├── lb.scenario
│ │ ├── memcached.scenario
│ │ ├── mongodb.scenario
│ │ ├── neutron.scenario
│ │ ├── nova.scenario
│ │ ├── rabbitmq.scenario
│ │ ├── readme.txt
│ │ ├── redis.scenario
│ │ ├── sahara.scenario
│ │ ├── serverprep.scenario
│ │ ├── swift.scenario
│ │ ├── test.sh
│ │ └── trove.scenario
│ ├── rabbitmq-config.md
│ ├── rabbitmq-restart.md
│ ├── redis-config.md
│ ├── sahara-config.md
│ ├── swift-config.md
│ └── trove-config.md
├── make-vm
└── pcmk/
├── NovaCompute
├── NovaEvacuate
├── baremetal-rollback.scenario
├── baremetal.scenario
├── basic-cluster.scenario
├── beaker.scenario
├── ceilometer-test.sh
├── ceilometer.scenario
├── cinder-test.sh
├── cinder.scenario
├── compute-cluster.scenario
├── compute-common.scenario
├── compute-managed.scenario
├── controller-managed.scenario
├── galera-test.sh
├── galera.scenario
├── gateway.scenario
├── glance-test.sh
├── glance.scenario
├── ha-collapsed.variables
├── ha-segregated.variables
├── hacks.scenario
├── heat-test.sh
├── heat.scenario
├── horizon-test.sh
├── horizon.scenario
├── keystone-test.sh
├── keystone.scenario
├── lb.scenario
├── memcached.scenario
├── mongodb.scenario
├── mrg.variables
├── neutron-agents.scenario
├── neutron-server.scenario
├── neutron-test.sh
├── nova-test.sh
├── nova.scenario
├── nova_client.py
├── rabbitmq-test.sh
├── rabbitmq.scenario
├── swift-aco.scenario
├── swift-test.sh
├── swift.scenario
├── virt-hosts.scenario
├── vmsnap-rollback.scenario
└── vmsnap.scenario
================================================
FILE CONTENTS
================================================
================================================
FILE: HA-keepalived.md
================================================
Introduction
------------
This document aims at defining a high level architecture for a highly available OpenStack setup using application-native options and keepalived. It will document the overall setup, the architecture limitations and any important item to keep in mind when deploying.
The document can be used to create a highly available architecture for:
- [RDO Liberty](http://www.rdoproject.org)
- [Red Hat Enterprise Linux Openstack Platform 8](http://www.redhat.com/openstack), when available
**Note**: Instructions for Liberty are still work in progress.
If you are looking for the Kilo edition, check out the [Kilo-RDO7 branch](../Kilo-RDO7/HA-keepalived.md).
Most of the time the instructions will apply to both distributions. If there is any difference, it will be specified throughout the document.
### Authors and changelog
- Javier Peña (jpena@redhat.com) commited the initial version.
Architecture overview
---------------------
### Scope
This document will define, at a high level, the placement and high availability strategy for the different OpenStack services, as well as the limitations of the architecture.
Service monitoring and recovery are outside the scope for this document. You may choose to monitor and recover the running services health using any generally available monitoring tool, or implement automated service restart in systemd.
### Server roles
A typical OpenStack architecture will consist of servers performing various roles. The following roles may be spread over different servers or collapsed into some of them:
- Controller nodes
- Load balancer nodes
- Database nodes
- Network nodes
- Storage nodes
- Compute nodes
High availability for storage and compute nodes is out of the scope for this document.
### High availability strategy
The following diagram shows a very simplified view of the different strategies used to achieve high availability for the OpenStack services:
Depending on the method used to communicate with the service, the following availability strategies will be followed:
- Keepalived, for the HAProxy instances.
- Access via an HAProxy virtual IP, for services accessed via a TCP socket than can be load balanced (e.g. httpd).
- Built-in application clustering, when available from the application (e.g. Galera) .
- Starting up one instance of the service on several controller nodes, when they can coexist and coordinate by other means (e.g. RPC, in the case of nova-conductor).
- No high availability, when the service can only work in active/passive mode.
The detailed high availability strategy for the OpenStack services is defined in the following table.
| Service | Process | Mode | HA stragegy |
|------------------|----------------------------|:------:|-------------|
| Support services |MariaDB - Galera | A/A | HAProxy / app cluster |
| Support services |RabbitMQ | A/A | App cluster / service config |
| Support services |HAProxy | A/A | Keepalived |
| Support services |MongoDB | A/A | App cluster |
| Support services |Memcached | A/A | Service configuration |
| Support services |Redis | A/A | App cluster (Sentinel)|
| Keystone |openstack-keystone | A/A | HAProxy |
| Glance |openstack-glance-api | A/A | HAProxy |
| Glance |openstack-glance-registry | A/A | HAProxy |
| Nova |openstack-nova-api | A/A | HAProxy |
| Nova |openstack-nova-cert | A/A | |
| Nova |openstack-nova-compute | A/A | |
| Nova |openstack-nova-scheduler | A/A | |
| Nova |openstack-nova-conductor | A/A | |
| Nova |openstack-nova-novncproxy | A/A | HAProxy |
| Cinder |openstack-cinder-api | A/A | HAProxy |
| Cinder |openstack-cinder-scheduler | A/A | |
| Cinder |openstack-cinder-volume | **A/P**| No HA |
| Cinder |openstack-cinder-backup | A/A | |
| Neutron |neutron-server | A/A | HAProxy |
| Neutron |neutron-dhcp-agent | A/A | Multiple DHCP agents |
| Neutron |neutron-l3-agent | A/A | L3 HA |
| Neutron |neutron-metadata-agent | A/A | |
| Neutron |neutron-lbaas-agent | **A/P**| |
| Neutron |neutron-openvswitch-agent | A/A | |
| Neutron |neutron-metering-agent | A/A | |
| Horizon |httpd | A/A | HAProxy |
| Ceilometer |openstack-ceilometer-api | A/A | HAProxy |
| Ceilometer |openstack-ceilometer-central| A/A | Workload partitioning: tooz + Redis|
| Ceilometer |openstack-ceilometer-compute| A/A | |
| Ceilometer |openstack-ceilometer-alarm-notifier| A/A | |
| Ceilometer |openstack-ceilometer-evaluator| A/A | |
| Ceilometer |openstack-ceilometer-notification| A/A | |
| Heat |openstack-heat-api | A/A | HAProxy |
| Heat |openstack-heat-cfn | A/A | |
| Heat |openstack-heat-cloudwatch | A/A | |
| Heat |openstack-heat-engine | A/A | |
| Swift |openstack-swift-proxy | A/A | HAProxy |
| Swift |openstack-swift-account | A/A | HAProxy |
| Swift |openstack-swift-container | A/A | HAProxy |
| Swift |openstack-swift-object | A/A | HAProxy |
| Sahara |openstack-sahara-api | A/A | HAProxy |
| Sahara |openstack-sahara-engine | A/A | |
| Trove |openstack-trove-api | A/A | HAProxy |
| Trove |openstack-trove-engine | A/A | |
| Trove |openstack-trove-conductor | A/A | |
**Notes:**
1. There are known issues with cinder-volume that recommend setting it as active-passive for now, see <https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support>
2. While there will be multiple Neutron LBaaS agents running, each agent will manage a set of load balancers, that cannot be failed over to another node.
Architecture limitations
------------------------
This architecture has some inherent limitations that should be kept in mind during deployment and daily operations. The following sections describe those limitations.
### Keepalived and network partitions
In case of a network partitioning, there is a chance that two or more nodes running keepalived claim to hold the same VIP, which may lead to an undesired behaviour. Since keepalived uses VRRP over multicast to elect a master (VIP owner), a network partition in which keepalived nodes cannot communicate will result in the VIPs existing on two nodes. When the network partition is resolved, the duplicate VIPs should also be resolved. Note that this network partition problem with VRRP is a known limitation for this architecture.
### Cinder-volume as a single point of failure
There are currently concerns over the cinder-volume service ability to run as a fully active-active service. During the Mitaka timeframe, this is being worked on, see [1](https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support). Thus, cinder-volume will only be running on one of the controller nodes, even if it will be configured on all nodes. In case of a failure in the node running cinder-volume, it should be started in a surviving controller node.
### Neutron-lbaas-agent as a single point of failure
The current design of the Neutron LBaaS agent using the HAProxy driver does not allow high availability for the tenant load balancers. The neutron-lbaas-agent service will be enabled and running on all controllers, allowing for load balancers to be distributed across all nodes. However, a controller node failure will stop all load balancers running on that node until the service is recovered or the load balancer is manually removed and created again.
### Service monitoring and recovery required
An external service monitoring infrastructure is required to check the OpenStack service health, and notify operators in case of any failure. This architecture does not provide any facility for that, so it would be necessary to integrate the OpenStack deployment with any existing monitoring environment.
### Manual recovery after a full cluster restart
Some support services used by RDO / RHEL OSP use their own form of application clustering. Usually, these services maintain a cluster quorum, that may be lost in case of a simultaneous restart of all cluster nodes, e.g. during a power outage. Each service will require its own procedure to regain quorum:
- Galera: [Galera bootstrap instructions](keepalived/galera-bootstrap.md)
- RabbitMQ: [RabbitMQ cluster restart](keepalived/rabbitmq-restart.md)
- MongoDB: [MongoDB cluster recovery](keepalived/mongodb-recovery.md)
Implementation
--------------
The implementation will be split into two articles:
- [Controller node implementation](keepalived/controller-node.md)
- [Compute node implementation](keepalived/compute-node.md)
================================================
FILE: README.md
================================================
Introduction
------------
This repository contains the description of two highly available OpenStack architectures using RDO or Red Hat Enterprise Linux OpenStack Platform:
- An architecture based on [Pacemaker](ha-openstack.md).
- An architecture based on [application-native tools and Keepalived](HA-keepalived.md).
Each architecture includes a description and implementation instructions. Be sure to understand the contrains and limitations of each architecture, and choose the one that better suits your needs.
How you can help
----------------
Feedback is encouraged for this project. Feel free to submit patches and report issues.
================================================
FILE: build-all.sh
================================================
#!/bin/bash
set -e
declare -A nodeMap
declare -A variables
declare -A cluster
nodeMap["baremetal"]="east-01 east-02 east-03 east-04 east-05 east-06 east-07"
nodeMap["virt-hosts"]="east-01 east-02 east-03 east-04"
nodeMap["galera"]="rdo7-db1.vmnet rdo7-db2.vmnet rdo7-db3.vmnet"
nodeMap["memcached"]="rdo7-memcache1.vmnet rdo7-memcache2.vmnet rdo7-memcache3.vmnet"
nodeMap["swift-aco"]="rdo7-swift-brick1.vmnet rdo7-swift-brick2.vmnet rdo7-swift-brick3.vmnet"
nodeMap["compute-nodes"]="east-05 east-06 east-07"
nodeMap["controller-managed"]="rdo7-node1.vmnet rdo7-node2.vmnet rdo7-node3.vmnet east-05 east-06 east-07"
nodeMap["vmsnap"]="east-02 east-03 east-04"
cluster["baremetal"]=0
cluster["gateway"]=0
cluster["virt-hosts"]=0
variables["nodes"]=""
variables["network_domain"]="lab.bos.redhat.com"
variables["deployment"]="collapsed"
variables["status"]=0
variables["components"]="lb db rabbitmq memcache mongodb keystone glance cinder swift-brick swift neutron-server neutron-agents ceilometer heat"
variables["scenarios-segregated"]="gateway virt-hosts vmsnap-hacks hacks vmsnap-lb lb vmsnap-galera galera vmsnap-rabbitmq rabbitmq vmsnap-memcached memcached vmsnap-mongodb mongodb vmsnap-keystone keystone vmsnap-glance glance vmsnap-cinder cinder vmsnap-swift-aco swift-aco vmsnap-swift swift vmsnap-neutron-server neutron-server vmsnap-neutron-agents neutron-agents vmsnap-nova nova vmsnap-ceilometer ceilometer vmsnap-heat heat vmsnap-horizon horizon compute-common vmsnap-compute compute-cluster"
variables["scenarios-collapsed"]="gateway virt-hosts vmsnap-hacks hacks vmsnap-basic-cluster basic-cluster vmsnap-lb lb vmsnap-galera galera vmsnap-rabbitmq rabbitmq vmsnap-memcached memcached vmsnap-mongodb mongodb vmsnap-keystone keystone vmsnap-glance glance vmsnap-cinder cinder vmsnap-swift-aco swift-aco vmsnap-swift swift vmsnap-neutron-server neutron-server vmsnap-neutron-agents neutron-agents vmsnap-nova nova vmsnap-ceilometer ceilometer vmsnap-heat heat vmsnap-horizon horizon compute-common vmsnap-compute"
function create_phd_definition() {
scenario=$1
definition=$2
snapshot_name=$3
rm -f ${definition}
nodes=${variables["nodes"]}
if [ "x$nodes" = x ]; then
nodes=${nodeMap[$scenario]}
fi
if [ "x$nodes" = "x" ]; then
for n in `seq 1 3`; do
nodes="$nodes rdo7-${scenario}${n}.vmnet"
done
fi
nodelist="nodes="
for node in $nodes; do
nodelist="${nodelist}${node}.${variables["network_domain"]} "
done
echo "$nodelist" >> ${definition}
if [ -n "${snapshot_name}" ]; then
echo "snapshot_name=${snapshot_name}" >> ${definition}
fi
cat ${definition}
}
generate=0
scenarios=""
while true ; do
case "$1" in
--help|-h|-\?)
echo "$0 "
exit 0;;
-n|--node) variables["nodes"]="${variables["nodes"]} $2"; shift; shift;;
-c|--collapsed) variables["deployment"]="collapsed"; shift;;
-s|--segregated) variables["deployment"]="segregated"; shift;;
-f|--from) fromscenario=$2; shift; shift;;
-t|--to) toscenario=$2; shift; shift;;
-S|--status) variables["status"]=1; shift;;
-g|--generate) generate=1; shift;;
--mrg)
variables["network_domain"]="mpc.lab.eng.bos.redhat.com"
variables["config"]="mrg";
nodeMap["baremetal"]="mrg-01 mrg-02 mrg-03 mrg-04 mrg-07 mrg-08 mrg-09"
nodeMap["virt-hosts"]="mrg-01 mrg-02 mrg-03 mrg-04"
nodeMap["vmsnap"]="mrg-02 mrg-03 mrg-04"
nodeMap["compute-nodes"]="mrg-07 mrg-08 mrg-09"
nodeMap["controller-managed"]="rdo7-node1.vmnet rdo7-node2.vmnet rdo7-node3.vmnet mrg-07 mrg-08 mrg-09"
shift;;
-m|--method) redeploy=$2; shift; shift;;
-i|--instance) instance=$2; shift; shift;;
-x) set -x ; shift;;
--) shift ; break ;;
-*) echo "unknown option: $1"; exit 1;;
"") break;;
*) scenarios="${scenarios} $1"; shift;;
esac
done
if [ -z ${variables["config"]} ]; then
variables["config"]=ha-${variables["deployment"]}
fi
if [ ${variables["status"]} = 1 ]; then
if [ ${variables["deployment"]} != "collapsed" ]; then
scenarios=node
elif [ "x${scenarios}" = x ]; then
scenarios=${variables["components"]}
fi
for scenario in $scenarios; do
ssh rdo7-${scenario}1.vmnet.${variables["network_domain"]} -- crm_mon -1
done
exit 0
fi
if [ "x${scenarios}" = x ]; then
deploy=scenarios-${variables["deployment"]}
scenarios=${variables[${deploy}]}
if [ -z "$redeploy" ]; then
redeploy=full
fi
case $redeploy in
rollback) scenarios="baremetal-rollback $scenarios";;
full) scenarios="beaker baremetal $scenarios";;
*) echo "unknown redeploy method"; exit 1;;
esac
if [ -z "$instance" ]; then
instance=ha
fi
case $instance in
ha) scenarios="$scenarios compute-managed controller-managed";;
single) scenarios="$scenarios compute-cluster";;
*) echo "unknown instance value"; exit 1;;
esac
fi
function run_phd() {
if [ ${generate} = 1 ]; then
scripts=$(phd_exec -s ./pcmk/${1}.scenario -d ${HOME}/phd.${scenario}.conf -V ./pcmk/${variables["config"]}.variables -p | grep PHD_SCPT | sort | awk -F= '{print $2}')
for script in $scripts; do
echo "#### $script"
more "$script"
done
else
phd_exec -s ./pcmk/${1}.scenario -d ${HOME}/phd.${scenario}.conf -V ./pcmk/${variables["config"]}.variables
fi
}
inscenario=0
for scenario in $scenarios; do
if [ -n "$fromscenario" ]; then
if [ "$fromscenario" = "${scenario}" ]; then
inscenario=1
fi
if [ "$inscenario" = 0 ]; then
continue;
fi
fi
snapshot_name=""
if [ ${variables["deployment"]} = "collapsed" ]; then
case $scenario in
vmsnap-rollback-*)
snapshot_name=$(echo $scenario | sed -e 's/vmsnap-rollback-//g')
scenario=vmsnap-rollback
nodeMap[$scenario]=${nodeMap[vmsnap]}
;;
vmsnap-*)
snapshot_name=$(echo $scenario | sed -e 's/vmsnap-//g')
scenario=vmsnap
;;
compute-*)
nodeMap[$scenario]=${nodeMap[compute-nodes]}
;;
beaker|baremetal|gateway)
nodeMap[$scenario]=${nodeMap[baremetal]}
;;
baremetal-rollback)
# stupid hack to avoid spending heaps of time waiting for nfs
# server timeout
list=""
for x in ${nodeMap[baremetal]}; do
list="$x $list"
done
nodeMap[$scenario]="$list"
;;
virt-hosts|controller-managed)
;;
*)
# Overwrite the node list to be the nodes of our collapsed cluster
nodeMap[$scenario]="rdo7-node1.vmnet rdo7-node2.vmnet rdo7-node3.vmnet"
;;
esac
fi
create_phd_definition ${scenario} ${HOME}/phd.${scenario}.conf ${snapshot_name}
if [ x${cluster[${scenario}]} = x0 ]; then
: no need to bootstrap a cluster
elif [ ${variables["deployment"]} != "collapsed" ]; then
: prep a new cluster for ${scenario}
echo "$(date) :: Initializing cluster for scenario $scenario"
run_phd basic-cluster
fi
echo "$(date) :: Beginning scenario $scenario"
run_phd ${scenario}
if [ -n "$toscenario" ] && [ "$toscenario" = ${scenario} ]; then
echo "$(date) :: Reached $scenario. Stop processing"
break;
fi
done
================================================
FILE: full-stack.md
================================================
This is what the result of `pcs status` should look like:
Cluster name: rhos-node
Last updated: Fri Mar 6 22:06:28 2015
Last change: Fri Mar 6 22:03:52 2015
Stack: corosync
Current DC: rhos6-node2 (2) - partition with quorum
Version: 1.1.12-a14efad
3 Nodes configured
121 Resources configured
Online: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Full list of resources:
fence1 (stonith:fence_xvm): Started rhos6-node1
fence2 (stonith:fence_xvm): Started rhos6-node2
fence3 (stonith:fence_xvm): Started rhos6-node3
Clone Set: lb-haproxy-clone [lb-haproxy]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
vip-db (ocf::heartbeat:IPaddr2): Started rhos6-node1
vip-qpid (ocf::heartbeat:IPaddr2): Started rhos6-node2
vip-keystone (ocf::heartbeat:IPaddr2): Started rhos6-node3
vip-glance (ocf::heartbeat:IPaddr2): Started rhos6-node1
vip-cinder (ocf::heartbeat:IPaddr2): Started rhos6-node2
vip-swift (ocf::heartbeat:IPaddr2): Started rhos6-node3
vip-neutron (ocf::heartbeat:IPaddr2): Started rhos6-node1
vip-nova (ocf::heartbeat:IPaddr2): Started rhos6-node2
vip-horizon (ocf::heartbeat:IPaddr2): Started rhos6-node3
vip-heat (ocf::heartbeat:IPaddr2): Started rhos6-node1
vip-ceilometer (ocf::heartbeat:IPaddr2): Started rhos6-node2
vip-rabbitmq (ocf::heartbeat:IPaddr2): Started rhos6-node3
Master/Slave Set: galera-master [galera]
Masters: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: mongodb-clone [mongodb]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: memcached-clone [memcached]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: rabbitmq-server-clone [rabbitmq-server]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: keystone-clone [keystone]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: glance-fs-clone [glance-fs]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: glance-registry-clone [glance-registry]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: glance-api-clone [glance-api]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: cinder-api-clone [cinder-api]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: cinder-scheduler-clone [cinder-scheduler]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
cinder-volume (systemd:openstack-cinder-volume): Started rhos6-node1
Clone Set: swift-fs-clone [swift-fs]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: swift-account-clone [swift-account]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: swift-container-clone [swift-container]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: swift-object-clone [swift-object]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: swift-proxy-clone [swift-proxy]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
swift-object-expirer (systemd:openstack-swift-object-expirer): Started rhos6-node2
Clone Set: neutron-server-clone [neutron-server]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: neutron-scale-clone [neutron-scale] (unique)
neutron-scale:0 (ocf::neutron:NeutronScale): Started rhos6-node3
neutron-scale:1 (ocf::neutron:NeutronScale): Started rhos6-node1
neutron-scale:2 (ocf::neutron:NeutronScale): Started rhos6-node2
Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
ceilometer-central (systemd:openstack-ceilometer-central): Started rhos6-node3
Clone Set: ceilometer-collector-clone [ceilometer-collector]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: ceilometer-api-clone [ceilometer-api]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: ceilometer-delay-clone [ceilometer-delay]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: ceilometer-alarm-evaluator-clone [ceilometer-alarm-evaluator]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: ceilometer-alarm-notifier-clone [ceilometer-alarm-notifier]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: ceilometer-notification-clone [ceilometer-notification]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: heat-api-clone [heat-api]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: heat-api-cfn-clone [heat-api-cfn]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
Clone Set: heat-api-cloudwatch-clone [heat-api-cloudwatch]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
heat-engine (systemd:openstack-heat-engine): Started rhos6-node1
Clone Set: horizon-clone [horizon]
Started: [ rhos6-node1 rhos6-node2 rhos6-node3 ]
PCSD Status:
rhos6-node1: Online
rhos6-node2: Online
rhos6-node3: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
================================================
FILE: ha-openstack.md
================================================
# Highly Available Openstack Deployments
The current target for this document is RDO 10, based on
the OpenStack Newton release.
Looking for an edition prior to Newton (RDO10)?
Check out the [Juno-RDO6](../Juno-RDO6/ha-openstack.md) or
[Mitaka-RDO9](../Mitaka-RDO9/ha-openstack.md) branches instead.
## Purpose of this Document
This document aims at defining a high level architecture for a highly
available RHEL OSP setup with the [Pacemaker](http://clusterlabs.org)
cluster manager which provides:
- detection and recovery of machine and application-level failures
- startup/shutdown ordering between applications
- preferences for other applications that must/must-not run on the same machine
- provably correct response to any failure or cluster state
It is important to understand the following definitions used to
describe the operational mode of services in a cluster:
- Active/active
Traffic intended for the failed node is either passed onto an
existing node or load balanced across the remaining nodes. This is
usually only possible when the nodes use a homogeneous software
configuration.
- Active/passive
Provides a fully redundant instance of each node, which is only
brought online when its associated primary node fails. This
configuration typically requires the most extra hardware.
In this document, all components are currently modelled as
active/active with the exception of:
- cinder-volume
Implementation details are contained in scripts linked to from the main document.
Read them carefully before considering to run them in your own environment.
## Historical Context
In the previous OpenStack HA architectures used by Red Hat, SuSE and
others, Systemd is the entity in charge of starting and stopping most
OpenStack services. Pacemaker exists as a layer on top, signalling
when this should happen, but Systemd is the part making it happen.
This is a valuable contribution for active/passive (A/P) services and
those that require all their dependancies be available during their
startup and shutdown sequences. However as OpenStack has matured, more
and more components are able to operate in an unconstrained
active/active capacity with little regard for the startup/shutdown
order of their peers or dependancies - making them well suited to be
managed exclusively by Systemd.
## Overall Design
With Newton, OpenStack has reached the point where it is now a good
idea to limit Pacemaker’s involvement to core services like Galera and
Rabbit as well as the few remaining OpenStack services, such as
cinder-volume, that run A/P.
This will be particularly useful as we look towards a containerised
future. It both allows OpenStack to play nicely with the current
generation of container managers which lack Orchestration
capabilities, as well as reducing recovery and down time by allowing
for the maximum possible parallelism.
Any objections to this architecture usually fall into one of three
main categories:
1. The use of Pacemaker as an alerting mechanism
1. The idea that Pacemaker provides better monitoring of systemd services
1. A believe that active/passive installations are suprior
If these concerns apply to you then, as the founding author of
Pacemaker, I would like to direct your attention to my
[post](http://blog.clusterlabs.org/blog/2016/next-openstack-ha-arch)
which will attempt to disuade you of their relevance.
This reference design is based around a single cluster of 3 or more
nodes on which every component is running.
This scenario can be visualized as below:
With the advent of composable roles however, it is certainly possible
to dedicate a subset of nodes for one or more components that are
expected to be a bottleneck.
It is also possible that these dedicated nodes run extra copies of
those service, in addition to the ones on a fully symmetrical core set
of nodes.
## Assumptions
It is required that the clusters contain at least three nodes so that
we take advantage of
[quorum](http://en.wikipedia.org/wiki/Quorum_(Distributed_Systems))
Quorum becomes important when a failure causes the cluster to split in
two or more paritions. In this situation, you want the majority to
ensure the minority are truely dead (through fencing) and continue to
host resources. For a two-node cluster, no side has the majority and
you can end up in a situations where both sides fence each other, or
both sides are running the same services - leading to data corruption.
Clusters with an even number of hosts suffer from similar issues - a
single network failure could easily cause a N:N split where neither
side retains a majority. For this reason, we recommend an odd number
of cluster members when scaling up.
You can have up to 16 cluster members (this is currently limited by
corosync's ability to scale higher). In extreme cases, 32 and even up
to 64 nodes could be possible however this is not well tested.
In some environments, the available IP address range of the public LAN
is limited. If this applies to you, you will need one additional node
to set up as a [gateway](pcmk/gateway.scenario) that will provide DNS
and DHCP for the guests containing the OpenStack services and expose
the required nova and horizon APIs to the external network.
## Solution Components
### Cluster Manager
At its core, a cluster is a distributed finite state machine capable
of co-ordinating the startup and recovery of inter-related services
across a set of machines.
Even a distributed and/or replicated application that is able to
survive failures on one or more machines can benefit from a
cluster manager:
1. Awareness of other applications in the stack
While SYS-V init replacements like systemd can provide
deterministic recovery of a complex stack of services, the
recovery is limited to one machine and lacks the context of what
is happening on other machines - context that is crucial to
determine the difference between a local failure, clean startup
and recovery after a total site failure.
1. Awareness of instances on other machines
Services like RabbitMQ and Galera have complicated boot-up
sequences that require co-ordination, and often serialization, of
startup operations across all machines in the cluster. This is
especially true after site-wide failure or shutdown where we must
first determine the last machine to be active.
1. A shared implementation and calculation of [quorum](http://en.wikipedia.org/wiki/Quorum_%28Distributed_Systems%29)
It is very important that all members of the system share the same
view of who their peers are and whether or not they are in the
majority. Failure to do this leads very quickly to an internal
[split-brain](https://en.wikipedia.org/wiki/Split-brain_(computing))
state - where different parts of the system are pulling in
different and incompatioble directions.
1. Data integrity through fencing (a non-responsive process does not imply it is not doing anything)
A single application does not have sufficient context to know the
difference between failure of a machine and failure of the
applcation on a machine. The usual practice is to assume the
machine is dead and carry on, however this is highly risky - a
rogue process or machine could still be responding to requests and
generally causing havoc. The safer approach is to make use of
remotely accessible power switches and/or network switches and SAN
controllers to fence (isolate) the machine before continuing.
1. Automated recovery of failed instances
While the application can still run after the failure of several
instances, it may not have sufficient capacity to serve the
required volume of requests. A cluster can automatically recover
failed instances to prevent additional load induced failures.
For this reason, the use of a cluster manager like
[Pacemaker](http://clusterlabs.org) is highly recommended.
### Proxy server
Almost all services in this stack are proxied.
Using a proxy server provides:
1. Load distribution
Many services can act in an active/active capacity, however they
usually require an external mechanism for distributing requests to
one of the available instances. The proxy server can serve this
role.
1. API isolation
By sending all API access through the proxy, we can clearly
identify service interdependancies. We can also move them to
locations other than `localhost` to increase capacity if the need
arises.
1. Simplified process for adding/removing of nodes
Since all API access is directed to the proxy, adding or removing
nodes has no impact on the configuration of other services. This
can be very useful in upgrade scenarios where an entirely new set
of machines can be configured and tested in isolation before
telling the proxy to direct traffic there instead.
1. Enhanced failure detection
The proxy can be configured as a secondary mechanism for detecting
service failures. It can even be configured to look for nodes in
a degraded state (such as being 'too far' behind in the
replication) and take them out of circulation.
The following components are currently unable to benefit from the use
of a proxy server:
- RabbitMQ
- memcached
- mongodb
However the reasons vary and are discussed under each component's
heading.
We recommend HAProxy as the load balancer, however there are many
alternatives in the marketplace.
We use a check interval of 1 second however the timeouts vary by service.
Generally we use round-robin to distriute load amongst instances of
active/active services, however Galera and Qpid use the `stick-table`
options to ensure that incoming connections to the virtual IP (VIP)
should be directed to only one of the available backends.
In Galera's case, although it can run active/active, this helps avoid
lock contention and prevent deadlocks. It is used in combination with
the `httpchk` option that ensures only nodes that are in sync with its
peers are allowed to handle requests.
Qpid however operates in a active/passive configuration, no built-in
clustering, so in it's case the `stick-table` option ensures that all
requests go to the active instance.
### Replicated Database
Most OpenStack components require access to a database.
To avoid the database being a single point of failure, we require that
it be replicated and the ability to support multiple masters can help
when trying to scale other components.
One of the most popular database choices is Galera for MySQL, it supports:
- Synchronous replication
- active/active multi-master topology
- Automatic node joining
- True parallel replication, on row level
- Direct client connections, native MySQL look & feel
and claims:
- No slave lag
- No lost transactions
- Both read and write scalability
- Smaller client latencies
Although galera supports active/active configurations, we recommend
active/passive (enforced by the load balancer) in order to avoid lock
contention.
### Database Cache
Memcached is a general-purpose distributed memory caching system. It
is used to speed up dynamic database-driven websites by caching data
and objects in RAM to reduce the number of times an external data
source must be read.
__Note__: Access to memcached is not handled by HAproxy because
replicated access is currently only in an experimental state. Instead
consumers must be supplied with the full list of hosts running
memcached.
### Message Bus
An AMQP (Advanced Message Queuing Protocol) compliant message bus is
required for most OpenStack components in order to co-ordinate the
execution of jobs entered into the system.
RabbitMQ and Qpid are common deployment options. Both support:
- reliable message delivery
- flexible routing options
- replicated queues
This guide assumes RabbitMQ is being deployed, however we also
[document Qpid (TODO)](pcmk/osp-qpid.scenario) for completeness. Pay
attention to the comments in that guide for how selecting `Qpid` affects
the rest of the configuration.
__Note__: Access to RabbitMQ is not handled by HAproxy. Instead
consumers must be supplied with the full list of hosts running
RabbitMQ with `rabbit_hosts` and `rabbit_ha_queues` options.
Jock Eck found the [core
issue](http://people.redhat.com/jeckersb/private/vip-failover-tcp-persist.html)
and went into some detail regarding the [history and
solution](http://john.eckersberg.com/improving-ha-failures-with-tcp-timeouts.html)
on his blog.
In summary though:
> The source address for the connection from HAProxy back to the
> client is the VIP address. However the VIP address is no longer
> present on the host. This means that the network (IP) layer deems
> the packet unroutable, and informs the transport (TCP) layer. TCP,
> however, is a reliable transport. It knows how to handle transient
> errors and will retry. And so it does.
In this case that is a problem though, because:
> TCP generally holds on to hope for a long time. A ballpark estimate
> is somewhere on the order of tens of minutes (30 minutes is commonly
> referenced). During this time it will keep probing and trying to
> deliver the data.
>
> It's important to note that HAProxy has no idea that any of this is
> happening. As far as its process is concerned, it called write()
> with the data and the kernel returned success.
The [resolution](https://review.openstack.org/#/c/146047/) is already
understood and just needs to make its way through review.
## Core OpenStack services
In contrast to earlier versions of this guide, with the exception of
Cinder Volume, there are no specific instructions with regards to the
installation core OpenStack services beyond:
1. Ensuring services that make use of RabbitMQ list all configured servers
1. Accessing Galera and all OpenStack peer APIs (keystone, etc) via the HAProxy and the VIPs
In all other respects, one should follow standard practices for
installing packages and instructing the system to start them at boot
time.
### Cinder
Cinder provides 'block storage as a service' suitable for performance
sensitive scenarios such as databases, expandable file systems, or
providing a server with access to raw block level storage.
Persistent block storage can survive instance termination and can also
be moved across instances like any external storage device. Cinder
also has volume snapshots capability for backing up the volumes.
In theory cinder can be run as active/active however there are
currently sufficient concerns that cause us to recommend running the
volume component as active/passive only.
Jon Bernard writes:
> Requests are first seen by Cinder in the API service, and we have a
> fundamental problem there - a standard test-and-set race condition
> exists for many operations where the volume status is first checked
> for an expected status and then (in a different operation) updated to
> a pending status. The pending status indicates to other incoming
> requests that the volume is undergoing a current operation, however it
> is possible for two simultaneous requests to race here, which
> undefined results.
>
> Later, the manager/driver will receive the message and carry out the
> operation. At this stage there is a question of the synchronization
> techniques employed by the drivers and what guarantees they make.
>
> If cinder-volume processes exist as different process, then the
> 'synchronized' decorator from the lockutils package will not be
> sufficient. In this case the programmer can pass an argument to
> synchronized() 'external=True'. If external is enabled, then the
> locking will take place on a file located on the filesystem. By
> default, this file is placed in Cinder's 'state directory' in
> /var/lib/cinder so won't be visible to cinder-volume instances running
> on different machines.
>
> However, the location for file locking is configurable. So an
> operator could configure the state directory to reside on shared
> storage. If the shared storage in use implements unix file locking
> semantics, then this could provide the requisite synchronization
> needed for an active/active HA configuration.
>
> The remaining issue is that not all drivers use the synchronization
> methods, and even fewer of those use the external file locks.
> A sub-concern would be whether they use them correctly.
You can read more about these concerns on the [Red Hat
Bugzilla](https://bugzilla.redhat.com/show_bug.cgi?id=1193229) and
there is a [psuedo roadmap](https://etherpad.openstack.org/p/cinder-kilo-stabilisation-work)
for addressing them upstream.
# Implementation
The best way to visualize the result of this architecture is to make
use of
[tripleo-quickstart](https://github.com/openstack/tripleo-quickstart/blob/master/README.rst)
which implements the described architecture.
This will take a bare metal installation of your favorite OS (surely CentOS 7.2) and:
1. create a 'stack' user
1. create several VMs representing the undercloud, control plane and computes
1. deploy the undercloud (TripleO uses a pre-rolled OpenStack image as a means for deploying and updating the user facing installation of OpenStack aka. the overcloud)
1. deploy the overcloud for you to investigate and compare your existing architecture against
You should digest the entire [README](https://github.com/openstack/tripleo-quickstart/blob/master/README.rst), however the quick version for the purposes of seeing this HA architecture in action is:
git clone git@github.com:openstack/tripleo-quickstart.git
cd tripleo-quickstart
./quickstart.sh -b -n -w $PWD -c config/general_config/ha.yml -p quickstart-extras.yml -r quickstart-extras-requirements.txt --tags all -R newton -T all ${the_machine_you_wish_to_install_to}
For those that would prefer not to deal with TripleO, you can see
roughly what TripleO does by examining the pseudo code for manually:
1. configuring a basic [pacemaker cluster](pcmk/basic-cluster.scenario)
1. deploying the [load balancer](pcmk/lb.scenario)
1. deploying [galera](pcmk/galera.scenario)
1. deploying [memcached](pcmk/memcached.scenario)
1. deploying [rabbitmq](pcmk/rabbitmq.scenario)
1. deploying [cinder volume](pcmk/cinder.scenario)
Here is a [list of variables](pcmk/ha-collapsed.variables) used when
executing the referenced scripts. Modify them to your needs.
## Disclaimer
- The referenced scripts contain many comments and warnings - READ THEM CAREFULLY.
- There are probably 2^8 other ways to deploy this same scenario. This is only one of them.
- Due to limited number of available physical LAN connections in the test setup, the instance IP traffic overlaps with the internal/management network.
- Distributed/Shared storage is provided via NFS from the commodity server due to lack of dedicated CEPH servers. Any other kind of storage supported by OpenStack would work just fine.
- Bare metal could be used in place of any or all guests.
- Most of the scripts contain shell expansion to automatically fill in some values. Use your common sense when parsing data. Example:
`openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address $(ip addr show dev vmnet0 scope global | grep inet | sed -e 's#.*inet ##g' -e 's#/.*##g')`
means that we want the IP address from vmnet0 as vncserver_proxyclient_address.
# Compute Nodes
We will usually need more than 16 compute nodes
which is beyond Corosync's ability to manage. So in order monitor the
healthiness of compute nodes and the services running on them, we
previously had to create single node clusters.
The current deployment model allows Pacemaker to continue this role,
but presents a single coherent view of the entire deployment while
allowing us to scale beyond corosync's limits. Having this single
administrative domain then allows us to do clever things like
automated recovery of VMs running on a failed or failing compute node.
The main difference with the previous deployment mode is that services
on the compute nodes are now managed and driven by the Pacemaker
cluster on the control plane. The compute nodes do not become full
members of the cluster and they no longer require the full cluster
stack, instead they run pacemaker_remoted which acts as a conduit.
> Implementation Details:
>
> - Pacemaker monitors the connection to pacemaker_remoted to verify
> that the node is reachable or not. Failure to talk to a node
> triggers recovery action.
>
> - Pacemaker uses pacemaker_remoted to start compute node services in
> the same sequence as before (neutron-ovs-agent ->
> ceilometer-compute -> nova-compute).
>
> - If a service fails to start, any services that depend on the
> FAILED service will not be started. This avoids the issue of
> adding a broken node (back) to the pool.
>
> - If a service fails to stop, the node where the service is running
> will be fenced. This is necessary to guarantee data integrity and
> a core HA concept (for the purposes of this particular discussion,
> please take this as a given).
>
> - If a service's health check fails, the resource (and anything that
> depends on it) will be stopped and then restarted. Remember that
> failure to stop will trigger a fencing action.
>
> - A successful restart of all the services can only potentially
> affect network connectivity of the instances for a short period of
> time.
With these capabilities in place, we can exploit Pacemaker's node
monitoring and fencing capabilities to drive nova host-evacuate for
the failed compute nodes and recover the VMs elsewhere.
When a compute node fails, Pacemaker will:
1. Execute 'nova service-disable'
2. fence (power off) the failed compute node
3. fence_compute off (waiting for nova to detect compute node is gone)
4. fence_compute on (a no-op unless the host happens to be up already)
5. Execute 'nova service-enable' when the compute node returns
Technically steps 1 and 5 are optional and they are aimed to improve
user experience by immediately excluding a failed host from nova
scheduling.
The only benefit is a faster scheduling of VMs that happens during a
failure (nova does not have to recognize a host is down, timeout and
subsequently schedule the VM on another host).
Step 2 will make sure the host is completely powered off and nothing
is running on the host. Optionally, you can have the failed host
reboot which would potentially allow it to re-enter the pool.
We have an implementation for Step 3 but the ideal solution depends on
extensions to the nova API. Currently fence_compute loops, waiting
for nova to recognise that the failed host is down, before we make a
host-evacuate call which triggers nova to restart the VMs on another
host. The discussed nova API extensions will speed up recovery times
by allowing fence_compute to proactively push that information into
nova instead.
To take advantage of the VM recovery features:
- VMs need to be running off a cinder volume or using shared ephemeral
storage (like RBD or NFS)
- If VM is not running using shared storage, recovery of the instance
on a new compute node would need to revert to a previously stored
snapshot/image in Glance (potentially losing state, but in some
cases that may not matter)
- RHEL7.1+ required for infrastructure nodes (controllers and
compute). Instance guests can run anything.
- Compute nodes need to have a working fencing mechanism (IPMI,
hardware watchdog, etc)
## Compute Node Implementation
Start by creating a minimal CentOS __7__ installation on at least one node.
Once the machine(s) have been installed, [prepare
them](pcmk/baremetal.scenario) for hosting OpenStack.
Next, you can configure them as [compute nodes](pcmk/compute-common.scenario).
We now add them to the cluster as [partial members](pcmk/compute-managed.scenario).
Once the compute nodes are configured as remote, they can be added
to the [controller backplane](pcmk/controller-managed.scenario)
> TODO: what if nova-compute fails to restart and there are scheduled
> instances? Those can still be accessed from outside but cannot be
> managed by nova. This might warrant a host-evacuate.
>
> Traditionally, HA systems would fence the node at this point.
================================================
FILE: iha-install-10.sh
================================================
#!/bin/bash
set -ex
#
#If your deployment includes this use case, include the no_shared_storage=1 option in step 7.
helper=~/.ssh/config
cat <<EOF >> $helper
Host overcloud-*
BatchMode yes
User heat-admin
StrictHostKeyChecking no
EOF
source stackrc
nova list | grep ctlplane | awk -F\| '{ print "Host "$3 "\n\tHostName " $7}' | sed 's/ctlplane=//' >> $helper
chmod 600 $helper
COMPUTES=$(nova list | grep novacompute | awk -F\| '{ print $3}' | tr '\n' ' ')
CONTROLLERS=$(nova list | grep controller | awk -F\| '{ print $3}' | tr '\n' ' ')
FIRST_COMPUTE=$(echo $COMPUTES | awk '{print $1}')
FIRST_CONTROLLER=$(echo $CONTROLLERS | awk '{print $1}')
# 15. Add stonith devices for the Compute nodes. fence_ironic is not
# a recommended approach, but is usable everywhere which makes it
# ideal for generic scripts like this
if [ 1 = 1 ]; then
helper=iha-helper-configure-virt-fencing.sh
cat <<EOF > $helper
cp fence_ironic.py /usr/sbin/fence_ironic
sudo chmod a+x /usr/sbin/fence_ironic
EOF
for node in $CONTROLLERS; do scp $helper fence_ironic.py ${node}: ; done
for node in $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done
hostmap="";
hostlist=$(nova list | grep ctlplane | awk -F\| '{print $3}')
for host in $hostlist; do hostmap="$hostmap $host:$(ironic node-list | grep $(nova list | grep $host | awk '{print $2}') | awk '{print $4}' )"; done
ssh ${FIRST_CONTROLLER} -- sudo pcs stonith create shooter fence_ironic auth-url=${OS_AUTH_URL} login=${OS_USERNAME} passwd=${OS_PASSWORD} tenant-name=${OS_TENANT_NAME} pcmk_host_map=\"${hostmap}\" op monitor interval=60s timeout=180s
fi
# 1. Begin by stopping and disabling libvirtd and all OpenStack services on the Compute nodes:
source overcloudrc
ssh ${FIRST_CONTROLLER} -- sudo pcs property set stonith-enabled=false
helper=iha-helper-stop-services.sh
cat <<EOF > $helper
set -ex
for s in openstack-nova-compute neutron-openvswitch-agent libvirtd; do
systemctl stop \${s}
systemctl disable \${s}
done
# Punch a hole for pacemaker-remote
iptables -I INPUT -p tcp --dport 3121 -j ACCEPT
service iptables save
EOF
for node in $COMPUTES; do scp $helper ${node}: ; done
for node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done
# 2. Create an authentication key for use with pacemaker-remote.
dd if=/dev/urandom of=./authkey bs=4096 count=1
# 3. Copy this key to the director node, and then to the remaining Compute and Controller nodes:
helper=iha-helper-fix-auth.sh
cat <<EOF > $helper
set -ex
mkdir -p --mode=0750 /etc/pacemaker/
chgrp haclient /etc/pacemaker
mv authkey /etc/pacemaker/
chown root:haclient /etc/pacemaker/authkey
EOF
for node in $COMPUTES $CONTROLLERS; do scp ./authkey $helper ${node}: ; done
for node in $COMPUTES $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done
# 4. Enable pacemaker-remote on all Compute nodes:
for compute in $COMPUTES; do ssh ${compute} -- sudo systemctl enable pacemaker_remote; done
for compute in $COMPUTES; do ssh ${compute} -- sudo systemctl start pacemaker_remote; done
# 5. Confirm that the required versions of the pacemaker (1.1.13-10.el7_2.2.x86_64), fence-agents (fence-agents-all-4.0.11-27.el7_2.5.x86_64) and resource-agents (3.9.5-54.el7_2.6.x86_64`) packages are installed on the controller and Compute nodes:
for compute in $COMPUTES; do ssh ${compute} -- rpm -qa | egrep '(pacemaker|fence-agents|resource-agents)' ; done
# 7. Create a NovaEvacuate active/passive resource using the overcloudrc file to provide the auth_url, username, tenant and password values:
# 8. Confirm that nova-evacuate is started after the floating IP resources, and the Image Service (glance), OpenStack Networking (neutron), Compute (nova) services:
helper=iha-helper-create-evacuate.sh
cat <<EOF > $helper
set -ex
pcs resource create nova-evacuate ocf:openstack:NovaEvacuate auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME domain=localdomain no_shared_storage=1 op monitor interval=60s timeout=240s
sudo pcs constraint order start haproxy-clone then nova-evacuate
sudo pcs constraint order start galera-clone then nova-evacuate
sudo pcs constraint order start rabbitmq-clone then nova-evacuate
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# Note: If you are not using shared storage, include the no_shared_storage=1 option in your resource create ... command above. See Exception for shared storage for more information.
# 10. Create a list of the current controllers using cibadmin data :
#controller-1 # controllers=$(sudo cibadmin -Q -o nodes | grep uname | sed s/.*uname..// | awk -F\" '{print $1}')
#controller-1 # echo $controllers
# 11. Use this list to tag these nodes as controllers with the osprole=controller property:
# 12. Build a list of stonith devices already present in the environment:
# 13. Tag the control plane services to make sure they only run on the controllers identified above, skipping any stonith devices listed:
helper=iha-helper-tag-controllers.sh
cat <<EOF > $helper
set -ex
for controller in ${CONTROLLERS}; do pcs property set --node \${controller} osprole=controller ; done
stonithdevs=\$(pcs stonith | awk '{print \$1}')
for i in \$(cibadmin -Q --xpath //primitive --node-path | tr ' ' '\n' | awk -F "id='" '{print \$2}' | awk -F "'" '{print \$1}' | uniq); do
found=0
if [ -n "\$stonithdevs" ]; then
for x in \$stonithdevs; do
if [ \$x = \$i ]; then
found=1
fi
done
fi
if [ \$found = 0 ]; then
pcs constraint location \$i rule resource-discovery=exclusive score=0 osprole eq controller
fi
done
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# 14. Begin to populate the Compute node resources within pacemaker, starting with neutron-openvswitch-agent:
helper=fix-compute.sh
cat <<EOF > $helper
set -x
grep crudini /usr/lib/ocf/resource.d/openstack/nova-compute-wait
if [ \$? != 0 ]; then
set -e
patch -p1 /usr/lib/ocf/resource.d/openstack/nova-compute-wait \${PWD}/bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch
fi
EOF
for node in $COMPUTES; do scp bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch $helper ${node}: ; done
for node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done
source overcloudrc
helper=iha-helper-create-compute-resources.sh
cat <<EOF > $helper
set -ex
pcs resource create neutron-openvswitch-agent-compute systemd:neutron-openvswitch-agent op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force
pcs constraint location neutron-openvswitch-agent-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute
# Then the Compute libvirtd resource:
pcs resource create libvirtd-compute systemd:libvirtd op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force
pcs constraint location libvirtd-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute
pcs constraint order start neutron-openvswitch-agent-compute-clone then libvirtd-compute-clone
pcs constraint colocation add libvirtd-compute-clone with neutron-openvswitch-agent-compute-clone
# Then the nova-compute resource:
pcs resource create nova-compute-checkevacuate ocf:openstack:nova-compute-wait auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME domain=localdomain op start timeout=300 --clone interleave=true --disabled --force
pcs constraint location nova-compute-checkevacuate-clone rule resource-discovery=exclusive score=0 osprole eq compute
pcs resource create nova-compute systemd:openstack-nova-compute op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force
pcs constraint location nova-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute
pcs constraint order start nova-compute-checkevacuate-clone then nova-compute-clone require-all=true
pcs constraint order start nova-compute-clone then nova-evacuate require-all=false
pcs constraint order start libvirtd-compute-clone then nova-compute-clone
pcs constraint colocation add nova-compute-clone with libvirtd-compute-clone
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# 16. Create a seperate fence-nova stonith device:
helper=iha-helper-create-nova-fence.sh
cat <<EOF > $helper
set -ex
pcs stonith create fence-nova fence_compute auth-url=$OS_AUTH_URL login=$OS_USERNAME passwd=$OS_PASSWORD tenant-name=$OS_TENANT_NAME domain=localdomain record-only=1 op monitor interval=60s timeout=180s --force
EOF
source stackrc
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# 17. Make certain the Compute nodes are able to recover after fencing:
ssh ${FIRST_CONTROLLER} -- sudo pcs property set cluster-recheck-interval=1min
# 18. Create Compute node resources and set the stonith level 1 to include both the nodes's physical fence device and fence-nova:
helper=iha-helper-create-computes.sh
cat <<EOF > $helper
set -ex
for node in $COMPUTES; do
pcs resource create \${node} ocf:pacemaker:remote reconnect_interval=60 op monitor interval=20
pcs property set --node \${node} osprole=compute
pcs stonith level add 1 \${node} shooter,fence-nova
done
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# 19. Enable the control and Compute plane services:
helper=iha-helper-enable-services.sh
cat <<EOF > $helper
set -ex
pcs resource enable neutron-openvswitch-agent-compute
pcs resource enable libvirtd-compute
pcs resource enable nova-compute-checkevacuate
pcs resource enable nova-compute
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# 20. Allow some time for the environment to settle before cleaning up any failed resources:
helper=iha-helper-cleanup.sh
cat <<EOF > $helper
set -ex
sleep 60
pcs resource cleanup
pcs status
echo pcs property set stonith-enabled=true
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
#Test High Availability
#Note: These steps deliberately reboot the Compute node without warning.
#1. The following step boots an instance on the overcloud, and then crashes the Compute node:
#stack@director # . overcloudrc
#stack@director # nova boot --image cirros --flavor 2 test-failover
#stack@director # nova list --fields name,status,host
#stack@director # . stackrc
#stack@director # ssh -lheat-admin compute-n
#compute-n # sudo su -
#root@compute-n # echo c > /proc/sysrq-trigger
#
#2. A short time later, the instance should be restarted on a working Compute node:
#
#stack@director # nova list --fields name,status,host
#stack@director # nova service-list
# 3 ip6tables -I INPUT -p tcp --dport 3121 -j ACCEPT
# 4 iptables -I INPUT -p tcp --dport 3121 -j ACCEPT
# 5 service iptables save
# 6 service ip6tables save
================================================
FILE: iha-install-9.sh
================================================
#!/bin/bash
set -ex
#
#If your deployment includes this use case, include the no_shared_storage=1 option in step 7.
helper=~/.ssh/config
cat <<EOF > $helper
Host overcloud-*
BatchMode yes
User heat-admin
StrictHostKeyChecking no
EOF
source stackrc
nova list | grep ctlplane | awk -F\| '{ print "Host "$3 "\n\tHostName " $7}' | sed 's/ctlplane=//' >> $helper
chmod 600 $helper
COMPUTES=$(nova list | grep novacompute | awk -F\| '{ print $3}' | tr '\n' ' ')
CONTROLLERS=$(nova list | grep controller | awk -F\| '{ print $3}' | tr '\n' ' ')
FIRST_COMPUTE=$(echo $COMPUTES | awk '{print $1}')
FIRST_CONTROLLER=$(echo $CONTROLLERS | awk '{print $1}')
# 15. Add stonith devices for the Compute nodes. fence_ironic is not
# a recommended approach, but is usable everywhere which makes it
# ideal for generic scripts like this
if [ 1 = 1 ]; then
helper=iha-helper-configure-virt-fencing.sh
cat <<EOF > $helper
cp fence_ironic.py /usr/sbin/fence_ironic
sudo chmod a+x /usr/sbin/fence_ironic
EOF
for node in $CONTROLLERS; do scp $helper fence_ironic.py ${node}: ; done
for node in $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done
hostmap=$(nova list | grep ctlplane | awk -F\| '{print $3 ":" $2}' | tr -d ' ' | tr '\n' ' ')
ssh ${FIRST_CONTROLLER} -- sudo pcs stonith create shooter fence_ironic auth-url=${OS_AUTH_URL} login=${OS_USERNAME} passwd=${OS_PASSWORD} tenant-name=${OS_TENANT_NAME} pcmk_host_map=\"${hostmap}\" op monitor interval=60s
fi
# 1. Begin by stopping and disabling libvirtd and all OpenStack services on the Compute nodes:
source overcloudrc
ssh ${FIRST_CONTROLLER} -- sudo pcs property set stonith-enabled=false
helper=iha-helper-stop-services.sh
cat <<EOF > $helper
set -ex
openstack-service stop
openstack-service disable
systemctl stop libvirtd
systemctl disable libvirtd
EOF
for node in $COMPUTES; do scp $helper ${node}: ; done
for node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done
# 2. Create an authentication key for use with pacemaker-remote.
dd if=/dev/urandom of=./authkey bs=4096 count=1
# 3. Copy this key to the director node, and then to the remaining Compute and Controller nodes:
helper=iha-helper-fix-auth.sh
cat <<EOF > $helper
set -ex
mkdir -p --mode=0750 /etc/pacemaker/
chgrp haclient /etc/pacemaker
mv authkey /etc/pacemaker/
chown root:haclient /etc/pacemaker/authkey
EOF
for node in $COMPUTES $CONTROLLERS; do scp ./authkey $helper ${node}: ; done
for node in $COMPUTES $CONTROLLERS; do ssh ${node} -- sudo bash $helper ; done
# 4. Enable pacemaker-remote on all Compute nodes:
for compute in $COMPUTES; do ssh ${compute} -- sudo systemctl enable pacemaker_remote; done
for compute in $COMPUTES; do ssh ${compute} -- sudo systemctl start pacemaker_remote; done
# 5. Confirm that the required versions of the pacemaker (1.1.13-10.el7_2.2.x86_64), fence-agents (fence-agents-all-4.0.11-27.el7_2.5.x86_64) and resource-agents (3.9.5-54.el7_2.6.x86_64`) packages are installed on the controller and Compute nodes:
for compute in $COMPUTES; do ssh ${compute} -- rpm -qa | egrep '(pacemaker|fence-agents|resource-agents)' ; done
# 6.a Apply the following constraint workarounds required for BZ#1257414:
#Note: This issue has been addressed in RHSA-2015:1862 and might not be required for your environment.
#controller-1 # sudo pcs constraint order start openstack-nova-novncproxy-clone then openstack-nova-api-clone
#controller-1 # sudo pcs constraint order start rabbitmq-clone then openstack-keystone-clone
#controller-1 # sudo pcs constraint order promote galera-master then openstack-keystone-clone
#controller-1 # sudo pcs constraint order start haproxy-clone then openstack-keystone-clone
#controller-1 # sudo pcs constraint order start memcached-clone then openstack-keystone-clone
#controller-1 # sudo pcs constraint order promote redis-master then start openstack-ceilometer-central-clone require-all=false
#controller-1 # sudo pcs resource defaults resource-stickiness=INFINITY
#6.b Apply the following constraint workarounds required for BZ#1295835:
#Note: This issue has been addressed in RHBA-2016:0264-1 and might not be required for your environment.
#sudo pcs config | grep systemd | awk '{print $2}' | while read RESOURCE; do sudo pcs resource update $RESOURCE op start timeout=200s op stop timeout=200s; done"
# 7. Create a NovaEvacuate active/passive resource using the overcloudrc file to provide the auth_url, username, tenant and password values:
# 8. Confirm that nova-evacuate is started after the floating IP resources, and the Image Service (glance), OpenStack Networking (neutron), Compute (nova) services:
helper=iha-helper-create-evacuate.sh
cat <<EOF > $helper
set -ex
pcs resource create nova-evacuate ocf:openstack:NovaEvacuate auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME no_shared_storage=1
for i in \$(sudo pcs status | grep IP | awk '{ print \$1 }'); do sudo pcs constraint order start \$i then nova-evacuate ; done
for i in openstack-glance-api-clone neutron-metadata-agent-clone openstack-nova-conductor-clone; do sudo pcs constraint order start \$i then nova-evacuate require-all=false ; done
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# Note: If you are not using shared storage, include the no_shared_storage=1 option in your resource create ... command above. See Exception for shared storage for more information.
# 9. Disable all OpenStack resources across the control plane:
# Depending on the time needed to stop Identity Service (and on the power of your hardware) you can consider increasing the timeout period (--wait).
# OSP8: ssh ${FIRST_CONTROLLER} -- sudo pcs resource disable openstack-keystone --wait=600s
# OSP9+:
ssh ${FIRST_CONTROLLER} -- sudo pcs resource disable openstack-core-clone --wait=600s
# 10. Create a list of the current controllers using cibadmin data :
#controller-1 # controllers=$(sudo cibadmin -Q -o nodes | grep uname | sed s/.*uname..// | awk -F\" '{print $1}')
#controller-1 # echo $controllers
# 11. Use this list to tag these nodes as controllers with the osprole=controller property:
# 12. Build a list of stonith devices already present in the environment:
# 13. Tag the control plane services to make sure they only run on the controllers identified above, skipping any stonith devices listed:
helper=iha-helper-tag-controllers.sh
cat <<EOF > $helper
set -ex
for controller in ${CONTROLLERS}; do pcs property set --node \${controller} osprole=controller ; done
stonithdevs=\$(pcs stonith | awk '{print \$1}')
for i in \$(cibadmin -Q --xpath //primitive --node-path | tr ' ' '\n' | awk -F "id='" '{print \$2}' | awk -F "'" '{print \$1}' | uniq); do
found=0
if [ -n "\$stonithdevs" ]; then
for x in \$stonithdevs; do
if [ \$x = \$i ]; then
found=1
fi
done
fi
if [ \$found = 0 ]; then
pcs constraint location \$i rule resource-discovery=exclusive score=0 osprole eq controller
fi
done
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# 14. Begin to populate the Compute node resources within pacemaker, starting with neutron-openvswitch-agent:
helper=fix-compute.sh
cat <<EOF > $helper
set -x
grep crudini /usr/lib/ocf/resource.d/openstack/nova-compute-wait
if [ \$? != 0 ]; then
set -e
patch -p1 /usr/lib/ocf/resource.d/openstack/nova-compute-wait \${PWD}/bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch
fi
EOF
for node in $COMPUTES; do scp bz1380314-nova-compute-wait-fix-invalid-hostname-issue.patch $helper ${node}: ; done
for node in $COMPUTES; do ssh ${node} -- sudo bash $helper ; done
helper=iha-helper-create-compute-resources.sh
cat <<EOF > $helper
set -ex
pcs resource create neutron-openvswitch-agent-compute systemd:neutron-openvswitch-agent op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force
pcs constraint location neutron-openvswitch-agent-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute
pcs constraint order start neutron-server-clone then neutron-openvswitch-agent-compute-clone require-all=false
# Then the Compute libvirtd resource:
pcs resource create libvirtd-compute systemd:libvirtd op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force
pcs constraint location libvirtd-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute
pcs constraint order start neutron-openvswitch-agent-compute-clone then libvirtd-compute-clone
pcs constraint colocation add libvirtd-compute-clone with neutron-openvswitch-agent-compute-clone
# Then the openstack-ceilometer-compute resource:
pcs resource create ceilometer-compute systemd:openstack-ceilometer-compute op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force
pcs constraint location ceilometer-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute
pcs constraint order start openstack-ceilometer-notification-clone then ceilometer-compute-clone require-all=false
pcs constraint order start libvirtd-compute-clone then ceilometer-compute-clone
pcs constraint colocation add ceilometer-compute-clone with libvirtd-compute-clone
# Then the nova-compute resource:
pcs resource create nova-compute-checkevacuate ocf:openstack:nova-compute-wait auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME domain=localdomain op start timeout=300 --clone interleave=true --disabled --force
pcs constraint location nova-compute-checkevacuate-clone rule resource-discovery=exclusive score=0 osprole eq compute
pcs constraint order start openstack-nova-conductor-clone then nova-compute-checkevacuate-clone require-all=false
pcs resource create nova-compute systemd:openstack-nova-compute op start timeout=200s op stop timeout=200s --clone interleave=true --disabled --force
pcs constraint location nova-compute-clone rule resource-discovery=exclusive score=0 osprole eq compute
pcs constraint order start nova-compute-checkevacuate-clone then nova-compute-clone require-all=true
pcs constraint order start nova-compute-clone then nova-evacuate require-all=false
pcs constraint order start libvirtd-compute-clone then nova-compute-clone
pcs constraint colocation add nova-compute-clone with libvirtd-compute-clone
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# 16. Create a seperate fence-nova stonith device:
helper=iha-helper-create-nova-fence.sh
cat <<EOF > $helper
set -ex
pcs stonith create fence-nova fence_compute auth-url=$OS_AUTH_URL login=$OS_USERNAME passwd=$OS_PASSWORD tenant-name=$OS_TENANT_NAME domain=localdomain record-only=1 --force
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# 17. Make certain the Compute nodes are able to recover after fencing:
ssh ${FIRST_CONTROLLER} -- sudo pcs property set cluster-recheck-interval=1min
# 18. Create Compute node resources and set the stonith level 1 to include both the nodes's physical fence device and fence-nova:
helper=iha-helper-create-computes.sh
cat <<EOF > $helper
set -ex
for node in $COMPUTES; do
pcs resource create \${node} ocf:pacemaker:remote reconnect_interval=60 op monitor interval=20
pcs property set --node \${node} osprole=compute
pcs stonith level add 1 \${node} shooter,fence-nova
done
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# 19. Enable the control and Compute plane services:
helper=iha-helper-enable-services.sh
cat <<EOF > $helper
set -ex
#pcs resource enable openstack-keystone
pcs resource enable openstack-core
pcs resource enable neutron-openvswitch-agent-compute
pcs resource enable libvirtd-compute
pcs resource enable ceilometer-compute
pcs resource enable nova-compute-checkevacuate
pcs resource enable nova-compute
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
# 20. Allow some time for the environment to settle before cleaning up any failed resources:
helper=iha-helper-cleanup.sh
cat <<EOF > $helper
set -ex
sleep 60
pcs resource cleanup
pcs status
echo pcs property set stonith-enabled=true
EOF
scp $helper ${FIRST_CONTROLLER}:
ssh ${FIRST_CONTROLLER} -- sudo bash $helper
#Test High Availability
#Note: These steps deliberately reboot the Compute node without warning.
#1. The following step boots an instance on the overcloud, and then crashes the Compute node:
#stack@director # . overcloudrc
#stack@director # nova boot --image cirros --flavor 2 test-failover
#stack@director # nova list --fields name,status,host
#stack@director # . stackrc
#stack@director # ssh -lheat-admin compute-n
#compute-n # sudo su -
#root@compute-n # echo c > /proc/sysrq-trigger
#
#2. A short time later, the instance should be restarted on a working Compute node:
#
#stack@director # nova list --fields name,status,host
#stack@director # nova service-list
================================================
FILE: iha-uninstall.sh
================================================
#!/bin/bash
set -ex
source stackrc
# If your machines don't conform to this structure, try calling the script like this:
# COMPUTE_PATTERN=mycompute ./iha-uninstall.sh upgrade
: ${COMPUTE_PATTERN=novacompute}
: ${CONTROLLER_PATTERN=controller}
COMPUTES=$(nova list | grep ${COMPUTE_PATTERN} | awk -F\| '{ print $3}' | tr '\n' ' ')
CONTROLLERS=$(nova list | grep ${CONTROLLER_PATTERN} | awk -F\| '{ print $3}' | tr '\n' ' ')
FIRST_COMPUTE=$(echo $COMPUTES | awk '{print $1}')
FIRST_CONTROLLER=$(echo $CONTROLLERS | awk '{print $1}')
ssh ${FIRST_CONTROLLER} -- sudo pcs property set stonith-enabled=false
if [ $1 = "upgrade" ]; then
SERVICES="nova-evacuate neutron-openvswitch-agent-compute libvirtd-compute-clone ceilometer-compute-clone nova-compute-checkevacuate-clone nova-compute-clone"
helper=iha-helper-remove.sh
cat <<EOF > $helper
set -ex
pcs property set maintenance-mode=true
for resource in $COMPUTES $SERVICES $FUDGE; do
pcs resource cleanup \${resource}
pcs --force resource delete \${resource}
done
for node in $COMPUTES; do
cibadmin --delete --xml-text "<node id='\${node}'/>"
cibadmin --delete --xml-text "<node_state id='\${node}'/>"
done
pcs property set maintenance-mode=false --wait
EOF
scp $helper heat-admin@${FIRST_CONTROLLER}:
ssh heat-admin@${FIRST_CONTROLLER} -- sudo bash $helper
fi
helper=iha-helper-reenable.sh
cat <<EOF > $helper
set -ex
for service in neutron-openvswitch-agent openstack-ceilometer-compute openstack-nova-compute libvirtd; do
systemctl enable \${service}
done
EOF
for node in $COMPUTES; do scp $helper heat-admin@${node}: ; ssh heat-admin@${node} -- sudo bash $helper ; done
================================================
FILE: keepalived/ceilometer-config.md
================================================
Introduction
------------
In terms of high availability, the Ceilometer central agent deserves special attention. This agent had to run in a single node until the Juno release cycle, since there was no way to coordinate multiple agents and ensure they would not duplicate metrics. Now, multiple central agent instances can run in parallel with workload partitioning among these running instances, using the tooz library with a Redis backend for coordination. See [here](http://docs.openstack.org/admin-guide-cloud/telemetry-data-collection.html#support-for-ha-deployment) for additional information.
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/ceilometer.scenario).
Install software
----------------
yum install -y openstack-ceilometer-api openstack-ceilometer-central openstack-ceilometer-collector openstack-ceilometer-common openstack-ceilometer-alarm python-ceilometer python-ceilometerclient python-redis
Configure ceilometer
--------------------
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_plugin password
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken username ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken password ceilometertest
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken project_name services
openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211
openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3
openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://controller-vip.example.com:5000/v2.0
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest
openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://hacontroller1,hacontroller2,hacontroller3:27017/ceilometer?replicaSet=ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf database max_retries -1
# keep last 5 days data only (value is in secs)
openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000
openstack-config --set /etc/ceilometer/ceilometer.conf api host 192.168.1.22X
Configure coordination URL
--------------------------
openstack-config --set /etc/ceilometer/ceilometer.conf coordination backend_url 'redis://hacontroller1:26379?sentinel=mymaster&sentinel_fallback=hacontroller2:26379&sentinel_fallback=hacontroller3:26379'
Enable and start Ceilometer services, open firewall ports
---------------------------------------------------------
systemctl start openstack-ceilometer-central
systemctl enable openstack-ceilometer-central
systemctl start openstack-ceilometer-collector
systemctl enable openstack-ceilometer-collector
systemctl start openstack-ceilometer-api
systemctl enable openstack-ceilometer-api
systemctl start openstack-ceilometer-alarm-evaluator
systemctl enable openstack-ceilometer-alarm-evaluator
systemctl start openstack-ceilometer-alarm-notifier
systemctl enable openstack-ceilometer-alarm-notifier
systemctl start openstack-ceilometer-notification
systemctl enable openstack-ceilometer-notification
firewall-cmd --add-port=8777/tcp
firewall-cmd --add-port=8777/tcp --permanent
firewall-cmd --add-port=4952/udp
firewall-cmd --add-port=4952/udp --permanent
Tests
-----
On any node:
. /root/keystonerc_admin
for m in storage.objects image network volume instance ; do ceilometer sample-list -m $m | tail -2 ; done
================================================
FILE: keepalived/cinder-config.md
================================================
Introduction
------------
Cinder will be configured in this example to use the NFS backend driver. Instructions for any other backend driver will only differ in the `volume_driver` config option and any driver-specific options.
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/cinder.scenario).
Install software
----------------
yum install -y openstack-cinder openstack-utils openstack-selinux python-memcached
Configure
---------
openstack-config --set /etc/cinder/cinder.conf database connection mysql://cinder:cindertest@controller-vip.example.com/cinder
openstack-config --set /etc/cinder/cinder.conf database max_retries -1
openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_plugin password
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken username cinder
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken password cindertest
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_name services
openstack-config --set /etc/cinder/cinder.conf DEFAULT notification_driver messaging
openstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder
openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_host controller-vip.example.com
openstack-config --set /etc/cinder/cinder.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211
openstack-config --set /etc/cinder/cinder.conf DEFAULT host rhos7-cinder
openstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen 192.168.1.22X
openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3
openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/cinder/cinder.conf keymgr encryption_auth_url http://controller-vip.example.com:5000/v3
**Note:** We are setting a single "host" entry for all nodes, this is related to the A/P issues with cinder-volume.
Configure NFS driver
--------------------
# Choose whatever NFS share is used
cat > /etc/cinder/nfs_exports << EOF
192.168.1.4:/volumeUSB1/usbshare/openstack/cinder
EOF
chown root:cinder /etc/cinder/nfs_exports
chmod 0640 /etc/cinder/nfs_exports
openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfs_exports
openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_sparsed_volumes true
openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_mount_options v3
openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.nfs.NfsDriver
Manage DB
---------
On node 1:
su cinder -s /bin/sh -c "cinder-manage db sync"
Start services
--------------
On node 1:
systemctl start openstack-cinder-api
systemctl start openstack-cinder-scheduler
systemctl start openstack-cinder-volume
systemctl enable openstack-cinder-api
systemctl enable openstack-cinder-scheduler
systemctl enable openstack-cinder-volume
**Note:** If this node crashes, it should be manually started on another node. Refer to [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1193229) for additional information.
On nodes 2 and 3:
systemctl start openstack-cinder-api
systemctl start openstack-cinder-scheduler
systemctl enable openstack-cinder-api
systemctl enable openstack-cinder-scheduler
Open firewall ports
-------------------
On all nodes:
firewall-cmd --add-port=8776/tcp
firewall-cmd --add-port=8776/tcp --permanent
Test
----
On any node:
. /root/keystonerc_demo
cinder create --display-name test 1
cinder extend test 4
cinder delete test
================================================
FILE: keepalived/compute-node.md
================================================
Introduction
------------
The compute node implementation is relatively straightforward, compared to the controller node. It will only be necessary to configure:
- OpenvSwitch and Neutron OpenvSwitch agent
- Nova compute
- Ceilometer compute agent
You can find a phd scenario file [here](phd-setup/compute.scenario).
Environment description
-----------------------
The network configuration was previously discussed in the [Controller node implementation](controller-node.md) section:
- The external network is used by the Neutron floating IPs, and for any external access. The hypervisor nodes (hacompute1 and hacompute2) do not need to be connected to this network, but in the demo setup they are connected for testing purposes.
- The internal network will carry all other traffic: API traffic, tenant networks and storage traffic.
- The router providing connectivity between the internal and external networks is only needed if Trove and/or Sahara are being deployed.
Remember this is a minimum test setup. Any production setup should separate internal and external API traffic, tenant networks and storage traffic in different network segments.
Compute node configuration
--------------------------
The following commands should be executed on each compute node to be added to the installation. There is no configuration required on the controller nodes, meaning compute nodes can be added anytime.
### Install software
yum install -y openstack-nova-compute openstack-utils python-cinder openstack-neutron-openvswitch openstack-ceilometer-compute openstack-neutron openstack-selinux
### Enable OpenvSwitch, start daemon and create integration bridge
systemctl enable openvswitch
systemctl start openvswitch
ovs-vsctl add-br br-int
### Configure Nova compute
openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211
openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.1.22X
openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 0.0.0.0
openstack-config --set /etc/nova/nova.conf vnc novncproxy_base_url http://controller-vip.example.com:6080/vnc_auto.html
openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@controller-vip.example.com/nova
openstack-config --set /etc/nova/nova.conf database max_retries -1
openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/nova/nova.conf glance host controller-vip.example.com
openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API
openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver
openstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver
openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron
openstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL
openstack-config --set /etc/nova/nova.conf conductor use_local false
openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3
openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True
openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True
openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest
openstack-config --set /etc/nova/nova.conf neutron url http://controller-vip.example.com:9696/
openstack-config --set /etc/nova/nova.conf neutron project_domain_id default
openstack-config --set /etc/nova/nova.conf neutron project_name services
openstack-config --set /etc/nova/nova.conf neutron user_domain_id default
openstack-config --set /etc/nova/nova.conf neutron username neutron
openstack-config --set /etc/nova/nova.conf neutron password neutrontest
openstack-config --set /etc/nova/nova.conf neutron auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/nova/nova.conf neutron auth_uri http://controller-vip.example.com:5000/
openstack-config --set /etc/nova/nova.conf neutron auth_plugin password
openstack-config --set /etc/nova/nova.conf neutron region_name regionOne
openstack-config --set /etc/nova/nova.conf libvirt nfs_mount_options v3
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_plugin password
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/nova/api-paste.ini filter:authtoken username compute
openstack-config --set /etc/nova/api-paste.ini filter:authtoken password novatest
openstack-config --set /etc/nova/api-paste.ini filter:authtoken project_name services
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_uri http://controller-vip.example.com:5000/
Only run the following command if you are creating a test environment where your hypervisors will be virtual machines.
openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu
### Configure Neutron on compute node
openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_plugin password
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password neutrontest
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name services
openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3
openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent tunnel_types vxlan
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 4789
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs enable_tunneling True
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_id_ranges 1:1000
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tenant_network_type vxlan
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs integration_bridge br-int
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_bridge br-tun
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs local_ip 192.168.1.22X
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent l2_population False
### Configure Ceilometer on compute node
openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit True
openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit_period hour
openstack-config --set /etc/nova/nova.conf DEFAULT notify_on_state_change vm_and_task_state
openstack-config --set /etc/nova/nova.conf DEFAULT notification_driver nova.openstack.common.notifier.rpc_notifier
sed -i -e 's/nova.openstack.common.notifier.rpc_notifier/nova.openstack.common.notifier.rpc_notifier\nnotification_driver = ceilometer.compute.nova_notifier/g' /etc/nova/nova.conf
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_plugin password
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken username ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken password ceilometertest
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken project_name services
openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211
openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3
openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://controller-vip.example.com:5000/v2.0
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest
openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://hacontroller1,hacontroller2,hacontroller3:27017/ceilometer?replicaSet=ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf database connection max_retries -1
# keep last 5 days data only (value is in secs)
openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000
**Note:** the following SELinux boolean allows QEMU to use NFS for Cinder volumes. A different boolean may apply if using another type of backend storage.
setsebool -P virt_use_nfs 1
### Set kernel TCP keepalive parameters
cat > /etc/sysctl.d/tcpka.conf << EOF
net.ipv4.tcp_keepalive_intvl = 1
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_time = 5
EOF
sysctl -p /etc/sysctl.d/tcpka.conf
### Enable and start services, open firewall ports
**Note:** we are enabling ports 5900-5999 for VNC access. If the compute node could host more than 100 VMs, we have to extend this range.
systemctl start libvirtd
systemctl start neutron-openvswitch-agent
systemctl enable neutron-openvswitch-agent
systemctl enable neutron-ovs-cleanup
systemctl start openstack-ceilometer-compute
systemctl enable openstack-ceilometer-compute
systemctl start openstack-nova-compute
systemctl enable openstack-nova-compute
firewall-cmd --add-port=4789/udp
firewall-cmd --add-port=4789/udp --permanent
firewall-cmd --add-port=5900-5999/tcp
firewall-cmd --add-port=5900-5999/tcp --permanent
================================================
FILE: keepalived/controller-node.md
================================================
Introduction
------------
Controller nodes will run all OpenStack services in this architecture, while compute nodes will only act as hypervisors.
Environment description
-----------------------
### Network setup
The basic requirements for this environment include 5 nodes, with the network setup described in the following diagram:
- The external network is used by the Neutron floating IPs, and for any external access. The hypervisor nodes (hacompute1 and hacompute2) do not need to be connected to this network, but in the demo setup they are connected for testing purposes.
- The internal network will carry all other traffic: API traffic, tenant networks and storage traffic.
- A router will provide connectivity for the controller nodes to the floating IP network as required by Sahara for instance management. In this configuration example, it will also allow virtual machines to access the controller nodes, allowing Trove instances access to the RabbitMQ server (**please note this is not recommended for a production setup**, refer to [trove-config.md](the Trove section) for details). If neitner Sahara or Trove are used, the router is not required.
Please note this is a minimum test setup. Any production setup should separate internal and external API traffic, tenant networks and storage traffic in different network segments, and route traffic between instances and the controller nodes via a firewall.
### Node setup
The following table provides the system details for the environment used during testing.
| Hostname | NIC 1 | NIC 2 | Disk 1 | Disk 2 |
|--------------------|--------------------------------------|-------------------|------------------|-----------------|
|hacontroller1 |eth0: no IP, used for provider network|eth1: 192.168.1.221| 60 GB (/dev/vda) | 8 GB (/dev/vdb) |
|hacontroller2 |eth0: no IP, used for provider network|eth1: 192.168.1.222| 60 GB (/dev/vda) | 8 GB (/dev/vdb) |
|hacontroller3 |eth0: no IP, used for provider network|eth1: 192.168.1.223| 60 GB (/dev/vda) | 8 GB (/dev/vdb) |
|controller-vip (virtual hostname)| |eth1: 192.168.1.220| | |
|hacompute1 |eth0: 10.10.10.224 (-priv) |eth1: 192.168.1.224| 60 GB (/dev/vda) | |
|hacompute2 |eth0: 10.10.10.225 (-priv) |eth1: 192.168.1.225| 60 GB (/dev/vda) | |
All nodes have SELinux set to *Enforcing* and an active firewall.
On the detailed installation notes, remember to substitute any occurrence of 192.168.1.22X with the IP of the node being configured.
#### Single vs multiple VIPs
Please note that the current document uses a single virtual IP for the whole stack of OpenStack API services. It is also possible to use multiple virtual IPs, one per API service. Using multiple virtual IPs will allow for load distribution between the three HAProxy instances. In contrast, the requirements for available IP addresses increase, and the initial configuration can be slightly more complex if performed manually.
#### Base operating system installation
All nodes start from a *minimal* CentOS 7 or RHEL 7 installation, then enabling the required software channels.
- For RDO Liberty, follow the steps specified in the [RDO wiki](https://openstack.redhat.com/Repositories)
**Note:** For now, use [this release RPM](http://rdoproject.org/repos/openstack-liberty/rdo-release-liberty.rpm) to setup the required repositories.
- For the [Red Hat Enterprise Linux OpenStack Platform](http://www.redhat.com/openstack), run the following commands to enable the required repositories:
<pre>subscription-manager register --username=${RHSMUSER} --password=${RHSMPASS}
subscription-manager attach --auto
subscription-manager repos --disable \*
subscription-manager repos --enable rhel-7-server-rpms
subscription-manager repos --enable rhel-7-server-rh-common-rpms
subscription-manager repos --enable rhel-7-server-openstack-7.0-rpms
yum -y update
reboot</pre>
The systems also had NetworkManager disabled:
systemctl disable NetworkManager
systemctl enable network
If the installed system cannot access the gateway, be sure there is a “GATEWAY=xxx” entry in the relevant ifcfg file:
NAME=eth0
...
GATEWAY=192.168.1.1
**Note:** It is very important to make sure that NTP is correctly configured on all nodes. Failure to do so can cause environment instability.
The following phd scenario files have been created:
- A file with the [VM creation and initial setup](phd-setup/hypervisors.scenario)
- A file with some [basic system preparation tasks](phd-setup/serverprep.scenario)
#### Configuration steps
**NOTE:** before moving on, please remember that the following instructions have been created for the above-mentioned scenario (3 controller nodes, with a specific network setup). If you are using this guide to implement your own environment, pay close attention to any IP substitution and other changes that may apply to your environment. Also, do not forget to set sensible passwords.
The configuration steps can be divided into:
- Installing/configuring/enabling all core non-Openstack services
- [HAProxy](haproxy-config.md)
- [Galera](galera-config.md)
- [RabbitMQ](rabbitmq-config.md)
- [Memcached](memcached-config.md)
- [Redis](redis-config.md)
- [MongoDB](mongodb-config.md)
- [Keepalived](keepalived-config.md)
- Installing/configuring/enabling all core Openstack services
- [Keystone](keystone-config.md)
- [Glance](glance-config.md)
- [Cinder](cinder-config.md)
- [Swift](swift-config.md)
- [Neutron](neutron-config.md)
- [Nova](nova-config.md)
- [Ceilometer](ceilometer-config.md)
- [Heat](heat-config.md)
- [Horizon](horizon-config.md)
- [Trove](trove-config.md)
- [Sahara](sahara-config.md)
On each section, a link to a [phd-based](https://github.com/davidvossel/phd) scenario file is provided, as a reference.
================================================
FILE: keepalived/galera-bootstrap.md
================================================
Introduction
------------
Here is an outline of the steps needed to re-establish/bootstrap Galera quorum.
1. Determine loss of quorum
2. Determine systems with last activity
3. Start first DB on first node
4. Start DB on remaining nodes
Determine loss of quorum
------------------------
Confirm in the */var/log/mariadb/mariadb.log* on each system, looking for Errors
140929 11:25:40 [ERROR] WSREP: Local state seqno (1399488) is greater than group seqno (10068): states diverged. Aborting to avoid potential data loss. Remove '/var/lib/mysql//grastate.dat' file and restart if you wish to continue. (FATAL)
140929 11:25:40 [ERROR] Aborting
[root@ospha2 ~]#
Also the clustercheck command should so that there are some systems not in sync
[root@ospha2 ~]# clustercheck
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 36
Galera cluster node is not synced.
[root@ospha2 ~]#
Determine systems with last activity
------------------------------------
In this section we attempt to determine which system or systems has the highest valid sequence number for the for the latest UUID.
### Orderly shutdown
If the cluster shutdown correctly the `/var/lib/mysql/grastate.dat` file will have positive numbers for the seqno. Note which system or systems have the greatest seqno. However, if any system has a `-1` value, that indicates the shutdown was not clean and another method to determine the seqno is needed.
[root@ospha2 ~]# cat /var/lib/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid: b048715d-4369-11e4-b7ef-af1999a6c989
seqno: -1
cert_index:
[root@ospha2 ~]#
### Disorderly Shutdown
The seqno is in the `/var/log/mariadb/mariadb.log` file. Search for lines with "Found save state", ignoring any -1 values. The last value on each line is in the form UUID:seqno.
[root@ospha1 ~]# tail -n 1000 /var/log/mariadb/mariadb.log | grep "Found saved state" | grep -v ":-1"
140923 17:49:19 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:2229
140924 15:37:13 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:2248
140929 11:24:26 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:10060
[root@ospha1 ~]#
[root@ospha2 ~]# tail -n 1000 /var/log/mariadb/mariadb.log | grep "Found saved state" | grep -v ":-1"
140926 14:58:16 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:171535
140929 11:24:28 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:1399488
[root@ospha2 ~]#
[root@ospha3 ~]# tail -n 2000 /var/log/mariadb/mariadb.log | grep "Found saved state" | grep -v ":-1"
140923 17:36:57 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:36
140923 17:43:18 [Note] WSREP: Found saved state: b048715d-4369-11e4-b7ef-af1999a6c989:785
[root@ospha3 ~]#
Notice all servers have the same UUID (b048715d-4369-11e4-b7ef-af1999a6c989), but server *ospha2* has the largest seqno (1399488).
Start first DB on first node
----------------------------
The following command will initiate the Galera cluster. Since ospha2 had the highest seqno, that is the node to start first.
[root@ospha2 ~]# sudo -u mysql /usr/libexec/mysqld --wsrep-cluster-address='gcomm://' &
[1] 1910
[root@ospha2 ~]# 140929 16:31:00 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295
140929 16:31:00 [Warning] Could not increase number of max_open_files to more than 1024 (request: 1835)
/usr/libexec/mysqld: Query cache is disabled (resize or similar command in progress); repeat this command later
Verify that this brought the this node into sync.
[root@ospha2 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32
Galera cluster node is synced.
[root@ospha2 ~]#
Start DB on remaining nodes
---------------------------
On another cluster member, start the database, and then verify this node reports synced.
[root@ospha1 ~]# systemctl start mariadb
[root@ospha1 ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32
Galera cluster node is synced.
[root@ospha1 ~]#
Once `clustercheck` returns 200 on all nodes, restart MariaDB on the first node.
kill <mysql PIDs>
systemctl start mariadb
================================================
FILE: keepalived/galera-config.md
================================================
Introduction
------------
MariaDB with Galera provides synchronous database replication, in an active-active, multi-master environment. High availability for the data itself is managed internally by Galera, while access availability is managed by HAProxy.
It is important to note that, while Galera supports multi-master access, there are known issues in an OpenStack environment due to its use of optimistic locking (see [this post](http://lists.openstack.org/pipermail/openstack-dev/2014-May/035264.html) for details). Due to this limitation, the HAProxy configuration used for Galera sets a single node to be accessed at all times, and a different node will only be used when the original node fails. This effectively turns the Galera configuration into an active-hot standby one.
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/galera.scenario).
Install software
----------------
yum install -y mariadb-galera-server xinetd rsync
Configure cluster check
-----------------------
cat > /etc/sysconfig/clustercheck << EOF
MYSQL_USERNAME="clustercheck"
MYSQL_PASSWORD="redhat"
MYSQL_HOST="localhost"
MYSQL_PORT="3306"
EOF
Start mysqld and create user for cluster check
----------------------------------------------
systemctl start mysqld
mysql -e "CREATE USER 'clustercheck'@'localhost' IDENTIFIED BY 'redhat';"
systemctl stop mysqld
Create Galera configuration
---------------------------
cat > /etc/my.cnf.d/galera.cnf << EOF
[mysqld]
skip-name-resolve=1
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
max_connections=8192
query_cache_size=0
query_cache_type=0
bind_address=192.168.1.22X
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_name="galera_cluster"
wsrep_cluster_address="gcomm://192.168.1.221,192.168.1.222,192.168.1.223"
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_notify_cmd=
wsrep_sst_method=rsync
EOF
Create Galera systemd configuration file, to allow a higher number of files to be opened
----------------------------------------------------------------------------------------
mkdir -p /etc/systemd/system/mariadb.service.d/
cat > /etc/systemd/system/mariadb.service.d/limits.conf << EOF
[Service]
LimitNOFILE=16384
EOF
Configure monitor service (used by HAProxy)
-------------------------------------------
cat > /etc/xinetd.d/galera-monitor << EOF
service galera-monitor
{
port = 9200
disable = no
socket_type = stream
protocol = tcp
wait = no
user = root
group = root
groups = yes
server = /usr/bin/clustercheck
type = UNLISTED
per_source = UNLIMITED
log_on_success =
log_on_failure = HOST
flags = REUSE
}
EOF
Start services and open firewall ports
--------------------------------------
systemctl daemon-reload
systemctl enable xinetd
systemctl start xinetd
systemctl enable haproxy
systemctl start haproxy
firewall-cmd --add-service=mysql
firewall-cmd --add-port=4444/tcp
firewall-cmd --add-port=4567/tcp
firewall-cmd --add-port=4568/tcp
firewall-cmd --add-port=4568/tcp --permanent
firewall-cmd --add-service=mysql --permanent
firewall-cmd --add-port=4567/tcp --permanent
firewall-cmd --add-port=4444/tcp --permanent
firewall-cmd --add-port=9300/tcp
firewall-cmd --add-port=9300/tcp --permanent
firewall-cmd --add-port=9200/tcp
firewall-cmd --add-port=9200/tcp --permanent
Start mariadb cluster
---------------------
**On all nodes:**
systemctl enable mariadb
**On node1:**
sudo -u mysql /usr/libexec/mysqld --wsrep-cluster-address='gcomm://' &
**On node2 and node3:**
systemctl start mariadb
Once clustecheck returns 200 on all nodes, restart galera on node 1:
kill <mysql PIDs>
systemctl start mariadb
Create users and databases
--------------------------
**On node 1:**
mysql
MariaDB [(none)]> use mysql;
MariaDB [mysql]> drop user ''@'hacontroller1.example.com';
MariaDB [mysql]> drop user 'root'@'hacontroller1.example.com';
MariaDB [mysql]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED by 'mysqltest' WITH GRANT OPTION;
MariaDB [mysql]> CREATE DATABASE keystone;
MariaDB [mysql]> GRANT ALL ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'keystonetest';
MariaDB [mysql]> CREATE DATABASE glance;
MariaDB [mysql]> GRANT ALL ON glance.* TO 'glance'@'%' IDENTIFIED BY 'glancetest';
MariaDB [mysql]> CREATE DATABASE cinder;
MariaDB [mysql]> GRANT ALL ON cinder.* TO 'cinder'@'%' IDENTIFIED BY 'cindertest';
MariaDB [mysql]> CREATE DATABASE neutron;
MariaDB [mysql]> GRANT ALL ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'neutrontest';
MariaDB [mysql]> CREATE DATABASE nova;
MariaDB [mysql]> GRANT ALL ON nova.* TO 'nova'@'%' IDENTIFIED BY 'novatest';
MariaDB [mysql]> CREATE DATABASE heat;
MariaDB [mysql]> GRANT ALL ON heat.* TO 'heat'@'%' IDENTIFIED BY 'heattest';
MariaDB [mysql]> CREATE DATABASE sahara;
MariaDB [mysql]> GRANT ALL ON sahara.* TO 'sahara'@'%' IDENTIFIED BY 'saharatest';
MariaDB [mysql]> CREATE DATABASE trove;
MariaDB [mysql]> GRANT ALL ON trove.* TO 'trove'@'%' IDENTIFIED BY 'trovetest';
MariaDB [mysql]> FLUSH PRIVILEGES;
MariaDB [mysql]> quit
mysqladmin flush-hosts
================================================
FILE: keepalived/glance-config.md
================================================
Introduction
------------
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/glance.scenario).
Install software
----------------
yum install -y openstack-glance openstack-utils openstack-selinux nfs-utils
Configure glance-api and glance-registry
----------------------------------------
openstack-config --set /etc/glance/glance-api.conf database connection mysql://glance:glancetest@controller-vip.example.com/glance
openstack-config --set /etc/glance/glance-api.conf database max_retries -1
openstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone
openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/
openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_plugin password
openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/glance/glance-api.conf keystone_authtoken username glance
openstack-config --set /etc/glance/glance-api.conf keystone_authtoken password glancetest
openstack-config --set /etc/glance/glance-api.conf keystone_authtoken project_name services
openstack-config --set /etc/glance/glance-api.conf DEFAULT notification_driver messaging
openstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host 192.168.1.22X
openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host controller-vip.example.com
openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3
openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/glance/glance-registry.conf database connection mysql://glance:glancetest@controller-vip.example.com/glance
openstack-config --set /etc/glance/glance-registry.conf database max_retries -1
openstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone
openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/
openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_plugin password
openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken username glance
openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken password glancetest
openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken project_name services
openstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host 192.168.1.22X
Manage DB
---------
On node 1:
su glance -s /bin/sh -c "glance-manage db_sync"
Configure backend
-----------------
For this setup, NFS will be used. Add the NFS mount to `/etc/fstab`, making sure it is mounted on `/var/lib/glance`. Be aware the last two columns in `fstab` need to be "0 0" on RHEL/CentOS 7, due to [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1120367). You may also find [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1203820) if using NFS v3 shares.
Also, note there is currently a known SELinux issue when using an NFS backend for Glance. See [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1219406) for a description and fix.
On all nodes:
chown glance:nobody /var/lib/glance
Start services and open firewall ports
--------------------------------------
systemctl start openstack-glance-registry
systemctl start openstack-glance-api
systemctl enable openstack-glance-registry
systemctl enable openstack-glance-api
firewall-cmd --add-port=9191/tcp
firewall-cmd --add-port=9191/tcp --permanent
firewall-cmd --add-port=9292/tcp
firewall-cmd --add-port=9292/tcp --permanent
Test
----
On any node:
. /root/keystonerc_admin
wget http://download.cirros-cloud.net/0.3.3/cirros-0.3.3-x86_64-disk.img
glance image-create --name "cirros" --disk-format qcow2 --container-format bare --file cirros-0.3.3-x86_64-disk.img --visibility public
glance image-list
================================================
FILE: keepalived/haproxy-config.md
================================================
Introduction
------------
A load-balancing proxy is used to provide scalability and load balancing for OpenStack API services and some of the supporting services. All requests will be distributed using a round-robin algorithm between all available controller nodes, and HAProxy itself will monitor the availability of each service. In case a node or service goes down, HAProxy will remove it from the active pool after a timeout has been reached, tipically a few seconds.
The following commands will be executed on all controller nodes.
You can find a phd scenario file [here](phd-setup/lb.scenario).
Install packages
----------------
yum install -y haproxy openstack-selinux
Allow binding to non-local IPs
------------------------------
echo net.ipv4.ip_nonlocal_bind=1 >> /etc/sysctl.d/haproxy.conf
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
Configure HAProxy
-----------------
cat > /etc/haproxy/haproxy.cfg << EOF
global
daemon
stats socket /var/lib/haproxy/stats
defaults
mode tcp
maxconn 10000
timeout connect 5s
timeout client 30s
timeout server 30s
listen monitor
bind 192.168.1.220:9300
mode http
monitor-uri /status
stats enable
stats uri /admin
stats realm Haproxy\ Statistics
stats auth root:redhat
stats refresh 5s
frontend vip-db
bind 192.168.1.220:3306
timeout client 90m
default_backend db-vms-galera
backend db-vms-galera
option httpchk
stick-table type ip size 1000
stick on dst
timeout server 90m
server rhos8-node1 192.168.1.221:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions
server rhos8-node2 192.168.1.222:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions
server rhos8-node3 192.168.1.223:3306 check inter 1s port 9200 backup on-marked-down shutdown-sessions
# Note the RabbitMQ entry is only needed for CloudForms compatibility
# and should be removed in the future
frontend vip-rabbitmq
option clitcpka
bind 192.168.1.220:5672
timeout client 900m
default_backend rabbitmq-vms
backend rabbitmq-vms
option srvtcpka
balance roundrobin
timeout server 900m
server rhos8-node1 192.168.1.221:5672 check inter 1s
server rhos8-node2 192.168.1.222:5672 check inter 1s
server rhos8-node3 192.168.1.223:5672 check inter 1s
frontend vip-keystone-admin
bind 192.168.1.220:35357
default_backend keystone-admin-vms
timeout client 600s
backend keystone-admin-vms
balance roundrobin
timeout server 600s
server rhos8-node1 192.168.1.221:35357 check inter 1s on-marked-down shutdown-sessions
server rhos8-node2 192.168.1.222:35357 check inter 1s on-marked-down shutdown-sessions
server rhos8-node3 192.168.1.223:35357 check inter 1s on-marked-down shutdown-sessions
frontend vip-keystone-public
bind 192.168.1.220:5000
default_backend keystone-public-vms
timeout client 600s
backend keystone-public-vms
balance roundrobin
timeout server 600s
server rhos8-node1 192.168.1.221:5000 check inter 1s on-marked-down shutdown-sessions
server rhos8-node2 192.168.1.222:5000 check inter 1s on-marked-down shutdown-sessions
server rhos8-node3 192.168.1.223:5000 check inter 1s on-marked-down shutdown-sessions
frontend vip-glance-api
bind 192.168.1.220:9191
default_backend glance-api-vms
backend glance-api-vms
balance roundrobin
server rhos8-node1 192.168.1.221:9191 check inter 1s
server rhos8-node2 192.168.1.222:9191 check inter 1s
server rhos8-node3 192.168.1.223:9191 check inter 1s
frontend vip-glance-registry
bind 192.168.1.220:9292
default_backend glance-registry-vms
backend glance-registry-vms
balance roundrobin
server rhos8-node1 192.168.1.221:9292 check inter 1s
server rhos8-node2 192.168.1.222:9292 check inter 1s
server rhos8-node3 192.168.1.223:9292 check inter 1s
frontend vip-cinder
bind 192.168.1.220:8776
default_backend cinder-vms
backend cinder-vms
balance roundrobin
server rhos8-node1 192.168.1.221:8776 check inter 1s
server rhos8-node2 192.168.1.222:8776 check inter 1s
server rhos8-node3 192.168.1.223:8776 check inter 1s
frontend vip-swift
bind 192.168.1.220:8080
default_backend swift-vms
backend swift-vms
balance roundrobin
server rhos8-node1 192.168.1.221:8080 check inter 1s
server rhos8-node2 192.168.1.222:8080 check inter 1s
server rhos8-node3 192.168.1.223:8080 check inter 1s
frontend vip-neutron
bind 192.168.1.220:9696
default_backend neutron-vms
backend neutron-vms
balance roundrobin
server rhos8-node1 192.168.1.221:9696 check inter 1s
server rhos8-node2 192.168.1.222:9696 check inter 1s
server rhos8-node3 192.168.1.223:9696 check inter 1s
frontend vip-nova-vnc-novncproxy
bind 192.168.1.220:6080
default_backend nova-vnc-novncproxy-vms
backend nova-vnc-novncproxy-vms
balance roundrobin
timeout tunnel 1h
server rhos8-node1 192.168.1.221:6080 check inter 1s
server rhos8-node2 192.168.1.222:6080 check inter 1s
server rhos8-node3 192.168.1.223:6080 check inter 1s
frontend nova-metadata-vms
bind 192.168.1.220:8775
default_backend nova-metadata-vms
backend nova-metadata-vms
balance roundrobin
server rhos8-node1 192.168.1.221:8775 check inter 1s
server rhos8-node2 192.168.1.222:8775 check inter 1s
server rhos8-node3 192.168.1.223:8775 check inter 1s
frontend vip-nova-api
bind 192.168.1.220:8774
default_backend nova-api-vms
backend nova-api-vms
balance roundrobin
server rhos8-node1 192.168.1.221:8774 check inter 1s
server rhos8-node2 192.168.1.222:8774 check inter 1s
server rhos8-node3 192.168.1.223:8774 check inter 1s
frontend vip-horizon
bind 192.168.1.220:80
timeout client 180s
default_backend horizon-vms
backend horizon-vms
balance roundrobin
timeout server 180s
mode http
cookie SERVERID insert indirect nocache
server rhos8-node1 192.168.1.221:80 check inter 1s cookie rhos8-horizon1 on-marked-down shutdown-sessions
server rhos8-node2 192.168.1.222:80 check inter 1s cookie rhos8-horizon2 on-marked-down shutdown-sessions
server rhos8-node3 192.168.1.223:80 check inter 1s cookie rhos8-horizon3 on-marked-down shutdown-sessions
frontend vip-heat-cfn
bind 192.168.1.220:8000
default_backend heat-cfn-vms
backend heat-cfn-vms
balance roundrobin
server rhos8-node1 192.168.1.221:8000 check inter 1s
server rhos8-node2 192.168.1.222:8000 check inter 1s
server rhos8-node3 192.168.1.223:8000 check inter 1s
frontend vip-heat-cloudw
bind 192.168.1.220:8003
default_backend heat-cloudw-vms
backend heat-cloudw-vms
balance roundrobin
server rhos8-node1 192.168.1.221:8003 check inter 1s
server rhos8-node2 192.168.1.222:8003 check inter 1s
server rhos8-node3 192.168.1.223:8003 check inter 1s
frontend vip-heat-srv
bind 192.168.1.220:8004
default_backend heat-srv-vms
backend heat-srv-vms
balance roundrobin
server rhos8-node1 192.168.1.221:8004 check inter 1s
server rhos8-node2 192.168.1.222:8004 check inter 1s
server rhos8-node3 192.168.1.223:8004 check inter 1s
frontend vip-ceilometer
bind 192.168.1.220:8777
timeout client 90s
default_backend ceilometer-vms
backend ceilometer-vms
balance roundrobin
timeout server 90s
server rhos8-node1 192.168.1.221:8777 check inter 1s
server rhos8-node2 192.168.1.222:8777 check inter 1s
server rhos8-node3 192.168.1.223:8777 check inter 1s
frontend vip-sahara
bind 192.168.1.220:8386
default_backend sahara-vms
backend sahara-vms
balance roundrobin
server rhos8-node1 192.168.1.221:8386 check inter 1s
server rhos8-node2 192.168.1.222:8386 check inter 1s
server rhos8-node3 192.168.1.223:8386 check inter 1s
frontend vip-trove
bind 192.168.1.220:8779
default_backend trove-vms
backend trove-vms
balance roundrobin
server rhos8-node1 192.168.1.221:8779 check inter 1s
server rhos8-node2 192.168.1.222:8779 check inter 1s
server rhos8-node3 192.168.1.223:8779 check inter 1s
EOF
Note we are **not** starting haproxy yet.
Once HAproxy is started, you can monitor progress of your service configuration by going to [<http://controller-vip.example.com:9300/admin>](http://controller-vip.example.com:9300/admin) (root/redhat, remember to set a sensible password). With this you will be able to see which services are running on which nodes, as seen by HAproxy.
================================================
FILE: keepalived/heat-config.md
================================================
Introduction
------------
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/heat.scenario).
Install software
----------------
yum install -y openstack-heat-engine openstack-heat-api openstack-heat-api-cfn openstack-heat-api-cloudwatch python-heatclient openstack-utils python-glanceclient
Configure Heat domain
---------------------
To allow non-admin users to create Heat stacks, a Keystone domain needs to be created. Run the following commands to create the Heat domain, and configure Heat to use it.
On node 1:
. /root/keystonerc_admin
openstack role create heat_stack_user
openstack token issue
Take note of the token ID issued, then:
openstack --os-token=${TOKEN_ID} --os-url=http://controller-vip.example.com:5000/v3 --os-identity-api-version=3 domain create heat --description "Owns users and projects created by heat"
openstack --os-token=${TOKEN_ID} --os-url=http://controller-vip.example.com:5000/v3 --os-identity-api-version=3 user create --password heattest --domain heat --description "Manages users and projects created by heat" heat_domain_admin
openstack --os-token=${TOKEN_ID} --os-url=http://controller-vip.example.com:5000/v3 --os-identity-api-version=3 role add --user heat_domain_admin --domain heat admin
On all nodes:
openstack-config --set /etc/heat/heat.conf DEFAULT stack_domain_admin_password heattest
openstack-config --set /etc/heat/heat.conf DEFAULT stack_domain_admin heat_domain_admin
openstack-config --set /etc/heat/heat.conf DEFAULT stack_user_domain_name heat
Configure Heat
--------------
openstack-config --set /etc/heat/heat.conf database connection mysql://heat:heattest@controller-vip.example.com/heat
openstack-config --set /etc/heat/heat.conf database max_retries -1
openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/
openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_plugin password
openstack-config --set /etc/heat/heat.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/heat/heat.conf keystone_authtoken username heat
openstack-config --set /etc/heat/heat.conf keystone_authtoken password heattest
openstack-config --set /etc/heat/heat.conf keystone_authtoken project_name services
openstack-config --set /etc/heat/heat.conf keystone_authtoken keystone_ec2_uri http://controller-vip.example.com:35357/v2.0
openstack-config --set /etc/heat/heat.conf keystone_authtoken identity_uri http://controller-vip.example.com:35357
openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_user heat
openstack-config --set /etc/heat/heat.conf keystone_authtoken admin_password heattest
openstack-config --set /etc/heat/heat.conf ec2authtoken auth_uri http://controller-vip.example.com:5000/v2.0
openstack-config --set /etc/heat/heat.conf DEFAULT memcache_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211
openstack-config --set /etc/heat/heat.conf heat_api bind_host 192.168.1.22X
openstack-config --set /etc/heat/heat.conf heat_api_cfn bind_host 192.168.1.22X
openstack-config --set /etc/heat/heat.conf heat_api_cloudwatch bind_host 192.168.1.22X
openstack-config --set /etc/heat/heat.conf DEFAULT heat_metadata_server_url controller-vip.example.com:8000
openstack-config --set /etc/heat/heat.conf DEFAULT heat_waitcondition_server_url controller-vip.example.com:8000/v1/waitcondition
openstack-config --set /etc/heat/heat.conf DEFAULT heat_watch_server_url controller-vip.example.com:8003
openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3
openstack-config --set /etc/heat/heat.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/heat/heat.conf DEFAULT rpc_backend rabbit
openstack-config --set /etc/heat/heat.conf DEFAULT notification_driver heat.openstack.common.notifier.rpc_notifier
openstack-config --set /etc/heat/heat.conf DEFAULT enable_cloud_watch_lite false
Manage DB
---------
On node 1:
su heat -s /bin/sh -c "heat-manage db_sync"
Start services, open firewall ports
-----------------------------------
On all nodes:
systemctl start openstack-heat-api
systemctl start openstack-heat-api-cfn
systemctl start openstack-heat-api-cloudwatch
systemctl start openstack-heat-engine
systemctl enable openstack-heat-api
systemctl enable openstack-heat-api-cfn
systemctl enable openstack-heat-api-cloudwatch
systemctl enable openstack-heat-engine
firewall-cmd --add-port=8000/tcp
firewall-cmd --add-port=8000/tcp --permanent
firewall-cmd --add-port=8003/tcp
firewall-cmd --add-port=8003/tcp --permanent
firewall-cmd --add-port=8004/tcp
firewall-cmd --add-port=8004/tcp --permanent
================================================
FILE: keepalived/horizon-config.md
================================================
Introduction
------------
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/horizon.scenario).
Install software
----------------
yum install -y mod_wsgi httpd mod_ssl python-memcached openstack-dashboard
Set secret key
--------------
On node 1:
openssl rand -hex 10
Take note of the generated random value, then on all nodes:
sed -i -e "s#SECRET_KEY.*#SECRET_KEY = 'VALUE'#g#" /etc/openstack-dashboard/local_settings
Configure local\_settings and httpd.conf
----------------------------------------
sed -i -e "s#ALLOWED_HOSTS.*#ALLOWED_HOSTS = ['*',]#g" \
-e "s#^CACHES#SESSION_ENGINE = 'django.contrib.sessions.backends.cache'\nCACHES#g#" \
-e "s#locmem.LocMemCache'#memcached.MemcachedCache',\n\t'LOCATION' : [ 'hacontroller1:11211', 'hacontroller2:11211', 'hacontroller3:11211', ]#g" \
-e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = "controller-vip.example.com"#g' \
-e "s#^LOCAL_PATH.*#LOCAL_PATH = '/var/lib/openstack-dashboard'#g" \
/etc/openstack-dashboard/local_settings
Restart httpd and open firewall port
------------------------------------
systemctl daemon-reload
systemctl restart httpd
firewall-cmd --add-port=80/tcp
firewall-cmd --add-port=80/tcp --permanent
================================================
FILE: keepalived/keepalived-config.md
================================================
Introduction
------------
[Keepalived](http://www.keepalived.org/) provides simple and robust facilities for load balancing and high-availability to Linux system and Linux based infrastructures. In this highly available OpenStack architecture, it is used to provide high availability to the virtual IP(s) used by HAProxy. High-availability is achieved by VRRP protocol, a fundamental brick for router failover.
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/keepalived.scenario).
Install software
----------------
yum -y install keepalived psmisc
Create configuration file
-------------------------
On all nodes:
cat > /etc/keepalived/keepalived.conf << EOF
vrrp_script chk_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
}
vrrp_instance VI_PUBLIC {
interface eth1
state BACKUP
virtual_router_id 52
priority 101
virtual_ipaddress {
192.168.1.220 dev eth1
}
track_script {
chk_haproxy
}
# Avoid failback
nopreempt
}
vrrp_sync_group VG1
group {
VI_PUBLIC
}
EOF
Open firewall rules and start services
--------------------------------------
On all nodes:
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -i eth1 -d 224.0.0.0/8 -j ACCEPT
firewall-cmd --direct --perm --add-rule ipv4 filter INPUT 0 -i eth1 -d 224.0.0.0/8 -j ACCEPT
systemctl start keepalived
systemctl enable keepalived
================================================
FILE: keepalived/keystone-config.md
================================================
Introduction
------------
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/keystone.scenario).
Install software
----------------
yum install -y openstack-keystone openstack-utils openstack-selinux httpd mod_wsgi python-openstackclient
Create service token and distribute to the other controllers
------------------------------------------------------------
On node 1:
export SERVICE_TOKEN=$(openssl rand -hex 10)
echo $SERVICE_TOKEN > /root/keystone_service_token
scp /root/keystone_service_token root@hacontroller2:/root
scp /root/keystone_service_token root@hacontroller3:/root
Configure Apache web server for Keystone
----------------------------------------
**NOTE:** running Keystone under eventlet has been deprecated as of the Kilo release. Support for utilizing eventlet will be removed as of the M-release. Thus, instructions are provided to run Keystone under the Apache web server, as a WSGI process.
On all nodes:
cp /usr/share/keystone/wsgi-keystone.conf /etc/httpd/conf.d/
sed -i -e 's/apache2/httpd/g' /etc/httpd/conf.d/wsgi-keystone.conf
sed -i -e 's/VirtualHost \*/VirtualHost 192.168.1.22X/g' /etc/httpd/conf.d/wsgi-keystone.conf
sed -i -e 's/Listen 5000/Listen 192.168.1.22X:5000/g' /etc/httpd/conf.d/wsgi-keystone.conf
sed -i -e 's/Listen 35357/Listen 192.168.1.22X:35357/g' /etc/httpd/conf.d/wsgi-keystone.conf
sed -i -e 's/^Listen.*/Listen 192.168.1.22X:80/g' /etc/httpd/conf/httpd.conf
Configure Keystone
------------------
On all nodes:
export SERVICE_TOKEN=$(cat /root/keystone_service_token)
openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token $SERVICE_TOKEN
openstack-config --set /etc/keystone/keystone.conf DEFAULT rabbit_hosts hacontroller1,hacontroller2,hacontroller3
openstack-config --set /etc/keystone/keystone.conf DEFAULT rabbit_ha_queues true
openstack-config --set /etc/keystone/keystone.conf eventlet_server admin_endpoint 'http://controller-vip.example.com:%(admin_port)s/'
openstack-config --set /etc/keystone/keystone.conf eventlet_server public_endpoint 'http://controller-vip.example.com:%(public_port)s/'
openstack-config --set /etc/keystone/keystone.conf database connection mysql://keystone:keystonetest@controller-vip.example.com/keystone
openstack-config --set /etc/keystone/keystone.conf database max_retries -1
openstack-config --set /etc/keystone/keystone.conf DEFAULT public_bind_host 192.168.1.22X
openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_bind_host 192.168.1.22X
openstack-config --set /etc/keystone/keystone.conf token driver keystone.token.persistence.backends.sql.Token
Create and distribute PKI setup, manage DB
------------------------------------------
On node 1:
keystone-manage pki_setup --keystone-user keystone --keystone-group keystone
chown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/
su keystone -s /bin/sh -c "keystone-manage db_sync"
cd /etc/keystone/ssl
tar cvfz /tmp/keystone_ssl.tgz *
scp /tmp/keystone_ssl.tgz hacontroller2:/tmp
scp /tmp/keystone_ssl.tgz hacontroller3:/tmp
Restore Keystone PKI setup from node 1
--------------------------------------
On nodes 2 and 3:
mkdir -p /etc/keystone/ssl
cd /etc/keystone/ssl
tar xvfz /tmp/keystone_ssl.tgz
chown -R keystone:keystone /var/log/keystone /etc/keystone/ssl/
restorecon -Rv /etc/keystone/ssl
touch /var/log/keystone/keystone.log
chown keystone:keystone /var/log/keystone/keystone.log
Create cron job to flush expired tokens
---------------------------------------
On all nodes:
echo "1 * * * * keystone keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1" >> /etc/crontab
Start services and open firewall ports
--------------------------------------
On all nodes;
firewall-cmd --add-port=5000/tcp
firewall-cmd --add-port=5000/tcp --permanent
firewall-cmd --add-port=35357/tcp
firewall-cmd --add-port=35357/tcp --permanent
systemctl start httpd
systemctl enable httpd
Create endpoints, services and users for all API services
---------------------------------------------------------
On node 1:
export OS_TOKEN=$(cat /root/keystone_service_token)
export OS_URL=http://controller-vip.example.com:35357/v2.0
export OS_REGION_NAME=regionOne
openstack service create --name=keystone --description="Keystone Identity Service" identity
openstack endpoint create --publicurl 'http://controller-vip.example.com:5000/v2.0' --adminurl 'http://controller-vip.example.com:35357/v2.0' --internalurl 'http://controller-vip.example.com:5000/v2.0' --region regionOne keystone
openstack user create --password keystonetest admin
openstack role create admin
openstack project create admin
openstack role add --project admin --user admin admin
openstack user create --password redhat demo
openstack role create _member_
openstack project create demo
openstack role add --project demo --user demo _member_
openstack project create --description "Services Tenant" services
# glance
openstack user create --password glancetest glance
openstack role add --project services --user glance admin
openstack service create --name=glance --description="Glance Image Service" image
openstack endpoint create --publicurl 'http://controller-vip.example.com:9292' --adminurl 'http://controller-vip.example.com:9292' --internalurl 'http://controller-vip.example.com:9292' --region regionOne glance
# cinder
openstack user create --password cindertest cinder
openstack role add --project services --user cinder admin
openstack service create --name=cinder --description="Cinder Volume Service" volume
openstack endpoint create --publicurl "http://controller-vip.example.com:8776/v1/\$(tenant_id)s" --adminurl "http://controller-vip.example.com:8776/v1/\$(tenant_id)s" --internalurl "http://controller-vip.example.com:8776/v1/\$(tenant_id)s" --region regionOne cinder
openstack service create --name=cinderv2 --description="OpenStack Block Storage" volumev2
openstack endpoint create --publicurl "http://controller-vip.example.com:8776/v2/\$(tenant_id)s" --adminurl "http://controller-vip.example.com:8776/v2/\$(tenant_id)s" --internalurl "http://controller-vip.example.com:8776/v2/\$(tenant_id)s" --region regionOne cinderv2
# swift
openstack user create --password swifttest swift
openstack role add --project services --user swift admin
openstack service create --name=swift --description="Swift Storage Service" object-store
openstack endpoint create --publicurl "http://controller-vip.example.com:8080/v1/AUTH_\$(tenant_id)s" --adminurl "http://controller-vip.example.com:8080/v1" --internalurl "http://controller-vip.example.com:8080/v1/AUTH_\$(tenant_id)s" --region regionOne swift
# neutron
openstack user create --password neutrontest neutron
openstack role add --project services --user neutron admin
openstack service create --name=neutron --description="OpenStack Networking Service" network
openstack endpoint create --publicurl "http://controller-vip.example.com:9696" --adminurl "http://controller-vip.example.com:9696" --internalurl "http://controller-vip.example.com:9696" --region regionOne neutron
# nova
openstack user create --password novatest compute
openstack role add --project services --user compute admin
openstack service create --name=compute --description="OpenStack Compute Service" compute
openstack endpoint create --publicurl "http://controller-vip.example.com:8774/v2/\$(tenant_id)s" --adminurl "http://controller-vip.example.com:8774/v2/\$(tenant_id)s" --internalurl "http://controller-vip.example.com:8774/v2/\$(tenant_id)s" --region regionOne compute
# heat
openstack user create --password heattest heat
openstack role add --project services --user heat admin
openstack service create --name=heat --description="Heat Orchestration Service" orchestration
openstack endpoint create --publicurl "http://controller-vip.example.com:8004/v1/%(tenant_id)s" --adminurl "http://controller-vip.example.com:8004/v1/%(tenant_id)s" --internalurl "http://controller-vip.example.com:8004/v1/%(tenant_id)s" --region regionOne heat
openstack service create --name=heat-cfn --description="Heat CloudFormation Service" cloudformation
openstack endpoint create --publicurl "http://controller-vip.example.com:8000/v1" --adminurl "http://controller-vip.example.com:8000/v1" --internalurl "http://controller-vip.example.com:8000/v1" --region regionOne heat-cfn
# ceilometer
openstack user create --password ceilometertest ceilometer
openstack role add --project services --user ceilometer admin
openstack role create ResellerAdmin
openstack role add --project services --user ceilometer ResellerAdmin
openstack service create --name=ceilometer --description="OpenStack Telemetry Service" metering
openstack endpoint create --publicurl "http://controller-vip.example.com:8777" --adminurl "http://controller-vip.example.com:8777" --internalurl "http://controller-vip.example.com:8777" --region regionOne ceilometer
# sahara
openstack user create --password saharatest sahara
openstack role add --project services --user sahara admin
openstack service create --name=sahara --description="Sahara Data Processing" data-processing
openstack endpoint create --publicurl "http://controller-vip.example.com:8386/v1.1/%(tenant_id)s" --adminurl "http://controller-vip.example.com:8386/v1.1/%(tenant_id)s" --internalurl "http://controller-vip.example.com:8386/v1.1/%(tenant_id)s" --region regionOne sahara
# trove
openstack user create --password trovetest trove
openstack role add --project services --user trove admin
openstack service create --name=trove --description="OpenStack Database Service" database
openstack endpoint create --publicurl "http://controller-vip.example.com:8779/v1.0/%(tenant_id)s" --adminurl "http://controller-vip.example.com:8779/v1.0/%(tenant_id)s" --internalurl "http://controller-vip.example.com:8779/v1.0/%(tenant_id)s" --region regionOne trove
Create keystonerc files for simplicity
--------------------------------------
On all nodes:
cat > /root/keystonerc_admin << EOF
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
export OS_PROJECT_NAME=admin
export OS_REGION_NAME=regionOne
export OS_PASSWORD=keystonetest
export OS_AUTH_URL=http://controller-vip.example.com:35357/v2.0/
export PS1='[\u@\h \W(keystone_admin)]\$ '
EOF
cat > /root/keystonerc_demo << EOF
export OS_USERNAME=demo
export OS_TENANT_NAME=demo
export OS_PROJECT_NAME=demo
export OS_REGION_NAME=regionOne
export OS_PASSWORD=redhat
export OS_AUTH_URL=http://controller-vip.example.com:5000/v2.0/
export PS1='[\u@\h \W(keystone_user)]\$ '
EOF
================================================
FILE: keepalived/memcached-config.md
================================================
Introduction
------------
Memcached is a general-purpose distributed memory caching system. It is used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source must be read.
**Note:** Access to memcached is not handled by HAproxy because replicated access is currently only in an experimental state. Instead consumers must be supplied with the full list of hosts running memcached.
The following commands will be executed on all controller nodes.
You can find a phd scenario file [here](phd-setup/memcached.scenario).
Install and enable memcached
----------------------------
yum install -y memcached
systemctl start memcached
systemctl enable memcached
firewall-cmd --add-port=11211/tcp
firewall-cmd --add-port=11211/tcp --permanent
================================================
FILE: keepalived/mongodb-config.md
================================================
Introduction
------------
MongoDB can provide high availability through the use of replica sets. A replica set in MongoDB is a group of mongod processes that maintain the same data set, where one of the nodes is specified as master and the rest as slaves. Clients are explicitly told to connect to the replica set, by specifying all its members. In case of a node failure, the client should transparently reconnect to a surviving replica.
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/mongodb.scenario).
Install packages
----------------
yum install -y mongodb mongodb-server
Listen to external connections, and configure replication set
-------------------------------------------------------------
sed -i -e 's#bind_ip.*#bind_ip = 0.0.0.0#g' /etc/mongod.conf
echo "replSet = ceilometer" >> /etc/mongod.conf
Start services and enable firewall ports
----------------------------------------
systemctl start mongod
systemctl enable mongod
firewall-cmd --add-port=27017/tcp
firewall-cmd --add-port=27017/tcp --permanent
Create replica set
------------------
On node 1:
mongo
> rs.initiate()
> sleep(10000)
> rs.add("hacontroller2.example.com");
> rs.add("hacontroller3.example.com");
And verify:
> rs.status()
Until all nodes show `"stateStr" : "PRIMARY"` or `"stateStr" : "SECONDARY"`, then:
> quit()
================================================
FILE: keepalived/mongodb-recovery.md
================================================
Introduction
------------
MongoDB usually does a good job at re-forming a replica set after a full cluster reboot. In case of any failure, [its documentation](http://docs.mongodb.org/v2.6/tutorial/troubleshoot-replica-sets/) provides an excellent reference on how to troubleshoot and fix any error.
================================================
FILE: keepalived/neutron-config.md
================================================
Introduction
------------
For this setup, we will configure an external provider network using eth0 (10.10.10.X network). A different setup would apply for bridged networks, although the HA features should be the same.
There are two services that require special attention for a highly available architecture:
- Neutron DHCP agent availability is obtained by assigning two or more agents to manage each tenant network, setting `dhcp_agents_per_network=2` in `/etc/neutron/neutron.conf`.
- Neutron L3 agent availability uses the L3 HA functionality using VRRP. Using this functionality, a new type of router, spawned on two or more different agents, is created. One agent will be in charge of the master version of this router, and the remaining L3 agents will be in charge of the slave routers. Refer to [this blog post](http://assafmuller.com/2014/08/16/layer-3-high-availability/) for a detailed description of the feature.
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/neutron.scenario).
Install software
----------------
yum install -y openstack-neutron openstack-neutron-openvswitch openstack-neutron-ml2 openstack-utils openstack-selinux
Configure Neutron server
------------------------
openstack-config --set /etc/neutron/neutron.conf DEFAULT bind_host 192.168.1.22X
openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://controller-vip.example.com:5000/
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_plugin password
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password neutrontest
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name services
openstack-config --set /etc/neutron/neutron.conf database connection mysql://neutron:neutrontest@controller-vip.example.com:3306/neutron
openstack-config --set /etc/neutron/neutron.conf database max_retries -1
openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier
openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3
openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/neutron/neutron.conf nova nova_region_name regionOne
openstack-config --set /etc/neutron/neutron.conf nova project_domain_id default
openstack-config --set /etc/neutron/neutron.conf nova project_name services
openstack-config --set /etc/neutron/neutron.conf nova user_domain_id default
openstack-config --set /etc/neutron/neutron.conf nova password novatest
openstack-config --set /etc/neutron/neutron.conf nova username compute
openstack-config --set /etc/neutron/neutron.conf nova auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/neutron/neutron.conf nova auth_plugin password
openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes True
openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_data_changes True
openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin neutron.plugins.ml2.plugin.Ml2Plugin
openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router
openstack-config --set /etc/neutron/neutron.conf DEFAULT router_scheduler_driver neutron.scheduler.l3_agent_scheduler.ChanceScheduler
openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 2
openstack-config --set /etc/neutron/neutron.conf DEFAULT api_workers 2
openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_workers 2
openstack-config --set /etc/neutron/neutron.conf DEFAULT l3_ha True
openstack-config --set /etc/neutron/neutron.conf DEFAULT min_l3_agents_per_router 2
openstack-config --set /etc/neutron/neutron.conf DEFAULT max_l3_agents_per_router 2
### ML2 configuration
ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini
openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 type_drivers local,gre,flat,vxlan,vlan
openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan
openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers openvswitch
openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks \*
openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_gre tunnel_id_ranges 10:10000
openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vni_ranges 10:10000
openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vxlan_group 224.0.0.1
openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_security_group True
openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
### LBaaS configuration (optional)
yum -y install openstack-neutron-lbaas
openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router,lbaas
openstack-config --set /etc/neutron/lbaas_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver
openstack-config --set /etc/neutron/lbaas_agent.ini DEFAULT device_driver neutron_lbaas.services.loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver
openstack-config --set /etc/neutron/lbaas_agent.ini haproxy user_group haproxy
### FWaaS configuration (optional)
yum -y install openstack-neutron-fwaas
openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router,firewall,lbaas
openstack-config --set /etc/neutron/fwaas_driver.ini fwaas enabled True
openstack-config --set /etc/neutron/fwaas_driver.ini fwaas driver neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas.IptablesFwaasDriver
Manage DB
---------
On node 1:
neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head
Start services, open firewall ports
-----------------------------------
On all nodes:
systemctl start neutron-server
systemctl enable neutron-server
firewall-cmd --add-port=9696/tcp
firewall-cmd --add-port=9696/tcp --permanent
OpenvSwitch configuration
-------------------------
systemctl enable openvswitch
systemctl start openvswitch
**Note:** we have seeing issues when trying to configure an IP on br-eth0 (specially ARP problems), so it is not recommended.
Assuming eth0 is your interface attached to the external network, create two files in _/etc/sysconfig/network-scripts/_ as follows (change MTU if you need):
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-eth0
ONBOOT=yes
BOOTPROTO=none
VLAN=yes
MTU="9000"
NM_CONTROLLED=no
EOF
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-br-eth0
DEVICE=br-eth0
DEVICETYPE=ovs
OVSBOOTPROTO=none
TYPE=OVSBridge
ONBOOT=yes
BOOTPROTO=static
MTU="9000"
NM_CONTROLLED=no
EOF
Restart the network for the changes to take effect.
systemctl restart network
OpenvSwitch agent
-----------------
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent tunnel_types vxlan
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 4789
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs local_ip 192.168.1.22X
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs enable_tunneling True
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs integration_bridge br-int
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_bridge br-tun
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs bridge_mappings physnet1:br-eth0
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs network_vlan_ranges physnet1
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent l2_population False
Metadata agent
--------------
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_strategy keystone
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_url http://controller-vip.example.com:35357/v2.0
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_host controller-vip.example.com
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT auth_region regionOne
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_tenant_name services
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_user neutron
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT admin_password neutrontest
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip controller-vip.example.com
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_port 8775
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret metatest
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_workers 4
openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_backlog 2048
DHCP agent
----------
openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver
openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_config_file /etc/neutron/dnsmasq-neutron.conf
The following will prevent issues from happening when the network card MTU is 1500. If we are using jumbo frames, it should not be required. Be aware that this only helps on certain operating systems with a well-behaving DHCP client. Windows is known to ignore it.
echo "dhcp-option-force=26,1400" > /etc/neutron/dnsmasq-neutron.conf
chown root:neutron /etc/neutron/dnsmasq-neutron.conf
chmod 644 /etc/neutron/dnsmasq-neutron.conf
L3 agent
--------
openstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver
openstack-config --set /etc/neutron/l3_agent.ini DEFAULT handle_internal_only_routers True
openstack-config --set /etc/neutron/l3_agent.ini DEFAULT send_arp_for_ha 3
openstack-config --set /etc/neutron/l3_agent.ini DEFAULT metadata_ip controller-vip.example.com
openstack-config --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge
Start services and open VXLAN firewall port
-------------------------------------------
systemctl start neutron-openvswitch-agent
systemctl start neutron-dhcp-agent
systemctl start neutron-l3-agent
systemctl start neutron-metadata-agent
systemctl start neutron-lbaas-agent
systemctl enable neutron-openvswitch-agent
systemctl enable neutron-dhcp-agent
systemctl enable neutron-l3-agent
systemctl enable neutron-metadata-agent
systemctl enable neutron-ovs-cleanup
systemctl enable neutron-lbaas-agent
firewall-cmd --add-port=4789/udp
firewall-cmd --add-port=4789/udp --permanent
**NOTE:** During a full cluster reboot, since Galera does not start cleanly neutron-server will wait for some time, then fail due to a service startup timeout (see [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1188198) for details). We can fix that by creating a file named `/etc/systemd/system/neutron-server.service.d/restart.conf` with the following contents:
[Service]
Restart=on-failure
Neutron server will try to restart indefinitely, then eventually succeed as soon as the Galera DB is running.
Create provider network
-----------------------
On node 1:
. /root/keystonerc_admin
neutron net-create public --provider:network_type flat --provider:physical_network physnet1 --router:external
neutron subnet-create --gateway 10.10.10.1 --allocation-pool start=10.10.10.100,end=10.10.10.150 --disable-dhcp --name public_subnet public 10.10.10.0/24
================================================
FILE: keepalived/nova-config.md
================================================
Introduction
------------
The following commands will be executed on all controller nodes, unless otherwise stated.
You can find a phd scenario file [here](phd-setup/nova.scenario).
Install software
----------------
yum install -y openstack-nova-console openstack-nova-novncproxy openstack-utils openstack-nova-api openstack-nova-conductor openstack-nova-scheduler python-cinderclient python-memcached
Configure Nova API
------------------
openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers hacontroller1:11211,hacontroller2:11211,hacontroller3:11211
openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_host 192.168.1.22X
openstack-config --set /etc/nova/nova.conf vnc novncproxy_base_url http://controller-vip.example.com:6080/vnc_auto.html
openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.1.22X
openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 192.168.1.22X
openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@controller-vip.example.com/nova
openstack-config --set /etc/nova/nova.conf database max_retries -1
openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen 192.168.1.22X
openstack-config --set /etc/nova/nova.conf DEFAULT metadata_host 192.168.1.22X
openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 192.168.1.22X
openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen_port 8775
openstack-config --set /etc/nova/nova.conf DEFAULT glance_host controller-vip.example.com
openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API
openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver
openstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver
openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron
openstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL
openstack-config --set /etc/nova/nova.conf conductor use_local false
openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts hacontroller1,hacontroller2,hacontroller3
openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True
openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True
openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest
openstack-config --set /etc/nova/nova.conf neutron url http://controller-vip.example.com:9696/
openstack-config --set /etc/nova/nova.conf neutron project_domain_id default
openstack-config --set /etc/nova/nova.conf neutron project_name services
openstack-config --set /etc/nova/nova.conf neutron user_domain_id default
openstack-config --set /etc/nova/nova.conf neutron username neutron
openstack-config --set /etc/nova/nova.conf neutron password neutrontest
openstack-config --set /etc/nova/nova.conf neutron auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/nova/nova.conf neutron auth_uri http://controller-vip.example.com:5000/
openstack-config --set /etc/nova/nova.conf neutron auth_plugin password
openstack-config --set /etc/nova/nova.conf neutron region_name regionOne
# REQUIRED FOR A/A scheduler
openstack-config --set /etc/nova/nova.conf DEFAULT scheduler_host_subset_size 30
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_plugin password
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_url http://controller-vip.example.com:35357/
openstack-config --set /etc/nova/api-paste.ini filter:authtoken username compute
openstack-config --set /etc/nova/api-paste.ini filter:authtoken password novatest
openstack-config --set /etc/nova/api-paste.ini filter:authtoken project_name services
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_uri http://controller-vip.example.com:5000/
Only run the following command if you are creating a test environment where your hypervisors will be virtual machines
openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu
Manage DB
---------
On node 1:
su nova -s /bin/sh -c "nova-manage db sync"
Start services, open firewall ports
-----------------------------------
On all nodes:
systemctl start openstack-nova-consoleauth
systemctl start openstack-nova-novncproxy
systemctl start openstack-nova-api
systemctl start openstack-nova-scheduler
systemctl start openstack-nova-conductor
systemctl enable openstack-nova-consoleauth
systemctl enable openstack-nova-novncproxy
systemctl enable openstack-nova-api
systemctl enable openstack-nova-scheduler
systemctl enable openstack-nova-conductor
firewall-cmd --add-port=8773-8775/tcp
firewall-cmd --add-port=8773-8775/tcp --permanent
firewall-cmd --add-port=6080/tcp
firewall-cmd --add-port=6080/tcp --permanent
================================================
FILE: keepalived/phd-setup/ceilometer.scenario
================================================
# This file can be used directly by 'phd', see 'build-all.sh' in this
# directory for how it can be invoked. The only requirement is a list
# of nodes you'd like it to modify.
#
# The scope of each command-block is controlled by the preceeding
# 'target' line.
#
# - target=all
# The commands are executed on evey node provided
#
# - target=local
# The commands are executed from the node hosting phd. When not
# using phd, they should be run from some other independant host
# (such as the puppet master)
#
# - target=$PHD_ENV_nodes{N}
# The commands are executed on the Nth node provided.
# For example, to run on only the first node would be target=$PHD_ENV_nodes1
#
# Tasks to be performed at this step include:
# - Install Ceilometer
# - Configuring Ceilometer
# - Start services and add firewall rules
#################################
# Scenario Requirements Section #
#################################
= VARIABLES =
PHD_VAR_network_nic_internal
PHD_VAR_network_nic_external
PHD_VAR_network_hosts_vip
PHD_VAR_network_hosts_controllers
PHD_VAR_network_hosts_rabbitmq
PHD_VAR_network_hosts_memcache
PHD_VAR_network_hosts_mongodb
PHD_VAR_network_ips_controllers
PHD_VAR_network_neutron_externalgateway
PHD_VAR_network_neutron_externalnetwork
PHD_VAR_network_neutron_allocpoolstart
PHD_VAR_network_neutron_allocpoolend
#################################
# Scenario Requirements Section #
#################################
= REQUIREMENTS =
nodes: 1
######################
# Deployment Scripts #
######################
= SCRIPTS =
target=all
....
myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)
yum install -y openstack-ceilometer-api openstack-ceilometer-central openstack-ceilometer-collector openstack-ceilometer-common openstack-ceilometer-alarm python-ceilometer python-ceilometerclient
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_user ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_password ceilometertest
openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}
openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}
openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://${PHD_VAR_network_hosts_vip}:5000/v2.0
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest
openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://${PHD_VAR_network_hosts_mongodb}:27017/ceilometer?replicaSet=ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf database max_retries -1
# keep last 5 days data only (value is in secs)
openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000
openstack-config --set /etc/ceilometer/ceilometer.conf api host ${myip}
IFS=', ' read -a controller_names <<< "${PHD_VAR_network_hosts_controllers}"
openstack-config --set /etc/ceilometer/ceilometer.conf coordination backend_url "redis://${controller_names[0]}:26379?sentinel=mymaster&sentinel_fallback=${controller_names[0]}:26379&sentinel_fallback=${controller_names[0]}:26379"
systemctl start openstack-ceilometer-central
systemctl enable openstack-ceilometer-central
systemctl start openstack-ceilometer-collector
systemctl enable openstack-ceilometer-collector
systemctl start openstack-ceilometer-api
systemctl enable openstack-ceilometer-api
systemctl start openstack-ceilometer-alarm-evaluator
systemctl enable openstack-ceilometer-alarm-evaluator
systemctl start openstack-ceilometer-alarm-notifier
systemctl enable openstack-ceilometer-alarm-notifier
systemctl start openstack-ceilometer-notification
systemctl enable openstack-ceilometer-notification
firewall-cmd --add-port=8777/tcp
firewall-cmd --add-port=8777/tcp --permanent
firewall-cmd --add-port=4952/udp
firewall-cmd --add-port=4952/udp --permanent
....
================================================
FILE: keepalived/phd-setup/cinder.scenario
================================================
# This file can be used directly by 'phd', see 'build-all.sh' in this
# directory for how it can be invoked. The only requirement is a list
# of nodes you'd like it to modify.
#
# The scope of each command-block is controlled by the preceeding
# 'target' line.
#
# - target=all
# The commands are executed on evey node provided
#
# - target=local
# The commands are executed from the node hosting phd. When not
# using phd, they should be run from some other independant host
# (such as the puppet master)
#
# - target=$PHD_ENV_nodes{N}
# The commands are executed on the Nth node provided.
# For example, to run on only the first node would be target=$PHD_ENV_nodes1
#
# Tasks to be performed at this step include:
# - Install Cinder
# - Configure Cinder, using a NFS v3 backend
# - Start services, open firewall rules
#################################
# Scenario Requirements Section #
#################################
= VARIABLES =
PHD_VAR_network_nic_internal
PHD_VAR_network_hosts_vip
PHD_VAR_network_ips_controllers
PHD_VAR_network_hosts_rabbitmq
PHD_VAR_network_hosts_memcache
PHD_VAR_network_nfs_cindershare
#################################
# Scenario Requirements Section #
#################################
= REQUIREMENTS =
nodes: 1
######################
# Deployment Scripts #
######################
= SCRIPTS =
target=all
....
myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)
yum install -y openstack-cinder openstack-utils openstack-selinux python-memcached
openstack-config --set /etc/cinder/cinder.conf database connection mysql://cinder:cindertest@${PHD_VAR_network_hosts_vip}/cinder
openstack-config --set /etc/cinder/cinder.conf database max_retries -1
openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_user cinder
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken admin_password cindertest
openstack-config --set /etc/cinder/cinder.conf DEFAULT notification_driver messaging
openstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder
openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_host ${PHD_VAR_network_hosts_vip}
openstack-config --set /etc/cinder/cinder.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}
openstack-config --set /etc/cinder/cinder.conf DEFAULT host rhos7-cinder
openstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen ${myip}
openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}
openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues true
cat > /etc/cinder/nfs_exports << EOF
${PHD_VAR_network_nfs_cindershare}
EOF
chown root:cinder /etc/cinder/nfs_exports
chmod 0640 /etc/cinder/nfs_exports
openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_shares_config /etc/cinder/nfs_exports
openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_sparsed_volumes true
openstack-config --set /etc/cinder/cinder.conf DEFAULT nfs_mount_options v3
openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.nfs.NfsDriver
....
target=$PHD_ENV_nodes1
....
su cinder -s /bin/sh -c "cinder-manage db sync"
systemctl start openstack-cinder-volume
systemctl enable openstack-cinder-volume
....
target=all
....
systemctl start openstack-cinder-api
systemctl start openstack-cinder-scheduler
systemctl enable openstack-cinder-api
systemctl enable openstack-cinder-scheduler
firewall-cmd --add-port=8776/tcp
firewall-cmd --add-port=8776/tcp --permanent
....
================================================
FILE: keepalived/phd-setup/compute.scenario
================================================
# This file can be used directly by 'phd', see 'build-all.sh' in this
# directory for how it can be invoked. The only requirement is a list
# of nodes you'd like it to modify.
#
# The scope of each command-block is controlled by the preceeding
# 'target' line.
#
# - target=all
# The commands are executed on evey node provided
#
# - target=local
# The commands are executed from the node hosting phd. When not
# using phd, they should be run from some other independant host
# (such as the puppet master)
#
# - target=$PHD_ENV_nodes{N}
# The commands are executed on the Nth node provided.
# For example, to run on only the first node would be target=$PHD_ENV_nodes1
#
# Tasks to be performed at this step include:
# - Install all required packages for a compute node
# - Configure nova-compute, neutron and ceilometer for the compute node
# - Tweak TCP keepalive parameters
# - Start services and open firewall ports
#################################
# Scenario Requirements Section #
#################################
= VARIABLES =
PHD_VAR_network_nic_internal
PHD_VAR_network_nic_external
PHD_VAR_network_hosts_vip
PHD_VAR_network_ips_controllers
PHD_VAR_network_hosts_rabbitmq
PHD_VAR_network_hosts_memcache
PHD_VAR_network_hosts_mongodb
#################################
# Scenario Requirements Section #
#################################
= REQUIREMENTS =
nodes: 1
######################
# Deployment Scripts #
######################
= SCRIPTS =
target=all
....
myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)
# Install packages
yum install -y openstack-nova-compute openstack-utils python-cinder openstack-neutron-openvswitch openstack-ceilometer-compute openstack-neutron
# Enable OpenvSwitch
systemctl enable openvswitch
systemctl start openvswitch
ovs-vsctl add-br br-int
# Configure Nova compute
openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers ${PHD_VAR_network_hosts_memcache}
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address ${myip}
openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen 0.0.0.0
openstack-config --set /etc/nova/nova.conf DEFAULT novncproxy_base_url http://${PHD_VAR_network_hosts_vip}:6080/vnc_auto.html
openstack-config --set /etc/nova/nova.conf database connection mysql://nova:novatest@${PHD_VAR_network_hosts_vip}/nova
openstack-config --set /etc/nova/nova.conf database max_retries -1
openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/nova/nova.conf glance host ${PHD_VAR_network_hosts_vip}
openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.neutronv2.api.API
openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver
openstack-config --set /etc/nova/nova.conf libvirt vif_driver nova.virt.libvirt.vif.LibvirtGenericVIFDriver
openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api neutron
openstack-config --set /etc/nova/nova.conf cinder cinder_catalog_info volume:cinder:internalURL
openstack-config --set /etc/nova/nova.conf conductor use_local false
openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}
openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues True
openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True
openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret metatest
openstack-config --set /etc/nova/nova.conf neutron url http://${PHD_VAR_network_hosts_vip}:9696/
openstack-config --set /etc/nova/nova.conf neutron admin_tenant_name services
openstack-config --set /etc/nova/nova.conf neutron admin_username neutron
openstack-config --set /etc/nova/nova.conf neutron admin_password neutrontest
openstack-config --set /etc/nova/nova.conf neutron admin_auth_url http://${PHD_VAR_network_hosts_vip}:35357/v2.0
openstack-config --set /etc/nova/nova.conf neutron region_name regionOne
openstack-config --set /etc/nova/nova.conf libvirt nfs_mount_options v3
openstack-config --set /etc/nova/api-paste.ini filter:authtoken auth_host ${PHD_VAR_network_hosts_vip}
openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_tenant_name services
openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_user compute
openstack-config --set /etc/nova/api-paste.ini filter:authtoken admin_password novatest
# Only if the hypervisor is a virtual machine
openstack-config --set /etc/nova/nova.conf libvirt virt_type qemu
# Configure Neutron for compute node
openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_host ${PHD_VAR_network_hosts_vip}
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_user neutron
openstack-config --set /etc/neutron/neutron.conf keystone_authtoken admin_password neutrontest
openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}
openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/neutron/neutron.conf DEFAULT notification_driver neutron.openstack.common.notifier.rpc_notifier
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent tunnel_types vxlan
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent vxlan_udp_port 4789
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs enable_tunneling True
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_id_ranges 1:1000
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tenant_network_type vxlan
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs integration_bridge br-int
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs tunnel_bridge br-tun
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs local_ip ${myip}
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini securitygroup firewall_driver neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent l2_population False
# Configure Ceilometer for compute node
openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit True
openstack-config --set /etc/nova/nova.conf DEFAULT instance_usage_audit_period hour
openstack-config --set /etc/nova/nova.conf DEFAULT notify_on_state_change vm_and_task_state
openstack-config --set /etc/nova/nova.conf DEFAULT notification_driver nova.openstack.common.notifier.rpc_notifier
sed -i -e 's/nova.openstack.common.notifier.rpc_notifier/nova.openstack.common.notifier.rpc_notifier\nnotification_driver = ceilometer.compute.nova_notifier/g' /etc/nova/nova.conf
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_host ${PHD_VAR_network_hosts_vip}
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_port 35357
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken auth_protocol http
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_user ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf keystone_authtoken admin_password ceilometertest
openstack-config --set /etc/ceilometer/ceilometer.conf DEFAULT memcache_servers ${PHD_VAR_network_hosts_memcache}
openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}
openstack-config --set /etc/ceilometer/ceilometer.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/ceilometer/ceilometer.conf publisher telemetry_secret ceilometersecret
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_auth_url http://${PHD_VAR_network_hosts_vip}:5000/v2.0
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_username ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_tenant_name services
openstack-config --set /etc/ceilometer/ceilometer.conf service_credentials os_password ceilometertest
openstack-config --set /etc/ceilometer/ceilometer.conf database connection mongodb://${PHD_VAR_network_hosts_mongodb}:27017/ceilometer?replicaSet=ceilometer
openstack-config --set /etc/ceilometer/ceilometer.conf database connection max_retries -1
# keep last 5 days data only (value is in secs)
openstack-config --set /etc/ceilometer/ceilometer.conf database metering_time_to_live 432000
# Cinder uses NFS
setsebool -P virt_use_nfs 1
# Kernel TCP keepalive parameters
cat > /etc/sysctl.d/tcpka.conf << EOF
net.ipv4.tcp_keepalive_intvl = 1
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_time = 5
EOF
sysctl -p /etc/sysctl.d/tcpka.conf
# Start services, open firewall ports
systemctl start libvirtd
systemctl start neutron-openvswitch-agent
systemctl enable neutron-openvswitch-agent
systemctl enable neutron-ovs-cleanup
systemctl start openstack-ceilometer-compute
systemctl enable openstack-ceilometer-compute
systemctl start openstack-nova-compute
systemctl enable openstack-nova-compute
firewall-cmd --add-port=4789/udp
firewall-cmd --add-port=4789/udp --permanent
firewall-cmd --add-port=5900-5999/tcp
firewall-cmd --add-port=5900-5999/tcp --permanent
....
================================================
FILE: keepalived/phd-setup/galera.scenario
================================================
# This file can be used directly by 'phd', see 'build-all.sh' in this
# directory for how it can be invoked. The only requirement is a list
# of nodes you'd like it to modify.
#
# The scope of each command-block is controlled by the preceeding
# 'target' line.
#
# - target=all
# The commands are executed on evey node provided
#
# - target=local
# The commands are executed from the node hosting phd. When not
# using phd, they should be run from some other independant host
# (such as the puppet master)
#
# - target=$PHD_ENV_nodes{N}
# The commands are executed on the Nth node provided.
# For example, to run on only the first node would be target=$PHD_ENV_nodes1
#
# Tasks to be performed at this step include:
# - Install MariaDB / Galera
# - Configure Galera
# - Bootstrap an initial Galera cluster
# - Create databases for OpenStack services
#################################
# Scenario Requirements Section #
#################################
= VARIABLES =
PHD_VAR_network_nic_internal
#################################
# Scenario Requirements Section #
#################################
= REQUIREMENTS =
nodes: 1
######################
# Deployment Scripts #
######################
= SCRIPTS =
target=all
....
yum install -y mariadb-galera-server xinetd rsync psmisc
cat > /etc/sysconfig/clustercheck << EOF
MYSQL_USERNAME="clustercheck"
MYSQL_PASSWORD="redhat"
MYSQL_HOST="localhost"
MYSQL_PORT="3306"
EOF
systemctl start mysqld
mysql -e "CREATE USER 'clustercheck'@'localhost' IDENTIFIED BY 'redhat';"
systemctl stop mysqld
myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)
cat > /etc/my.cnf.d/galera.cnf << EOF
[mysqld]
skip-name-resolve=1
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
max_connections=2048
query_cache_size=0
query_cache_type=0
bind_address=${myip}
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_name="galera_cluster"
wsrep_cluster_address="gcomm://192.168.1.221,192.168.1.222,192.168.1.223"
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_notify_cmd=
wsrep_sst_method=rsync
EOF
mkdir -p /etc/systemd/system/mariadb.service.d/
cat > /etc/systemd/system/mariadb.service.d/limits.conf << EOF
[Service]
LimitNOFILE=16384
EOF
cat > /etc/xinetd.d/galera-monitor << EOF
service galera-monitor
{
port = 9200
disable = no
socket_type = stream
protocol = tcp
wait = no
user = root
group = root
groups = yes
server = /usr/bin/clustercheck
type = UNLISTED
per_source = UNLIMITED
log_on_success =
log_on_failure = HOST
flags = REUSE
}
EOF
systemctl daemon-reload
systemctl enable xinetd
systemctl start xinetd
systemctl enable haproxy
systemctl start haproxy
firewall-cmd --add-service=mysql
firewall-cmd --add-port=4444/tcp
firewall-cmd --add-port=4567/tcp
firewall-cmd --add-port=4568/tcp
firewall-cmd --add-port=4568/tcp --permanent
firewall-cmd --add-service=mysql --permanent
firewall-cmd --add-port=4567/tcp --permanent
firewall-cmd --add-port=4444/tcp --permanent
firewall-cmd --add-port=9300/tcp
firewall-cmd --add-port=9300/tcp --permanent
firewall-cmd --add-port=9200/tcp
firewall-cmd --add-port=9200/tcp --permanent
systemctl enable mariadb
....
target=$PHD_ENV_nodes1
....
# This is required to allow sudo execution without a tty
sed -i 's/Defaults requiretty/Defaults !requiretty/g' /etc/sudoers
nohup sudo -u mysql /usr/libexec/mysqld --wsrep-cluster-address='gcomm://' < /dev/null > /dev/null 2>&1 &
sleep 30
# A little cleanup
sed -i 's/Defaults !requiretty/Defaults requiretty/g' /etc/sudoers
....
target=$PHD_ENV_nodes2
....
systemctl start mariadb
sleep 10
....
target=$PHD_ENV_nodes3
....
systemctl start mariadb
sleep 10
....
target=$PHD_ENV_nodes1
....
cat > /tmp/mysql.sql << EOF
use mysql;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED by 'mysqltest' WITH GRANT OPTION;
CREATE DATABASE keystone;
GRANT ALL ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'keystonetest';
CREATE DATABASE glance;
GRANT ALL ON glance.* TO 'glance'@'%' IDENTIFIED BY 'glancetest';
CREATE DATABASE cinder;
GRANT ALL ON cinder.* TO 'cinder'@'%' IDENTIFIED BY 'cindertest';
CREATE DATABASE neutron;
GRANT ALL ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'neutrontest';
CREATE DATABASE nova;
GRANT ALL ON nova.* TO 'nova'@'%' IDENTIFIED BY 'novatest';
CREATE DATABASE heat;
GRANT ALL ON heat.* TO 'heat'@'%' IDENTIFIED BY 'heattest';
CREATE DATABASE sahara;
GRANT ALL ON sahara.* TO 'sahara'@'%' IDENTIFIED BY 'saharatest';
CREATE DATABASE trove;
GRANT ALL ON trove.* TO 'trove'@'%' IDENTIFIED BY 'trovetest';
FLUSH PRIVILEGES;
EOF
killall mysqld
# it takes some time for mysqld to actually stop after you kill it
sleep 30
systemctl start mariadb
mysql < /tmp/mysql.sql > /tmp/mysql.out
rm -f /tmp/mysql.sql
mysqladmin flush-hosts
....
================================================
FILE: keepalived/phd-setup/glance.scenario
================================================
# This file can be used directly by 'phd', see 'build-all.sh' in this
# directory for how it can be invoked. The only requirement is a list
# of nodes you'd like it to modify.
#
# The scope of each command-block is controlled by the preceeding
# 'target' line.
#
# - target=all
# The commands are executed on evey node provided
#
# - target=local
# The commands are executed from the node hosting phd. When not
# using phd, they should be run from some other independant host
# (such as the puppet master)
#
# - target=$PHD_ENV_nodes{N}
# The commands are executed on the Nth node provided.
# For example, to run on only the first node would be target=$PHD_ENV_nodes1
#
# Tasks to be performed at this step include:
# - Installing Glance
# - Configuring Glance
# - Starting services and opening firewall rules
#################################
# Scenario Requirements Section #
#################################
= VARIABLES =
PHD_VAR_network_nic_internal
PHD_VAR_network_hosts_vip
PHD_VAR_network_ips_controllers
PHD_VAR_network_hosts_rabbitmq
PHD_VAR_network_nfs_glanceshare
#################################
# Scenario Requirements Section #
#################################
= REQUIREMENTS =
nodes: 1
######################
# Deployment Scripts #
######################
= SCRIPTS =
target=all
....
myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)
yum install -y openstack-glance openstack-utils openstack-selinux nfs-utils
openstack-config --set /etc/glance/glance-api.conf database connection mysql://glance:glancetest@${PHD_VAR_network_hosts_vip}/glance
openstack-config --set /etc/glance/glance-api.conf database max_retries -1
openstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone
openstack-config --set /etc/glance/glance-api.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/
openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_user glance
openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_password glancetest
openstack-config --set /etc/glance/glance-api.conf DEFAULT notification_driver messaging
openstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host ${myip}
openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host ${PHD_VAR_network_hosts_vip}
openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_hosts ${PHD_VAR_network_hosts_rabbitmq}
openstack-config --set /etc/glance/glance-api.conf oslo_messaging_rabbit rabbit_ha_queues true
openstack-config --set /etc/glance/glance-registry.conf database connection mysql://glance:glancetest@${PHD_VAR_network_hosts_vip}/glance
openstack-config --set /etc/glance/glance-registry.conf database max_retries -1
openstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone
openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken identity_uri http://${PHD_VAR_network_hosts_vip}:35357/
openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_tenant_name services
openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_user glance
openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_password glancetest
openstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host ${myip}
....
target=$PHD_ENV_nodes1
....
su glance -s /bin/sh -c "glance-manage db_sync"
....
target=all
....
echo "${PHD_VAR_network_nfs_glanceshare} /var/lib/glance nfs vers=3 0 0" >> /etc/fstab
# Workaround for bz#1203820
systemctl start rpcbind
systemctl start nfs-config
systemctl start rpc-statd
mount -a
chown glance:nobody /var/lib/glance
systemctl start openstack-glance-registry
systemctl start openstack-glance-api
systemctl enable openstack-glance-registry
systemctl enable openstack-glance-api
firewall-cmd --add-port=9191/tcp
firewall-cmd --add-port=9191/tcp --permanent
firewall-cmd --add-port=9292/tcp
firewall-cmd --add-port=9292/tcp --permanent
....
================================================
FILE: keepalived/phd-setup/ha-collapsed.variables
================================================
# Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD
# Each scenario file will verify that values have been provided for the variables it requires.
deployment: collapsed
hypervisors:
controllers: controller1:oslab1,controller2:oslab2,controller3:oslab3
computenodes: compute1:oslab1,compute2:oslab3
rhn:
user: jpena@redhat.com
pass: OSA07novp7her
network:
ips:
vip: 192.168.1.220
controllers: 192.168.1.221,192.168.1.222,192.168.1.223
computeinternal: 192.168.1.224,192.168.1.225
computeexternal: 10.10.10.224,10.10.10.225
gateway: 192.168.1.1
netmask: 255.255.255.0
nfs:
glanceshare: 192.168.1.4:/volumeUSB1/usbshare/openstack/glance
cindershare: 192.168.1.4:/volumeUSB1/usbshare/openstack/cinder
nic:
base: 54:52:00
internal: eth0
external: eth1
hosts:
vip: controller-vip.example.com
controllers: controller1.example.com, controller2.example.com, controller3.example.com
compute: compute1.example.com, compute2.example.com
mongodb: controller1,controller2,controller3
memcache: controller1:11211,controller2:11211,controller3:11211
rabbitmq: controller1,controller2,controller3
domain: example.com
neutron:
externalgateway: 10.10.10.1
externalnetwork: 10.10.10.0/24
allocpoolstart: 10.10.10.100
allocpoolend: 10.10.10.150
ssh:
pubkey: ssh-dss AAAAB3NzaC1kc3MAAACBANTRU30h4EC1sv5kb/KJBDYx0D/YKWAVzMJpeWLzFrECF41Tymte8W5s4oz0CzpQQH4r19YFpeS+jingeakkRc749ImwhJE77yRmPsjaEDe3OZXW+Udz4dikE5HBjuVlEL9UN3yoGfBZCXo9Jm+oRJCH+wx36Pl69IHRZwJIAvQfAAAAFQDTdIeHmLP77PvjG5jo6eEYDTedXwAAAIEAvwKlws7V3qy6Ckd26FcAr1noN+myBdVVpb960F7BPWIxXkA9z12o+3y5JEUJRVTplRf8pBz1JwKdsYS7WJj8teSslmwbQz0kQ61uSNkWqBZGOLE/ddQxCJbD3OTXRZ289A/2mRBcil8mIg82JY1AzNr1OaQQ82ZP35nISA/jeQMAAACBAIeuHCLqkYRf7Xuv2ztFOhPzLrJRRSm7Zcpdxp5Ano7R68Yq1RQbMn0lUIG4QMssviQPyklF/xHYGJLUXBIUfNORn7BYqHikbcguUHKiwiC9io9xR9+zlvgIAIU6bbnsNT9MofTdrRT+H7df6cDP8NO43w8dWFXWX4K3NUU0YjbM
components: serverprep lb galera rabbitmq keystone memcache glance cinder swift-brick swift neutron nova horizon heat mongodb ceilometer node
osp:
major: 7
env:
password: cluster
# TODO...
password:
cluster: foo
keystone: bar
================================================
FILE: keepalived/phd-setup/heat.scenario
================================================
# This file can be used directly by 'phd', see 'build-all.sh' in this
# directory for how it can be invoked. The only requirement is a list
# of nodes you'd like it to modify.
#
# The scope of each command-block is controlled by the preceeding
# 'target' line.
#
# - target=all
# The commands are executed on evey node provided
#
# - target=local
# The commands are executed from the node hosting phd. When not
# using phd, they should be run from some other independant host
# (such as the puppet master)
#
# - target=$PHD_ENV_nodes{N}
# The commands are executed on the Nth node provided.
# For example, to run on only the first node would be target=$PHD_ENV_nodes1
#
# Tasks to be performed at this step include:
# - Installing Heat
# - Creating the required Heat domain
# - Configuring Heat
# - Starting services and opening firewall rules
#################################
# Scenario Requirements Section #
#################################
= VARIABLES =
PHD_VAR_network_nic_internal
PHD_VAR_network_nic_external
PHD_VAR_network_hosts_vip
PHD_VAR_network_ips_controllers
PHD_VAR_network_hosts_rabbitmq
PHD_VAR_network_hosts_memcache
PHD_VAR_network_neutron_externalgateway
PHD_VAR_network_neutron_externalnetwork
PHD_VAR_network_neutron_allocpoolstart
PHD_VAR_network_neutron_allocpoolend
#################################
# Scenario Requirements Section #
#################################
= REQUIREMENTS =
nodes: 1
######################
# Deployment Scripts #
######################
= SCRIPTS =
target=all
....
yum install -y openstack-heat-engine openstack-heat-api openstack-heat-api-cfn openstack-heat-api-cloudwatch python-heatclient openstack-utils python-glanceclient
....
target=$PHD_ENV_nodes1
....
. /root/keystonerc_admin
openstack role create heat_stack_user
TOKEN_ID=$(openstack token issue --format value --column id)
openstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 domain create heat --description "Owns users and projects created by heat"
openstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 user create --password heattest --domain heat --description "Manages users and projects created by heat" heat_domain_admin
openstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-version=3 role add --user heat_domain_admin --domain heat admin
....
target=all
....
myip=$(ip a |grep ${PHD_VAR_network_nic_internal} | grep inet | awk '{print $2}' | awk -F/ '{print $1}' | head -n 1)
. /root/keystonerc_admin
TOKEN_ID=$(openstack token issue --format value --column id)
HEAT_DOMAIN_ID=$(openstack --os-token=${TOKEN_ID} --os-url=http://${PHD_VAR_network_hosts_vip}:5000/v3 --os-identity-api-versio
gitextract_4wkbeqjk/
├── HA-keepalived.md
├── README.md
├── build-all.sh
├── full-stack.md
├── ha-openstack.md
├── iha-install-10.sh
├── iha-install-9.sh
├── iha-uninstall.sh
├── keepalived/
│ ├── ceilometer-config.md
│ ├── cinder-config.md
│ ├── compute-node.md
│ ├── controller-node.md
│ ├── galera-bootstrap.md
│ ├── galera-config.md
│ ├── glance-config.md
│ ├── haproxy-config.md
│ ├── heat-config.md
│ ├── horizon-config.md
│ ├── keepalived-config.md
│ ├── keystone-config.md
│ ├── memcached-config.md
│ ├── mongodb-config.md
│ ├── mongodb-recovery.md
│ ├── neutron-config.md
│ ├── nova-config.md
│ ├── phd-setup/
│ │ ├── ceilometer.scenario
│ │ ├── cinder.scenario
│ │ ├── compute.scenario
│ │ ├── galera.scenario
│ │ ├── glance.scenario
│ │ ├── ha-collapsed.variables
│ │ ├── heat.scenario
│ │ ├── horizon.scenario
│ │ ├── hypervisors.scenario
│ │ ├── keepalived.scenario
│ │ ├── keystone.scenario
│ │ ├── lb.scenario
│ │ ├── memcached.scenario
│ │ ├── mongodb.scenario
│ │ ├── neutron.scenario
│ │ ├── nova.scenario
│ │ ├── rabbitmq.scenario
│ │ ├── readme.txt
│ │ ├── redis.scenario
│ │ ├── sahara.scenario
│ │ ├── serverprep.scenario
│ │ ├── swift.scenario
│ │ ├── test.sh
│ │ └── trove.scenario
│ ├── rabbitmq-config.md
│ ├── rabbitmq-restart.md
│ ├── redis-config.md
│ ├── sahara-config.md
│ ├── swift-config.md
│ └── trove-config.md
├── make-vm
└── pcmk/
├── NovaCompute
├── NovaEvacuate
├── baremetal-rollback.scenario
├── baremetal.scenario
├── basic-cluster.scenario
├── beaker.scenario
├── ceilometer-test.sh
├── ceilometer.scenario
├── cinder-test.sh
├── cinder.scenario
├── compute-cluster.scenario
├── compute-common.scenario
├── compute-managed.scenario
├── controller-managed.scenario
├── galera-test.sh
├── galera.scenario
├── gateway.scenario
├── glance-test.sh
├── glance.scenario
├── ha-collapsed.variables
├── ha-segregated.variables
├── hacks.scenario
├── heat-test.sh
├── heat.scenario
├── horizon-test.sh
├── horizon.scenario
├── keystone-test.sh
├── keystone.scenario
├── lb.scenario
├── memcached.scenario
├── mongodb.scenario
├── mrg.variables
├── neutron-agents.scenario
├── neutron-server.scenario
├── neutron-test.sh
├── nova-test.sh
├── nova.scenario
├── nova_client.py
├── rabbitmq-test.sh
├── rabbitmq.scenario
├── swift-aco.scenario
├── swift-test.sh
├── swift.scenario
├── virt-hosts.scenario
├── vmsnap-rollback.scenario
└── vmsnap.scenario
SYMBOL INDEX (13 symbols across 1 files)
FILE: pcmk/nova_client.py
function register_named (line 25) | def register_named(cls):
function named (line 40) | def named(method_name, positionals=[], opts_with_val=[], opts_without_va...
function shell_fields (line 58) | def shell_fields(fields):
class NovaClientWrapper (line 67) | class NovaClientWrapper(object):
method __init__ (line 70) | def __init__(self, version, username, password, tenant_name, auth_url):
method service_list (line 82) | def service_list(self, host=None, binary=None):
method service_enable (line 88) | def service_enable(self, host, binary):
method migration_list (line 95) | def migration_list(self, host=None, status=None, cell_name=None):
method _server_migrate (line 101) | def _server_migrate(self, server):
method host_servers_migrate (line 117) | def host_servers_migrate(self, host):
method handle_method_and_args (line 126) | def handle_method_and_args(self, args_list):
function print_list (line 166) | def print_list(objs, fields):
function main (line 193) | def main():
Condensed preview — 102 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (524K chars).
[
{
"path": "HA-keepalived.md",
"chars": 9668,
"preview": "Introduction\n------------\n\nThis document aims at defining a high level architecture for a highly available OpenStack set"
},
{
"path": "README.md",
"chars": 642,
"preview": "Introduction\n------------\n\nThis repository contains the description of two highly available OpenStack architectures usin"
},
{
"path": "build-all.sh",
"chars": 7053,
"preview": "#!/bin/bash\n\nset -e\n\ndeclare -A nodeMap\ndeclare -A variables\ndeclare -A cluster\n\nnodeMap[\"baremetal\"]=\"east-01 east-02 e"
},
{
"path": "full-stack.md",
"chars": 5903,
"preview": "This is what the result of `pcs status` should look like:\n\n Cluster name: rhos-node\n Last updated: Fri Mar 6 22:0"
},
{
"path": "ha-openstack.md",
"chars": 24399,
"preview": "# Highly Available Openstack Deployments\n\nThe current target for this document is RDO 10, based on\nthe OpenStack Newton "
},
{
"path": "iha-install-10.sh",
"chars": 10968,
"preview": "#!/bin/bash\n\nset -ex\n#\n#If your deployment includes this use case, include the no_shared_storage=1 option in step 7.\n\n\nh"
},
{
"path": "iha-install-9.sh",
"chars": 13039,
"preview": "#!/bin/bash\n\nset -ex\n#\n#If your deployment includes this use case, include the no_shared_storage=1 option in step 7.\n\n\nh"
},
{
"path": "iha-uninstall.sh",
"chars": 1681,
"preview": "#!/bin/bash\n\nset -ex\n\nsource stackrc\n\n# If your machines don't conform to this structure, try calling the script like th"
},
{
"path": "keepalived/ceilometer-config.md",
"chars": 4489,
"preview": "Introduction\n------------\n\nIn terms of high availability, the Ceilometer central agent deserves special attention. This "
},
{
"path": "keepalived/cinder-config.md",
"chars": 4166,
"preview": "Introduction\n------------\n\nCinder will be configured in this example to use the NFS backend driver. Instructions for any"
},
{
"path": "keepalived/compute-node.md",
"chars": 11255,
"preview": "Introduction\n------------\n\nThe compute node implementation is relatively straightforward, compared to the controller nod"
},
{
"path": "keepalived/controller-node.md",
"chars": 6097,
"preview": "Introduction\n------------\n\nController nodes will run all OpenStack services in this architecture, while compute nodes wi"
},
{
"path": "keepalived/galera-bootstrap.md",
"chars": 4490,
"preview": "Introduction\n------------\n\nHere is an outline of the steps needed to re-establish/bootstrap Galera quorum.\n\n1. Determin"
},
{
"path": "keepalived/galera-config.md",
"chars": 5878,
"preview": "Introduction\n------------\n\nMariaDB with Galera provides synchronous database replication, in an active-active, multi-mas"
},
{
"path": "keepalived/glance-config.md",
"chars": 4337,
"preview": "Introduction\n------------\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYo"
},
{
"path": "keepalived/haproxy-config.md",
"chars": 9307,
"preview": "Introduction\n------------\n\nA load-balancing proxy is used to provide scalability and load balancing for OpenStack API se"
},
{
"path": "keepalived/heat-config.md",
"chars": 5075,
"preview": "Introduction\n------------\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYo"
},
{
"path": "keepalived/horizon-config.md",
"chars": 1325,
"preview": "Introduction\n------------\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYo"
},
{
"path": "keepalived/keepalived-config.md",
"chars": 1651,
"preview": "Introduction\n------------\n\n[Keepalived](http://www.keepalived.org/) provides simple and robust facilities for load balan"
},
{
"path": "keepalived/keystone-config.md",
"chars": 11060,
"preview": "Introduction\n------------\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYo"
},
{
"path": "keepalived/memcached-config.md",
"chars": 837,
"preview": "Introduction\n------------\n\nMemcached is a general-purpose distributed memory caching system. It is used to speed up dyna"
},
{
"path": "keepalived/mongodb-config.md",
"chars": 1469,
"preview": "Introduction\n------------\n\nMongoDB can provide high availability through the use of replica sets. A replica set in Mongo"
},
{
"path": "keepalived/mongodb-recovery.md",
"chars": 300,
"preview": "Introduction\n------------\n\nMongoDB usually does a good job at re-forming a replica set after a full cluster reboot. In c"
},
{
"path": "keepalived/neutron-config.md",
"chars": 12741,
"preview": "Introduction\n------------\n\nFor this setup, we will configure an external provider network using eth0 (10.10.10.X network"
},
{
"path": "keepalived/nova-config.md",
"chars": 5188,
"preview": "Introduction\n------------\n\nThe following commands will be executed on all controller nodes, unless otherwise stated.\n\nYo"
},
{
"path": "keepalived/phd-setup/ceilometer.scenario",
"chars": 4691,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/cinder.scenario",
"chars": 3905,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/compute.scenario",
"chars": 9811,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/galera.scenario",
"chars": 5117,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/glance.scenario",
"chars": 4167,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/ha-collapsed.variables",
"chars": 2206,
"preview": "# Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD\n# Each scenario file will verify that value"
},
{
"path": "keepalived/phd-setup/heat.scenario",
"chars": 5962,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/horizon.scenario",
"chars": 2662,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/hypervisors.scenario",
"chars": 6755,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/keepalived.scenario",
"chars": 2044,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/keystone.scenario",
"chars": 13200,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/lb.scenario",
"chars": 9404,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/memcached.scenario",
"chars": 1336,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/mongodb.scenario",
"chars": 1773,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/neutron.scenario",
"chars": 11726,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/nova.scenario",
"chars": 5668,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/rabbitmq.scenario",
"chars": 3743,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/readme.txt",
"chars": 474,
"preview": "Assumptions:\n\n1. We have a basic rhel7-base image. It is just a cloud image where we have manually injected a root ssh k"
},
{
"path": "keepalived/phd-setup/redis.scenario",
"chars": 2728,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/sahara.scenario",
"chars": 3878,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/serverprep.scenario",
"chars": 2404,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/swift.scenario",
"chars": 6033,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/phd-setup/test.sh",
"chars": 1478,
"preview": "#!/bin/bash\n\nset -e\n\ndeclare -A nodeMap\ndeclare -A variables\ndeclare -A cluster\n\nnodeMap[\"hypervisors\"]=\"oslab1 oslab2 o"
},
{
"path": "keepalived/phd-setup/trove.scenario",
"chars": 11431,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "keepalived/rabbitmq-config.md",
"chars": 3491,
"preview": "Introduction\n------------\n\nRabbitMQ can create a native cluster, by grouping several nodes and replicating all message q"
},
{
"path": "keepalived/rabbitmq-restart.md",
"chars": 2000,
"preview": "Introduction\n------------\n\nIn general, RabbitMQ does a good job at restarting the cluster when all nodes are stared at t"
},
{
"path": "keepalived/redis-config.md",
"chars": 1852,
"preview": "Introduction\n------------\n\nRedis in a key-value cache and store, used by Ceilometer with the [tooz](https://github.com/o"
},
{
"path": "keepalived/sahara-config.md",
"chars": 3878,
"preview": "Introduction\n------------\n\n**Important:** this configuration assumes that the controller nodes can access the floating I"
},
{
"path": "keepalived/swift-config.md",
"chars": 5856,
"preview": "Introduction\n------------\n\nWe need to have an additional disk, `/dev/vdb` in our test available for Swift usage.\n\nThe fo"
},
{
"path": "keepalived/trove-config.md",
"chars": 9812,
"preview": "Introduction\n------------\n\n**WARNING:** this configuration uses the same RabbitMQ instance as the rest of the OpenStack "
},
{
"path": "make-vm",
"chars": 2861,
"preview": "#!/bin/bash\n\nonlyone=$1\n\nexport PHD_VAR_network_nic_base=\"54:52:00\"\nexport PHD_VAR_vm_base=\"/srv/rhos6-rhel7-vms/rhos6-r"
},
{
"path": "pcmk/NovaCompute",
"chars": 10645,
"preview": "#!/bin/sh\n#\n#\n# NovaCompute agent manages compute daemons.\n#\n# Copyright (c) 2015\n#\n# This program is free software; you"
},
{
"path": "pcmk/NovaEvacuate",
"chars": 8442,
"preview": "#!/bin/sh\n#\n#\n# NovaCompute agent manages compute daemons.\n#\n# Copyright (c) 2015\n#\n# This program is free software; you"
},
{
"path": "pcmk/baremetal-rollback.scenario",
"chars": 2260,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/baremetal.scenario",
"chars": 5546,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/basic-cluster.scenario",
"chars": 4471,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/beaker.scenario",
"chars": 2307,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/ceilometer-test.sh",
"chars": 227,
"preview": ". ${PHD_VAR_env_configdir}/keystonerc_admin\nfor m in storage.objects image network volume instance ; do ceilometer sampl"
},
{
"path": "pcmk/ceilometer.scenario",
"chars": 6242,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/cinder-test.sh",
"chars": 193,
"preview": ". ${PHD_VAR_env_configdir}/keystonerc_admin\n\nopenstack volume list\nopenstack volume create --size 10 test-volume\nopensta"
},
{
"path": "pcmk/cinder.scenario",
"chars": 5081,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/compute-cluster.scenario",
"chars": 2362,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/compute-common.scenario",
"chars": 9648,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/compute-managed.scenario",
"chars": 1721,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/controller-managed.scenario",
"chars": 8073,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/galera-test.sh",
"chars": 74,
"preview": "clustercheck\n\n# verify sync is done\nmysql\nSHOW STATUS LIKE 'wsrep%';\nquit\n"
},
{
"path": "pcmk/galera.scenario",
"chars": 5091,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/gateway.scenario",
"chars": 9811,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/glance-test.sh",
"chars": 418,
"preview": ". ${PHD_VAR_env_configdir}/keystonerc_admin\n\nif [ ! -f ${PHD_VAR_env_configdir}/cirros-0.3.2-x86_64-disk.img ]; then\n\twg"
},
{
"path": "pcmk/glance.scenario",
"chars": 5457,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/ha-collapsed.variables",
"chars": 2873,
"preview": "# Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD\n# Each scenario file will verify that value"
},
{
"path": "pcmk/ha-segregated.variables",
"chars": 2027,
"preview": "# Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD\n# Each scenario file will verify that value"
},
{
"path": "pcmk/hacks.scenario",
"chars": 1569,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/heat-test.sh",
"chars": 1685,
"preview": "# TEST:\n# Requires a compute node!\n\n. ${PHD_VAR_env_configdir}/keystonerc_admin\n\nnova keypair-add --pub_key ~/.ssh/autho"
},
{
"path": "pcmk/heat.scenario",
"chars": 4840,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/horizon-test.sh",
"chars": 181,
"preview": "\nIt should be possible now to login to http://vip-horizon/dashboard with admin account.\nNote that it is still not possib"
},
{
"path": "pcmk/horizon.scenario",
"chars": 3077,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/keystone-test.sh",
"chars": 284,
"preview": "# TEST (might require logout/login to reset the environmet that was set before \n# during initial bootstrap)\n\nunset SERVI"
},
{
"path": "pcmk/keystone.scenario",
"chars": 9716,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/lb.scenario",
"chars": 8105,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/memcached.scenario",
"chars": 1094,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/mongodb.scenario",
"chars": 1883,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/mrg.variables",
"chars": 2755,
"preview": "# Expanded to $PHD_VAR_network_domain, $PHD_VAR_network_internal, etc by PHD\n# Each scenario file will verify that value"
},
{
"path": "pcmk/neutron-agents.scenario",
"chars": 6921,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/neutron-server.scenario",
"chars": 7359,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/neutron-test.sh",
"chars": 782,
"preview": "# TEST/create your first network\n# It is not possible to test neutron completely until the full deployment is complete a"
},
{
"path": "pcmk/nova-test.sh",
"chars": 70,
"preview": ". ${PHD_VAR_env_configdir}/keystonerc_admin\nnova usage\nnova usage-list"
},
{
"path": "pcmk/nova.scenario",
"chars": 6770,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/nova_client.py",
"chars": 8547,
"preview": "#!/usr/bin/python -tt\n\nimport argparse\nimport collections\nimport inspect\nimport logging\nimport sys\n\ntry:\n from novacl"
},
{
"path": "pcmk/rabbitmq-test.sh",
"chars": 53,
"preview": "rabbitmqctl cluster_status\nrabbitmqctl list_policies\n"
},
{
"path": "pcmk/rabbitmq.scenario",
"chars": 2185,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/swift-aco.scenario",
"chars": 4486,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/swift-test.sh",
"chars": 380,
"preview": ". ${PHD_VAR_env_configdir}/keystonerc_admin\n\nopenstack container list\nopenstack container create test\nopenstack containe"
},
{
"path": "pcmk/swift.scenario",
"chars": 4830,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/virt-hosts.scenario",
"chars": 11069,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/vmsnap-rollback.scenario",
"chars": 4073,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
},
{
"path": "pcmk/vmsnap.scenario",
"chars": 2474,
"preview": "# This file can be used directly by 'phd', see 'build-all.sh' in this\n# directory for how it can be invoked. The only r"
}
]
About this extraction
This page contains the full source code of the beekhof/osp-ha-deploy GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 102 files (491.7 KB), approximately 136.9k tokens, and a symbol index with 13 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.