Additional Steps for Multi-node Setups
To ensure your multi-node cluster runs seamlessly during failover state, complete the following additional steps. The example below assumes the cluster has three (3) nodes.
GEOREDUNDANCY:
In a Georedundant setup, the following procedure should be performed in each data center.
In a Georedundant setup, the following procedure should be performed in each data center.
To configure ElasticSearch
1. Create a new file with the name es_nodeport.yaml:
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"labels": {
"common.k8s.elastic.co/type": "elasticsearch",
"elasticsearch.k8s.elastic.co/cluster-name": "elasticsearch-arcus"
},
"name": "ice-arcus-es-client-np",
"namespace": "ice-arcus",
"selfLink": "/api/v1/namespaces/ice-arcus/services/ice-arcus-es-client-np"
},
"spec": {
"externalTrafficPolicy": "Cluster",
"ports": [
{
"name": "arcus-es",
"nodePort": 30029,
"port": 9200,
"protocol": "TCP",
"targetPort": 9200
}
],
"selector": {
"common.k8s.elastic.co/type": "elasticsearch",
"elasticsearch.k8s.elastic.co/cluster-name": "elasticsearch-arcus"
},
"sessionAffinity": "None",
"type": "NodePort"
}
}
2. Create the nodeport service:
kubectl -n ice-arcus create -f es_nodeport.yaml
3. Scale-up the ElasticSearch deployment using kubectl on any one node:
# the following command must be on a single line
ESS=$(kubectl -n ice-arcus get secrets elasticsearch-arcus-es-elastic-user -o jsonpath --template '{.data.elastic}' | base64 -d)
# the following command must be on a single line
LIP=$(ip route get 1 | awk '{print $NF;exit}')
# the following command must be on a single line
kubectl -n ice-arcus patch elasticsearches.elasticsearch.k8s.elastic.co elasticsearch-arcus --type='json' --patch='[{"op":"replace","path":"/spec/nodeSets/0/count","value":3}]'
4. ElasticSearch scales up to three nodes with green status:
$ kubectl -n ice-arcus \
get elasticsearches.elasticsearch.k8s.elastic.co \
elasticsearch-arcus
NAME HEALTH NODES VERSION PHASE AGE
elasticsearch-arcus green 3 7.6.2 Ready 2d22h
5. Define replicas for each ElasticSearch index:
for INDEX in $(curl -k --user elastic:${ESS} https://${LIP}:30029/_cat/indices 2>/dev/null | awk '{print $3}')
do
curl -k --user elastic:${ESS} -XPUT \
"https://${LIP}:30029/${INDEX}/_settings?pretty" \
-H 'Content-Type: application/json' \
-d' { "number_of_replicas": 0 }'
done
6. Verify each index is now replicated across all three nodes:
$ curl -k --user elastic:${ESS} https://${LIP}:30029/_cat/shards
channels 0 r STARTED 5 37.4kb 10.90.0.9 elasticsearch-arcus-es-member-0
channels 0 r STARTED 5 37.4kb 10.90.2.21 elasticsearch-arcus-es-member-2
channels 0 p STARTED 5 37.4kb 10.90.1.22 elasticsearch-arcus-es-member-1
geofence 0 r STARTED 0 283b 10.90.0.9 elasticsearch-arcus-es-member-0
geofence 0 p STARTED 0 283b 10.90.2.21 elasticsearch-arcus-es-member-2
geofence 0 r STARTED 0 283b 10.90.1.22 elasticsearch-arcus-es-member-1
persons 0 r STARTED 4 87.8kb 10.90.0.9 elasticsearch-arcus-es-member-0
persons 0 r STARTED 4 87.8kb 10.90.2.21 elasticsearch-arcus-es-member-2
persons 0 p STARTED 4 87.8kb 10.90.1.22 elasticsearch-arcus-es-member-1
filedata 0 p STARTED 0 283b 10.90.0.9 elasticsearch-arcus-es-member-0
filedata 0 r STARTED 0 283b 10.90.2.21 elasticsearch-arcus-es-member-2
filedata 0 r STARTED 0 283b 10.90.1.22 elasticsearch-arcus-es-member-1
auditlog-02042022 0 p STARTED 30 43.2kb 10.90.0.9 elasticsearch-arcus-es-member-0
auditlog-02042022 0 r STARTED 30 43.2kb 10.90.2.21 elasticsearch-arcus-es-member-2
auditlog-02042022 0 r STARTED 30 43.2kb 10.90.1.22 elasticsearch-arcus-es-member-1
auditlog-02012022 0 r STARTED 102 61.7kb 10.90.0.9 elasticsearch-arcus-es-member-0
auditlog-02012022 0 p STARTED 102 61.7kb 10.90.2.21 elasticsearch-arcus-es-member-2
auditlog-02012022 0 r STARTED 102 61.7kb 10.90.1.22 elasticsearch-arcus-es-member-1
textmessage 0 p STARTED 0 283b 10.90.0.9 elasticsearch-arcus-es-member-0
textmessage 0 r STARTED 0 283b 10.90.2.21 elasticsearch-arcus-es-member-2
textmessage 0 r STARTED 0 283b 10.90.1.22 elasticsearch-arcus-es-member-1