Elasticsearch 8 Cluster Installation

In this step-by-step guide, we’ll show you how to set up an Elasticsearch cluster with 3 nodes and Kibana. In addition, we’ll use an NGINX reverse proxy and Let’s Encrypt for SSL termination.

Prerequisites

• At least 3 Elasticsearch servers (Debian, pre-installed only, with pre-installed NGINX reverse proxy, default configuration)
• This guide requires SSH access via root

---

Preparing Node 1

Limiting Available Memory

To ensure the system remains operational even under a heavy workload, we recommend that you always limit the available memory to prevent an OOM (Out-Of-Memory) kill.

We use the following formula (max server RAM - 3 GiB = Elasticsearch RAM limit) to set the following limits on a server with 8 GiB of RAM:

nano /etc/elasticsearch/jvm.options.d/creoline.options

Insert the following section and adjust the values as needed:

# min ram
-Xms5g

# max ram
-Xmx5g

Configure Elasticsearch Basic Settings for Node 1:

First, the first node of the cluster must be configured. It is important to note that the following changes are made only on the first node, and the Elasticsearch server is enabled and started only here.

Open the Elasticsearch server configuration as follows.

nano /etc/elasticsearch/elasticsearch.yml

Modify the configuration as follows:

cluster.name: <cluster-name>

node.name: <server-name> 

node.roles: [ master, data ]

network.host: <VPC-IP-address>

discovery.seed_hosts: ["<node-1-VPC-IP>:9300", "<node-2-VPC-IP>:9300", "<node-3-VPC-IP>:9300"]

cluster.initial_master_nodes: ["<node-1-fqdn>", "<node-2-fqdn>", "<node-3-fqdn>"]

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

http.host: <VPC IP address>

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

Save the modified configuration and enable the Elasticsearch service:

systemctl enable elasticsearch.service

You can then start it:

systemctl restart elasticsearch.service

Adjusting the NGINX Configuration for Nodes 1–3:

If you want to use a domain that differs from the server’s working domain, you must add it as the server name in the NGINX configuration. Alternatively, you can simply replace the working domain with your custom domain:

nano /etc/nginx/conf.d/elasticsearch-proxy.conf

server {
    server_name localhost sXXXXX.creoline.cloud <Custom-Domain.tld>;

Next, you must update the IP address for port forwarding to port 9200 to match the server’s VPC IP address and disable SSL verification for proxy connections. This prevents errors caused by the internal, SSL-secured communication between the Elasticsearchnodes and Kibana:

server {
    server_name localhost sXXXXX.creoline.cloud <Custom-Domain.tld>;

    location / {
# proxy_pass http://127.0.0.1:9200;
 proxy_pass https://<VPC IP address>:9200;
 proxy_ssl_verify off;

To apply the changes, the NGINX service must be reloaded:

systemctl reload nginx.service

Issuing a Let’s Encrypt Certificate for a Custom Domain:

Use the following command to issue a free SSL certificate from Let’s Encrypt for your custom domain:

certbot --nginx -d <custom-domain> --non-interactive --agree-tos -m <customer’s technical contact email>

Custom Domain for Kibana Node 1:

If you’re using a custom domain, it must also be specified in Kibana’s configuration:

nano /etc/kibana/kibana.yml

#server.name: "XXXXXX.creoline.cloud"
server.name: <"Custom-Domain.tld">

Additionally, the IP address of the Elasticsearch host must be updated in the following two lines of the configuration so that Kibana can access the Elasticsearch server via the server’s VPC IP address:

# This section was automatically generated during setup.
#elasticsearch.hosts: ['https://127.0.0.1:9200']
elasticsearch.hosts: ['https://<VPC IP address>:9200']

#xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://127.0.0.1:9200'], ca_trusted_fingerprint: e69ac43bd506dfae1d65962422a0746e899a9449751b0199f464e73d25028510}]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://<VPC IP address>:9200'], ca_trusted_fingerprint: e69ac43bd506dfae1d65962422a0746e899a9449751b0199f464e73d25028510}]

Restart the Kibana service for the changes to take effect:

systemctl restart kibana.service

---

Cluster Setup

In the next step, generate a cluster enrollment token on Node 1 and make a note of it.

This will be needed later to add Node 2 and Node 3 to the cluster.

You can generate the enrollment token as follows:

 /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node

eyJ2ZXIiOiI4LjEyLjIiLCJhZHIiOlsiNS4xLjc4LjE3OjkyMDAiXSwiZmdyIjoiMDJjMGRkMDdlNDNjM2NmYTFlMzk3YTFmM
GM2MjIzOTM2YjMwYjYwZTk0ZGM5ODA3YTgxNzA0NzM3NTIxNzljZSIsImtleSI6IktRNHU5WTBCdUN2VlJ5bm0yRVJ4Ojd4Sj
c3UWoxUVdLQ054UXdSTDRWNUEifQ==

Next, you need to generate an enrollment token for the later integration of the Kibana instances from Node 2 and Node 3. Be sure to save the generated token separately as well:

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token --scope kibana

eyJ2ZXIiOiI4LjEyLjIiLCJhZHIiOlsiNS4xLjc4LjE3OjkyMDAiXSwiZmdyIjoiMDJjMGRkMDdlNDNjM2NmYTFlMzk3YTFmMGM
2MjIzOTM2YjMwYjYwZTk0ZGM5ODA3YTgxNzA0NzM3NTIxNzljZSIsImtleSI6IlVLTjc5WTBCVWtnSDVsMndzM3FIOnJibFk4Q2
5TU3ppXzdxNjlYT0lqSEEifQ==

Joining the Elasticsearch Cluster

Run the following commands to uninstall Elasticsearch and Kibana, clean up any remaining files, and reinstall them:

apt purge elasticsearch kibana

rm -rf /etc/elasticsearch
rm -rf /var/lib/elasticsearch
rm -rf /etc/kibana
rm -rf /var/lib/kibana

apt update && apt install -y elasticsearch kibana

Enable automatic service restart

Copy the entire command below and run it:

echo "[Service]
Restart=on-failure" > /etc/systemd/system/elasticsearch.service.d/override.conf

Then run the following command to apply the configuration changes:

systemctl daemon-reload

Joining Nodes 2 and 3 to the Cluster

After installation on Nodes 2 and 3, run the following command immediately to join the cluster. Replace <Enrollment-Token-Node-1> with the enrollment token previously generated for cluster enrollment:

/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <Enrollment-Token-Node-1>

Afterward, the following entry should have been added to the configuration file /etc/elasticsearch/elasticsearch.yml:

discovery.seed_hosts: ["<VPC-IP-Address>:9300"] -> The IP address corresponds to that of Node 1!

---

Setting Up Node 2 and Node 3

After successfully joining the cluster, you can proceed to adjust the configuration of Elasticsearch and Kibana.

Limiting Available Memory

Example for a server with 8 GiB of memory:

nano /etc/elasticsearch/jvm.options.d/creoline.options

Add the following:

# min ram
-Xms5g

# max ram
-Xmx5g

Adjusting the Basic Elasticsearch Configuration for Nodes 2 & 3:

Now you can adjust the Elasticsearch configuration similarly to Node 1:

nano /etc/elasticsearch/elasticsearch.yml

Set the settings as follows and save the file:

cluster.name: <cluster-name>

node.name: <server-name> 

node.roles: [ master, data ]

network.host: <10.20.0.x (VPC network/IP address)>

cluster.initial_master_nodes: ["<node-1-fqdn>", "<node-2-fqdn>", "<node-3-fqdn>"]

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

http.host: <VPC IP address>

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

Enable and Start Elasticsearch + Kibana:

To apply the changes, you must enable and start Elasticsearch and Kibana:

systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service

Generating the Kibana Encryption Keys

Generate the Kibana encryption keys as follows so that Kibana will be able to communicate with the Elasticsearch service later on:

/usr/share/kibana/bin/kibana-encryption-keys generate -q >> /etc/kibana/kibana.yml

Basic Kibana Configuration:

Open the Kibana configuration on Node 2 and Node 3:

nano /etc/kibana/kibana.yml

Adjust the configuration as follows.

If you are using a custom domain, it must also be specified for Kibana:

server.host: "127.0.0.1"
server.port: 5601
#server.name: "sXXXXX.creoline.cloud"
server.name: <"Custom-Domain.tld">
server.basePath: "/kibana"
server.rewriteBasePath: true
server.publicBaseUrl: "http://127.0.0.1/kibana"

Restart Kibana for the changes to take effect:

systemctl enable kibana.service
systemctl start kibana.service

Connecting Elasticsearch and Kibana on Nodes 2 & 3 to Node 1

If you are using the server’s work domain, you can access Kibana in your browser as follows:

https://XXXXXX.creoline.cloud/kibana

If you are using a custom domain, you can access Kibana as follows:

https://<Custom-Domain.tld>/kibana

In the following screen, enter the enrollment token you generated earlier for Kibana and confirm the details.

Generate the Kibana Verification Code to Complete the Setup

You can generate the verification code to complete the Kibana setup as follows. Run the command on the node where you are currently setting up Kibana (Node 2 or Node 3):

/usr/share/kibana/bin/kibana-verification-code

Enter the 6-digit code and wait until the setup is complete.

Once the setup is complete, the following should have been added to kibana.yml:

# This section was automatically generated during setup.
elasticsearch.hosts: ['https://<VPC-IP-Node-1>:9200']
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE3NDQ4Nzc5MjI0NTY6TWxDY2JtUEtTa0NONkxkeTVaN0VkUQ
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1744877923188.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://<VPC-IP-Node-1>:9200'], ca_trusted_fingerprint: e69ac43bd506dfae1d65962422a0746e899a9449751b0199f464e73d25028510}]

---

Completing the Cluster Setup

Finalizing the Cluster Setup

Once Kibana has been set up, the cluster setup is nearly complete.

The following final changes must now be made to the elasticsearch.yml file on all nodes:

nano /etc/elasticsearch/elasticsearch.yml

→ Comment out cluster.initial_master_nodes

# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]

→ Adjust discovery.seed_hosts (add nodes 2 and 3)

# Discover existing nodes in the cluster
discovery.seed_hosts: ["VPC-IP-Node-1:9300", "VPC-IP-Node-2:9300", "VPC-IP-Node-3:9300"]

Restart Elasticsearch:

systemctl restart elasticsearch.service

Run checks

Check inter-node communication using curl:

curl -k -XGET "https://<FQDN>/_cat/nodes?v" -u elastic
Enter host password for user 'elastic':
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
<VPC-IP-Node-1>   9 99   1    0.00    0.03     0.06 dm *      Node-1
<VPC-IP-Node-2>  11 91   3    0.17    0.11     0.09 dm - Node-2
<VPC-IP-Node-3>   4 92   2    0.16    0.18     0.12 dm - Node-3

Perform a health check using curl:

curl -k -XGET "https://<FQDN>/_cat/health?v" -u elastic
Enter host password for user 'elastic':
epoch timestamp cluster     status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1710497165 10:06:05  cluster-name green 3 3     61  30    0    0 0 0 - 100.0%

Alternatively, you can also enter the following URLs in your browser to view the corresponding output after successful authentication:

https://XXXXX.creoline.cloud/_cat/nodes?v

https://XXXXX.creoline.cloud/_cat/health?v

---

Cloud Firewall Recommendations

We recommend that you configure the following firewall rules in this order in the cloud firewall for all cluster nodes. Access to ports 80 and 443 is required for the automatic renewal of Let’s Encrypt certificates:

• Allow access to port 443
• Allow access to port 80
• Allow access to port 22 only for specific IPv4/IPv6 addresses
• Explicitly block access to port 9200 on the WAN interface
• Explicitly block access to port 9300 on the WAN interface
• Explicitly block access to port 5601 on the WAN interface
• Block all other access