Deploying AWS OpenSearch: A Step-By-Step Guide

Introduction

AWS OpenSearch is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud.

It is open-source and has a distributed search and analytics engine forked from Elasticsearch.

OpenSearch enables you to perform real-time searches, analyze big data sets, and visualize the data with rich dashboards.

Prerequisites

Before provisioning AWS OpenSearch, you must do a few things for the following project.

• You need an AWS Account: Ensure you have an active AWS account. Click the link to install: AWS Console — Signup
• You need Terraform installed on your local machine: Install Terraform on your local machine. Here’s the link to do it: Install Terraform | Terraform | HashiCorp Developer
• You need to install AWS CLI on your computer: Install and configure the AWS CLI with appropriate credentials. Install, Update, and Uninstall the AWS CLI — AWS Command Line Interface

Deploying OpenSearch Using Terraform

What is Terraform?

Terraform is an open-source infrastructure-as-code (IaC) tool developed by HashiCorp.

It allows you to define and provision infrastructure using a high-level configuration language known as HashiCorp Configuration Language (HCL) or JSON.

Terraform enables you to manage a variety of cloud service providers, such as AWS, Azure, and Google Cloud, as well as on-premises infrastructure, through declarative configuration files.

By describing your infrastructure in code, Terraform allows you to version, share, and reuse configurations, ensuring consistent and reproducible deployments.

The tool translates these configurations into API calls to the appropriate providers, orchestrating resource creation, modification, and deletion.

One of Terraform's critical features is its execution plan, which outlines the changes needed to achieve the desired state described in the configuration files.

This plan allows users to review changes before applying them, ensuring transparency and control over infrastructure modifications.

Terraform also supports the concept of state, which keeps track of your infrastructure's current state and helps manage resources over time.

With its modular approach, Terraform enables you to create reusable components, enhancing team efficiency and collaboration.

By integrating with version control systems, Terraform promotes best practices in infrastructure management, such as code reviews and continuous integration/continuous deployment (CI/CD), making it a powerful tool for modern DevOps practices.

The concept:

• Define the AWS provider, which will specify the AWS region.
• Create necessary VPC and subnet resources to ensure your OpenSearch domain is within a VPC.
• Define security group rules to allow appropriate ingress and egress traffic.
• Configure the OpenSearch domain to set up the domain, the desired cluster configuration, the EBS options, the necessary access policies, and VPC options.
• Initialize and apply the configuration to create the resources using terraform init and terraform apply.

Step One: Create The Necessary Terraform Configuration Files

Create a variables.tf file and Add this code:

variable "region" {
  description = "The AWS region to deploy the resources"
  type     = string
}

variable "vpc_cidr" {
  description = "CIDR block for the VPC"
  type     = string
}

variable "subnet_cidr" {
  description = "CIDR block for the subnet"
  type =      string
}

variable "opensearch_domain_name" {
  description = "Name of the OpenSearch domain"
  type     = string
}

variable "opensearch_instance_type" {
  description = "Instance type for OpenSearch nodes"
  type    = string
}

variable "opensearch_instance_count" {
  description = "Number of OpenSearch nodes"
  type     = number
}

variable "opensearch_volume_size" {
  description = "Size of EBS volumes for OpenSearch nodes (in GB)"
  type    = number
}

variable "opensearch_version" {
  description = "OpenSearch engine version"
  type     = string
}

Create a terraform.tfvars file and add this script:

region                   = "us-east-1"
vpc_cidr                 = "40.0.0.0/16"
subnet_cidr              = "40.0.1.0/24"
opensearch_domain_name   = "t2s-dev"
opensearch_instance_type = "t3.small.search"
opensearch_instance_count = 2
opensearch_volume_size   = 10
opensearch_version       = "OpenSearch_1.0"

Create an Access Policies JSON file and add this content:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-2:123456789012:domain/t2s-academy.tech/*"
    }
  ]
}

Please ensure the JSON above is in the same folder (directory) as all your Terraform configuration files. Also, replace this, 12345678901, with the correct Account ID.

Create a main.tf file and add this code:

provider "aws" {
  region = var.region
}

resource "aws_vpc" "t2s-VPC-dev" {
  cidr_block = var.vpc_cidr
}

resource "aws_subnet" "t2s-SUBNET-dev" {
  vpc_id     = aws_vpc.t2s-VPC-dev.id
  cidr_block = var.subnet_cidr
  availability_zone = "${var.region}a"
}

resource "aws_security_group" "t2s-sg-dev" {
  vpc_id = aws_vpc.t2s-VPC-dev.id

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_opensearch_domain" "t2s-OpenSearch" {
  domain_name    = var.opensearch_domain_name
  engine_version = var.opensearch_version

  cluster_config {
    instance_type  = var.opensearch_instance_type
    instance_count = var.opensearch_instance_count
  }

  ebs_options {
    ebs_enabled = true
    volume_size = var.opensearch_volume_size
    volume_type = "gp2"
  }

  access_policies = file("access_policies.json")

  vpc_options {
    subnet_ids          = [aws_subnet.t2s-SUBNET-dev.id]
    security_group_ids  = [aws_security_group.t2s-sg-dev.id]
  }
}

Create an output.tf file and add this script:

output "opensearch_endpoint" {
  value = aws_opensearch_domain.t2s-OpenSearch.endpoint
}

Step Two: Apply the Configuration

Initialize Terraform:

terraform init

Review the Configuration:

terraform plan

Apply the Configuration:

terraform apply

Clean Up

To clean up, use this command:

terraform destroy

Deploying OpenSearch Using The AWS CLI

The create-elasticsearch-domain command can be used to deploy OpenSearch, formerly known as Amazon Elasticsearch Service, on AWS using the AWS CLI.

Here is a basic example of how you can create an OpenSearch domain using the AWS CLI:

aws es create-elasticsearch-domain --domain-name YOUR_DOMAIN_NAME --elasticsearch-version "7.10" --elasticsearch-cluster-config InstanceType=t2.small.elasticsearch,InstanceCount=1 --ebs-options EBSEnabled=true,VolumeType=gp2,VolumeSize=10

Here’s how you should use the above command:

• Replace YOUR_DOMAIN_NAME with your desired domain name. In my case, I will use t2s-academy.tech.
• Depending on your requirements, You can customize other parameters, such as InstanceType, InstanceCount, VolumeType, and VolumeSize.

Ensure that the AWS CLI is configured with the necessary permissions for your AWS account before running this command.

My command, as a result, looks like this:

 aws es create-elasticsearch-domain --domain-name t2s-academy-tech --elasticsearch-version "7.10" --elasticsearch-cluster-config InstanceType=t2.small.elasticsearch,InstanceCount=1 --ebs-options EBSEnabled=true,VolumeType=gp2,VolumeSize=10

First, let’s check the Amazon OpenSearch Service Dashboard. There, you can see there is no domain yet on our account.

Now, after running the above command, here’s what we see: only a portion of the screen.

Your screen will end with a “:” and press Enter to move to the next page.

To quit, enter the letter “q” from your keyboard.

The above screenshot shows you that your OpenSearch domain has been created.

Go to the console, and you will see our domain, t2s-academy-tech, and the progress line.

The OpenSearch domain will become active once it finishes being created.

Click on View details (at the top of the screen with the blue background) to see the progress.

The changes that are being applied include:

• Creating a new environment that requires the necessary resources.
• Provisioning new nodes. For our project, we will be creating only 1 node.
• Traffic routing on new nodes. The service will enable traffic routing through our newly created node.

Once processing updates are 100% complete, you can return to the Dashboard and examine your OpenSearch domain, t2s-academy-tech.

Deploying OpenSearch Using CloudFormation

What is CloudFormation?

WS CloudFormation is a service provided by Amazon Web Services (AWS) that enables you to define and provision infrastructure as code (IaC).

Using CloudFormation, you can create, update, and manage a collection of related AWS resources through templates in a predictable and repeatable manner.

These templates, written in JSON or YAML, describe the resources and properties needed for your application.

CloudFormation manages the lifecycle of these resources by handling dependencies and executing operations like creating, updating, or deleting stacks, which are collections of AWS resources managed as a single unit.

Additionally, CloudFormation supports features like change sets, which preview changes before they are applied, and drift detection, which identifies discrepancies between actual resource configurations and those defined in the template.

By leveraging these capabilities, CloudFormation facilitates the automation and consistency of infrastructure management, reducing manual interventions and errors while ensuring reliable and scalable deployments across different environments.

Using CloudFormation

First, you need to create a template file in a JSON or YAML format.

We will use a YAML template for our project. So, create a file named opensearch.yaml, and add the following content:

Resources:
  OpenSearchDomain:
    Type: AWS::OpenSearchService::Domain
    Properties:
      DomainName: "t2s-academy"   # Replace with the correct Domain Name
      EngineVersion: "OpenSearch_1.0"  # Replace with the correct engine version
      ClusterConfig:
        InstanceType: "m5.large.search"
        InstanceCount: 2
      EBSOptions:
        EBSEnabled: true
        VolumeSize: 10
        VolumeType: gp2
      AccessPolicies:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: "arn:aws:iam::730335276920:role/aws-service-role/opensearchservice.amazonaws.com/AWSServiceRoleForAmazonOpenSearchService"
            Action: "es:*"
            Resource: "arn:aws:es:us-east-1:730335276920:domain/t2s-academy/*" # Replace with the correction region and account ID

You need to do the following:

• Replace t2s-academy with your desired Domain.
• Adjust the OpenSearchVersion, InstanceType, InstanceCount, VolumeType, and VolumeSize per your requirements.
• Ensure you have an IAM Role for OpenSearch, AWSServiceRoleForAmazonOpenSearchService. You must attach it to the YAML (JSON template file if you use this format).

Next, go to the CloudFormation Dashboard and upload the template. Click Create stack.

Then, select Choose an existing template. For the template source, choose Upload a template file. And upload it using the Upload a template file button.

Click Next.

Give the stack a name, and we will call it opensearch-t2s for our project. Click Next since we will not need to configure any Parameters.

Here, we will everything as defaults.

Go to the bottom of the page and click Submit.

The creation of your CloudFormation stack will begin.

Go to the OpenSearch Dashboard to check if your domain is running now. Of course, you will see it and can also check progress.

What’s left to do is wait until the Processing updates are 100%.

This will complete our project, which shows you how to provision an OpenSearch Domain using Terraform, AWS CLI, and CloudFormation.

Conclusion

If you followed all the steps, you now have the skills to deploy an OpenSearch Domain using three different methods: Terraform, AWS CLI, or CloudFormation. Happy provisioning. God bless you!