Study guide
Technical reference and lesson notes
Purpose of This Lesson
Amazon OpenSearch Service is a managed, distributed search and analytics platform for text, logs, metrics, and other unstructured or semi-structured data. For SAP-C02 scenarios, the important decisions involve cluster architecture, VPC placement, access control, high availability, data ingestion, storage tiers, and operational tradeoffs.
Amazon OpenSearch Service is the successor to Amazon Elasticsearch Service. Exam questions may use either name, and the core architectural concepts are substantially similar.
Key Concepts
What OpenSearch Service Provides
Amazon OpenSearch Service manages OpenSearch domains, which are clusters composed of search and data nodes. It is designed for:
- Full-text search
- Log and application analytics
- Infrastructure and application monitoring
- Security analytics
- Data visualization and troubleshooting
- Searching and aggregating unstructured or semi-structured data
The service is managed by AWS, but you still choose instance types, instance counts, storage, networking, security settings, and scaling strategies. It is not serverless by default; traditional domains run on provisioned infrastructure.
A domain can be created through the AWS Management Console, API, or AWS CLI.
Cluster and Node Design
A domain can use multiple Availability Zones to improve resilience. Data is distributed across nodes, and replicas provide additional copies of index data.
Important design practices include:
- Distribute data nodes across multiple Availability Zones.
- Prefer three Availability Zones where supported for stronger fault tolerance.
- Use equal numbers of data nodes per Availability Zone to maintain balanced distribution.
- Configure at least one replica for important indexes when the data must remain available after a node failure.
- Use dedicated master nodes for larger or production domains so cluster management is not competing with search and indexing workloads.
- When using dedicated master nodes, three is a common resilient configuration because it supports quorum-based cluster management.
OpenSearch scales by adding or removing instances and, depending on the workload, adjusting storage and node roles. Scaling should account for both storage capacity and search/indexing performance.
Storage Options
OpenSearch domains can use different storage approaches depending on how frequently data is accessed:
- Standard or hot storage: Intended for frequently searched and actively indexed data.
- UltraWarm storage: Lower-cost storage for less frequently accessed data that still needs to remain searchable.
- Cold storage: Intended for infrequently accessed data where minimizing storage cost is more important than immediate access.
- Snapshots: Used for backup and recovery. Snapshots are an important part of a recovery strategy but should not be confused with replicas; replicas provide operational availability, while snapshots provide backup and restore capability.
The appropriate tier depends on access frequency, retention requirements, performance expectations, and cost.
Data Ingestion and Visualization
Data can be ingested from several sources, including:
- Amazon Kinesis Data Firehose
- Logstash
- OpenSearch or Elasticsearch APIs
- Applications and custom producers
A common log analytics pattern is:
- Applications, infrastructure, or security systems produce events.
- A collection or ingestion component sends the data to OpenSearch.
- OpenSearch indexes and stores the data.
- OpenSearch Dashboards is used to search, visualize, and analyze the results.
The historical ELK stack refers to Elasticsearch, Logstash, and Kibana. In AWS exam material, you may also encounter the equivalent OpenSearch terminology: OpenSearch, Logstash or another ingestion service, and OpenSearch Dashboards.
Public and VPC Domains
A domain can use a public endpoint or be deployed inside an Amazon VPC.
#### VPC deployment
A VPC domain is generally preferred when producers, consumers, and administrative tools already operate inside private AWS networking. It supports private communication between the domain and other VPC resources, avoiding exposure of search traffic to the public internet.
Access from outside the VPC requires an appropriate private connectivity path, such as:
- AWS Site-to-Site VPN
- AWS Direct Connect
- A proxy or other controlled access component
- A connected network architecture using supported AWS networking services
#### Public endpoint
A public domain can be reached directly over the internet, but it must be protected with restrictive domain access policies, identity-based permissions, network controls, and encryption. Public accessibility does not eliminate the need for authentication and authorization.
#### VPC deployment limitations and gotchas
Several deployment choices are difficult or impossible to change later:
- A domain cannot simply be converted from a VPC deployment to a public endpoint, or vice versa.
- A domain deployed in one VPC cannot be moved directly to a different VPC.
- Subnet and security group settings can be changed within the selected VPC, subject to service constraints.
- VPC domains cannot use IP-based access policies in the same way as public domains.
- A domain cannot be launched in a VPC configured with dedicated tenancy.
These constraints make VPC placement an early architecture decision rather than an easily reversible setting.
OpenSearch Access Control
OpenSearch access control is layered. The main mechanisms are:
#### Domain access policies
These are resource-based policies attached to the OpenSearch domain. They define which principals or network sources can access the domain and which actions are allowed.
Restrictive domain access policies should be used instead of broad permissions such as allowing all principals or all network locations.
#### IAM identity-based policies
IAM policies can be attached to users, groups, or roles. They authorize AWS API operations and can be combined with domain-level resource policies.
#### IP-based policies
IP-based controls restrict access to specified IP addresses or CIDR blocks. They are useful for public endpoint scenarios, but they are not available for VPC domains in the same manner as public domains.
#### Fine-grained access control
Fine-grained access control provides role-based permissions at more detailed levels, including:
- Index level
- Document level
- Field level
- OpenSearch Dashboards access
- Dashboards multi-tenancy
This is appropriate when different teams or applications must access different indexes or fields within the same domain.
Authentication Options
OpenSearch can integrate with different authentication models, including:
- IAM-based access
- SAML federation with enterprise identity directories, such as an on-premises Active Directory environment
- Amazon Cognito integration with supported identity providers and social identity providers
- HTTP-based authentication options used with fine-grained access control
Authentication identifies the caller; authorization determines what that caller can do. A strong design addresses both.
Encryption
For sensitive workloads, enable:
- Encryption at rest: Protects stored indexes, automated snapshots, and related data on supported storage.
- Node-to-node encryption: Protects traffic between nodes in the domain.
- HTTPS or encryption in transit: Protects client-to-domain communications.
Encryption should be considered together with access policies, IAM, authentication, and network placement. Encryption alone does not provide authorization.
Exam-Relevant Takeaways
- Amazon OpenSearch Service is the successor to Amazon Elasticsearch Service; exam questions may use either name.
- An OpenSearch domain is a managed cluster, not simply an API endpoint with no infrastructure considerations.
- Use multiple Availability Zones for resilience and balance the data nodes across those zones.
- Replicas improve availability and read capacity, while snapshots provide backup and recovery.
- Three dedicated master nodes are a common production design for resilient cluster management.
- VPC deployment provides private connectivity but introduces connectivity requirements for external clients and limits some policy options.
- Public and VPC endpoint choices are not freely interchangeable after deployment.
- Use domain access policies, IAM policies, and fine-grained access control together where appropriate.
- Fine-grained access control is the choice when permissions must be restricted by index, document, or field.
- Use Kinesis Data Firehose or Logstash when the requirement is continuous ingestion from logs or streaming sources.
- Use OpenSearch Dashboards for interactive search, visualization, and analysis.
- Choose UltraWarm or cold storage for older, less frequently accessed data when the workload does not require all data to remain on hot storage.
Architecture Decision Guide
| Requirement | Appropriate design choice | Reason or tradeoff |
|---|---|---|
| Keep search traffic private inside AWS | Deploy the domain in a VPC | Avoids public exposure, but external clients need private connectivity such as VPN or Direct Connect |
| Allow direct internet access to the domain | Use a public endpoint with restrictive policies and strong authentication | Easier external access, but increases exposure and requires careful security controls |
| Survive an Availability Zone or node failure | Use multiple AZs and index replicas | Improves availability and resilience; increases storage consumption and cost |
| Protect cluster management from data workloads | Use dedicated master nodes | Provides more predictable cluster coordination; adds node cost |
| Restrict users to specific indexes or fields | Enable fine-grained access control | Provides detailed authorization beyond broad domain permissions |
| Ingest continuous logs or event streams | Use Kinesis Data Firehose, Logstash, or an application producer | Decouples producers from the search cluster and supports streaming ingestion |
| Search recent, frequently accessed data | Use hot or standard storage | Best performance for active indexing and queries |
| Retain older data at lower cost | Use UltraWarm or cold storage | Reduces cost, with different performance and access characteristics |
| Recover deleted or corrupted data | Use snapshots | Provides backup and restore; replicas are not a substitute for backups |
| Federate access with an enterprise directory | Use SAML federation | Allows existing identity management and centralized authentication |
Common Exam Traps
- Confusing replicas with backups: Replicas help maintain service availability; snapshots are required for backup and recovery.
- Assuming VPC domains are reachable directly from the internet: Private domains require a network path into the VPC.
- Assuming a domain can be moved between VPC and public deployment later: Endpoint type is a foundational deployment decision.
- Using IP policies for a VPC domain: IP-based access policy behavior differs from public endpoint deployments and is not the normal control for VPC domains.
- Treating OpenSearch as fully serverless: Traditional OpenSearch domains use provisioned instances and require capacity planning.
- Using IAM alone for field-level protection: Fine-grained access control is needed for index-, document-, or field-level authorization.
- Confusing Kibana with the search engine: Kibana or OpenSearch Dashboards is the visualization and analysis interface; the OpenSearch domain stores and indexes the data.
- Putting all data nodes in one AZ: This creates an avoidable single-AZ failure risk.
- Assuming encryption solves access control: Encryption protects data, while IAM, domain policies, authentication, and fine-grained permissions control access.
- Choosing hot storage for every retained log: Older data may be better suited to UltraWarm or cold storage to reduce cost.
Real-World Engineer Notes
- Design the index lifecycle before deploying the domain. Retention, rollover, shard sizing, replicas, and storage tiers strongly influence cost and performance.
- Avoid sending uncontrolled high-cardinality data into indexes. Excessive fields, large documents, and poorly designed mappings can increase storage and query costs.
- Treat VPC placement, subnet selection, security groups, DNS, and administrative access as part of the initial design.
- Monitor cluster health, CPU, memory pressure, storage utilization, indexing latency, search latency, and rejected requests.
- Plan ingestion backpressure. A streaming producer should not overwhelm the cluster during spikes or recovery events.
- Use restrictive access policies and separate human administration from application write access whenever possible.
- Keep snapshots and test restoration. A replica does not protect against accidental deletion, corruption propagated to replicas, or an operational mistake.
- Use hot storage only where its performance is justified. Tiering older data can significantly reduce the cost of long-retention analytics.
Quick Reference Summary
- Service: Managed search, logging, visualization, and analytics platform.
- Legacy name: Amazon Elasticsearch Service.
- Cluster name: OpenSearch domain.
- Common ingestion: Kinesis Data Firehose, Logstash, APIs, and applications.
- Visualization: OpenSearch Dashboards; historically Kibana in the ELK stack.
- Resilience: Multiple AZs, balanced nodes, replicas, and dedicated master nodes.
- Backup: Snapshots.
- Private design: Deploy in a VPC and provide VPN, Direct Connect, or another private access path for external clients.
- Detailed authorization: Fine-grained access control.
- Security layers: IAM, domain access policies, authentication, fine-grained permissions, encryption, and network controls.
- Storage optimization: Hot storage for active data, UltraWarm or cold storage for less frequently accessed data.
Flashcards
1. What is an OpenSearch domain?
A managed OpenSearch cluster containing the configured search, data, and optional dedicated master nodes.
2. What service preceded Amazon OpenSearch Service?
Amazon Elasticsearch Service.
3. What is the main reason to deploy OpenSearch in a VPC?
To keep communication with other VPC resources private and avoid exposing the domain through a public endpoint.
4. How can an on-premises client reach a VPC-based OpenSearch domain?
Through private connectivity such as Site-to-Site VPN or Direct Connect, or through a controlled proxy architecture.
5. What do index replicas provide?
Additional copies of index data that improve availability and can support read workloads; they are not backups.
6. What provides backup and restore capability?
Snapshots.
7. Why use dedicated master nodes?
To isolate cluster-management responsibilities from data indexing and search workloads.
8. What is fine-grained access control used for?
Role-based authorization at index, document, and field levels, along with Dashboards access and multi-tenancy.
9. What is the difference between a domain access policy and an IAM identity policy?
A domain access policy is a resource-based policy on the domain; an IAM identity policy is attached to a user, group, or role.
10. What is the ELK stack?
Elasticsearch, Logstash, and Kibana—a common search, ingestion, and visualization combination. OpenSearch uses corresponding successor terminology.
11. When should UltraWarm or cold storage be considered?
When data must be retained but is accessed less frequently than active, hot data.
12. Can a domain freely switch between VPC and public deployment?
No. Endpoint placement is a significant deployment decision and cannot simply be reversed later.
Practice Questions
Question 1
A company runs applications in private subnets and must store and analyze application logs. Security policy prohibits search data from traversing the public internet. Which design best meets the requirement?
A. Deploy OpenSearch with a public endpoint and restrict access by IP address.
B. Deploy OpenSearch in the application VPC and connect the applications using private networking.
C. Deploy OpenSearch publicly and encrypt all data at rest.
D. Store logs in Amazon S3 only and use an internet-facing dashboard.
Correct answer: B
A VPC-based OpenSearch domain keeps traffic private between the applications and the search cluster. Encryption and access policies are still required, but a public endpoint would not meet the stated network requirement.
Question 2
A production OpenSearch domain must continue serving queries if an individual data node fails. Which configuration is most appropriate?
A. One data node with frequent snapshots.
B. Multiple data nodes across Availability Zones with at least one replica per index.
C. One data node with encryption in transit.
D. Three dedicated master nodes and no replicas.
Correct answer: B
Multiple AZs and replicas provide operational availability when a data node fails. Dedicated master nodes help cluster coordination but do not replace data replicas.
Question 3
An organization needs to allow analysts to query one index, prevent them from viewing sensitive fields in another index, and provide separate Dashboards tenants. Which OpenSearch capability is required?
A. Security groups only.
B. IP-based domain access policies only.
C. Fine-grained access control.
D. Encryption at rest only.
Correct answer: C
Fine-grained access control supports role-based authorization at index, document, and field levels and can control Dashboards access and multi-tenancy.
Question 4
A company retains seven years of logs. Most queries target the last 30 days, while older data is rarely accessed but must remain available for investigation. Which storage strategy is most cost-conscious?
A. Keep all data on hot storage indefinitely.
B. Delete all data older than 30 days.
C. Move older data to UltraWarm or cold storage according to access requirements.
D. Replace replicas with encryption at rest.
Correct answer: C
Tiering older, infrequently accessed data reduces storage cost while preserving retention. The precise tier depends on required access performance and retrieval expectations.
Question 5
An administrator deploys an OpenSearch domain into a VPC and later discovers that an operations team outside AWS needs access. What is the appropriate solution?
A. Change the domain to a public endpoint with no other changes.
B. Add the team’s public IP range to an IP-based policy on the VPC domain.
C. Establish private connectivity, such as VPN or Direct Connect, or use a controlled proxy path.
D. Disable encryption so the external team can connect.
Correct answer: C
A VPC domain is private. External users need a supported network path into the VPC or a controlled access component. Changing endpoint type is not a simple post-deployment adjustment, and disabling encryption would weaken security rather than solve connectivity.