Guidance for running Elasticsearch on Azure
Elasticsearch is a scalable open source search engine and database that has been gaining popularity among developers building cloud-based systems. When suitably configured, it is capable of ingesting and efficiently querying large volumes of data very rapidly.
It’s reasonably straightforward to build and deploy an Elasticsearch cluster to Azure. You can create a set of Windows or Linux VMs, then download the appropriate Elasticsearch packages to install it on each VM. Alternatively, we published an ARM template you can use with the Azure portal to automate much of the process.
Elasticsearch is highly configurable, but we’ve witnessed many systems where a poor selection of options has led to slow performance. One reason for this is that there are many factors you need to take into account in order to achieve the best throughput and most responsive system, including:
•The cluster topology (client nodes, master nodes and data nodes)
•The structure of each index (the number of shards and replicas to specify)
•The virtual hardware (disk capacity and speed, amount of memory, number of CPUs)
•The allocation of resources on each cluster (disk layout, Java Virtual Machine memory usage, Elasticsearch queues and threads, I/O buffers)
You cannot consider these items in isolation, because the nature of workloads you are running will also have great bearing on the performance of the system. An installation optimized for data ingestion might not be well-tuned for queries, and vice versa. Therefore, you need to balance the requirements of the different operations your system needs to support. For these reasons, we spent considerable time working through a series of configurations, performing numerous tests and analyzing the results.
The purpose was to illustrate how you can design and build an Elasticsearch cluster to meet your own requirements, and to show how you can test and tune performance. This guidance is now available in Azure documentation. We provided a series of documents covering:
•General guidance on Elasticsearch, describing the configuration options available and how you can apply them to a cluster running on Azure
•Specific guidance on deploying, configuring, and testing an Elasticsearch cluster that must support a high level of data ingestion operations
•Guidance and considerations for Elasticsearch systems that must support mixed workloads and/or query-intensive systems
We used Apache JMeter to conduct performance tests and incorporated JUnit tests written using Java. Then we captured the performance data as a set of CSV files and used Excel to graph and analyze the results. We also used Elasticsearch Marvel to monitor systems while the tests were running.
If you'd like to repeat these tasks on your own setup, the documentation provides instructions on how to create your own JMeter test environment and gather performance information from Elasticsearch, in addition to providing scripts to run our JMeter tests.
Read more about the comparison from LinkedIn here
Learn about how you could qualify for a free Microsoft trial from us here!
Microsoft Azure Stack Technical Preview is now available for download
The first Technical Preview of Azure Stack is now available for download!
This Technical Preview includes these fundamental capabilities to help you deliver Azure services in your datacenter:
-
Developer and IT professional experiences: Azure portal experiences for application developers and service administrators
-
Unified application model: Azure Resource Manager
-
Foundational services: Compute (Virtual Machines and Virtual Machine Extensions), Networking (Virtual Network, Software Load Balancer, Distributed Firewall) and Storage (Blobs and Tables)
-
Core services: Subscription management (identity and quotas), role-based access control, metering and audit
-
Application components: Curated Azure Resource Manager templates and Azure-consistent VM extensions (such as Desired State Configuration and Docker/Linux integration) to help you jumpstart your Azure Stack efforts
Hybrid Cloud without the Hassle, Simply Connect to Azure with Availability on Demand
Availability is not merely Disaster Recovery in the cloud, it’s the empowering connection between a datacenter and the cloud for protection and value creation. With today’s announcement, Microsoft provides comprehensive Availability on Demand in Azure for hybrid and heterogeneous environments, allowing organizations to harness a near unlimited amount of compute and storage in the cloud for dev/test, cloud bursting, migration, reporting/analytics, recovery, backup and long-term data retention.
Microsoft offers an unrivaled integration of on-premises assets with Azure in one unified solution, eliminating the hassle of managing multiple point solutions, and a mix of cloud providers. Whether an environment has Hyper-V, Physical (in preview) or VMware (in preview) assets, with Availability on Demand, the power of Azure is just a few clicks away. Merely connect your datacenter to Azure with Availability on Demand, and your data and applications will be available in Azure when needed.
-
Eliminate the need to build and maintain a secondary DR site.
-
Save time and money, by eliminating tape backup and house up to 99 years of backup data in Azure.
-
Easily migrate running Hyper-V, Physical (in preview) and VMware (in preview) workloads into Azure to leverage the economics and elasticity of the cloud.
-
Run compute intensive reports or analytics on a replicated copy of your on-premises asset in Azure, without impacting production workloads.
-
Burst into the cloud and run on-premises workloads in Azure, with larger compute templates when needed, to give you the power you need, when you need it.
-
Create multi-tier development environments in Azure with a few clicks – even replicate live production data to your dev/test environment to keep it in near real-time sync.
Availability on Demand provides a seamless connection between your on-premises production systems and Azure or a secondary site, leveraging the replication capabilities of Azure Site Recovery and the data protection of Azure Backup. With the ability to provide both on-premises support and connect to Azure and replicate Hyper-V and now Physical and VMware workloads (in preview), organizations can implement a true hybrid model, and take advantage of compelling cloud economics and elasticity.
