Enterprise Software are known to be business-critical and business-oriented software that each company develops its own one to serve their business needs. Today, we gonna talk about Java Enterprise Applications. Java Enterprise Apps are multi-tier Enterprise Software that consists of -generally- from 3 tiers: Web Interface, Application Processing Tier and a Back-end DB. Each tier isn’t a single instance -either physical or virtual-, but each tier consists of a cluster of instances to serve the purpose of the tier. This makes Java Enterprise Applications excellent Virtualization candidates.
vSphere 5.x with its enormous features, can deliver the required performance, scalability and availability level for Java Enterprise Applications clusters. With HA, DRS, vMotion and other many features, vSphere Platform can extremely decrease the physical hardware required for these applications clusters and hence, decrease the required cost and management and in the same time, without affecting their performance.
I’ll follow the same schema of my previous posts and relate these best practices to our five Design Qualifiers (AMPRS – Availability, Manageability, Performance,Recoverability and Security) in addition to Scalability. For more information, check the References section below.
Availability:
1-) Try to use vSphere HA with your Java VMs to provide decent level of availability.
2-) Try to separate your Java VMs, from the same tier, on different Racks, Blade Chassis and Storage Arrays if available using VMs Affinity/Anti-affinity rules and Storage Affinity/Anti-affinity rules for highest level of availability. Keep in mind that, when HA restart a VM, it’ll not respect the anti-affintiy rule, but on the following DRS invocation, the VM will be migrated to respect the rule. In vSphere 5.5, configure the vSphere Cluster with “das.respectVmVmAntiAffinityRules” set to 1 to respect all VMs Anti-affinity rules.
3-) Try to leverage VM Monitoring to mitigate the risk of Guest OS failure. VMware Tools inside Java VMs will send heartbeats to HA driver on the host. If it’s stopped because Guest OS failure, the host will monitor IO and network activity of the VM for certain period. If there’s also no activity, the host will restart the VM. This add additional layer of availability for Java VMs. For more information, refer to: vSphere HA VM Monitoring – Back to Basics | VMware vSphere Blog – VMware Blogs.
4-) Try to leverage Symantec Application HA Agent for Oracle with vSphere HA for max. availability. Using Application HA, the monitoring agent will monitor Java Application instance and its services, sending heartbeats to HA driver on ESXi host. In case of application failure, it may restart services and any dependent resources. If Application HA Agent can’t recover the application from that failure, it’ll stop sending heartbeats and the host will initiate a VM restart as a HA action. For more information about Symantec Application HA, refer to this pdf.
Performance:
1-) Follow all VMware best practices for Latency-Sensitive Applications in this pdf: Tuning Latency-Sensitive Workloads on vSphere.
2-) Configure the following BIOS Settings on each ESXi Host:
| Settings | Recommended Value | Description |
| Virtualization Technology | Yes | Necessary to run 64-bit guest operating systems. |
| Turbo Mode | Yes | Balanced workload over unused cores. |
| Node Interleaving | No | Disables NUMA benefits if set to Yes. |
| VT-x, AMD-V, EPT, RVI | Yes | Hardware-based virtualization support. |
| C1E Halt State | No | Disable if performance is more important than saving power. |
| Power-Saving | No | Disable if performance is more important than saving power. |
| Virus Warning | No | Disables warning messages when writing to the master boot record. |
| Hyperthreading | Yes | For use with some Intel processors. Hyperthreading is always recommended with Intel’s newer Core i7 processors such as the Xeon 5500 series. |
| Video BIOS Cacheable | No | Not necessary for database virtual machine. |
| Wake On LAN | Yes | Required for VMware vSphere Distributed Power Management feature. |
| Execute Disable | Yes | Required for vMotion and VMware vSphere Distributed Resource Scheduler (DRS) features. |
| Video BIOS Shadowable | No | Not necessary for database virtual machine. |
| Video RAM Cacheable | No | Not necessary for database virtual machine. |
| On-Board Audio | No | Not necessary for database virtual machine. |
| On-Board Modem | No | Not necessary for database virtual machine. |
| On-Board Firewire | No | Not necessary for database virtual machine. |
| On-Board Serial Ports | No | Not necessary for database virtual machine. |
| On-Board Parallel Ports | No | Not necessary for database virtual machine. |
| On-Board Game Port | No | Not necessary for database virtual machine. |
3-) During testing and planning phase, try to establish a baseline of the ratio:
HTTP requests: Java Heaps: DB Connections required.
4-) CPU Sizing:
a- CPU Over-commit is allowed in Java Enterprise Applications VMs so that total physical CPUs utilization doesn’t exceed 80%. Performance monitoring is mandatory in this case to maintain a baseline of normal-state utilization.
b- Assign vCPUs as required and don’t over-allocate to the VM to prevent CPU Scheduling issues at hypervisor level and high RDY time.
c- Enable Hyperthreading when available. It won’t double the processing power –in opposite to what shown on ESXi host as double number of logical cores- but it’ll give a CPU processing boost up to 20-25% in some cases. Don’t consider it when calculating Virtual: Physical Cores ratio.
d- ESXi Hypervisor is NUMA aware and it leverages the NUMA topology to gain a significant performance boost. Try to size your Java VMs to fit inside single NUMA node to gain the performance boost of NUMA node locality.
5-) Memory Sizing:
a- Don’t over-commit memory, as Java Applications are memory-intensive. If needed, reserve the configured memory to provide the required performance level. Keep in mind that memory reservation affects as aspects, like: HA Slot Size, vMotion chances and time. In addition, reservation of memory removes VM swapfiles from datastores and hence, its space is usable for adding more VMs. For some cases, like testing environments, over-commitment is allowed to get higher consolidation ratios. Performance monitoring is mandatory in this case to maintain a baseline of normal-state utilization.
b- Don’t disable Ballon Driver installed with VMware Tools. Balloning is the last line of defense of ESXi Host before compression and swapping to disk when its memory becomes too low. When Balloning is needed, Ballon Driver will force Guest OS to swap the idle memory pages to disk to return the free memory pages to the Host, i.e. swapping is done according to Guest OS techniques. Swapping to disk is done by ESXi Host itself. Host’d swap memory pages from physical memory to VM swap file on disk without knowing what these pages contain or if these pages are required or idle. Balloning hit to the performance is somehow much lower than Swapping to disk. Generally speaking as mentioned in the previous point, don’t over-commit memory for business critical Java VMs and if you’ll do some over-commitment, don’t push it to these extreme limits.
c- Configure Memory Reservation on your Java VMs according to:
“Reserved memory= VM Memory= Guest OS Memory+ Java Memory (JVM Memory)”
d- Leverage Large Memory Pages feature. It can give a performance boost for your Java VMs. Keep in mind not to configure all Java VM configured memory as Large Pages and leave some memory to be used by small pages for processes that can’t leverage Large Pages.
6-) Monitoring:
Try to establish a performance baseline for your Java VMs and VI by monitoring the following:
– ESXi Hosts and VMs counters:
| Resource | Metric (esxtop/resxtop) | Metric (vSphere Client) | Description |
| CPU | %USED | Used | CPU used over the collection interval (%) |
| %RDY | Ready | CPU time spent in ready state | |
| %CSTP | Co-Stop | Percentage of time a vCPU spent in read, co-descheduled state. Only meaningful for SMP virtual machines. | |
| %MLMTD | Percentage of time a vCPU was ready to run but was deliberately not scheduled due to CPU limits. | ||
| %SWPWT | Virtual machine waiting on swapped pages to be read from disk. This can indicate overcommitted memory. | ||
| %SYS | System | Percentage of time spent in the ESX/ESXi Server VMKernel | |
| Memory | Swapin, Swapout |
Swapinrate, Swapoutrate | Memory ESX/ESXi host swaps in/out from/to disk (per virtual machine, or cumulative over host) |
| MCTLSZ (MB) | vmmemctl | Amount of memory reclaimed from resource pool by way of ballooning | |
| N%L | If less than 80, the virtual machine is experiencing poor NUMA locality. If the virtual machine has memory size greater than the amount of memory local to each processor, the ESXi scheduler does not attempt to use NUMA optimizations for that virtual machine. | ||
| Disk | READs/s, WRITEs/s | NumberRead, NumberWrite | Reads and Writes issued in the collection interval |
| DAVG/cmd | deviceLatency | Average latency (ms) of the device (LUN) | |
| KAVG/cmd | KernelLatency | Average latency (ms) in the VMkernel, also known as Queuing Time‖ | |
| ABRTS/s | Aborts are issued by the virtual machine because the storage is notresponding. For Windows virtual machines, this happens after a 60-second by default. This issue can be caused by path failure, or when the storage array is not accepting I/O. | ||
| RESET/s | The number of command resets per second. | ||
| Network | MbRX/s, MbTX/s | Received, Transmitted | Amount of data received/transmitted per second |
| PKTRX/s, PKTTX/s | PacketsRx, PacketsTx | Received/Transmitted Packets per second | |
| %DRPRX, %DRPTX | DroppedRx, DroppedTx | Receive/Transmit Dropped packets per second |
Manageability:
1-) Time Synchronization is one of the most important things in Java Enterprise Applications environments. It’s recommended to do the following:
a- Let all your Java VMs sync their time according to the following best practices:
Windows: VMware KB: Timekeeping best practices for Windows, including NTP.
Linux: VMware KB: Timekeeping best practices for Linux guests.
b- Disable time-sync between Java VMs and Hosts using VMware Tools totally (Even after uncheck the box from VM settings page, VM can sync with the Host using VMware Tools in case of startup, resume, snapshotting, etc.) according to the following KB: VMware KB: Disabling Time Synchronization.
c- Sync all ESXi Hosts in the VI to the same Startum 1 NTP Server which is the same time source of your forest/domain.
2-) Try to leverage vSphere-aware load balancers that can integrate with vSphere API and automatically add newly added Java VM to its corresponding load-balancing pool.
3-) Use load balancers with known algorithms that you can understand so that you can test and configure them efficiently to make sure that each Java VM has equal share of requests and load is balanced.
Recoverability:
1-) Use VMware Site Recovery Manager (SRM) if available for Disaster Recovery. With SRM, automated failover to a replicated copy of the VMs in your DR site can be carried over in case of a disaster or even a failure of single critical SQL VM in your environment.
Security:
1-) All security procedures done for securing physical Java Enterprise Application environments should be done in virtual environment, like: Role-based Access Policy.
2-) Follow VMware Hardening Guide (v5.1/v5.5) for more security procedures to secure both of your VMs and vCenter Server.
Scalability:
1-) Leverage CPU/Memory Hot add with Java VMs to scale them up as needed according to needed performance.
2-) When increasing your VM Java Heap Size, increase your vCPUs as well for max. performance
3-) When adapting Scale-out approach of your Java VMs, try to use symmetric VMs in your scaling, so that load balancers can effectively load-balancing requests between them. Load balancers aren’t aware of VMs sizing and hence, non-symmetric VMs would lead to non-efficient load-balancing unless you configure VMs sizes on your load balancers algorithms, which is time consuming and sometimes, it’s hard to code.
I tried as much as possible to gather all the available information about best practices for virtualizing Java Enterprise Applications. Unfortunately, sources are rare and even VMware itself provides only few old documents which included in References section. I hope that I helped somehow. 🙂
References:
** Java Enterprise Application on VMware – Best Practices.
** vSphere Design Sybex 2nd Edition by Scott Lowe, Kendrick Coleman and Forbes Guthrie.
Leave a Reply