Last weekend, I was engaged by one of our biggest clients, a bank, to upgrade its virtual infrastructure to vSphere 6 and to migrate their infrastructure to new VMAX 3 storage arrays in both main and DR site. It’s one of only two customers in Egypt that uses vSphere SRM with VMAX Arrays for DR purposes.
VMAX3 Arrays are enterprise-level arrays from EMC. They’re the descendant of the great Symmetrix Arrays and they use their own replication engine, Symmetrix Remote Data Facility (SRDF).
To manage SRDF and integrate it with any other solution for DR Automation and Management, it requires a software component called Solutions Enabler (SE). This software consists of a client and a server component. SE server is either installed on a separate physical server (just like) in the old days, a VM or a virtual appliance or currently (it) is shipped on VMAX3 directors as a standalone virtual appliance. The common scenario for deployment with SRM is to use it in a server-client topology, in which the Solutions Enabler client is installed on SRM server and it’s connected to Solutions Enabler server. The following screenshot from EMC SRDF Guide with VMware shows the topology.
Back to my client. According to the design agreed upon, we had to use the vCenter Appliance for management in both the main and the DR site and this was the first issue. It prevented me from using SRDF Array Utility, which is a GUI tool to manage SRDF settings and its integration with SRM using vSphere Web Client. It supports only the Windows vCenter and its Web Client, not the appliance.
Note: SRDF Array utility only supports Windows vCenter. For more information and a comprehensive guide, check this blog post. Future releases may support vCenter Appliance.
The second issue was a technical issue in the Solutions Enabler virtual appliance (that is) shipped on VMAX3 Directors. We opened a ticket with EMC Support and we are still working on it. As a workaround, we decided to use a standalone VM SE Server installation on a Windows 2012R2 VM per site.
The following steps show a guideline of the configurations done to integrate Solutions Enabler 8.1 with the VMAX3 Array and with vSphere SRM 6.0 (done on each site separately: Main and DR):
1- We made sure that time-sync between each vCenter and SRM machines are accurately configured. Any time lag/lead between SRM and vCenter Server will cause connection issues between them and hence your DR capabilities will be threaten.
2- We installed Solutions Enabler Server on a Windows VM (Keep a note of the installation path). During installation, components selected by default should be sufficient. Also, we selected the following daemons to be installed:
Base Daemon, SYMAPI Server Daemon, RDF Daemon and Event Daemon
3- After installation of the SE Server, the Gatekeepers should be added. The Solutions Enabler server receives the commands from Solutions Enabler client installed on SRM using TCP/IP. To pass the commands to the VMAX Array in form of low-level commands (syscalls), it either should have direct access to the devices using HBA (in case of physical server installation) or using a small devices (3MB in size) called Gatekeepers. Number of Gatekeepers are calculated as following
Number of Gatekeepers= 6+ 0.3* Number of Consistency Group created on the Array
As we were using Solutions Enabler server as a Windows VM, we created 8 RDMs on the VMAX array, each with 3MB size, and added them as disks on Solutions Enabler Windows VM. To allow for vMotion and DRS operations, these RDMs are masked to the entire ESXi Clusters hosting this VM. RDMs of the main site Solutions Enabler Server are from local VMAX Array and the same for DR site.
4- After we had finished the setup of the Solutions Enabler Server, and the SRM VM, we installed the SRDF Storage Replication Adapter (SRA) downloaded from VMware.com then we installed Solutions Enabler Client Component. The Solutions Enabler Client component installation was done using the same file used in step (1). We only selected Base Daemon and Event Daemon to be installed (Keep a note of installation path).
5- On Solutions Enabler server, we modified the nethost file -located in Installation_Directory/config/– to allow only SRM servers to access the Solutions Enabler server using the same account SRM service runs with using the following syntax:
Main_Site_SRM_FQDN <tab> SRM_Service_Account_without_any_domain_prefix DR_Site_SRM_FQDN <tab> SRM_Service_Account_without_any_domain_prefix
These two lines should be the same in both Solutions Enabler Servers in main and DR sites.
6- On SRM/Solutions Enabler Client server, we modified the netcnfg file -located in Installation_Directory/config/– to allow connection between client component and server component. We used the following syntax:
SYMAPI_SERVER - TCPIP FQDN_of_Solutions_Enabler_Server IP_of_Solutions_Enabler_Server Port_of_Connection Security_Level_used
Port_of_Connection: if not set, it's 2707 by default. Security_level_used: can be ANY, SECURE or NONSECURE.
This should be done on SRM/Solutions Enabler Client Server in both sites.
7- On the SRM/Solutions Enabler Client server, we navigated to C:\Program Files\emc\SYMCLI\bin\ and using an Admin CMD, we set SE Connection type to local using the following command:
set SYMCLI_CONNECT_TYPE=LOCAL
This should be done on SRM/Solutions Enabler Client Server in both sites.
8- On the SRM/Solutions Enabler Client server, we navigated to C:\Program Files\emc\SYMCLI\bin\ and using an Admin CMD, we authorized vCenter Server using the following command:
symcfg authorization add -host FQDN_of_vCenter_Server -username vCenter_Admin_Account_Username -password vCenter_Admin_Account_Password-vmware -namespace "vmware/vc"
This should be done on SRM/Solutions Enabler Client Server in both sites.
9- We restarted the SRM services to make sure that the services are refreshed after these configurations.
10- After doing all of this, we added the two sites in vSphere Web Client, then we added Array Manager Pair. After a successful re-sync, all replicated LUNs are shown correctly and we started creating Protection Groups and Recovery Plans.
I hope this LONG guide to be helpful and will save some of you lots of time and effort of troubleshooting I had to waste time on 😀
References: (Documents may need EMC Partner Account for access)
** Solutions Enabler Installation Guide.
** Using EMC SRDF Adapter for VMware Site Recovery Manager.
** EMC VMAX3 Security Configuration Guide.
** Solutions Enabler Array Management: CLI User Guide.
Leave a Reply