You Have Aligned IT with Business - Are You Protecting the Business?
The financial industry must have the appropriate technologies in place to quickly adapt and change to evolving regulations, technology advancements and business process changes. PEFCU’s IT department must be visionaries to keep up with rapidly changing technology and maintain regulatory compliance. This requires long-term strategic planning and strong teamwork within the organization. Performing an analysis of the five year business plan and goals allows IT to strategically plan for those changes and determine the impact to the infrastructure and security of the business.
PEFCU invested in EMC and Cisco technologies ensuring we were responsive enough to face any challenge. Our strategy evolved to keep up with technology and regulations we must adhere to. In choosing vendors, most notably, EMC, we provide members and staff maximum uptime, security, and performance. More importantly, strategic planning for the growth of data and software applications is considered along with data retention, backup, archiving, and data security.
“In choosing vendors, most notably, EMC, we provide members and staff maximum uptime, security and performance”
PEFCU has branches in Florida and Georgia with centralized IT at our Lakeland, Florida office. Our Disaster Recovery site in Jacksonville is an exact mirror of the Lakeland office with a Tertiary site in Norcross GA, which hosts a VNX system for critical applications only.
Challenges in Protecting the Business
For business continuity/disaster recovery, we installed several EMC products to protect our business. Disaster Recovery is “top-of-mind” and availability is critical. We require 365/24/7 uptime for our customer facing e-services applications. In a competitive market we must differentiate ourselves from competitors through the use of technology. In order to protect the business and financial data we have had to evolve our Archiving, Backup and Recovery to reduce complexity, recurring costs, and expand data protection service levels.
To meet business and financial goals we consolidated backup and archiving on to EMC Data Domain appliances (DDRs) and replicated that data to 3 sites, maintaining data integrity and reducing business risk by ensuring quick data restoration. Operationally we improved replication performance, reduced RTO for backup and Disaster Recovery and operational efficiency by streamlining management of backup and recovery operations.
Environment & Existing Applications
We use EMC VMware Vsphere 5.0 on a Cisco UCS Blade system running 120 VMs. This system is attached to an EMC VMAX 10k with a mix of Flash, Fibre Channel and SATA drives using Fast VP.
PEFCU implemented an EMC Enterprise Storage Network that comprises two EMC VMAX 10K storage arrays located in Lakeland and Jacksonville. Each unit is configured with a total of 40 TB useable capacity.
The units employ three distinct storage tiers using EMC FAST technology to migrate data amongst the tiers depending on usage. Tier 1 is the highest performing and contains 1 TB of EFD/Flash Drives. 15K fiber channel drives (18 TB) on Tier 2 and 7200 RPM SATA drives (21 TB) on Tier 3. EMC FAST technology uses sophisticated algorithms to determine proper tier placement for data and dynamically moves data amongst the tiers to ensure optimal performance.
The ESN is comprised of two Cisco MDS 9148 switches in Lakeland, two Cisco MDS 9124 switches in Jacksonville and EMC Recover Point Appliances (RPA). Each switch constitutes a complete VSAN in the storage network providing a logical security boundary. PEFCU’s ESN employs the concept of mapping, zoning and LUN masking to restrict and enforce access to SAN Data.
EMC RPAs are designed to incorporate Disaster Recover/ Business continuity with one cluster at Lakeland and another cluster at Jacksonville, allowing bi-directional SAN based replication. Each cluster consists of four 1U Recover Point appliances that provide asynchronous replication between Lakeland and Jacksonville across a 200 Mbps Ethernet Private Line (EVPL). This mode of replication sends point in time copies and allows for the recovery of replicas on a very granular level and protection against replication of deletes or data corruption from site to site so we can return to a restore point an instant before the deletion or corruption occurred.
These RPA’s work with the EMC VMware SRM solution to providing an efficient disaster failover/failback solution requiring minimal user intervention. This allows for the failover of one or more VMs from Lakeland to Jacksonville and back.
Archiving, Backup and Recovery Overview
The Credit Union employs several other EMC products to protect the business and member data.
SourceOne was implemented for archiving and 5 year data retention of e-mails. Exchange E-mails are journaled and SourceOne pulls these emails from the Journal to a staging area on the server and written to the SourceOne archive on the DDRs. SourceOne provides E-discovery for all archived content and Email supervision for regulatory compliance.
After 90 days, emails are shortcutted and represented as a stub in Outlook clients. After one year the stub is removed from Outlook and Exchange and the user can search for e-mails using many parameters in the web client of SourceOne.
All applications write to production DDRs and backups replicate from the production DDRs to the hot and warm sites using cascading replication. The basic diagram of our current VMAX environment includes Recover Point and Cisco MDS switches along with Site Recovery Manager (SRM), allowing for full site failover in approximately 1 hour.
VMDKs are backed up to the DDR’s and EMC DiskXtender backs up our Imaging system to the DDRs as well.
Data is written to DDRs over Ethernet via CIFS shares or NFS mounts. The retention of the files is defined by the backup applications, not Data Domain. All data on the appliances are deduplicated and compressed using CPU and memory, is continually verified by the appliance, and available whenever needed. DDRs maintain statistics on all content written and report their overall statistics and health daily.
Data Domain Statistics
An example of data reported upon is the amount of data written before deduplication, as well as disk capacity .
TCO savings achieved are the consolidation of archive data onto DDRs as the sets are deduplicated together, resulting in disk capacity savings overall and reduction ratios directly correlate to WAN bandwidth savings, as data is replicated deduplicated between facilities.
Because Data Domain is used as a NAS, many applications can write to it over Ethernet directly. This results in software license cost savings, as applications can write backups directly to DDRs without requiring additional backup software costs.
In summary, EMC provides PEFCU industry leading hardware and software for application protection, backup, recovery, archive, and compliance.