Symptom: Infrastructure constraints are limiting server sprawl at many enterprises.

Diagnosis: With advanced 64-bit x86 server technology now widely available, why not consolidate existing servers and applications onto fewer, more powerful servers?

It's a good remedy and one that's being implemented by data center managers far and wide, including IT executives at AMD. In December 2005, the AMD technology execs, who operate eight data centers around the world, realized that many of their legacy servers were nearing end of life. In addition, many running a single application were underutilized. To boot, AMD's main corporate data center in Austin, Texas, was essentially maxed out. Trying to squeeze more servers into the space there and in other data centers would have required big investments in a power and cooling infrastructure.

In response, the executives proposed "AMD on AMD," an ambitious, fast-track server consolidation project aimed at conserving space and lowering costs. The idea was to move the company's production applications onto a virtualized server infrastructure powered by AMD Opteron" processors and VMware ESX Server 3.0. The timeframe to accomplish this was one year, and the plan was to start in Austin and spread worldwide.

"AMD on AMD delivered the obvious and expected benefits of virtualization: eliminating physical servers by running multiple virtualized applications on one," says Walter Smith, director of IT strategy and architecture at AMD. He also cites "the flexibility of the environment, the potential for better availability and disaster recovery, and real improvements in customer-centric services delivery."

Virtues of Virtualization

Physical consolidation can be done by re-architecting applications to share servers, but this often causes problems in load balancing, server failure recovery and resource management. AMD executives decided to consolidate through virtualization, using the existing application architecture. Virtual servers would run on a farm of standardized commodity servers powered by AMD Opteron processors.

The next decision was choosing a virtualization platform. Due diligence led AMD to VMware and its ESX Server, a robust virtualization layer that installs on the "bare metal" and allows multiple unmodified operating systems and their applications to run in virtual machines that share physical resources.

AMD executives liked the new capabilities in ESX Server 3.0, such as support for large server farms and advanced memory management. VMware's VMotion software, which allows virtual machines to be easily moved from one physical server to another, also factored into the decision.

"VMotion was important to us, because early on, our ability to categorize and model the individual virtual servers was less than perfect," says Smith. "Having the ability to easily adjust the model by moving the [virtual servers] around provided a great safety net."

One challenge involved the release date of ESX Server 3.0: It was in beta when the project started, but was expected to be released during the project. Not wanting to face a costly upgrade from ESX Server 2.5, the team decided to go with the new version and test its stability on nonessential applications first.

Designing the Infrastructure

After choosing VMware, AMD brought in RapidApp, a Chicago-based IT consulting firm with a track record in large-scale server consolidation using VMware. The consulting team implemented a four-stage methodology (design, planning, implementation and transition to support) to move the project along its aggressive timeline.

The design phase started by surveying the Austin data center to decide which servers were candidates for virtualization. After reviewing existing server hardware specifications, CPU utilization, memory utilization and I/O rates, the team considered factors such as type and age of physical servers.

"If you find that 20 percent of your architecture is running at 5 percent [utilization], those [applications] are probably good candidates to put on a virtual server, because your consolidation rates will be highest," says Robert Koury, vice president at RapidApp.

Enter Hewlett-Packard. Next, the team designed an ESX Server 3.0 infrastructure based on HP ProLiant DL585 rack-mount servers with 48 gigabytes of memory. These four-processor 8-core servers based on AMD Opteron processors are the "sweet spot for both consolidation ratio and cost efficiency," Koury says. "We have worked extensively on virtualization projects with servers powered by AMD Opteron processors, and the results have been excellent."

The HP ProLiant DL585 servers are designed for customers that need outstanding performance in a smaller space. For a separate ESX server farm in AMD's Internet environment, the team chose HP ProLiant DL385 servers powered by two AMD Opteron processors.

The idea was that each physical server would run ESX Server 3.0, while all virtual machine configuration and application data would reside separately on the storage area network (SAN). In the event of a server failure, virtual machines could quickly be switched to another host.

Implementation Planning

Next, the team developed an implementation plan, establishing evening hours each week as time slots for migration. The implementation included three major phases: process development and testing, migration of nonproduction systems onto ESX Server 3.0 release-candidate code and migration of production systems onto ESX Server 3.0 production code.

The team worked with application owners by advertising migration time slots and allowing them to sign up on a first-come, first-served basis. They migrated approximately 25 nonproduction systems using release-candidate code, and then upgraded to production code and continued the migration of production systems. After successful application migration and testing, the legacy servers were decommissioned and removed from the data center.

"Our experience during implementation was consistent with what the infrastructure sizing analysis predicted," says Michael Winslett, IT architecture manager at AMD. "We experienced very few performance problems once the systems were migrated."

To ensure a smooth transition, processes for automating server and virtual machine configurations were developed during the design phase, as well as processes needed for successful operation and support of the virtual server environment.

"It's key to work out your operational and support processes in advance, and test them thoroughly before you put any system you care about in the environment," Winslett says.

After the implementation, the IT staff transitioned into supporting the virtualized production environment. The transition went smoothly due to the thorough testing of all support processes. These included processes for virtual machine requests, deployments and chargebacks; preventive maintenance; quality assurance for host and virtual machine builds; and monitoring configuration standards for virtual machines and hosts.

Rewarding ROI

Assisted by RapidApp, AMD consolidated and virtualized servers over a 10-month period at data centers in Austin, Texas; Sunnyvale, Calif.; and Singapore. In Austin, 117 servers were initially consolidated into seven active ESX 3.0 servers. Today, approximately 200 servers are running, leading to a consolidation ratio of 26:1. This number should increase as new virtual machines are added to the server farm, which still has considerable headroom.

All in all, the AMD on AMD project is expected to pay for itself in less than a year, compared to what AMD would have spent had it not virtualized its data center. For example, power savings in Austin are substantial. Additional savings have been achieved due to the conservation of data center space and infrastructure. Other savings accrue from procuring and supporting fewer servers. But the benefits extend well beyond hard dollar savings.

"In the past, when a customer needed a new application, we had to meet with the application staff to specify the operating system, the server and the configuration, and then order the server from the vendor," Smith says. "Weeks later, we would receive it, put in on the rack and get it all connected.

"In the virtual environment, that customer comes in, and, in a matter of minutes, we can have a new server running because it's all software," says Smith. "It's truly an agile and streamlined environment."

New Possibilities

Smith also sees a positive impact on application availability and disaster recovery. "Since the servers are now software, nothing prevents us from replicating them to the SAN in our Sunnyvale data center," he says. "If Austin went off the map, I could choose to boot up a copy without missing a beat. "

These possibilities are leading AMD to move full-speed ahead with virtualization worldwide. The company hopes to complete the initiative by the end of 2006.

"We are leveraging virtualization's power wherever it's feasible across all our major locations," Smith says. "Virtualization makes the IT function more nimble and effective in providing consistent, customer-centric service delivery."

Maximizing Benefits

AMD Opteron processors provide a robust foundation for maximizing the benefits of virtualization software such as VMware's ESX Server. These benefits include the following:

Performance: AMD64 technology with Direct Connect Architecture seamlessly connects the processors, memory controller and I/O to the central processing unit. This, combined with hardware-based memory management, can improve overall virtualization performance and efficiency.

Enterprise-class scalability: By providing efficient CPU utilization and I/O capabilities, AMD Opteron processor-based systems can support an exceptional number of virtual machines within a single physical system. For targeted applications, one AMD processor-based server has the ability to support large numbers of users on multiple virtual machines.

Effective workload management: The integrated memory controller offers high-speed, low-latency access to memory. Thus, host systems and virtual machines can operate effectively under peak loads and enable IT staff to efficiently balance workloads across the virtual environment.

These benefits can be seen in the four-processor HP ProLiant DL585 servers used in the AMD on AMD project, says Robert Koury, vice president at RapidApp, a Chicago-based IT consulting firm.

"In real-world situations, when you combine the advantages of the underlying AMD technology with the price point of the HP ProLiant DL585 servers, you receive excellent price-performance for virtualization," Koury says. "We're recommending these servers regularly now, because they can offer a significant advantage in cost per virtual server."