7 Steps to Success with an Automated Dev/Test Cloud

Although cloud computing is continuing its relentless march forward, many enterprises are still striving to identify the best scenarios for deploying this exciting new architecture. For reasons that we'll soon describe, the job of building, testing, and deploying software is an ideal use case for cloud computing. However, until recently there have been relatively few proven, real-world guidelines that explain exactly how to proceed down this path. In this whitepaper, we illustrate why well-designed, specialized software delivery technology paired with automated cloud infrastructure provisioning is a marriage made in heaven. This powerful combination – known as the Agile Development Cloud – helps you deliver high-quality software more quickly and effectively than ever before. And to assist you on your journey, we present a collection of best practices that show you how to arrive at this desirable destination by using cloud infrastructure software from Eucalyptus and software delivery solutions from Electric Cloud.

The intended audience for this paper includes software development managers as well as IT infrastructure leaders and managers.

Why the Software Build/Test/Deploy Process is a Great Fit for Cloud Computing There are two key drivers why the software development (build/test/deploy) process is a great fit for cloud computing:

  1. The intrinsic nature of complex software development, which demands lots of resources for short bursts of time.
  2. The rapid growth of Agile development methodologies, which mandate process automation and infrastructure technologies.

Let's examine each of these drivers in more detail. First, building, testing, and deploying complex software encompasses a number of characteristics that are ideal beneficiaries for the elasticity and scalability enhancements fundamental to cloud computing. These traits include:

  • Highly variable workloads, with minimal predictability
  • Each project is in its own unique phase of the lifecycle
  • Instant need for resources
  • Relatively brief processing windows
  • Software development teams distributed around the world

Therefore it's no wonder that software development is frequently the first use case for the cloud-based technologies that support these requirements so well.

Secondly, many enterprises are adopting the Agile software development methodology. It's composed of a number of techniques designed to help deliver incremental releases of high-quality software more quickly than ever before. The iterative Agile software development process leads to dramatically increased number of build/test/deploy cycles – often tens to hundreds of times per week.

Why IaaS is the Right Cloud Computing Choice for the Software Build/Test/Deploy Process

As we've just described, cloud computing is a great solution for the problems facing modern software development organizations. But the only way that the cloud can be properly deployed is if IT managers are able to do their part and supply the right kind of Infrastructure as a Service (IaaS).

First, it's essential that the selected IaaS solution be capable of offering sufficient scalability (defined as the ability to allocate additional processing firepower) and elasticity (defined as the ability to quickly supply and then release these resources) to support the unique computational demands of the software build/test/ deploy process. Secondly, these capabilities need to be available across all three major cloud computing IaaS flavors: public, private, and hybrid. Finally, since the IaaS landscape is so dynamic, it's incumbent upon infrastructure managers to remain focused on several IaaS traits that are particularly relevant for supporting the software build/test/deploy process. Key questions that you should ask include:

  • Is it specifically designed for cloud computing? Far too many IaaS technologies are repurposed legacy platforms. Avoid "cloudwashing" by choosing something purposely built for the cloud.
  • Is it enterprise-ready? The IaaS solution must be robust and architected to cope with the heavy processing loads present in every Agile environment.
  • Is it designed for extensibility? Stay away from closed, proprietary technologies; instead, seek out open solutions with a diverse ecosystem.
  • Has it been proven by thousands of implementations? There are many new startups and approaches to IaaS. Your critical software delivery process is no place to experiment.

Now that we've described why cloud computing IaaS is a logical choice for the software delivery process, let's examine a series of seven guidelines to help you implement an Agile Development Cloud.

Seven Steps to Create the Agile Development Cloud

For each of these actions – which we advocate performing in sequence – we'll explain why it's important, how to perform it, and the advantages you'll realize. To bring these guidelines to life, we'll supply real-world examples using software from Electric Cloud and Eucalyptus. Table 1 provides a brief summary of each step.

Step Purpose
1. Model and Map your Software Development Process Gain a baseline understanding of how your build/test/ deploy process really works.
2. Enable Automation of your Software Delivery Procedures Apply specialized, integrated, and secure technology for thorough streamlining.
3. Report and Analyze Process Execution and Workflow Close the loop with comprehensive analytics; stop errors before they start.
4. Define Valuable Infrastructure Services Define and quantify the case for cloud computing. Be specific, and focus on business benefits.
5. Plan and Standardize the Cloud Platform Identify the resources that comprise the reference architecture for your cloud computing environment.
6. Plan and Architect the Deployment Get ready to support real-world processing loads. Focus on elasticity and scalability.
7. Bring the Power of Self-Service Cloud Computing to your Build/Test/Deploy Process Reap the rewards of complete and transparent software delivery automation. All necessary resources are automatically allocated and released, without the need for developer, tester, or IT intervention.

Table 1: Seven Steps to the Agile Development Cloud

1. Model and Map your Software Development Process

Today's software projects commonly require close collaboration among many dynamic, distributed teams. There are numerous obstacles that can derail this teamwork. First, the complexity of the software delivery process itself can be overwhelming. To make matters worse, a significant number of these procedures are specific to a single team, they're poorly documented, and there's little or no sharing of best practices. Even if the entire enterprise shared a single set of software delivery techniques, there are still many incompatible tools that remain to be integrated.

To help prepare your software build/test/deploy process for the Agile Development Cloud, follow these steps:

  1. Analyze and document your existing processes and team interactions – including handoffs. Make sure that all relevant team members are included. Try to find out what really happens, as opposed to what 'should' happen.
  2. Examine and model your existing scripts. This is one of the best ways to figure out exactly what's going in in your software delivery process. Plus, as we'll illustrate later, the Agile Development Cloud will be able to employ these scripts.
  3. Understand the integrations among your existing tools. You've made a significant investment in these tools; the Agile Development Cloud will protect these expenditures.
  4. Identify and inventory your most effective best practices. These will be carried forward in the Agile Development Cloud's easily accessible, centralized repository of these techniques.

Performing these tasks will help provide visibility into your process and results, which boosts compliance for both internal and regulatory policies. It will also identify any inefficiencies or other issues with your existing procedures, while improving the likelihood of sharing best practices across teams, geographies, business units, and so on.

2. Enable Automation of your Software Delivery Procedures

IT organizations continually strive to streamline as many tasks as possible. It's understandable why this is such an appealing approach: it provides maximum ROI on existing assets, reduces one-off procedures, and makes automated management possible.

However, in today's intricate software development environments, bare-boned virtual instances with pre-configured templates don't go far enough. For example, the typical build/test/deploy procedure consists of dozens of moving parts, which means that each time one of these platforms is needed, someone — usually on the development team — must spend a sizable amount of time to properly configure it. While this may barely suffice in a traditional software delivery scenario, the hundreds of monthly builds mandated by Agile development render manual configuration obsolete.

Instead of labor-intensive provisioning, try to follow these steps to help automate the software delivery process:

  1. Implement technology to automatically set up and then tear down resources upon completion of the build/test/release cycle. This should occur as part of the normal software build process, and should not require any developer or tester intervention.
  2. Make sure that disparate software development and testing systems are integrated. Your project structure should be flexible, and support cross-team operations.
  3. Strive to improve performance. Parallelize and distribute as many tasks as possible, making sure to account for dependencies and optimized network traffic.
  4. Implement fine-grained security. This is an often-overlooked but essential responsibility.
  5. Ensure repeatability and reliability. All of this work is for naught if you can't easily reprise these procedures at will.

Optimizing your software delivery process helps everyone. First, software developers and testers improve their workflow, throughput, accuracy, and performance. Secondly, IT becomes more productive, and is thus able to provide better service while making more proficient use of all your resources. Finally, the entire enterprise experiences a positive cloud computing experience, which boosts agility and innovation while controlling costs.

3. Report and Analyze Process Execution and Workflow

With hundreds of build/test/deploy cycles a week, even the smallest error will be magnified and multiplied. Even though automation is a great technique to boost efficiency and accuracy, it can only go so far if you still rely on manual analysis and error diagnosis techniques. The best way to identify and correct anomalies is to run comprehensive reports and then take action on them. To make this happen, follow these steps:

  1. Define and manage metadata. This knowledge will be used to drive many aspects of error correction and reporting. Luckily, many of these metrics will be determined during the exercise of automating your software delivery process.
  2. Set up a consolidated repository of all logs and relevant software delivery data. This information forms the foundation of all error correction and management reporting, so it's critical that you make it consistent and easily accessible.
  3. Implement a comprehensive, dedicated reporting platform. Rather than attempting to customize a generic, all-purpose reporting product, it's much wiser to employ specialized reports that are fine-tuned to the unique details generated during the software delivery process.
  4. Create specialized dashboards. Although reports are an extremely useful element of your software toolkit, real-time dashboards will supply team members, managers, and executives with actionable, up•to-the-minute intelligence. Closing the loop with all-inclusive, precise reports and dashboards enables continual software delivery process improvement. In turn, this increases the reliability of project plans and lets supervisors optimize their time and manage by exception.

Up to this point, we've been describing the steps necessary to prepare your software delivery process for the Agile Development Cloud. However, to successfully automate these important tasks, it's vital that you also deploy the right blend of cloud computing technology. We describe these strategies next.

4. Define Valuable Infrastructure Services

At the beginning of this paper, we explained that while most enterprises realize that cloud computing is a big part of their future, a significant number of them have very unfocused goals for this new technology. This fuzziness inevitably results in a host of undesirable side-effects, including delays, cost overruns, security gaps, and a general perception that the investment in cloud computing wasn't worth it.

To keep the transition to cloud computing on the right track, follow these steps:

  1. Recognize that cloud computing is a blanket term, composed of compute, network, storage, identity, and other services. Each of these should be analyzed and planned for separately.
  2. Come up with the top 3 business-oriented use cases for cloud computing. Keeping a limited list helps you stay focused; naturally, the build/test/deploy process should be one of these use cases.
  3. Determine requirements for infrastructure consumers. You need to establish who will be accessing your Agile Development Cloud, along with their needs. The answer to this question will help you determine the exact scope of your infrastructure.
  4. Define your needs for SLAs, accounting, and chargebacks. You should consult with the consumers from the previous step, along with the people who sign the checks. As we'll see later, following best practices will let you provide them with accurate metrics, but you should bear in mind that these policies will evolve over time. Targeted and well-justified cloud computing implementations always prevail over "build it and they will come" initiatives. Understanding your constituencies and their workloads in advance makes it much easier to come up with the right configurations. And since many cloud computing undertakings stumble over financial matters, defining reasonable goals in advance will help set proper expectations.

5. Plan and Standardize the Cloud Platform

Without standards and procedures, it's easy for a cloud computing initiative to become chaotic. For example, in the absence of well-defined, functioning infrastructure that is accessible via self-service, many developers and testers will turn to the public cloud to get their jobs done. They'll upload and install components necessary for their application, plus any custom-built software and data. This exposes your enterprise to many risks, including loss of control, licensing issues, potential liabilities, and excessive costs. Plus, intellectual property and potentially sensitive data is now offsite.

To forestall this undesirable possibility, follow these procedures to help select a reference cloud computing architecture:

  1. Define how you'll use virtualization. Since virtualization is a key underpinning of cloud computing, you'll need to determine if you'll rely on virtual machines from a single vendor or multiple vendors. You should also bear in mind that virtualization also applies to network and storage, and can be separately scaled for each one.
  2. Survey your entire existing infrastructure. Many enterprises have large numbers of isolated silos, often containing enormous amounts of duplicated assets. Failing to take advantage of these resources often leads to unnecessary hardware purchases, which can have a devastating impact on the economic justification for cloud computing.
  3. Determine your infrastructure automation plan and tools. There's no need to manually configure and monitor your cloud computing infrastructure. Instead, it's much wiser to put the power of automation to work for these essential tasks.

Standards keep your software delivery teams concentrated on their core responsibilities, rather than on cobbling together costly, one-off cloud computing assemblies. It reduces the risk of unauthorized offsite initiatives and improves the ROI of existing assets. It also prepares you for future cloud computing scenarios such as sharing, hybrid, and other variants. Figure 1 displays an example of setting up a private cloud using Eucalyptus.

Figure 1: Configuring a private cloud

Configuring a private cloud

6. Plan and Architect the Deployment

Now that you've standardized your services as well as related infrastructure, it's time to turn your vision into reality. This phase identifies any gaps between planning and implementation. It also ensures that you'll get maximum reuse from your existing technology investments while reducing the likelihood of extraneous purchases. Major steps include:

  1. Understand workloads and security needs. Consulting with all interested parties will help make this possible. Pay particular attention to security, being sure to factor in how you'll comply with the laws and regulations for all the jurisdictions in which you operate.
  2. Make sure your plan allows for separate scaling of major resources. As we described earlier, cloud computing is a blanket term consisting of compute, storage, identity, network, and other resources. Each of these elements will have their own usage patterns.
  3. Prepare for private and hybrid cloud configurations. Certain assets should not be placed in the public cloud. However, seasonal and other usage variations might necessitate "cloudbursting" processing workloads or data from your private cloud into the public cloud. Now is the time to plan for these types of events.
  4. Set up formal management and reporting processes. This step is essential if you want to demonstrate the economic benefits of cloud computing.

You're now prepared for the most likely processing scenarios, including variability triggered by seasonal, business process, and other factors. You've incorporated elasticity and scalability up front, and you're now ready to take advantage of all 3 major cloud computing styles: public, private, and hybrid. Finally, by incorporating accurate reporting and monitoring, you've set up a feedback loop to help continually correct, tune, and quantify your cloud computing initiatives.

7. Bring the Power of Self-Service Cloud Computing to your Build/Test/Deploy Process

The best cloud computing infrastructure and process automation won't live up to their full potential if developers and testers are forced to continually request outside assistance from IT: the number of compilations and builds mandated by Agile development alone makes this implausible. Instead, everything should be geared toward giving these users the freedom to help themselves. This means that all of your internal technology assets should be enlisted by your process automation solution to complete the software build/test/deploy process as efficiently as possible. You'll reap the following rewards from this smooth integration:

  • Complete customization for developer/tester needs. As we described earlier, modern software development environments are extremely intricate and consume enormous amounts of time to manually configure. The Agile Development Cloud performs all necessary steps to ensure that each resource is properly staged, all without needing any intervention from developers or testers.
  • Automatic setup and teardown. All necessary resources are acquired and then immediately released upon completion of the software delivery process. This eliminates the serious problem of 'virtual machine sprawl', which occurs whenever developers or testers manually requisition a virtual machine yet neglect to free it after they're finished.
  • Transparent load balancing. The Agile Development Cloud offers two levels of automatic workload management to optimize resource utilization. First, the software delivery platform leverages parallel processing to spread the work across multiple systems. These distributed platforms then collaborate to deliver results more quickly than ever before. Second, the IaaS cloud platform implements the proven strategies of scalability and elasticity to further drive high performance.
  • End-to-end reporting. Since the Agile Development Cloud serves two constituencies, it's natural that it offers all-inclusive business intelligence for both communities. For example, development managers are able to monitor resource usage patterns by project, process, user, and so on. At the same time, IT is able to analyze workload levels for all of their assets – including public, private, and hybrid cloud components – which results in much more accurate chargeback, planning, and budgeting.
  • Operational cloud computing. At the start of the paper, we explained that numerous enterprises are seeking scenarios that will benefit from cloud computing. Once the Agile Development Cloud is operational, many of these organizations elect to extend the power of this IaaS cloud implementation beyond the software build/test/deploy process to support other categories of production systems.

Figure 2 supplies an example of the close integration between the Eucalyptus and Electric Cloud solutions. In this case, a user is being defined with a specific purpose of supporting the software delivery process.

Eucalyptus and Electric Cloud Integration

Conclusion

With more enterprises adopting Agile software development methodologies, the already-frenetic pace of the software build/test/deploy process will continue to accelerate. Specialized software delivery technology such as provided by Electric Cloud brings order and predictability to these increasingly chaotic operations. But it's equally important to buttress the build/test/deploy automation solution with cloud-based Infrastructure as a Service. IaaS – with its unlimited scale and fast resource setup – provides the ideal solution to support the heavy resource requirements endemic to Agile development processes.

Automating your software delivery process and then integrating it with cloud-based IaaS is the best way to keep pace with today's demands. However, this approach is not for everyone. To help determine if it's a good match for your organization, consider whether any of the following statements describe your environment:

  • Migration towards Agile development methodologies, with a focus on speedy process execution
  • A plan to move toward shared resources to enable infrastructure efficiencies
  • A goal of sharing best practices across teams
  • A desire to automate wherever possible
  • Interest in employing virtualization to put unused assets to work
  • A strategy of standardizing on core cloud infrastructure
  • A need to leverage public, private, and hybrid clouds

Getting started with the Agile Cloud is easy:

About Electric Cloud

Electric Cloud is the smart development cloud company. Electric Cloud bridges the gap between physical and virtual IT environments and development processes and tools to create smart development clouds. The company's patented and award-winning products help development organizations to speed time to market, boost developer productivity, and improve software while leveraging the operational efficiencies provided by virtualized/cloud infrastructures. Leading companies across a variety of industries, including financial services, ISVs, mobile devices, semiconductors, and transactional websites rely on Electric Cloud's development cloud solutions. For more information about Electric Cloud, please visit www.electric-cloud.com.

Get Started with Eucalyptus

Download FastStart for free to deploy a Eucalyptus cloud in 30 minutes.