Adopting modern engineering processes and tools at Microsoft IT

As written on
Microsoft IT is adopting agile development methods as part of the move to modern engineering. Learn how we’re using Azure and Visual Studio Team Foundation Server, and about our processes for agile development and automated testing.
To support our customers and partners, Microsoft IT must respond more quickly to evolving business needs. Taking six months to deliver an application or an update is no longer an option. We need to deliver value faster and more efficiently. To increase our responsiveness, we started the journey to modern engineering. Our goal is to release new functionality every day in a process called continuous integration, continuous delivery.
An important part of this journey is adopting agile development processes and tools. We’re moving to an iterative design process and using agile methods to develop new services and features. And to help our engineers understand how to apply agile methods in their day-to-day work, we’ve embedded a set of practices, called Engineering Fundamentals, into our development environment. We’ve also set up tools and processes to run a project using agile methodology. This includes:
  • Moving to Visual Studio Team Services to manage our backlog and work items.
  • Reducing the Dev/Test environment footprint using virtualized environments hosted by Microsoft Azure.
  • Improving build and test automation.
  • Establishing a pipeline for continuous integration and continuous delivery using Visual Studio Team Services.
While we’re still working toward our goal of releasing every day, we’ve already shortened our release tempo from as long as six months to as little as two weeks.
Using agile methodology for planning and design
In the modern engineering model, design and architecture are iterative processes. Instead of using the traditional, slow process of creating detailed plans up front, we create sketches of the future state design at a relative level of detail and fidelity. Then we use rapid prototyping to validate and refine our design and get feedback about how it works. Everyone on the team applies deep and shared customer knowledge in every design decision. Throughout the design process, we solicit ideas from a diverse audience and draw upon the work of other teams.
Applying the Engineering Fundamentals
Each sprint team applies them to each project. A checklist defines every step that an engineer should take in the modern engineering approach to do each development task. This encourages engineers to think not only about what they’re delivering, but also about how they’re delivering it.
Engineering Fundamentals serve as requirements, and each fundamental has acceptance criteria that features inherit. Sprint teams make sure that the service they deliver adheres to the fundamentals. New employees use them as guiding principles to learn how to do engineering in our organization. And the Engineering Fundamentals are embedded in Visual Studio Team Services—where the engineers do their daily work—so they’re easy to find and use, as shown here.
The Engineering Fundamentals can be summarized as follows.
Continuous delivery
Dynamic and on-demand environment provisioning. Engineers can, at any time, provision an environment that includes all of the prerequisite components and dependency services, so that when code is deployed, it can run the designated functions.
Continuous integration. Engineers can, at any time, start an unattended build for a check-in that produces a functional build in minutes, which is deployable to any environment without competing for resources.
Continuous validation. Engineers can, at any time, start an automated validation process for a deployment that increases release readiness without manual effort, in minutes.
Continuous service deployment. Engineers can, at any time, start an unattended deployment process for a build that takes minutes in a functional environment without competing for resources.
Customer focus
Safe testing in production. Engineers can, at any time, start tests to learn from experimentation or prove service health.
Know the stakeholders deriving value from the service. Engineers keep a relentless focus on stakeholders: users, customers, and partners. This helps them communicate requirements and translate them into features, user stories, and tasks that produce an output that delights the customer.
Core engineering practices
Ready to code. Engineers can, at any time, check out code to any development environment (local or remote) to compile, run, debug—all within minutes.

Ready to build. Engineers can, at any time, provision a development environment with all of the prerequisite components, so that code can be checked out, compiled, and run.

Componentization. Engineers use componentization to make the codebase easier to build, compose, and deploy.

Security and privacy compliance. All services are highly secure and comply with security and privacy standards. Engineers integrate security infrastructure and tools into the continuous delivery process/pipeline.

Service health, analytics, and monitoring. Engineers use telemetry and data to form insights about the user experience, system health, the business value of the service, and to support automation.

Using Azure to improve agility

On-premises deployment takes a lot of manual work to configure and manage servers. Also, automated deployment in this scenario requires custom PowerShell scripts. To support continuous integration and agile release processes, we moved projects from Microsoft Team Foundation Server running on-premises to Visual Studio Team Services on Azure. Dev/Test environments are now virtualized. We deploy a complete virtual environment with a click. And we use Visual Studio Team Services to create automation pipelines to promote code, create environments, and to kick off tests.

Using Visual Studio Team Services agile and scrum templates, we set up projects in a few minutes. Templates also support uniform processes across teams, so it’s easier for engineers to move between projects. And because release manager includes a library of scripts, we need less custom code to seamlessly promote a build through development, staging, and production.

To increase speed and agility, we incorporate other Azure services into our solutions. When solution components must remain on premises for security reasons, we use Microsoft Azure Hybrid Connection Manager to integrate them with cloud services. New applications are entirely Azure-based.

Automating build and test

The biggest challenge in moving to an agile, continuous delivery model is ensuring the quality of the code. Pushing code into production without enough testing could break the build. Handling this issue in an agile manner requires evaluating risks and mitigating them proactively. Safety is important, but so is knowing which risks are acceptable.

To manage risks, we put modern engineering tools and processes in place that help prevent production problems and that mitigate problems before customers experience them. Test automation is an important part of this. Previously, code was handed off to the test team, and they executed one or more tests against each component of a feature or solution. For example, on a Bing site, the ability to navigate to a search page might be one test case. Another might be the ability to see the search box and another the ability to type in the search box. Another might be to get a set of results. Each test succeeds or fails. Testing each component separately in this manner is time-consuming.

Today we use a different approach. Engineers write code for functional tests based on end-to-end scenarios. Scenario-based testing puts the customer experience first. It addresses these questions: What is a customer really going to do? What are the key scenarios? What must work? What are the critical elements to test? How can we automate the tests? We no longer have explicit tests for each component, but rather use functional tests of outcomes. For example, a functional test might get search results for a query. To get faster turnaround, the team determines what minimum level of quality is required to ship the code. The goal is to test and correct failures faster.

Production-ready check-ins

We use build definitions in Visual Studio Team Services to support gated, production-ready check-ins. Code must pass tests to be promoted to the next phase of testing or deployment. If code fails to pass a test in any phase, the engineer must fix the code and then check it back in for retesting.

The phases are as follows:

  • An engineer tests code on a local computer.
  • The code is checked into a branch and incorporated into the daily build that includes all check-ins from the last 24 hours in an integrated environment with other systems that the new code has dependencies on. First, an automated build verification test runs, then automated tests run against the environment.
  • We manually test the build against scenarios that weren’t covered in automated testing.
  • Stakeholders/customers test the feature or application to verify that it meets their requirements.
  • The code is deployed into production. It’s deployed to a limited set of users first, and then if there are no issues, released to additional sets of users.

Improving business value and customer satisfaction

We still have a long way to go on our journey to modern engineering, but our efforts are already yielding benefits. We’re able to maintain a consistent and predictable release cycle and deliver updates and enhancements frequently. This way our development efforts yield business value faster. Using single source control within Visual Studio Team Services has helped us increase code reuse for more efficient development. Also, applications are “production ready” at release, and rework is reduced. Finally, breaking down releases into smaller chunks has reduced risk because all features represent two weeks of effort rather than several months.

Best practices

We found some practices that worked well when adopting the processes and tools for modern engineering.

To run fast, first slow down

Balancing the need to make infrastructure improvements with ongoing business needs for new functionality was a challenge. Stakeholders agreed on priorities and timelines to make sure that we would lay a good foundation. We made the right investments early, such as in test automation. Changing the infrastructure and moving from six-month to two-week release cycles took about a year. Before we could run fast, we first had to slow down.

Let one team manage migration to Visual Studio Team Services

A single team was responsible for moving projects from Team Foundation Server to Visual Studio Team Services. Team members became familiar with the steps and tools involved and handled the process more efficiently as they gained experience with multiple migration projects. This process saved time and headaches.

Digital Crimes Unit uses Microsoft data analytics stack to catch cybercriminals

Microsoft Digital Crimes Unity

The Microsoft Digital Crimes Unit consistently leverages the latest in analytics technology, relying on some of the brightest employees, some of the smartest scientists, and certainly some of the company’s best partners in law enforcement, to disrupt and dismantle devious cybercriminals. Learn how Microsoft used some of our best technology to uncover the behavior of one cybercriminal ring, and how the Digital Crimes Unit worked in partnership with Microsoft IT and federal law enforcement, to shut down one of the nation’s most prolific cybercrime operations.
Business Problem
It’s not hard to find a good deal on the Internet, but this deal looked a little too good. Kelly Reynolds, a small-time operator in Des Moines, Iowa, was offering Windows software online at prices that were a small fraction of retail. In November 2013, an agent from the US Department of Homeland Security purchased a copy of the software, including a product key to activate and use it, and sent the key to Microsoft, along with a question: Was the product key legitimate or stolen?
They say timing is everything in life. In this case, it was true. Had the question been asked just a few years earlier, Microsoft probably would have passed it on to its Product ID Center, which would have checked the product key number against a database and identified it as a real number that hadn’t yet been activated. Microsoft probably would have answered that, as far as it could tell, the key was legitimate and unused. No flags would have been raised. And that might have been the end of the investigation.
Instead, it was only the start. That’s because Microsoft had already brought together leading data scientists, forensics specialists, and former law-enforcement officers; equipped them with the company’s own advanced
data-mining and analysis tools; installed them in the Digital Crimes Unit (DCU) of the newly created Cybercrime Center located on the Redmond, Washington, campus; and tasked these individuals to fight cybercrime worldwide.
Thanks to the involvement of the DCU, the inquiry about the suspect product key in Des Moines resulted in the identification of tens of thousands of stolen product keys, the disruption of a multimillion dollar criminal operation, and the generation of leads that are now helping to identify half a dozen more criminal enterprises. (Some names and locations have been changed due to ongoing investigations.)
This is a story of collaboration—starting with a team of Microsoft analysts who worked closely with law-enforcement agents in a public-private partnership at every stage of the investigation, from their earliest suspicions to the early-morning SWAT-team raid that busted the Des Moines operation.
Another partnership was equally crucial to the success of the case, this one a partnership wholly within Microsoft itself. It was an example of a model that sees business units—in this case, the DCU—working in collaboration with Microsoft IT, with each party playing to its distinctive strengths. Microsoft IT took the lead in providing and supporting the technology infrastructure on which the data analysis was based, and the DCU led in creating the data sets and models that would yield the most effective solutions. It’s a marked evolution from the traditional way that IT has been handled in most companies, with a centralized IT organization providing infrastructure and the business solutions that run on that infrastructure.
Here, Microsoft IT gathered and integrated data from 20 databases throughout the company, established a highly automated and efficient means of updating the system, and managed it on a 24 x 7 basis for optimal accuracy and availability. But it was the data scientists in the DCU who best understood the data and invented highly innovative ways to use it.
Yet another piece to the story is the collection of technologies for mining and analyzing big data that the investigators used to uncover the scope of the global conspiracy from a single set of numbers. It’s a collection of technologies that is proving increasingly useful not only to Microsoft but also to other corporations. And not only in the fight against cybercrime, but also in making sense of big data and propelling better, data-driven decisions in fields as diverse as physical sciences and financial services.
Those technologies include some of the newest Microsoft big data mining and analysis tools, including an Analytics Platform System to manage the massive volume of data; Azure HDInsight for big-data analysis; Azure Machine Learning for predictive analysis; and Power BI and Power Maps to give the Microsoft analysts a highly visual and easy-to-use tool to gain insights from the data.
When law enforcement asked about the Des Moines product key, the Microsoft DCU investigators were ready. They checked it against the 650 million product keys and 7 billion rows of data—growing at a rate of 4 million rows a day—in its product key activation database. No one had previously attempted to activate the key—a good sign. But then the key turned up in a Microsoft database of known stolen keys. It was one of more than 300,000 keys stolen from a Microsoft-contracted facility in the Philippines and resold and distributed by another rogue operator in China. That didn’t mean that Reynolds, in Des Moines, knew the key was stolen nor that he had any other stolen keys—but it was enough to raise suspicion.
It was enough for law enforcement to search his curbside trash and discover records of another 30,000 product keys, which also turned up in the stolen-key database. Now, Microsoft and law enforcement had enough to act—but they wanted more. Analyzing a database of PCs with stolen software keys—a traditional way to look for patterns of fraud—turned up nothing suspicious about the Des Moines location. So how was an online seller in Des Moines connected to a stolen product-key ring halfway around the world? Both Microsoft and law enforcement wanted to know.
“We took datasets about product keys shipped worldwide and merged them with datasets about key activation—and we did it in ways we’d never tried to do before,” says Donal Keating, Senior Manager of Cyberforensics at the DCU. “That requires some heavy lifting to manage the data volumes, especially when you’re asking new questions and want the answers quickly. At a different moment in time, we wouldn’t have had these tools—and we wouldn’t have gotten our answers, certainly not as quickly and easily as we did. What happened in minutes might otherwise have taken days.”
When Keating and his team looked at the data in an untraditional way, the answers instantly became clear. Instead of focusing on the PCs on which product keys were activated, they decided to look only at the activations themselves—and then an IP address in Des Moines suddenly appeared as the most prominent site in the US (see map, below.). Law enforcement used the information to obtain warrants to connect the IP address to the location of the suspect activity.
More than 2,800 copies of Microsoft Office had been loaded and activated on just four computers there. “We don’t expect to see Microsoft Office loaded on a PC 700 times—let alone see it loaded 700 times onto each of four PCs,” says Keating, with some understatement. “We didn’t understand it, but it confirmed that whatever was going on in Des Moines wasn’t legitimate.”
When law enforcement entered and secured the house, they found plenty of evidence, including invoicing and purchasing records, and emails indicating the imminent delivery of another 300 stolen product keys.
The officers also found one of the PCs on which Reynolds had activated hundreds of stolen product keys. And from him, law enforcement got the answer to the mystery of why he had done so. Reynolds confessed that he had activated the keys—a bit less than 10 percent of his inventory—to test them, much as a drug dealer tests random samples of a new narcotics delivery to ensure its quality.
“That was a new insight into the behavior of the bad guys,” says Keating. “And it gave us a new pattern—the ‘test spike’ algorithm—to put into the big-data warehouse to help detect new cases.”
Already, leads and lessons from the Des Moines case have helped DCU identify other suspected stolen key operations at home and abroad. And Microsoft IT is helping the DCU make the data discoveries in this case a standard part of its cyberforensics toolkit for future investigations.
“The bad news is that cybercriminals have never been as brazen and as sophisticated as they are today. But there’s good news: our tools and technologies are better than ever, and that means we can do more to disrupt the cybercriminals. We leverage big data and technologies like Azure HD Insight, PowerBI, and PowerMaps to understand and glean behaviors on how they operate and anticipate their next moves. And we have deeper partnerships with industry, academic experts, and law enforcement, too—all of which helps us drive greater impact,” says David Finn, Executive Director & Associate General Counsel, Digital Crimes Unit.
Organizations realize a competitive edge when more employees are empowered with data. The unique approach that Microsoft has to data technology delivers this capability—whether through insights and analytics or with powerful reporting for line-of-business applications. In a world where business demands the speed to compete, Microsoft data solutions cut the time it takes to go from raw data to results for everyone.

Contact us Today!

Chat with an expert about your business’s technology needs.