Cloud Data Challenges tackles common data issues you're likely facing as an app developer. Whether it's security, globalization, big data, AI, or something else, you're under pressure from management and your customers to deliver breakthrough experiences, and this e-book provides actionable answers for the hurdles in your path.
We wrote this book for developers focused on the requirements of their next cloud project and its critical data challenges. We've included the technical details specific to each scenario, and, if you're working these issues, you'll be familiar with the concepts we unpack and, hopefully, find value in the solutions we explore.
If you're like most developers, you spend about two-thirds of your time building and the rest of your time learning. Sometimes your research is proactive, based on personal interest or a desire to maintain your skills, but more likely it's out of necessity—learning what you need to tackle your next big project.
Not long ago, new projects assumed an on-premises or hosted approach, forcing you to spend considerable time on essential requirements like scalability, availability, and security. Today, you still must deliver on those essentials, but the apps you're building need to do a lot more: support millions of users, span the globe, make sense of petabytes of data, and delight users in new ways.
Cloud services not only deliver "built-in" scalability and availability, but also provide an unprecedented range of prebuilt functionality for driving innovation into your apps—ready to evaluate for your next project, without your having to worry about physical infrastructure and the related overhead. The biggest challenge: there's so much out there that it can be hard to know where to start.
As a developer, you're probably already engaged with the cloud at some level, and you may be asking yourself, "Am I getting the most out of the data and AI technologies that are available in the cloud, or am I missing the key ingredients to deliver my next breakthrough application?"
If you're not sure of the answer to this question, you're not alone—especially when it comes to data. Microsoft internal research shows that, while 75 percent of developers are building data into their apps and 64 percent have a high level of confidence working with databases, only 24 percent of developers are comfortable activating cloud data services on their own.
This paper examines six of the most common data scenarios that you're likely to face and maps them to the services you might want to consider as a starting point:
Like most developers, you have a myriad of tools and technologies at your disposal and want to use what works best for you, without compromising how you work. With Microsoft Azure, it's easy to integrate data and AI into your applications using your preferred data engine and the languages, frameworks, and tools of your choice. Read on to discover how to unlock the full potential of your data—wherever it lives, in whatever form it resides.
No matter what your app does, you need to ensure that it delivers on the "essentials." Let's face it, every user demands a flawless experience with every interaction, including fast performance, high availability, ironclad security, and uncompromised privacy. But as a developer, while you know you need to deliver these core capabilities, you probably don't want to think about them—let alone be burdened with all the details necessary to ensure their delivery.
To determine whether you can trust a cloud platform to deliver on the basics—in a way that lets you stay flexible and focus on delivering new experiences that delight users—it's worth asking yourself a few questions:
More than any other factor, customer experiences determine whether companies thrive and profit, or struggle and fade. "Outside In: The Power of Putting Customers at the Center of Your Business" by Harley Manning and Kerry Bodine (New Harvest, 2012).
With your short-term investment in choosing the right cloud platform for your business, you'll soon be reaping the long-term rewards of stability, efficiency, and, ultimately, happy customers.
Microsoft Azure services are built cloud-first—ready to deliver all the above, no matter where you are in the world. Azure runs on a worldwide network of Microsoft-managed datacenters, across 36 (and growing) regions, ensuring global coverage.
With Azure, you don't need to worry about capacity or performance. You can easily and instantly scale up when more capacity is needed, and then scale down to save money when demand subsides. You can configure Azure to let you know when capacity or performance need attention, or you can configure your Azure apps to scale up and down on their own.
Azure managed services help ensure availability through automatic patching and backups, built-in monitoring and security, and more, so you can stay focused on building great apps. Many Azure services are backed by service-level agreements (SLAs) that include uptime guarantees and downtime credit policies. To ensure business continuity and disaster recovery, you can choose from locally redundant storage (where data is replicated locally within your primary region) or geographically redundant storage, where data is replicated to a secondary region that's at least 250 miles (400km) away but within the same geography.
What's more, because Azure is available in 140 countries/regions, you can put your data where its users are. Storage of data can be restricted to a single geography, region, or country, so you always have control over how close your data and apps are to the people who use them.
Azure also provides comprehensive security and identity management tools and services, including advanced threat detection, Azure Security Center, Azure Active Directory, Azure Key Vault, and Azure Multi-Factor Authentication. And Azure offers the most comprehensive set of compliance offerings (including certifications and attestations) of any cloud service provider. (Security is such an important topic that we cover it separately later, in Scenario 6: Security and compliance.)
Whatever you're building, Azure helps ensure that your customers get enterprise-proven levels of scalability, availability, and security—everything your customers expect. And while Azure takes care of these fundamentals, you get the space to focus on the work you do best—development.
Figure 1. Microsoft Azure runs on a worldwide network of Microsoft-managed datacenters, across 36 (and growing) regions.
Based in Seattle, Washington, GeekWire is a rapidly growing technology news site with a global readership. Started by veteran tech journalists in 2011, GeekWire quickly established a reputation for fast-breaking industry news and expert analysis.
In the past, GeekWire ran on open-source, WordPress software with a MySQL database, and the site was hosted by a managed services provider dedicated to WordPress. Although that relationship worked well in the beginning, as GeekWire's popularity and site traffic increased, so did its performance concerns. "We rely heavily on people visiting us from social media and search engines, where performance is a ranking factor, so we need to deliver content very quickly," says Kevin Lisota, Web Developer at GeekWire.
Eventually, the time came for a better solution. Lisota and his team needed:
When Lisota learned that Microsoft was launching a fully managed service for MySQL databases, moving to Azure made sense. "Azure Database for MySQL was what we were looking for," he says. "I don't have to deal with managing any of the details of the database—I don't have to patch, scale, or back it up and can rely on Microsoft to take care of that part of our infrastructure."
GeekWire's move to Azure delivered several immediate benefits, including on-demand scalability and reduced administrative overhead. "The biggest benefit of Azure Database for MySQL will be to have Microsoft manage and back up that resource for us so that we can focus on other aspects of the site," Lisota says. "Plus, we will be able to scale up temporarily as traffic surges and then bring it back down when it is not needed. That's a big deal for us."
Our page load times are very low, and we're able to do it on a more powerful and scalable infrastructure that costs us 45 percent less. Kevin Lisota, Web Developer, GeekWire
Read the entire GeekWire case study.
Hear from other customers who are using Azure.
Software-as-a-service (SaaS) apps are becoming more and more popular across organizations of all sizes. This is happening for many reasons, including faster time-to-value, reduced up-front investment, and freedom from worrying about running and supporting apps.
To effectively build a SaaS app that serves hundreds or thousands of customers, delivers essential elements such as data isolation and security, and helps protect your margins, you may want to start by asking these key questions:
If these questions resonate, then you probably already have a firm grasp of what you need from a cloud database solution.
Azure SQL Database, an intelligent relational cloud database service, helps you avoid the typical tradeoffs between development efficiency, manageability, performance, and security for SaaS applications. As your app runs, SQL Database employs built-in machine learning to continuously assess your app's behavior, tune performance, and automatically improve reliability and data protection— freeing you to focus on other things. And as demand for your SaaS app grows, SQL Database scales on the fly, with virtually no app downtime.
With SQL Database, you can take advantage of:
Azure SQL Database supports multiple data models for multitenant SaaS apps, so you can choose the one that delivers the optimal balance of isolation, cloud resource costs (compute and storage), and DevOps complexity for your unique situation. This also allows for a more flexible business model that includes a trial/free tier experience, serves a breadth of moderately active customers, and offers highend, dedicated premium offerings for your most demanding customers.
Figure 2. Azure SQL Database supports multiple data models for multitenant SaaS apps.
A list of all major Azure SQL Database features (with links to more information) can be found on the SQL Database features page.
With a multitenant app, you need a way to organize and manage user accounts. More likely than not, your customers already have an identity management and authentication mechanism—and don't want to use and maintain a different one just to access your app.
You can use Azure Active Directory (Azure AD)—a multitenant, cloud-based directory and identity management service—to easily add single sign-on (SSO) to your SaaS app for greater value and differentiation in the marketplace. With Azure AD, you make it quick and simple for your customers to integrate with an existing Windows Server Active Directory, enabling them to leverage existing onpremises investments to manage access to your app.
Few SaaS apps are complete without some sort of reporting environment. But all too often, such reports are static and deliver only limited insights. Embedding a rich, interactive analytics environment into your app is another way to increase its usefulness and help your customers make quick, datadriven decisions.
Power BI Embedded is an Azure service you can use to bring data to life within your SaaS app—without the time and expense of building and maintaining your own analytics environment. You can easily embed stunning, fully interactive reports in a cost-effective and scalable way—providing yet another source of value and differentiation, and potentially an additional revenue stream if you choose to position such functionality as a premium offering. You don't need to change the design of your app to use Power BI Embedded, and your users can sign in to your application just like before.
Umbraco is a popular open-source content management system. To deliver new customer value, Umbraco used Azure SQL Database and other Microsoft Azure services to build Umbraco-as-a-Service (UaaS), a multitenant SaaS app that lets customers self-provision new environments.
To make new customer environments available virtually immediately, Umbraco maintains a pool of pre-provisioned databases using elastic database pools. As the pre-provisioned databases are used, new elastic database pools are created and new databases are pre-provisioned. An automated process restores databases that are no longer needed to the elastic database pool. As of October 2016, Umbraco had nearly 3,000 databases across 19 elastic database pools.
Remarkably, Umbraco doesn't employ any database administrators. "We wanted to spend our time on solving our customers' problems, not managing infrastructure," says Niels Hartvig, founder of Umbraco. "We initially considered hosting the servers ourselves, but capacity planning would have been a nightmare."
By choosing Azure, Umbraco is able to provide customers with optimal performance, without the investment in IT resources that a self-hosted solution would require. "We love the developer convenience and scalability that Azure gives us, and our customers are thrilled with the features and reliability," says Morten Christensen, Technical Lead at Umbraco.
Elastic database pools are a perfect fit for our SaaS offering because we can dial capacity up and down as needed. Provisioning is easy, and with our setup, we can keep utilization at a maximum. Morten Christensen, Technical Lead, Umbraco
Read the entire Umbraco case study.
In today's global economy, apps and their users span the planet. To support rich, real-time personalization at global scale, your app needs fast access to the data that drives this personalization regardless of where the user is located.
To achieve this, from a data perspective, you need to consider how to:
In addition, to deliver the greatest value, you need to handle a variety of unstructured data at scale. Social media posts, mobile data, document text, application telemetry, and website content are just a few examples of today's data that doesn't always fit neatly into rows and columns. You also need to collect and analyze the data needed to drive real-time personalization before you can operationalize it for real-time access.
Ideally, you need a way to meet all these requirements without the hassle of deploying and managing servers, complex configurations of multiple datacenters across several regions, and complex big data, advanced analytics, and machine learning technology to drive personalization.
Azure Cosmos DB is designed to provide low latency access to rich data anywhere in the world— making real-time personalization at global scale exactly the type of scenario that Azure Cosmos DB is designed to support. What's more, with Azure Cosmos DB, you can model your data the way your app requires, using familiar APIs, tools, and frameworks.
Azure Cosmos DB provides:
The capabilities provided by Azure Cosmos DB make it well suited for many use cases beyond realtime personalization at global scale—including IoT and telematics, retail and marketing, gaming, web and mobile applications, banking, and other mission-critical, global scenarios.
By taking advantage of the native integration between Azure Cosmos DB and Azure Functions, you can quickly and easily build and deploy event-driven, serverless, personalized apps that have lowlatency access to rich data on a global scale. Because Azure Functions are event-driven, you can simply listen to a change feed from Azure Cosmos DB instead of creating your own listening logic. You can also bind an Azure Function to an Azure Cosmos DB collection using an input binding (which reads data from a container when a function executes) or using an output binding (which writes data to a container when a function completes).
Figure 3. Native integration between Azure Cosmos DB and Azure Functions makes it easy to build fast, serverless, personalized apps at global scale.
With Azure Functions, there are no limits—you set the parameters and the functions execute in parallel, with the Azure Functions service spinning up as many as times you need. The service creates new instances of functions whenever an event fires and closes them as soon as the function completes. This makes functions good for quick tasks and enables you to only pay for the time your functions are running.
Used together, Azure Cosmos DB and Azure Functions deliver a compelling set of benefits: eventdriven, serverless computing at near-infinite global scale, with low-latency access to rich data for serverless apps, app performance that enables a real-time user experience, and freedom from infrastructure through fully managed services.
You need to collect and analyze the data needed to drive real-time personalization before you can operationalize it and push it into Azure Cosmos DB for real-time access. The Azure big data and advanced analytics services you can use to do this—such as Azure HDInsight for analyzing data in Spark or Hadoop environments and Microsoft Power BI to bring that analysis to life—are covered later, in Scenario 4: Big data and advanced analytics.
Next Games, maker of the popular mobile strategy game The Walking Dead: No Man's Land, has been with Azure from the start. By 2017, No Man's Land had 16 million installations, had generated 120 GB of new data each day, and was handling 11,500 database requests per second. To support continued growth, Next Games needed more storage, increased flexibility, and the ability to maintain availability and performance for users who aren't geographically close to one another.
To meet those needs, the company adopted Azure Cosmos DB. "We want our game data to always reside in the datacenter closest to the player so that we can load the player data as fast as possible," says Kalle Hiitola, Chief Technology Officer at Next Games. "[The Azure Cosmos DB] replication feature allows us to do this, so that we always have the data close by the player, no matter which datacenter it comes from."
Currently, all player data is in Azure Cosmos DB except for the actual saved player file, which is stored in Azure Blob storage. In addition to Azure Cosmos DB and, soon, Service Fabric, Next Games is using Media Services for streaming video, Event Hubs for analytics, Notification Hubs to send push notifications to clients, Traffic Manager for load balancing, and Web Apps for the dashboard, which provides game management.
The good thing about Azure is that we can incrementally add new features Microsoft brings on to Azure into our platform and switch them behind the scenes. You just start a new service, and off you go exploring what it can do for you. Kalle Hiitola, Chief Technology Officer, Next Games
Read the entire Next Games case study.
Customers today have high expectations when it comes to speed, consistency, personalization, and more. While failing to meet those expectations might cause customers to leave, simply delivering what they already expect might not be enough to delight them. So how can you exceed their expectations?
The answer lies in your data—or, more accurately, what you do with it. By capturing more of the data within your enterprise, augmenting it with additional data from your customers and other sources, pulling it all together (both historical and fresh), and examining it in new ways, you can extract the insights you need to deliver compelling digital experiences for every type of customer, across virtually every industry and scenario.
To get the most out of your data, you need a single, scale-out infrastructure on which you can capture and analyze a variety of data at unprecedented volumes and velocity. Specifically, you need to:
With the big data and advanced analytics services in Azure, you have a tool set that enables you to do all the above. What's more, you can provision fast and scale even faster—paying for only what you use. You also benefit from granular security, automatic threat detection, market-leading SLAs, broad compliance certifications, a datacenter network with unprecedented worldwide coverage, and your choice of tools across both Microsoft and open-source services.
Figure 4. Microsoft Azure supports a single, scale-out architecture for capturing and analyzing a variety of data.
You can use the following services to store and manage all types of data at hyper-scale:
Figure 5. Big data and data warehouse.
You can use the following services to find the hidden patterns in your data and derive new insights:
Figure 6. Analytics and data science.
You can use the following services to orchestrate stream processing and information management:
Figure 7. Stream processing and information management.
PROS offers a dynamic pricing service that analyzes thousands of internal and external factors to help customers keep their pricing accurate, competitive, and profitable. PROS achieves this by running enormously complex calculations on multiple terabytes of data. When the company made its move to the cloud, compatibility with Apache Spark, speed, scalability, and high availability were all essential requirements.
PROS chose Microsoft Azure and implemented Microsoft Azure HDInsight, which meant that the company could run its entire service platform on a single architecture. "Moving from on-premises to the cloud was essential, but we also hoped we could find a cloud system that offered us a complete computing platform," says Weiping Wang, Senior Software Architect at PROS. "This would free us up from managing the Spark cluster ourselves, which would become a much bigger job as we grew."
Apache Spark for Azure HDInsight offered the compatibility with Apache Spark technology that PROS required—including parallel processing, which multiplied the volume of data that could be processed in a given time. Apache Spark for Azure HDInsight also offered functionality specific to running large computations with vast amounts of data. Jobs could be set up easily—regardless of their size and in a fraction of the time previously required—by spinning up an instance of Azure HDInsight. "Rather than having to configure hardware to keep up with demand, Azure HDInsight allows us to keep up with demand dynamically," says Wang.
Azure HDInsight also gives PROS an industry-leading SLA, with 99.9 percent availability, to help ensure business continuity and help protect against catastrophic events. Plus, changing the company's compute platform to HDInsight has helped the company convert manual functionality into automation in its product, making it easier for customers to update their pricing models as their businesses evolve.
In HDInsight, a process that formerly took several days now takes just a few minutes. Ed Gonzalez, Product Manager, PROS
Read the entire PROS case study.
People don't want to adapt to technology. They want it to adapt to them—to help them "get things done" in a way that's easy and natural, and that complements how they perceive the world around them, process information, and interact with their surroundings. Recent advances in artificial intelligence (AI)—the computer simulation of intelligent human behavior—are making this possible to an extent that, only a few years ago, seemed like science fiction. But how can you harness the power of AI within your own apps, augmenting them with human characteristics such as speech, vision, language, and knowledge to make them even more compelling and useful?
At Microsoft, AI is all about amplifying human ingenuity with intelligent technology. That's why we're infusing AI into Cortana, our personal digital assistant; building AI into our database engines for greater performance and security; and equipping organizations with the power of AI through business solutions, accelerators, and practices.
It's also why we're building an AI platform with cutting-edge cognition technologies that empower developers to make their own apps more intelligent and delight their customers. For instance, you could use the Microsoft AI platform to build an "intelligent chatbot" that uses speech, vision, language, and knowledge to connect with users through your own apps or via Facebook Messenger, Cortana, Slack, Skype, and Bing.
You can use Microsoft Cognitive Services and the Microsoft Bot Framework, two components of the Microsoft AI platform, to add cognitive and conversational capabilities to your own apps—in a way that lets you reach out to users more naturally and contextually, wherever they are.
Microsoft Cognitive Services are a collection of powerful APIs that you can use to support natural and contextual interactions in your apps, enabling them to intelligently interpret the world and engage your users. The APIs are implemented as REST calls, so you can easily take advantage of them with just a few lines of code—knowing they work with any language, framework, or platform.
Microsoft Cognitive Services include the following categories:
Try the Cognitive Services APIs for free, in minutes.
The Microsoft Bot Framework is a platform for building, connecting, testing, and deploying intelligent bots that can naturally interact with your users at scale, in a conversational way—for instance, to provide or capture information, make recommendations, or perform tasks.
With the Bot Framework, you get:
Figure 8. The Microsoft AI platform makes it easy to build intelligent apps that connect with users more contextually and naturally, wherever they are.
Nedbank, a major bank in South Africa, wants to engage with their clients through the communication channels they already use, such as WhatsApp, Facebook Messenger, Slack, and other messaging apps. As a first step toward that goal, Nedbank built a proof-of-concept virtual call center solution based on Microsoft Cognitive Services and the Microsoft Bot Framework. The virtual call center agent recognizes the context of clients' questions and answers the 10 inquiries for which it's programmed at 10 percent of the cost of a live agent.
"With the Microsoft Bot Framework, Nedbank gets the power and flexibility it needs to make virtual agents a success," says Paul Cartmel, Managing Director at Atura, the technology provider that assisted with the project. "For example, [the virtual agent] can distinguish what callers want when they ask the same questions in different contexts. It was easy for us to get up to speed, and [the Bot Framework] remains easy to work with."
The bank's long-term goal is to set up Nedgroup Investments, the asset management division of Nedbank, for aggressive, planned growth without having to add staff.
The shift to serving individual investors is a priority for us in the next 12 months. Bot technology plays a big role in that. Steven Goodrich, Head of Technology, Nedbank Wealth Division
Read the entire Nedbank case study.
No matter what you're building, you need to protect users' data and ensure compliance with any applicable regulations. You can do some of this within your app, which is why key security features that are specific to certain Azure services are covered earlier in this e-book, under the relevant scenarios for those services. However, when you rely on a cloud service provider for your infrastructure, you're also relying on that provider to help you keep your customers' data secure. It's the same with regulatory compliance; if the cloud services you're using aren't compliant, then neither is your app.
One of the best reasons to use Azure for your applications and services is to take advantage of its many built-in security tools and capabilities—additional layers of protection that you can use to achieve a defense-in-depth approach. For example, across all scenarios, you can use Azure Security Center for increased visibility and control over the security of all your Azure resources, use Azure Active Directory to help secure access to on-premises and cloud applications, use Azure Key Vault to safeguard cryptographic keys and other secrets used by your cloud apps and services, and use Azure Multi-Factor Authentication to provide even more security for your data and apps.
The Microsoft Azure infrastructure is designed from facility to applications for hosting millions of customers simultaneously, providing a trustworthy foundation upon which you can meet your needs. Security and privacy capabilities are built-in from the start, beginning with the Security Development Lifecycle (SDL), which addresses security at every development phase and ensures that Azure is continually updated to make it even more secure. Operational Security Assurance builds on SDL knowledge and processes to supply a framework that helps provide secure operations throughout the lifecycle of cloud-based services.
Depending on the cloud service model, responsibility varies for managing security at different solution layers. Azure provides a wide array of configurable security options so you can customize security to meet your unique requirements.
Security capabilities built into the Azure platform cover all major functional areas, including network security, database security, storage security, compute security, operational security, security monitoring and management, and more. The introduction to Azure security provides an overview of all these areas.
Figure 9. Shared security responsibilities for different cloud service models.
In a world where data breaches and government requests for access to online customer information happen daily, you need a cloud platform that helps you ensure regulatory compliance. Microsoft provides the most comprehensive set of compliance offerings, including certifications and attestations, of any cloud service provider.
For example, in May 2018, a European privacy law, the General Data Protection Regulation (GDPR), is scheduled to take effect. The GDPR imposes new rules on companies, government agencies, nonprofits, and other organizations that offer goods and services to people in the European Union (EU), or that collect and analyze data tied to EU residents, and it applies no matter where you are located. Microsoft is committed to GDPR compliance across its cloud services when enforcement begins May 25, 2018, and provides GDPR related assurances in its contractual commitments.
Founded in 1668, Merck KGaA, Darmstadt, Germany, has grown to become one of today's leading healthcare, life sciences, and performance materials companies in the world. When the company first replaced several disparate internal platforms with EVA, an integrated digital workplace and social collaboration tool, employees connected to it and other internal applications via a virtual private network (VPN). As the company grew, routing all traffic through its on-premises environment became less and less tenable. And access via VPN didn't permit the use of multifactor authentication, which the company needed.
The company used Azure Active Directory Premium to give employees highly secure, single sign-on access to both the company's core cloud-based SaaS services and on-premises applications from their desktop computers or mobile devices. "By using Microsoft Azure, we are able to create a secure platform that allows us to embrace digital opportunities—SaaS applications and mobility device connectivity—that we knew would make our teams more productive," says Conor O'Halloran, Head of Identity Management for Merck KGaA, Darmstadt, Germany.
With Azure services, we have streamlined our identity environment and made it more secure by giving employees access to a platform we trust in the Microsoft cloud. Marc Autenrieth, Head of Core Solutions, Merck KGaA, Darmstadt, Germany
Read the entire Merck KGaA, Darmstadt, Germany case study.
With its broad portfolio of data services, Microsoft Azure can help you build breakthrough new apps for any scenario, across your choice of stack, with innovative technology like AI built-in to wow your customers while making you more productive. The Microsoft Cloud supports more than a billion customers in more than 140 countries and regions, providing a unique platform that helps you achieve your business goals.
Azure supports a fully hybrid architecture, which helps you deliver the functionality you need regardless of where your data or compute power resides: your own datacenter, an Azure datacenter, other public cloud datacenters, or even a mobile device. A hybrid architecture also facilitates a phased approach. For instance, it helps you focus on harnessing the data within your walls before starting to tie in data from customers, suppliers, and other sources.
With Azure, you can: