I started this blog post series with one thing in mind: help SQL Server administrators prepare for the journey towards the cloud. I may be a little bit ambitious to use hybrid SQL Server high availability and disaster recovery (HA/DR) solutions as a starting point.  But since SQL Server HA/DR solutions have been around for a while, I might as well use it as a frame of reference. Besides, its easier to learn something new by associating it with something that we already know. Why do you think I use stories in my blog posts?

Each of the blog posts somehow poses a question:

• Where are you at in your journey?
• Are you thinking differently?
• Are you setting and managing the right expectations?
• What are the chances of making this mistake?
• Do you know what parts to use?
• Do you really need everything you have right now?

Notice that none of the questions are technical. It feels unnatural – a bit awkward. Because database professionals were trained to ask technical questions. You can rephrase that statement and say something like “technology professionals were trained to ask technical questions.”

Now, read those blog posts one more time and tell me you didn’t feel uneasy. Where’s the discussion about IOPs, backup frequency, server configuration? Nothing.

But didn’t I say this is your first pit stop?

Pit stops help drivers in a car race make the necessary adjustments with the goal of winning the race. The keywords there are “necessary adjustments.” Often times, those adjustments are trivial yet impact the final outcome. Small changes here and there but add up over time. Sometimes, the adjustments can be massive. A car can’t move forward if the tire blew up. So, a new tire needs to be replaced.

Designing SQL Server HA/DR solutions using the old paradigm may have worked in the past. That won’t work anymore in the cloud. That’s why you need to change the way you think about designing solutions. Ask different questions. Look at things differently.

It was interesting to see the changes in the design of a solution that a friend of mine did as we were discussing it last week. I can’t share the technical details but all I can say is that the design made a complete turn when I asked non-technical questions. Same thing happened to another guy who contacted me thru my blog. The questions I asked made them go back to the drawing board and rethink their designs.

A simple, four-word question is all you need to make the necessary adjustments in your design: does this make sense?

We’ve moved quite a bit as a family within the last 15 years – from moving between cities, different countries to different continents. Each move was different.

The first one was pretty easy since we didn’t have a lot of stuff. All we needed was a medium-sized pickup truck. The next move was to a different country. At this point, we’ve accumulated several personal belongings. But because the move was no longer a driving distance away, we had to be very intentional about what we want to bring with us. Two adults, two toddlers and four large suitcases were all that made it to the new country. That meant discarding a lot of unnecessary baggage that we won’t be needing in our new home.

The last major move was to a different continent. It required a large crate, six large suitcases and all four of us. The suitcases went with us while the crate had to be shipped separately and would arrive 2 months after. We still had to discard a lot of unnecessary baggage. The amount of personal belongings that we had were just a by-product of a growing family.

Asking The Right Question

Your databases are also moving on a regular basis – be it to a newer version of the database engine, a larger machine, a larger storage capacity, a hypervisor or the public cloud. I bet you’ve done a fair amount of moving databases around in your career.

A standard practice for moving databases has always been to move everything. That’s why a backup-copy-restore process is pretty common for database upgrade and migration. Whether you’re using log shipping, database mirroring or Availability Groups in SQL Server to move databases, you still need to do the standard backup-copy-restore process.

A common question we ask when moving databases is this: “where are we moving?” Because it’s always been the question that everyone ask.

I think we need to change the question to this: “do we really need everything for the move?” Because asking this question forces us to think about the following:

• Data Lifecycle Management. Far more often, you’ve only considered maintaining databases. You worry about backups, think about optimizing queries and protecting from unauthorized access (recall treating your databases to a S.P.A.) You are too focused on the present that you fail to consider the past and the future. Business intelligence has started to consider the future with analytics. But what about the past? How do you archive old data? How do you deal with old data governed by compliance requirements?
• Operational Efficiency. I’m sure you’ll agree that care and feeding of current data is more important than old data. Some folks categorize this as hot versus cold data. In fact recovery objectives for hot data are different from cold data. In case something goes wrong, hot data should be recovered as fast as you possibly can. Not so much if the data was from five years ago. But when you move all of your data, you will be forced to treat them the same. Hot and cold data is on the same storage subsystem and same machine.  Same recovery objectives and service level agreements. To a point where you are sacrificing your SLAs because of cold data.
• Operational Cost. A server with 24-CPU cores, 256 GB of memory and solid state drives. Add SQL Server licensing on top of that. That ain’t cheap. All because you have to manipulate records in a very large table stored in a single filegroup. You’re running index maintenance jobs for a table where majority of the records were from five years ago. It’s like paying someone to clean your storage facility when you haven’t even touched your personal stuff for years.  And I haven’t even covered storage cost for backup, for data replicated in an Availability Group configuration, for buffer pool memory, etc.

What Does This Have to Do With the Cloud?

With on-premises deployments, we barely look at the cost of doing something. That’s because the SAN has already been purchased and the hardware lease extended. It’s the bean counters that get to see how much it costs to deploy something. And it’s not even specific to your database. The cost gets buried under the sum total for all IT costs – the SAN is shared and the physical servers run virtual machines from all business units.

Not so with the cloud. You get a line item for everything that you use. Need fast machines and storage for your database? Larger VM sizes and premium storage cost more. Want additional storage for your backups? You need to pay for additional capacity. Most cloud providers won’t charge you for ingress – getting your data into their facilities. But they will charge for egress – getting your data outside of their facilities.

Asking the question “do we really need everything for the move?” does change the tone of the conversation. As you are designing your hybrid SQL Server high availability and disaster recovery (HA/DR) solution, start thinking about whether or not you want to bring all of your data to the cloud. Because I can guarantee you, this won’t be the last time you’ll move databases around.

I’m sure you’re familiar with the famous brand and its product line of toys. You’ve probably seen one of the recent movies – The Lego® Movie and The Lego® Batman Movie. You may have played with it yourself when you were a kid. I know I did.

I got fascinated with Lego® when I was in pre-school. I like the idea of putting together different interlocking pieces and blocks to create something – anything. Just like a painter working with a blank canvas to create a wonderful piece of art. Or the musician playing the instrument that produces a relaxing sound. Creativity at its best. The only limits are your imagination.

CREATIVITY when solving problems. That's the heart of every design.
Share on X

To create something from the Lego® pieces, you need to know what the piece does and what it doesn’t. If a piece cannot achieve what you had in mind, you grab another piece, try it out and see if fits your overall “design.” The Lego® toys currently available in the market are not as much fun as they were before. In the past, you were given generic pieces of the same size. You supply the creativity. Today, you are provided a packaged set with instructions on how to build something. You don’t have to design anything. You just need to follow instructions and build what somebody else already did.

What’s In The Packaged Set?

One comment I typically hear from SQL Server professionals who are given responsibilities to deploy resources on the cloud is that “it is very confusing.” And I get it. You have to deal with storage, compute and networking resources – all of which are outside the scope of a typical SQL Server DBA’s job responsibilities.  In a large enterprise environment, somebody takes care of the storage (the storage engineers), the compute (the server administrators) and the networking (the network engineers). The SQL Server DBA’s responsibility is mainly the database server – tuning queries, applying security patches, performing backups, etc.

But not so with the cloud.

Before you can even deploy a database on an Azure virtual machine (VM) running a SQL Server instance, you need to understand what the available Azure storage types are, the available Azure VM sizes, how Azure networking works and how SQL Server depends on all of these components.

The Storage Piece

As far as I can recall, a SQL Server database still requires some form of storage – be it on-premises, an Azure SQL Database or inside an Azure VM. A database backup also requires some form of storage. When you’re designing a SQL Server solution on Microsoft Azure, you need to know what the available storage options are and how you can use them with your overall solution. Just like that Lego® piece that you need.

Start with this: Introduction to Microsoft Azure Storage. Whether you’re stretching your SQL Server Availability Group to Azure for disaster recovery or using it simply to send your backups to an offsite location, you need to know what is available and what you can use them for. Hint: You’ll most likely be working initially with general-purpose storage on Azure. Between standard and premium, that’s for you to decide. Or should I say, that’s for your performance numbers and recovery objectives to decide.

The Compute Piece

This is just a fancy name that cloud providers came up with. But it is simply the server that runs your workload. The server has CPU, memory and local disk resources. In the good-old-days of physical machines, this could be an HPE DL 380 machine or a Lenovo X-series machine. In today’s world of virtualization, this could mean anything.

Start with this: Azure Virtual Machine series. You know your workload and your applications better than anyone else. Collect performance metrics, run tests, validate assumptions. For a hybrid SQL Server high availability and disaster recovery solution, you might be tempted to go cheap on the Azure VM size. Again, that’s for your performance numbers and recovery objectives to decide.

The Networking Piece

Despite the fact that I knew networking quite a bit, I still struggled with this when I was starting out with cloud computing. That’s because everything is abstracted. With on-premises networking, you can still unplug cables to check if the physical connection is damaged (I worked part-time doing structured cabling installations to partially pay for my last year in university). This is something you cannot do on the cloud. You “assume” that layers 1 and 2 of the OSI model will not have any issues because your cloud providers will take care of it. How you wished that plugging the server to the network is all you need to do.

I’m tempted to say “start with this: Azure virtual networks.” I realized that it doesn’t work. Unlike the storage and compute pieces, the networking piece is something most SQL Server DBAs rarely get the chance to explore. But I would say this: “grab your network administrators and ask them to explain the basics of TCP/IP networking to you.” You don’t need to be an expert on configuring supernets or defining routing tables. You just need to understand the basics and how to use them when designing your solutions on the cloud.

Putting Different Pieces Together

I’m sure you’re still telling yourself, “it’s still very confusing.” But before you get too overwhelmed with the available information, why not think about each of the components as pieces in a Lego® set. The pieces have already been created for you. Your business requirement paints the entire picture of what you need to build. Identify what type of storage that you need, choose the Azure VM size for your workload and get your network administrators to assist you in designing the virtual network for your solution.

Besides, I bet you enjoyed playing with Lego® at some point in your life.

I couldn’t figure out why I got two of everything I purchased from Amazon – two copies of a book, two pieces of lavalier microphone, RAM modules for my laptop and cables for my digital audio interface. I went back to my purchase history and realized that I accidentally ordered two of everything. I only wanted two of the RAM modules and cables, not the microphone and the book. No wonder I paid more than what I expected.

Depending on your recovery objectives and service level agreements, designing a SQL Server high availability and disaster recovery (HA/DR) solution will require you to have at least two of everything – two servers for local high availability (HA) plus another one for disaster recovery (DR), two or more copies of your databases, two data centers. You get the idea.

Which is why I started the previous statement with “depending on..”

The Confusion Around Azure Availability Sets

Leveraging Microsoft Azure for DR requires understanding what it offers so you don’t end up accidentally “clicking the button twice.”

Something that is frequently brought up during HA/DR conversations involving Azure is the concept of an availability set. No, this is not SQL Server Availability Group. Availability Sets allow you to logically group one or more (emphasis mine) Azure virtual machines (VM). The Azure platform takes care of where the VMs are placed within the underlying hardware infrastructure. In order to understand this concept better, let’s introduce additional concepts by illustrating a typical server rack inside a data center.

Fault Domains

Modern data centers are filled with racks containing physical servers, just like the image above. Each physical server can run one or more VMs, depending on the available resources. Most system administrators nowadays don’t even know how a rack-mounted server look like because they are all accessed remotely. I still remember the time when we had to move servers across data centers using trolleys. Think Compaq ProLiant servers.

A rack contains a power distribution unit (PDU) that is used to distribute electrical power to all of the mounted physical servers. All of the physical servers are connected to the PDUs which, in turn, are attached to an uninterrupted power supply (UPS) to protect against power interruption. This configuration is to provide availability to the physical servers. A rack also contains a network switch to connect all of the physical servers to the network.

But the reality is, the PDU or the network switch is a single point of failure. If the PDU fails for whatever reason – blown fuse, burnt wiring due to overloading, wear and tear, etc. – all of the physical servers in the rack will be powered off together with all of the VMs running on the servers. Similarly, all of the physical servers get disconnected from the network when the network switch fails.

Given this standard layout of how physical servers are mounted on racks in data centers, the concept of a fault domain is introduced.

According to Microsoft’s documentation, a fault domain defines the group of machines that share a common PDU and network switch. Hence, a fault domain is simply the server rack.

Update Domains

Each physical server on the rack will need to be updated and maintained on a regular basis – OS updates, firmware updates, security patches, hardware replacements, etc. You’re still responsible for the VMs but Microsoft is responsible for the underlying physical server that runs your VM. Because the physical servers need to be maintained, we introduce the concept of an update domain. An update domain is a logical boundary that defines how Microsoft performs planned maintenance. It would not make sense to maintain and/or reboot several servers at the same time (I know this first hand when – almost 12 years ago –  two domain controllers in our data center were rebooted at the exact same time). Hence, an update domain is simply the physical server and how they are maintained.

Each fault domain (rack) will have several update domains (physical servers). When you deploy one or more VMs in an Availability Set, Azure will place them in different fault (rack) and update (physical servers) domains. This provides high availability by making sure that the VMs do not end up in the same rack and same physical servers, thus, meeting the 99.95% SLA.

True…until you read the fine print.

It does tell you that you need to have at least more than one VM in an Availability Set to meet the 99.95% SLA. It’s the reason I kept emphasizing one or more.

About that second serving…

You might think that you need more than one VM on Azure to meet that 99.95% SLA. But as I mentioned in the previous blog post, if you don’t need 99.95% SLA for your DR requirements, you might want to hold off creating that Availability Set. And maybe ask your boss if he can take you out for lunch. You just saved the company a few hundred (or even thousand) dollars a month.

You’re on the phone to get a technical issue resolved. You checked the service provider website and noticed that their phone support is available 24/7. You dial the direct number, expecting a quick response. Ah, the sound of a live phone operator – not a system generated voice prompt – in less than a minute. Within a few minutes, the phone operator decides to escalate the issue to a more senior engineer. “Now, that doesn’t happen often“, you think. Your call was forwarded for escalation and the next thing you know, you’ve been waiting on the phone for 45 minutes.

Does that sound familiar?

Businesses who rely on Amazon Web Services have been impacted by the recent Amazon outage. I, for one, scrambled to test accessing my video files when I found out about the incident. My online course video files are stored on Amazon S3, particularly on the US-EAST-1 region. The good thing is that my online course wasn’t affected by the outage. But I did send an email update to my course subscribers regarding the potential impacts of the Amazon outage.

Did you read the fine print?

Read thru my blog archive and you will notice a common theme when it comes to high availability and disaster recovery (HA/DR). I’m a big fan of processes. In fact, whenever I get called in to do a HA/DR project, I always start off with what I refer to as the “Alphabet Soup of HA/DR” followed by explaining the importance of “The Lion, The Switch and the Wardrobe.”

Designing & implementing HA/DR solutions should be focused more on the process & people aspect.
Share on X

The more dependencies you have, the more you need to focus on the people and process aspect of your design. Especially when you are considering moving some of your workloads to the public cloud. Here’s what I mean.

I’ve taken screenshots of the virtual machine service level agreements (SLA) – both for Amazon and Microsoft Azure – and highlighted what we technical professionals usually tend to focus on.

Amazon EC2 SLA

Microsoft Azure VM SLA

Microsoft has also introduced an improved SLA for single-instance virtual machines. I highlighted virtual machines and not the database-as-a-service platform – Amazon RDS and Azure SQL Database – because VMs are the shortest path towards your journey to the public cloud.

But what you might end up missing is the fact that meeting the 99.95% SLA, for Microsoft Azure VMs, you need to have

• at least two or more VM instances
• the VM instances should be deployed in the same Availability Set (I’ll cover the concepts behind Availability Sets in a future blog post)
• only the VM instances are protected, not the workloads inside the VMs

The same principles apply to Amazon EC2 VMs. In fact, they apply to any Infrastructure-as-a-Service (IaaS) platform, regardless of the service provider. For now. Until we see additional enhancements in virtualization platforms.

What the cloud providers say…

In an ideal world, a 99.95% uptime means a potential service downtime/unavailability of

• Daily: 43.2 seconds
• Weekly: 5 minutes and 2.4 seconds
• Monthly: 21 minutes and 54.9 seconds
• Yearly: 4 hours, 22 minutes and 58.5 seconds

Source: Uptime.is

But didn’t I just highlight the prerequisites in order to satisfy this SLA? It means

• having at least two or more VMs; translating to increased acquisition and operations cost
• having the VMs deployed in an Availability Set; requiring an understanding of what Availability Sets are and knowing how to configure them resulting in increased complexity
• having a high availability solution for the workloads running inside the VM; again, additional complexity and cost

Whoever said moving to the cloud was cost-effective was not considering the economics of managing the right expectations.

Managing The Right Expectations

Ask yourself the following questions as you are in the process of designing hybrid SQL Server HA/DR solutions:

• How critical is this database to the overall business operations?
• What’s the agreed upon RPO/RTO/SLA as defined by the business stakeholders – both for HA and DR?
• What’s the working budget to get this solution implemented?

Based on your responses to these questions, you can then proceed to identify the right solution and the right cloud provider. For example, in a hybrid HA/DR solution, you would be more concerned about local high availability RP/RTO/SLA. Would you need more than one VM on the cloud for your SQL Server databases to meet your DR requirements? Probably not. Unless you require more than 99.95% uptime – even for DR.

Additional Resources

• Why People and Processes Matter More Than Technology
• Disaster Recovery is More Than Just Technology Part 2: The Alphabet Soup
• Disaster Recovery Is More Than Just Technology Part 3: The Lion, The Switch and The Wardrobe
• The Unrealistic HA/DR Belief That Is Costing You A Lot Of Money
• Recent Amazon and Skype Outages and The Cost Of NOT Having A Proper HA/DR Strategy

As I was reading the news this week, it’s hard to ignore the announcement of General Motors laying off hundreds, if not thousands, of employees in their Michigan-based manufacturing plants. This also has severe impacts in their Ontario-based manufacturing plant in Canada where some of their production work will be moved to Mexico. Some 600+ jobs will be cut. This is due to increase in manufacturing costs and reduction in sales.

In contrast, Tesla has experienced tremendous growth in the previous years with increase in sales and demand. Just out of curiosity, I compared the stock prices of other car companies in relation to Tesla. Note that I don’t have any plans to buy a new car nor invest in any of these companies.

It’s interesting to see how a relatively new company is beating the pioneers and the giants in the automotive industry. Tesla defined a new category in an existing industry – electric cars. While the automotive giants dismissed electric cars as a potential market disruption, Tesla capitalized on it.

The Market Is Constantly Shifting – FAST

We’ve seen this happen in other industries

• Mobile phones beat land lines
• NetFlix beats the DVD rentals and cable TV business
• Amazon beats the bookstore business
• iPhone beats the BlackBerry

Only those who are smart and wise enough to see the market changes – and act accordingly – will survive. And whether we like it or not, that’s just how things are.

Running Data Centers or Innovating the Business?

You might be wondering, “what does this have to do with designing and implementing hybrid SQL Server HA/DR solutions with Microsoft Azure?” A lot, actually.

When I used to work as a data center engineer, we advised our customers on the appropriate technical solution for their business processes. I have to admit that, in most cases, our recommendations were based on our existing skills and technical expertise. As a Windows/SQL Server engineer, I make recommendations based on what I know about Windows and SQL Server. If the customer asks for a DR solution, guess what I would recommend?

But the one thing I’ve learned from working as a data center engineer is that the customer should not be focused on the technical aspects of a business process. We will make the recommendation, get it approved and take care of the rest. The customers can continue on becoming better at what they do – innovating the business and making their customers happy. In fact, I’ve created a tagline for what we do as a technology service provider: helping customers become better at what they do best. Besides, we were “the cloud.”

As a SQL Server DBA or consultant, a HA/DR solution that you can think of might consist of running a SQL Server failover clustered instance for local high availability and possibly a log shipping configuration (or an Availability Group replica, depending on your budget and RPO/RTO/SLA) for disaster recovery. And, that’s OK. Besides, that’s what we’re comfortable with. It’s our key area of expertise, our bread and butter.

As a CIO or a CTO, you might be thinking about the investments that you’ve already made in hardware and data center space so you might as well make the most out of it. Again, that’s OK. It’s not practical to throw away investments that have already been made.

But the one question that we need to constantly ask ourselves in the IT industry is this: Are we really in the business of running databases?

The Cloud As A Way To Innovate and Improve The Business

Whether we like it or not, the goal of every business is to be profitable. Otherwise, it won’t survive. It’s the reason why General Motors is moving some of its production to Mexico – to lower operations costs. It’s the same reason  why large companies have outsourced some of their operations to China or India. It’s reality. We need to accept the fact and get on with it.

But that doesn’t mean we’ll just wait for our turn to get that dreaded pink slip and be unemployed while our jobs get outsourced to somewhere cheaper. That’s the wrong way to approach it. Take NetFlix as an example.

NetFlix was the foster child of the public cloud, running all of their customer-facing systems on Amazon Web Services. One might think that their move to the public cloud would mean a reduction in their IT operations. On the contrary, they still run a big IT operation. Their IT staff – database administrators, server administrators, network engineers, developers, etc. – have truly embraced the concept of DevOps (a term that a lot of the hard-core IT folks refer to as a marketing hype.) But the IT roles have evolved over the years. Their database administrators have started doing analytics (they still do traditional OLTP-based DBA work.) They’ve created roles that we’re probably not even aware of. Who knows what roles they’ll create within the next five years or so.

NetFlix understood that they are not in the business of running data centers. They are in the business of providing customers with the experience of Internet TV. They are simply leveraging technology to constantly innovate and improve the business, regardless of whether that technology leverages an on-premises solution or the public cloud.

What Is Edwin Up To?

My goal has always been to help IT professionals and organizations grow and develop their full potential. But growth starts in the mind. I want you to start thinking about the business aspect of what you do as a technology professional. I’m sure you enjoy fixing corrupted data pages, reading SQL Server memory dumps and analyzing execution plans. But understand that none of that matters if it does not contribute to innovating and improving the business. Growth may look like shifting focus from purely technical to some business. Calculating the cost of that Azure blob storage for backup archival – that’s important to the business. Because you are thinking of ways to not just meet your database RPO/RTO/SLA but also to lower operations cost. You need to learn how to ask questions that matter to the business: where is the business heading? What are their goals and how can technology help meet those goals? And when you start doing that, you develop a mindset that enables you to properly design and implement a hybrid HA/DR solution – be it for SQL Server or other platforms.

Besides, I’m sure your organization is not in the business of running data centers anyway.

I’m sure you’ve seen a lot of status updates on your social media feeds this past Valentine’s Day. Whether you’re searching for a new partner or in an on-going relationship, I’m sure you would agree that the dating phase often sets the tone for your relationship. Now, I don’t claim to be an expert on relationships but bear with me for a moment here.

I just finished a whole-day workshop on designing and implementing a hybrid SQL Server high availability and disaster recovery (HA/DR) solution with Microsoft Azure for SQLSaturday Melbourne. Whenever I do these workshops, I always ask the attendees where they are at in their journey to the cloud – be it with Amazon, Microsoft or even Google. Here are the most common response that I get:

• currently evaluating
• deployed dev-test environments
• new projects/applications
• disaster recovery
• some mission-critical production workload

A small fraction of the attendees in my workshops have deployed mission-critical production workloads on the cloud. Now, I would assume it’s because the workshop is very specific to hybrid SQL Server HA/DR solutions with Microsoft Azure that it only attracts a specific segment of the IT professionals population.

One of the things that I highlight in the workshop is understanding what I call the “dating phase” of their journey to the cloud. What I mean by this is that their journey to the cloud is in the early stages of the relationship. And whether we like it or not, majority of the customers will be in the “dating phase” with the cloud within the next five (5) years. Why do I say that? The fact that Azure (formerly known as Windows Azure before it was renamed to Microsoft Azure) is almost 10 years old now (it was announced in 2008 and made publicly available in 2010) and customers still have not fully adopted it makes the case.

Early Stages in a Relationship

We go thru different stages in a relationship. I bet you can relate to this in one way or another. As you go thru your journey to the cloud, keep the following in mind:

1. Attraction. You met somebody at work, in school, at church, at a social event. You got attracted. The beautiful smile (it’s what got me attracted to my wife). Those lovely eyes. Maybe the dress. Or the way he talked. Similarly, your CTO/CIO/IT manager might have seen an advertisement on a magazine as he’s on his way to a meeting. Or maybe the keynote speaker at a conference mentioned it briefly. There’s something about the cloud that could reduce your company’s IT operational costs. Or maybe lower IT acquisition costs. He or she got attracted to the idea. It’s time to explore.
2. Introduction. Depending on your personality type, you can walk up to the person and introduce yourself. Or ask somebody else to get introduced. Taking your phone out, you look up his or her social media profile – works in finance, loves the outdoors, volunteers at the local children’s hospital, is health conscious. Looks like an ideal “prospect.” Now, you have a bit of an idea about the person even without talking to them. It’s no different when exploring the different cloud providers. You read the documentation (assuming it’s not convoluted and complex), you watch a webinar, you attend a workshop or a technical presentation. Sure, you “know” what the cloud is capable of and what it has to offer. You “know” the difference between an A-series and a DS-series VM on Microsoft Azure. You “have an idea” on how you can store your SQL Server database files or backups on Azure blob storage and potentially save money on archived backups. “Currently evaluating” might still be in this stage.
3. Dating. You finally had the courage to ask the person out. You set the date, the time and the place. It could just be a coffee date or at a fancy restaurant. At this stage, you start making small commitments. That schedule on your calendar is a commitment. The choice to go outside of your normal routine to spend some time with this person is a commitment. And with these commitments come expectations – even if it’s only the first date. You expect that the person will show up on the agreed upon time and place. You expect to have a face-to-face conversation. At the end of the date, you can expect whether or not you’ll get a chance to have a second or a third or a fourth date.This is where most customers are at in their journey to the cloud. They start deploying dev-test environments. While not mission-critical, it’s an opportunity to “test the waters.” The fact that they are running some type of workload on the cloud is already a type of commitment.This is also where “getting to know each other” becomes real. You notice some mannerisms that slightly annoy you. You realize that their social media profiles do not really reflect who they really are. Some people under-promise yet over-deliver (that person you thought was so boring made you laugh the whole time you were together). Some were just really good at creating an “ideal social media profile” you assume that they are just really good at marketing.

   And about the cloud? You thought moving an Azure VM from one region to another (or from classic to resource manager deployment model) was just a matter of a few mouse clicks on the portal. Or the set of deployment scripts that you wrote a few weeks ago would still work today (don’t get me started with this.) But that new project costs a lot cheaper on the cloud than it will be on-premises.

   How you treat the dating phase sets the tone for the success or failure of the relationship. Maybe it’s just a miscommunication. Or a misunderstanding. Maybe you should spend more time together to get to know each other better. Just because you didn’t get that cost savings or that IOPS requirements you have for your new database project doesn’t mean the cloud cannot deliver. Maybe you should explore other design alternatives. Like maybe combining scaling up the Azure VM with adding a few premium storage accounts. Be creative. You’ll be surprised at the outcome if you just put in more effort.

   But sometimes…as IT professionals, we act like we’re parents of a young adult who is considering a serious relationship with another fine human being. We’re afraid. We hold back to the past, thinking that we can stop time. We think that this new relationship will rob us of the opportunity to be parents. We don’t realize that it actually creates an opportunity for us to redefine our role. We’re not just parents anymore, we can be advisers and mentors.

   We look at the cloud as if it’s going to rob us of our jobs. Sure, we will no longer do the boring, repetitive tasks of patching servers, taking backups, managing user accounts – unless you find these tasks enjoyable. We ignore the endless opportunities that the cloud provides – becoming a data strategist for moving workloads to the cloud, a solutions architect for designing hybrid SQL Server HA/DR solutions, analysts for identifying data patterns that can improve business operations, even a BI professional. The possibilities are infinite, as my former employer would claim.

4. Commitment. At this stage, you are now serious with the relationship. After a few dates, you decide to  go steady. Boyfriend/girlfriend/partner. Call it whatever you want, all I know is that you are serious with the relationship at this stage. You make commitments. You go out every weekend or so. You spend the holidays with family. Maybe you start planning for the future, who knows.

One of my customers considered getting serious with Microsoft Azure. They were required by their stakeholders to have a DR strategy for their mission-critical applications. They asked me to create an action plan for them to use Microsoft Azure as their DR strategy. They currently have a SQL Server failover clustered instance for their database high availability solution. They have database backups. That was their DR strategy.

As always, I ask for their recovery objectives (RPO/RTO) and service level agreements (SLA) to understand whether or not the cloud would meet their needs without spending too much. It seems like a perfect fit. The idea was to create an automation framework that uses the Azure PowerShell modules to (1) copy their database backups to an Azure blob storage, (2) create a SQL Server VM from the available Azure VM templates that matches their on-premises version and (3) restore the database backups after the SQL Server VM was created. The database was running on SQL Server 2014 and I could have used the Backup to URL feature to store their database backups directly to Azure blob storage. But I’m not a fan of NOT having local copies of database backups. Besides, you would be paying for egress/outbound data. That means additional cost just to download your backups from Azure blob storage in case you need to recover a corrupted data page.

The project achieved two main goals: it gave them the opportunity to cost-effectively (1) test their database backups regularly and (2) create a DR strategy for their mission-critical applications. The customer has gone thru the different stages in their journey to the cloud until they’re ready to make the commitment.

Question: Where are you at in your journey to the cloud? Or are you even considering moving some of your workloads to the cloud? Share your story in the Comments section below.

Growing up, my mom required us to help with household chores. And, I’m so glad she did. I remember doing my own laundry when I was 9 years old – with no washing machine nor dryer. Being in a private school didn’t help either – I had to get my school uniform ready before the school week started. So, my weekend mornings were usually spent doing my (just mine) laundry. At the back of my mind, I knew there was an “unlimited” supply of water and laundry detergent so I used up as much as I can. I hung my clothes on a clothesline at our backyard to dry. But I didn’t understand whenever my mom would say “don’t waste water” or “don’t waste laundry detergent.” How could I be wasting when I was using them? I also didn’t understand whenever my mom would say “collect your laundry at the end of the day.” Why do it at the end of the day when I can do it the next day?

But when one of my socks went missing because I forgot to collect them the night before, the lesson stuck. And when – 10 years later – I saw how much I was paying for my water bill every month, my mom’s reminder came echoing thru one of the tellers at my bank. Because my actions now COST me something.

When Somebody Else Is Paying

As a SQL Server professional – whether you’re a database developer, a database administrator or a business analyst – our main focus is to get our job done. You write code to implement a functionality, you take backups to recover the database, and you run SELECT * FROM statements to perform analysis. Notice how all of these tasks focus on our jobs, our roles and our responsibilities. We complain when we don’t get that new version of SQL Server to test out a new feature or that shiny, new server from HPE. And we make all calls to the database impersonate the sa login. Like me, you probably reason that you are being a responsible professional who maximizes the available resources. We rarely pay attention to the cost associated with what we are doing. And this is the typical mindset that we IT professional have when we started our careers.

Becoming A Better SQL Server Professional With Cloud Computing

Cloud computing changes that. With organizations continuously looking at different ways to drive down cost, IT needs to be aware of how it contributes to the bottom line. Think about it. Whenever you use a resource on the cloud, you get visibility on how much that resource is costing you. Let’s see how using cloud computing can help make us better SQL Server professionals by evaluating it against the S.P.A. I’ll use Microsoft Azure as a reference but this applies to any cloud provider.

• Security. How many database developers think about firewall rules when they access the database from the application? More so, how many database developers apply the concept of least privilege when writing code? When you create an Azure SQL Database, you would be asked to provide the IP address that you want to allow access to the database. You will also be asked to create different database roles. This approach already forces you to think about security even before you create your tables or connect your application. The same thing applies to a SQL Server instance in an Azure virtual machine. You will be asked whether you want connectivity from within the Azure virtual network or thru the public Internet (which is certainly not a good idea.) Not thinking about security can cost you in terms of data exposure and potential security breach.
• Performance. As I mentioned in a previous blog post, performance seems to be the top priority of every database professional. And this can be easily addressed on the cloud compared to on-premise deployments. Want a faster Azure SQL Database? Upgrade an S0 instance to a P1. That’s an easy task with literally no downtime. Same thing when provisioning an Azure virtual machine with a SQL Server instance. Do you want an Enterprise Edition to go with that? Or maybe local SSDs? But wait until you see the bill at the end of the month. As a database administrator, this forces us to rethink the way we do database deployments. Developers will start evaluating the use of appropriate data types for the table columns because you need to pay monthly for the storage cost. Outbound data transfer for BI purposes will have to be re-evaluated to minimize bandwidth usage. If you have an MSDN subscription, try using your Microsoft Azure credits just for a month. With a cap on your monthly budget, you start to think differently in terms of maximizing your available resources while still getting your job done efficiently.
• Availability. You say you need all of your databases in FULL recovery model? Or all of them in an Availability Group? FULL recovery model will require you to take regular log backups which will require storage space. Your Availability Groups will require Enterprise Edition licenses (as of SQL Server 2014) and at least two SQL Server machines. We don’t see the cost associated with an on-premise deployment. The cloud will tell you all about it at the next billing cycle. This forces us to think about focusing on recovery objectives and service level agreements. Don’t get me wrong. As somebody who focuses on HA/DR, I would love all of my databases to be highly available. But as I tell my customers, we need to balance the HA/DR requirement with the cost associated with it. The cloud forces us to do that.

It’s a natural tendency for us to resist change, especially if the change pose a risk to ourselves. IT professionals hated the idea of cloud computing for fear of losing their jobs in the long run. But if you think about it, the cloud is actually helping us change our mindset in terms of managing an IT asset. That also prepares us in facing the challenges of our next role. Just think of the cloud as an opportunity to give yourself a promotion – from a data professional to a solutions strategist.

Do you still remember the very first time you have had your hands on a mobile phone? I sure do. Because back in those days, you need to sign up with a mobile phone service provider to have a phone and use their service. You choose from the different plans available from the service provider based on your preferences and usage history. My challenge back then was I paid more for a plan with more minutes of airtime than I actually need. And if I decide to downgrade the plan but would require more minutes, I still get charged more for overage.

If you look at the total cost of ownership (TCO) of an IT resource, the biggest bulk of it comes from capital expenditures – hardware, software licenses and human resources.  Imagine having to build a data center from scratch – you have to architect the design (human resource,) buy the hardware and software licenses and eventually install and configure. That doesn’t even include the time it takes roll out the solution into a fully functional system.

You wonder why the cloud has such an appeal to the business decision maker. While IT assets have become a necessity to improve business processes, we’re not in the business of building and supporting IT resources (not unless you work for companies like IBM, HP, Microsoft, Google, Amazon, etc.) Organizations aim to lower down the cost of acquiring IT assets while maximizing their productivity benefits. Here are five (5) reasons why Microsoft Azure can be a cost effective data center solution for small and medium-sized businesses.

1. You dictate when you get your IT resources. Tired of waiting for the whole procurement processes? Even small and medium-sized businesses are affected by this. Resellers will tell you when your order will be available and when it will be delivered. Remember the hard drive shortage of 2012?  Even the big players like EMC and Hitachi could not fulfill the requirements due to the shortage. With Microsoft Azure, you can get a server in less than 20 minutes, depending on the configuration you would like. No more waiting for days to provision a server for your online store.
2. You decide how your bill would look like. Compare the cloud with the traditional way of building data centers or acquiring IT assets, your costs will be very different. Instead of having to pay upfront for the resources, you take advantage of the pay-as-you-use model. Want a server with 12 CPU cores, 32 GB of memory and fast SSD storage running SQL Server 2014 Standard Edition? That’s all that you have to pay for. Microsoft will take care of the underlying networking and maintenance.
3. You determine if you need to increase your IT assets. Launching a product but not sure how much traffic your marketing campaign will generate? In the past, you are stuck with over-provisioning hardware and software resources in the hopes that your investments will allow for growth. If you under-provision your resources, you run the risk of losing potential customers. Microsoft Azure’s scalable architecture allows for the flexibility to scale up or down your IT resources depending on the workload demands. And, depending on the service that you choose – be it the Infrastructure-as-a-Service (IaaS) or Platform-as-a-Service (PaaS) – you can even do so without the downtime. So, if your online store is experiencing a massive amount of requests, scaling up can be as easy as clocking thru the Microsoft Azure portal or even automated using Windows PowerShell or Azure Automation.
4. You define your pace of innovation. A lot of successful startups and small businesses depend on technology to sustain their growth. Companies like AirBnB, Uber and NetFlix started leveraging the existing technologies to become successful. You’re probably aware that analytics and business intelligence has been and still is a hot topic in organizations because it allows for quicker decision making based on data. But if you have to rely on internal resources to build analytical models when you don’t even have a full time IT staff, it would take some time to leverage the availability of technologies. Microsoft Azure has these capabilities that organizations can use – Azure SQL Data warehouse for building and running data warehouses, Power BI for providing interactive visualization tools, Azure SQL Database with elastic scale for dynamic workload demands and many more.
5. You depend on a reliable platform. As your organization grows, high availability and disaster recovery become a priority. I’m a bit biased because this is my field of expertise. But when you need to keep your IT assets highly available and  make sure that your customer data secure, it comes as a cost.  Microsoft Azure’s Platform-as-a-Service (PaaS) offerings are highly available, by default. I use Azure SQL Database as an example for this purpose. Any database you create in Azure SQL Database will automatically have three local copies for redundancy. If you want to have it replicated across different geographical locations for even better resiliency, you can sign up for the active geo-replication feature.

These are just some of the options available from Microsoft Azure that give you the flexibility of provisioning IT resources at your own pace depending on your goals and budget.

Not sure if Microsoft Azure is for your organization, you can sign up for a free trial to test it out.

When I first started driving in North America, one of the things that I noticed along the freeway is a self-storage facility. Having not seen one before, I asked my dad what they were for – they are space rented out on a short-term basis where you can keep your personal stuff. If people don’t have enough space to keep their stuff at home, they can either move into a larger house or rent one of these self-storage spaces.

self storage units (9) by Scott Myers

Every business that stores their data in databases will eventually face this challenge. Be it for a compliance requirement or just for archival purposes, data collected needs to be stored and managed in the long term. And especially in these age of data analytics, there’s even more justification keep on keeping data. There’s a reason the prices of storage media has dropped significantly throughout the years.

Similar to us hoarding personal stuff at home, there are those that we use on a regular basis – like your favorite sweater that you wear almost every week – and those that don’t – like that vintage fretless bass guitar that you haven’t touched for years.  We like to keep the stuff that we use on a regular basis within reach for ease of access and maybe those that we rarely use stored elsewhere. And as we accumulate more stuff, the amount of storage space we need increases.

I’m sure every (SQL Server) DBA can relate to this. The database has grown huge that some data processing tasks that used to take a few minutes are now taking ages to complete. What used to fit in a 500 GB-sized disk could no longer fit in a 1 TB-sized drive. And it still continues to grow. You now worry about data archival, backups, data processing and consistency checks. And while your organization is not in the business of hoarding large amounts of data, you do it to support business operations.

Stretch Database Feature in SQL Server 2016

The upcoming version of SQL Server introduced a new feature called Stretch Database. As per Books Online, this feature allows you to “migrate your historical data transparently and securely to the Microsoft Azure cloud.” Think of it this way: your on-premise database has some historical tables on Microsoft’s Azure SQL Database.

I see a couple of different use cases for this feature but one that stands out is the ability to “outsource your data archival process.” As a high availability and disaster recovery expert, the biggest concern that most of my customers have is meeting recovery objectives and service level agreements (SLAs.) The larger the database becomes, the longer it takes to restore it from backup or bring it online. As a side effect of that, you also need a large amount of storage space to keep up with the requirement – the database files, the backups and the copy of backups that you need to maintain for your retention policies. I’ve sat down with some of my customers to define recovery objectives and SLAs on a per-database object basis. While that may be an unusual exercise for some DBAs, the reality is that we will have to deal with managing a large database in terms of storage requirements.

Similar to how we deal with our personal stuff, we can keep hot data in our on-premise SQL Server database and move archived data on Azure SQL Database via the Stretch Database feature. It could be a table exclusively for historical data or a table that has a combination of current and historical data.

What Are Your Concerns?

Looking at a new feature always raises several concerns. Microsoft Certified Master and Data Platform MVP Brent Ozar (blog | Twitter) raised some valuable concerns about the Stretch Database feature in this blog post.

While SQL Server 2016 has not been released to manufacturing (RTM) just yet, you can already test it out to see if it meets your requirements. If you have questions and concerns about the Stretch Database feature, let me know or you can file a feedback/comment on the Microsoft Connect website for SQL Server.

In a future blog post, I’ll walk you thru the process of enabling the Stretch feature an on-premise SQL Server 2016 database and looking at the data archival process happening in the background. It will feel like somebody else is doing the data archival process for you.

Additional Resources

• Stretch Database: SQL Server 2016
• Backup and restore Stretch-enabled databases
• Enable Stretch Database for a database (I feel like the title is a bit redundant)