TRUST

Given Microsoft’s recent announcement of SQL Server 2017 general availability, you somehow get an idea of the frequency of releases for future SQL Server versions. This translates to more database upgrade and migration work that SQL Server DBAs have to do. However, keeping up with the latest and greatest version is not practical from a business standpoint. What might end up happening is that there would be more frequent installations of service packs and cumulative updates to keep a specific version of SQL Server in a supported configuration.

In my online course Windows Server Failover Clustering for the Smart SQL Server DBA, I walk thru the process of installing service packs on a multi-instance, multi-node SQL Server failover cluster. The process is basically the same if you want to perform an in-place upgrade. Majority of SQL Server DBAs are mostly focused on the actual upgrade process. And, why not? This is where the real action happens. But from my experience, the actual upgrade process is just but a small portion of the overall activity. In fact, what happens before the actual upgrade process determines whether or not the upgrade becomes successful.

Here’s another thing that I observed in my experience: service pack and cumulative update installations are treated like normal operations tasks. That’s a huge mistake, especially if you do not have a high availability solution in place. I remember the days of installing service packs on a SQL Server 2000 failover cluster. I think I spent more time “hoping and praying” for the entire installation process to succeed than doing the actual work. Otherwise, I’ll be stuck with rebuilding an entire cluster.

Whether you are installing a service pack or upgrading from a lower version to a higher version of SQL Server, this video will show you the real secret to successfully perform these tasks.

I can guarantee you this. Because no one brags about their failures.

The year was 1999. A small business owner approached a young guy – fresh out of university with no database nor computer programming background – to write a software for their retail and consumer goods business. The business owner wanted a software that can be used as a point-of-sale system as well as managing inventory. The young guy declined, explaining that he didn’t have any background in computer programming nor database design. The business owner insisted: he fully trusted the young guy to deliver.

The young guy learned as much as he can about computer programming and database design. He read books, asked questions on forums and newsgroups. Tried sample codes. A lot of sleepless nights. After 24 days of banging on the keyboard writing code, he had a working prototype. He showed the business owner how it worked. It was very intuitive, easy to use and fully functional. He just got himself one happy customer.

Except…

The database had only one table…with 234 columns. It could have had about 17 tables if the design was properly normalized. What’s worse, all records were accessed using the RBAR database programming method. One “row by agonizing row” at a time.

The year was 2008. A conference room was packed with about four hundred plus people, mostly SQL Server database administrators. The topic: database recovery techniques. The speaker was showing a dozen different ways on how to recover from a database disaster – from somebody accidentally dropping a table to fixing a corrupted data page. The members of the audience were holding their breath. They just saw a disk volume disappear from underneath SQL Server, losing one of the database files in a filegroup. But the transactions kept going. Inserting records into the partitioned table, despite having one of the data files in the missing disk volume, kept going. Everyone in the room tried to guess what’s going to happen next. How will the database with a missing data file be recovered? Anticipation was building.

Except…

The speaker didn’t have a working database backup. How on earth can you recover that missing database file if you don’t have a working backup? One of the attendees even shouted, “well, that could be my production database right there.” And he was right.

The year was 2010. A consultant was working on designing and deploying System Center Operations Manager (OpsMgr) and System Center Configuration Manager (ConfigMgr) for a customer. The goal was to operationalize the enterprise software deployment process and systems monitoring. The servers were prepared, SQL Server was installed, OpsMgr and ConfigMgr configured. The project was in the stabilization phase waiting for the go-live schedule.

Except…

The consultant received an email alert. The SQL Server machine hosting both the OpsMgr and ConfigMgr databases was taken offline. And there’s no way the server is coming back online – ever. Because the system has still not reached “production phase,” only the databases had backups. It turned out that an image of a pre-built Windows XP Professional system was accidentally deployed specifically to the SQL Server machine. The process had a step to reformat the machine and install the Windows XP image. What’s worse, the deployment process was configured by the consultant himself.

In case you’re wondering, these three events have one thing common – ME.

I was the one who wrote the point-of-sale system and designed the database with a single table. I was the speaker who was right in the middle of a database disaster talking about, ironically, recovering from database disasters – in front of about four hundred plus people. I was the one who automated the deployment of the Windows XP image that took down the SQL Server machine.

But you probably didn’t notice that I called them events. They don’t define who I am.

I used to try to forget my failures. I was raised in a culture where failure is shameful. Until I realized that every failure contributed to where I am today. When I started seeing failure as a learning event – not something that defines who I am – I embraced it. Because it’s in the failures that I learn the most.

So, don’t be afraid to fail. Fail fast. Fail forward. Learn as much as you can. Just don’t make the same mistakes over and over again.

You still won’t find this in my resume. But I do talk about my failures. In fact, talking about my failures got me this wonderful prize.

I started this blog post with a warning mainly because I’ve seen a lot of engineers and database administrators spend so much time figuring out the root cause of why a SQL Server database running on a WSFC is offline – while it is still offline. I know because I used to be like that. I want to find out the real reason why so that I can take proactive steps in preventing it from happening again.

There’s nothing wrong with finding the root cause of a problem. In fact, we should try our best to “really” find out why the problem occurred in the first place. But do you remember the main goal of why we are running SQL Server on a WSFC? It is to keep it highly available (refer to Focus on The Main Goal of Part 1 of this series). You only start doing root cause analysis (RCA) after the issue has been resolved.

Your Last Option

If the methods provided in part 1, part 2 and part 3 of this series of blog posts did not work for you and your SQL Server databases are still offline, then, analyzing the cluster debug log maybe your last option. The cluster debug log (or more commonly known as the cluster.log) provides a more detailed logging information about what’s going on in the WSFC. You will have more (and potentially overwhelming) information available to help with the troubleshooting process. Let’s start with generating the cluster debug log.

Generating the Cluster Debug Log

In the past, I’ve used cluster.exe command to generate the cluster debug logs for analysis. When Windows Server 2008 introduced PowerShell cmdlets for WSFC, I immediately ditched cluster.exe.

You can generate the cluster debug log by running the Get-ClusterLog PowerShell cmdlet. This will generate a log file for all (or specific) nodes in the WSFC. By default, it will store the cluster debug log files in the C:\Windows\Cluster\Reports folder.

Being a lazy guy that I am, here are the two parameters that I frequently use with the Get-ClusterLog PowerShell cmdlet and why I use them.

• Destination – Didn’t I say I’m lazy? I don’t want to navigate to the default path just to retrieve the generated files. I use the Destination parameter, passing the dot (.) value to store the generated log files wherever I am in the PowerShell command line. This makes it easy for me to open the log files using Notepad – I can just tab-complete the filename within the PowerShell command window. Below is how I use the Destination parameter.

PS C:\> Get-ClusterLog -Destination .
PS C:\> notepad hostname_cluster.log

• TimeSpan – Sometimes, when I get called in to resolve an availability issue for SQL Server databases running on WSFC, the first thing that I do is run this PowerShell cmdlet and pass the TimeStamp parameter to include the events leading up to the incident. For example, if I get called in at 11:00 AM and the customer tells me that the databases (SQL Server failover clustered instance or Availability Group) have been offline for roughly 30 minutes, I immediately collect – not analyze, just collect – the cluster debug log so that I don’t get overwhelmed with the amount of data that I need to analyze later on when doing the RCA. Below is how I use the TimeSpan parameter, passing 40 (minutes) as a value. I used 40 and not 30 because I assume that the customer may not have immediately reported the issue. The log files will only contain the events that happened in the last 40 minutes. This parameter makes targeting a specific time frame much easier.

PS C:\> Get-ClusterLog -Destination . -TimeSpan 40

One thing to note about the timestamps written in the log files – they are in UTC format. This is because you can have WSFC nodes in different geographical regions and time zones. Think SQL Server Availability Groups with replicas on a different data center for disaster recovery purposes. So, when you’re correlating the events in the log, be sure to use the UTC time as a reference. Alternatively, you can use the -UseLocalTime parameter to generate the logs using the servers’ local time.

How I Read The Log Files

Once the log files are generated, I open one up (the number of log files depend on the number of nodes in the WSFC) using any text editor, Notepad being the most common one because it’s available on every Windows installation.

Here’s how I read the log files to start analyzing the issue. Again, I only do this after the issue has been resolved or when all the first three options I mentioned in the earlier blog posts were not enough for me to fix the issue.

1. Hit Ctrl + End. This will move the cursor the the very end of the log file. Why would i want to do that? Because the log files are generated sequentially, with the earliest event at the top and the latest one at the bottom.
2. Hit Ctrl + F. This will open the Find window so I can start searching for specific text. Here are the two that I look for
   • <space>ERR<space><space> – note that I included a space before the text ERR and two blank spaces afterwards. I use this to locate the latest error event written in the log without having to go thru some names or keywords that have the ERR text in them, for example, an Availability Group named TERRA. Refer to the screenshot below.

   

   • <space>WARN<space><space> – same as with the ERR text. I use this to locate the latest warning event written in the log. Refer to the screenshot below.

   

3. Hit Alt + U. Since I’m at the end of the file, I want to start searching for the latest entry – be it for ERR or WARN. This will search the file upwards instead of the default downwards.
4. Hit Alt + F. Start the search.

As you go thru the cluster debug log, you need to know what the event messages mean so that you don’t waste time trying to understand everything. I still don’t know every single event message but I try to focus first on what the ERR or WARN events are telling me to make sense of what happened and why.

In the next blog post, we’ll look at reading the event messages and how to decipher what those messages mean.

SQL Server Failover Clustered Instances (FCI) and Availability Groups (AG) depend a lot on Windows Server Failover Clustering (WSFC). Understanding how the underlying WSFC platform works can help us maintain availability of our databases

As I was driving out of my driveway, I noticed something that is considered to be every car owner’s worst nightmare: the check engine light. And what’s frustrating about it is that it comes without a warning and no explanation whatsoever.

I’ve seen this light go off in the past. So, I did what I did before to get it to turn off. I opened the gas tank, cleaned the gas cap and the surrounding area and closed it real tight. That didn’t work. I was hoping to save a few hundred bucks and a trip to the car mechanic. I was unlucky. I had to take it to the mechanic.

Modern vehicles are equipped with sensors and mini-computers that track what’s wrong with your car. Your car dealer or mechanic will have devices that can read these sensors and mini-computers to identify the code that is triggered the check engine light. If you really want to know what caused that check engine light to turn on, you have to read the error codes with a scanner tool. See, even cars have error codes.

The SQL Server Error Log

The SQL Server error log contains information about the operational aspects of your databases. Depending on how you log operational events, it can contain everything from failed logins, corrupted data pages, failed (and annoyingly successful) backups, out-of-memory issues, etc. Reading the SQL Server error log can be overwhelming, especially if you don’t know what you’re looking for.

Troubleshooting availability issues with a SQL Server failover clustered instance (FCI) or Availability Groups (AGs) can feel like reading the codes that triggered your car’s check engine light. But, hey, you don’t need any special code reader to do it. You can use any text editor/reader to open the SQL Server error log even when the SQL Server instance is offline. In fact, this is one common misconception about reading the SQL Server error log: you can read it even when the SQL Server instance is offline. You just need to know where to find it.

Two Common Keywords That Identify Potential Availability Issues

I could list a ton of reasons why a SQL Server FCI or AG is offline. Which is why my first step in troubleshooting availability issues on a SQL Server FCI or AG is to look at the Cluster Dependency Report. More often than not, you can quickly identify availability issues for a SQL Server FCI using the Cluster Dependency Report. The most common availability issues for a SQL Server FCI are caused by the underlying Windows Server Failover Cluster (WSFC). Just take a look at this Failover Cluster Troubleshooting guide to see what I mean.

But what about SQL Server AGs?

Because SQL Server AGs work on the database-level, anything related to availability issues at the database-level can cause issues at the AG-level. Below are two of the common keywords that identify SQL Server AG availability issues. You can use them to filter the results in the SQL Server error log. These are simply meant to give you a sense of how you can use the SQL Server error log to start identifying availability issues.

RESOLVING

This could be caused by a number of different reasons. Here’s one example.

Error 983, Severity 14: Unable to access database ‘%.*ls’ because its replica role is RESOLVING which does not allow connections.

This could be caused by a network communication issue between the primary and the secondary.

Here’s a tip that helps me simplify this idea: Think of this as a typical client-server communication issue with the primary replica acting as the client (sending log records) to the secondary replica and vice versa.

• firewall – Windows Firewall or otherwise
• endpoint permissions – the SQL Server service account does not have access to the endpoint
• bad cable – you need to keep rats off your data center
• corrupted storage on the secondary – this renders your database offline regardless
• offline secondary WSFC node – this renders your database server offline regardless
• large number of VLFs on database that is taking time to analyze during a restart of the secondary – this is just like any other databases undergoing crash recovery
• Transparent Data Encryption enabled on the primary but not on the secondary

I could go on and on and list other reasons. But the point is to find this keyword in the SQL Server error log and think of reasons why your primary replica could not connect to the secondary and vice versa.

Now, this does not mean that your SQL Server AG is offline. If the WSFC is properly configured and you have proper quorum, your SQL Server AG will still be online with the primary replica just happily doing its thing. It’s like being able to drive your car even with the check engine light on.

FAILOVER

This could be related to the previous item RESOLVING. Now, you can be as paranoid as I am to get alerted when a SQL Server AG (or FCI) failover occurred so that you can quickly identify what caused it. But you can also be confident that your system just works as expected.

The failover process is initiated by the WSFC. But in order for the WSFC to automatically failover the SQL Server AG from the primary to the secondary replica, the following conditions should be met.

• The WSFC has quorum and can decide to automatically failover
• Maximum failure threshold has not been exceeded
• Cluster service on the secondary replica is running and healthy
• The secondary replica configured for automatic failover is fully synchronized

Noticed that the first three items in the list all pertain the WSFC. The last item applies specifically to the database. Which is why you need to quickly identify and resolve any error messages that contain the keyword RESOLVING to guarantee automatic failover.

Here’s an example error message.

Error 19406, Severity 10: The state of the local availability replica in availability group ‘%.*ls’ has changed from ‘%ls’ to ‘%ls’. 

Of course, you won’t see anything like this when a SQL Server FCI fails over because that is no different from starting a SQL Server instance. The SQL Server error log would be recycled as a side effect of the failover.

NOTE: Automatic failover is not available on SQL Server AGs when one of the replicas is running a SQL Server FCI because the SQL Server FCI automatic failover takes precedence over the SQL Server AG automatic failover.

Reading the SQL Server Errror Log

I’ve been using the undocumented sp_readerrorlog (was xp_readerrorlog) system stored procedure ever since I can remember to search for keywords inside the SQL Server error log. I’m still not good with SQL Server Management Studio, even SQL Server Enterprise Manager from the good old days of SQL Server 2000 and earlier. To use the sp_readerrorlog stored procedure to search the current SQL Server error log for the RESOLVING keyword,

--to search for the RESOLVING keyword

--in the current SQL Server error log

sp_readerrorlog 0,1, 'RESOLVING'

My friend and SQL Server senior escalation engineer Balmukund Lakhani (blog | Twitter) wrote about the different parameters that you can use when calling this system stored procedure. This makes it easy for me to search the SQL Server error log to look for these keywords.

UPDATE:  I’ve included Rob Sewell’s (blog | Twitter) one-line PowerShell script example below. This just demonstrates how powerful PowerShell is when performing tasks like this.

(Get-SQLErrorLog -Server SERVER/INSTANCE).Where{$_.Message -like '*Resolving*' -or $_.Message -like '*Failover*'}

Additional Resources

• Troubleshooting automatic failover problems in SQL Server 2012 AlwaysOn environments
• Troubleshoot SQL Server AlwaysOn
• Troubleshoot Always On Availability Groups Configuration (SQL Server)
• Failover Cluster Troubleshooting guide
• Troubleshooting AlwaysOn Issues

For one, transparent data encryption is an Enterprise Edition-only feature whereas backup encryption exists in Standard Edition (I can stop right here and move on to the next topic of discussion.)  The price tag of an Enterprise Edition license is more than enough justification to consider using backup encryption. I’ve worked with customers in the past who had to use encryption utilities like TrueCrypt and GnuPG to encrypt their backups because they do not have Enterprise Edition. When customers have Enterprise Edition, my default recommendation is to use transparent data encryption.  I wrote an article back in 2008 about how we can implement transparent data encryption to provide encryption at rest as well as for the backups we take on the databases.

So, What’s The Real Difference?

Both TDE and backup encryption provides encryption – the former encrypting the MDF/NDF and LDF files together with the backups taken, the latter just the backups. That also means that, whether you use TDE or backup encryption, your database backups will be encrypted. But with backup encryption, only the backups are secured. Try detaching and re-attaching the database files to a different SQL Server instance and you can easily do so. TDE does not allow you to re-attach database files unless the target SQL Server instance has the certificate used to encrypt the database files. Below is a sample workflow and script to test it out (I’m *stealing* the code from the script in the article.)

1. Create a database master key on the master database
   USE MASTER
GO
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'mY_P@$$w0rd'

2. Create a certificate protected by the master key
   CREATE CERTIFICATE NorthwindCert
WITH SUBJECT = 'My DEK Certificate for Northwind database'
3. Create sample databases – one that is encrypted using TDE and one that isn’t

From here on, you can perform the test. We’ll use the Northwind_TDE database as the one that has TDE enabled and the Northwind_BackupEncryption as the one that does not have TDE enabled but backups will be encrypted.

NOTE: Make sure you backup the encryption key immediately after enabling TDE on your databases or encrypting your backups. And, don’t forget to update your disaster recovery strategies.

Now, because the Northwind_TDE database has TDE enabled, you can’t just detach and re-attach it to any SQL Server instance without first restoring the certificate on the target instance. Doing so will throw this error message.

We won’t have this error message when you try to attach the Northwind_BackupEncryption database on a different SQL Server instance because it’s no different from any other user databases. Again, the only thing encrypted will be the backups, not the database. But this would give you the confidence to store your database backups on Microsoft Azure or any other cloud provider. Or, even better, when you need to transfer backups to initialize Availability Group replication, log shipping, database mirroring or replication for a remote data center. I used to rely on either WinZip or WinRar to compress and encrypt backups to initialize log shipping and database mirroring in previous versions of SQL Server. This meant submitting a separate change request just to install these utilities on the remote SQL Server instance. Now, I have both compression and encryption built into SQL Server for this purpose.

But What if I Only Have Backup Encryption?

So, you only have Standard Edition and are concerned about your databases being compromised. Sure, you can encrypt your backups, but what about someone with administrative privilege and malicious intent started detaching and copying those database files? It’s the same reason I wrote this blog post – I have the same concerns as you do.

This is where the principle of defense in-depth comes in. As a SQL Server DBA, it is our responsibility to make sure that our databases are secure. We need to provide layers upon layers of protection to minimize, if not avoid these types of security breaches. Limiting access to both the host operating system and the SQL Server instance, coupled with auditing access can prevent unauthorized access.

Practice Your Database Recovery Strategies

The beauty of this feature is that you can restore your databases to lower editions like Web or Express Editions. Although you have a database size limitation of 10 GB for Express Editions, it still is a good opportunity to test restoring databases from encrypted backups to make sure that we still meet our recovery objectives.

Additional Resources:

• SQL Server 2014 Backup Encryption
• SQL Server Transparent Data Encryption
• Implementing Transparent Data Encryption in SQL Server 2008 (still applies to later versions)

University of the Philippines Oblation by randyg

“With rights come responsibilities. If we aren’t willing to take responsibilities, how dare we claim the rights.” 

I’m rarely vocal about my political views for “fear” of being branded as a leftist (I used the word fear because leaders are not immune to the feeling: they just know how to deal with it.) When I was in the university, I was almost always associated with the leftist primarily because of what I wear and who I hang out with. Most people know me as somebody who’s passionate about my country – the Philippines – even when I was still a kid. And I consider myself as an “unofficial Philippine ambassador.”

Last Friday, I noticed several media personnel in the University of the Philippines (UP) (Manila campus.) I wasn’t aware of any issues surrounding the premiere state university in the country because I try to avoid reading the news. I was intrigued. When I got back home, the first thing that I did was check the news. It turns out that there was a freshman university student who committed suicide because she was forced to take a leave of absence for failure to pay tuition fees. This became instant news, especially with the upcoming senatorial elections in May of this year. A friend of mine posted a very intriguing question on Facebook regarding the incident: is the University of the Philippines for (a) Poor Pinoy students, (b) Academically Excellent Pinoy students or (c) Academically Excellent Poor Pinoy students? I responded. Not because I wanted to defend my alma mater. I wanted others to understand that there was a much deeper issue that had to be dealt with. It was an issue of the heart. An issue of entitlement. It’s sad to hear about the loss of a loved one. More sad to realize that people are blaming the system without first looking for answers from within. Some say the student was killed by the system. Others say she was a victim of  a repressive policy that wasn’t in favor of education as a right. A characteristic of true leadership is taking responsibilities for any actions done. Below was my response to the question. 

I try to avoid telling others that I went to University of the Philippines (Diliman campus) primarily because (a) I failed “17 courses” in my undergraduate program and (b) nobody wanted to hire me because of the former  Now, when the UP system was founded back in 1908, the ultimate goal was to provide a high quality of education to the Filipino people. Nowhere in the original goal did it mention that it was for “anybody poor.” The system has evolved into what it is today primarily because it was partly subsidized by the government. With the economic journey of the Philippines after World War II, where it used to be a first-world country in Southeast Asia, most people in the government took advantage of the UP system as a means to advance their political career. Because there was an increase in the population of those living below the poverty line, the UP STFAP was born. The UP STFAP program was conceived in the late 80s, following the People Power revolution. This was the time where there was an increasing gap between the rich and the poor. This created an ever increasing “entitlement mindset” among those who are poor-yet-deserving students to go to the UP system. The very reason why tuition in the UP system is way higher than the other state universities is because they subsidize other state universities. So, to answer your question, the UP system is for the academically excellent Filipino students, regardless of economic status. For those who feel that they deserve to go to UP, go ahead and prove that they indeed deserve to go there. I couldn’t afford a UP education because it was relatively expensive for me. I had a choice between not pursuing a university degree (cheaper,) going to other state universities (a little more costly,) going to other private institutions (most expensive) or going to UP (bordering between cheap and expensive.) I chose to go to UP. I fought to stay in UP. I sacrificed a lot to go to UP. Now, do I deserve to go to UP because I was poor? Absolutely not! But I made a choice to go there, stay there and earn my university degree for there no matter what. I knew how it felt like to go from UP STFAP’s bracket E1 to bracket A (during my time, the UP STFAP used a numeric system for the brackets.) I knew how it felt like to scour for funds to finish my degree regardless of what my STFAP’s status was. Education, I believe, should be a right. 

With rights come responsibilities.If we aren’t willing to take responsibilities,how dare we claim the rights.
Share on X

I was installing CentOS 5.2 on a VMWare Workstation image when I suddenly hit a wall with this error

No Drives Found
An error has occurred – no valid devices were found on which to create new file systems. Please check your hardware for the cause of this problem.

Now, this isn’t the first time I’m installing CentOS on a virtual machine nor on a physical hardware but it definitely is the first time to install version 5.2. Back in the old versions, everything was pretty straight-forward and that I had never encountered this error message before. I was beginning to be tempted to use an iSCSI disk for the installation with another virtualized iSCSI disk but I wouldn’t want to go down that road unless I will be configuring this virtual machine as a clustered server. Having searched thru a ton of newsgroup and blog posts on similar issues, a few of them mentioned changing the Operating System to Red Hat Enterprise Linux or Other Linux 2.4.x kernel to make it work. I did find a recommendation to change the virtual disk from SCSI (which happens to be the default setting when you configure your virtual machine) to IDE. That did the trick, although I needed to create a new virtual machine in the process which was the quickest way to do it.

So, remember – use an IDE disk in your VMWare image when installing CentOS 5.2