It's been a while...and my MSDB has gotten fat!

New job, new house, new year...lots of changes within the last 12 months. What hasn't changed is my love of working with SQL Server. And my annoyance with certain things.

Latest: The MSDB on a server has grown immense. What the heck is going on? So I took a look, it turns out that the MSDB backup job history has never been cleaned up.  Due to the high volume of databases on this particular server combined with tlog backups every 15 minutes it bloated the MSDB database...like ridiculously, 15 GB.

Now there's a fairly easy fix for this, you could use sp_delete_backuphistory to help clean up the MSDB database. One thing though, I would recommend scheduling it to occur after hours because with an MSDB of this size it may take a while and could potentially cause performance issues.  And increasing the size of the drive would only be a temporary solution, the real problem is that the MSDB backup history is not being cleaned up.

Why isn't there a default cleanup job in SQL set for, say, every 45 days? We clean up Agent error log files, why not the MSDB history?

Schedule this cleanup to delete entries older than a certain (rolling) date and you've got yourself a permanent fix.

What do you mean my query is broken? Search isn't working? The Fulltext Phantom.

I have a love/hate relationship with 'problems'. I really like troubleshooting, because I learn so much. Sometimes though it can be really frustrating. Luckily (and I'm sure it was mostly luck that something tickled the back of my memory on this one) I was able to quickly resolve this issue.

Ticket:

I can search and find articles with up to 2 words but anything 3 words and beyond fails to return QA results. 

What? OK, first thing is to ask lots of questions, but yes, the data they wanted was there in all environments, and the query should have returned results, and it WAS returning results, but only in Test, not QA.

Here is the query (First the base one, then a modified one, neither worked in QA):

SELECT *

  FROM [MyDB].[this].[table]

  where value like 'diana arriving at the studio%'

******second query*******

USE [MyDB]

GO

DECLARE       @return_value int,

              @RowCount int

EXEC   @return_value = [this].[table]

              @SearchText = N'FORMSOF (INFLECTIONAL,"diana") AND FORMSOF (INFLECTIONAL,"arriving") AND FORMSOF (INFLECTIONAL,"at") AND FORMSOF (INFLECTIONAL,"the") AND FORMSOF (INFLECTIONAL,"studio")',

              @Start = 0,

              @End = 100,

              @RowCount = @RowCount OUTPUT

SELECT @RowCount as N'@RowCount'

SELECT 'Return Value' = @return_value

GO

So one of my first questions is: what's different. So I look through some of the basic configurations and data, and yep all of the data, views, access etc are there. Then I stopped to think. They're doing a search. What's one of the best ways to make searches like this faster? Using Fulltext! So I went to look at the settings for Fulltext on all of the servers (prod, QA, test, etc) and found a difference. Looks like they had Transform Noise Words OFF on the offending servers. So a quick turn on:

sp_configure 'show advanced options', 1;

RECONFIGURE;

GO

sp_configure 'transform noise words', 1;

RECONFIGURE;

GO

And the query is returning results as it should. Kind of a sneaky setting, and if you haven't had exposure to Fulltext you might not think of it. So here's your reminder!

Changing the service account for SSIS/SSAS, service won't start, error 1069

I've decided I'm going to post answers to issues that make me go ARGH instead of just sticking them into my Evernote; especially because it's frustrating when I Google the issue and can't find an answer then have to stumble around til I can get it fixed. The one today made me walk around with a red mark on my forehead from slapping myself when I finally got it figured out. Here's the story:

Where I work we had one domain acct for SQL. This was used to run all of the services, and also was a Domain Admin. I know, real secure. So we were migrating everything to a new, non-Domain Admin account, but still it's one account to run everything.

I'm sure you're rolling your eyes by now but hey, one step at a time!

Anyways, we had to change the accounts. So I wrote up a how-to: changing it in Configuration Manager, don't ever start services from there only in Failover Cluster Manager, don't forget to check the runas account in jobs,check credentials and linked servers etc.

So everything is going smoothly...until we get to SSAS. It wouldn't start after we changed the service account! This is the error:

analysis service could not be brought online with error 1069

So I tried it with my domain account. Still a no go, and giving me a bad credentials error.

I'm thoroughly confused because I'm not seeing any more information on what's going on, just the dang thing won't start. And, as we update more servers, I'm getting pings that SSIS is having the same issue.

Looking in error log, just says bad credentials, Looking in Cluster logs, no info there either.

And then the slap!

What do these things have in common? They're SERVICES! What do services need? LOG ON AS A SERVICE right! Yep, even though they're local administrators on the machine, that account also needs to be explicitly granted the Log on as a Service right, in Local Security Policy > Local Policies > User Rights Assignment.

I deserve my red forehead, but it's fixed :)

Remove-SqlAvailabilityDatabase -Path

Been playing a lot with PoSH lately, am now writing a script to automate the restore of a database that's in an availability group. One of the issues I kept running into was getting the correct path for the Remove-SqlAvailabilityDatabase cmdlet. The scripts I'd see online would only have a 'standard' path, like this:

Remove-SqlAvailabilityDatabase `
-Path SQLSERVER:\Sql\SecondaryComputer\InstanceName\AvailabilityGroups\MyAg\Databases\MyDb

Unfortunately that wasn't the path to my DB (and yes, before you say 'did you change the parameters'...yes....yes I did).

So anyways, to make a long story shorter, to find the path to your AG database, open up SSMS, expand AlwaysOn High Availability > Availability Groups > <your group>  and right-click on your database, choosing  Start Powershell. The prompt in the window that opens is the path to your database! So copy that and try the PoSH command Remove-SqlAvailabilityDatabase again and it'll work.

I'll probably post my automated restore solution when I'm done but wanted to get this off my chest.

How to be an amazing DBA

When you ask what makes a great DBA, what answers do you get?

-Get lots of certs!

-Oh you have to have a blog!

-Become a speaker at SQL Saturday! (or any of the other respected trade conferences)

-Mentor juniors!

....etc


While all of these are good answers, and will definitely help your intellectual credibility, none of this is worth a job if you're not customer-focused, result driven, and generally easy to work with.

A story. I tried for many years to break into the DBA world. I had quite a few jobs where I got to touch the database server occasionally: troubleshooting connectivity, doing backups, giving users permissions...simple, very junior administration. But I didn't have the experience to get a job as an actual database administrator, and where I worked doing application support there was absolutely no room (nor need) for a DBA. So I decided that getting my certifications was a good way to break into the business.

I went ahead and studied (and received) my MCDBA. I went to a small technical training firm to do this as I was able to pay for the materials and test in one lump sum, and had access to live help if I had additional questions. After receiving my certifications, the training company asked me if I wanted to work for them! They were so impressed with the fact that, when I was at the center in the evenings, I would help other students and was generally friendly and outgoing.

I worked there for three months as a trainer while trying to find that elusive DBA job. I saw an ad online for a Jr DBA position and applied. I was asked some fairly simple questions (like the difference between a user and an login) and then was invited to come in for a team interview. They asked me many more questions, most of which I knew from my training, and a few where I had no idea if they were even speaking English! Overall I thought the interview went well, I liked the team and we all seemed to mesh.

I got the job. My first REAL Database Administration job. I was so happy! I did very well, moving from Jr DBA to Sr and Team Lead within a few years of putting in a LOT of hours, studying my butt off, and working like a dog.

After I left that company, I had a conversation with my old boss. Got the news that after my interview the team didn't actually recommend that I be hired, they didn't think that I had the technical skills to be a good addition to the team. I was crushed! But luckily, my boss overrode their advice and hired me. Why? Because of one thing I said in the interview when asked about how I approach my job. I said "We're all on the same team, trying to get the same problems solved. While it may be frustrating at times, you have to put yourself in other people's shoes and see things from their point of view. In the end my job is to make their job easier."

Think about that. How much more smoothly would your company run if everyone had this attitude? You'd never hear "It's not my problem" "It can't be done" or just flat out "No". Instead you'd hear things like "Hmm, well I can't help you, but I think I know someone who might be able to...let me send them an email and I'll CC you" or "Well, it can't be done that way, what result are you trying to achieve and why? Maybe we can go about it a different way." or "Why do you need this? What's the end result you want to see?" (which brings me to something I will blog about in the future, always ask one more question, because the user may SAY they need a mauve database, when actually they might just need a report emailed to them or to be taught how to do something. Get to the root need before throwing solutions at something that may not even be an issue).

Overall what I'm saying is: to be not just a good but an amazing DBA you need both the technical chops AND the empathy to put yourself into the end-users situation. Having both of these skills will make you a very valuable employee, well-liked and respected, and never wanting for work (or friends)!

"Attempted to read or write protected memory" error

At my current workplace we have actual desktop computers, but are issued laptops in case we have to work remotely. If we want to access any of our installed programs etc we RDP into our desktop. This works out fairly well as my connection is quick enough for me to be able to do most everyday tasks when working remotely.

Then I got to thinking...

What happens if my workplace becomes a smoking hole? What if I'm unable to RDP to my desktop? I have SQL set up with my registered servers, RDCMan set up with all of the SQL instances I may need to connect to, all of my miscellaneous text files and screenshots that I haven't gotten around to archiving on the network...basically I'd be hurting.

So I decided to knuckle down and install all of my tools onto my laptop and then use a dock to connect to my monitors when I'm in the office, a fairly straight-forward process.

Until I installed SQL 2014 and tried to import my registered servers.

BAM


"Attempted to read or write protected memory" error. SQL crashes and restarts. Repeatedly. Whenever I try to log into an instance. When I try to add a registered server. Basically when I try to do anything.

What fresh hell is this??

So I did what any other good DBA would do and Googled the error. I found all sorts of links telling me to do everything from reboot my computer to replace my disk drives to give up and go home (I liked that one best!). I tried a few fixes and got a whole lot of nowhere.

Then I decided to look at what was different between my two computers. I looked at what was recently updated and saw that on my laptop that .NET had been updated to 4.5.1, Hmm I thought...I remember seeing something about this. I started digging through my emails and found one mentioning that this update 'breaks things' (yeah, real specific) and to run this to see if it helps:

netsh winsock reset

Well I ran it and BINGO, everything is running smoothly again, I'm able to log into instances using SSMS, can add my registered servers, and everything is as peachy as it can be when you're stuck in an office on a beautiful summer day.

I'm sure there are other reasons the protected memory error comes up, but this seems to be rather obscure and I wasn't able to find a fix easily so hopefully this post can help someone else!

SP_Help_Revlogin w/o the SP

I've been saving up a few ideas for blog posts for a while now. Nothing worse than having a blog and noting to say...wait, OK there are worse things but I'm being a bit megalomaniacal right now and am just focusing on the blog, not war or famine. 

So. Ran into an issue the other day. We're setting up a new proto-Prod/QUAT environment for our developers to destroy...I mean...use. It's set up nicely with a new VM and new install of SQL, the databases are copied over using SAN replication and then attached to the new server. Wonderful idea and an easy way to get a copy of Prod into a QA/DEV environment, especially when you have a TB of data to move.

Now here comes the fun part. You may have figured this out already, but the new installation doesn't come with the existing production Master (or MSDB) databases. Now we don't need the jobs per-se so we're not as worried about the MSDB, but permissions are a different story. My first thought was 'Oh I can just restore the Master DB from production!'. And then my second though was 'Dumbass, production is SQL 2008 and the new environment is SQL 2012, you can't do that.'

SIDE NOTE:
If you've ever tried to restore or rebuild the Master DB it can be a royal pain (actually, if you've never tried it, you really should. It's gotten easier to do, and much easier to practice this BEFORE you need to know it than try to muddle through it afterwards. Here's a link: http://technet.microsoft.com/en-us/library/ms190679.aspx. Also practice REBUILDING the system databases in case of disaster: http://technet.microsoft.com/en-us/library/dd207003.aspx)


The new environment is to have new logins, and there's a tangled mess of permissions on Production that need to mapped to these new logins. First thing first is to get the logins over. 

'Hey, I can just use SP_Help_Revlogin!'

Easy-peasy right? DOH! It's not on the server. And our approval process for installing anything useful in production is a nightmare worthy of Elm Street (don't get me started!). So I needed a way to use SP_Help_Revlogin without the stored procedure part of it. I needed a stand-alone way to get the logins from one machine to the other. Hence this:


*****************************************************************************
USE [master]
GO


DECLARE @login_name sysname = NULL
DECLARE @name sysname
DECLARE @type varchar (1)
DECLARE @hasaccess int
DECLARE @denylogin int
DECLARE @is_disabled int
DECLARE @PWD_varbinary varbinary (256)
DECLARE @PWD_string varchar (514)
DECLARE @SID_varbinary varbinary (85)
DECLARE @SID_string varchar (514)
DECLARE @tmpstr varchar (1024)
DECLARE @is_policy_checked varchar (3)
DECLARE @is_expiration_checked varchar (3)

DECLARE @defaultdb sysname

IF (@login_name IS NULL)
DECLARE login_curs CURSOR FOR

SELECT p.sid, p.name, p.type, p.is_disabled, p.default_database_name, l.hasaccess, l.denylogin FROM 
sys.server_principals p LEFT JOIN sys.syslogins l
ON ( l.name = p.name ) WHERE p.type IN ( 'S', 'G', 'U' ) AND p.name <> 'sa'
ELSE
DECLARE login_curs CURSOR FOR


SELECT p.sid, p.name, p.type, p.is_disabled, p.default_database_name, l.hasaccess, l.denylogin FROM 
sys.server_principals p LEFT JOIN sys.syslogins l
ON ( l.name = p.name ) WHERE p.type IN ( 'S', 'G', 'U' ) AND p.name = @login_name
OPEN login_curs

FETCH NEXT FROM login_curs INTO @SID_varbinary, @name, @type, @is_disabled, @defaultdb, @hasaccess, @denylogin
IF (@@fetch_status = -1)
BEGIN
PRINT 'No login(s) found.'
CLOSE login_curs
DEALLOCATE login_curs
RETURN
END
SET @tmpstr = '/* sp_help_revlogin script '
PRINT @tmpstr
SET @tmpstr = '** Generated ' + CONVERT (varchar, GETDATE()) + ' on ' + @@SERVERNAME + ' */'
PRINT @tmpstr
PRINT ''
WHILE (@@fetch_status <> -1)
BEGIN
IF (@@fetch_status <> -2)
BEGIN
PRINT ''
SET @tmpstr = '-- Login: ' + @name
PRINT @tmpstr
IF (@type IN ( 'G', 'U'))
BEGIN -- NT authenticated account/group

SET @tmpstr = 'CREATE LOGIN ' + QUOTENAME( @name ) + ' FROM WINDOWS WITH DEFAULT_DATABASE = [' + @defaultdb + ']'
END
ELSE BEGIN -- SQL Server authentication
-- obtain password and sid

--in-line sp_hexadecimal password

SET @PWD_varbinary = CAST( LOGINPROPERTY( @name, 'PasswordHash' ) AS varbinary (256) )


DECLARE @binvalue varbinary(256) = @PWD_varbinary
DECLARE @charvalue varchar (514)
DECLARE @i int
DECLARE @length int
DECLARE @hexstring char(16)
SELECT @charvalue = '0x'
SELECT @i = 1
SELECT @length = DATALENGTH (@binvalue)
SELECT @hexstring = '0123456789ABCDEF'
WHILE (@i <= @length)
BEGIN
DECLARE @tempint int
DECLARE @firstint int
DECLARE @secondint int
SELECT @tempint = CONVERT(int, SUBSTRING(@binvalue,@i,1))
SELECT @firstint = FLOOR(@tempint/16)
SELECT @secondint = @tempint - (@firstint*16)
SELECT @charvalue = @charvalue +
SUBSTRING(@hexstring, @firstint+1, 1) +
SUBSTRING(@hexstring, @secondint+1, 1)
SELECT @i = @i + 1
END



--in-line sp_hexadecimal SID

SELECT @PWD_string = @charvalue


DECLARE @binvalue1 varbinary(256) = @SID_varbinary
DECLARE @charvalue1 varchar (514)
DECLARE @i1 int
DECLARE @length1 int
DECLARE @hexstring1 char(16)
SELECT @charvalue1 = '0x'
SELECT @i1 = 1
SELECT @length1 = DATALENGTH (@binvalue1)
SELECT @hexstring1 = '0123456789ABCDEF'
WHILE (@i1 <= @length)
BEGIN
DECLARE @tempint1 int
DECLARE @firstint1 int
DECLARE @secondint1 int
SELECT @tempint1 = CONVERT(int, SUBSTRING(@binvalue1,@i1,1))
SELECT @firstint1 = FLOOR(@tempint1/16)
SELECT @secondint1 = @tempint1 - (@firstint1*16)
SELECT @charvalue1 = @charvalue1 +
SUBSTRING(@hexstring1, @firstint+11, 1) +
SUBSTRING(@hexstring1, @secondint+11, 1)
SELECT @i1 = @i1 + 1
END

SELECT @SID_string = @charvalue1


--EXEC sp_hexadecimal @PWD_varbinary, @PWD_string OUT
--EXEC sp_hexadecimal @SID_varbinary,@SID_string OUT

-- obtain password policy state
SELECT @is_policy_checked = CASE is_policy_checked WHEN 1 THEN 'ON' WHEN 0 THEN 'OFF' ELSE NULL END FROM sys.sql_logins WHERE name = @name
SELECT @is_expiration_checked = CASE is_expiration_checked WHEN 1 THEN 'ON' WHEN 0 THEN 'OFF' ELSE NULL END FROM sys.sql_logins WHERE name = @name

SET @tmpstr = 'CREATE LOGIN ' + QUOTENAME( @name ) + ' WITH PASSWORD = ' + @PWD_string + ' HASHED, SID = ' + @SID_string + ', DEFAULT_DATABASE = [' + @defaultdb + ']'

IF ( @is_policy_checked IS NOT NULL )
BEGIN
SET @tmpstr = @tmpstr + ', CHECK_POLICY = ' + @is_policy_checked
END
IF ( @is_expiration_checked IS NOT NULL )
BEGIN
SET @tmpstr = @tmpstr + ', CHECK_EXPIRATION = ' + @is_expiration_checked
END
END
IF (@denylogin = 1)
BEGIN -- login is denied access
SET @tmpstr = @tmpstr + '; DENY CONNECT SQL TO ' + QUOTENAME( @name )
END
ELSE IF (@hasaccess = 0)
BEGIN -- login exists but does not have access
SET @tmpstr = @tmpstr + '; REVOKE CONNECT SQL TO ' + QUOTENAME( @name )
END
IF (@is_disabled = 1)
BEGIN -- login is disabled
SET @tmpstr = @tmpstr + '; ALTER LOGIN ' + QUOTENAME( @name ) + ' DISABLE'
END
PRINT @tmpstr
END

FETCH NEXT FROM login_curs INTO @SID_varbinary, @name, @type, @is_disabled, @defaultdb, @hasaccess, @denylogin
END
CLOSE login_curs
DEALLOCATE login_curs
RETURN

GO

*****************************************************************************

The original script that is floating around out there does a call to sp_hexadecimal. Obviously this won't work if I can't install anything on the server, so I had to make it an inline function. You'll see two comments denoting where the password and SID are converted.

This has been tested on SQL 2008 and SQL 2012.