• Pyramid of Diet Coke or Mountain Dew cans
• Pile of takeout containers (half-eaten because there’s no time)
• Wild, bloodshot eyes
• Caffeine-induced shakes
• Sleeping bag and/or pillow

…Back away slowly. DO NOT, under any circumstances, ask the developer if the documentation is up to date. You may lose a valuable body part before you can wedge yourself behind the file cabinet.

I love the beginning of a project. It’s full of resolutions and promises about how this time it’s going to be different. A quick run-down of statements I’ve heard (and said):

1. “We’re building time into the schedule to keep the documentation up to date”
2. “All of my time estimates will include some buffer for documentation”
3. “QA will not sign off without documentation”
4. “We’ve made accurate documentation part of the critical path”
5. “We’ll document as we go”

What ends up happening? I can sum it up in 4 words: “We’ll fix it later”. But we never do. Then we leave and the next poor soul comes in blind and realizes that the documentation is either non-existent or woefully out of date. He gets to spend the next few weeks being indignant and self-righteous because the documentation is bad. He quickly forgets the undocumented tangle of code he left behind at his last job.

So, what exactly is documentation? Is it a necessary evil, a sacrifice to the development lifecycle gods? Is it yet another bit of unproductive administration work you have to do to check all the boxes on the project plan? Is it some antiquated legacy of waterfall methods? No. It is none of these things. Documentation, when done well, is a conversation between you and the next guy who comes along and looks at your work. You might not be there to explain why you had to do that weird thing in that stored procedure. You might not be able to tell the story behind the funky flat table you wish your name wasn’t associated with. You might not be able to sit down over lunch and talk about the wacky business rules you’re trying to enforce. But, your documentation will be there. More than anything else, this is your legacy.

Am I proposing that you create exhaustive documentation explaining every decision and line of code? No. That’s totally unrealistic, and I believe that it’s what gets us into trouble. Overwhelmed by the goal of perfect documentation, we end up not writing any documentation at all. I fall into the good enough camp in my approach. Even then, it can be a struggle to keep up. In that spirit, I have a few must-haves and guidelines for what and when to document. Also, I’m sort of a function over form kind of girl. If it works, do it. If it’s pretty, well, that’s nice too. Of course, I’m a developer. If you’re a DBA, your list will probably look different. But, I recommend the same basic approach. What are my must-haves and what are my triggers for a bit of explanatory prose? Or, you know, poetry… but if you have time to write documentation in iambic pentameter, you probably need a more challenging job.

Must-Haves

Data Dictionary - If nothing else, get the basics down:

1. Entities and their purpose
2. Attributes and their definition
3. Business names for entities and attributes
4. Explanation or examples of what data can end up in a column

Source to Target Mapping – if you’re moving data around, make a map

1. Where does the data come from?
2. Where is it going?
3. What are the stops along the way?

Anything Manual - if you have to do anything out of process or manually, write it down

1. Bob from Accounting asked you to run a “quick little report” for him in 2009. Somehow, you’re still running it every Monday
2. You manually kick off that pesky job every Thursday evening instead of scheduling it
3. You had to (gasp!) manually update data in the database

Other Guidelines

In-Line Documentation – (Stored Procedures/Functions/Scripts/etc.)

1. Set up a nice little template inside any procedure/function/script that tells: Author/Date/Purpose/Major Revisions
2. If you did something unconventional inside some encapsulated code, make a note of why you went this direction
3. If some piece of code started out pretty and then grew horns and a tail and bought a pitchfork from Evil Processes R Us, explain what happened
4. If you’re working around some limitation of the application/database/etc., say how and why

In Case I Win the Lottery Documentation (aka, In Case I Get Hit by a Bus)

Imagine that you walk out the door this afternoon, and for some wonderful (not getting hit by a bus) reason, you never step foot inside your office again. You never log into the network again, and no one knows how to get in touch with you. What do only you know? Trust me, if you’ve been with an organization for any length of time, I can guarantee that you have some proprietary knowledge. This can be the least formal of all. Just make a list of things you hope someone else knows how to do when you’re sitting in Tahiti sipping fruity drinks, ogling the very cute cabana boy as you ask him to bring you yet another extra towel.

In our little SQL Server community, we talk a lot about being part of the community. (I know… totally Meta) Documentation is a great way to be a good citizen inside our world. Chances are someone you know, or someone who knows someone you know, will be coming into an organization after you leave. Helping the next guy ramp up more quickly and avoid your early stumbles is just about the best you can do to support your fellow database professional.

Technical skills allow me to execute on a plan. The good news is that if I don’t know how to do something, there is a wealth of resources out there to help me out. I can probably pilfer a bit of code from a blog, find a checklist, or even call a friend. Knowing the plan? That’s the hard part.

Brent Ozar (Blog|Twitter) wrote a brilliant post about being a consultant. It was so brilliant and apropos that I e-mailed him to ask for advice on a few things I’m dealing with. He was awesome, and thoughtful, and gave me some great ideas. He even recommended a book, The Secrets of Consulting by Gerald Weinberg. I read it, and I immediately felt better. Everyone should read it. It validated something that had been creeping around in the recesses of my brain: I didn’t have a good plan.

Why not? Well, I’d been so busy executing that I’d forgotten to take a step back and think. It happens to the best of us. I tend to be an intuitive person. I feel like something’s not right long before I can put my finger on it. It makes me crazy. It’s like the intelligent part of my brain is whispering, “Audrey… Audrey… Pay attention. This isn’t working”, while the rest of my brain is totally focused on crossing things off the to-do list. The problem? Maybe the to-do list is dead wrong.

So that’s what development managers and project managers are for, right? They put the plan together. They figure out the what, we figure out the how. Right. Right? Hogwash! Yeah, I said it. Hogwash.

Here’s what I believe. Every person involved in a project, from the college intern to the CTO needs to do a personal assessment of the project they’re on. I’d flat forgotten this personal belief of mine. In the rush to deliver, I’d jumped headfirst into a project without first grounding myself. So, after talking with Brent and reading Weinberg’s book, I assigned myself a task: Assess the Project.

Now, I’m not the first person to do this, and I’m certainly not the first person to talk about this. But I constantly have to remind myself to actually do it. It is a liberating exercise. Putting onto paper what’s worrying me about a project makes it real. If it’s real I can do something about it, or at least see it coming before it smacks me in the face.

I have a few personal rules for my assessments:
1) It is a personal document. For my benefit. I might use it as a reference for later communication, but for now, it’s just me and my stream of consciousness. I don’t worry about sounding negative or hurting feelings. If I think it’s going to be really bad, I might even write it at home and on the personal computer.

2) It is mostly a problem-defining exercise, not a problem-solving exercise.

3) No edits till I’m done. None. No tweaking, rewording, or rethinking. This one is hardest for me. I can’t help myself sometimes, and the urge to soften a harsh word or begin in-line rationalizing is tough to resist.

4) This is not a technical document. It is an emotional document. Everything from “I don’t know how to do X” to “Mr. End User refuses to cooperate” is fair game.

Anyway, here’s my process. I ask myself some questions, and answer them. Really, it’s just a set of lists.

1) What is the current state? – What’s going on in the business that prompted this project in the first place? What needs to be improved/created/maintained? Is the system too slow? Are users complaining? Are we losing customers?

2) What is the desired state? – What does everyone want the world to look like when this project is done? Is capacity higher? Turnaround faster? Errors reduced? Is there a totally new process? Is there a shiny new system?

3) What are the problems? – (Remember, we can use the politically incorrect “problem” because it’s a personal document) What’s keeping us from getting to the desired state? What issues do we keep tripping over? Who’s being difficult or unrealistic? Is the schedule reasonable? What am I awake at 4:00 AM worrying about?

4) What can I fix? – Here, I sort of break one of my rules. I try to identify what I can fix that’s broken. Key point: What, not how.

a. Right Now – What can I do right now without anything else happening first? I don’t worry about time or resources; I just list everything I could theoretically fix.
b. Right After – If I fix the things I could fix right now, what’s next?
c. And Then? – If I can theoretically get through the “Right Now” and “Right After”, what could I do?

TANGENT 1: It’s interesting to see if putting together these three lists naturally gets me to the desired state I defined in List 2. BUT… I resist the urge to force it. Be honest.

5) What is my conclusion? – This is the part where I get to rant. I just start writing about where I think this project is headed. Hopefully, the first 4 lists I’ve put together have helped me get my head on straight. If not, well, that tells me something too. Seriously, I rant. I tell it like I think it is. No one is going to read what I say except for me. Do I believe the project is going to fail? I say it. Then say why. Do I think we need to go in a different direction? I put it down. Do I think I’m failing to deliver? Why? It’s the most liberating part of this process. Feels like confession.

6) What questions do I have? – I read back over my first 5 lists, and start writing down any questions I can’t answer. Doesn’t matter how big or stupid or rude. In the past, I’ve written thing like, “Does anyone care if this project succeeds?”, and “Can we hit deadline if [Name Redacted] keeps screwing up?” I might never ask these questions out loud, but it’s therapeutic to ask myself. I might even come up with a few that need real answers that I can ask in public and look proactive and smart.

TANGENT 2: If I can’t put the 6 lists together off the top of my head, this is a giant, flaming red flag. If I can’t define where we are, or where we want to go, or what I can do to help get us there, I’ve got real problems.

So, I’ve poured my thoughts and worries and soul into answering 6 basic questions. What now? I put it away. I leave it alone to marinate for a day. Then, I open it back up again. I read it and try to see what the basic feel is. Is it optimism or despair? Was I overly negative, or did I apply false optimism to my lists? And, most important, do I see the beginnings of a plan?

Ninety-nine times out of a hundred, I see things I could be doing differently. I can begin to filter out the things I can improve versus the things I have no control over. I usually see a plan emerge. I see a way out of whatever hole I’ve dug for myself (or been thrown into). Most importantly, I have a lodestone in this assessment next time the manager asks me what I think. I’ve already thought about it and put it on paper, and I’m not fumbling around trying to describe some general feeling of “Not Rightness”.

I don’t believe that there are impossible projects, but I do believe that there are impossible plans. More than there should be, actually. I know that Development Managers get sick of hearing us whine about the plan. But, if I can say, “I have a few specific questions and ideas about the plan”, they usually sit up and listen. See, there’s another dirty little secret that my dear friend Josh Lane told me once: We all think there are these experts out there that have all the answers. Guess what? There isn’t. It’s us. We’re it.

Scary as hell? Yes. But it also means that it’s on us as developers to not just solve problems, but to help define them as well. Ask anyone in our business… it really is harder than it looks.

Lately, I feel a bit like the scarred anti-hero. I’m doing some things in a production environment that I’m not totally thrilled about, and I wish I could stop the line and do things the “right” way. I want to use SSIS to transform data. I want to encapsulate processes into neat, repeatable, parameterized modules. But, you know what? When there’s a same-day turnaround on a request, you make do. You go a little Noir on your T-SQL, know what I mean?

I want to show you two things that I’ve actually done in the past few weeks. No, given a nice, neat environment, this SQL might never have been written. Am I proud of it? Well, yes. Yes I am. At the end of the day, I got the customer what he needed. Was it pretty? No. I’m cool with that. Being the anti-hero is kind of fun every once in a while.

Fixed-Width Output

I needed to give a guy a text file in fixed-width format. I had a process from my predecessor that just wasn’t working. The file was already late. So here’s what I did. I’m using the AdventureWorks database to show an example.

SELECT
	LEFT((ISNULL(Title,'')+SPACE(50)), 8)+
	LEFT((ISNULL(FirstName,'')+SPACE(100)), 20)+
	LEFT((ISNULL(LastName,'')+SPACE(100)), 30)+
	LEFT((ISNULL(MiddleName,'')+SPACE(100)), 5)+
	LEFT((ISNULL(EmailAddress,'')+SPACE(100)), 35)+
	LEFT((ISNULL(Phone,'')+SPACE(100)), 25)
FROM AdventureWorks.Person.Contact	;

The result:

Paste it into Notepad and see how it looks:

I save the text file and send it on. Pour myself a whiskey, neat, and light up an unfiltered Lucky Strike. Okay, not really, but you know what I mean.

A quick run-down:

ISNULL: If any of the values I’m concatenating are NULL, then the entire string will come back as NULL. I wrap all of my columns in ISNULL like so:

ISNULL(Title, ‘’)

This sets the value to an empty string if the value is NULL.

SPACE: This handy little string function will pad the given number of spaces onto the result you return. I want to make sure I end up with enough padded spaces to fill out the fixed-width portion of that column. So, I pad the output:

ISNULL(Title, ‘’)+SPACE(50)

This will give me the output from the Title column, plus 50 spaces.

LEFT: Now, not every value coming out of the database is going to have the exact same number of columns. So, I use the LEFT function to trim it down to the exact length I want. LEFT will take the left-most number of characters you tell it to. If I say,

LEFT((ISNULL(Title,”)+SPACE(50)), 8 )

I’m telling it to give me characters 1-8 that are returned. Since I’ve padded my output with spaces, it’ll be the result from the column, plus as many spaces as I need to pad the output to 8.

Pretty? No. Functional? Yes. Noir SQL? Absolutely.

Remove Unwanted Characters

Next up, I have a source file I use from another department. It comes in Excel format, and includes a phone number. I’m supposed to get something that looks like this: 1112223333. Nice, neat, simple. What do I get? A hodge-podge of phone number formats. I’m looking at something like this:

CREATE TABLE PhoneNumber
(
	PhoneNumber varchar(50)
); 

INSERT INTO PhoneNumber(PhoneNumber)
VALUES
	('1112223333'), ('(111) 222-3333'), ('111-222-3333'), ('111 222 3333'); 	

SELECT PhoneNumber
FROM PhoneNumber

Okay. So I need to clean these numbers up quickly. Destination, not journey, my friends. I’m the anti-hero. I import the data into SQL Server using the Import/Export utility so I can manipulate the data. Then, I run this nifty little REPLACE statement:

SELECT PhoneNumber,
	CASE
	WHEN ISNUMERIC(PhoneNumber) = 0
		THEN REPLACE(
			REPLACE(
				REPLACE(
					REPLACE(PhoneNumber, '-', ''),			--Strip out dashes
				' ', ''),							--Strip out spaces
			')', ''),								--Strip out close parenthesis
		'(', '')									--Strip out open parenthesis
		ELSE PhoneNumber
	END as FormattedPhoneNumber
FROM dbo.PhoneNumber

Check out the results:

Sweet. It’s quick, it’s dirty, and it saved me having to wait on the source data provider to clean things up on his end. I turn the query into an UPDATE statement, and I’ve got clean data to import. Again, a run-down of the functions:

ISNUMERIC: Tells me whether the value I’m passing is qualifies as a number or not. NOTE: It recognizes hexadecimal as a number, so use carefully. I set up a CASE statement that asks if the value is numeric. If it is, that means I don’t have any characters like “(“, “)”, or “-“ in there. If not, I apply a nested REPLACE to the value.

REPLACE: Replace is awesome. I can say something like this: REPLACE(PhoneNumber, ‘-‘, ‘’). This is saying that if I find a dash, I want to replace it with an empty string. What’s really cool is that I can nest them. So, I can tell it to remove the dashes, then the spaces, then the open parenthesis, and finally the close parenthesis in one statement.

Bottom line: Sometimes things just have to get done. The difference between an anti-hero and a true antagonist is that we anti-heroes know to go back and do things the right way as soon as we get a moment to breathe. In the meantime, don’t apologize for leaving behind a few unmarked graves when you need to get the job done. We’re anti-heroes. We have the scars to prove it.

WHAT: Atlanta SQL Server Writer’s Workshop
WHO: Anyone who writes, presents, or is thinking of getting started in either
WHEN: Thursday, January 6, beginning at 6:30 PM (First Thursday of the month going forward)
WHERE: 12000 Findley Rd, Ste 150, John’s Creek, GA 30097*
WHY: To write! To socialize! To finish that abstract that’s been calling your name for the past two weeks!

* More detailed instructions: This is the John’s Creek City Hall building right off Peachtree Pkwy (Hwy 141). If you go into the front entrance of the building, hang a right and go down the hallway. We’re the last door on the left. If you come in the side entrance, we’re the first door on your right. The door is locked by default, but if you know the secret knock, we’ll let you in.

The Explanation:

Writing and Presenting is fun, satisfying, and good for your career. It’s also intimidating, nerve-wracking, and hard. If you’re just getting into the game, it’s a scary proposition. You’re not sure how to get started, what to write or present about, or how to get your brilliant ideas to the masses. If you’ve been doing it for a while… well, all of the above probably applies to you too.

Some days, you feel like this guy:

The words and ideas flow. You amuse yourself with your witty prose, and it comes naturally. Gwyneth Paltrow falls madly in love with you and you live happily ever after.

But some days, you’re this guy:

All work and no play… Need I say more?

A few weeks ago my fellow Datachix, Julie Smith (Blog|Twitter) made a comment that we should have an abstract writing party. I think that everyone went, “Huh, that actually sounds fun.” (Go ahead, insert geek joke here. I’ll wait.) Then, Aaron Nelson (Blog|Twitter) and I had lunch last week and talked about it again. The idea that Like-Minded Individuals within the SQL Server community would get together to socialize and write all at the same time sounded pretty cool.

So here we are, a week later. Logistics have been worked out, permission has been granted, and the invitation is being put forward. We’re starting a monthly Writer’s Workshop for anyone who writes or presents about SQL Server in the Atlanta area (or elsewhere if you want to travel and/or happen to be in town). We need a cooler name than Writer’s Workshop, but I’ll leave that open up for discussion. For now, SQL Server Writer’s Workshop it is. Even given the name, if you want to write or present about any technical topic, come on. We are an inclusive bunch.

Here’s the setup: Bring your laptop, ideas, half-finished blog posts, nearly-due abstracts, writer’s block, and your happy self. This being a working/social gathering, it’s BYOB (or soda/coffee/whatever – we don’t judge). We’ll probably order take-out or make a food run, but feel free to bring some grub. Settle in and work on whatever it is you’re trying to start or finish, knowing that the room has people willing to answer a shouted question, do a bit of quick copy-editing, or make suggestions about your work. That’s right, fellow writers. This is a community gathering; be prepared to listen and offer suggestions. If someone wants to get a second set of eyes, they might be yours. On the other hand, if you need a look over your shoulder, there’s someone there. Also, with a bit of advance notice, I can make sure we have a projector on site for presentation practice.

If you’re thinking of starting a blog or submitting your first presentation abstract, we especially want you to come out. I would argue that Atlanta has one of the best SQL Server communities out there, and we want to reinforce that reputation. Getting new speakers and writers up and running is a win for everyone. So seriously, join us. We promise to be nice.

My boss (and the President of Key2 Consulting) Brian Thomas (LinkedIn), has been gracious enough to allow us to use the Key2 offices in John’s Creek after-hours for this. We have open space, good wifi, whiteboards, an as-needed projector, and a fridge. It’s not fancy (we are a small company after all), but it’s got a good setup and it’s free. We have a decent-sized room with tables, chairs, and a couch for those of you who like to get comfy while writing.

If you have questions feel free to e-mail me at audrey.hammonds@key2consulting.com. I’m also on Twitter as Datachix2.

And now, the official disclaimers:

1) Please be respectful of the offices we’re using. They’re being offered up free of charge, and my boss is doing our community a solid by letting us use the space. Don’t take what’s not yours, and clean up after yourself.
2) It’s BYOB. You are a grown-up. Be responsible and know your limits. Key2 Consulting nor the people you’re surrounded by are responsible for your actions or behavior during or after the gathering.
3) You are solely responsible for what you publish or present. Do your homework.
4) That being said, if someone gives you a great idea or really helps you out, give them some credit. A name-drop is a good thing in this context.

Now, take a knife and cut into the cake. You’ve defined a set. It’s a collection of Tuples. Cool, huh? Now, stop cutting! I have to get through a basic set before we cut (SELECT) a whole slice of data out of that decadent, delicious cake. As I mentioned before, I’m using the Microsoft SQL Server 2008 MDX Step-by-Step book as my primary resource. Much credit to these guys for their excellent tome.

The best way to illustrate a SET is to build up a SELECT statement in MDX. So, that’s just what we’re going to do.

Defining a set gives you a lot of power over the way that your result is presented to you as well as what’s included in it. In most cases, you’re going to define what’s on COLUMNS and ROWS, the first two of 128 possible axes you can define. I’d love to talk to the sadistic you-know-what at Microsoft that thought it would be funny to try to make my brain fry by encouraging me to even attempt to visualize how a result set would be presented on 128 different axes. It’s okay, though, somebody probably failed to tell him that SSMS only allows you to return two axes in a result set. If you try to define a third, PAGE, for those of you keeping track at home, you’ll get an error message instead of results. Ha! Take that, Mr. Microsoft Over-Achiever!

Anyway, let’s start building us a SELECT statement in Management Studio. First, make it as basic as possible:

SELECT
FROM [Adventure Works];

You get this:

Um, okay, that’s nice. 80 million dollars. That tells me… nothing useful. But, we have a syntactically correct MDX query, so I’m not complaining.

TANGENT:

By the way, what does that ~80 million represent? Reseller Sales Amount. Why? Because it’s the default measure for the Adventure Works cube. How do we know? Open up BIDS. Open the Analysis Services Database, Adventure Works. Open the Adventure Works cube, and go to the Cube Structure tab. Right-click on the Adventure Works cube in the Measures section (top-left corner) and select Properties. There’s a defaultmember property. It says Reseller Sales Amount. There you go.

END TANGENT

But, we can do better. Let’s define a set that will give us column headers:

SELECT
{
	 ([Sales Territory].[Sales Territory Country].[Australia])
	,([Sales Territory].[Sales Territory Country].[Canada])
	,([Sales Territory].[Sales Territory Country].[Germany])
	,([Sales Territory].[Sales Territory Country].[United Kingdom])
	,([Sales Territory].[Sales Territory Country].[United States])
} ON COLUMNS
FROM [Adventure Works];

That thing up there in the SELECT clause? A SET! Note that it’s enclosed in curly brackets ({}). Yes, I know they’re called braces. I call them curly brackets. More descriptive. Also note that each thing between the commas is a Tuple. Therefore, Collection of Tuples! There is an important, nay, vital rule that is being followed here: When I explicitly name a dimension in my tuple, each of the tuples in the set references the same hierarchy. Now, I don’t have to define the SAME LEVEL of the hierarchy in all of my tuples. I can do something like this:

SELECT
{
	 ([Sales Territory].[Sales Territory Country].[Australia])
	,([Sales Territory].[Sales Territory Country].[Canada])
	,([Sales Territory].[Sales Territory Country].[Germany])
	,([Sales Territory].[Sales Territory Country].[United Kingdom])
	,([Sales Territory].[Sales Territory Country].[United States])
	,([Sales Territory].[Sales Territory Country])
} ON COLUMNS
FROM [Adventure Works];

Cool. But, I can’t reference two different hierarchies from the Sales Territory dimension in one set. Check this out:

SELECT
{
	 ([Sales Territory].[Sales Territory Country].[Australia])
	,([Sales Territory].[Sales Territory Country].[Canada])
	,([Sales Territory].[Sales Territory Country].[Germany])
	,([Sales Territory].[Sales Territory Country].[United Kingdom])
	,([Sales Territory].[Sales Territory Country].[United States])
	,([Sales Territory].[Sales Territory Region].[Northeast])
} ON COLUMNS
FROM [Adventure Works];

Ooh, error. Back to the cake analogy… This would be sort of like starting to cut into the cake, and then picking up the knife and stabbing it into another part of the cake. You wouldn’t expect a clean slice, and the same goes for the query. It just doesn’t know how to pull this data back. By the same token, I can’t reference two different hierarchies either. Really, why would you do this to your lovely chocolate ganache, anyway?

Okay, there’s more we can do with these column headers that are being returned. We can define a more detailed tuple. Maybe I want to see why people bought products in Australia. Watch this:

SELECT
{
	 ([Sales Territory].[Sales Territory Country].[Australia], [Sales Reason].[Sales Reason].[Quality])
	,([Sales Territory].[Sales Territory Country].[Australia], [Sales Reason].[Sales Reason].[Price])
	,([Sales Territory].[Sales Territory Country].[Australia], [Sales Reason].[Sales Reason].[Magazine Advertisement])
	,([Sales Territory].[Sales Territory Country].[Australia], [Sales Reason].[Sales Reason].[Review])
	,([Sales Territory].[Sales Territory Country].[Australia], [Sales Reason].[Sales Reason].[Manufacturer])
} ON COLUMNS
FROM [Adventure Works];

Sweet. Remember the rules from the Tuple episode? When a tuple is defined, every single dimension is actually represented in the query, even if you don’t explicitly name it. It defines the tuple members used according to the Other Three Rules*: Default Member, then (All) Members, then First Member. Before, the Sales Reason dimension was accounted for, but it was using the (All) Members rule because a Default Member isn’t defined. This time around, we’re telling the query exactly which members from the Sales Reason dimension to return, as well as which order to return them in. I could go on. I could define this tuple out to my heart’s content. BUT, there is one big rule to follow: The Set requires that the dimensions are given in the same order in every tuple. The following query will return an error:

SELECT
{
	 ([Sales Reason].[Sales Reason].[Quality], [Sales Territory].[Sales Territory Country].[Australia])
	,([Sales Territory].[Sales Territory Country].[Australia], [Sales Reason].[Sales Reason].[Price])
	,([Sales Territory].[Sales Territory Country].[Australia], [Sales Reason].[Sales Reason].[Magazine Advertisement])
	,([Sales Territory].[Sales Territory Country].[Australia], [Sales Reason].[Sales Reason].[Review])
	,([Sales Territory].[Sales Territory Country].[Australia], [Sales Reason].[Sales Reason].[Manufacturer])
} ON COLUMNS
FROM [Adventure Works];

Again, this query is like stabbing your knife into the cake and expecting to come out with a beautiful slice. MDX likes clean cuts. So, it wants consistently defined tuples. Humor it.

Okay. Remember how I told you to quit after making the first cut into your cake? Go ahead, make the second cut. I’ll wait…… Oh good, you’re back. Hey, you have a little icing on your chin. Right there. No, there. To the left. There you go, got it. So, you cut twice (asked for two Sets) and ended up with a nice piece of cake (Data) didn’t you? Awesome. Let’s continue to wring the life out of this analogy and look at the MDX.

SELECT
{
	 ([Sales Territory].[Sales Territory Country].[Australia])
	,([Sales Territory].[Sales Territory Country].[Canada])
	,([Sales Territory].[Sales Territory Country].[Germany])
	,([Sales Territory].[Sales Territory Country].[United Kingdom])
	,([Sales Territory].[Sales Territory Country].[United States])
} ON COLUMNS
,
{
	 ([Date].[Calendar Year].[CY 2005])
	,([Date].[Calendar Year].[CY 2006])
	,([Date].[Calendar Year].[CY 2007])
} ON ROWS
FROM [Adventure Works];

Okay, so what’s this doing? Well, it’s saying, “Hey, MDX, I want you to go out and find the Reseller Sales Amount. Then, I want you to break it down for me. I want column headers that show the Countries I’ve specified. Then, I want row headers that show the years 2005 – 2007. Finally, I want the portion of the overall Reseller Sales Amount in a cell at the intersection of the Country and the Year.”

I said that we weren’t going to get into functions yet, but I do have one little thing I want to close with. The Members function. This is sort of like the “SELECT *” of MDX. You can tag a “.Members” onto the end of a [Dimension].[Hierarchy].[Level] reference (or even a [Dimension].[Hierarchy] reference) inside a tuple. I’m going to re-write the COLUMN set to return pretty much the same data, but with less carpal-tunnel syndrome.

SELECT
{
	 ([Sales Territory].[Sales Territory Country].Members)
} ON COLUMNS
,
{
	 ([Date].[Calendar Year].[CY 2005])
	,([Date].[Calendar Year].[CY 2006])
	,([Date].[Calendar Year].[CY 2007])
} ON ROWS
FROM [Adventure Works];

Check that out. It even gives us members we didn’t know to ask for, including an (All) Members summary. This function is great for a couple of reasons: 1) You don’t have to type so much. 2) If you don’t know all of the hierarchy members, you don’t have to go look them up. And, if the members change down the road, you’re not slogging through MDX queries manually updating them.

Alright, so maybe we’re not to awesome-ness yet, but you have to admit, not too shabby. There are about a bazillion other things you can do with these sets, and we’ll get to them. But for now, let’s take a break and enjoy the lovely piece of cake… er, data we’ve created.

Query on, my friends.

*Other Three Rules because the Three Rules are strictly reserved for references to Isaac Asimov’s I, Robot and the Foundation series. If you’ve only seen he movie, for the love of all that is good in this world, go read the books. While you’re at it, go read Starship Troopers by Robert Heinlein. That book got the short end of the movie stick too. Seriously, Denise Richards? Denise Richards?!? They should have made her shave her head to stay true to the story.

My favorite movie is My Fair Lady. I love Audrey Hepburn. I love the Pygmalion story. Quick aside: I used to tell people that my parents named me after her. They didn’t, and the true story is convoluted. My mom loved the name Audrey Dalton (my middle name is Dalton), which was the name of a movie star. My great-great grandmother was Dalton Harris, and she thought it would be cool to name me after her and the actress. Then she met my dad. His mom’s name was Audrey. (She went by her middle name, Geraldine, which I never understood… but I digress.) Anyway, when I was born, she told everyone that I was named for my paternal grandmother and my maternal great-great grandmother. When, secretly, I was just named after an actress with a name she liked. I’m glad she told me this. (Audrey Dalton was a total hottie.)

<– Audrey Dalton, HOTTIE (courtesy of Ballybane Enterprise Centre http://www.bbec.ie/blog/?p=708)

This week, I’ve decided to start digging into MDX. There are three reasons for this:

1) It’s PASS Summit week. While I’m not there, I’m trying to get into the spirit of things by learning something new.
2) I’m just not good at MDX. There is no excuse for this.
3) I’m gearing up for my MCITP exam in Business Intelligence 2008. I hear rumor that there are MDX questions.

So anyway, I feel a lot like Eliza Doolittle this week. If you’re not familiar with the story, she is the subject of a bet between Henry Higgins and Colonel Pickering. They bet that Prof. Higgins can’t pass her off as a Lady (with a capital “L”) in a year. She’s just a lowly flower girl, complete with cockney accent. In order to refine her, he has to teach her how to speak again. It’s her own language, but she has to learn how to use it in a totally unfamiliar way. Instead of saying, “In ‘artford, ‘ereford, and ‘ampshire, ‘urricanes ‘ardly h-ever ‘appen”, she has to learn to say, “In Hartford, Hereford, and Hampshire, hurricanes hardly ever happen”. (Swear to cheesus, I haven’t hit IMDB yet… I really love this movie) Same words, same meaning, totally different accent.

Rather than a flower girl trying to sound like a Lady, I’m a T-SQL girl trying to sound like an MDX Lady. Or something like that. You know what I mean.

<– T-SQL Flower Girl

To get started, I picked up Microsoft SQL Server 2008 MDX Step by Step (by Brian C. Smith, C. Ryan Clay, and Hitachi Consulting). I’m starting with the basics, so right now I’m in “SELECT * FROM” territory. Or, “SELECT FROM ” territory, since we’re talking MDX.

Transitioning from T-SQL to MDX is not easy. The syntax is just familiar enough to me to trip me up. I keep catching myself trying to equate a query against a cube to a query against a relational data store. It’s not the same, and it has been tough for me to wrap my head around it. But, “I washed my face and ‘ands before I come, I did”, so I think I’m ready to get started.

So far, I’ve learned about one important concept: Tuples. The point of this blog post is to force myself to regurgitate what I’ve learned, because to paraphrase something Jen McCown (Blog | Twitter) said the other day, you don’t really know something until you’ve taught it. True that. Please keep reading, but also read a book by an expert. I’ve been happy with the Step by Step book so far.

Wait… one more silly analogy. Writing T-SQL is a bit like cutting out paper dolls. It can be complex, but it’s just two dimensional space. Writing MDX is like chiseling a hole into a big rock at a specific point. It’s n-dimensional space. While a bit goofy, this visualization has really helped me draw a line between T-SQL and MDX.

Tuples (as Translated by Me)

A tuple is basically the identifying characteristics of a cell inside a cube. Really, a data point inside a cube. Say I have three dimensions, Actor, Movie, and Year. Say I have a Measure Group that includes Budget Amount. Say I wanted to find the cell, or data point, identifying the budget for the movie My Fair Lady starring Audrey Hepburn that came out in 1964. I’d look at the attribute-hierarchies Audrey Hepburn, My Fair Lady, and 1964. (Hey, I didn’t say it was a well designed cube!) Those identifying characteristics of the cell are the tuple, which would be formatted something like this in MDX;

(
[Actor].[Audrey Hepburn]
,[Movie].[My Fair Lady]
,[Year].[1964]
,[Measures].[Budget Amount]
)

Another way to look at it is in terms of math. I always swore that geometry and algebra were pointless in high school. Well, Mr. Smith, you were right. I’m about to talk axes. (axises? axii?) Each of my attribue-hierarchies and my measure group make up an axis within my cube. Don’t even try to visualize a 4-dimensional cube. I did, and it made my head hurt when I ran out of 3-dimensional space. Let’s label each axis:

[Actor].[Audrey Hepburn] = a1
[Movie].[My Fair Lady] = a2
[Year].[1964] = a3
[Measures].[Budget Amount] = a4

Now, if I want to identify the point that is the intersection, my notation would look something like this: (a1, a2, a3, a4). I also imagine four lines (in 2-dimensional space, all intersecing one another at one point. That point is my tuple.

The MDX syntax for my query looks like this:

SELECT
FROM [Pretend Movie Cube]
WHERE
(
[Actor].[Audrey Hepburn]
,[Movie].[My Fair Lady]
,[Year].[1964]
,[Measures].[Budget Amount]
);

It would return one value: $17,000,000

Some Key Points:

1) Every attribute-hierarchy gets an axis, NOT every Dimension. So, if I had two attribute-hierarchies within my Actor dimension, Audrey Hepburn and Rex Harrison, they all have an axis. I could actually reference the same Dimension multiple times like so:

(
[Actor].[Audrey Hepburn]
,[Actor].[Rex Harrison] <–Henry Higgins!
,[Movie].[My Fair Lady]
,[Year].[1964]
,[Measures].[Budget Amount]
);

2) Measures each get an axis. They are treated differently at design time, but for the purposes of seeking out that one cell or set of cells, it’s treated just the same as an attribute hierarchy.

3) Analysis Services allows you to be lazy. You can define what’s called a Partial Tuple, leaving out some axis references. But… it’s going to try to figure out where on that missing axis you were headed. It’s going to go in this order:
1 – Default member (defined at design time)
2 – (All) member –remember that Measures don’t have an (All) member
3 – First member

Am I getting this? Have I missed the boat? Close, but no cigar? Any other cliche suggesting I don’t know what I’m talking about?

I think of them as rogue tables. They’re quick and dirty and cause you a world of hurt before it’s all over. We’ve all got them. Like those photos from college that guarantee you’ll never run for public office, rogue tables are best left hidden. But, you’re always wondering when they’re going to show up in public. I admit, I have one. There, I said it. It’s a config table for our ETL processes that we threw out there at the last minute to handle a data-driven filter on an import process. Didn’t stop and think about a primary key or constraints, just threw it into the database to get something done before the production push. Yes, you heard right. Production. Oy vey.

The other day, I was making use of my roguish table to add a few rows (in my development environment, thank the database gods). Trying to multi-task, I ran a scripted insert on it. Then I answered the phone, responded to an IM, read an e-mail, and turned around and executed the same blasted statement. Without constraints of any kind to save my distracted soul, I now had two of each row. I don’t care what those guys at Wrigley’s say, sometimes two of something doesn’t double the fun. It did double the headache I was already nursing from a morning status meeting.

I needed to get those extra rows out with as little pain as possible. I needed to make it interesting. Look, I find my thrills wherever I can. I took a CTE/ROW_NUMBER() approach to finding and removing my duplicate rows. First, let’s talk about these two constructs.

Common Table Expressions (CTE)

I’ve heard CTE’s described a few different ways: in-line temp table, in-line view, work area, etc. What it does is allow you to create a temporary, named result set. It persists (is scoped) for a single SELECT, INSERT, UPDATE, or DELETE statement. It is a lot like creating a temporary table or using a table variable, but with about half the hassle. The syntax is crazy-simple:

WITH <any name you want> AS
(
SELECT col1, col2
FROM tblx
)
<Your SELECT, INSERT, UPDATE, or DELETE goes here>;

Basically, you can prep data to be used in the statement that immediately follows your WITH. It’s great for any pesky operation that just won’t work well in a single statement. Personally, I think it is easier to read, too. One note: If you’re running multiple statements in a batch, make sure you end the statement just prior to the WITH with a semi-colon. In fact, just end everything with a semi-colon. It makes you look detail-oriented.

ROW_NUMBER()

ROW_NUMBER() falls into the “ranking functions” category. With this quite functional function, you can number rows in your result set. Even better, you can PARTITION BY to split your result set up into groups. I might not want to see 1-10 as my row numbers, I might want to see 1-5 and 1-5 based on some column that I decide to partition the data by. Note, this is a horizontal partition of rows. If you’re trying to partition your columns vertically, we might need to talk over a beer or two. You’ve got bigger issues than duplicate rows. The syntax takes a little getting used to, but once you break it down, it makes pretty decent sense:

ROW_NUMBER() OVER (PARTITION BY colx, coly… ORDER BY colz) as aliasname

Let’s take a closer look:

• ROW_NUMBER() – you’re instructing the query engine to give you back a column with row numbers. These come back as a bigint.
• OVER – you’re telling it that you’re about to give it some more information. Specifically, an ORDER BY and an optional PARTITION BY.
• PARTITION BY – you’re providing instructions about how to group the rows. You can partition by multiple columns. This works a little like a GROUP BY clause.
• ORDER BY – what order do you want your rows numbered in? If you have a PARTITION BY, it’ll order within each partition. If you’ve left the PARTITION BY out, it’ll order the entire result set
• alias – you’re going to have to alias this new column so that you can reference it later on

Now that we’re all CTE and ROW_NUMBER() experts, let’s talk about how we put these guys to work to undo my bone-headed duplicate row insert. I’m scripting an example here, with bonus semi-witty comments.

–Create the rogue table
IF EXISTS (SELECT * FROM sys.tables WHERE name = N’TableOfShame’)
BEGIN
DROP TABLE TableOfShame
END

CREATE TABLE TableOfShame
(
ShameCode varchar(4) NULL,
ShameType varchar(15) NULL,
ShamePriority varchar(10) NULL
);

–Insert the rows you really wanted in your table
INSERT INTO TableOfShame
VALUES
(’01′, ‘Chagrin’, ‘Low’),
(’02′, ‘Disgust’, ‘High’),
(’03′, ‘Abashment’, ‘Medium’),
(’04′, ‘Embarassment’, ‘Low’),
(’05′, ‘Humiliation’, ‘Medium’);

/* Answer the phone, check your e-mail, listen to your co-worker tell hilarious story, get generally distracted */

–Oops, insert them again (Note the sleek and modern Table Value Constructor)
INSERT INTO TableOfShame
VALUES
(’01′, ‘Chagrin’, ‘Low’),
(’02′, ‘Disgust’, ‘High’),
(’03′, ‘Abashment’, ‘Medium’),
(’04′, ‘Embarassment’, ‘Low’),
(’05′, ‘Humiliation’, ‘Medium’);

–Look what you’ve done! Damn that funny anecdote that completely derailed your train of thought.
SELECT * FROM TableOfShame;

–Find the duplicates, give them something differentiating (a row number!)
/* Based on my made-up business rules, I’ve partitioned by something resembling a business key. It’ll give me unique groups, which is sort of an oxymoron, but you know what I mean. */
WITH cte_FindDuplicateShame as
(
SELECT ShameCode, ShameType, ShamePriority,
ROW_NUMBER() over(PARTITION BY ShameCode, ShameType ORDER BY ShameCode DESC) as RowNum
FROM dbo.TableOfShame
)
SELECT ShameCode, ShameType, ShamePriority, RowNum
FROM cte_FindDuplicateShame
ORDER BY ShameCode, ShameType, RowNum;

–Now, we know what we have, let’s delete the duplicates
/*Note that I’m actually issuing the DELETE against the CTE. Keep in mind that the CTE is only a temporary, named result set off of a physical table (sort of like an in-line view). Running the DELETE against the CTE will affect the physical table that was used to create the result set. */

WITH cte_FindDuplicateShame as
(
SELECT ShameCode, ShameType, ShamePriority,
ROW_NUMBER() over(PARTITION BY ShameCode, ShameType ORDER BY ShameCode DESC) RowNum
FROM dbo.TableOfShame
)
DELETE cte_FindDuplicateShame
WHERE RowNum <> 1;

–Aha! Distraction-created rows are gone.
SELECT *
FROM TableOfShame
ORDER BY ShameCode;

So there you have it. A mildly interesting way to get myself out of the hole I dug by getting F5 happy. CTE on, my friends.

If you’re not familiar with PIVOT, go back and read the original post. If I’ve done my job properly, it should make sense. So, here’s what I did… I resisted the urge to hit Google to find a solution to the dynamic pivot problem. I opened SSMS and said, “Self, you’re under a deadline. Write it and see if you can get it to work all by your lonesome”. 45 minutes later, I had a working script that produced some cool real-world output, if I do say so myself.

Then, I hit Google. Then I saw Itzik Ben-Gan’s solution. My first response was, “Crap!” Actually, it was a much less ladylike expletive than that. The solution was… Beautiful. Elegant. Blew my method out of the water. You know how athletes have muscle memory? Well, developers have it too. We fall back to what’s comfortable and familiar. Sort of like our own version of T-SQL sweatpants and chocolate ice cream. Before I start in on the comparison of my solution and Itzik’s, let me say this: His is so much better than mine. Did I mention that it was elegant? And beautiful? But you know what? In a real development environment, with deadlines and giant to-do lists, I would have fallen back to my own comfort zone. I know this. I also know that next time I need to write a dynamic PIVOT, I’m going to know how to use his method.

Authors, when asked to give advice to aspiring writers, always say the same thing. “Write what you know.” For us IT Folk, there’s a corollary. “Write what you know. Hit the deadline. Then, go learn a better way.” Am I proud that I figured a solution out on my own? Yup. Am I a bit deflated that I didn’t come up with the same solution as Itzik Ben-Gan? Nope. Come on, it’s Itzik.

Personal note: I hate when I run across someone else’s T-SQL and ask them, “How does this work?”, and their response is, “I don’t know, I found it on a blog post/Google/forum.” Peeps, this is unacceptable. Don’t copy and paste until you understand what you’re seeing. Because someday you’re going to have to maintain that pilfered bit of code. If you don’t know what it does, then don’t use it. Comprehend your own code. We all borrow from the experts, but make sure you can explain it in 50 words or less. If you can’t, then back away from the Ctrl+V. Stretch your skills, learn new things, just don’t jeopardize a project by jumping the gun.

Okay, enough commentary. On to the solutions. The trick in a dyamic PIVOT is to create a string that has all of the column headers you need. This is where he and I diverged wildly. I fell back on a WHILE Loop over a set of rows contained in a table variable, he used the STUFF function with a FOR XML Path() query output. I wrote my solution to address the same example from BOL that I ranted about in my first post. I modified his solution to produce the same output, and to clean out some unused variables that were in the sample I found. I’ve also resisted the urge to make little tweaks to my script after doing some extra research. Truly, I want to make the point that there’s what works… and what works beautifully.

My solution:

SET NOCOUNT ON;

DECLARE @vEmployeeIDTable as TABLE
(
EmployeeID varchar(20) NOT NULL
,ProcessedFlag bit NOT NULL DEFAULT(0)
)

DECLARE @vEmployeeID varchar(20)
DECLARE @vSQLString varchar(max) = ”
DECLARE @vEmployeeIDSELECT varchar(max) = ”
DECLARE @vEmployeeIDFOR varchar(max) = ”
DECLARE @vLoopCounter varchar(50) = 1

INSERT INTO @vEmployeeIDTable(EmployeeID)
SELECT DISTINCT EmployeeID
FROM Purchasing.PurchaseOrderHeader;

WHILE (SELECT count(ProcessedFlag) FROM @vEmployeeIDTable WHERE ProcessedFlag = 0) > 0
BEGIN

SELECT @vEmployeeID = ‘['+cast(MIN(EmployeeID) as varchar(20)) +']‘
FROM @vEmployeeIDTable
WHERE ProcessedFlag = 0

SET @vEmployeeIDSELECT = @vEmployeeIDSELECT + @vEmployeeID + ‘ as Emp’+@vLoopCounter+’,’
SET @vEmployeeIDFOR = @vEmployeeIDFOR + @vEmployeeID +’,’

UPDATE @vEmployeeIDTable
SET ProcessedFlag = 1
WHERE EmployeeID = cast(substring(@vEmployeeID,2, LEN(@vEmployeeID)-2) as int)

SET @vLoopCounter = @vLoopCounter + 1

END

SET @vEmployeeIDSELECT = SUBSTRING(@vEmployeeIDSELECT,1, len(@vEmployeeIDSELECT)-1)
SET @vEmployeeIDFOR = SUBSTRING(@vEmployeeIDFOR,1, len(@vEmployeeIDFOR)-1)

SET @vSQLString = ‘
SELECT VendorID, ‘+@vEmployeeIDSELECT +’
FROM
(SELECT PurchaseOrderID, EmployeeID, VendorID
FROM Purchasing.PurchaseOrderHeader) p
PIVOT
(
COUNT (PurchaseOrderID)
FOR EmployeeID IN
(‘+@vEmployeeIDFOR+’)
) AS pvt
ORDER BY pvt.VendorID; ‘

PRINT @vSQLString

EXECUTE (@vSQLString)

So, a quick rundown of what I did:

1) Create a table variable (@vEmployeeIDTable). Populate it with DISTINCT EmployeeID’s from Purchasing.PurchaseOrderHeader.
2) Declare the following variables:
a) @vEmployeeID – holds the EmployeeID I’m concatenating into the string during the WHILE loop
b) @vEmployeeIDSELECT – holds the EmployeeID string that I’ll use in the SELECT clause of my PIVOT. I separate this one out because I want to concatenate the column aliases just as they were in the BOL example.
c) @vEmployeeIDFOR – holds the EmployeeID string that I use in the FOR clause of my PIVOT. I don’t need column aliases here.
d) @vLoopCounter – holds a counter as I loop through the string concatenation. I use it to help name my column aliases (Emp1, Emp2…). The 1 and 2 are coming from this variable
3) While I have unprocessed rows in my table variable, I loop through with a WHILE
a) Set @vEmployeeID to the minimum EmployeeID that hasn’t been processed. I also concatenate on the brackets I need since these will become column names. (Those brackets were a pain. I kept having to work around them. Another place where Ben-Gan’s method was more elegant)
b) Set @vEmployeeIDSELECT to itself plus the EmployeeID being processed (@vEmployeeID), and then set up the alias. (as ‘Emp’+@vLoopCounter). Important note: I initialized the variable as an empty string (”). This is so that I’m not trying concatenate a NULL value to a string on the first go-round.
c) Set @vEmployeeIDFor to itself plus the EmployeeID being processed
d) Update @vEmployeeIDTable to indicate that the EmployeeID has been added to the string variables
e) Update @vLoopCounter so that the next table alias will be the next number
4) Clean up the extra commas at the end of the string variables
5) Put the whole thing together in @vSQLString
a) Place the @vEmployeeIDSELECT variable where it needs to go
b) Place the @vEmployeeIDFOR variable where it needs to go
6) Execute the variable @vSQLString

This is the output:

Okay, not bad. Now, the elegant Itzik Ben-Gan solution:

DECLARE
@cols AS NVARCHAR(MAX),
@sql AS NVARCHAR(MAX);

SET @cols = STUFF(
(SELECT N’,’ + QUOTENAME(EmployeeID) AS [text()]
FROM (SELECT DISTINCT EmployeeID FROM Purchasing.PurchaseOrderHeader) AS Y
ORDER BY EmployeeID
FOR XML PATH(”)),
1, 1, N”);

SET @sql = N’SELECT ‘+@cols +’
FROM (SELECT VendorID, EmployeeID, PurchaseOrderID
FROM Purchasing.PurchaseOrderHeader) AS D
PIVOT(COUNT(PurchaseOrderID)
FOR EmployeeID IN(‘ + @cols + N’)) AS P
ORDER BY P.VendorID;’;

PRINT @SQL

EXEC sp_executesql @sql;
GO

I know, right? Elegant. So what did he do?

1) Declared a couple of variables
a) @cols – holds the string of column values for the PIVOT
b) @sql – holds the SQL statment that gets executed
2) Used a FOR XML PATH(”) command to concatenate the string. This is cool. The query pulls EmployeeID’s out of a derived table in the FROM Clause. He orders by EmployeeID (which is not required), and outputs the result of this query using FOR XML PATH(”). The FOR XML PATH(”) clause creates a single row that looks like this:

,[250],[251],[252],[253],[254],[255],[256],[257],[258],[259],[260],[261]

Wow, exactly what we need for the PIVOT. Well, almost. That’s what the STUFF function is for. Getting rid of “almost”.

3) Also, see how he used QUOTENAME to add the brackets he needed?

QUOTENAME(EmployeeID) AS [text()]

4) Then, since that leading comma (,[250]) is not needed, he uses the STUFF command to strip it off. STUFF looks like this:

STUFF ( character_expression , start , length ,character_expression )

a) character_expression – the results of the query containing the FOR XML PATH(”) output
b) start – first character
c) length – how many characters to replace with what we’re “stuffing” in. In this case, a length of 1.
d) character_expression – an empty string, which is what’s’ “stuffed” into the first character expression, eliminating the comma.

Try this to illustrate it much more simply:

SELECT STUFF(‘abcdef’, 1, 1, ”);

Your result is: ‘bcdef’. The empty string he specified basically replaces the first character which is the comma we don’t want. Seriously, I had to run the baby STUFF to understand it properly. The beauty of STUFF over SUBSTRING is that SUBSTRING requires you to tell the function the length of the resulting string, which would require a LEN function over the entire subquery to get it right. It saves you having to execute that bad boy more than once.

5) Finally, he just puts the PIVOT query into @sql, concatenating in @cols where he needs to, and then executes it.

This is his output:

So he didn’t do pretty column aliases, but the important data is the same. And just take a look at the execution plans. That’s where I do feel just a bit deflated. Mine is monstrous. His? TWO queries. TWO! But that’s not the point. The point is, I had a blast figuring out how to write my own dynamic PIVOT. I had even more fun dissecting Itzik Ben-Gan’s method. (Yeah, I know. I’m a dork.) And, you can bet your sweet bippy that I’ll be working to make sure that FOR XML PATH, STUFF, and QUOTENAME all become part of my T-SQL muscle memory.

I went to BOL, and looked it up. Now, I’m a fan of Books Online. It saves my tush daily. But in this case… I’m sorry, but the explanation is nonsensical. I mean, I read it, and what I comprehend is, “blah, blah, PIVOT, blah, you’re an idiot, Audrey, just give up now”.

So, being forced to use a PIVOT, I had to break it down into chunks that my tiny brain could consume. So, first, let’s look at the BOL syntax:

SELECT <non-pivoted column>,

[first pivoted column] AS <column name>,

[second pivoted column] AS <column name>,

…

[last pivoted column] AS <column name>

FROM

(<SELECT query that produces the data>)

AS <alias for the source query>

PIVOT

(

<aggregation function>(<column being aggregated>)

FOR

[<column that contains the values that will become column headers>]

IN ( [first pivoted column], [second pivoted column],

… [last pivoted column])

) AS <alias for the pivot table>

<optional ORDER BY clause>;
Hoo-kay. I’m going to step you through my process of understanding this so I could construct my own PIVOT. I’m even going to use the complex pivot example from BOL, the AdventureWorks2008 database. We’re going in this order: FROM, PIVOT, FOR, SELECT.

But first, some rules. There are always rules:

RULES:
1) You have to know how many columns you’re going to end up with after the PIVOT. This means that this operation is great for things like months in a year, not so great for a varying number of pivoted columns. You can tell it which columns to return, but the bottom line is you need to know what your output should look like. If you want to break this rule, you’re writing dynamic SQL.
2) You’re going to have to aggregate. Even if you don’t really want to. It’s required, but as always, there are ways to work the syntax.

THE BOL QUERY EXAMPLE:

SELECT VendorID, [250] AS Emp1, [251] AS Emp2, [256] AS Emp3, [257] AS Emp4, [260] AS Emp5
FROM
(SELECT PurchaseOrderID, EmployeeID, VendorID
FROM Purchasing.PurchaseOrderHeader) p
PIVOT
(
COUNT (PurchaseOrderID)
FOR EmployeeID IN
( [250], [251], [256], [257], [260] )
) AS pvt
ORDER BY pvt.VendorID;

THE BOL QUERY OUTPUT:

THE BREAKDOWN:

1) FROM (Source Query): This is the derived table that lives in the FROM clause. It produces the data that is going to be aggregated and pivoted. Write this first. Get familiar with what data you’re working with. Don’t forget to give it an alias. I like the ever-creative “as SourceQuery” to help me remember what that derived table’s doing there in the first place.

FROM

(<SELECT query that produces the data>)

AS <alias for the source query>

In the BOL example, this is the Source Query:

FROM (
SELECT PurchaseOrderID, EmployeeID, VendorID
FROM Purchasing.PurchaseOrderHeader) as p

It returns this:

This is our raw data. By the time we get to the bottom of this blog post, we’re going to COUNT PurchaseOrderID’s by EmployeeID, set some EmployeeID’s as column headers, and return what looks like a cross-tab report with VendorID’s as row headers, EmployeeID’s as column headers, and PurchaseOrder COUNT as detail data. Really. I promise.

2) PIVOT (Aggregation/Summarization): This is where you’re saying how to aggregate, or summarize what will end up in the cells. Think of it this way: If this were a spreadsheet, with column headers and row headers, the data produced by the PIVOT clause is the detail data living in the cells. Now, you don’t always want to aggregate. Sometimes you don’t have anything to aggregate, you just want to flip your data from rows to columns. Too bad. You’re aggregating something. The solution I’ve seen is to do a MIN or MAX, but to make sure that the MIN or MAX is of a unique thing. You’ll have to examine your data to see what works for you. But back to PIVOT…

PIVOT
(
<aggregation function>(<column being aggregated>)

In the BOL example, it looks like this:

PIVOT
(
COUNT (PurchaseOrderID)

So, what it’s saying is that the “detail” data (think like you’re in Excel for a moment) should be the count of PurchaseOrderID’s. Simple enough. But where’s my GROUP BY? It feels like heresy, aggregating something without a GROUP BY. Hang in there…

3) FOR (Sort-of GROUP BY): FOR establishes what will be column headers for the PIVOT-ed (aggregated) data. One cool thing about it not being a true GROUP BY is that I don’t have to include everything from my Source Query (FROM). If you look at the BOL example, VendorID from my Source Query (FROM) isn’t included in the PIVOT or FOR clauses. It’s a pass-through column. It’s going to be there in the SELECT, and therefore in the output, but it isn’t part of the PIVOT process. In fact, you don’t have to include VendorID at all. The data probably wouldn’t make sense, but to each his own, right?

FOR

[<column that contains the values that will become column headers>]

IN ( [first pivoted column], [second pivoted column],

… [last pivoted column])

) AS <alias for the pivot table>

In the BOL example, the query developer chooses to return the number of purchase orders for a specific set of Employees. Yes, in the example it’s arbitrary, because they return 5 and there are actually 12 distinct EmployeeID’s in the Purchasing.PurchaseOrderHeader table, but I’m not here to judge. How do they do this? Like this:

FOR EmployeeID IN
( [250], [251], [256], [257], [260] )
) AS pvt

This is telling the PIVOT to produce 5 columns, [250], [251], [256], [257], and [260]. (You don’t have to have the brackets, except that “250″ wouldn’t be a valid column name without them.) Those numbers are the actual EmployeeID’s returned from the Source Query. You’re saying “FOR” an EmployeeID “IN” a specific set of values that were returned in the Source Query (FROM). You’re essentially establishing a GROUP BY on EmployeeID. What’s being “grouped” by the FOR clause? The data that you’re aggregating in the PIVOT clause. Cool, huh? The COUNT of PurchaseOrderID’s will be placed underneath the column corresponding to the EmployeeID it belongs to. Don’t forget to alias the FOR clause. Something like “IRockBecauseIFiguredThisOut” works well. Also, this is where you’re going to close the parenthesis that you opened up in the FROM clause.

Personal Note: This clause is one of the reasons I hate this BOL example. It doesn’t make sense that I would hard-code EmployeeID’s. A PIVOT example with months or years or something would be a more likely real-world scenario. Making it an example implies that it’s a good idea, and that every person reading BOL knows not to assume that Employee 257 will be a lifer at Adventure Works. But like I said, I don’t judge.

4) SELECT (Presentation): Why is it that SELECT is always the simplest part of a query? It seems so important, but it really doesn’t do much. It’s like the presentation layer of the query. Here, you’re telling the query what to output. As long as it was part of the Source Query (FROM), or defined as a column header in the FOR clause, you can include it in the SELECT clause. In fact, if you’re feeling frisky, you can leave off columns. The query doesn’t care, because the SELECT is just there to make things pretty.

SELECT <non-pivoted column>,

[first pivoted column] AS <column name>,

[second pivoted column] AS <column name>,

…

[last pivoted column] AS <column name>

In the BOL example, it looks like this:

SELECT VendorID, [250] AS Emp1, [251] AS Emp2, [256] AS Emp3, [257] AS Emp4, [260] AS Emp5

VendorID is a pass-through (non-pivoted) column. It’s there to supplement the PIVOTed data. The other columns are the ones we established in the FOR clause. Just remember that everything you want to work with needs to be included in that Source Query (FROM clause).

Putting it all together, it looks like this:

SELECT VendorID, [250] AS Emp1, [251] AS Emp2, [256] AS Emp3, [257] AS Emp4, [260] AS Emp5
FROM
(SELECT PurchaseOrderID, EmployeeID, VendorID
FROM Purchasing.PurchaseOrderHeader) p
PIVOT
(
COUNT (PurchaseOrderID)
FOR EmployeeID IN
( [250], [251], [256], [257], [260] )
) AS pvt
ORDER BY pvt.VendorID;

The output looks like this:

So there you have it. A peek into my thought process as I worked to overcome my fear of PIVOT. I’m good now. I’ll still have to look up the syntax whenever I write it, but at least I won’t break out into a cold sweat next time. And next up for me… PIVOT with an unknown/dynamic number of output columns. Woo-hoo! Dynamic SQL!

Query on, my friends.



Anyway, I was working my way through the Self-Paced Training Kit, and stumbled across one thing that I hadn’t known before and am so excited about that I want to share it.  Here we go…

First, because good bloggers give credit where credit is due, major props to the team that wrote the training kit for exam 70-433.  They include:  Tobias Thernstrom, Ann Weber, Mike Hotek, and GrandMasters.  It’s a well-written book, and they obviously snuck some things in there that were more about good design and development and less about answering test questions.  Kudos, gentlemen and lady!  I’m basically re-writing something that you’ve already covered in your book, but I think it’s really, really cool and want everyone to see it regardless of whether they’ve bought the book or not.  Given that, buy the book, dear readers.  Even if you’re not studying for the exam.  I promise you’ll learn something new.

Okay, to the point… Do me a favor, get on your local database and run this query.  Go ahead, I’ll wait for you. 

SELECT name, is_not_trusted
FROM sys.foreign_keys; 

Good to have you back.  I missed you.  I stared pensively into the horizon while I awaited your return.  I even wrote a poem and a folk song.  Sorry, I digress… one too many rom-coms lately.  Did you see any “1” values in the is_not_trusted column?  Did you even know that foreign keys could be trustworthy?  Nope, neither did I.  What does it mean?  It means that your foreign key hasn’t been verified by the system.  How does this happen?  Well, remember that optional clause called WITH CHECK | NOCHECK when you create a foreign key constraint?  Yup.  That did it. 

So why does this matter? Well, it actually has an effect on your query execution plan in some cases.  You know, that Query Optimizer is pretty darn smart.  Let’s look at an example from the trusty old AdventureWorks database.  I’m using AdventureWorks2008R2, but it should work with the older AdventureWorks databases.  The scenario is this:  I want to know if I have any sales orders that have invalid customers.  CustomerID is a NOT NULL column in Sales.SalesOrderHeader.  I know that if I count all the rows in Sales.SalesOrderHeader and the number of rows returned in the query we’re about to run, I should get the same number of rows back each time.  But, sometimes rogue values slip in, especially when the database design has been refined over time.  This could happen for any number of reasons:  constraints that were added after the fact, legacy data, disabled constraints, etc. 

Run this query, but turn on Include Actual Execution Plan (Ctrl+M) before you do. 

SELECT soh.*
FROM Sales.SalesOrderHeader soh
WHERE EXISTS (SELECT * FROM Sales.Customer c WHERE soh.CustomerID = c.CustomerID);

Note that I’m using WHERE EXISTS rather than an IN clause with a subquery.  This is because in this business scenario, I just want a boolean result (true or false), and I want it to run FAST. 

Check out the execution plan.  Note that the Customer table was never accessed.  Why?  Because that foreign key is trusted!  Since it was verified on creation, we know that no rogue CustomerID’s snuck into the Sales.SalesOrderHeader table.  It doesn’t even need to look at it.  
 

Now, let’s muck with the foreign key and make it un-trusted.  We’re disabling the foreign key with this statement.  Books Online has a good article about what this means. 

ALTER TABLE Sales.SalesOrderHeader
 NOCHECK CONSTRAINT FK_SalesOrderHeader_Customer_CustomerID;

Verify that the foreign key is disabled by checking sys.foreign_keys again: 

SELECT name, is_disabled, is_not_trusted
FROM sys.foreign_keys
WHERE name = ‘FK_SalesOrderHeader_Customer_CustomerID’;

If we run our query again, we see a completely different execution plan.  Now, the query optimizer has to go look at the Sales.Customer table to get us an answer: 

SELECT soh.*
FROM Sales.SalesOrderHeader soh
WHERE EXISTS (SELECT * FROM Sales.Customer c WHERE soh.CustomerID = c.CustomerID);

 
The execution plan had to change because SQL Server cannot guarantee that a CustomerID wasn’t entered while the foreign key constraint was disabled. 

Here’s where it gets interesting.  Enable the foreign key by executing the following: 

ALTER TABLE Sales.SalesOrderHeader
 CHECK CONSTRAINT FK_SalesOrderHeader_Customer_CustomerID;

Check out your sys.foreign_keys table again. 

SELECT name, is_disabled, is_not_trusted
FROM sys.foreign_keys
WHERE name = ‘FK_SalesOrderHeader_Customer_CustomerID’;

What?  It’s enabled, but it’s still not trusted!  If we execute our query, we’re still going to get the execution plan that looks at Sales.Customer.  Why?  Well, that CHECK keyword up there just said to enable the foreign key, it didn’t say verify it.  We have to issue this statement (Of course, if an invalid CustomerID snuck in while your FK was disabled, this ALTER is going to fail): 

ALTER TABLE Sales.SalesOrderHeader
 WITH CHECK —> this clause will make your FK trustworthy again
 CHECK CONSTRAINT FK_SalesOrderHeader_Customer_CustomerID;  

If we run our query now, we’ll get the sleeker, more efficient plan, because now our foreign key is enabled and trusted. 

SELECT soh.*
FROM Sales.SalesOrderHeader soh
WHERE EXISTS (SELECT * FROM Sales.Customer c WHERE soh.CustomerID = c.CustomerID);

 
Cool, huh?  The point is, this one teensy-tiny flag in the foreign key metadata makes a huge difference in how the query optimizer handles your query.  It might not be much, but why not make sure that you can get as many trusted foreign keys as possible?  You might just end up looking like a rock star for improving performance without having to modify any actual queries. 

Now if I could just figure out put to get a is_not_trusted flag on people…