Topic: SSIS Logging and Auditing Framework
Level: Beginer / Intermediate

Meeting Place
Matrix Resources
115 Perimeter Center Place
Suite 250 (South Terraces Building)
Atlanta, GA 30346

Overview: Proper logging in your SSIS packages can help save hours in determining what went wrong with your package execution. Auditing may be required to track what package touched what data and when. This session will review some best practices in setting up logging and auditing, walk through a standardized SSIS template that can be easily used as a starting point for all SSIS packages and review a reporting solution to quickly monitor your new package logs.

About Me: Matt Wollner in a Business Intelligence consultant with Key2 Consulting. He has extensive experience in data warehousing design, SQL Server database, ETL processes, multidimensional database and reporting systems. He has been working with SQL server products for the past 13 years with a focus on BI for the last 10 years. Matt has successfully delivered large scale data warehousing solutions utilized by companies in the telecom, financial services, information aggregation, pharmaceutical and state and local governments.

Todd McDermind has a good post on how to query the sysssislog table, if you are using the built in SQL SSIS logging.

http://toddmcdermid.blogspot.com/2011/04/transforming-ssis-sysssislog-entries.html

1. The XML file can only contain 1 namespace. I had to do a string replace in my TSQL to remove the namespaces and keep the element names unique.

Original XML

<root>
<h:table xmlns:h="http://www.w3.org/TR/html4/">
<h:tr>
<h:td>Apples</h:td>
<h:td>Bananas</h:td>
</h:tr>
</h:table>
<f:table xmlns:f="http://www.w3schools.com/furniture">
<f:name>African Coffee Table</f:name>
<f:width>80</f:width>
<f:length>120</f:length>
</f:table>
</root>

TSQL Replace

SELECT
REPLACE(
REPLACE(
REPLACE(@XML,'<f:table xmlns:f="http://www.w3schools.com/furniture">','<f:table>')
,'<h:table xmlns:h="http://www.w3.org/TR/html4/">','<h:table>')
 ,':','-')

Final XML

<root>
<h-table>
<h-tr>
<h-td>Apples</h-td>
<h-td>Bananas</h-td>
</h-tr>
</h-table>
<f-table>
<f-name>African Coffee Table</f-name>
<f-width>80</f-width>
<f-length>120</f-length>
</f-table>
</root>

2. Each child element in the XML file will get its own data flow output. If you are looking to flatten out the XML file you will need to join each output together. The XML source conveniently added a _id for each level in the XML. If you sort these, they can be joined together. In the XML below both the food and nutrition outputs with have a food_id. We you join the two outputs on the food_id the calories and fat data will be flattened out to the same row as the other food information.

<breakfast_menu>
<food>
<name>Belgian Waffles</name>
<price>$5.95</price>
<description>two of our famous Belgian Waffles with plenty of real maple syrup</description>
<nutrition>
<calories>650</calories>
<fat>13</fat>
</nutrition>
</food>
<food>
<name>Strawberry Belgian Waffles</name>
<price>$7.95</price>
<description>light Belgian waffles covered with strawberries and whipped cream</description>
<nutrition>
<calories>900</calories>
<fat>27</fat>
</nutrition>
</food>
</breakfast_menu>

3. The XML source can pull data from an XML file on disk or from a variable. If you are moving you package to multiple environments you will want to have the path of the XML file to be dynamic. This is easy by either creating a variable that contains the full file path of the XML file (XML file from variable) or in the control flow, pulling the value of the XML document and storing the XML in a string variable (XML data from variable). The one problem of this is that, in the data flow, there is no way to make the XSD dynamic. In order to add an expression to the XML Source, you must go back to the Control Flow and set an expression to the Data Flow for the [XML Source].[XMLSchemaDefinition].
4. And finally, your XML may not be formatted. If you open the XML in Management Studio you can format it quickly. Click on Edit –>Advanced –> Format (Ctrl+K, Ctrl+D)

Format XML in SSMS

I did the following to get my list of files

1. Create an FTP Connection manager.  I named mine connFTP.  You do not have to file in all the info here.  We will overwrite it in a Script Task.  You will need to put something in the server name.  
2. Create package variables
   1. One for each of the following  FTP server, port, user name and password. 
   2. A variable for the path to the files, if the files are not in the FTP root directory, vFTPFileDir.  In my case this looked like “/user name/working folder”
   3. A variable to store the file list, of type Object,   vFTPFileList.
3. Create a script Task (this example is in VB)

 

'1st we must set up the connection to the FTP server

Dim cmFTP As ConnectionManager = Dts.Connections("connFTP")

'Set the FTP connection properties like username & password

cmFTP.Properties("ServerName").SetValue(cmFTP, Dts.Variables("vFTPConnectionString").Value

cmFTP.Properties("ServerUserName").SetValue(cmFTP, Dts.Variables("vFTPUserName").Value)cmFTP.Properties("ServerPassword").SetValue(cmFTP, Dts.Variables("vFTPPassword").Value)cmFTP.Properties("ServerPort").SetValue(cmFTP,  Dts.Variables("vFTPPort").Value)

'Next we will connect to the FTP Server

'create the FTP object that sends the files and pass it the connection created above.

Dim FtpCC As FtpClientConnection = New FtpClientConnection(cmFTP.AcquireConnection(Nothing))

'Connects to the ftp server

 FtpCC .Connect()

'Navigates to the directory in the FTP site

 ftpVentive.SetWorkingDirectory(Dts.Variables("vFTPFileDir").Value)

'Finally  we want to store the list of files to the variable

'Declare the string arrays

Dim saFileNames() As String = Nothing

Dim saFolderNames() As String = Nothing

'Store a list of folders and files into string arrays

FtpCC .GetListing(saFolderNames, saFileNames)

'Set the variable to the file list

 Dts.Variables("vFTPFileList").Value = saFileNames

'This part is optional to loop through each file

 If Not saFileName Is Nothing Then

For Each sFileName As String In saFileNames

'Do Something

Next

End If

'Close the FTP connection

ftpVentive.Close()

Dts.TaskResult = ScriptResults.Success

 

4. Add a Foreach Loop Container
   1. Set the enumerator to Foreach From Variable Enumerator
   2. Set the variable to vFTPFileList

There you have it.  You can now store a list of all the files in an FTP directory and loop through it in a Script Task or in a Foreach Task.

I am using data that I pulled off the US Census web page. I was able to download three CSV files (Population Change, Population Density, Apportionment ). I did a couple of quick data modifications.  The data in the files is stored with a new column for each year. I created a new column called year and unpivoted the data. I then took the data from the three files and merged them into one Excel spreadsheet. I also create a 2nd workbook for the region information.

…Dowload Source Spreadsheet

There are two easy ways to pull data from an existing Excel spreadsheet. You can open an existing spreadshhet and convert the data into Excel Table or you can open a blank spreadsheet and import the data. I will first show how to import data within a existing spreadsheet.

1. Start open the census spreadsheet in Excel 2010 (Make sure PowerPivot is already installed)
2. Under the PowerPivot ribbon select “Create Linked Table”. Select the entire set of census data. This will convert the existing set of data into an Excel Table and will open the PowerPivot browser.
   1. Inside the PowerPivot browser rename the table to Population
   2. Now that the data is in PowerPivot, the data can be sorted by any column, really quickly.

     

3. Close the PowerPivot browser and import the region data.
   1. Open the Region worksheet. Hit Create Linked Table. Select the data range.
   2. Now the Region data has been added as a 2nd table in PowerPivot.
   3. Rename the table to Region.
4. We now have two separate datasets. We need to tell PowerPivot how these two sets of data are related.
   1. Under the Design Ribbon select Create Relationship. The Create Relationship popup will appear.
   2. Select the State Column from each of the two tables that have been created.
   3. You can see that this column now has a relationship.

    

5. Now lets build a simple report.
   1. From the PowerPivot browser (or back in Excel under the PowerPivot Ribbon), select the PivotTable dropdown and select PivotChart.
   2. Place the Chart on a new Worksheet
   3. Under the PowerPivot Field List you will see two data sets (Population and Region)
   4. Add the following
      1. Slicers Vertical: Region, Division, State (you can use the state from either Population or Region)
      2. Slicers Horizontal: Year
      3. Legend Fields: Year
      4. Axis Fields: State
      5. Values: Population(The default will make this a SUM)

The slicers that have been added are a fast and easy way for you to slice and dice (filter) your data.  Each slicer is aware of the other slicers in the PowerPivot dataset.  You will notice that if you click on the Region Slicer, lets say South, both the Division and State slicers will be updated to list the Divisions and States in the South.

…View the final spreadsheet (In order to see PowerPivot in action, you will need to open the document in Excel)