All of the RDMS systems I’ve worked with have built in System Views ( or tables ) that contain data about the structure and contents of your database.  These views are there to help you with non data tasks.  For the sake of this post we’re only going mention the INFORMATION_SCHEMA, which is the system view I use most in SQL Server, and MySQL.  For your reference, here are links to INFORMATION_SCHEMA docs for four popular RDMS systems.

1. SQL Server – http://msdn.microsoft.com/en-us/library/ms186778.aspx
2. MySQL – http://dev.mysql.com/doc/refman/5.5/en/information-schema.html
3. Oracle – http://download.oracle.com/docs/cd/E17952_01/refman-5.0-en/information-schema.html
4. DB2 – http://www.tar.hu/sqlbible/sqlbible0100.html

If you work with SQL Server, you’ll be happy to know you can see all of the System Views in Management Studio by default.

System Views in the AdventureWorks database

If you work with MySQL, you can see the INFORMATION_SCHEMA database in MySQL Workbench, but you’ll have to enable it in Preferences first.

Edit > Preferences > SQL Editor > Check 'Show Metadata Schemata'

Now for the MySQL users, turning on the Metadata Schemata is not required in order to query those system tables.  Assuming your user has appropriate permissions, you can always fire up Workbench and fire a query like this.

SELECT *
FROM INFORMATION_SCHEMA.Columns
WHERE Column_Name LIKE '%user%'

MySQL has INFORMATION_SCHEMA tables too!

Now that you know about INFORMATION_SCHEMA, you should be happy to know this is part of the SQL-92 standard. Which means whatever sql scripts you write in MySQL, will most likely work with SQL Server as well. So go ahead and start querying your database, you’ll probably find uses for the INFORMATION_SCHEMA in your daily life really quickly.

I’m going to assume you’ve got the general idea here, so here are a few sql scripts that I’ve used over the years.  The first three are informational queries, and the last three are stored procedures that generate .NET or Coldfusion code based off of a table name.

Find all tables with the column EmployeeID

SELECT	*
FROM	INFORMATION_SCHEMA.COLUMNS
WHERE	COLUMN_NAME LIKE 'EmployeeID'

Which tables have an EmployeeID column?

How many tables have the word employee in the name

SELECT	*
FROM	INFORMATION_SCHEMA.TABLES
WHERE	LOWER(TABLE_NAME) LIKE '%employee%'

How many %employee% tables are in the DB?

How many SPROCs and UDFs does our database contain?

SELECT	s.SPROCs, f.UDFs
FROM
(
	SELECT	COUNT(ROUTINE_NAME) AS 'SPROCs', NULL AS 'UDFs'
	FROM	INFORMATION_SCHEMA.ROUTINES
	WHERE	ROUTINE_TYPE = 'PROCEDURE'
) AS s,
(
	SELECT	NULL AS 'SPROCs', COUNT(ROUTINE_NAME) AS 'UDFs'
	FROM	INFORMATION_SCHEMA.ROUTINES
	WHERE	ROUTINE_TYPE = 'FUNCTION'
) AS f

How many stored procedures and user defined functions are in the DB?

The stored procedures listed below all accept an incoming table name, then reads data from INFORMATION_SCHEMA to generate code for the specified database table.  Instead of listing the raw SQL, I’m just showing a sample result and allowing you to download the raw SQL sprocs.

Generate C# ADO.NET to VO Stored Procedure

Build custom C# ADO.NET to custom VO fillerup code

download stored procedure

Generate .NET VO class Stored Procedure

download stored procedure

Generate Coldfusion CFFunction with CFQUERY Stored Procedure

Generate Coldfusion CFFunction with CFQUERY for specified table name

download stored procedure

I use these System Views almost daily, and there are a lot of scenarios where my job would take a lot longer to do if I didn’t know about this information.  The INFORMATION_SCHEMA also feeds my passion to build code that makes code, just love it.  Hopefully this nickel tour was enough information to help somebody out.

Instead of boring you with a work scenario, let’s take a simpler one that uses the AdventureWorks database.  This example will create a list of sales people, total order count for each person, and store this list a single table variable to be used as the final data table.

Should be three simple steps right?

1. Create @Table variable

DECLARE @SalesPeople TABLE
(
  EmployeeID int NOT NULL,
  SalesPersonID int NOT NULL,
  FullName varchar(200) NOT NULL,
  Title varchar(200) NOT NULL,
  sales_count int NULL default 0
)

2. INSERT sales people into @Table

-- HACKISH : Match SalesPersonID to EmployeeID, and fill @SalesPeople
INSERT INTO @SalesPeople
( EmployeeID, SalesPersonID, FullName, Title )
SELECT	e.EmployeeID, sp.SalesPersonID,
		c.FirstName + ' ' + c.LastName as FullName,
		e.Title
FROM	Sales.SalesPerson sp,
		HumanResources.Employee e,
		Person.Contact c
WHERE	sp.SalesPersonID = e.EmployeeID
AND		e.ContactID = c.ContactID

3. UPDATE @Table with COUNT

UPDATE	@SalesPeople
SET
	sales_count = COUNT( soh.SalesOrderID )
FROM	@SalesPeople sp, Sales.SalesOrderHeader soh
WHERE EXISTS (
	SELECT DISTINCT SalesPersonID FROM @SalesPeople WHERE SalesPersonID = soh.SalesPersonID
)
AND	sp.SalesPersonID = soh.SalesPersonID

Not COUNT allowed in an UPDATE SET statement

The third step is where the original error comes in, so let’s update this to four steps and see how a Table Variable gets through this.

1 & 2 – Repeat from above

3. Create Table Variable of order counts

SELECT	soh.SalesPersonID, COUNT( soh.SalesOrderID ) AS sales_count
INTO	#SalesOrderCounts
FROM	Sales.SalesOrderHeader soh
WHERE EXISTS (
	SELECT DISTINCT SalesPersonID FROM @SalesPeople WHERE SalesPersonID = soh.SalesPersonID
)
GROUP BY soh.SalesPersonID

4. Update @Table with order counts

UPDATE	@SalesPeople
SET		sales_count = tmp.sales_count
FROM	@SalesPeople sp, #SalesOrderCounts tmp
WHERE	sp.SalesPersonID = tmp.SalesPersonID

And here’s the full script from start to finish with the table variable in use.

-- Master table of sales people
DECLARE @SalesPeople TABLE
(
  EmployeeID int NOT NULL,
  SalesPersonID int NOT NULL,
  FullName varchar(200) NOT NULL,
  Title varchar(200) NOT NULL,
  sales_count int NULL default 0
)

-- Match SalesPersonID to EmployeeID, and fill @SalesPeople
INSERT INTO @SalesPeople
( EmployeeID, SalesPersonID, FullName, Title )
SELECT	e.EmployeeID, sp.SalesPersonID,
		c.FirstName + ' ' + c.LastName as FullName,
		e.Title
FROM	Sales.SalesPerson sp, HumanResources.Employee e, Person.Contact c
WHERE	sp.SalesPersonID = e.EmployeeID
AND		e.ContactID = c.ContactID

-- put sales counts into the other kind of #tableVariable
SELECT	soh.SalesPersonID, COUNT( soh.SalesOrderID ) AS sales_count
INTO	#SalesOrderCounts
FROM	Sales.SalesOrderHeader soh
WHERE EXISTS (
	SELECT DISTINCT SalesPersonID FROM @SalesPeople WHERE SalesPersonID = soh.SalesPersonID
)
GROUP BY soh.SalesPersonID

-- Update our master @table with data from #tableVariable
UPDATE	@SalesPeople
SET		sales_count = tmp.sales_count
FROM	@SalesPeople sp, #SalesOrderCounts tmp
WHERE	sp.SalesPersonID = tmp.SalesPersonID

-- dump the results
SELECT	FullName, Title, sales_count
FROM	@SalesPeople

-- cleanup
drop table #SalesOrderCounts

There is no rhyme or reason here, just some things I felt need to be repeated.

1. PreserveSingleQuotes()

This one came in really handy on a project requiring large text files to be imported into a MySQL database.  I used Coldfusion to upload and read the files into large INSERT chunks using MySQL’s multi – row INSERT syntax. Code built the VALUES portion of the SQL, then I just fed the data into a function for insertion.

INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);

	<cfquery name="insert_data" result="insert_result" DATASOURCE="#request.dsn#" USERNAME="#request.dbuser#" PASSWORD="#request.dbpswd#">
	INSERT INTO table
	( column1, column2, column3, column4, column5, column6, column7 )
	VALUES
	#PreserveSingleQuotes( insert_values )#
	</cfquery>

2. What if GENERATED_KEY doesn’t work?

If you’re using MySQL and the GENERATED_KEY property of your cfquery objects isn’t populating, you can use LAST_INSERT_ID instead.

SELECT LAST_INSERT_ID();

3. Use ArrayAppend when building strings

Classic performance tuning tip for just about any programming language.  Here I’ll give a Coldfusion example and keep it dead simple.  If you ever have to concatenate strings in code, user ArrayAppend instead.  Here are two loops that do the same thing.  If you run this code, you should notice loop1 takes forever, and loop2 is smoking fast.

slow…..

<cfscript>
xx				= 100000;
insertString	= "";

// do the loop
while( xx > 0 ) {
	insertString &= xx & " ";
	xx--;
}

WriteOutput( insertString);
</cfscript>

FAST!

<cfscript>
xx				= 1000000;
insertArray		= ArrayNew(1);
// do the loop
while( xx > 0 ) {
	ArrayAppend( insertArray, xx & " " );
	xx--;
}

WriteOutput( ArrayToList( insertArray, " " ) );
ArrayClear( insertArray );
</cfscript>

4. If CSV, then CHR

This one is simple, if you find yourself creating CSV or any other text file, use special characters when dealing with single and double quotes, etc.

• chr(9) = Tab
• chr(34) = ” double quote
• chr(39) = ‘ single quote

And if you’re not sure of the correct code for the character you’re looking to use, just wrap that character in ASC() and WriteOutput to the page.

5. Use CFMail with GMail

This is a no brainer, but with how difficult sending email can be with other languages, I’m mentioning it here.

Application.cfc

<cfscript>
	APPLICATION.mail.server		= "smtp.gmail.com";
	APPLICATION.mail.port		= "465";
	APPLICATION.mail.ssl		= true;
	APPLICATION.mail.user		= "gmailAccount";
	APPLICATION.mail.pswd		= "gmailPasssword";
</cfscript>

Emailer.cfm

<cfmail to="work@ericfickes.com"
		bcc=""
		from="web@master.com"
		subject="sending mail is easy with Coldfusion"
		server="#application.mail.server#"
		useSSL="#application.mail.ssl#"
		port="#application.mail.port#"
		username="#application.mail.user#"
		password="#application.mail.pswd#">
#emailBody#
</cfmail>

6. CFSCRIPT doesn’t know NULL?

Another tip from the land of importing and exporting data.  While working with query objects in CFScript, for some reason I could never accurately detect for NULL values.  I tried all sorts of detection schemes and ended up just writing a hacky fail safe.  Please, if you have a better suggestion for *easy* NULL detection in CFSCript, add it in the comments below.

utils.cfc

<!---  Simple value getter with try / catch to get around NULL values
This function originated in a script where we always needed a " " even if
the value from the database was null.
--->
<cffunction name="qryGetString" access="public" returntype="string">

<cfargument name="data" type="string">

<cftry>
	<cfscript>
	return #data# & " ";
	</cfscript>

	<cfcatch type="Any">
		<cfreturn " "/>
	</cfcatch>
</cftry>

</cffunction>

Exporter.cfm

// largeish loop
tab = chr(9);
for( xx = 1; xx <= queryObj.RecordCount; xx++ )
{
	// start the row
	this_row = 	utils_cfc.qryGetString( queryObj.FirstName[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.MiddleName[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.LastName[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.Suffix[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.MedicalTitle[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.email[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.practice_phone[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.practice_fax[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.practice_name[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.practice_address1[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.practice_address2[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.practice_city[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.practice_state[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.practice_zipcode[xx] ) & tab &
			utils_cfc.qryGetString( queryObj.hospital_affiliation[xx] ) & tab;

	// do stuff with the data
}

7. How I find list items

Ever notice the different behavior in the Coldfusion ListFind commands?  I ended up writing my own ListHasValue function in order to find exact pattern matching in a list.  I had a list of role ids in a list, and just couldn’t get the built in functions to tell me when my id was in the list without also matching on other ids.  This one makes sense when you run some code.

The top of this sample as my custom ListHasValue() command, and the lower half does three simple loops counting from 1 to 100, and using ListFind, ListContains, and ListHasValue for number checking against the same list.

<!---
	Use this to do an exact pattern check for a value in a list.
	This is useful inside of a loop checking numbers against a number list.

	EX :

	list = "91, 92"

	if you loop from 1 - 100, ListFind and related CF functions will match
	on 1, 2, and 9.  Not just 91 and 92

	Use this when you're looping and you ONLY want to match on 91, or 92

--->
<cffunction name="ListHasValue" access="public" returntype="boolean">

	<cfargument name="list" required="yes" type="string">
	<cfargument name="value" required="yes" type="any">

	<cfscript>
		// clean up to be safe
		list = trim( toString( list ) );

		// check to see if we have a *possible* match
		position = ListContains( list, value ) ;

		if( position > 0 )
		{
			// NOTE : KEEP THE TRIM AND TOSTRING
			found_value = trim( toString( ListGetAt( list, position ) ) );

			if( Compare( value, found_value ) == 0 )
			{
				return true;
			}
		}

		// no match for you!
		return false;
	</cfscript>

</cffunction>

<cfscript>
list	= "1, 11, 21, 31, 41, 51, 61, 71, 81, 91, 99";
xx		= 1;

WriteOutput( "List = " & list & "<hr>");

while( xx < 100 ) {
	if( ListContains( list, xx ) > 0 )
	{
		WriteOutput( "ListContains found " & xx & "<br>" );
	}
    xx++;
}

WriteOutput("<hr />");

xx = 1;
while( xx < 100 ) {
	// ListFind
	if( ListFind( list, xx ) > 0 )
	{
		WriteOutput( "ListFind found " & xx & "<br>" );
	}
    xx++;
}

WriteOutput("<hr />");

xx = 1;
while( xx < 100 ) {
    // Eric's ListHasValue
	if( ListHasValue( list, xx ) )
	{
		WriteOutput( "ListHasValue found " & xx & "<br>" );
	}
	xx++;
}
</cfscript>

If you run this code on your Coldfusion server, you should notice the following results.  ListContains matches single digits from the loop that do not really exist in the list.  ListFind only finds the number 1?  And finally, my function does exactly what I needed it to do.  Tell me when a specific number exists in a list.

Coldfusion ListFind functions don't always behave how I want them to

8. Make PDFs faster

This could easily be it’s own topic, but I’ll say one thing about making PDFs faster with CFDocument.  Only put final content between <cfdocument> and </cfdocument>.  That is, if you have any processing code, cfqueries, cfloops, inside of your cfdocument tag, your cfml page is running slower than it needs to be.  Here’s a simple example of one of my cfml pages that has only final content in the cfdocument tags.

The key to this example is moving all of my content creation code into an external file, then including at the top of my page. I always do a check for my main PDF_BODY variable, and then spit out my PDF document.

<cfinclude template="code/export_pdf_codefile.cfm">

<cfif PDF_BODY NEQ "">

    <cfdocument	name="provider_profile"
                format="PDF"
                pagetype="A4"
                mimetype="application/pdf"
                orientation="portrait"
                margintop="0"
                marginbottom="0.2"
                marginleft="0.2"
                marginright="0.2"
                >
        <cfoutput>#PDF_BODY#</cfoutput>
    </cfdocument>

    <!--- send directly to client --->
    <cfheader name="Content-Disposition" value="attachment; filename=#filename#">
    <cfcontent type="application/pdf" variable="#provider_profile#">

<cfelse>
	No PDF content found
</cfif>

9. Use parameterized queries

This is a tip for all server side programmers whether you use Coldfusion, ASP, JSP, PBJ.  Use parameterized queries when doing any database interaction.  It’s too easy not to use, and you get protection from SQL Injection, as well as enforcing proper data types when speaking to your database.  This is something all server programmers should do regardless of your language, the sample below is for Coldfusion.

BAD

<cfquery name="tblInsert" datasource="myDb">
INSERT INTO	myTable
( col1, col2, col3, col4 )
VALUES
( '#Form.field1#', '#Form.field2#', '#Form.field3#' )
</cfquery>

GOOD

<cfset val1 = Form.field1>
<cfset val2 = Form.field1>
<cfset val3 = Form.field1>

<cfquery name="tblInsert" datasource="myDb">
INSERT INTO	myTable
( col1, col2, col3, col4 )
VALUES
(
    <cfqueryparam cfsqltype="cf_sql_varchar" value="#val1#" />,
    <cfqueryparam cfsqltype="cf_sql_varchar" value="#val2#" />,
    <cfqueryparam cfsqltype="cf_sql_varchar" value="#val3#" />
)
</cfquery>

10. Where’d the time go?

For the good programmers already using parameterized queries, ever insert a timestamp into your database and find out the date is correct, but the time is always 12:00:00?  Take a closer look at the cfsqltype in your cfqueryparam, I had this exact problem and here’s what happened.

Using cf_sql_date does not include the full date and timestamp, just the date.

<cfqueryparam cfsqltype="cf_sql_date" value="#paymentDate#" />

Using cf_sql_timestamp includes the full date and timestamp I was looking for.

<cfqueryparam cfsqltype="cf_sql_timestamp" value="#paymentDate#" />

11. Stored Procedures are a little different

This last one isn’t much of a tip, but more of a reminder to myself.  I do so much database work that stored procedures are just queries to me, but not so to Coldfusion and the CFQuery tag.  If you want to get data from a stored procedure, you need to use the CFStoredproc tag.  Here’s a sample of passing one argument into a stored procedure, and how to get the resulting data.

<cfstoredproc	datasource="myDb" procedure="GetDataFaster">

	<cfprocparam type="in" cfsqltype="cf_sql_integer" value="#inputVar#" />

	<!--- specify sproc result here, cfstoredproc res != returned recordset --->
	<cfprocresult name = sprocResult>

</cfstoredproc>

<cfreturn #sprocResult.ColumnFromQuery#>

Here the contents of my batch file.

### DELETE TEMP FILES
del /q /f /s Z:\_FavoriteClient\_GIT_\webapp\invoices\* /f /s

### RESTORE DATABASE
sqlcmd -S "localhost" -i Z:\_FavoriteClient\_GIT_\sql\restoreBackup.sql

To use this in a .bat file, copy and paste the text into Notepad, then save as “restore.bat”.  Remember, you will need to wrap your filename.bat with quotes, otherwise Notepad will save the file as filename.bat.txt.

* The contents of restoreBackup.sql is listed below.

The web application I’m working on is a payment processor made up of five steps.  In order to test I have to setup multiple sales for multiple clients, then log in as three different admin users to push the transactions through the system.  This gets my test transactions to the proper testable state.  The process of setting up testable transactions takes over 30 clicks, and once I hit step 3 of the wizard, my test transactions are completed in a way that I have to re-setup the test data ( 20 GOTO 10 ).

I thought about a handful of options, and ended up going with this solution.

1. Setup all test transactions in web application by hand ( get the data ready )
2. Take a full database backup ( freeze the data )
3. Use TSQL script to drop all connections to my app’s db, then restore the database to the “testable” state from step 1 ( reset the DB )

Here is my TSQL script

DECLARE @sessID int,
		@dbName varchar(50),
		@userName varchar(50),
		@backupFile varchar(200)

SET @dbName		= 'DA413'	-- your database name
SET @userName	= 'DA413'	-- sql user account to look for
SET @backupFile = 'D:\DB\Backup\DA413.bak' -- path to SQL backup file

-- use a cursor to store all session_ids
DECLARE session_cursor CURSOR
FOR
	SELECT	session_id
	FROM	sys.dm_exec_sessions
	WHERE	original_login_name = @userName

	-- open cursor and grab first row
	OPEN session_cursor
	FETCH NEXT FROM session_cursor INTO @sessID

	-- loop through session_ids
	WHILE @@FETCH_STATUS = 0
	BEGIN

		-- kill it
		-- using EXEC because the sproc kill does not like @variables
		EXEC('kill ' + @sessID)

		-- get the next session_id
		FETCH NEXT FROM session_cursor INTO @sessID
	END

-- cursor cleanup
CLOSE session_cursor
DEALLOCATE session_cursor

-- restore backup
USE master
	RESTORE DATABASE DA413
	FROM DISK = @backupFile
GO

A few notes about this script :

• I only need to disconnect my web application’s database user, not any user
• I’m using the latest full backup, and not a specific database snapshot
• The SPROC kill doesn’t like @variables as input, so use EXEC
• Your web server doesn’t know the user was disconnected, so you’ll have to log back into your application.
• If you use this technique, be sure to RERUN your backup if you add anything to the database ( EX : new table column, stored procedure, etc )
• If you’re not the only person connected to this database, make sure you don’t disconnect anybody else using the same database name

This solution is perfect for me because I have full control over my code, database, and server.  It’s also great because I can test my application, run a single sql script, and 10 seconds later I can test my application again.  While this solution is perfect for me, it’s probably best used as reference for others.  However, this technique of rolling back the database could be applied to any software application using SQL Server for it’s datasource.

Hope this helps somebody.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

-- init vars
DECLARE @sessID int,
		@dbName varchar(50),
		@userName varchar(50)
SET @dbName	= 'DA413'	-- your database name
SET @userName	= 'DA413'	-- sql user account to look for
-- use a cursor to store all session_ids
DECLARE session_cursor CURSOR
FOR
	SELECT	session_id
	FROM	sys.dm_exec_sessions
	WHERE	original_login_name = @userName
	-- open cursor and grab first row
	OPEN session_cursor
	FETCH NEXT FROM session_cursor INTO @sessID
	-- loop through session_ids
	WHILE @@FETCH_STATUS = 0
	BEGIN
		-- kill it
		-- using EXEC because the sproc kill does not like @variables
		EXEC('kill ' + @sessID)
		-- get the next session_id
		FETCH NEXT FROM session_cursor INTO @sessID
	END
-- cursor cleanup
CLOSE session_cursor
DEALLOCATE session_cursor

Permalink

| Leave a comment

1
2
3
4
5
6
7
8
9
10

-- Give me all tables ending with an UPPERCASE X

SELECT	table_name, SUBSTRING( table_name, LEN(table_name), 1 ) as 'end'

FROM	INFORMATION_SCHEMA.TABLES

-- SELECT ascii('X') = 88
-- SELECT ascii('x') = 120
WHERE	ASCII( SUBSTRING( table_name, LEN(table_name), 1 ) ) = 88

Permalink

| Leave a comment

At first I thought no sweat.  My dataset is still small ( only 2000 rows ), so if we want uber control I could do the math and generate my id lists by hand.  Yes, hand crafting is possible and under a deadline that kind of logic almost makes sense.  However, I already know the table I’m working with will grow in the future, and I’ll probably have to do this data update again, so why not do this right?  While playing around with different select statements I had a “EUREKA!” moment.  SQL Server’s TOP operator supports PERCENT, not just number.  I couldn’t believe it.  I use TOP at least once a week and I always forget about TOP PERCENT.  Since I already know how to select random rows via CTE, it was time to put it all together.

Before giving you the final SQL, here are the important parts to be familiar with.  Also, for the sake of example I’m using the AdventureWorks database so you can play along at home.

TOP PERCENT

If you just need 50% of the rows in a table, but you’re not concerned about the sequence returned, you can fire this query.  This will give you a sequential listing of ProductIDs

SELECT TOP 50 PERCENT ProductID
FROM Production.Product
ORDER BY ProductID

Which will look something like this.

COMMON TABLE EXPRESSION

Now let’s say you want to randomly pull all rows from a table.  This can be achieved using this CTE.

WITH data( ProductID ) AS (
	SELECT	ProductID
	FROM	Production.Product
)
SELECT	ProductID
FROM	data
ORDER BY NEWID()

Which will look like this

If you’re looking to randomly select values from a pre-determined list, see my CTE sample here.

So now that you’ve seen TOP PERCENT and CTE in action, it’s time to put these together and solve my initial task of creating randomly selected groups of ids, of a percent size.

RANDOMLY SELECT TOP PERCENT

Putting it all together, here is the query I used to create my first status group.

WITH data( ProductID ) AS (
	SELECT	ProductID
	FROM	Production.Product
)
SELECT TOP 20 PERCENT ProductID
FROM	data
ORDER BY NEWID()

Which gives me a dataset that is 20% of all rows in Production.Product, and the ids are in random order.

And there you have it. Randomly selecting a percent sized data set from a table in SQL Server. The SQL here is really pretty simple, but for some reason I always forget TOP PERCENT. I’m hoping this post will help me remember TOP PERCENT, and maybe even help somebody else with some TSQL.

The base concept is using your knowledge of SQL’s UNION operator to add a temp value to the beginning of a list of data from a sql query.  In the past I’ve done this countless times via code, and recently I didn’t have the time to do this, so I updated the query to a UNION, and I was good to go.

Now I’m not selling this as a ‘best practice’, but I do consider this one more reason why it’s good to know SQL.

Problem : Using a SQLDataSource to populate a DropDown component, how do you inject a spacer value in position 0? EX : “- select value -”

Solution : Inject your spacer value in your SelectCommand via sql’s UNION operator

ComboBox

<asp:DropDownList runat="server" ID="meter_manufacturer_dd" DataSourceID="sql_meterManufacturer" DataTextField="Manufacturer" />

DataSource

<asp:SqlDataSource runat="server" ID="sql_meterManufacturer"
    SelectCommand="
    SELECT '- Choose Manufacturer -' as Manufacturer
    UNION
    SELECT DISTINCT Manufacturer FROM Smart_Meter_DEF"
    />

What this solution gets you.

1. Your spacer value shows up in position 0 ( because the first character is – and not alphanumeric )

2. Auto ViewState caching ( EG : going straight .NET solution, .NET handles persisting your dropdown selection between postbacks )

* in this sample, the connectionString for the SqlDataSource is set in code.

I had this need on an app recently and built a little dynamic sql action that does just this.  The example below however, uses the the DatabaseLog table in the AdventureWorks sample database to return a count of Events logged for each Schema.  Before jumping into the PIVOT, here’s a simple query that gives you the same information, all Schemas, Events, and Event counts.

SELECT      [Schema], [Event], COUNT( [Event] ) AS 'event_count'
FROM        DatabaseLog
GROUP BY    [Schema], [Event]
ORDER BY    [Schema]

Running this query should give you a long result looking something like this.

While this query returns the same information to you, I don’t like this format as much as using PIVOT.  This query result is long and requires a bit of manipulation to get into a readable format.

Now let’s have a look at retrieving the same information using the PIVOT function.

/*
Example of a dynamic PIVOT against a varchar column from the Adventureworks database

References :
PIVOT & UNPIVOT function

http://msdn.microsoft.com/en-us/library/ms177410.aspx

AdventureWorks sample Databases

http://msdn.microsoft.com/en-us/library/ms124501(v=SQL.100).aspx

AdventreWorks.DatabaseLog

http://msdn.microsoft.com/en-us/library/ms124872.aspx

*/

USE AdventureWorks

-- populate temp Event table
SELECT DISTINCT [Event] as 'Event'
INTO	#events
FROM	DatabaseLog

-- this var will hold a comma delimited list of [Event]
DECLARE	@eventList nvarchar(max)

-- create a flattened [Event], list for the PIVOT statement
SELECT	@eventList = COALESCE( @eventList + ', ', '') + CAST( QUOTENAME( [Event] ) AS VARCHAR(1000) )
FROM	#events
ORDER BY [Event]

-- drop table var since our data now lives in @eventList
DROP TABLE #events

-- this var will hold the dynamic PIVOT sql
DECLARE @pvt_sql nvarchar(max)

-- NOTE : we're using dynamic sql here because PIVOT
-- does not support sub SELECT in the 'FOR Event IN ( )'
-- part of the query.
-- If we don't use dynamic SQL here, the PIVOT function
-- requires you to hard code each 'Event'
-- Using SELECT * here so the [Event] columns are auto included
SET @pvt_sql = 'SELECT	*
                FROM
                (
                    SELECT	[Event], [Schema]
                    FROM	DatabaseLog
                ) AS data
                PIVOT
                (
                    COUNT( Event )
                    FOR Event IN
                    ( ' + @eventList + ' )
                ) AS pvt'

-- run the query
EXEC sp_executesql @pvt_sql

Assuming you have the AdventureWorks database installed on your server, running this sql should give you a result looking something like this.

Show all Schemas and count of each Event type

This query result was truncated to fit in this post, but just know the query above creates a column for every Event in the Databaselog table.

A quick explanation of what’s happening in this sql

1. First you fill a table variable ( #events ) with all Events from DatabaseLog
2. Next create a comma delimited list of the Events inside of the table variable
3. Drop the table variable now that we’ve got our delimited list of Events
4. Build the PIVOT statement as a string so you can inject the Events list
5. Fire the dynamic SQL via EXEC

Dynamic SQL is something that comes in handy from time to time, but I do my best to only use it if I absolutely have to.  In this case we’re using it because the PIVOT function does not allow sub SELECT statements.  This is also why we create a specially formatted delimited list of Events prior to building the dynamic sql.

So there you have it, one example of using PIVOT against a varchar column instead of an integer column.  Also, this is a pretty good example of a dynamic PIVOT since it’s pretty simple.  I hope this makes sense, and if you have any suggestions of better techniques, I’d love to hear it.