All of the RDMS systems I’ve worked with have built in System Views ( or tables ) that contain data about the structure and contents of your database.  These views are there to help you with non data tasks.  For the sake of this post we’re only going mention the INFORMATION_SCHEMA, which is the system view I use most in SQL Server, and MySQL.  For your reference, here are links to INFORMATION_SCHEMA docs for four popular RDMS systems.

1. SQL Server – http://msdn.microsoft.com/en-us/library/ms186778.aspx
2. MySQL – http://dev.mysql.com/doc/refman/5.5/en/information-schema.html
3. Oracle – http://download.oracle.com/docs/cd/E17952_01/refman-5.0-en/information-schema.html
4. DB2 – http://www.tar.hu/sqlbible/sqlbible0100.html

If you work with SQL Server, you’ll be happy to know you can see all of the System Views in Management Studio by default.

System Views in the AdventureWorks database

If you work with MySQL, you can see the INFORMATION_SCHEMA database in MySQL Workbench, but you’ll have to enable it in Preferences first.

Edit > Preferences > SQL Editor > Check 'Show Metadata Schemata'

Now for the MySQL users, turning on the Metadata Schemata is not required in order to query those system tables.  Assuming your user has appropriate permissions, you can always fire up Workbench and fire a query like this.

SELECT *
FROM INFORMATION_SCHEMA.Columns
WHERE Column_Name LIKE '%user%'

MySQL has INFORMATION_SCHEMA tables too!

Now that you know about INFORMATION_SCHEMA, you should be happy to know this is part of the SQL-92 standard. Which means whatever sql scripts you write in MySQL, will most likely work with SQL Server as well. So go ahead and start querying your database, you’ll probably find uses for the INFORMATION_SCHEMA in your daily life really quickly.

I’m going to assume you’ve got the general idea here, so here are a few sql scripts that I’ve used over the years.  The first three are informational queries, and the last three are stored procedures that generate .NET or Coldfusion code based off of a table name.

Find all tables with the column EmployeeID

SELECT	*
FROM	INFORMATION_SCHEMA.COLUMNS
WHERE	COLUMN_NAME LIKE 'EmployeeID'

Which tables have an EmployeeID column?

How many tables have the word employee in the name

SELECT	*
FROM	INFORMATION_SCHEMA.TABLES
WHERE	LOWER(TABLE_NAME) LIKE '%employee%'

How many %employee% tables are in the DB?

How many SPROCs and UDFs does our database contain?

SELECT	s.SPROCs, f.UDFs
FROM
(
	SELECT	COUNT(ROUTINE_NAME) AS 'SPROCs', NULL AS 'UDFs'
	FROM	INFORMATION_SCHEMA.ROUTINES
	WHERE	ROUTINE_TYPE = 'PROCEDURE'
) AS s,
(
	SELECT	NULL AS 'SPROCs', COUNT(ROUTINE_NAME) AS 'UDFs'
	FROM	INFORMATION_SCHEMA.ROUTINES
	WHERE	ROUTINE_TYPE = 'FUNCTION'
) AS f

How many stored procedures and user defined functions are in the DB?

The stored procedures listed below all accept an incoming table name, then reads data from INFORMATION_SCHEMA to generate code for the specified database table.  Instead of listing the raw SQL, I’m just showing a sample result and allowing you to download the raw SQL sprocs.

Generate C# ADO.NET to VO Stored Procedure

Build custom C# ADO.NET to custom VO fillerup code

download stored procedure

Generate .NET VO class Stored Procedure

download stored procedure

Generate Coldfusion CFFunction with CFQUERY Stored Procedure

Generate Coldfusion CFFunction with CFQUERY for specified table name

download stored procedure

I use these System Views almost daily, and there are a lot of scenarios where my job would take a lot longer to do if I didn’t know about this information.  The INFORMATION_SCHEMA also feeds my passion to build code that makes code, just love it.  Hopefully this nickel tour was enough information to help somebody out.

Instead of boring you with a work scenario, let’s take a simpler one that uses the AdventureWorks database.  This example will create a list of sales people, total order count for each person, and store this list a single table variable to be used as the final data table.

Should be three simple steps right?

1. Create @Table variable

DECLARE @SalesPeople TABLE
(
  EmployeeID int NOT NULL,
  SalesPersonID int NOT NULL,
  FullName varchar(200) NOT NULL,
  Title varchar(200) NOT NULL,
  sales_count int NULL default 0
)

2. INSERT sales people into @Table

-- HACKISH : Match SalesPersonID to EmployeeID, and fill @SalesPeople
INSERT INTO @SalesPeople
( EmployeeID, SalesPersonID, FullName, Title )
SELECT	e.EmployeeID, sp.SalesPersonID,
		c.FirstName + ' ' + c.LastName as FullName,
		e.Title
FROM	Sales.SalesPerson sp,
		HumanResources.Employee e,
		Person.Contact c
WHERE	sp.SalesPersonID = e.EmployeeID
AND		e.ContactID = c.ContactID

3. UPDATE @Table with COUNT

UPDATE	@SalesPeople
SET
	sales_count = COUNT( soh.SalesOrderID )
FROM	@SalesPeople sp, Sales.SalesOrderHeader soh
WHERE EXISTS (
	SELECT DISTINCT SalesPersonID FROM @SalesPeople WHERE SalesPersonID = soh.SalesPersonID
)
AND	sp.SalesPersonID = soh.SalesPersonID

Not COUNT allowed in an UPDATE SET statement

The third step is where the original error comes in, so let’s update this to four steps and see how a Table Variable gets through this.

1 & 2 – Repeat from above

3. Create Table Variable of order counts

SELECT	soh.SalesPersonID, COUNT( soh.SalesOrderID ) AS sales_count
INTO	#SalesOrderCounts
FROM	Sales.SalesOrderHeader soh
WHERE EXISTS (
	SELECT DISTINCT SalesPersonID FROM @SalesPeople WHERE SalesPersonID = soh.SalesPersonID
)
GROUP BY soh.SalesPersonID

4. Update @Table with order counts

UPDATE	@SalesPeople
SET		sales_count = tmp.sales_count
FROM	@SalesPeople sp, #SalesOrderCounts tmp
WHERE	sp.SalesPersonID = tmp.SalesPersonID

And here’s the full script from start to finish with the table variable in use.

-- Master table of sales people
DECLARE @SalesPeople TABLE
(
  EmployeeID int NOT NULL,
  SalesPersonID int NOT NULL,
  FullName varchar(200) NOT NULL,
  Title varchar(200) NOT NULL,
  sales_count int NULL default 0
)

-- Match SalesPersonID to EmployeeID, and fill @SalesPeople
INSERT INTO @SalesPeople
( EmployeeID, SalesPersonID, FullName, Title )
SELECT	e.EmployeeID, sp.SalesPersonID,
		c.FirstName + ' ' + c.LastName as FullName,
		e.Title
FROM	Sales.SalesPerson sp, HumanResources.Employee e, Person.Contact c
WHERE	sp.SalesPersonID = e.EmployeeID
AND		e.ContactID = c.ContactID

-- put sales counts into the other kind of #tableVariable
SELECT	soh.SalesPersonID, COUNT( soh.SalesOrderID ) AS sales_count
INTO	#SalesOrderCounts
FROM	Sales.SalesOrderHeader soh
WHERE EXISTS (
	SELECT DISTINCT SalesPersonID FROM @SalesPeople WHERE SalesPersonID = soh.SalesPersonID
)
GROUP BY soh.SalesPersonID

-- Update our master @table with data from #tableVariable
UPDATE	@SalesPeople
SET		sales_count = tmp.sales_count
FROM	@SalesPeople sp, #SalesOrderCounts tmp
WHERE	sp.SalesPersonID = tmp.SalesPersonID

-- dump the results
SELECT	FullName, Title, sales_count
FROM	@SalesPeople

-- cleanup
drop table #SalesOrderCounts

Let’s take a simple example.  Say you’ve got a comment form that you want to auto close, or reload after the form was posted.  Below is a simple single file style asp.net page with a simple javascript function that reloads this page.

<%@ Page Language="C#" %>

<script runat="server">
/// <summary>
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
protected void Page_Load(object sender, EventArgs e)
{
    if (!IsPostBack)
    {

    }
}

///////////////////////////////////////////////////////////////////////////////
/// Do stuff with the form data, then refresh page using javascript
protected void submitComments(object sender, EventArgs e)
{

    try
    {
	//
	// do stuff here
	//

	// set javascript timer to reload page afer 3 seconds
	js_target.Text = "setTimeout('reload()', 3000);";

    }
    catch (Exception exc)
    {
        Response.Write( "ERROR : " + exc.Message );
    }
}
</script>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head>
	<title>Comments</title>
	<script type="text/javascript">
	// page reload helper
	function reload() {
		document.location.replace( document.location );
	}
	</script>
</head>
<body>

<form id="form1" runat="server" method="post">

<script type="text/javascript"><asp:Literal runat="server" id="js_target" /></script>

	Comments
	<asp:TextBox runat="server" ID="comment_box" Width="200" />
	<br><br>

	Your name
	<asp:TextBox runat="server" ID="fullname" Width="200" />
	<br><br>
	<asp:Button runat="server" ID="submit_btn" onclick="submitComment" Text="submit" />

</form>
</body>
</html>

If you look just under the form tag you’ll see the key to this technique, an asp literal wrapped by an open and close script tag.

<script type="text/javascript"><asp:Literal runat="server" id="js_target" /></script>

When you load your page and view the source you’ll just see an empty script tag, so it shouldn’t interfere with the execution or rendering of your page.

The last part of this technique is simple, in your server code just set your Literal control’s .Text value to your javascript code. In this case when I post my comment form, after handling the input data I display a thank you message, then set some javascript to reload the page.

ltl_js.Text = "setTimeout('reload()', 3000);";

That’s all there is to it. Drop a literal in an empty script block and BAM!, you have an easy way to add javascript to your asp.net page.

Here the contents of my batch file.

### DELETE TEMP FILES
del /q /f /s Z:\_FavoriteClient\_GIT_\webapp\invoices\* /f /s

### RESTORE DATABASE
sqlcmd -S "localhost" -i Z:\_FavoriteClient\_GIT_\sql\restoreBackup.sql

To use this in a .bat file, copy and paste the text into Notepad, then save as “restore.bat”.  Remember, you will need to wrap your filename.bat with quotes, otherwise Notepad will save the file as filename.bat.txt.

* The contents of restoreBackup.sql is listed below.

The web application I’m working on is a payment processor made up of five steps.  In order to test I have to setup multiple sales for multiple clients, then log in as three different admin users to push the transactions through the system.  This gets my test transactions to the proper testable state.  The process of setting up testable transactions takes over 30 clicks, and once I hit step 3 of the wizard, my test transactions are completed in a way that I have to re-setup the test data ( 20 GOTO 10 ).

I thought about a handful of options, and ended up going with this solution.

1. Setup all test transactions in web application by hand ( get the data ready )
2. Take a full database backup ( freeze the data )
3. Use TSQL script to drop all connections to my app’s db, then restore the database to the “testable” state from step 1 ( reset the DB )

Here is my TSQL script

DECLARE @sessID int,
		@dbName varchar(50),
		@userName varchar(50),
		@backupFile varchar(200)

SET @dbName		= 'DA413'	-- your database name
SET @userName	= 'DA413'	-- sql user account to look for
SET @backupFile = 'D:\DB\Backup\DA413.bak' -- path to SQL backup file

-- use a cursor to store all session_ids
DECLARE session_cursor CURSOR
FOR
	SELECT	session_id
	FROM	sys.dm_exec_sessions
	WHERE	original_login_name = @userName

	-- open cursor and grab first row
	OPEN session_cursor
	FETCH NEXT FROM session_cursor INTO @sessID

	-- loop through session_ids
	WHILE @@FETCH_STATUS = 0
	BEGIN

		-- kill it
		-- using EXEC because the sproc kill does not like @variables
		EXEC('kill ' + @sessID)

		-- get the next session_id
		FETCH NEXT FROM session_cursor INTO @sessID
	END

-- cursor cleanup
CLOSE session_cursor
DEALLOCATE session_cursor

-- restore backup
USE master
	RESTORE DATABASE DA413
	FROM DISK = @backupFile
GO

A few notes about this script :

• I only need to disconnect my web application’s database user, not any user
• I’m using the latest full backup, and not a specific database snapshot
• The SPROC kill doesn’t like @variables as input, so use EXEC
• Your web server doesn’t know the user was disconnected, so you’ll have to log back into your application.
• If you use this technique, be sure to RERUN your backup if you add anything to the database ( EX : new table column, stored procedure, etc )
• If you’re not the only person connected to this database, make sure you don’t disconnect anybody else using the same database name

This solution is perfect for me because I have full control over my code, database, and server.  It’s also great because I can test my application, run a single sql script, and 10 seconds later I can test my application again.  While this solution is perfect for me, it’s probably best used as reference for others.  However, this technique of rolling back the database could be applied to any software application using SQL Server for it’s datasource.

Hope this helps somebody.

I ran into this while building a PayPal payment processor, and wasted too much time thinking this was a username and password issue, or possibly a problem with the host name I was using, but it wasn’t.  It was extra frustrating because my test PayPal code worked just fine if I created an html form and posted directly to PayPal.  Also, the same code worked perfectly in production, just not on my development server.

After hours of googling and trial and error I found my solution thanks to Steve Erat.  Basically Coldfusion 7 isn’t able to handshake securely with PayPal ( Verisign ) because the server doesn’t have PayPal’s certificate in the key store.  You can read in depth about what’s happening on Steve’s blog post here.

So my solution ended up being this

1. Download the verisign.cer certificate from my production server
2. Import verisign.cer into the JDK that comes with Coldfusion
3. Restart Coldfusion and IIS

I tried all sorts of things from installing every windows certificate management features, upgrading my JDK and having Coldfusion use that, apply all the latest service packs and updates to my server, changing any paypal setting I can think of.  All I needed to do was this.

1. Drop to CMD prompt
2. Run the keytool that came installed with Coldfusion
   “C:\CFusionMX7\runtime\jre\bin\keytool -import -trustcacerts -keystore C:\CFusionMX7\runtime\jre\lib\security\cacerts -storepass changeit -noprompt -alias PayPal -file c:\verisign.cer”

I couldn’t find this solution anywhere else online besides Steve’s site, so this is a rebroadcast in case somebody else hits this issue.  Also, at the time of writing, the test payment hostname for PayPal is pilot-payflowpro.paypal.com.

Test Credit Card Account Numbers

While testing, use only the credit card numbers listed here. Other numbers produce an error.

Expiration Date

must be a valid date in the future (use the mmyy format).

Test Credit Card Account Numbers

Permalink

| Leave a comment

Endless Mural wins FWA for HTML5

I’m ecstatic to announce the Endless Mural HTML5 project has won the prestigious FWA Site of the day award.  While this isn’t the first project I’ve worked on that has won the FWA, it is the first NON-Flash, HTML5 and ASP.NET project that has.  No Flash, and we still got the FWA site of the day, SWEET!.

I need a SQL Ninja

On Friday July 23rd I got the chance to skateboard with my good buddy and personal hero Joshua Davis. I feel lucky being able to say we’ve actually been skating together for a few years now, but this was certainly my favorite session we’ve had so far. We started out at Broomfield’s new park because I had to show Josh the new mini bowl.

Two hot dogs ( http://yfrog.com/n3hs5j )

After Broomfield we made a quick stop for lunch, and then on to the Denver Skatepark in downtown.  It was great showing Josh the lines at my hometown skateparks, as well as a few radical maneuvers.

Now that Josh had seen my non-frontside airs, it was time to wrap things up.  As we were saying our goodbyes I decided to ask about work.  Normally I don’t talk about work at the skatepark, but I was thinking about going indie again, and figured what the heck.

Turns out Josh was about to start a project for Microsoft ( WHAT?!?! ) and he need to “find a SQL Ninja” ( DOUBLE WHAT?!?! ).  I’ve been actively working with MS SQL Server since version 6.5, so I let him know he was looking at his sql ninja.  Josh was interested, but gave me the “just cause we’re bros, doesn’t put you on the team”.

HTML5 drawing tool in one night

Driving home from the skatepark I called my wife super giddy.  ”Honey, I may be going indie sooner than we planned”.  I gave her the rundown of the potential project Josh and I just spoke about, and let her know I had some homework to do.  That night I went home and built out a distant cousin of the endless mural project.

HTML5 drawing tool powered by ASPX and MySQL

HTML5 drawing tool powered by ASPX and MySQL

EFDRAW is a really simple HTML5 drawing tool powered by ASP.NET and MySQL. It has most of the features of the mural ( draw, save, replay, share ), but this was only a proof of concept.  This was my first dive into HTML5 development, and it’s pretty sweet.

If you’re interested in HTML5 drawing tools, feel free to play around with EFDRAW, view source, help yourself.

There’s one thing…. Azure

A few days after building EFDRAW my band The Compilers played at Ignite Denver 7. Right before starting our first set my phone rings and it’s Josh! OMG I think, this is either the “you got it” or the “sorry bud, we’ll skate again” call. I decide to take the call even though we were locked and loaded, standing on stage with our gear waiting for the house music to go down. I answer and it’s Josh, but not the usual hyperactive Josh I’m accustomed to. I ask about the gig and he says “Well, there’s one thing. We have to use Azure”.

I let him know I’ve worked with other cloud platforms already, just not Microsoft’s. So we talk a little more and Josh passes the phone to Branden so we can talk 0s and 1s. After talking to Branden “mega brain” Hall for a few minutes he asks if I can do this. I tell him yes, he says yes, I get excited, he gets excited. Branden passed me back to Josh and I’m in shock at this point.

Now the boring stuff

So that’s the story of how I landed the Endless Mural gig, now the boring technical details.

The drawing portion of the mural was built by Branden Hall of Automata Studios.  I did the backend which is made up of ASP.NET ( C# ), SQL Azure, and Windows Azure.  We’re using Azure blob storage to save and serve up the PNGs created at the mural.  To access SQL server, I wrote a super lightweight data access library using all native .NET.

For the most part the backend was very much like every other .NET SQL Server project I build, but Azure did introduce a few gotchas.

1. The publishing and management of your cloud site mainly goes through http://windows.azure.com/.
2. You can deploy your site from Visual Studio which proved to be immensely helpful after my Azure deploy package grew beyond 100 MB.
3. You can access SQL Azure directly from SQL 2008+ management tools.
4. You can not FTP single files up to the cloud, only the full ball of wax.
5. You can still use web.config for configuration storage, but Azure also has it’s own version of web.config.
6. If you need to edit your settings after deploying, store those settings in your Azure service config, not web.config
7. SQL Azure requires all tables to use clustered indexes
8. SQL Azure has it’s own TSQL restrictions ( not many, but be aware )
9. On average, doing a full republish of an Azure site took a full hour.

I could probably ramble on and on about Azure, but I’ll cut it short.  If you happen to have any questions about Azure feel free to hit me up or leave a comment.  I would also like to say that I know Microsoft is and has been actively improving Azure by the day.  The state of Azure today is most likely even better than when we built the mural, so my experiences may not be your own.

The toolbox

• Windows Azure SDK
• Windows Azure Platform Kit June 2010
• Windows Azure Tools for Visual Studio ( v1.2 )
• Microsoft Seadragon Ajax library
• Microsoft SQL Server 2008 R2
• Microsoft SQL Azure
• ASP.NET 4 ( C# )
• Windows Azure
• Azure Storage Explorer

Here is the toolbox that Branden used on the client side.

Hotlinks from the server guy

• Developing and Deploying with SQL Azure
• SQL Azure DB T-SQL reference
• Azure w/Intellitrace
• Using Windows Azure Page Blobs .. how to upload / download blobs
• Windows Azure Deep Dive: Working with Configuration
• Seadragon Ajax – Getting Started
• Creating Content : Deep Zoom Tools
• Deep Zoom Blog
• Twitter TWEET Button

It’s a wrap

This project was the most concentrated five weeks I’ve had in quite some time. I still wonder if we were only given five weeks because this was an HTML5 project. Either way, the mural team made some magic and now you can too. If you’re like me and just want to doodle, go make some art at the mural. If you’re a developer interested in HTML5 and Javascript programming, go check out the javascript library okapi.js which Branden Hall recently open sourced.

Also be sure to visit the magicians, I mean artists, who made the amazing patterns you see when using the mural. I’m a life long doodler, but can’t art myself out of a paper bag.

Evgeny Kiselev

Evgeny Kiselev - www.ekiselev.com

Guilherme Marconi

Guilherme Marconi - brain.marconi.nu

Joshua Davis

Joshua Davis - www.joshuadavis.com

Matt Lyon

Matt Lyon - www.c8six.com

And lastly I put up a photo album on Facebook of all my camera phone pictures from the trip.  Check out the endlessmural photo album.

Go make art at endlessmural.com

Endlessmural.com is a generative drawing tool written in HTML5, Javascript, CSS3, on top of a Microsoft Azure backend.  Go, make art, share the url.

Here is a piece I made today.

ScreenFlow

Price : $99.00

URL : http://www.telestream.net/screen-flow/

Notes : ScreenFlow looks freaking amazing.  If I had to do regular screencasting, I would spend the $99 to get this application.  It appears to do, save, and export everything.  This is the best solution for any professional screencaster.

Snapz Pro X

Price : $69.00

URL : http://www.ambrosiasw.com/utilities/snapzprox/

Notes : If I was going to pay money for a screencast app, it would be ScreenFlow.  However, I do use Ambrosia Soft’s Wiretap Studio, so I know they make great software.  If $30 is enough to affect your decision, I would highly recommend looking at this before ScreenFlow.

Jing Free

Price : free

URL : http://www.jingproject.com/features/

Notes : Jing Free is pretty sweet, but only captures a maximum of five minutes of video.  For my assignment that ended up being good so I could keep my videos short.  The pay service looks really helpful for sharing captures ( images, videos, etc ), but I just can’t justify signing up for this yet.

Silverback App

Price : $69.95

URL : http://silverbackapp.com/

Notes : This capturing app is geared towards taking videos for usability studies.  This is a really cool application that I’d take a closer look if I wasn’t such a server side programmer.  Be sure to watch the demo, it’s a really cool application, just not good for what I was doing.

Screentoaster.com

Price : FREE

URL : http://www.screentoaster.com

Notes : This is a free web based screen recording service that I use fairly often.  You can’t beat free, and this will run on your MAC or PC since it’s browser based.  Definitely worth a look if you need something quick and want to sharing capabilities.

This is great, but what happens when you have a mixed bag of clients that all need to upload to the same location?  Sometimes I work with completely ASP.NET or CF web apps, but more often than not I’m dealing with Flash clients as well as HTML clients.

Recently I ran into this upload scenario and built this simple ASP.NET uploader script.  This feels a bit old school since it uses .NET’s built in Request.Files collection, instead of a fancy new ‘all in one’ server control, but I actually prefer this method.

Here’s all you need :

// Check for posted files
for (int xx = 0; xx < Request.Files.Count; xx++)
{
    // UPLOAD FILE
    HttpPostedFile _file = Request.Files[xx];

    // make sure we're not finding empty filename
    if (_file.FileName.Trim() != string.Empty)
    {
        // NOTE : IE < 8 reports full path of file, not just filename
        // Parse out filename, then create full upload path
        var fileName = _file.FileName;
        if (fileName.Contains("\\"))
        {
            var aFile = fileName.Split('\\');
            fileName = aFile[ aFile.Length - 1 ].ToString();
        }

        // create full save path for uploaded file
        var full_file_path = Server.MapPath( UP_FOLDER ) + "\\" + fileName;

        try
        {
            // save file to server
            _file.SaveAs(full_file_path);
        }
        catch (Exception exc)
        {
            var emsg = "Unable to upload file : " + exc.Message;

            Response.Write( emsg );
            Response.Flush();
            Response.End();
        }

        // show result
        Response.Write( _file.FileName + " uploaded! <br>" );
    }
}

That’s all there is to it codewise. Before using this code you will need to give the NETWORK SERVICES user write permissions to your upload folder. Other than that, that’s all she wrote!

Here is a zip of all the code for you to download.

Inside this zip you will find :

• flashclient.fla – Flash upload client ( *be sure to update the upload path before building )
• flexclient.mxml – Flex upload client ( *also update upload path before building )
• uploader.aspx - ASP.NET file upload handler
• uploadform.html - sample HTML upload form ( again, update path )

Hope somebody finds this useful.