BizTalk 2010 Rollup – Sequence to Flat File Mapping

Feb 032012

Earlier in the week I spent some time trying to accomplish what I thought was going to be a trivial task – mapping from a sequence structure to a flat file structure using BizTalk Development Tools.

The main idea is to take a looping structure on the left hand side and map it to a flat structure on the right hand side.

As it turns out, it’s pretty easy to do, and not complicated. I spent a lot of time trying to do something more dramatic, if you’re looking for this info, I think you’ll be surprised by how logical the solution is.

Take a schema like the following:

Which defines a structure along the lines of this:

<ns0:Account xmlns:ns0="http://Mapping_Project.Inbound">
<FirstName>Jed</FirstName>
<LastName>Bartlett</LastName>
<Address>
    <Line1>1 Constitution Ave</Line1>
    <Line2></Line2>
    <Suburb>Lincoln</Suburb>
    <State>Nebraska</State>
    <Country>USA</Country>
    <PostCode>90210</PostCode>
    <Type>Home</Type>
</Address>
<Address>
    <Line1>Suite 10</Line1>
    <Line2>1090 Eggplant Blvd</Line2>
    <Suburb>Washington</Suburb>
    <State>Nebraska</State>
    <Country>USA</Country>
    <PostCode>90211</PostCode>
    <Type>Work</Type>
</Address>
</ns0:Account>

We’re going to try and transform it into something ugly, like this:

Which means flattening the data from the left hand side to the right hand side – but how?

First up, generate some test data. It’s a very good habit to get into, and it’s easy as pie. Right click on a schema and select “Generate Instance”. Modify the values so they’re easy to debug with and you’re set.

Let’s look at how we transform this simple XML. The answer, as it turns out is blindingly simple.

Open up your map..

Now, let’s assume we want to apply a condition to how we separate the data on the left hand side. Let’s assume there will be two types of addresses (Home and Work). Our aim is to assign all the “Address 1” address details as “Home”, and all the “Address 2” details as “Work”.

Drop two Logical EQUALs functoids onto the map and join them to the “Type” attribute. Then, open each functoid and we’ll set the comparison value to ‘Home’ and ‘Work’ respectively (omit the quotes). This applies the logic to split the output.

1. 2.

Now, assign all the attributes directly across to their destination.

Next, connect each equals functoid to the destination it reflects, i.e. connect the functoid for “Home” to each of the “Address 1” properties, etc. Don’t forget to map the “Type” too, if you want to capture that detail.

Given the data I’ve specified above – let’s give it a test! With the map open in Visual Studio, open the Properties Window and take a look at the following details:

Change the “Test Map Input” to “XML” and then you can specify the path to a test XML document (which you might have created earlier). Here’s how mine ended up:

To test, right click on the map file in Solution Explorer and select “Test Map”. You should get some results in the output window:

TestMap used the following file: <file:///C:\Dev\BizTalk\Mapping\Mapping Project\Test Data\Inbound_Data.xml> as input to the map.
Test Map success for map file C:\Dev\BizTalk\Mapping\Mapping Project\Maps\InboundToOutbound.btm. The output is stored in the following file: <file:///C:\Dev\BizTalk\Mapping\Mapping Project\Test Output\Output.xml>
Component invocation succeeded.

Which provides a generated instance which looks like this:

<ns0:AccountDetails xmlns:ns0="http://Mapping_Project.Outbound">
<Name>Jed Bartlett</Name>
<Address1_Line1>1 Constitution Ave</Address1_Line1>
<Address1_Line2 />
<Address1_Suburb>Lincoln</Address1_Suburb>
<Address1_State>Nebraska</Address1_State>
<Address1_Country>USA</Address1_Country>
<Address1_PostCode>90210</Address1_PostCode>
<Address1_Type>Home</Address1_Type>
<Address2_Line1>Suite 10</Address2_Line1>
<Address2_Line2>1090 Eggplant Blvd</Address2_Line2>
<Address2_Suburb>Washington</Address2_Suburb>
<Address2_State>Nebraska</Address2_State>
<Address2_Country>USA</Address2_Country>
<Address2_PostCode>90211</Address2_PostCode>
<Address2_Type>Work</Address2_Type>
</ns0:AccountDetails>

There were some warnings about multiple mappings, but I didn’t find the absence of a looping functoid to be a problem. I realise this is a pretty obvious and simple mapping, but when I went looking for examples about how to solve this type of mapping, I couldn’t find much at all (including from e-books).

There’ll be more on BizTalk mapping and other fun stuff in some upcoming articles.

Changing the Remote Desktop Security Layer

Tech Meme, Tips and Tricks No Responses »

Jan 232012

Ever wondered how to configure when you get prompted for user credentials when logging onto a remote system via Remote Desktop (Terminal Services)?

Well, chances are that this might come in handy – especially fi you encounter issues which stem from an expired password (hence, the title of this article). I was prompted with a pre-connection Authentication challenge/response dialog, when I entered the correct details, it simply displayed a dialog with an “OK” and “Cancel” button and the text “"You must change your password before logging on the first time.".

Trawling the Internet for some hints, I came across a thread on the MSDN Forums suggesting that the Security setting be changed to “RDP Security Layer” from “Negotiate”. Do do this you need to open the Terminal Services configuration, which can be done by executing the tsconfig.msc console.

Double click on the Connection:

..and you can change the default Security Layer (as shown).

This seems to only apply to Windows Server 2008, as I could not locate a similar application on Windows 7. It may be possible to run this console on Windows 7

In the end, it didn’t help my specific scenario (sorry), but I did like that it changed the authentication process. For my situation, I had someone clear the password change setting on the domain account in question, then happily logged on and changed the password manually.

Deploying BizTalk 2010 Assembly Dependencies

BizTalk Server, Infrastructure, Tips and Tricks No Responses »

Jan 062012

Introduction

From time to time, it is natural for us to deploy BizTalk solutions with Assembly dependencies. Usually this is because we have created helper classes (for mapping, or querying) or perhaps it reuses some common logic shared between application suites.

Whatever the reason, it can be very handy to pick up all assemblies when we export the BizTalk Application from BizTalk Administration, when we use the Export MSI feature/wizard.

The Problem

The one drawback is that, since BizTalk requires assemblies to be GAC’d (that is, strong named and added to the Global Assembly Cache), it’s hard for a tool, like the export to MSI wizard, to know what are custom assemblies which the BizTalk assemblies rely on (that aren’t system or framework assemblies).

In other words, you need a way to call out the dependencies, so that the export wizard will package the entire application. The alternative would be that you would have to ensure any custom dependencies are already deployed to your (new) target environment.

The Solution

As you might be aware, you can view a BizTalk application’s resources (assemblies) from within BizTalk Administrator. What you may not know is that you can add “Resources” to this location. In particular, you can add (or call out) dependency assemblies.

Just right click in the Resources view, and select Add->Resources..

A dialog pops up and allows you to browse for the required assemblie(s). You have some options here, you can force the assemblies to overwrite, you can select if and when the chosen assemblie(s) are GAC’d (registered with the GAC), registered as COM components or made visible to COM components.

The dependencies tab allows you to quickly see if all the dependencies for a specific assembly are to be found. This will help you check and ensure your solution has everything it needs.

When you decide to do an “Export to MSI”, you’ll notice that it now includes any dependency resource Assemblies which you have added to your application:

If the assembly you are adding has a dependency on another assembly that is not included in the application, the add operation will fail.

Great, but what if I want to automate this procedure as part of an automated deployment?

I’m glad you asked. Naturally, you can script commands which will accomplish this task for you. You could easily script this command, to be included in automated builds or deployments.

Note that, as a general rule, it is not advised to automatically register any assemblies in the Global Assembly Cache of a build machine/build server.

Steps to add assemblies from command line:

Open a command prompt as follows: Click Start, click Run, type cmd, and then click OK.

Note: you may require elevated permissions to accomplish this, you’ll certainly need permissions to administer BizTalk Server as well.
Type the following command, substituting the appropriate values, as described in the following table.

Command Syntax:

BTSTask AddResource [/ApplicationName:value] /Type:System.BizTalk:BizTalkAssembly [/Overwrite] /Source:value [/Destination:value] [/Options:GacOnAdd|GacOnInstall|GacOnImport] [/Server:value] [/Database:value]

Example:

BTSTask AddResource /ApplicationName:MyApplication
/Type:System.BizTalk:BizTalkAssembly /Overwrite
/Source:"C:\BizTalk Assemblies\MyOrchestration.dll"
/Destination:"C:\New BizTalk Assemblies\
MyOrchestration.dll " /Server:MyDatabaseServer
/Database:BizTalkMgmtDb

The following is a complete list of options lifted from MSDN, but they approximate what you can do via the BizTalk Administrator console.

Parameter	Value
/ApplicationName	Name of the BizTalk application to which to add the BizTalk assembly. If the application name is not specified, the default BizTalk application is used. If the name includes spaces, you must enclose it in double quotation marks (").
/Type	System.BizTalk:BizTalkAssembly
/Overwrite	Option to update an existing assembly. If not specified, and an assembly already exists in the application that has the same LUID as the assembly being added, the AddResource operation fails. You can view the LUIDs for the artifacts in an application by using the ListApp Command. If another application depends on the assembly being overwritten, the AddResource operation fails, even when this parameter is specified.
/Source	Full path of the assembly file, including the file name. If the path includes spaces, you must enclose it in double quotation marks (").
/Destination	Full path of the location where the assembly file is to be copied when the application is installed from the .msi file. If not provided, the assembly file is not copied to the local file system during installation. If the path includes spaces, you must enclose it in double quotation marks (").
/Options	GacOnAdd: Specify to install the assembly to the global assembly cache (GAC) on the local computer during the AddResource operation. GacOnInstall: Specify to install the assembly to the GAC when the application is installed from the .msi file. GacOnImport: Specify to install the assembly to the GAC when the application .msi file is imported. You must separate multiple options with a comma.
/Server	Name of the SQL Server instance hosting the BizTalk Management database, in the form ServerName\InstanceName,Port. Instance name is only required when the instance name is different than the server name. Port is only required when SQL Server uses a port number other than the default (1433). Examples: Server=MyServer Server=MyServer\MySQLServer,1533 If not provided, the name of the SQL Server instance running on the local computer is used.
/Database	Name of the BizTalk Management database. If not specified, the BizTalk Management database running in the local instance of SQL Server is used.

Introduction

As I alluded to at the very end of 2011, I’ve been spending more and more time with BizTalk 2010. It’s been a little while since I’ve had to support/build/maintain BizTalk artefacts, but it doesn’t take long to reacquire “the groove”.

To gently thrust you into the dark underbelly of the land of BizTalk development, I thought I’d start off by exploring some fairly basic BizTalk 101 type experiences (and solutions!). This article focuses on generic issues with XML schemas, with an eye to BizTalk’s usage of said schema.

Sadly, some of what you might read here (and later) might smack a tad of ‘What? You didn’t know that?”, but I’m willing to swallow some pride if it helps out other Architects and Programmers.

What you need (to follow along)

This article is (IMHO) decent enough for the generalist (non-BizTalk specialist) and especially may be handy for those unfamiliar in working with XML schemas.

There is a big part where I’m playing with BizTalk’s Developer Tools (in Visual Studio 2010), so if you’d like to give it a go, I’d suggest you pick up a copy of BizTalk Server 2010 Developer Edition (click here for more info on this edition of BizTalk), which – I believe – you need to have to install the developer tools.

If I write subsequent article on BizTalk, the BizTalk developer tools will likely play a significant role, so it might be worth investigating. As always, it’s worth bookmarking the BizTalk Development Center link.

The Joys of Schema

If you know anything about BizTalk, you’ll know that it’s all about the schemas. Everything ends up mapping to a document schema type whether it’s inbound or outbound. Schemas (XSD) can be an acquired taste, especially if you’ve been off doing something more exciting.

Last month, handwritten schemas tripped me up quite a bit. I was used to using Wizards and Adapters to generate a lot of the schemas for me, so when I went to design some basic input and output schemas, I encountered some “fun” issues.

Something funny happened…

Now, I was putting together some very basic sample schema to use as an example for demonstrating a really neat technique with maps (transforms).

Just in case someone else encounters this (or, like me, found it mildly amusing) I thought I’d document some cause and effect of why you might get some head scratching error messages when designing in Visual Studio 2010 with the BizTalk designer.

Given the following (flawed) schema, which I built within Visual Studio:

<?xml version="1.0" encoding="utf-16"?>

<xs:schema xmlns="http://Sandbox.InputSchema" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://Sandbox.InputSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema">

<xs:element name="Root">

<xs:complexType>

<xs:sequence>

<xs:element minOccurs="2" maxOccurs="3" name="Customer">

<xs:complexType>

<xs:sequence>

<xs:element name="Address">

<xs:complexType>

<xs:attribute name="Street" type="xs:string" />

<xs:attribute name="State" type="xs:string" />

<xs:attribute name="Country" type="xs:string" />

</xs:complexType>

</xs:element>

</xs:sequence>

<xs:attribute name="ID" type="xs:integer" />

<xs:attribute name="FirstName" type="xs:string" />

<xs:attribute name="Surname" type="xs:string" />

</xs:complexType>

</xs:element>

</xs:sequence>

</xs:complexType>

</xs:element>

</xs:schema>

Which, when loaded in the BizTalk Schema Designer looks like this:

Has an unfortunately obvious flaw if you consider I was expecting the schema to facilitate the following format of XML data:

<ns0:Root xmlns:ns0="http://Sandbox.InputSchema">

</Customer>

</Customer>

</ns0:Root>

In other words, I was expecting to store multiple “Customer” elements in my XML document.

Debugging

When I loaded the document and tested it in Visual Studio 2010, I received the following, very weird error message: “The element ‘Root’ in namespace ‘http://Sandbox.InputSchema’ has invalid child element ‘Customer’.” – huh?

The problem, I thought, might be related to not expressly prefixing the namespace directive to the child element, so I made some edits.

This resulted in an even more ambiguous error message, funny even:

“The element ‘Root’ in namespace ‘http://Sandbox.InputSchema’ has invalid child element ‘Customer’ in namespace ‘http://Sandbox.InputSchema’. List of possible elements expected: ‘Customer’.”

or, in generic terms:

“The element ‘<element>’ in namespace ‘<namespace>’ has invalid child element ‘<child element>’ in namespace ‘<namespace>’. List of possible elements expected: ‘<child element>’.”

WTF?

You may have to squint to read the text…

At this point, I felt like I was going backwards. So I undid my changes and took a look at the schema definition – which, upon second inspection was clearly wrong.

The Solution

See if you can spot the difference?

My mistake was in not declaring a sequence after the root element. Thus rectified, my XML now matched the fixed schema, and I was able to continue on my merry way.

Filtering in a Transform/Map

This all brings us back to the original intention of my article – to highlight the use of functoids for filtering multiple nodes in a transform.

Given that we want to transform our input schema (from before) into another format, with filtered data, here’s the schema for the output:

The Mapping

We can construct a map which assigns the fields we want per the following example:

A basic map

Now, if we wanted to only take Customers who live in the state of New South Wales, we need to validate each element in the input XML recursively.

A Functoid for all occasions

The easiest means is to use a equality functoid, in this example I’ve used an equals functoid, as per below. You must still map the field directly to the output schema, but also to the functoid – which is, in turn, mapped to the logical root of the target (per the example above).

When the functoid returns a true value, the element is copied to the output. If the result is fale, the entire element (children etc) are skipped. Thus, from the following input XML:

<ns0:Root xmlns:ns0="http://Sandbox.InputSchema">

</Customer>

</Customer>

</ns0:Root>

When run through this map (using trusty ‘Test Map’ functionality), will generate the following output, filtering out that pesky Queensland (QLD) customer:

<ns0:Root xmlns:ns0="http://Sandbox.OutputSchema">

</ns0:Root>

Helpful Support in Visual Studio 2010

Before you start despairing about schema development and BizTalk maps, there are a bunch of very helpful tools built into Visual Studio 2010 (and prior versions).

Generating Schema Instances

Given you have Schema files in your project, you can right click them and select “Generate Instance”. This will create an XML file using intelligent default values, according to the schema definition. These files can be modified and used to test out other artefacts who use this schema.

Validating Schema Instances

If you want to test if a particular XML file is valid, you can do the reverse – by clicking on the schema properties, you can set the “input XML” for a given schema. Then, when right clicking on the Schema in Solution Explorer, you can select “Validate Instance”. This will highlight any issues with your schema.

Testing a Map/Transform

Once you have some valid XML for your schema, you can then use it as test input data for various things including maps.

The following shows how to establish test data parameters for a map within Visual Studio 2010:

When you have configured a map to use a specified XML input, you can then right click on the map (in Solution Explorer) and select “Test Map”.

Other Options – Xml Schema/Data Type Utility

A handy note for future reference – if you have an XML structure in mind, but (like me) you are rusty with your XSD designing skillz, there’s another way forward which may prove to be better.

There is a tool which helps you work with XML and XSD document formats, and it ships by default with a number of Microsoft products and SDKs, it is called “Xml Schemas/DataTypes support utility” or XSD.exe.

Important Disclaimer: In nearly almost all circumstances, it is considered a better practice to produce or design the XML schema first. Generating a schema from XML data can be fraught with peril, especially if your XML sample doesn’t accurately reflect the full spectrum of supported formats.

For example, min/max occurrences, type lengths and other constraints, vague data types (where the type or precision is “guessed”), elements supported by the schema which are not required – and thus missing from your sample XML – to name but a few.

Where Do I Find This Tool?

Chances are that you have it already. I’ve located it installed in the following locations (check the (x86) folder as well if you are on a 64 bit machine):

Directory of C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin

Directory of C:\Program Files\Microsoft SDKs\Windows\v7.0A\Bin

Directory of C:\Program Files\Microsoft SDKs\Windows\v7.0A\Bin\NETFX 4.0 Tools

Directory of C:\Program Files\Microsoft SDKs\Windows\v7.0A\Bin\x64

What does it do?

Keeping in mind that this utility was really meant for the days of evil DataSets, it still can be quite useful.

Using the following (poor) sample XML structure:

<?xml version=”1.0” encoding=”utf-8”?>
<Country Name="Australia">
     <State Name="NSW">
           <Capital Name="Sydney"/>
     </State>
     <State Name="QLD">
           <Capital Name="Brisbane"/>
     </State>
     <State Name="WA">
           <Capital Name="Perth"/>
     </State>
     <State Name="TAS">
           <Capital Name="Hobart"/>
     </State>
     <State Name="VIC">
           <Capital Name="Melbourne"/>
     </State>
</Country>

You can use the XSD.exe utility to generate an approximation of the schema, like so:

C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin>xsd Input.xml

Microsoft (R) Xml Schemas/DataTypes support utility

[Microsoft (R) .NET Framework, Version 2.0.50727.3038]

Writing file ‘C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\Input.xsd’.

Which would produce the following output:

<?xml version="1.0" encoding="utf-8"?>

<xs:schema id="NewDataSet" xmlns="" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">

<xs:element name="Country">

<xs:complexType>

<xs:sequence>

<xs:element name="State" minOccurs="0" maxOccurs="unbounded">

<xs:complexType>

<xs:sequence>

<xs:element name="Capital" minOccurs="0" maxOccurs="unbounded">

<xs:complexType>

<xs:attribute name="Name" type="xs:string" />

</xs:complexType>

</xs:element>

</xs:sequence>

<xs:attribute name="Name" type="xs:string" />

</xs:complexType>

</xs:element>

</xs:sequence>

<xs:attribute name="Name" type="xs:string" />

</xs:complexType>

</xs:element>

<xs:element name="NewDataSet" msdata:IsDataSet="true" msdata:UseCurrentLocale="true">

<xs:complexType>

<xs:choice minOccurs="0" maxOccurs="unbounded">

<xs:element ref="Country" />

</xs:choice>

</xs:complexType>

</xs:element>

</xs:schema>

Now, given I mentioned that bit about the DataSet support.. You’ll need to remove the element called “NewDataSet”. This is included because the utility assumes you want to generate a schema for use with DataSets.

Removing the element, renders a nice XSD in Visual Studio:

But wait.. There’s more

Given a valid XSD schema, you can now use the same utility (isn’t it wonderful?) to generate valid C# classes by using the following syntax:

C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin>xsd Input.xsd /c

Microsoft (R) Xml Schemas/DataTypes support utility

[Microsoft (R) .NET Framework, Version 2.0.50727.3038]

Writing file ‘C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\Input.cs’.

The result of executing this command produces the following classes (which you could now serialize into and out of XML):

Summary

You may not be able to teach a dog new tricks, but I’m willing to bet he remembers a bunch of old ones!

Although I’m a bit rusty, I’m able to overcome obstacles reasonably quickly. This article might not make me look like Einstein, but I thought I’d write up some of my experiences in the hope it helps others.

We’ve only just touched the tip of the iceberg with BizTalk 2010. The development environment is challenging and prickly (which makes for great sources of inspiration for blog articles) so you can be sure there will be more here soon, so check back!

Some INFORMATION_SCHEMA magic

Database Magic, Tips and Tricks No Responses »

Nov 082011

There are times when you want to get a little creative about your investigative work when it comes to SQL Server. In this case, this is a very short article highlighting some efficiencies in locating information about specific programmatic resources within a database.

If you haven’t used the INFORMATION_SCHEMA views before, well you’re in for a treat.
Let’s say you need to query the details of a Stored Procedure, and all you know is the name of the proc.

How would you go about retrieving the information?

One way is to use the INFORMATION_SCHEMA.ROUTINES. For this example, let us assume our proc is called “usp_TestProcedure”, and execute the following:

SELECT *
FROM INFORMATION_SCHEMA.ROUTINES
WHERE ROUTINE_TYPE = ‘PROCEDURE’
AND ROUTINE_NAME = ‘usp_TestProcedure’
ORDER BY ROUTINE_NAME

This results in a plethora of information (some of it not used). Of interest are the following columns:

ROUTINE_CATALOG – location of the routine
ROUTINE_SCHEMA – schema of the routine
ROUTINE_NAME – name of the routine

ROUTINE_TYPE – Type of routine, e.g. Procedure/Function etc
ROUTINE_BODY – Whether it is T-SQL, other etc
ROUTINE_DEFINITION – The actual text of the routine

CREATED – Created Date
LAST_ALTERED – Modified date

All well and good.. but what of those Parameters? We can query those details from INFORMATION_SCHEMA.PARAMETERS as follows:

SELECT
COALESCE(PARAMETER_NAME, ‘<no params>’) as ‘Parameter’,
COALESCE(UPPER(DATA_TYPE) +
CASE
    WHEN DATA_TYPE IN (‘NUMERIC’, ‘DECIMAL’) THEN
     ‘(‘ + CAST(NUMERIC_PRECISION AS VARCHAR)
        + ‘, ‘ + CAST(NUMERIC_SCALE AS VARCHAR) + ‘)’
    WHEN RIGHT(DATA_TYPE, 4) = ‘CHAR’ THEN
        ‘(‘ + CAST(CHARACTER_MAXIMUM_LENGTH AS VARCHAR) + ‘)’
        ELSE ” END + CASE PARAMETER_MODE
    WHEN ‘INOUT’ THEN ‘ OUTPUT’ ELSE ‘ ‘
    END, ‘-’) as ‘Data Type’
FROM INFORMATION_SCHEMA.PARAMETERS
WHERE SPECIFIC_NAME = ‘usp_TestProcedure’
ORDER BY ORDINAL_POSITION

Obviously this applies a bit of logic to format the output, depending on the type of parameter, there are plenty of useful columns in this view which would help you determine more about each parameter, such as numeric precision, maximum lengths, parameter direction (in/out) and so on.

Now, what happens when you alter a Stored Procedure and the information isn’t immediately refreshed in the INFORMATION_SCHEMA? Well, for SQL Server, try running a Stored Procedure recompile by executing the following:

EXECUTE sp_recompile <proc name>

Enjoy using INFORMATION_SCHEMA!

Analyze .NET Applications with Visual Studio 2010 (Part 1)

Design Concepts and Code, Tips and Tricks No Responses »

Oct 212011

Introduction

If you have a copy of Visual Studio 2010 Premium or Ultimate, you’ll likely have a menu in your Integrated Development Environment (IDE) called “Analyze”.

Chances are, you might not have investigated the true potential of the profiling tools in VS 2010. This article will demonstrate usage of some of the profiling techniques, and then discuss how to interpret and act on the results.

You’ll need:

A copy of Visual Studio 2010 Premium or Ultimate
A .NET solution for testing purposes, the more complex the better
- Preferably a WinForms or Console application, but a WCF service application or ASP.net website will work just fine
Memory (at least 4 GB) and a decent CPU

Profiling is an expensive operation in terms of system resources. Trying to profile a .NET application on a slow, low memory machine is going to be a long and frustrating task, so choose and configure your environment well!

My Environment

At the moment, I’ve configured two independent environments – a desktop machine, and a Hyper-V virtual client OS, specifications are as below.

Desktop

Windows 7 x64 + Service Pack 1
Intel Core i5 2400 3.10 GHz (dual core with hyper-threading)
4.0 GB Memory
222 GB HDD (120 GB Free)

Hyper-V Server Client OS

Windows Server 2008 R2 x64+ Service Pack 1
AMD Opteron (Dual Core) 2216 HE 1.58 GHz
6.0 GB Memory
60 GB HDD (20 GB free)

Prerequisites/Considerations

Profiling is not an inexpensive operation. You will require a decent amount of hard drive space (so that Visual Studio can record profiling information as you run your tests) and plenty of memory. Remember, most of the time the focus should be on determining what your application or services does, not so much why it labours when in a resource constrained environment.

I’m not saying that there’s no value in profiling under a resource constrained environment – there likely is – however it will be a terribly frustrating endeavour and it will be hard to differentiate between overhead from running your application or service, and the overhead of the act of profiling itself. You can likely extrapolate the impact of low resource environments based on the metrics collected during profiling anyway.

In short: the better the environment you profile under, the more likely the data is of use to you. Just keep it in mind.

In my more recent experience, I found that a memory profile session used almost 8 GB of hard drive space to track/record an application which had a memory footprint of between 500mb to 650 mb. I’m not sure if this is much of a guide. The more HDD space the better is a good general rule.

Method

Open your solution in Visual Studio, for best results, I’d recommend profiling in Debug configuration and preferably according to the processor architecture of your machine (i.e. x64 if your machine is 64 bit). Ensure your solution successfully compiles – always helps – and you should be ready to proceed. Note – if you are using a processor specific configuration, all modules to be profiled must match (no mix ‘n match of 32 and 64 bit modules). Assuming you have Visual Studio 2010 Premium or Ultimate, you should have an “Analyze” menu:

The best way to proceed, the first time, is to “Launch Performance Wizard”:

There are quite a few decent options here, but what do they cover? Here’s a more in depth look at each configuration:

Method	Description
Sampling	Collects statistical data about the work performed by an application.
Instrumentation	Collects detailed timing information about each function call.
Concurrency	Collects detailed information about multi-threaded applications.
.NET memory	Collects detailed information about .NET memory allocation and garbage collection.
Tier interaction	Collects information about synchronous ADO.NET function calls to a SqlServer database.

For this and the next article in this series, I’m mostly going to focus on the more useful profiling methods – .NET Memory and Instrumentation, but concurrency, tier interaction and sampling all have their uses.

What is useful in understanding up front, is that no matter what your method of data collection (profiling) is, the template within which the profile reports are rendered remains more or less the same. The varying factor is the X-factor – i.e. the “what is collected”. Memory is pretty obvious – the focus of sampling is on the allocation (exclusive and inclusive) of memory and the performance of Garbage Collection. With instrumentation, the sampling focus is on call frequency, call chaining, call durations and the distribution of call frequency between functions and modules.

The more information that can be supplied (i.e. symbols, debugging info such as .pdbs and line numbers/source code) the better when it comes to exhaustive analysis.

Some Notes on Running the Profiler

You will almost invariably need to run Visual Studio 2010 with elevated permissions. This (some days) seems like a universal constant as we start to require a heavier hand in constructing, tuning and testing our software. For the best results, the more control and permissions you have (administration rights help the profiler collect performance counter stats information, for example) the better. Some profiling (such as concurrency) may require changes to the system configuration (and may require a system reboot).

In the next article…

That’s it for our introductory article – the next article will dive into the actually profiling itself, and how to interpret the results. Stay tuned for more.

A Custom Background for Windows 7 Logon Screen

Tips and Tricks No Responses »

Sep 302011

Recently I came across this little gem of a registry change –
allowing a custom logon screen background, using OEM registry keys.

Essentially, you simply have to set a specific key, then create a folder under the Windows directory, and then insert a JPG picture of your choosing.
It’s pretty simple and will take a few minutes, if you have the interest.

Honestly, the hardest part is choosing an appropriate image! The requirements are that the image not exceed 256k in size.

In order for it to look half decent, it is highly recommended that you resize the image to the same dimensions as your screen resolution (if possible).

Once ready, open your registry editor (Start->regedit) – you may be required to allow regedit to elevate (UAC). Once in, open HKEY_LOCAL_MACHINE and expand the path:
HKLM\Software\Microsoft\Windows\CurrentVersion\Authentication\LogonUI\Background

If this path doesn’t exist, feel free to create any of the sub keys as necessary. At this location, create a new DWORD (32 bit) value and call it OEMBackground. Set the value to 1. You are now done with the registry, close Regedit.

Next Click open Windows Explorer and navigate to: <system drive>\Windows\System32\oobe\

If there is no subfolder here called ‘info’, then feel free to create it.
If not, open the folder and create a new subfolder called ‘backgrounds’.

Copy your image into this folder and rename the image ‘backgroundDefault.jpg’.
All things being equal, you should have the following:

You might get prompted for some security elevation, since you’re making some file system changes to structures under the (protected) Windows system folders.

Note that my chosen image is under 256kb in size. Log off and you should see your new Logon Screen background.

Automating Visual Studio 2010 – Part 2

Build and Config, Projects, Tips and Tricks No Responses »

Aug 232011

Continuing on from Part 1, our objective in this article is to create a new Visual Studio solution, and then populate it with some existing projects. To facilitate this, it would be beneficial if you had a few projects already created. For this example, I have defined the following structure:

C:\UnitTesting – Root Folder

C:\UnitTesting\TestSolution – An existing folder which contains subdirectories which hold project files.

The sample structure is as follows:

So basically, there are three projects and two reference each other. Our goal is to create a new solution file in the root directory which is then populated with references to the existing project files.

Please bear in mind that this is all demo code, and lacks the normal hardening and checks which production code should have. Please use wisely, as there are no refunds.. ha!

I’ve reworked the previous example to be a little more robust, as follows:

Encapsulated the functionality to create a new blank solution file (a method called CreateSolution())
Second, I’ve added a new function called AddProjects which will take a path, plus details about the target solution file.

I’ve moved the DTE2 object to be a static object, and refactored CreateSolution thus:

private static DTE2 _dte2 = (DTE2)Microsoft.VisualBasic.Interaction.CreateObject("VisualStudio.DTE.10.0", "");

public static void CreateSolution(string path, string solutionName)
{
    Solution4 solutionObject = (Solution4)_dte2.Solution;
    solutionObject.Create(path, solutionName);
    solutionObject.Close(true);
}

Unit testing for this was easy enough:

[TestMethod]
public void CreateEmptySolution()
{
    SolutionHelper.CreateSolution(@"C:\UnitTesting\", "UnitTestSolution");
}

[TestMethod]
public void AddProjects()
{
    File.Delete(@"C:\UnitTesting\UnitTestSolution.sln");
    SolutionHelper.AddProjects(@"C:\UnitTesting\UnitTestSolution.sln", @"C:\UnitTesting\TestSolution", "*.*proj");
}

As you can see, all pretty straightforward. So reviewing our CreateSolution function, it now just takes a path and solution name (omit the “.sln” extension) and creates the empty solution.

The next step is to iterate and find projects, and finally – add them to a solution file. I’ve added some logic which creates the target Solution File if it does not exist.

The AddProjects function is pretty easy to understand:

public static void AddProjects(string solutionFile, string projectPath, string projectWildCardMatch)
{
    if (!File.Exists(solutionFile))
    {
        CreateSolution(Path.GetDirectoryName(solutionFile), Path.GetFileName(solutionFile));
    }

    Solution4 solutionObject = (Solution4)_dte2.Solution;
    solutionObject.Open(solutionFile);

    var fileNames = Directory.GetFiles(projectPath, projectWildCardMatch, SearchOption.AllDirectories);

    foreach (string projectFile in fileNames)
    {
        solutionObject.AddFromFile(projectFile, false);
    }

    solutionObject.Close(true);
}

Hopefully you can see how easy it is to add projects to the solution. The boolean value in the call Solution.AddFromFile is important, as it elects to use the currently loaded solution. If omitted, it is either attached to the project’s original solution (in my experience). Make sure you save the solution when closing it.

If we open the generated solution file in Visual Studio 2010, you’ll see that not only are the projects added, but they also show the references:

Here is a complete copy of the source file:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using EnvDTE80;
using EnvDTE100;
using System.IO;
using EnvDTE;
using System.Diagnostics;

namespace VSAutomation.Toolkit
{
    public static class SolutionHelper
    {
        private static DTE2 _dte2 = (DTE2)Microsoft.VisualBasic.Interaction.CreateObject("VisualStudio.DTE.10.0", "");

        public static void CreateSolution(string path, string solutionName)
        {
            Solution4 solutionObject = (Solution4)_dte2.Solution;
            solutionObject.Create(path, solutionName);
            solutionObject.Close(true);
        }

        public static void AddProjects(string solutionFile, string projectPath)
        {
            if (!File.Exists(solutionFile))
            {
                CreateSolution(Path.GetDirectoryName(solutionFile), Path.GetFileName(solutionFile));
            }

            Solution4 solutionObject = (Solution4)_dte2.Solution;
            solutionObject.Open(solutionFile);

            var fileNames = Directory.GetFiles(projectPath, "*.*proj", SearchOption.AllDirectories);

            foreach (string projectFile in fileNames)
            {
                solutionObject.AddFromFile(projectFile, false);
            }

            solutionObject.Close(true);
        }
    }
}

Visual Studio 2010 and MSDN Licensing White Paper

Infrastructure, Team Foundation Server, Tips and Tricks No Responses »

Aug 102011

Not too long ago I wrote an article about Team Foundation Server licensing. Now that it is 2011, and after some recent access requests for TFS, I’ve decided it is time for an update.

Does TFS cover SQL Server Licensing?

One instance of Microsoft SQL Server 2008 Standard Edition or SQL Server 2008 R2 Standard Edition can be used as the Team Foundation Server 2010 database.

That’s right – TFS 2010 includes a Standard Edition license for SQL Server but only when used exclusively as the data store for TFS and nothing else. The paper doesn’t stipulate if the license is only valid for single tier or dual tier configurations, but since the database instance must be used exclusively by TFS, I can’t see why it wouldn’t cover dual tier deployments. Note that if you want to use a different version of SQL (Enterprise, DataCenter) you need a separate license.

How about TFS user licensing?

A few years ago, Microsoft eased up a little bit on the licensing restrictions which caused a user to require a Client Access License (which are expensive). The clause in the TFS licensing is explicitly:

Client Access License Exception for Certain Work Items
A user does not need a CAL or External Connector License to create new work items or to update work items that that same user has created. This exception applies only to work items related to defect filing or enhancement requests. However, a CAL is required when a user views or modifies a work item created by another user or interacts with Team Foundation Server in any other way.

Which opens up TFS for people who wish to create defects (bugs) and feature requests. Accessing TFS for this purpose is fairly obvious and easy to open to a large host of users – via Web Access. However, how does a TFS Administrator set permissions appropriately? It turns out that, via the TFS Administration Console, Active Directory groups and users can be added to a special group called the “Work Item Only View Users”. For more information on configuring this, see the following article: Work Item Only View (WIOV) Users in TFS 2010

Members of the Work Item Only View Users group are restricted to a limited view, which is indicated by the “Work Item Only View” tag in the top right hand corner of the web application page:

By default, I noticed that restricted use users did not automatically have permissions to create work items! This rendered the whole exercise rather pointless. Digging deeper, I realised I’d have to grant permissions on a per-Team Project basis so that the limited permission group could create work items. Given that each Team Project could have a different template base, I believe you’ll also have to grant permissions separately for work item types as well – where the work item type relates to a defect or enhancement.

Note that external users (users who do not belong to the parent organization) are required to be licensed both with a CAL (if working beyond the scope of the CAL exception) and an External Connector License (which is a nice way of raising some extra revenue).

I can think of ways around that licensing model..

Now, don’t think that “multiplexing” or pooling access to TFS will avoid the need for CALS for your users!

End users or devices that access Team Foundation Server in any way-other than the Client Access License Exception for Certain Work Items exception noted above-are required to have the appropriate licenses, regardless of whether they are using a direct or indirect connection to the software.

Which translates into: “no matter how a user accesses TFS or TFS data, Microsoft want their money”. Even automated builds aren’t devoid of licensing creep. An automated deployment or build which does basically anything other than compile requires a Visual Studio license:

CAL Requirements for Builds
Users who merely execute and review builds are only required to have a Team Foundation Server CAL; however, users who author tests or other types of validation and include them as part of a build must be licensed for the appropriate Visual Studio 2010 individual development tool.

This extends to the execution of: unit tests, code coverage tests, web or load testing, architectural tests (validations) or any code analysis or coded UI or database unit testing.

Also note that the following shows the many ways TFS can be accessed:

Accessing Team Foundation Server
The Team Foundation Server 2010 data can be accessed in several ways, including:
Visual Studio Team Explorer 2010, which is included with Visual Studio 2010 Ultimate, Visual Studio 2010 Premium, Visual Studio 2010 Professional, and Visual Studio Test Professional 2010, and available for download to the general public. Team Explorer also can be installed as a standalone client, and on as many physical devices as you like.
Visual Studio Team Web Access 2010, which is a browser-based version of the Team Explorer client.
Microsoft Office ExcelR or Microsoft Office Project, which can access Team Foundation Server using add-ins for those programs that are provided with Team Explorer.
Microsoft Expression Blend. 3, which can access Team Foundation Server without requiring Team Explorer.
Microsoft Expression Studio 4 (all editions), which can access Team Foundation Server without requiring Team Explorer.
Programmatically, as enabled through the Team Foundation Server 2010 application programming interfaces (APIs) or by other means.

For more information, why not check out the whitepaper directly? See below for further reading.

Visual Studio

Further Reading

Visual Studio 2010 and MSDN Licensing White Paper
Work Item Only View (WIOV) Users in TFS 2010
Team Foundation Server Permissions
Visual Studio 2010 and MSDN Licensing White Paper (updated)

F8S7T93JTXQR

Automating Visual Studio 2010 – Part 1

Build and Config, Design Concepts and Code, Tips and Tricks No Responses »

Aug 082011

Now for something completely different. Recently I had the desire to build a Visual Studio 2010 solution file for a number of project files under a directory. There were many projects, and each one linked to a solution file – but I wanted a “master” solution file.. one to rule them all(?)

Generally, I hate manual repetitious tasks, and laboriously adding all the projects to a new, blank, solution file seemed tedious and not fun one bit. Then I decided that I’d write a program which would iterate all the projects and create a solution file for me. Sounds simple enough, right?

Well, it took me on a journey into Visual Studio automation. There are many facets of working with VS, as I found out, and backwards compatibility reigns supreme. What I’m going to introduce you to is pretty much just for Visual Studio 2010, but in the right hands a design could come which supports versions dating back to Visual Studio 2005.

The Prerequisites

Visual Studio 2010 (obviously)
Visual Studio 2010 Service Pack 1

Optional Extras

In order to follow along, you may need to download the source of the MPF for Projects – Visual Studio 2010 project from codeplex for future parts to this article series.
There is also a chance you might require the Visual Studio 2010 SP1 SDK, although I haven’t had a need for it just yet.
These can’t hurt either: Visual Studio 2010 Productivity Power Tools (February 2011) if you don’t have it already (the screenshots reflect this being installed)

Once you’ve got your system all nicely configured, we’re ready to begin.

The Beginning

To begin with, I created a Class Library. My intention was to create a large amount of nice, reusable automation for integration into Team Build, utilities and so forth. You never know when this is going to be very, very handy.

To start our journey, I decided that my requirement was going to be very simple (for Part 1): create a blank solution file using Visual Studio 2010. Easy enough, right? Well, as you’d expect, the devil is in the details. This took me quite a bit of time to get right, but in the end the outcome was most pleasing.

So open up Visual Studio, create a project type you’d prefer and then ready the project, as so:

Add a reference
Pick the COM tab
Scroll and select “Microsoft Development Environment 10.0”
This will add several COM interop assemblies to your project (and also include the previous versions)
Click on the .Net tab and select “Microsoft.VisualBasic” (even if you aren’t coding in Visual Basic)

Now it is time to go code us up a simple solution. I’m going to be working on a far more robust library, but in the meantime this will introduce you to the extremely powerful DTE and DTE2 objects.

We have to (sadly) instantiate a COM object wrapper, which will actually load devenv in a non-interactive mode, from which we can then cherry pick functionality we’d like to use. For this demo, we’re just going to create a Visual Studio 2010 compatible solution file.

Consider the following very simple class:

namespace VSAutomation.Toolkit
{
    public static class SolutionHelper
    {
        public static void CreateSolution(string path, string solutionName)
        {
            DTE2 dte2 = (DTE2)Microsoft.VisualBasic.Interaction.CreateObject("VisualStudio.DTE.10.0", "");
            Solution4 solutionObject = (Solution4)dte2.Solution;
            solutionObject.Create(path, solutionName);
            solutionObject.Close(true);
        }
    }
}

You will notice that we create a COM object instance using the VisualBasic namespace. When the code actually creates the object, under the hood, the machine will see an instance of Visual Studio created (but no GUI):

Once it is loaded, we can get access to the VS Solution functionality. From here, it is a simple matter of supplying the path and solution name (omit the .sln extension) and, assuming all goes to plan, you will have the following output:

Obviously this is just the tip of the iceberg.

We have so much powerful functionality to exploit, so we won’t stop at this. Since we’ve achieved our objective for Part 1, we’ll wrap up here, but check back for Part 2 where we’ll populate the solution file with existing (or new) projects!

In the next part, we’ll also discuss ways of supporting the older versions (Visual Studio 2008 and 2005) as well as providing a more verbose sample.

Further Reading

How to: Get References to the DTE and DTE2 Objects

Solution4.Create Method (String, String)

Older Entries

Further Reading

Introduction

The Problem

The Solution

Great, but what if I want to automate this procedure as part of an automated deployment?

Further Reading

Introduction

What you need (to follow along)

The Joys of Schema

Something funny happened…

Debugging

The Solution

Filtering in a Transform/Map

The Mapping

A Functoid for all occasions

Helpful Support in Visual Studio 2010

Generating Schema Instances

Validating Schema Instances

Testing a Map/Transform

Other Options – Xml Schema/Data Type Utility

Where Do I Find This Tool?

What does it do?

But wait.. There’s more

Summary

Introduction

My Environment

Prerequisites/Considerations

Method

Some Notes on Running the Profiler

In the next article…

Further Reading

Don’t Miss This Section!

Wine Reviews

Stuff to cover the hosting costs

Pages

Translate this Page

Categories

Tag Cloud

Blogroll

Disclaimer

Aussie Wine Guy

Archives