BizTalk 2010 Rollup – Sequence to Flat File Mapping

Feb 032012

Earlier in the week I spent some time trying to accomplish what I thought was going to be a trivial task – mapping from a sequence structure to a flat file structure using BizTalk Development Tools.

The main idea is to take a looping structure on the left hand side and map it to a flat structure on the right hand side.

As it turns out, it’s pretty easy to do, and not complicated. I spent a lot of time trying to do something more dramatic, if you’re looking for this info, I think you’ll be surprised by how logical the solution is.

Take a schema like the following:

Which defines a structure along the lines of this:

<ns0:Account xmlns:ns0="http://Mapping_Project.Inbound">
<FirstName>Jed</FirstName>
<LastName>Bartlett</LastName>
<Address>
    <Line1>1 Constitution Ave</Line1>
    <Line2></Line2>
    <Suburb>Lincoln</Suburb>
    <State>Nebraska</State>
    <Country>USA</Country>
    <PostCode>90210</PostCode>
    <Type>Home</Type>
</Address>
<Address>
    <Line1>Suite 10</Line1>
    <Line2>1090 Eggplant Blvd</Line2>
    <Suburb>Washington</Suburb>
    <State>Nebraska</State>
    <Country>USA</Country>
    <PostCode>90211</PostCode>
    <Type>Work</Type>
</Address>
</ns0:Account>

We’re going to try and transform it into something ugly, like this:

Which means flattening the data from the left hand side to the right hand side – but how?

First up, generate some test data. It’s a very good habit to get into, and it’s easy as pie. Right click on a schema and select “Generate Instance”. Modify the values so they’re easy to debug with and you’re set.

Let’s look at how we transform this simple XML. The answer, as it turns out is blindingly simple.

Open up your map..

Now, let’s assume we want to apply a condition to how we separate the data on the left hand side. Let’s assume there will be two types of addresses (Home and Work). Our aim is to assign all the “Address 1” address details as “Home”, and all the “Address 2” details as “Work”.

Drop two Logical EQUALs functoids onto the map and join them to the “Type” attribute. Then, open each functoid and we’ll set the comparison value to ‘Home’ and ‘Work’ respectively (omit the quotes). This applies the logic to split the output.

1. 2.

Now, assign all the attributes directly across to their destination.

Next, connect each equals functoid to the destination it reflects, i.e. connect the functoid for “Home” to each of the “Address 1” properties, etc. Don’t forget to map the “Type” too, if you want to capture that detail.

Given the data I’ve specified above – let’s give it a test! With the map open in Visual Studio, open the Properties Window and take a look at the following details:

Change the “Test Map Input” to “XML” and then you can specify the path to a test XML document (which you might have created earlier). Here’s how mine ended up:

To test, right click on the map file in Solution Explorer and select “Test Map”. You should get some results in the output window:

TestMap used the following file: <file:///C:\Dev\BizTalk\Mapping\Mapping Project\Test Data\Inbound_Data.xml> as input to the map.
Test Map success for map file C:\Dev\BizTalk\Mapping\Mapping Project\Maps\InboundToOutbound.btm. The output is stored in the following file: <file:///C:\Dev\BizTalk\Mapping\Mapping Project\Test Output\Output.xml>
Component invocation succeeded.

Which provides a generated instance which looks like this:

<ns0:AccountDetails xmlns:ns0="http://Mapping_Project.Outbound">
<Name>Jed Bartlett</Name>
<Address1_Line1>1 Constitution Ave</Address1_Line1>
<Address1_Line2 />
<Address1_Suburb>Lincoln</Address1_Suburb>
<Address1_State>Nebraska</Address1_State>
<Address1_Country>USA</Address1_Country>
<Address1_PostCode>90210</Address1_PostCode>
<Address1_Type>Home</Address1_Type>
<Address2_Line1>Suite 10</Address2_Line1>
<Address2_Line2>1090 Eggplant Blvd</Address2_Line2>
<Address2_Suburb>Washington</Address2_Suburb>
<Address2_State>Nebraska</Address2_State>
<Address2_Country>USA</Address2_Country>
<Address2_PostCode>90211</Address2_PostCode>
<Address2_Type>Work</Address2_Type>
</ns0:AccountDetails>

There were some warnings about multiple mappings, but I didn’t find the absence of a looping functoid to be a problem. I realise this is a pretty obvious and simple mapping, but when I went looking for examples about how to solve this type of mapping, I couldn’t find much at all (including from e-books).

There’ll be more on BizTalk mapping and other fun stuff in some upcoming articles.

Deploying BizTalk 2010 Assembly Dependencies

BizTalk Server, Infrastructure, Tips and Tricks No Responses »

Jan 062012

Introduction

From time to time, it is natural for us to deploy BizTalk solutions with Assembly dependencies. Usually this is because we have created helper classes (for mapping, or querying) or perhaps it reuses some common logic shared between application suites.

Whatever the reason, it can be very handy to pick up all assemblies when we export the BizTalk Application from BizTalk Administration, when we use the Export MSI feature/wizard.

The Problem

The one drawback is that, since BizTalk requires assemblies to be GAC’d (that is, strong named and added to the Global Assembly Cache), it’s hard for a tool, like the export to MSI wizard, to know what are custom assemblies which the BizTalk assemblies rely on (that aren’t system or framework assemblies).

In other words, you need a way to call out the dependencies, so that the export wizard will package the entire application. The alternative would be that you would have to ensure any custom dependencies are already deployed to your (new) target environment.

The Solution

As you might be aware, you can view a BizTalk application’s resources (assemblies) from within BizTalk Administrator. What you may not know is that you can add “Resources” to this location. In particular, you can add (or call out) dependency assemblies.

Just right click in the Resources view, and select Add->Resources..

A dialog pops up and allows you to browse for the required assemblie(s). You have some options here, you can force the assemblies to overwrite, you can select if and when the chosen assemblie(s) are GAC’d (registered with the GAC), registered as COM components or made visible to COM components.

The dependencies tab allows you to quickly see if all the dependencies for a specific assembly are to be found. This will help you check and ensure your solution has everything it needs.

When you decide to do an “Export to MSI”, you’ll notice that it now includes any dependency resource Assemblies which you have added to your application:

If the assembly you are adding has a dependency on another assembly that is not included in the application, the add operation will fail.

Great, but what if I want to automate this procedure as part of an automated deployment?

I’m glad you asked. Naturally, you can script commands which will accomplish this task for you. You could easily script this command, to be included in automated builds or deployments.

Note that, as a general rule, it is not advised to automatically register any assemblies in the Global Assembly Cache of a build machine/build server.

Steps to add assemblies from command line:

Open a command prompt as follows: Click Start, click Run, type cmd, and then click OK.

Note: you may require elevated permissions to accomplish this, you’ll certainly need permissions to administer BizTalk Server as well.
Type the following command, substituting the appropriate values, as described in the following table.

Command Syntax:

BTSTask AddResource [/ApplicationName:value] /Type:System.BizTalk:BizTalkAssembly [/Overwrite] /Source:value [/Destination:value] [/Options:GacOnAdd|GacOnInstall|GacOnImport] [/Server:value] [/Database:value]

Example:

BTSTask AddResource /ApplicationName:MyApplication
/Type:System.BizTalk:BizTalkAssembly /Overwrite
/Source:"C:\BizTalk Assemblies\MyOrchestration.dll"
/Destination:"C:\New BizTalk Assemblies\
MyOrchestration.dll " /Server:MyDatabaseServer
/Database:BizTalkMgmtDb

The following is a complete list of options lifted from MSDN, but they approximate what you can do via the BizTalk Administrator console.

Parameter	Value
/ApplicationName	Name of the BizTalk application to which to add the BizTalk assembly. If the application name is not specified, the default BizTalk application is used. If the name includes spaces, you must enclose it in double quotation marks (").
/Type	System.BizTalk:BizTalkAssembly
/Overwrite	Option to update an existing assembly. If not specified, and an assembly already exists in the application that has the same LUID as the assembly being added, the AddResource operation fails. You can view the LUIDs for the artifacts in an application by using the ListApp Command. If another application depends on the assembly being overwritten, the AddResource operation fails, even when this parameter is specified.
/Source	Full path of the assembly file, including the file name. If the path includes spaces, you must enclose it in double quotation marks (").
/Destination	Full path of the location where the assembly file is to be copied when the application is installed from the .msi file. If not provided, the assembly file is not copied to the local file system during installation. If the path includes spaces, you must enclose it in double quotation marks (").
/Options	GacOnAdd: Specify to install the assembly to the global assembly cache (GAC) on the local computer during the AddResource operation. GacOnInstall: Specify to install the assembly to the GAC when the application is installed from the .msi file. GacOnImport: Specify to install the assembly to the GAC when the application .msi file is imported. You must separate multiple options with a comma.
/Server	Name of the SQL Server instance hosting the BizTalk Management database, in the form ServerName\InstanceName,Port. Instance name is only required when the instance name is different than the server name. Port is only required when SQL Server uses a port number other than the default (1433). Examples: Server=MyServer Server=MyServer\MySQLServer,1533 If not provided, the name of the SQL Server instance running on the local computer is used.
/Database	Name of the BizTalk Management database. If not specified, the BizTalk Management database running in the local instance of SQL Server is used.

Introduction

As I alluded to at the very end of 2011, I’ve been spending more and more time with BizTalk 2010. It’s been a little while since I’ve had to support/build/maintain BizTalk artefacts, but it doesn’t take long to reacquire “the groove”.

To gently thrust you into the dark underbelly of the land of BizTalk development, I thought I’d start off by exploring some fairly basic BizTalk 101 type experiences (and solutions!). This article focuses on generic issues with XML schemas, with an eye to BizTalk’s usage of said schema.

Sadly, some of what you might read here (and later) might smack a tad of ‘What? You didn’t know that?”, but I’m willing to swallow some pride if it helps out other Architects and Programmers.

What you need (to follow along)

This article is (IMHO) decent enough for the generalist (non-BizTalk specialist) and especially may be handy for those unfamiliar in working with XML schemas.

There is a big part where I’m playing with BizTalk’s Developer Tools (in Visual Studio 2010), so if you’d like to give it a go, I’d suggest you pick up a copy of BizTalk Server 2010 Developer Edition (click here for more info on this edition of BizTalk), which – I believe – you need to have to install the developer tools.

If I write subsequent article on BizTalk, the BizTalk developer tools will likely play a significant role, so it might be worth investigating. As always, it’s worth bookmarking the BizTalk Development Center link.

The Joys of Schema

If you know anything about BizTalk, you’ll know that it’s all about the schemas. Everything ends up mapping to a document schema type whether it’s inbound or outbound. Schemas (XSD) can be an acquired taste, especially if you’ve been off doing something more exciting.

Last month, handwritten schemas tripped me up quite a bit. I was used to using Wizards and Adapters to generate a lot of the schemas for me, so when I went to design some basic input and output schemas, I encountered some “fun” issues.

Something funny happened…

Now, I was putting together some very basic sample schema to use as an example for demonstrating a really neat technique with maps (transforms).

Just in case someone else encounters this (or, like me, found it mildly amusing) I thought I’d document some cause and effect of why you might get some head scratching error messages when designing in Visual Studio 2010 with the BizTalk designer.

Given the following (flawed) schema, which I built within Visual Studio:

<?xml version="1.0" encoding="utf-16"?>

<xs:schema xmlns="http://Sandbox.InputSchema" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://Sandbox.InputSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema">

<xs:element name="Root">

<xs:complexType>

<xs:sequence>

<xs:element minOccurs="2" maxOccurs="3" name="Customer">

<xs:complexType>

<xs:sequence>

<xs:element name="Address">

<xs:complexType>

<xs:attribute name="Street" type="xs:string" />

<xs:attribute name="State" type="xs:string" />

<xs:attribute name="Country" type="xs:string" />

</xs:complexType>

</xs:element>

</xs:sequence>

<xs:attribute name="ID" type="xs:integer" />

<xs:attribute name="FirstName" type="xs:string" />

<xs:attribute name="Surname" type="xs:string" />

</xs:complexType>

</xs:element>

</xs:sequence>

</xs:complexType>

</xs:element>

</xs:schema>

Which, when loaded in the BizTalk Schema Designer looks like this:

Has an unfortunately obvious flaw if you consider I was expecting the schema to facilitate the following format of XML data:

<ns0:Root xmlns:ns0="http://Sandbox.InputSchema">

</Customer>

</Customer>

</ns0:Root>

In other words, I was expecting to store multiple “Customer” elements in my XML document.

Debugging

When I loaded the document and tested it in Visual Studio 2010, I received the following, very weird error message: “The element ‘Root’ in namespace ‘http://Sandbox.InputSchema’ has invalid child element ‘Customer’.” – huh?

The problem, I thought, might be related to not expressly prefixing the namespace directive to the child element, so I made some edits.

This resulted in an even more ambiguous error message, funny even:

“The element ‘Root’ in namespace ‘http://Sandbox.InputSchema’ has invalid child element ‘Customer’ in namespace ‘http://Sandbox.InputSchema’. List of possible elements expected: ‘Customer’.”

or, in generic terms:

“The element ‘<element>’ in namespace ‘<namespace>’ has invalid child element ‘<child element>’ in namespace ‘<namespace>’. List of possible elements expected: ‘<child element>’.”

WTF?

You may have to squint to read the text…

At this point, I felt like I was going backwards. So I undid my changes and took a look at the schema definition – which, upon second inspection was clearly wrong.

The Solution

See if you can spot the difference?

My mistake was in not declaring a sequence after the root element. Thus rectified, my XML now matched the fixed schema, and I was able to continue on my merry way.

Filtering in a Transform/Map

This all brings us back to the original intention of my article – to highlight the use of functoids for filtering multiple nodes in a transform.

Given that we want to transform our input schema (from before) into another format, with filtered data, here’s the schema for the output:

The Mapping

We can construct a map which assigns the fields we want per the following example:

A basic map

Now, if we wanted to only take Customers who live in the state of New South Wales, we need to validate each element in the input XML recursively.

A Functoid for all occasions

The easiest means is to use a equality functoid, in this example I’ve used an equals functoid, as per below. You must still map the field directly to the output schema, but also to the functoid – which is, in turn, mapped to the logical root of the target (per the example above).

When the functoid returns a true value, the element is copied to the output. If the result is fale, the entire element (children etc) are skipped. Thus, from the following input XML:

<ns0:Root xmlns:ns0="http://Sandbox.InputSchema">

</Customer>

</Customer>

</ns0:Root>

When run through this map (using trusty ‘Test Map’ functionality), will generate the following output, filtering out that pesky Queensland (QLD) customer:

<ns0:Root xmlns:ns0="http://Sandbox.OutputSchema">

</ns0:Root>

Helpful Support in Visual Studio 2010

Before you start despairing about schema development and BizTalk maps, there are a bunch of very helpful tools built into Visual Studio 2010 (and prior versions).

Generating Schema Instances

Given you have Schema files in your project, you can right click them and select “Generate Instance”. This will create an XML file using intelligent default values, according to the schema definition. These files can be modified and used to test out other artefacts who use this schema.

Validating Schema Instances

If you want to test if a particular XML file is valid, you can do the reverse – by clicking on the schema properties, you can set the “input XML” for a given schema. Then, when right clicking on the Schema in Solution Explorer, you can select “Validate Instance”. This will highlight any issues with your schema.

Testing a Map/Transform

Once you have some valid XML for your schema, you can then use it as test input data for various things including maps.

The following shows how to establish test data parameters for a map within Visual Studio 2010:

When you have configured a map to use a specified XML input, you can then right click on the map (in Solution Explorer) and select “Test Map”.

Other Options – Xml Schema/Data Type Utility

A handy note for future reference – if you have an XML structure in mind, but (like me) you are rusty with your XSD designing skillz, there’s another way forward which may prove to be better.

There is a tool which helps you work with XML and XSD document formats, and it ships by default with a number of Microsoft products and SDKs, it is called “Xml Schemas/DataTypes support utility” or XSD.exe.

Important Disclaimer: In nearly almost all circumstances, it is considered a better practice to produce or design the XML schema first. Generating a schema from XML data can be fraught with peril, especially if your XML sample doesn’t accurately reflect the full spectrum of supported formats.

For example, min/max occurrences, type lengths and other constraints, vague data types (where the type or precision is “guessed”), elements supported by the schema which are not required – and thus missing from your sample XML – to name but a few.

Where Do I Find This Tool?

Chances are that you have it already. I’ve located it installed in the following locations (check the (x86) folder as well if you are on a 64 bit machine):

Directory of C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin

Directory of C:\Program Files\Microsoft SDKs\Windows\v7.0A\Bin

Directory of C:\Program Files\Microsoft SDKs\Windows\v7.0A\Bin\NETFX 4.0 Tools

Directory of C:\Program Files\Microsoft SDKs\Windows\v7.0A\Bin\x64

What does it do?

Keeping in mind that this utility was really meant for the days of evil DataSets, it still can be quite useful.

Using the following (poor) sample XML structure:

<?xml version=”1.0” encoding=”utf-8”?>
<Country Name="Australia">
     <State Name="NSW">
           <Capital Name="Sydney"/>
     </State>
     <State Name="QLD">
           <Capital Name="Brisbane"/>
     </State>
     <State Name="WA">
           <Capital Name="Perth"/>
     </State>
     <State Name="TAS">
           <Capital Name="Hobart"/>
     </State>
     <State Name="VIC">
           <Capital Name="Melbourne"/>
     </State>
</Country>

You can use the XSD.exe utility to generate an approximation of the schema, like so:

C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin>xsd Input.xml

Microsoft (R) Xml Schemas/DataTypes support utility

[Microsoft (R) .NET Framework, Version 2.0.50727.3038]

Writing file ‘C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\Input.xsd’.

Which would produce the following output:

<?xml version="1.0" encoding="utf-8"?>

<xs:schema id="NewDataSet" xmlns="" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">

<xs:element name="Country">

<xs:complexType>

<xs:sequence>

<xs:element name="State" minOccurs="0" maxOccurs="unbounded">

<xs:complexType>

<xs:sequence>

<xs:element name="Capital" minOccurs="0" maxOccurs="unbounded">

<xs:complexType>

<xs:attribute name="Name" type="xs:string" />

</xs:complexType>

</xs:element>

</xs:sequence>

<xs:attribute name="Name" type="xs:string" />

</xs:complexType>

</xs:element>

</xs:sequence>

<xs:attribute name="Name" type="xs:string" />

</xs:complexType>

</xs:element>

<xs:element name="NewDataSet" msdata:IsDataSet="true" msdata:UseCurrentLocale="true">

<xs:complexType>

<xs:choice minOccurs="0" maxOccurs="unbounded">

<xs:element ref="Country" />

</xs:choice>

</xs:complexType>

</xs:element>

</xs:schema>

Now, given I mentioned that bit about the DataSet support.. You’ll need to remove the element called “NewDataSet”. This is included because the utility assumes you want to generate a schema for use with DataSets.

Removing the element, renders a nice XSD in Visual Studio:

But wait.. There’s more

Given a valid XSD schema, you can now use the same utility (isn’t it wonderful?) to generate valid C# classes by using the following syntax:

C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin>xsd Input.xsd /c

Microsoft (R) Xml Schemas/DataTypes support utility

[Microsoft (R) .NET Framework, Version 2.0.50727.3038]

Writing file ‘C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\Input.cs’.

The result of executing this command produces the following classes (which you could now serialize into and out of XML):

Summary

You may not be able to teach a dog new tricks, but I’m willing to bet he remembers a bunch of old ones!

Although I’m a bit rusty, I’m able to overcome obstacles reasonably quickly. This article might not make me look like Einstein, but I thought I’d write up some of my experiences in the hope it helps others.

We’ve only just touched the tip of the iceberg with BizTalk 2010. The development environment is challenging and prickly (which makes for great sources of inspiration for blog articles) so you can be sure there will be more here soon, so check back!

Have you met……… BizTalk?

BizTalk Server, Projects, Technology No Responses »

Dec 292011

Hi Everyone,

Well, it’s nearly the end of the year 2011, and hasn’t it been a doozy? I don’t know about you, but my year has been haywire, to say the least. I started the year living in China, moved back to Australia, set up in Sydney and then finished the year in Canberra (the capital of Australia).

With all that going on, you’d think I’d be hard pressed to find time to write anything substantiative. Of course, you’d be right. Therefore, I’m aiming to write at least one more article of value before the year is out. Since the bulk of the work I did this month was BizTalk related, I’m going to write a little bit about some problems I had to solve with BizTalk 2010.

I realise that I had previous written about a few other topics I wanted to cover off.. and I will.. but for now, BizTalk is the most convenient topic, so please bear with me Smile

There’s also quite a lot to look forward to in 2012. For one thing, we’re expecting the next version of SQL Server (codename Denali) to drop – that should be a biggie – and there are the usual soft spots (WCF, Web Services, IIS) and so on.

Hopefully, you’ll see my BizTalk article before the new year, but in the meantime.. thanks for reading this year and all the best for 2012!

Rob Sanders: Sanders Technology

BizTalk 2010 Rollup – Sequence to Flat File Mapping

Deploying BizTalk 2010 Assembly Dependencies

Introduction

The Problem

The Solution

Great, but what if I want to automate this procedure as part of an automated deployment?

Further Reading

Schema Fun with BizTalk Server 2010 (and more!)

Introduction

What you need (to follow along)