Article “Microsoft BizTalk Server seen by the programmer’s eyes”

Posted: November 1, 2011  |  Categories: BizTalk
Tags: BizTalk

This is the article that I published in the magazine “Programar” which can be found here (in Portuguese) and that I now decided to share with you in English…

Much has been said already about the BizTalk Server platform, what it is, and the advantages offered to the organizations. For the more distracted ones, BizTalk Server is the Microsoft platform of excellence for the integration of systems and business processes.

But what are the benefits that this platform offers to programmers (or developers)?

Job Market

Where the product is used? In what contexts? What are the job opportunities it offers? Remuneration? These are some questions that a programmer wants to know before dedicating himself to a technology/product.

Microsoft BizTalk Server 2010 is the seventh version of the product, making it one of the most mature and stable products from Microsoft.

BizTalk Server versions

Over the years the product has become a benchmark success between the integration systems, being currently one of most used product in the world in its area, as demonstrate the following facts

  • More than 10,000 customers all over the world
  • 81% of the “Top 100” Fortune Magazine
  • 12 of the 15 largest Retailers in the World run Microsoft BizTalk
  • 5 of 10 largest Hotel Chains in the World with over 2 Million rooms use Microsoft BizTalk
  • 9 of 10 largest U.S. Telecommunications Companies use Microsoft BizTalk
  • 6 of the 8 largest U.S. Pharmaceutical Companies use Microsoft BizTalk
  • 4 of the 5 largest U.S. Electronics Parts Manufacturers use Microsoft BizTalk
  • 9 of the 10 largest Aerospace and Defense Companies in the U.S. run Microsoft BizTalk
  • 5 of the 8 largest U.S. Chemical Companies run Microsoft BizTalk
  • 4 of the 5 largest Railroads in the U.S. run Microsoft BizTalk
  • 9 of the 10 largest Insurance Companies in the World run Microsoft BizTalk
  • 23 of 27 EU member governments use Microsoft BizTalk to provide more efficient government services

With a robust messaging infrastructure, dehydration and rehydration functionalities, more than 25 multi-platform adapters, rules engine (BRE), ability to obtain performance information on critical business processes, debug, persistence, treatment and error recovery, transactions, and many more features.

Makes BizTalk Server a tool and infrastructure unique, ideal to be used primarily for Enterprise Application Integration (EAI), Business to Business (B2B) Integration and Business Process Management (BPM) solutions.

BizTalk Server B2B EIA

Regarding job opportunities, we can define 3 profiles:

BizTalk Architect: knower of the entire system integration: backup and recovery plans, security, logging, message flow, communication interface… Using his knowledge as well as a programmer, the architect should know the capabilities and limitations of the tools at his disposal (BizTalk engine, BAM, BRE) and design projects making the best choices for each situation.

BizTalk Developer: implements and extends the basic functionalities, taking advantage of the different tools. Here there are many areas completely orthogonal and a programmer may not master them all at the same level: Orchestration, Adapters, Pipelines, Mappings, Functoids, Routing, Rules, Tracking, OLAP, and much more.

BizTalk Administrator: a system administrator will have other concerns such as the health of servers and their activity (HAT – Health and Activity Tracking), he has overall responsibility for the BizTalk Environment, such as installation, configuration, and system maintenance, deploying and managing BizTalk Applications, monitoring (unlocking messages and processes or ensuring proper flow of messages), disaster recovery

Here are some real examples of job opportunities (source LinkedIn):

  • BizTalk Developers needed in: Miami, NYC, Jacksonville, Austin, Sacramento, Louisville, Hawaii, Panama City, Montevideo, London, Toronto, Preston, Porto.
  • BizTalk Administrator needed in Ft. Lauderdale FL 75K – 100K depending on experience.
  • BizTalk Developer/Architect needed for International Law Firm in NY. 110k-130k.
  • 6 Month BizTalk 2009 Consultant required for Public Sector project in London. Rates negotiable but c£500-£550 per day.
  • 2 BizTalk Developer Needed in NYC – Healthcare, mid-level $95-110K and senior-level $105-120K.

Features available to programmers

There is nothing that BizTalk do that can be done with code written from scratch! The question is how long it takes to implement a solution without using a Middleware like BizTalk?

You can easily make an integration between two systems with the use of communication technologies and data transport such as Microsoft WCF or Web Service, but when we look more closely at some of the requirements, is easy to identify that some challenges can become quite complex: implement persistence, correlation of asynchronous messages, recovery in case of failures, complex mappings, atomicity, long-running transactions, monitoring and visibility of the process, … and all this is reflected in huge implementation costs.

It is here that BizTalk enters and does what he does best, because all these features are available “out of the box” with the product, thereby freeing the developers the necessity of having to reimplement those features again and again and thereby allowing to have focus in the most critical aspect: the implementation of business logic associated with integration flows.

There are several features or modules that I could list. Some help simplifying interoperability, others reducing other costs in the implementation:

BizTalk Orchestration Designer

Before learning to program is taught to all programmers how to represent all the steps necessary for the execution of a process in the form of a flowchart because it is more readable and illustrates in a simplified manner the progress of implementation and execution.

BizTalk includes an Orchestration Designer, integrated into Visual Studio that enables developers to represent the business process in a visual way (association of links between shapes, representation of ports and some configurations) making it easier to manage and read than textual language (C#, Java).

BizTalk Server Orchestration Designer

BizTalk Expression Editor

BizTalk Expression Editor allows entering .NET code, with IntelliSense support, in inside orchestrations. We can put all the code within the orchestrations or we can invoke external libraries, which in some scenarios are quite useful, eg: manipulate values of the messages inside Message Assignment shape, manipulate variables inside Expression shape, construction of Boolean expressions in Loop and Decide shapes, set pause time in Delay Shape or even configure dynamic send ports.

BizTalk Server Expression Editor

Rule Engine (BRE)

BizTalk Server includes the Business Rules Framework that allows us to create and isolate rules of a business decision. The primary modules include the Business Rule Composer for constructing policies, the Rule Engine Deployment Wizard for deploying policies created in the Business Rule Composer, and the Run-Time Business Rule Engine that executes policies on behalf of a host application.

The task of creating and changing the rules may not be the programmer. Analysts, consultants or those responsible for the business can create and update them at any time. The programmer can then reuse the business rules in their orchestrations to support a variety of scenarios, eg: to determine the execution path of a business process or value to apply to a transaction. More than one application parameter, this engine allows the value or condition being completely dynamic and determined only at runtime context.

BizTalk Server Rule Engine (BRE)

Using rules that change constantly rather than code, allows you to avoid having to redo their applications systematically.

For more information: The Business Rules Framework

BizTalk Mapper Designer

BizTalk includes a Mapper Designer, integrated into Visual Studio, which enables us to perform transformations of complex messages in a visual and extremely simple way.

BizTalk Server Mapper Designer

This tool allows us to map elements from one schema to another, use predefined functions to transform values (functoids), using custom XSLT transformations, custom .NET/C#, COM, VBscript code or using external XSLT, but the use of these options rely heavily on the experience of the programmer.

In fact, this editor is generating an XSLT file that can be used in other .NET (non-BizTalk) applications.

Transactions, Exceptions and Data Persistence

It is common in Service-Oriented Architecture (SOA) or Business Process applications, where processes can cover several endpoints (systems), that the operations cannot be handled within a short period of time, which leads that ACID transactions are not the most appropriate for this type of scenario.

Unlike traditional programming, BizTalk Server supports two distinct types of transactions:

  • Atomic: allows a transaction to automatically return to its previous state, if not completed successfully. We can implement orchestrations to support ACID (Atomic, Consistent, Isolated and Durable) by configuring the Scope, or the orchestration itself, as atomic.
  • Long-Running: These processes can remain active for days, weeks or longer periods of time, can contain multiple transactions or nested transactions, and allows us to treat exceptions for failure recovery. Supports consistency and durability.

Among several options, you can configure the type of transaction at the orchestration level:

transactions at the orchestration level

As set the transaction to the Scope level in order to encapsulate a unit of work behind a transaction context:

transaction at the Scope level

We have also at our disposal the possibility to define multiple blocks or Handlers for the treatment of exceptions:

multiple Handlers treatment exceptions

When dealing with business processes, particularly the long-term process, we always need to talk about the persistence of data in order to safeguard from fault and in order to reprocess them from a given point in the process. The orchestration engine persists automatically the state of running instances of orchestrations based on the design made by the programmer, existing some events or steps in which is triggered the operation persistence, called persistence points:

  • Persistence at the BizTalk Engine level: When an Orchestration Instance is suspended, when the system shutdowns in a controlled manner, when the engine determines it wants to dehydrate or when an orchestration instance is finished
  • Persistence in Orchestration level: When the end of a transactional scope is reached (atomic or long-running), at the execution of other orchestrations through the Start Orchestration shape, at the Send shape, or at debugging breakpoints

Each persistence point hits the database to store the current instance state. The state includes the internal state of the engine, including its current progress, the state of any .NET components that maintain state information and are being used by the orchestration and the message and variable values.

For more information: BizTalk Orchestration – Understanding Persistence points

Adapters

Includes over 25 multi-platform adapters that simplify the integration with Line of Business (LOB) Applications (such as Siebel, SAP, JD Edwards, Oracle, and Dynamics CRM), database (Microsoft SQL Server, Oracle, DB2) and other technologies (Tibco, Java EE).

BizTalk Server Adapters Feature

WCF LOB Adapter SDK

Provides a rich programming model for developing adapters based on Communication Foundation. (WCF LOB Adapter SDK)

Powerful and complete support for EDI/AS2

Electronic Data Interchange (EDI) is one of the most common forms of electronic communication between organizations (invoices, orders, debit notes). These standards (EDIFACT, ASC X12) are designed for reading electronic, and therefore inappropriate for human reading, reflecting the exchange of structured business data between systems using a standardized data format that ensures the reliability of data through different checksums.

  • The BizTalk EDI Application contains artifacts (including pipelines, orchestrations, and schemas) that are needed to process EDI documents.
  • The BizTalk EDI Receive Pipeline parses EDI-encoded documents, splits EDI batches, converts the EDI-encoded documents into XML encoding, performs EDI and XSD validation, and performs HIPAA X12 sub-document splitting. For more information, see EDI Receive Components.
  • The BizTalk EDI Send Pipeline converts XML documents into X12 or EDIFACT encoding, serializes EDI-encoded documents, and performs EDI and XSD validation. For more information, see EDI Send Components.
BizTalk Server EDI Pipelines
  • The batching orchestration batches EDI interchanges and sets context properties for sending of the batched interchange. The routing orchestration handles the instances in which messages match multiple batches, creating as many copies of the message as required. For more information, see Processing Incoming Batches and Batching Outgoing EDI Messages.
  • The status reporting user interface provides a comprehensive status of EDI interchanges and correlated acknowledgments. For more information, see EDI and AS2 Status Reporting.
  • A schema repository includes X12, EDIFACT, HIPAA X12N 4010A XSD, EANCOM, and control schemas. For more information, see EDI Document Schema Support.
  • A migration tool (Party Migration Tool) enables you to migrate EDI party data from BizTalk Server 2006 R2 or BizTalk Server 2009 to BizTalk Server 2010. For more information, see Migrating EDI Artifacts from a Previous Version of BizTalk Server.
  • The Trading Partner Management (TPM) user interface enables you to set processing properties for trading partners engaging in EDI document exchange and AS2 document transport. For more information, see The Role of Agreements in EDI Processing and EDI and AS2 UI Help.
Trading Partner Management

For more information:

Business Activity Monitoring (BAM)

Business Activity Monitoring or BAM provides visibility on business processes by tracking process milestones and business data (KPIs). BAM allows business users and decision-makers to gain insight into the current health of in-flight processes they are responsible for. Up-to-date information is accessible to users via Microsoft Office BI tools or a designated portal and through automatic business alerts. This allows users to streamline and improve daily business operations by proactively escalating issues and mitigating problems sooner.

Business Activity Monitoring tools

These tools allow developers to be free of the concern, at the time of development, of what information you want to collect and from the implementation of the monitoring process because analysts can develop your model with an extremely familiar tool (Microsoft Excel) and the BizTalk administrator can then connect the model with the processes using the Tracking Profile Editor tool.

Business Activity Monitoring tools

The big benefit is that BAM provides visibility into business processes. What may be ideal to involve those responsible for the business in the active operation of the processes, creating an important feedback loop for continuous improvement of process automation (BPA).

Since the implementation of the BAM is often only the definition of the model and the whole implementation is automated (generators), the effort to include BAM is very simplified for programmers.

BizTalk WCF Service Publishing Wizard and BizTalk WCF Service Consuming Wizard

Even when a process must be exposed as a Web service, there are wizards that help us to create both the WCF projects, as well as the installation of these on your Web server (IIS). The same is true when we want to include in our processes, calls to other Web Services.

These tools allow developers to get away from repetitive programming, streamlining the integration process.

BizTalk Flat File Schema Wizard

One of the older patterns for the exchange of messages is the use of text files (Flat Files) such as CSV or TXT files, many of them custom-made for their systems. But with the adoption of XML as the standard in the message exchange, it is often necessary to transform text files into XML and vice versa.

BizTalk Flat File Schema Wizard tool allows us to easily and visually make the transformation of positional text files:

HEADERXXXXXXXXXXXXXXXXXXXXXXX
BODYXXXXXXXXXXXXXXXXXXXXXXXXX
BODYXXXXXXXXXXXXXXXXXXXXXXXXX
FOOTERXXXXXXXXXXXXXXXXXXXXXXX

transformation of positional text files

Note: Header is in Portuguese “Cabeçalho”, therefore the text from the image is in Portuguese.

Or delimited by symbols:

1999990;1;P0110;1;1;20110307;
1999990;2;P0529;2;2;20110307;
1999990;3;P0530;3;3;20110307;

transformation delimited by symbols

The definition of the rules of parsing is embedded in XSD schema, thereby simplifying the reuse of all these schemes in different parts of the process. At any point, the document can be translated back into flat-file because the definition is declarative and symmetric

Testing, debugging and support for Team Foundation Server

The BizTalk development tools used folders, files, and projects in Visual Studio. So the teams can take advantage of the entire environment ALM (Application Lifecycle Management) collaborating with other programmers, project managers and testers who are part of the team.

Of course, within Visual Studio you will find support for debugging orchestrations, maps (XSLT), or pipeline components, as well to perform unit tests. You will also be able to use many other tools in Visual Studio (Microsoft or other vendors).

Conclusion

As presented in this article, with a focus on developing solutions for enterprise integration, BizTalk Server 2010 simplifies the integration of applications, regardless of their interfaces is on-premise (EAI), with its partners (B2B) or housed in the cloud.

This type of middleware becomes increasingly important when the number of endpoints grows increasingly and the number of traded messages becomes increasingly critical in a market, day by day, more electronic and automated.

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.

8 thoughts on “Article “Microsoft BizTalk Server seen by the programmer’s eyes””

    1. Keep an Eye on the pipeline, I am unsure if this is really a valid point of view considering the talk coming out of Microsoft. I would not suggest getting into this position over any other. 2 years ago that would have been different. If you are new to development there is a lot more room in Azure, SharePoint, Custom app development. Futhermore BizTalk is no longer the “beast” it once was, meaning that you no longer have to find the guys who can stomache being up for days for a simple configuration error. It’s all on MSDN and they have a developer version, which killed the old barrier to entry. Long story short, this will pay the bills today and tomorrow I do not think it will pay the bills next week. Look at AppFabric and Workflow.

      1. Hi,

        This topic would make a long conversation and different people have different views, I respect that.

        I do not want to discuss here if BizTalk will die or not (I hear this story from the first day I started working with this technology) and I am perfectly aware of the Roadmap (http://www.microsoft.com/biztalk/en/us/roadmap.aspx) and Tony Meleg session (http://digitalwpc.com/Videos/AllVideos/Permalink/e821e9f8-e379-45b0-8879-12fe271c86be#fbid=4OjgVgr6sAY), however I leave with you an extract from Richard Seroter’s article (http://seroter.wordpress.com/2011/07/14/is-biztalk-server-going-away-at-some-point-yes-dead-nope/)

        What Does This Mean to Developers?

        …except for BizTalk orchestrations, you’re seeing from Tony’s session that nearly all of the BizTalk-oriented components (adapters, pipelines, EDI management, mapping, BAM, BRE) will be part of the Microsoft integration server moving forward. Investments in learning and building solutions on those components today is far from wasted and immensely relevant in the future. Not to mention that understanding integration patterns like service bus and pub/sub are critical to excelling on the future platforms…

        this just to say that for me, it worth investing in BizTalk… and believe me, there are many demand for BizTalk professionals

Leave a Reply

Your email address will not be published. Required fields are marked *

turbo360

Back to Top