BizTalk DevOps: Monitor your BizTalk environment using PowerShell – Manage messages being archived locally into the hard drive

Is very common in integration scenarios to see messages being archived locally into the hard drive – either by using a pipeline component like BizTalk Archiving – SQL and File (an old component that was available on Codeplex) or by simply using the default functionalities in BizTalk like filters:

This can happen for several reasons, for example:

  • The messages need to be archived for legal reasons (normally (recommended) we use an “external” repository – not a local hard drive in the server)
  • or because it is practical and useful to have the file so it can easily be reprocessed in the case of failures

The common problem with some of these approaches is that normally we tend to implement the archiving mechanism and we forget to implement a cleaning mechanism and we end up with several problems after a while:

  • The most critical –> disk full, causing processes to fail or other problems even more critical
  • And, according to some articles, having many files in a folder, will cause performance problems when we are trying to perform operations on it. The lookup time of a directory increases proportionally to the square of the number of entries and performance seems to drop between 1000 and 3000 files (the important is not the exact number itself but knowing that a high number may cause problems).

And if you have an integration process that processes hundreds of messages, you can quickly get this kind of problem.

In my demo scenario we have a common Message Archive folder:

  • D:\BizTalkApplications\Archive

This folder is organized by sub-folders describing the processes and the communication, for example:

  • System A
    • Inbound
    • Outbound
  • System B
    • Inbound
    • Errors
  • Canonical Messages
    • Inbounded
    • Archived

And so on… that I need to monitor and constantly clean it.

So how can PowerShell help us?

With this script, you can be able to automatically monitor these folders on the servers using PowerShell.

This script allows you to set:

  • The numbers of hours, minutes or days that you want to archive the files, leaving only the recent ones – if the creation time of the file is older than this parameter then it will be archived and deleted from the folder
 $Hours = "12"
 $LastWrite = $Now.AddHours(-$Days)
 
  • The directory you want to monitor
 $Directories = Get-ChildItem "D:\BizTalkApplications\Archive" -Recurse | ?{ $_.PSIsContainer } | Select-Object FullName
 
  • And for each subdirectory, it will check if exist files to be archived
    • if so, it will create a zip file, using the 7-zip tool, in a different location “BackupFileWarehouse”
    • Otherwise, will not create any file and move to the next directory
foreach ($directory in $Directories)
{
    $zipFile = "E:\BackupFileWarehouse\" + $directory.FullName.Substring($directory.FullName.LastIndexOf(":")+2).Replace("\","_") + "_" +  (get-date -f yyyyMMdd_hhss).ToString() + ".zip"

    $filelist = get-childitem $directory.FullName |
                                where-object {$_.LastWriteTime -le (get-date).AddHours(-$Hours)} |
                                where-object {-not $_.PSIsContainer}

    if($filelist.Count -gt 0)
    {
        $filelist | format-table -hideTableHeaders FullName | out-file -encoding utf8 -filepath lastmonthsfiles.txt
                    & 'C:\Program Files (x86)\7-Zip\7z.exe' a $zipFile `@lastmonthsfiles.txt

        $filelist | %{Remove-Item $_.FullName }
        rm lastmonthsfiles.txt
    }
}

Of course, if this script will not move/create the archived files in a shared location or something similar, then you will need an additional mechanism to maintain and delete these zip files

Requirements

Download

THIS POWERSHELL IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND.

You can download Manage messages being archived locally into the hard drive with PowerShell from GitHub here:

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.

1 thought on “BizTalk DevOps: Monitor your BizTalk environment using PowerShell – Manage messages being archived locally into the hard drive”

  1. Very timely. I was just thinking of putting together something like this. As always your posts are very useful. 🙂

Leave a Reply

Your email address will not be published.

BizTalk360
BizTalk Server

Over 650+ customers across
30+ countries depend on BizTalk360

Learn More
Serverless360
Azure

Operate efficiently with enterprise-grade Azure monitoring,
tracing, remediation & governance in one platform

Learn More

Back to Top