BizTalk DevOps: Monitor your BizTalk environment using PowerShell – Manage messages being archived locally into the hard drive

Is very common in integration scenarios to see messages being archived locally into the hard drive – either by using a pipeline component like BizTalk Archiving – SQL and File (an old component that was available on Codeplex) or by simply using the default functionalities in BizTalk like filters:

• Create a Send Port with a filter expression equal to the name of Receive Port or type of the message you want to archive. (How to Configure Filters for a Send Port)

This can happen for several reasons, for example:

• The messages need to be archived for legal reasons (normally (recommended) we use an “external” repository – not a local hard drive in the server)
• or because it is practical and useful to have the file so it can easily be reprocessed in the case of failures

The common problem with some of these approaches is that normally we tend to implement the archiving mechanism and we forget to implement a cleaning mechanism and we end up with several problems after a while:

• The most critical –> disk full, causing processes to fail or other problems even more critical
• And, according to some articles, having many files in a folder, will cause performance problems when we are trying to perform operations on it. The lookup time of a directory increases proportionally to the square of the number of entries and performance seems to drop between 1000 and 3000 files (the important is not the exact number itself but knowing that a high number may cause problems).

And if you have an integration process that processes hundreds of messages, you can quickly get this kind of problem.

In my demo scenario we have a common Message Archive folder:

• D:\BizTalkApplications\Archive

This folder is organized by sub-folders describing the processes and the communication, for example:

• System A
  • Inbound
  • Outbound
• System B
  • Inbound
  • Errors
• Canonical Messages
  • Inbounded
  • Archived

And so on… that I need to monitor and constantly clean it.

So how can PowerShell help us?

With this script, you can be able to automatically monitor these folders on the servers using PowerShell.

This script allows you to set:

• The numbers of hours, minutes or days that you want to archive the files, leaving only the recent ones – if the creation time of the file is older than this parameter then it will be archived and deleted from the folder

 $Hours = "12"
 $LastWrite = $Now.AddHours(-$Days)
 

• The directory you want to monitor

 $Directories = Get-ChildItem "D:\BizTalkApplications\Archive" -Recurse | ?{ $_.PSIsContainer } | Select-Object FullName
 

• And for each subdirectory, it will check if exist files to be archived
  • if so, it will create a zip file, using the 7-zip tool, in a different location “BackupFileWarehouse”
  • Otherwise, will not create any file and move to the next directory

foreach ($directory in $Directories)
{
    $zipFile = "E:\BackupFileWarehouse\" + $directory.FullName.Substring($directory.FullName.LastIndexOf(":")+2).Replace("\","_") + "_" +  (get-date -f yyyyMMdd_hhss).ToString() + ".zip"

    $filelist = get-childitem $directory.FullName |
                                where-object {$_.LastWriteTime -le (get-date).AddHours(-$Hours)} |
                                where-object {-not $_.PSIsContainer}

    if($filelist.Count -gt 0)
    {
        $filelist | format-table -hideTableHeaders FullName | out-file -encoding utf8 -filepath lastmonthsfiles.txt
                    & 'C:\Program Files (x86)\7-Zip\7z.exe' a $zipFile `@lastmonthsfiles.txt

        $filelist | %{Remove-Item $_.FullName }
        rm lastmonthsfiles.txt
    }
}

Of course, if this script will not move/create the archived files in a shared location or something similar, then you will need an additional mechanism to maintain and delete these zip files

Requirements

• 7-Zip utility tool

Download

THIS POWERSHELL IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND.

You can download Manage messages being archived locally into the hard drive with PowerShell from GitHub here: