I have an insane roadmap of things to be published but if I didn’t have this week was fertile in providing me with new content. Full week troubleshooting and fixing issues in several clients… what an amazing week! But the best one arrived almost on the last day.
Do you know the feeling when you develop a solution that has been working properly for a while and stops working for no logical reason? I bet you do. And this happened to me this week at a client where we are using the BizTalk Server Logic App Adapter to extend part of our logical business to the cloud. And without we make any changes to the Logic Apps or any changes in our BizTalk Server environment (aka no new development installed, no patch’s or hotfixes installed, no restarts, …) everything stops to work, we were not receiving any new messages on our Logic Apps.
When I access to BizTalk Server environment, I realize that all the requests were suspended with the following error:
System.ServiceModel.CommunicationException: An error occurred while making the HTTP request to https://prod-….logic.azure.com/workflows/…/triggers/manual/paths/invoke?queryparameters . This could be due to the fact that the server certificate is not configured properly with HTTP.SYS in the HTTPS case. This could also be caused by a mismatch of the security binding between the client and the server. —> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult a
The adapter failed to transmit message going to send port “LOGICAPP_SEND_PORT” with URL “https://prod-….logic.azure.com/workflows/…/triggers/manual/paths/invoke?queryparameters“. It will be retransmitted after the retry interval specified for this Send Port. Details:”System.ServiceModel.CommunicationException: An error occurred while making the HTTP request to https://prod-….logic.azure.com/workflows/…/triggers/manual/paths/invoke?queryparameters. This could be due to the fact that the server certificate is not configured properly with HTTP.SYS in the HTTPS case. This could also be caused by a mismatch of the security binding between the client and the server. —> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
— End of inner exception stack trace —
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
— End of inner exception stack trace —
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)
— End of inner exception stack trace —
Server stack trace:
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndRequest(IAsyncResult result)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.Channels.IRequestChannel.EndRequest(IAsyncResult result)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.RequestCallback(IAsyncResult result)”.
The other curious thing was that all my environments DEV, QA, and PROD were with the same symptom and all of them start at the same time which made this problem even more intriguing.
However, I was able to connect with the adapter, see and select all the Logic Apps in my subscription.
To demystify if that was a problem with the adapter, I switch to the HTTP adapter in order to try. Nevertheless, I got identical problems that made me dismiss the idea that there could be something wrong with the adapter.
The next step was to see if there was a network issue, maybe a firewall or proxy, which should be because there was no intervention in the environment, but it is always worth testing. To do that, I used one of my favorite tools, postman, and to my surprise, everything was working fine. I was able to communicate and send requests to all my Logic Apps in all my environments!
Cause
If you don’t are really familiar with this type of problem, you may be pointed for a certificate issue, but in our case, we were not using any kind of certificates to call our Logic Apps and no additional type of authentication.
But I was sure that I had already seen that error in the past, and I was related to security protocol (TLS 1.0 or TLS 1.2) used on the HTTPS communication and yes HTTPS because all Logic Apps endpoints are in HTTPS.
BizTalk Server came out-of-the-box and works very well with SSL (Secure Socket Layer) 3.0 or TLS (Transport Layer Security) 1.0, and these are the security protocol used. New versions of BizTalk Server allow us to use TLS 1.2, but that required extra manual configurations that we need to do in the environment and I know that everyone is deprecating TLS 1.0 and 1.1 because of the well know vulnerabilities, nevertheless, everything was working fine I have been doing this kind of hybrid solution connecting with Logic Apps.
What I discovered on the official documentation was that they say that: The Request trigger supports only Transport Layer Security (TLS) 1.2 for incoming calls. Outgoing calls continue to support TLS 1.0, 1.1, and 1.2.
Nevertheless, that wasn’t true until a few days ago, and my guess is that they actually discontinue support for TLS 1.0 and 1.1 on the incoming calls without any notice, something that, in my opinion, you shouldn’t do.
Solution
The solution was very simple, we just need to specify that the default security protocol on BizTalk Server is TLS 1.2. And to do that you need:
- To make some changes on the registry to set TLS 1.2 as default security protocol TLS1.2;
- Or creating a Custom BizTalk Server behavior;
I choose the first option, making some changes to the registry. And to accomplish that we need to add the below DWORD values in our registry:
- On the [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client]
- Create the following DWORD (32-bit) values
- Name: DisabledByDefault
- Value Data: 0
- Name: Enabled
- Value Data: 1
- Name: DisabledByDefault
- Create the following DWORD (32-bit) values
- On the [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Server]
- Create the following DWORD (32-bit) values
- Name: DisabledByDefault
- Value Data: 0
- Name: Enabled
- Value Data: 1
- Name: DisabledByDefault
- Create the following DWORD (32-bit) values
- On the [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]
- Create the following DWORD (32-bit) values
- Name: SchUseStrongCrypto
- Value Data: 1
- Name: SchUseStrongCrypto
- Create the following DWORD (32-bit) values
- On the [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319]
- Create the following DWORD (32-bit) value
- Name: SchUseStrongCrypto
- Value Data: 1
- Name: SchUseStrongCrypto
- Create the following DWORD (32-bit) value
If you try resending a message to your Logic App, everything should work properly again.
They indicate last year that they were going to deprecate TLS 1.0 and 1.1 in Azure and asked for feedback. But yes, never saw the announcement that they were doing it either.