Last week I have encountered an unusual situation in my BizTalk Server 2010 production environment… when I was trying to restart BizTalk Host Instance after publishing a small change in one of my applications, one of the host instances got stuck in “Stop pending” state.
The major problem of this stage is that we cannot perform any kind of operations on it through the BizTalk Server Administration Console or Service window
Cause
Well, unfortunately, I don’t know exactly why this problem has occurred, but I came across these posts:
- FIX: “Error saving map. Stored procedure returned a non-zero result.” error message when you deploy the BizTalk Server 2010 applications in BizTalk Server 2010 Administration Console.
- Stuck in a SERVICE_STOP_PENDING state when restarting host instances while deploying an application
And I believe that this problem was due to the deploy, unfortunately, and against my suggestions, the client had not installed the latest CU (I think that the last CU installed, was the CU2)
Solution
First of all, I suggest that you install the latest CU available.
But because this was a production environment and I needed to solve the problem quickly the solution to solve this problem is to kill the process via Windows Task Manager! Is not an elegant solution, I know and please avoid doing it.
Then go to BizTalk Server Administration Console or Service window and start the process.
Another precaution that you must have if you have multiple Host Instances, is to know what is the process id of the host instance in order to terminate the correct one, you can accomplish this following the steps in this post:
- How to determine the process ID of BizTalk Host Instances with PowerShell
- or in this Debugging External assembly’s or pipeline components – Attach to Process – which BizTalk process to use?
Related links:
Another useful way to determine the PID of the service is to use the sc queryex {servicename} command which returns like this;
sc queryex BTSSvc$BizTalkServerApplication
SERVICE_NAME: BTSSvc$BizTalkServerApplication
TYPE : 10 WIN32_OWN_PROCESS
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
PID : 11116
FLAGS :
The SC command also supports the continue command
C:>sc continue
DESCRIPTION:
Sends a CONTINUE control request to a service.
USAGE:
sc continue [service name]
This sometimes forces a service in a pending state to complete the pending operation.
Excellent feedback Mike! This is really a more elegant solution than killing the process.Kill the process for me should be the last resource.
Thanks.
The (possible) cause
Stop pending error is reproducible. I have observed it after delivery notification could not have been delivered from send port to caller orchestration (i.e. orchestration stops existing before send port succesfully sends the message). Such send port would not exit properly and leave its service instance “active” forever (this is btw visible in BizTalk administration console). That would then cause issue when you restart host instance (probably BizTalk has a problem serializing that service instance). I have observed such behaviour in FTP send port, while FILE send port works fine in this situation.
1. I have an orchestration with delivery notification set on a send port. Message is sent by the send port using FTP adapter. FTP transmission fails because password is incorrect. Orchestration catches DeliveryException, logs it and then finishes.
In administration console I can still see that the not delivered message is suspended on the send port. I adjust password on FTP server. Then I resume the message. Message is delivered to FTP location succesfully and delivery notification is sent again, but the orchestration is already gone. Service instance remains “active”.
2. You can also observe this error when you have delivery notification in orchestration, but unaware of that you send your message via send port group to 2 send ports (FTP adapter). Then 2 delivery notifications will be created. If both delivery notifications are fast enough your orchestration will exit with 1 non consumed message. But if second notification is created after orchestration has exited there are no subscribers waiting for it anymore and send port remains in “active” state.
BizTalk Server 2006 would suspend such service instance gracefully so I think this issue is actually a bug in BizTalk Server 2010.
The PID is available under Services tab in Task Manager.
Regards
Sonya
PID is also available under Process tab, but you have to select this column. The great advantage of Services tab is that you actually can see the host instance name, this way you can kill the right process in the processes tab… excellent tip Sonya.
Hi Sonya, again get tip but unfortunately Services tab is only available starting from windows server 2008, so if you are using other SO you have to use another approach to check the PID.
Thank you Sandro,
I have used that approach since it is the fasted way to kill a BizTalk instance or find the right process to attach to while debugging. I didn’t know it is a new feature of 2008 OS. Thank you for your note It is really useful.
We had a similar issue in production but with BizTalk 2009 and the latest CU patched. We found that the root cause was a broken connection to SQL whilst stopping the host instance. The only way forward was to reboot the box as killing the process did not work.