Several customers have observed problems with installing the Windows Jazz Agent using bulk provisioning tools.
- Prior to a reboot, an agent may continue reporting in activity events, but not process actions or
- Agents stop reporting in any events or heartbeat messages, even if the machine is running
- The Jazz Agent can no longer be removed from Add/Remove programs
- Attempting to install a new version of the Jazz agent "over the top" is unsuccessful
- Bulk software provisioning tools, including (but not limited to):
- Group Policy Application Management (GPO)
- System Center Configuration Manager (SCCM)
- Batch processes run as SYSTEM
- Any upgrade or downgrade pushed via these tools
- Any version of the Windows Jazz agent that does not include the MSIRMSHUTDOWN
- Up to and including 4.0.0
- Agents installed manually (or via msiexec) as a standard admin user
- Auto-upgrades pushed from the Jazz platform, assuming they do not then conflict with a bulk provisioning tool and lead to a subsequent downgrade
- Upgrades to agent version 4.0.1 onwards
The problem is caused by the fact that the Agent process has to run as the local SYSTEM user, and that the majority of bulk provisioning tools work by triggering the Windows Installer (msiexec.exe) also running as the SYSTEM user.
- The agent is installed as normal and runs as SYSTEM with a supervisor process (winsuper.exe) and several child processes (agent.exe)
- When an upgrade is triggered as SYSTEM, the installer immediately terminates the winsuper.exe process and service, but does not exit the agent.exe cleanly
- The agent.exe process still has active connections to the Jazz kernel drivers
- The installer attempts to stop the drivers, which refuse as they are still active
- After a 30 second timeout, the installer attempts to roll back the upgrade, but cannot stop the running driver(s)
- The installer fails with a fatal error (1603)
- The existing agent.exe process will keep running and report events back to the Jazz platform until the system is restarted, heartbeats however will be terminated, and agent actions may no longer be processed
- At this point, the drivers will have been deleted and not re-instated by the attempted rollback, meaning the agent will not start
Note: Those users installing or upgrading the MSI package manually, or invoking msiexec as a standard Administrator user are unaffected, as in this case the installer cleanly terminates the agent process and drivers.
The only way to repair a system once it is in this state is to perform a hard purge of all Jazz application and driver binaries, as well as clearing any associated registry entries. Jazz Support can provide either Powershell or Command Prompt batch scripts that can perform these actions.
Once the agent has been fully purged, it should be possible to push a new agent installer and automatically pick up the existing enrollment configuration.
It is important to ensure that any upgrade pushed via a SYSTEM user includes the MSIRMSHUTDOWN=2 flag to msiexec. This ensures that the Windows Restart Manager cleanly terminates the agent process before upgrading. This has been included as standard in all version of the auto-update process, and is included in agent 4.0.1 or later MSI packages.