Environment:

My customer is using Cisco AnyConnect not just as VPN client, either for NAC (Network Admission Control) aka. 802.1x authentication. That means, without this AnyConnect software and other requirements (ex. certificate), the workstation is not able to access to the corporate network. So it’s a NOGO, if this client during the upgrade (software/OS) process is not working properly.

Situation:

Following issue has been realised during the Windows 10 feature upgrade (1511 –> 1607). But we had the same problem when the AnyConnect client has to be updated (4.4.04030 –> 4.5.02036) regarding the Krack vulnerability. 

Issues:

• Some Windows 10 clients had a BSOD during the feature upgrade from version 1511 to 1607. Hereby, the upgrade parameter /MigrateDrivers all has been used.
• Cisco AnyConnect client couldn’t be updated from version 4.4.04030 to 4.5.02036. Hereby, the precise issue is, the Network Access Manager Filter Driver (3.1.6010 –> 4.3.5009) couldn’t be renewed.

Workaround by Cisco:

We opened an official Cisco ticket to solve this issue. They mentioned, the software upgrade process has to be straight forward. It means, uninstall old version of AnyConnect client –> reboot the client –> install the new client version –> reboot the client.

It’s not working correctly, because the old driver cannot be uninstalled successfully. The new software version can be installed without any issue, but under the hood the driver cannot be renewed.

Anyway, this workaround is not really a good option in our case. Because as I said, without this AnyConnect client the workstation is definitely offline and ex. cannot continue any software deployment process.

Resolution:

1. Copy the AnyConnectFix.ps1 script locally

   #Define variables
   [string]$acdriverpath  = ${env:ProgramFiles(x86)} + "\Cisco\Cisco AnyConnect Secure Mobility Client\drv"
   [string]$exe           = $acdriverpath + "\acnaminstfd.exe"
   #-----------------------------------------------------------------------------------
   #Stop NAM service
   Stop-Service nam -ErrorAction SilentlyContinue
   
   #Move wrong NAM driver
   Move-Item -Path ($env:windir+"\system32\drivers\acnamfd.sys") -Destination ($env:systemdrive+"\Temp") -ErrorAction SilentlyContinue
   
   #Copy correct NAM sys file
   Copy-Item ($acdriverpath+"\acnamfd.sys") -Destination ($env:windir+"\system32\drivers") -ErrorAction SilentlyContinue
   
   #Copy correct NAM dll file
   Copy-Item ($acdriverpath+"\acnamfdbctl.dll") -Destination ($env:windir+"\system32") -ErrorAction SilentlyContinue
   
   #Start NAM service
   Start-Service nam -ErrorAction SilentlyContinue
   
   #Register new ACNAMFD.SYS driver
   &$exe /F /LM /V /PATH "$acdriverpath\acnamfd.inf"
   
   #Disable scheduled task
   Get-ScheduledTask -TaskName "Enable Net Adapter" | Disable-ScheduledTask -ErrorAction SilentlyContinue
   
   #Reboot force
   Restart-Computer -Force

2. Create a scheduled task which has to be run with highest privilege after the first startup where the workstation is offline.
3. Install Cisco AnyConnect Secure Mobility Client (maybe you have to kill first the VPNUI.EXE process).
   • copy VPN service profile
4. Install Cisco AnyConnect Network Access Manager.
   • copy NAM profile
5. Unregister old NAM driver (ACNAMFD.SYS) with the following script

   #Define variables
   [string]$acdriverpath  = ${env:ProgramFiles(x86)} + "\Cisco\Cisco AnyConnect Secure Mobility Client\drv"
   [string]$acnamfdsys    = Select-String -Path "$env:windir\inf\*.inf" -pattern acnamfd.sys | Select-object -first 1 | select -ExpandProperty Path
   [string]$exe           = "$acdriverpath\acnaminstfd.exe"
   #-----------------------------------------------------------------------------------
   & $exe /U /D /V /PATH "$acnamfdsys"

6. Configure Wired AutoConfig (Dot3svc) service start for demand.
7. Configure Cisco AnyConnect Network Access Manager (nam) service stat for auto.
8. Configure Cisco AnyConnect Network Access Manager Logon Module (namlm) service stat for auto.
9. Install Cisco AnyConnect Diagnostic and Reporting Tool
10. Reboot
    • Starting the AnyConnectFix.ps1 upgrade script at the startup
11. Unregister scheduled task and cleanup the left-over files

This workaround has been officially confirmed by Cisco.

Microsoft case:

For this case Cisco has involved Microsoft as well, because Cisco said, ‘it’s an operating system issue‘. As I have already heard many times…

“… if a migration has every occurred on the machine, numerous artifacts are left behind which will interfere with subsequent installs or upgrades of the AnyConnect VPN’s (or any VPN for that matter).   The combination of various keys varies and cleaning some of them will sometimes work.

For example, we see at least 3 scenarios:

1. ACNAMFDBCTL.DLL is not removed or unregistered. 
2. ACNAMFD.SYS is not removed and service remains active
3. VPNVA is not removed and service remains active (sometimes not active but present)
4. Or any combination of the above (this is what made this complex since the repro was not consistent and varied between machines… what works for one machine may fail on another).

Each of these will cause a failure of subsequent installs of AnyConnect….” 

Conclusion:

Both cases are still open, but our “self designed” upgrade process, we can detect the correct version of each software components and which is more important, the correct NAM filter driver has been successfully registered. It has to be used as workaround, but our Cisco AnyConnect client upgrade for more than 1’400 workstations has been worked properly.

Our customer is happy –> we are happy!