A story of having to take drastic action in order to figure out a fairly simple problem.
The issue:
I need to use the company’s VPN to access the servers. I can do literally nothing if I can’t connect to the VPN. One fine Monday morning, I open Cisco AnyConnect, click the button, enter credentials… After the posture assessment, it runs its Hostscan utility for a while and then just goes back to a “Ready to connect” state. I try a couple more times, including trying our secondary connection and nothing. No connection.
I talk to support, get kicked up a couple levels of the support hierarchy, and no one has any concrete ideas. AnyConnect usually just works.
We try uninstalling and reinstalling… We try the very latest version (4.10.05xxx), we go back to 4.6.x (VERY bad idea, btw: I’m on Big Sur, and AnyConnect 4.6 and Big Sur do NOT like each other). I find this blog post from MIT detailing how to remove every last vestige of AnyConnect from your system, and do that before installing each version to test.
No dice. At a certain point, I broke it even further – Hostscan wouldn’t even run, and all I would get is a “User credentials entered” message for a while before actually giving me a failure message.
In the end, few of the details of the messages or the behavior we saw matter other than noting that Hostscan was specifically timing out, not erroring out. If it runs too long, it just stops and goes back to saying “Ready to connect.”
The Solution… Kinda:
By Thursday of that week we had run out of things to try, got fed up, and decided to completely erase my hard drive and start over.
After doing that, we get all the basic company stuff reinstalled, get the required security software installed and set up, and check AnyConnect: Success! We get a connection on the first try. And Hostscan runs really quickly, too.
OK fine, but now I need to get the rest of my stuff back in there. I have a TimeMachine backup (which is very handy, btw) and it being a Mac, I can just drag and drop applications from the backup to the Applications folder of my squeaky clean install.
I do this slowly, and continually check AnyConnect to make sure I haven’t broken anything. And something finally does break it – in the same way as the first time: Hostscan runs for a long while and then the dreaded “Ready to connect” appears.
It was MAMP. For those that don’t know, MAMP is a piece of software that, on a Mac, bundles Apache, MySQL, and PHP to allow you to stand up a local server to run a php-based website locally on your computer. I use it to run a full instance of our website for first-step feature development and bug fixes.
MAMP lives in the Applications folder with everything else, but it is itself a folder, rather than just an app, like “Google Chrome.app”. Within the MAMP folder is another folder called htdocs, and this is where all your files go for the content of your website.
Well, I had been testing various things, and had, like 5 instances of our site in there in various stages. I also had some vanilla Drupal instances in there as well from some Drupal-specfic courses I was taking. All of these also had an associated database that MAMP stores locally. So this MAMP folder was very very large – like 60gb or something. And it seems Hostscan was trying to scan all of it and giving up.
The Solution, for real:
Well, it turns out MAMP doesn’t actually care where you put your files, you can just point the server at whatever folder you want, and it can be one outside of MAMP. More importantly, it can be outside of the Applications folder!
So I moved all my files out of the htdocs/ directory and into something outside of the Applications folder. Hoping that was good enough (I didn’t want to research how to move the databases) I tried AnyConnect, and lo and behold: connection!
So, if you’ve gotten this far and didn’t get the hint from the title: If you are not getting any other messages from AnyConnect, and it seems Hostscan is just timing out and bringing you back to that “Ready to connect” state, check your Applications folder. Something in there might be unnecessarily large, and if you can make it smaller (much smaller) you can get connected again.
Oh and if you’re on a windows machine, sorry, I don’t know anything about AnyConnect on those.
Have you tried turning it off and on again?
Comments