Veeam Restore Issue #VEEAM #BACKUPS #vEXPERT #VMware #vSphere

So recently I came across an issue where Veeam would restore from a backup and it would be marked as successful, but when you went to power the VM on it would fail with a CID mismatch (of course you can fix this manually by messing around in the vmdk files, but you shouldn’t need to do this). On closer inspection of the restored VM, you would see that the VEEAM RESTORE SNAPSHOT was still in place.

In the restore logs I noticed :

Create snapshot, ref “vm-401631”, name “VEEAM RESTORE SNAPSHOT”, description “”, memory “False”, quiesce “False”
[16.02.2018 16:01:37] <01> Info [Soap] Snapshot VM ‘vm-401631’ was created. Ref: ‘snapshot-401632’
[Soap] Reverting snapshot snapshot-401632
[16.02.2018 16:07:53] <01> Info [VimApi] RevertSnapshot, type “VirtualMachineSnapshot”, ref “snapshot-401632”
[16.02.2018 16:07:53] <01> Error Snapshot finalization failed
[16.02.2018 16:07:53] <01> Error RevertSnapshot failed, snapshoID snapshot-401632, timeout 3600000 (System.Exception)
[16.02.2018 16:07:53] <01> Error at Veeam.Backup.ViSoap.CSoapConnection.RevertSnapshot(String snapshoID, Int32 timeout)
[16.02.2018 16:07:53] <01> Error at Veeam.Backup.ViSoap.CAutoReleaseSnapshot.Dispose()
[16.02.2018 16:07:53] <01> Error The request was aborted: Could not create SSL/TLS secure channel. (System.Net.WebException) 

By default I opened  a case with Veeam support and did some digging on my own and in the #vExpert slack. I came across this forum post:

https://forums.veeam.com/veeam-backup-replication-f2/ssl-tls-error-since-veeam-upgrade-to-version-9-0-t34686-15.html

Now this seemed to fit my case, I was running Veeam 9.5U3 on 2008r2, I had only recently upgraded to U3. I had another member of staff mention a similar issue a while ago on U1, but I was unable to replicate the issue. But in U3 it was happening consistently.

Support did get back to me and told me about the reg fix :

Registry path: HKLM SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL
Parameter: ClientCacheTime
Type: REG_DWORD
Value: 0

and then reboot the Veeam server.

Now you could do that or migrate to Windows Server 2012+. So I applied the patched as an interim measure and then used this issue as an excuse to move another server away from 2008r2.

Now this is where I came to a couple of hiccups. I had recently patched the current Veeam server to U3. So when I was creating this new 2012R2 box I downloaded the U3 ISO for a fresh install, the build numbers matched. But when I tried to point the new Veeam install to the SQL db, it kept throwing an error:

Unable to use database VeeamBackup, because it was created with a later version of Veeam Backup and Replication

I found this kb:

https://www.veeam.com/kb1845

Veeam support said that I shouldn’t have had this issue since they are both the same….but such is life. I created a new temp db and installed Veeam from the 9.5 U3 ISO. I then used the update patch for U3, even though I had installed from the U3 ISO, the patch still fully installed.

Then I used the DBConfig utility which is in the folder where you installed Veeam, and pointed Veeam to by existing db. this is where I hit another snag:

Even though the old vm was off, it appeared to be locked, so Veeam Support gave me an SQL query to run to clear it up:
SQL Query to remove lock:
delete from [options] where name = ‘lock_info’

Then I was able to get Veeam up and running again. Very interesting overall.

You could also use a config backup and restore that to the blank db, remember both ways are supported by Veeam, So use whatever floats your boat.

There does seem to be some discrepancy between the 9.5U3 ISO and the U3 patch, the patch must have a newer build version over the U3 ISO.

Now make sure you know all the credentials needed, as regardless of the way you do it, you need to know them and input them again as they do not cross over! I made sure of this before hand, and I came across some credentials that were very old/before my time and no one could remember what they were, and they were not stored securely anywhere as a result. So I used this as an excuse to change them out and use some new secure passwords and made sure they were stored securely away and could be referenced easily.

NOTE:

I had done a full test of all my backup jobs back in December (I hope you guys do too?!) and I never ran into this issue. So it looks like a mixture of Windows patching and/or Veeams update allowed this Windows issue to rear its nasty head.