WARNING: Veeam Backup VDDK error 1 and 2 ESXi 5.5/6 Veeam B&R v8/9

Also read the follow up post: VeeAM Backup VDDK error Follow-up and solution

So recently I came across a strange problem.

Random VMs from various different jobs, would fail to back up, I would get a VDDK error.

Now the last time I had this error, was when I had just joined a company and it turned out to be a faulty disk on a local host that had failed,  and it had been in a failed state for so long that the whole vdisk on the Dell host was compromised and I had a punctured RAID. There was no monitoring in place! rest assured that’s sorted now.

But these were all SAN based, on different datastores and on different SANs.

07/06/2016 00:26:26 :: Processing VMNAME Error: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to read from a virtual disk
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to read from a virtual disk
Unable to retrieve next block transmission command. Number of already processed blocks: [1].
Failed to download disk.

So as any admin would do, I did some more digging, as I was initially worried that this could be something very bad, as it spanned multiple jobs, across multiple datastores.

It turns out  that every single one of the VMs, that wouldn’t backup, were on the same host. The host was up and e, no errors in any logs, the VMs were running fine. Yet Veeam couldn’t back them up.

So a good place to start after pondering over it would be that it could be an agent issue, so I did a services.sh restart, we don’t run LACP here so it wouldn’t cause an issue. Once I had done that I ran the job again, and it finished no problem at all!

I logged a support call with Veeam, and compiled all the logs and sent them across, I then went on the Veeam Forum and it turns out it is a growing problem:

https://forums.veeam.com/vmware-vsphere-f24/repeted-vddk-error-1-and-2-t32723.html

It seems to span across ESXi 5.5 (any build) and ESXi 6 (any build) and Veeam B&R v8 and v9.

The work around seem to be :

1. vMotion the impacted VMs (this moves the Vm to another host, so I’M not surprised this worked)
2. Restart management agents (worked for me)
3. Storage vMotion (works for some people, but not others)

The problem is these are just work arounds to do after the fact, and the issue can happen again. It only impacts jobs that use nbd (network) mode, the general consensus is that it is something to do with ESXi Buffer Cache, and it seems to happen on hosts that are quite loaded? maybe?

I like other people have cases logged with Veeam and with VMware, but it seems Veeam are telling people to contact VMware, and VMware are like there is nothing wrong.

So there is no conclusion yet