I’ve been running a few VMs on Promox, including the Pfsense, for over a year without a single issue. This evening, however, I noticed I wasn’t able to access any local DNS as the VM hosting DNS server was not responding.
On checking the Proxmox console, the VM banan with ID 102 was not responding. I tried to reboot, stop and reset. None of which worked. Knowing that Proxmox is based on or using KVM, I guess there must be many commands that can be run directly on the Promox host.
Proxmox doesn’t make use of virsh, and based on this site [1], we can instead use qm command (qm - QEMU/KVM Virtual Machine Manager) to stop/start VM etc.
First thing I attempted was to stop the VM from the command line after logging into the Proxmox host. I already knew the my VM ID is 102, however, you can list
the VMID from the /etc/pve/.vmlist file.
root@pve:~# cat /etc/pve/.vmlist
{
"version": 1,
"ids": {
"100": { "node": "pve", "type": "qemu", "version": 3 },
"102": { "node": "pve", "type": "qemu", "version": 5 },
"104": { "node": "pve", "type": "qemu", "version": 2 },
"101": { "node": "pve", "type": "qemu", "version": 1 },
"103": { "node": "pve", "type": "qemu", "version": 4 }}
}
To stop the VM with ID 102, I executed the command:
root@pve:~# qm stop 102
trying to acquire lock...
can't lock file '/var/lock/qemu-server/lock-102.conf' - got timeout
From that above error message, my next sensible attempt was to unlock this VM.
root@pve:~# qm unlock 102
trying to acquire lock...
can't lock file '/var/lock/qemu-server/lock-102.conf' - got timeout
Again, it didn’t work. Based on [1], “The timeout error occurs when the virtual machine is locked or the process is still running in the background.”
So what to do? I didn’t have much choice but to kill the kvm process responsible for running this VM with ID 102, but first I needed to find that kvm process ID:
root@pve:~# ps aux | grep "/usr/bin/kvm -id 102"
root 3226657 9.4 39.2 8867996 6372952 ? SLl Feb11 9944:45 /usr/bin/kvm -id 102 -name banan,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/102.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/102.pid -daemonize -smbios type=1,uuid=68b9da26-8ae7-447f-9856-7b4951b8009d -drive if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd -drive if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/dev/pve/vm-102-disk-0,size=540672 -smp 2,sockets=1,cores=2,maxcpus=2 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/102.vnc,password=on -cpu host,+aes,+kvm_pv_eoi,+kvm_pv_unhalt -m 6144 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device vmgenid,guid=73137e8b-2349-4db3-943b-9ffeb480fdfa -device qemu-xhci,p2=15,p3=15,id=xhci,bus=pci.1,addr=0x1b -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device vfio-pci,host=0000:01:00.0,id=hostpci0,bus=pci.0,addr=0x10 -device usb-host,bus=xhci.0,port=1,vendorid=0x13fd,productid=0x3940,id=usb0 -device VGA,id=vga,bus=pcie.0,addr=0x1 -chardev socket,path=/var/run/qemu-server/102.qga,server=on,wait=off,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on -iscsi initiator-name=iqn.1993-08.org.debian:01:108c80afcd6 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/dev/pve/vm-102-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -netdev type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=5E:54:C7:B6:57:5D,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=1024,bootindex=101 -machine type=q35+pve0
root 3637798 0.0 0.0 6376 708 pts/0 S+ 20:56 0:00 grep /usr/bin/kvm -id 102
Now that I had the process ID of kvm process as 3226657, to kill that process, just run:
root@pve:~# kill -9 3226657
Just like that, my VM was back and running again.
Reference: