If packer fails to delete a machine the execution is blocked indefinitely

[fernflower]

Given that IBM Cloud can be unstable at times and vm cleanup may not be finished due to some cloud-related underlying problem we should not be forever blocked imo.

Currently if packer guest termination fails, the packer run will be blocked with

==> powervs: Capturing Instance
    powervs: Job state: queued, progress: , message:
    powervs: Sleeping for 5m0s
    powervs: Job state: completed, progress: capture, message: capture vm has successfully completed
==> powervs: Pausing after run of step 'StepCaptureInstance'. Press enter to continue. 
==> powervs: Pausing before cleanup of step 'StepCaptureInstance'. Press enter to continue. 
==> powervs: Pausing before cleanup of step 'StepPrepare'. Press enter to continue. 
==> powervs: Pausing before cleanup of step 'StepProvision'. Press enter to continue. 
==> powervs: Pausing before cleanup of step 'StepConnect'. Press enter to continue. 
==> powervs: Pausing before cleanup of step 'StepCreateInstance'. Press enter to continue. 
==> powervs: Deleting the Instance
    powervs: VM still exists, state: SHUTOFF
    powervs: VM still exists, state: SHUTOFF
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR
    powervs: VM still exists, state: ERROR

...

[mkumatag]

@fernflower, can you provide more information on why this VM got into that state? I've noticed that this issue occurs after deleting the VM. Please run ibmcloud pi ins get to gather more details.

btw: in this case, the cleanup job failed towards the end of the workflow, but the image should still be consumable even if the tool fails to terminate.

I need to find a solution to address this situation and timeout with an error instead of entering an infinite loop.

[fernflower]

Sure, from the vm details

Fault Details              Failed to terminate instance: Failed to unplug virtual interface: HTTP error 500 for method DELETE on path /rest/api/uom/LogicalPartition/6DE81A92-D225-46FD-941D-B7CE9CC0A6D5/ClientNetworkAdapter/3843f387-25e3-3ac7-9d09-1e279437f045: Internal Server Error
Last Health Update         2025-06-16T10:34:10.894043
Health Reason              vm_state - error
Health Status              CRITICAL

I filed a support ticket with ibm, to sum up - in order to proceed with the deletion they had to reset the machine from their end, so there is nothing we can really do. For now told us to treat it as a fluke.

But a timeout instead of infinite loop would be a nice to have :)

[fernflower]

Just in case - I managed to hit the issue once again (a bit different cause of ERROR though, but behavior is the same)

Fault Date                 2025-06-18T13:33:39.000Z
Fault Message              Failed to terminate instance: HTTP error 500 for method POST on path /rest/api/uom/VirtualIOServer/14065DA8-F60C-4EAB-8FBC-2BFC59F4E39A?group=ViosFCMapping,ViosSCSIMapping,ViosStorage: Internal Server Error -- Device 'vfchost9' is currently busy.  Try aga
Fault Details              Failed to terminate instance: HTTP error 500 for method POST on path /rest/api/uom/VirtualIOServer/14065DA8-F60C-4EAB-8FBC-2BFC59F4E39A?group=ViosFCMapping,ViosSCSIMapping,ViosStorage: Internal Server Error -- Device 'vfchost9' is currently busy.  Try aga