should group destroyed be unwatched automatically?

[linsun]

I had a group that I destroyed, but I noticed that it is still being watched... is this expected? I would expect it is auto unwatched when user has requested the group to be destroyed.

[wfarner]

That is not expected. Are you able to put together a scenario we can use to reproduce? I tried myself with a trivial sequence and had no such luck.

[linsun]

The scenario I have was very simple, I deployed a swarm worker group using the terraform instance plugin (for SL), e.g. build/infrakit group watch {path}/group.json then I did build/infrakit group destroy {group-id} to destroy the group. Then I realized my infrakit-instance-terraform terminal still flashes a lot with attempt to do terraform apply/show etc. This is based on my memory, let me know if you need precise steps.

[wfarner]

More precise steps would help; a minimal set of commands i can repeat would be ideal. I'll also be interested in determining which plugin the problem is associated with (group or terraform). While you're observing the behavior, output of build/infrakit group ls would be useful to help narrow down.

[wfarner]

@linsun have you encountered this again? If not, and without steps to reproduce, i suggest we close this until it becomes actionable.

[linsun]

@wfarner sorry for the delay, i tried to recreate this today, here are the steps:

1. launch the following plugins:

# linsun at linsun in ~/go/src/github.com/docker/infrakit on git:master ✖︎ [21:55:34]
→ build/infrakit-group-default 
INFO[0000] Listening at: /Users/linsun/.infrakit/plugins/group 
INFO[0253] Watching group 'terraform_demo_swarm_mgr_sl' 
INFO[0253] Adding 1 instances to group to reach desired 1 
INFO[0253] Created instance instance-1479265228 with tags map[infrakit.group:terraform_demo_swarm_mgr_sl infrakit.config_sha:zpXBA-c_noNozNR6fbDqD959jIo=] 
INFO[1004] Destroying instance instance-1479265228    

# linsun at linsun in ~/go/src/github.com/docker/infrakit on git:master ✖︎ [21:56:25]
→ build/infrakit-flavor-vanilla
INFO[0000] Listening at: /Users/linsun/.infrakit/plugins/flavor-vanilla 

# linsun at linsun in ~/go/src/github.com/docker/infrakit on git:master ✖︎ [21:58:30]
→ build/infrakit-instance-terraform --dir $(pwd)/example/instance/terraform/sl-swarm-mgr/
INFO[0000] Listening at: /Users/linsun/.infrakit/plugins/instance-terraform 
INFO[0110] Tuesday, 15-Nov-16 22:00:28 EST Applying plan 
INFO[0110] [WARN] /Users/linsun/go/bin/terraform-provider-softlayer overrides an internal plugin for softlayer-provider.
  terraform=apply

1. group watch cmd, simple group for swarm mgr

# linsun at linsun in ~/go/src/github.com/docker/infrakit on git:master ✖︎ [22:00:06]
→ build/infrakit group watch /Users/linsun/go/src/github.com/docker/infrakit/example/instance/terraform/sl-swarm-mgr/group.json 
watching terraform_demo_swarm_mgr_sl

1. destroy the group.

# linsun at linsun in ~/go/src/github.com/docker/infrakit on git:master ✖︎ [22:00:28]
→ build/infrakit group destroy terraform_demo_swarm_mgr_sl                                                                     
destroy terraform_demo_swarm_mgr_sl initiated

1. go back to the infrakit-instance-terraform console, terraform apply still runs there every second or so, even after the instance is completely removed in soft layer (IBM Bluemix). I did saw there is 1 destroyed in terraform apply as the instance is being removed.

[below occurs every second after the instance is completely removed ]
terraform=apply

INFO[1458] Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

[wfarner]

@linsun reading the code, it like this is expected behavior for the terraform plugin. Once an action is performed (e.g. Provision/Destroy), it begins apply-ing every second. It just so happens that the state being applied may be empty.

@chungers can you confirm?

[linsun]

@wfarner @chungers As part of the group destroy successful, can we also perform the unwatch the group action? I don't believe terraform keep apply-ing every second if I issue unwatch cmd, so we should have unwatch issued for the user automatically when group destroy is successful.

[wfarner]

That's the thing - the group is inwatched with the destroy action. The terraform plugin, however, continues running its background action indefinitely regardless of whether there are any remaining instances or groups. It's a noop and benign, albeit confusing.

[chungers]

@linsun - This is normal behavior of the terraform plugin -- it continuously call terraform apply to have terraform synchronize the desire state (the instances written as instance-xxxx.tf.json files on disk). So when a group is destroyed, the instance plugin is called to remove each one of the instance .tf.json files for that group.

It's important to note that an instance plugin can be used to provision multiple-groups with calls from possibly multiple group-plugins. So while one group has been destroyed/ unwatched, the instance plugin still has to keep calling Terraform to reconcile the state because other groups may be around, or even other group plugins.

[linsun]

@chungers I noticed this behavior when I do group unwatch cmd too... once I issue group watch, I saw terraform apply continuously in the terraform instance plugin... but once I issue group unwatch, I would expect it to stop, especially when it is not watching anything else. Is it possible to add unwatch hook to plugin so we can stop the terraform apply continuously?