**Describe the bug** This issue has been observed in the tutorials: https://github.com/Project-MONAI/tutorials/issues/2030#event-19777226570. **Expected behavior** The out layer should not be set in the forward loop because: - The device is wrong in this scenario - The parameters from this layer are not included in the optimizer trainable args. - **Additional context** This was caused by fix https://github.com/Project-MONAI/MONAI/pull/8514.