otelcol: use sync.Once in Shutdown

[axw]

Description

Make sure that Shutdown closes the shutdown channel at most once, but don't use the collector state to determine this. That's because the state may be "closing" during a config reload.

Link to tracking issue

Fixes #13740

Testing

Applied diff in the linked issue, does not trigger the bug anymore.

Documentation

N/A

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.39%. Comparing base (f8af86c) to head (60985e4).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13741      +/-   ##
==========================================
- Coverage   91.42%   91.39%   -0.03%     
==========================================
  Files         641      641              
  Lines       42670    42666       -4     
==========================================
- Hits        39011    38996      -15     
- Misses       2839     2847       +8     
- Partials      820      823       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dmathieu]

Have you tried reproducing the issue in a unit test?

[axw]

@dmathieu only briefly. I can take another look tomorrow - basically we'd need to block the internal service shutdown on reload, and check that a Shutdown call isn't ignored.

[axw]

@dmathieu done - the test now reliably triggers the bug on main

[bogdandrutu]

Should we have a "reloading" state instead and shutdown even in that?

[axw]

@bogdandrutu IMO yes, but not as a solution to this particular problem

[bogdandrutu]

I think also we need to make a decision about Run after Shutdown and make sure clearly implemented in the code.