Add retry to running demo projects for macOs platforms in CI

[Yarwin]

What does this PR solve?

Stops running examples on macOS platforms which fail semi-randomly for unknown reason, see: #12. We weren't able to find the cause for two months.

What was the problem extacly?

Our examples run sometimes do fail with an libc++abi: Pure virtual function called! error. Nobody knows what extacly causes it, thus the band-aids.

outdated description

What does this PR solve?

In practice, nothing, it just piles more workarounds on top of each other. According to the latest godot poll https://docs.google.com/forms/d/e/1FAIpQLScKWGJoLEeNW1qrsDfZRfk7gHultapacH5ZhQmo9XRZADW1IQ/viewanalytics 30% of all the Mac users are still using intel CPUs (see question: What type of Mac do you use? (Optional: only if you are using Mac)) so removing it completely from the pipeline would be way too silly.

This PR introduces new variable in run matrix - failable - which tries to run example three times. Additionally continue-on-error should make whole pipeline pass even if failable job fails. Better than nothing, eh?

[Yarwin]

huh, that's new - MacOS ARM (NOT x86) failed with the same error 🤔

[Bromeon]

So, does retrying fix the issue or not?

If failing the job doesn't contribute to the CI failing, that makes the job rather meaningless -- sure, people can manually check the result, but then what's the cause of action? How to decide if it's a "real" or "irrelevant" failure?

Are there maybe other ways we could automate this?

[Bromeon]

Btw, I brought this up in RocketChat a while ago. Unfortunately no answer. Might be worth trying again? 🤔

[Yarwin]

So, does retrying fix the issue or not?

If failing the job doesn't contribute to the CI failing, that makes the job rather meaningless -- sure, people can manually check the result, but then what's the cause of action? How to decide if it's a "real" or "irrelevant" failure?

Are there maybe other ways we could automate this?

That makes sense – I checked it in isolation few times and concluded that RETRY it is reliable enough to leave it be, but really serves no real purpose – and I'm pretty sure that everybody will forget to actually check the runs upon failing 🤔. I think I'll just disable the last step for macOS builds for now.

Btw, I brought this up in RocketChat a while ago. Unfortunately no answer. Might be worth trying again? 🤔

I re-forwarded given message and noted that it happens for macOS-ARM as well 👍

[Bromeon]

That makes sense – I checked it in isolation few times and concluded that RETRY it is reliable enough to leave it be, but really serves no real purpose – and I'm pretty sure that everybody will forget to actually check the runs upon failing 🤔. I think I'll just disable the last step for macOS builds for now.

Not sure I understand this right, if RETRY is reliable enough, we can leave that for now?

At least, CI would still fail in case of systematic errors (which is what we want) 🤔

[Yarwin]

Not sure I understand this right, if RETRY is reliable enough, we can leave that for now?

I checked it in isolation few times and it seems to work reliably (outside the fact that it initially did not fail 😅 – I added exit to handle that, let me know if it is the best/good solution).

I think we can use retry for now; if macOS jobs fail at least once then we should move to only compiling against given platform 🤔.

[Bromeon]

I think it's good, thanks! 👍 minor suggestion

[Bromeon]

Suggested change

	if [[ $RETRY == "true" ]]; then
	# MacOS – retry running demo projects several times on fail.
	echo "Running examples with retry"
	RUN="./.github/other/retry.sh ./.github/other/check-example.sh"
	else
	RUN="./.github/other/check-example.sh"
	fi
	# List all folders in current directory. Don't quote $files variable.
	for demo in $files; do
	# Strip trailing '/' from folder name.
	./.github/other/check-example.sh "${demo%/}"
	$RUN "${demo%/}"
	done
	if [[ $RETRY == "true" ]]; then
	# macOS – retry running demo projects several times on fail.
	echo "Running examples with retry"
	RETRY_CMD="./.github/other/retry.sh"
	else
	RETRY_CMD=""
	fi
	# List all folders in current directory. Don't quote $files variable.
	for demo in $files; do
	# Strip trailing '/' from folder name.
	$RETRY_CMD ./.github/other/check-example.sh "${demo%/}"
	done

The else branch might not even necessary, usually undefined env vars evaluate to "". But it's a bit more explicit.