Amazon linux 2023 fix

[pragnesh]

This PR makes the following changes:

• added ebs-volume-type config which allow ebs volume type from configuration since amazon linux 2023 by default comes with gp3 volume image
• updated setup-ephemeral-storage.py since new version of lsblk doesn't produce same output for setup.
• moved python package installation before running setup-ephemeral-storage.py
• instance data IMDSv2 fix for spark public hostname

I tested this PR by...
launching spark cluster with latest amazon linux 2023 and amazon linux 2 image. both working fine

[nchammas]

Thanks for this PR @pragnesh and sorry to create a conflict.

Could you explain the motivation for the following changes? They seem unrelated to the subject of this PR.

• moved python package installation before running setup-ephemeral-storage.py
• instance data IMDSv2 fix for spark public hostname

[nchammas]

Did you test this PR against a private VPC btw? I'm wondering if #346 is affected at all by these changes.

[pragnesh]

Hi @nchammas ,

setup-ephemeral-storage.py is not running without python package,

amazon linux 2 has default python installed so it was working earlier but amazon linux 2023 need python to be installed so moved it before setting running setup-ephemeral-storage.py

instance data IMDSv2 is enabed by default for amazon linux 2023, so SPARK_PUBLIC_DNS need to be adjusted for functional spark web UI

link for Instance data difference between amazon linux 2 and amazon linux 2023

[pragnesh]

Did you test this PR against a private VPC btw? I'm wondering if #346 is affected at all by these changes.

no i have not. we do you VPC but , launch it with public ip enabled.

I have face similar issue with IMDSV2 so hence this fix for SPARK_PUBLIC_DNS.

I have rebased my PR.

[pragnesh]

Did you test this PR against a private VPC btw? I'm wondering if #346 is affected at all by these changes.

I have updated PR to correctly set SPARK_PUBLIC_DNS in private VPC.
I have tested change on amazon linux 2 ami.
I have tested amazon linux 2023 ami with java preinstalled since Adoptium does not work with amazon linux 2023. Hardcoded releasever to 2 since Adoptium does not have amazon linux 2023 jdk
I have tried testing private vpc but due to some vpc network configuration i am unable to launch but SPARK_PUBLIC_DNS is set correctly within private vpc.

[nchammas]

Sorry to leave this PR hanging. It is still on my mind and I will hopefully pick it up after I get Flintrock 2.1 out.