Skip to content

Fix tarball extraction when using wildcards #15695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 3, 2023
Merged

Conversation

mthalman
Copy link
Member

@mthalman mthalman commented Mar 3, 2023

The VerifyVersionFile test in source-build is failing in the Ubuntu leg with the following failure:

System.InvalidOperationException : Failed to execute tar xzf /vmr/artifacts/x64/Release/dotnet-sdk-8.0.100-preview.3.23152.1-ubuntu.20.04-x64.tar.gz -C /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net8.0/sourcebuilt-artifacts ./sdk/*/.version
Exit code: 2

tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: ./sdk/*/.version: Not found in archive
tar: Exiting with failure status due to previous errors

It seems the version of tar in Ubuntu has different behavior compared to the other distros where this was working. Because the file path being targeted to extract has a wildcard in it, tar requires that the --wildcards option be explicitly set. Otherwise, it defaults to not treating it as a wildcard, which then leads to the failure to find a file path of ./sdk/*/.version.

I've fixed this by explicitly setting the --wildcards option. The use of this option is compatible with the other distros as well.

Fixes dotnet/source-build#3294

@mthalman mthalman requested a review from a team as a code owner March 3, 2023 14:51
@mthalman mthalman enabled auto-merge (squash) March 3, 2023 14:53
@mthalman mthalman merged commit 9bd65f4 into dotnet:main Mar 3, 2023
mthalman added a commit to mthalman/installer that referenced this pull request Mar 7, 2023
Comment on lines 64 to +66
private void ExtractFileFromTarball(string tarballPath, string filePath, string outputDir)
{
ExecuteHelper.ExecuteProcessValidateExitCode("tar", $"xzf {tarballPath} -C {outputDir} {filePath}", OutputHelper);
ExecuteHelper.ExecuteProcessValidateExitCode("tar", $"--wildcards -xzf {tarballPath} -C {outputDir} {filePath}", OutputHelper);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this net8.0 project can reference https://github.com/dotnet/sdk/blob/5638e0ffabae3eb5505d7aaf81e3449ce3641c36/src/RazorSdk/Tasks/Microsoft.NET.Sdk.Razor.Tasks.csproj#L43 and use:

using System.Formats.Tar;
using System.IO;
using System.IO.Compression;
using Microsoft.Extensions.FileSystemGlobbing;
...
private void ExtractFileFromTarball(string tarballPath, string filePath, string outputDir)
{
    Matcher matcher = new();
    matcher.AddInclude(filePath);

    using FileStream fileStream = File.OpenRead(tarballPath);
    using GZipStream decompressorStream = new(fileStream, CompressionMode.Decompress);
    using TarReader reader = new(decompressorStream);

    TarEntry entry;
    while ((entry = reader.GetNextEntry()) is not null)
    {
        if (matcher.Match(entry.Name).HasMatches)
        {
            string outputPath = Path.Join(outputDir, entry.Name);
            Directory.CreateDirectory(Path.GetDirectoryName(outputPath));

            using FileStream outputFileStream = File.Create(outputPath);
            entry.DataStream.CopyTo(outputFileStream);
            // break; ?
        }
    }
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kasperk81 - Fixing this in #15765

@mthalman mthalman deleted the sb3294 branch March 8, 2023 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

VerifyVersionFile test is failing in Ubuntu CI leg
3 participants