Allow GitHub origin to accept any host

To better support GitHub enterprise, the GitHub Origin will just assume any host it's given is a GitHub or Github enterprise host.

Author: chris-campos

docs/reference.md

@@ -3379,7 +3379,6 @@ Parameter | Description
 Name | Type | Description
 ---- | ---- | -----------
 <span style="white-space: nowrap;">`--allstar-app-ids`</span> | *list* | Flag used to set AllStar GitHub app id aliases. See https://github.com/ossf/allstar.
-<span style="white-space: nowrap;">`--github-allowed-hosts`</span> | *list* | If using GitHub Enterprise, one needs to specify valid hosts. By default only `github.com` is supported.
 <span style="white-space: nowrap;">`--github-api-bearer-auth`</span> | *boolean* | If using a token for GitHub access, bearer auth might be required
 <span style="white-space: nowrap;">`--github-destination-delete-pr-branch`</span> | *boolean* | Overwrite git.github_destination delete_pr_branch field
 <span style="white-space: nowrap;">`--gql-commit-history-override`</span> | *list* | Flag used to target GraphQL params 'first' arguments in the event the defaults are over or underusing the api ratelimit. The flag value should be semicolon separated. This should be rarely used for repos that don't fit well in our defaults. E.g. '50;5;5' represent 50 commits, 5 PRs for each commit, 5 reviews per PR
@@ -3428,7 +3427,6 @@ Name | Type | Description
 <span style="white-space: nowrap;">`--git-destination-push`</span> | *string* | If set, overrides the git destination push reference.
 <span style="white-space: nowrap;">`--git-destination-url`</span> | *string* | If set, overrides the git destination URL.
 <span style="white-space: nowrap;">`--git-skip-checker`</span> | *boolean* | If true and git.destination has a configured checker, it will not be used in the migration.
-<span style="white-space: nowrap;">`--github-allowed-hosts`</span> | *list* | If using GitHub Enterprise, one needs to specify valid hosts. By default only `github.com` is supported.
 <span style="white-space: nowrap;">`--github-api-bearer-auth`</span> | *boolean* | If using a token for GitHub access, bearer auth might be required
 <span style="white-space: nowrap;">`--github-destination-delete-pr-branch`</span> | *boolean* | Overwrite git.github_destination delete_pr_branch field
 <span style="white-space: nowrap;">`--gql-commit-history-override`</span> | *list* | Flag used to target GraphQL params 'first' arguments in the event the defaults are over or underusing the api ratelimit. The flag value should be semicolon separated. This should be rarely used for repos that don't fit well in our defaults. E.g. '50;5;5' represent 50 commits, 5 PRs for each commit, 5 reviews per PR
@@ -3437,7 +3435,7 @@ Name | Type | Description
 <a id="git.github_origin" aria-hidden="true"></a>
 ### git.github_origin
 
-Defines a Git origin for a Github repository. This origin should be used for public branches. Use github_pr_origin for importing Pull Requests.
+Defines a Git origin for a GitHub or GitHub Enterprise repository. This origin should be used for public branches. Use github_pr_origin for importing Pull Requests.
 
 <code><a href="#origin">origin</a></code> <code>git.github_origin(<a href=#git.github_origin.url>url</a>, <a href=#git.github_origin.ref>ref</a>=None, <a href=#git.github_origin.submodules>submodules</a>='NO', <a href=#git.github_origin.excluded_submodules>excluded_submodules</a>=[], <a href=#git.github_origin.first_parent>first_parent</a>=True, <a href=#git.github_origin.partial_fetch>partial_fetch</a>=False, <a href=#git.github_origin.patch>patch</a>=None, <a href=#git.github_origin.describe_version>describe_version</a>=None, <a href=#git.github_origin.version_selector>version_selector</a>=None, <a href=#git.github_origin.primary_branch_migration>primary_branch_migration</a>=False, <a href=#git.github_origin.enable_lfs>enable_lfs</a>=False, <a href=#git.github_origin.credentials>credentials</a>=None)</code>
 
@@ -3470,7 +3468,6 @@ Name | Type | Description
 <span style="white-space: nowrap;">`--git-origin-log-batch`</span> | *int* | Read the origin git log in batches of n commits. Might be needed for large migrations resulting in git logs of more than 1 GB.
 <span style="white-space: nowrap;">`--git-origin-non-linear-history`</span> | *boolean* | Read the full git log and skip changes before the from ref rather than using a log path.
 <span style="white-space: nowrap;">`--git-origin-rebase-ref`</span> | *string* | When importing a change from a Git origin ref, it will be rebased to this ref, if set. A common use case: importing a Github PR, rebase it to the main branch (usually 'master'). Note that, if the repo uses submodules, they won't be rebased.
-<span style="white-space: nowrap;">`--github-allowed-hosts`</span> | *list* | If using GitHub Enterprise, one needs to specify valid hosts. By default only `github.com` is supported.
 <span style="white-space: nowrap;">`--github-api-bearer-auth`</span> | *boolean* | If using a token for GitHub access, bearer auth might be required
 <span style="white-space: nowrap;">`--github-destination-delete-pr-branch`</span> | *boolean* | Overwrite git.github_destination delete_pr_branch field
 <span style="white-space: nowrap;">`--gql-commit-history-override`</span> | *list* | Flag used to target GraphQL params 'first' arguments in the event the defaults are over or underusing the api ratelimit. The flag value should be semicolon separated. This should be rarely used for repos that don't fit well in our defaults. E.g. '50;5;5' represent 50 commits, 5 PRs for each commit, 5 reviews per PR
@@ -3566,7 +3563,6 @@ Name | Type | Description
 <span style="white-space: nowrap;">`--git-destination-push`</span> | *string* | If set, overrides the git destination push reference.
 <span style="white-space: nowrap;">`--git-destination-url`</span> | *string* | If set, overrides the git destination URL.
 <span style="white-space: nowrap;">`--git-skip-checker`</span> | *boolean* | If true and git.destination has a configured checker, it will not be used in the migration.
-<span style="white-space: nowrap;">`--github-allowed-hosts`</span> | *list* | If using GitHub Enterprise, one needs to specify valid hosts. By default only `github.com` is supported.
 <span style="white-space: nowrap;">`--github-api-bearer-auth`</span> | *boolean* | If using a token for GitHub access, bearer auth might be required
 <span style="white-space: nowrap;">`--github-destination-delete-pr-branch`</span> | *boolean* | Overwrite git.github_destination delete_pr_branch field
 <span style="white-space: nowrap;">`--github-destination-pr-branch`</span> | *string* | If set, uses this branch for creating the pull request instead of using a generated one
@@ -3634,7 +3630,6 @@ Name | Type | Description
 <span style="white-space: nowrap;">`--git-origin-log-batch`</span> | *int* | Read the origin git log in batches of n commits. Might be needed for large migrations resulting in git logs of more than 1 GB.
 <span style="white-space: nowrap;">`--git-origin-non-linear-history`</span> | *boolean* | Read the full git log and skip changes before the from ref rather than using a log path.
 <span style="white-space: nowrap;">`--git-origin-rebase-ref`</span> | *string* | When importing a change from a Git origin ref, it will be rebased to this ref, if set. A common use case: importing a Github PR, rebase it to the main branch (usually 'master'). Note that, if the repo uses submodules, they won't be rebased.
-<span style="white-space: nowrap;">`--github-allowed-hosts`</span> | *list* | If using GitHub Enterprise, one needs to specify valid hosts. By default only `github.com` is supported.
 <span style="white-space: nowrap;">`--github-api-bearer-auth`</span> | *boolean* | If using a token for GitHub access, bearer auth might be required
 <span style="white-space: nowrap;">`--github-destination-delete-pr-branch`</span> | *boolean* | Overwrite git.github_destination delete_pr_branch field
 <span style="white-space: nowrap;">`--github-force-import`</span> | *boolean* | Force import regardless of the state of the PR
@@ -3674,7 +3669,6 @@ Parameter | Description
 Name | Type | Description
 ---- | ---- | -----------
 <span style="white-space: nowrap;">`--allstar-app-ids`</span> | *list* | Flag used to set AllStar GitHub app id aliases. See https://github.com/ossf/allstar.
-<span style="white-space: nowrap;">`--github-allowed-hosts`</span> | *list* | If using GitHub Enterprise, one needs to specify valid hosts. By default only `github.com` is supported.
 <span style="white-space: nowrap;">`--github-api-bearer-auth`</span> | *boolean* | If using a token for GitHub access, bearer auth might be required
 <span style="white-space: nowrap;">`--github-destination-delete-pr-branch`</span> | *boolean* | Overwrite git.github_destination delete_pr_branch field
 <span style="white-space: nowrap;">`--gql-commit-history-override`</span> | *list* | Flag used to target GraphQL params 'first' arguments in the event the defaults are over or underusing the api ratelimit. The flag value should be semicolon separated. This should be rarely used for repos that don't fit well in our defaults. E.g. '50;5;5' represent 50 commits, 5 PRs for each commit, 5 reviews per PR

java/com/google/copybara/git/GitHubOptions.java

@@ -89,26 +89,8 @@ public class GitHubOptions implements Option {
       arity = 1)
   public boolean gitHubApiBearerAuth = false;
 
-  @Parameter(
-      names = "--github-allowed-hosts",
-      description = "If using GitHub Enterprise, one needs to specify valid hosts. By default only `github.com` is supported."
-  )
-  public List<String> gitHubAllowedHosts = ImmutableList.of("github.com");
-
-  public GitHubHost getGitHubHost(String url) throws EvalException  {
-      GitHubHost host = GitHubHost.fromUrl(url);
-      StarlarkUtil.check(gitHubAllowedHosts.contains(host.getHost()), "'%s' is not a valid GitHub url", host.getHost());
-      return host;
-  }
-
-  public boolean isGithubUrl(String url)
-  {
-      GitHubHost host = GitHubHost.fromUrl(url);
-      if(gitHubAllowedHosts.contains(host.getHost())){
-          return host.isGitHubUrl(url);
-      }
-
-      return false;
+  public GitHubHost getGitHubHost(String url)  {
+    return GitHubHost.fromUrl(url);
   }
 
   public GitHubOptions(GeneralOptions generalOptions, GitOptions gitOptions) {
@@ -151,6 +133,26 @@ public GitHubApi newGitHubRestApi(GitHubHost ghHost,
         gitHubProject, /* checker= */ null, credentials, generalOptions.console());
   }
 
+  /**
+   * Returns a new Github.com specific {@link GitHubApi} instance for the given project enforcing the given {@link
+   * Checker}.
+   *
+   * <p>The project for 'https://github.com/foo/bar' is 'foo/bar'.
+   *
+   * @param gitHubProject the project
+   * @param checker the checker to enforce
+   * @param credentials the credentials to use for GitHub API auth
+   * @param console the console, used for logging
+   * @return the instance
+   * @throws RepoException if there is a failure in using the credentials
+   */
+  public GitHubApi newGitHubRestApi(String gitHubProject,
+      @Nullable Checker checker,
+      @Nullable CredentialFileHandler credentials,
+      Console console) throws RepoException {
+    return newGitHubRestApi(GitHubHost.GITHUB_COM, gitHubProject, checker, credentials, console);
+  }
+
   /**
    * Returns a new {@link GitHubApi} instance for the given project enforcing the given {@link
    * Checker}.
@@ -187,6 +189,27 @@ public GitHubGraphQLApi newGitHubGraphQLApi(GitHubHost ghHost,
         gitHubProject, /* checker= */ null, credentials, generalOptions.console());
   }
 
+  /**
+   * Returns a new GitHub.com specific {@link GitHubApi} instance for the given project enforcing the given {@link
+   * Checker}.
+   *
+   * <p>The project for 'https://github.com/foo/bar' is 'foo/bar'.
+   * @param gitHubProject the GitHub project
+   * @param checker the checker to enforce
+   * @param credentials the credentials to use for the GitHub API
+   * @param console the console, for logging
+   * @return the instance
+   * @throws RepoException if there is an issue using the provided credentials
+   */
+  public GitHubGraphQLApi newGitHubGraphQLApi(
+      String gitHubProject,
+      @Nullable Checker checker,
+      @Nullable CredentialFileHandler credentials,
+      Console console)
+      throws RepoException {
+    return newGitHubGraphQLApi(GitHubHost.GITHUB_COM, gitHubProject, checker, credentials, console);
+  }
+
   /**
    * Returns a new {@link GitHubApi} instance for the given project enforcing the given {@link
    * Checker}.

java/com/google/copybara/git/GitModule.java

@@ -356,7 +356,8 @@ public GitOrigin origin(
     CredentialFileHandler credentialHandler = getCredentialHandler(fixedUrl, credentials);
 
     GitHubOptions githubOptions = options.get(GitHubOptions.class);
-    boolean isGitHubUrl = githubOptions.isGithubUrl(url);
+    // This does not support GitHub Enterprise. For that, use githubOrigin.
+    boolean isGitHubUrl = GITHUB_COM.isGitHubUrl(url);
 
     return GitOrigin.newGitOrigin(
         options,
@@ -1263,7 +1264,7 @@ public Origin<GitRevision> githubPrOrigin(
   @StarlarkMethod(
       name = "github_origin",
       doc =
-          "Defines a Git origin for a Github repository. This origin should be used for public"
+          "Defines a Git origin for a GitHub or GitHub Enterprise repository. This origin should be used for public"
               + " branches. Use "
               + GITHUB_PR_ORIGIN_NAME
               + " for importing Pull Requests.",
@@ -3400,7 +3401,7 @@ private String fixHttp(String url, Location location) {
   /** Do not use this for github origins */
   protected ApprovalsProvider approvalsProvider(String url) {
       Preconditions.checkArgument(
-              !options.get(GitHubOptions.class).isGithubUrl(url),
+              !GITHUB_COM.isGitHubUrl(url),
               "Git origins with github should use github approval providers!");
     return options.get(GitOriginOptions.class).approvalsProvider;
   }
@@ -3517,13 +3518,9 @@ protected LazyResourceLoader<EndpointProvider<?>> maybeGetGerritApi(
   protected LazyResourceLoader<EndpointProvider<?>> maybeGetGitHubApi(
       String url, @Nullable Checker checker, @Nullable CredentialFileHandler creds,
       StarlarkThread thread) {
-      try {
-          GitHubOptions githubOptions = options.get(GitHubOptions.class);
-          GitHubHost ghHost = githubOptions.getGitHubHost(url);
-          if (!ghHost.isGitHubUrl(url)) {
-              return null;
-          }
-      } catch (EvalException e) {
+      GitHubOptions githubOptions = options.get(GitHubOptions.class);
+      GitHubHost ghHost = githubOptions.getGitHubHost(url);
+      if (!ghHost.isGitHubUrl(url)) {
           return null;
       }
     return (console) -> {
@@ -3553,15 +3550,11 @@ protected LazyResourceLoader<EndpointProvider<?>> maybeGetGitHubApi(
   @Nullable protected CredentialFileHandler getCredentialHandler(
       String url, @Nullable Object starlarkValue) {
     try {
-        try {
-            GitHubOptions githubOptions = options.get(GitHubOptions.class);
-            GitHubHost ghHost = githubOptions.getGitHubHost(url);
-            if (ghHost.isGitHubUrl(url)) {
-                url = ghHost.normalizeUrl(url);
-            }
-        } catch (EvalException e) {
-            // nothing to-do, it is valid that this is not an GitHub URL.
-        }
+      GitHubOptions githubOptions = options.get(GitHubOptions.class);
+      GitHubHost ghHost = githubOptions.getGitHubHost(url);
+      if (ghHost.isGitHubUrl(url)) {
+          url = ghHost.normalizeUrl(url);
+      }
       URI uri = URI.create(url);
       return getCredentialHandler(uri.getHost(), uri.getPath(), starlarkValue);
     } catch (ValidationException | IllegalArgumentException parseEx) {

java/com/google/copybara/git/github/util/GitHubHost.java

@@ -41,10 +41,8 @@ public GitHubHost(String host) {
     this.gitHubPrUrlPattern = Pattern.compile("https://\\Q" + host + "\\E/(.+)/pull/([0-9]+)");
   }
 
-  static public GitHubHost fromUrl(String url)
-  {
-      url = url.replaceAll("http\\:\\/\\/|https\\:\\/\\/|git\\+|git@", "").replaceAll(":.*|/.*", "");
-      return new GitHubHost(url);
+  static public GitHubHost fromUrl(String url) {
+    return new GitHubHost(URI.create(url).getHost());
   }
 
   /**
@@ -90,7 +88,7 @@ public String getProjectNameFromUrl(String url) throws ValidationException {
     return name;
   }
 
-  /** Returns true if url is a GitHub url for a given GitHub or Enterprise host. */
+  /** Returns true if URL belongs to the host that this object is initialized with. */
   public boolean isGitHubUrl(String url) {
     try {
       getProjectNameFromUrl(url);

javatests/com/google/copybara/git/GitOriginTest.java

@@ -338,7 +338,6 @@ public void testGithubOrigin() throws Exception {
 
   @Test
   public void testGithubOriginForEnterpriseUrl() throws Exception {
-    options.github.gitHubAllowedHosts = ImmutableList.of("some.github-enterprise.net");
     origin = skylark.eval("result",
         "result = git.github_origin(\n"
         + "    url = 'https://some.github-enterprise.net/copybara',\n"
@@ -354,40 +353,6 @@ public void testGithubOriginForEnterpriseUrl() throws Exception {
                 + "}");
   }
 
-  @Test
-  public void testInvalidGithubUrl() throws Exception {
-    ValidationException expected =
-        assertThrows(
-            ValidationException.class,
-            () ->
-                skylark.eval(
-                    "result",
-                    "result = git.github_origin(\n"
-                        + "    url = 'https://foo.com/copybara',\n"
-                        + "    ref = 'main',\n"
-                        + ")"));
-    console
-        .assertThat()
-        .onceInLog(MessageType.ERROR, ".*'foo.com' is not a valid GitHub url.*");
-  }
-
-  @Test
-  public void testInvalidGithubUrlWithGithubString() throws Exception {
-    ValidationException expected =
-        assertThrows(
-            ValidationException.class,
-            () ->
-                skylark.eval(
-                    "result",
-                    "result = git.github_origin(\n"
-                        + "    url = 'https://foo.com/github.com',\n"
-                        + "    ref = 'main',\n"
-                        + ")"));
-    console
-        .assertThat()
-            .onceInLog(MessageType.ERROR, ".*'foo.com' is not a valid GitHub url.*");
-  }
-
   @Test
   public void testResolveWithUrl() throws Exception {
     assertThat(origin.resolve(defaultBranch).getUrl()).isEqualTo(url);