start github-and-away tech note #1167
Conversation
Aariq
left a comment
Aariq
left a comment
There was a problem hiding this comment.
I think the most confusing part about this to me is the directionality of the phrase "to mirror". Is that from the "main home" to another remote or vice versa? I'm already a bit confused in the "Mirroring on codeberg" section because I'm not sure whether it's talking about pushing to codeberg and having codeberg push updates to GitHub (I think it's this) or if it means continuing to push to GitHub but having changes mirrored on codeberg.
|
|
||
| ## Managing one repository across multiple platforms | ||
|
|
||
| With due apologies for repeittion, git is a centralised version control system, and is thus not the best system for managing multiple remote sources. |
There was a problem hiding this comment.
Suggest using American English for consistency, "centralised" here, but on line 23 it is "centralized"
| doi: "" | ||
| --- | ||
|
|
||
| rOpenSci peer-review has to date been exclusively conducted [on GitHub](https://github.com/ropensci/software-review/issues?q=sort%3Aupdated-desc%20is%3Aissue%20state%3Aclosed). |
There was a problem hiding this comment.
why start with peer-review? also present other infrastructure aspects (registry, R-universe with docs building for rOpenSci packages). Also, is this post only about rOpenSci?
As new to other platforms, I'd actually be more worried about how to transfer my data to the other platform.
I think it'd make sense to say why a package would need a mirror on GitHub.
I'm not even sure it's needed for the registry, as I think we can clone from GitLab or codeberg: https://github.com/ropensci-org/makeregistry/blob/bc06da8f87da659a5e123f44603fb9737b969821/R/track-repos.R#L20
There was a problem hiding this comment.
@maelle These are tricky yet important questions...
is this post only about rOpenSci?
I'm not sure 😕 I think so, at the moment, because I think it should clearly describe how people can use other platforms to host/mirror their rOpenSci repositories? Making it more general would require describing rOpenSci-specific procedures in separate section, which might just add to confusion. It would also make it harder to clarify initial motivation. But I'm unsure, and open to different opinions? @adamhsparks @Aariq Any thoughts?
|
General question, I've migrated {getCRUCLData} to Codeberg, https://codeberg.org/ropensci/getCRUCLdata. It works great! Of course I had to change GitHub URLs in the DESCRIPTION and README, clean up GitHub workflows, etc. But I'm left wondering what to do with the GitHub repo now. Do I delete and create a new one and then use your instructions for adding a mirror? |
|
Is a mirror really needed? Could the current repo be wiped and contain just a README pointing to the new source home? |
|
well, that's another question we could answer here. :) it depends on whether it's personal or rOpenSci, I suppose. Do we have a policy on the best approach? |
@adamhsparks My personall recommendation would be to leave GitHub version, and use Codeberg's mirroring setup to reflect everything back. You then shouldn't need to do anything else here (on GitHub), and can happily just use Codeberg from here on. But I'll try to clarify different approaches in the text ...
@maelle Yes, of course, but this whole thing is supposed to be about mirroring. Simply pointing to a new source home is just like adding a hyperlink from somewhere to somewhere, with all the burden of keeping in mind all the places you've linked from and to everything. Mirroring is ... mirroring. So you only ever have to keep one place in mind, and all other places with automagically reflect any and all changes without you need to remember or do anything (except may look at |
maelle
left a comment
maelle
left a comment
There was a problem hiding this comment.
Thanks! A few further comments 😸
|
|
||
|  | ||
|
|
||
| {{< figure src = "local-remote.png" alt = "Local and remote repositories." class = "pull-left" caption = "Local and remote repositories.">}} |
There was a problem hiding this comment.
- The alt text should be more precise unless the info is all in the blog post text in which case the alt text should be empty but present.
"". - I think the yellow double arrows could be a tad prettier. Maybe use Excalidraw pre-built shapes? I do like the hand-drawn effect though!
There was a problem hiding this comment.
@maelle Does the alt text all seem better now? If not, could you please suggest improvements? Thanks!
There was a problem hiding this comment.
I think if you follow my suggestions to add a bit more detail to the text preceeding the image, you can replace the alt text with "", implying that the information in the Caption and Text is sufficient to understand the article (which it is).
@adamhsparks Check out state of this PR now, which has a great answer for you, thanks to @dgkf! |
|
@adamhsparks @Aariq @dgkf Thank you all for the very helpful input here! It's taken a while, but I think I'll now aim to publish it this coming Friday, 7th Nov. I've added all of your names as editors, but you are also very welcome to be co-authors - just let me know, and feel free to edit anything you like within the next 2-3 days. (@maelle same invitation applies to you of course, but you already indicated you were happy not being an author here.) |
|
One big downside of this fragmentation for e.g r-universe is that we lose contributor statistics when you host the package on another platform where the commits cannot be mapped to usernames because those users are not on that platform. Perhaps we can retrieve those statistics from a github mirror, but that would add a layer of complexity as a registry would need to specify one origin to pull source and another to get metadata, so once those are out of sync things become ambiguous. In my experience, mirroring eventually always silently breaks and you end up with dead mirrors that contain outdated code. Also metadata on "releases" and release assets is unavailable on e.g. codeberg afaik. You will lose those things if you move a repo. You will only keep "tags" but that is different. |
|
Thanks @jeroen, those are helpful insights to keep in mind.
Totally agree, and have done my best to strongly recommend people act as if there is a single primary repo. I'd say the best we can do it to try to get everybody to adhere to that. Committer retrieval and re-mapping across platforms is possible, just difficult. I've got prototype re-maps working for the dashboard stuff, which maps git logs to potentially changed names, emails, and github logins. Committers are not too difficult; contributors via issues, discussions, whatever, are the things we'd potentially lose.
Yep, that's definitely true. I'll add that as an important downside. |
|
Not sure how much moralizing you want to do in the post -- if you want to keep it rather neutral and pragmatic that's fine -- but I was also hoping to see a bit of an explainer for why someone may want to move away from GitHub. Since Just off the top of my head, some reasons that I see pop up frequently are:
|
|
Hi @mpadge, acknowledging as an editor is perfect for what I've contributed here. I've not contributed to writing much of this, even though I would have liked to. Time and personal circumstances intervened and prevented me from being more involved. |
The deploy preview didn't work, but I read the source and saw @dgkf's suggestion for the GitHub README addition. Thanks for that, both of you. |
That's fine! I won't have time to look at it again this week, but looking forward to reading the final product. |
|
I just pushed some changes. Including some changes in the introduction and some others simplifying language later on. However, the post did leave me a bit confused and I think could use some restructuring. Basically, I think it's important to provide the simplest instructions for the main use-case, and only after go into various alternative approaches or complexities (if at all). I think the simplest and most relevant case is using GitLab/Codeberg as the primary remote, using their tools to continually push to GitHub, and avoiding the need to push to multiple repos. So I think the figures should not show multiple mirrors, but just one. The labels should show the primary as GitLab/Codeberg and GitHub as the mirror. Only show the double arrow from local to primary, and the single arrow from primary to mirror. A more complex figure could go later when we explore other set-ups. Any digression beyond the basic setup should be at the end of the post, under a header like "Other Configurations", with sections like
I'd like to hear @steffilazerte's opinion, too, though! |
steffilazerte
left a comment
steffilazerte
left a comment
There was a problem hiding this comment.
Hi @mpadge!
I really enjoyed this post. I was a bit confused at times (see below for suggestions to clarify), but once I worked out what was confusing me I found it really interesting and helpful and it helped me understand what the slack conversations have been all about!
I agree with @noamross that some restructuring could be useful. I was confused because the first couple of sections ("Mirroring on Codeberg or GitLab" through Transferring issues) are actually about migrating, except of the titles, mirroring isn't mentioned at all.
However, I don't think that you need to remove references to multiple mirrors, as the instructions for having multiple mirrors isn't really that different from having a single mirror (it's just "repeat the step").
I do think, though, that it could benefit from a bit more explicit statements along the lines of "Our next step is this for these reasons", as well as reducing duplication (especially in the Transferring issues section) and tweaking the language (i.e. use migration instead of mirror for the migration step).
I've gone a bit rogue and made a large number of opinionated suggestions for a) reducing repetition (I combined the migration and transferring issues sections and reduced what was left to a smaller note, then moved the Migrating elsewhere to the end of the Migration section), clarifying language (I renamed the mirror sections with "Migration", but also added explicit statements to the effect that this was the first step in the whole process of setting up mirrors).
The rest are minor fixes for missing words etc.
I hope this isn't too complicated to dig through, feel free to ignore or omit any that don't resonate with you and let me know if you need any more details or explanations!
- post follows Content Guidelines
- post follows Style Guide
- title is in Title Case
- publication date is ok
- slug is ok
- alternative text of images is informative
- author metadata is provided with correct folder name for each language
- html not included in pull request of Rmd post
- I ran
roblog::ro_lint_md()on index.md - I ran
roblog::ro_check_urls()on index.md - I ran a spell-check on index.md
- YAML subject tags are ok ("tech notes" for tech notes; "community" for non-staff non-editor)
- YAML field 'editor' is filled with your name
- YAML field 'translator' is filled as needed
- YAML field 'interviewee' is filled as needed
- YAML field 'preface' is present if necessary
- YAML field 'params.doi' is filled with a new DOI
| In this diagram, the large yellow arrow represents the only connection where both `push` and `pull` events are allowed. | ||
| All other arrows are `push` events only. | ||
|
|
||
|  |
There was a problem hiding this comment.
Remove duplicated image?
|  |
|
|
||
|  | ||
|
|
||
| {{< figure src = "local-remote.png" alt = "Local and remote repositories." class = "pull-left" caption = "Local and remote repositories.">}} |
There was a problem hiding this comment.
I think if you follow my suggestions to add a bit more detail to the text preceeding the image, you can replace the alt text with "", implying that the information in the Caption and Text is sufficient to understand the article (which it is).
|
|
||
| To be clear, this entire post is about moving away from having code hosted on a single platform, to distributing across multiple platforms through _mirroring_. | ||
| If, for example, you only want to migrate away from GitHub to some single, other platform, then both Codeberg and GitLab already offer the full solutions described above. | ||
| We nevertheless recommend mirroring across multiple platforms, to reduce risks associated with dependence on any single platform. |
| The pure Git way of managing multiple remote sources is to take advantage of `git remote set-url --add` to add additional URLs to a single remote identifier. | ||
| [This blog post](https://jeffkreeftmeijer.com/git-multiple-remotes/) details how to do that safely, to ensure only one primary remote is configured to `fetch`, while allowing `push` events to all others. | ||
| An alternative option would be to initially create an additional remote like `git remote add other https://codeberg.org/ropensci/my-package`. | ||
| You can then extend that with each additional remote URL with `set-url --add`. |
There was a problem hiding this comment.
I don't understand this line, how does this work?
| You can then extend that with each additional remote URL with `set-url --add`. | ||
| Running `git push other <branch>` will then push that branch to all remote URLs specified in `other`. | ||
|
|
||
| Yet another approach is to define a [custom command](https://stackoverflow.com/questions/60060217/how-do-i-make-custom-git-commands) for `git push` to call a local script. |
There was a problem hiding this comment.
Is this paragraph really necessary? I find it complex and a bit confusing and possibly only useful to people trying to mass migrate & mirror a bunch of repositories. Could it be put in an appendix at the end perhaps?
| rOpenSci makes heavy use of GitHub for our projects and services, including [software peer-review](https://github.com/ropensci/software-review/issues?q=sort%3Aupdated-desc%20is%3Aissue%20state%3Aclosed). | ||
| GitHub is by far the most widely used git or code-hosting platform, and the combination of its popularity and freemium services have made it central to open-source and R communities. | ||
| However, for a variety of reasons, some of our community members or potential members may prefer or need to use other platforms. | ||
| These reasons may include concerns about privacy, including [identifying information required for age verification in some jurisdictions](https://www.abc.net.au/news/2025-09-24/digital-dilemna-social-media-age-ban-platforms/105807302). Other reasons include a desire to support platforms with different ownership or business models, based in other countries, or supprting alternatives to avoid the risk of hegemony. |
There was a problem hiding this comment.
I like this section, it's absolutely needed and helpful especially for people like me that might have a hazy idea but would like more.
| These reasons may include concerns about privacy, including [identifying information required for age verification in some jurisdictions](https://www.abc.net.au/news/2025-09-24/digital-dilemna-social-media-age-ban-platforms/105807302). Other reasons include a desire to support platforms with different ownership or business models, based in other countries, or supprting alternatives to avoid the risk of hegemony. | |
| These reasons may include concerns about privacy, including [identifying information required for age verification in some jurisdictions](https://www.abc.net.au/news/2025-09-24/digital-dilemna-social-media-age-ban-platforms/105807302). Other reasons include a desire to support platforms with different ownership or business models, based in other countries, or supporting alternatives to avoid the risk of hegemony. |
| author: | ||
| - Mark Padgham | ||
| editor: | ||
| date: '2025-11-07' |
There was a problem hiding this comment.
Don't forget to update the date here and in the folder when we have a new date for publication.
| - Maëlle Salmon | ||
| - Adam Sparks | ||
| - Eric Scott | ||
| - Doug Kelkhoff |
There was a problem hiding this comment.
Just dropping in my info here:
---
name: Doug Kelkhoff
mastodon: https://fosstodon.org/@dgkf
bio: Research software engineer in healthcare. Chair of the R Validation Hub, an R Consortium Working Group.
github: dgkf
gitlab: dgkf
codeberg: dgkf
orcid: 0009-0003-7845-4061
---
| description: How to manage and mirror code repositories across different platforms. | ||
| output: hugodown::md_document | ||
| tags: | ||
| - tech notes |
There was a problem hiding this comment.
I think we can add some more tags! What do you think about...
| - tech notes | |
| - tech notes | |
| - github | |
| - maintenance | |
| - help | |
| - infrastructure | |
| - git |
| --- | ||
| title: Code hosting options beyond GitHub | ||
| author: | ||
| - Mark Padgham |
There was a problem hiding this comment.
@noamross needs to be here somewhere, author or editor? If you added content, I'd say author.
|
Hey! Not sure if this is still at a good stage to receive additional input, but I spent today writing some tooling to mirror I ran some workflows over in this test repo. The user-facing setup is exactly the same as with I think we'll probably add a few bespoke examples that align a bit better with Codeberg's minimalist resource use philosophy, but this is a promising start. Even running the Next step is to see if |
|
@dgkf That's brilliant! I've been travelling the last couple of weeks, so haven't had a chance to finish off and publish. But that news will make a great addition. I'll update and hope to publish next week (start of Dec). I'll ping you for feedback when I add info on your codeberg enhancements. And i'll also start using them straight away too - thanks a load! |
8 participants
No description provided.