GitHub is facing HUGE problems!
MMaximilian Schwarzmüller
Computing/SoftwareBusiness NewsManagementInternet Technology
Transcript
00:00:00GitHub is in a very dire, a very bad situation.
00:00:04There are many, many problems, many of them related to AI,
00:00:08but maybe not for the reasons you think,
00:00:10but I'll get back to that.
00:00:11And of course that matters.
00:00:13That matters because GitHub is the backbone
00:00:16of modern development work.
00:00:17No matter if you're doing open source development,
00:00:20if you're maintaining some open source projects,
00:00:22if you're working just on your own projects,
00:00:24your personal projects, your side projects,
00:00:26if you're running a small business, a small company,
00:00:29or maybe if you're in a bigger company,
00:00:32it's used for all kinds of things as a code archive
00:00:35for CI/CD workflows, for collaboration,
00:00:38for working on projects together, through issues,
00:00:42for pull requests and many, many other things and use cases.
00:00:47So that matters, but as mentioned,
00:00:49there are many, many problems.
00:00:51And let's start with what's wrong
00:00:53before we take a look at the why
00:00:54and what that means for the future.
00:00:57And let's start with a big one.
00:00:59There was a big, a huge,
00:01:02an unbelievable security vulnerability reported yesterday
00:01:07when I'm recording this.
00:01:09A remote code execution on github.com.
00:01:12I mean, reading that is just insane.
00:01:16It was discovered by Viz, a security company,
00:01:19and it was not exploited.
00:01:21So it was discovered, it was reported, it was fixed.
00:01:25No damage was done.
00:01:28According to GitHub,
00:01:31they also published an answer to this report.
00:01:33Now, I won't go into the details
00:01:36of how that vulnerability worked.
00:01:39I'll link the article below though.
00:01:42But in the end, it all worked through git push.
00:01:44So no phishing involved,
00:01:46no account takeover of some employee,
00:01:49no supply chain attack.
00:01:51We've seen plenty of that stuff over the last weeks,
00:01:54but no, nothing like that was involved.
00:01:56Instead, it was just git push,
00:01:58and then specifically the standard push option feature
00:02:03that you can add to the git push command
00:02:05to attach extra options to that push command.
00:02:10And through that options feature and the vulnerability
00:02:13and the way GitHub handled pushes,
00:02:17the security researchers here were able to attach code
00:02:22that would execute just like that on the GitHub servers.
00:02:27Now, again, the exact details are in this report,
00:02:31but in the end, they abused the fact
00:02:34that you could add extra metadata to an xstat header
00:02:39that would be populated with help of that push options flag.
00:02:44And that metadata, that information that you could pass along
00:02:49with the push request through that header in the end
00:02:52was not sanitized by GitHub.
00:02:54They just authenticated the push request in the end,
00:02:58the push command.
00:02:59They checked if you are allowed to push
00:03:01to the repository you try to push to,
00:03:03but then they took that options data
00:03:07and built that xstat header without sanitizing that data.
00:03:12And that allowed the security researchers
00:03:15to execute command that was then not restricted
00:03:18to the repository to which they pushed,
00:03:21but that instead freely ran on GitHub servers
00:03:24and was able to access other repositories as well,
00:03:27including private repositories.
00:03:29Now, again, this vulnerability was reported and was fixed
00:03:33and it no longer exists,
00:03:35but it's a huge one obviously.
00:03:39I mean, that is such a big deal having a vulnerability
00:03:43that allows for a remote code execution on github.com.
00:03:45It's really huge.
00:03:47So yeah, that's a big one,
00:03:48but of course it's not the only problem.
00:03:51On April the 23rd, so just a few days earlier,
00:03:56there was a huge incident related to GitHub merge queues.
00:04:01Now, GitHub merge queues, in case you don't know,
00:04:04is a GitHub feature that's meant to be used for repositories
00:04:07where you have a lot of activity, a lot of active work,
00:04:11a lot of pull requests coming in.
00:04:13And in order to make sure that you don't have to merge
00:04:16every pull request before a new one can be sent,
00:04:19because of course you wanna like have a pull request
00:04:21against the latest state of the repository,
00:04:24of the main branch, for example,
00:04:26in order to make sure that you don't have to merge
00:04:28every pull request before a new one can be opened.
00:04:30In the end, this merge queue feature exists,
00:04:34which has the simple goal of effectively creating
00:04:38like an intermediate merge already
00:04:42of creating a new state of the repository of the branch
00:04:46you were trying to merge against for every pull request.
00:04:49And if a new pull request is added
00:04:51to the chain of pull requests,
00:04:53that is also already merged combined with the pull requests
00:04:57in front of it into the main branch
00:04:58so that new pull requests are opened
00:05:01as if the prior pull requests had already been merged.
00:05:05And that simply allows teams to work quicker
00:05:08because you can open more and more pull requests
00:05:10without having the ones in front of it first.
00:05:13At some point, of course, they will be merged,
00:05:15but it allows you to keep on working,
00:05:17which of course is important for big teams, for example.
00:05:19Now what's also important related to that feature
00:05:22is of course that it works correctly.
00:05:24And what happened on April the 23rd was
00:05:28that there was an error, an internal logic error
00:05:32in how GitHub resolved these different pull requests
00:05:37so that ultimately it would create a merge
00:05:41that would drop some information that would lead
00:05:45to an invalid commit and get rid of parts
00:05:49of the Git history there.
00:05:50Now the data was not actually lost,
00:05:53but this feature worked incorrectly
00:05:55and produced that incorrect commit.
00:05:57That's the short version of it, the gist of it.
00:06:00And of course, also totally unacceptable
00:06:03if you were a big company or any company relying
00:06:06on that feature and suddenly your project ends up
00:06:09in a broken state without you having a clear explanation
00:06:13for it, that is unacceptable, of course.
00:06:16And your first thought of course is probably not
00:06:19that there is some internal bug in that merge queue feature.
00:06:23It's probably that you did something wrong.
00:06:26So you spend a lot of time searching for the error
00:06:28until you find out, oh no, it's GitHub.
00:06:30And that of course all comes in addition
00:06:33to the ongoing uptime downtime issues GitHub has.
00:06:38Now the official status page looks bad,
00:06:42but maybe okay, but we don't have three nines
00:06:46of uptime here either, at least for most systems.
00:06:49They do report uptime separately for different systems.
00:06:53But things look a bit different if we look
00:06:55at the missing GitHub status page,
00:06:57which tracks uptime in a different way
00:07:00and counts every small incident as a problem,
00:07:04as a downtime in the end.
00:07:05Here we have a horrible uptime for such a crucial system
00:07:10like GitHub, totally unacceptable, of course.
00:07:14So we had uptime issues for the last months
00:07:18and even in the last year already.
00:07:20And there also have been smaller bugs here and there,
00:07:23just not as big as this one or as important
00:07:26as this security vulnerability.
00:07:28But yeah, there are many problems
00:07:31and GitHub has definitely become a unreliable platform
00:07:36at this point, unfortunately,
00:07:38which is a disaster given its role and its importance
00:07:43in, as I said initially, modern development,
00:07:47no matter which kind of development work you're doing.
00:07:50Another problem is that communication from GitHub site
00:07:54has been, let's say, not a lot.
00:07:59There hasn't been a lot of communication,
00:08:01but there has been a blog post shared on April 28th
00:08:06before that security vulnerability,
00:08:10where they kind of explain what's going on,
00:08:14where the problems are coming from,
00:08:16that they understand that their communication strategy
00:08:19hasn't been ideal and that things will get better.
00:08:23That's now the next part.
00:08:25Where are the problems coming from?
00:08:28The official statement here states AI as a reason,
00:08:32but not in the sense of GitHub engineers
00:08:36at Microsoft using AI and shipping broken software,
00:08:40broken updates to GitHub.
00:08:43That may be happening, but we have no proof for that.
00:08:47But instead, the main reason cited here is, of course,
00:08:51that because of AI, there are so many more projects
00:08:57being created, so much more code being generated,
00:09:00and ultimately all those projects and all that code
00:09:03being pushed to GitHub.
00:09:04And they share some, well, yeah, not super helpful,
00:09:09but they share some charts here.
00:09:12They're not super helpful because we have no y-axis.
00:09:14We don't see the absolute numbers,
00:09:17but of course we can see the relations here.
00:09:20And we can, of course, see that over 2025,
00:09:23there was a steep increase in pull requests merged,
00:09:28commits pushed, and of course also new repos being opened.
00:09:32That's all our site projects we're now creating
00:09:34and not finishing with AI.
00:09:36And then in 2026, obviously for all these metrics,
00:09:41the chart goes, just skyrockets into, well, the sky, I guess.
00:09:46So yeah, that is of course a pretty clear trend.
00:09:49And this traffic, this kind of increase in traffic
00:09:54would of course put any system under stress.
00:09:58It's particularly problematic for GitHub
00:10:01because they are in the midst of migrating away
00:10:05from a monolithic structure and from their own dedicated
00:10:09data centers or systems into the Azure cloud
00:10:13and into a more broken up system, a microservices system,
00:10:17you could say, instead of that monolithic structure.
00:10:21That was an ongoing process before we entered 2026 already.
00:10:26But of course it means that now this migration process
00:10:31is hit with that spike in demand,
00:10:34which means even though you're migrating,
00:10:36you have to kind of stabilize the current system
00:10:39while continuing the migration,
00:10:40which then hopefully will help with that increase
00:10:44in traffic in the future.
00:10:46That's the hope, of course, no guarantee.
00:10:50But of course it is something GitHub has to deal with.
00:10:52Now they're stating here that they started executing a plan
00:10:56to increase GitHub's capacity by 10x in October, 2025.
00:11:01So you could say around here they saw,
00:11:04well, this is all going up.
00:11:06I mean, they could see that from before already,
00:11:09but it's here where they decided we need to 10x our capacity.
00:11:13And then in February, 2026, they saw,
00:11:16okay, we need 30x, not 10x because, well,
00:11:20because of that development here, right?
00:11:22That of course must be done in addition to that migration.
00:11:28And that is a huge task, obviously.
00:11:33Now it is part of Microsoft, so it's not some small startup,
00:11:37but nonetheless, it's a daunting task.
00:11:39And this is one aspect of this entire GitHub problem
00:11:44where I have some sympathy because I think it's easy
00:11:47to hate on GitHub, to scoff at GitHub.
00:11:51And you definitely can.
00:11:52And I'll get back to more problems, which are really bad.
00:11:56But this kind of traffic increase would be a huge problem
00:11:59for any system, for any company out there.
00:12:03And it's hard to believe that any GitHub competitor
00:12:07would do better in this situation.
00:12:09Still, of course, that's no excuse.
00:12:10It's part of Microsoft.
00:12:12And therefore, they of course definitely have the resources
00:12:16to go through that transition and adjust their systems
00:12:20to this new world and to this new amount of traffic.
00:12:24But there is another important problem here with GitHub.
00:12:28And that is that it has no longer a CEO.
00:12:32The previous CEO, Thomas, Thomas Domke,
00:12:37retired or stepped down or announced that he would step down
00:12:41in August, 2025.
00:12:43And Microsoft did not assign a new CEO.
00:12:48Instead, GitHub became part of Core AI,
00:12:51an internal division at Microsoft that, as the name suggests,
00:12:56is all about AI and building AI tools and platforms.
00:13:01And GitHub is part of that.
00:13:03So clearly the mission of GitHub from Microsoft's perspective
00:13:07is to become part of that AI tool chain,
00:13:11of that AI revolution.
00:13:13And obviously Microsoft is pushing Co-Pilot
00:13:15into all their products.
00:13:16And indeed at GitHub Universe 2023,
00:13:20they already said that they will transform GitHub
00:13:24into the AI-powered developer platform
00:13:28with GitHub everywhere.
00:13:30That includes stuff like new features
00:13:32that help with creating issues with GitHub Co-Pilot,
00:13:36which is a huge problem for open source maintainers,
00:13:39but also just the pure presence of GitHub Co-Pilot
00:13:43everywhere on GitHub.
00:13:44There is this Agent HQ thing here on GitHub,
00:13:48github.com/copilot,
00:13:49where you can interact with GitHub Co-Pilot
00:13:52and work on your code right from inside GitHub Co-Pilot
00:13:55without ever opening up a local IDE or coding agent tool
00:14:00and many, many more parts.
00:14:02GitHub Co-Pilot is everywhere in GitHub,
00:14:05just like Co-Pilot is everywhere
00:14:07in all Microsoft products, I guess.
00:14:10And that of course is a clear strategic decision
00:14:14which kind of goes against the actual mission of GitHub,
00:14:19at least the mission GitHub had in the past.
00:14:23Because as I mentioned at the very beginning,
00:14:25GitHub is important for different kinds of developers
00:14:29for all kinds of use cases.
00:14:31Open source maintainers use it to have their source code
00:14:36there and collaborate with other maintainers
00:14:39and other contributors from the community.
00:14:41Issues are vital for detecting, well, issues
00:14:45and working on them.
00:14:46Pull requests are important for having other people
00:14:50contribute to the code base.
00:14:52Discussions can be great for discussing new features
00:14:55or directions of a repository or of a library and so on.
00:15:01There are many features related here
00:15:03that help open source maintainers
00:15:04or at least helped in the past.
00:15:07Other people are using GitHub just as a resource
00:15:11for hosting links or documents
00:15:13like all these awesome repositories, awesome Go, awesome Rust
00:15:17and so on, which you can use to easily find resources
00:15:20if you wanna work with Go or Rust.
00:15:22I'm using GitHub also for hosting my course resources
00:15:26like here for my Codex course, for example,
00:15:29and for many other courses as well.
00:15:31So you can even abuse GitHub
00:15:33as just a kind of a document storage.
00:15:36And then of course you can also use GitHub for CI/CD work.
00:15:40In a company, you may be using GitHub
00:15:43to of course have your source code there,
00:15:46to have your team members collaborate on that source code
00:15:50with pull requests and so on.
00:15:52And then of course, GitHub very often
00:15:54is also part of the CI/CD pipeline
00:15:57where a new push to the main branch, for example,
00:15:59triggers a CI/CD pipeline.
00:16:02That could be with help of GitHub actions,
00:16:05though that product has its own problems.
00:16:08But of course it could also be to trigger a CI/CD pipeline
00:16:12on any other CI/CD provider, not just GitHub action.
00:16:16So GitHub of course has a very important role
00:16:20for classic traditional development work.
00:16:24But of course, Microsoft decided that nope,
00:16:27it should become an AI powered developer platform,
00:16:31not just a developer platform.
00:16:33And that of course is kind of a mismatch here.
00:16:37Developers don't necessarily want co-pilot
00:16:41in every aspect of GitHub.
00:16:43I guess users of Microsoft products in general
00:16:46don't want GitHub in all their products,
00:16:48but that's a different story.
00:16:49And GitHub has been neglecting the core features
00:16:53that are important for developers.
00:16:56And I mean, take open source development work.
00:17:00Open source project maintainers are drowning
00:17:03in AI generated issues and pull requests.
00:17:07Now the problem here of course is asymmetry.
00:17:10It's easy to use AI to generate code or issues.
00:17:14It's way harder to review all that stuff.
00:17:19So to review that generated code and those generated issues.
00:17:24And I mean, that's something every developer knows
00:17:26who ever worked with AI.
00:17:27You can easily spin up three AI agents or more
00:17:30and have them work on your projects,
00:17:32totally outside of GitHub.
00:17:33You can do that on your machine with codecs,
00:17:35cloth code and so on.
00:17:36But then if you're not going down the wipe coding route,
00:17:39which you shouldn't in my opinion,
00:17:41you have to review that code at some point.
00:17:44And that takes time.
00:17:45And it's not a lot of fun, at least for me.
00:17:48Now, if you spin up three agents,
00:17:51you have to review the output of three agents.
00:17:54You can reduce the amount of agents if that's too much
00:17:57for you and you find that you're not really productive
00:17:59that way.
00:18:00Now, when you're an open source maintainer on GitHub,
00:18:03you're drowning in AI generated issues and pull requests
00:18:07and you never have two main options.
00:18:09You can ignore them and that kind of defeats their purpose
00:18:13of course, but it is a valid strategy obviously.
00:18:16Or you try to work your way through them
00:18:18and you get burned out because it's just too much
00:18:21because unlike with your own personal development work,
00:18:25you can't just reduce the amount of incoming issues
00:18:29and pull requests.
00:18:30You can use less agents on your own
00:18:33if you find that you're not effective or not productive
00:18:36with all the agents you're trying to run.
00:18:38You can't do that with the public repositories.
00:18:41You can't control how many people will post AI generated
00:18:45issues or share pull requests with you.
00:18:49So that is a huge issue for open source maintainers
00:18:53and why the entire open source scene
00:18:56and the philosophy behind open source
00:18:59is in huge problems right now because of AI.
00:19:04And GitHub is not helping with that.
00:19:06Instead, they're doing the opposite.
00:19:08They are actively making it easier for AI slop issues
00:19:13to be shared and so on.
00:19:15What maintainers and developers would need
00:19:18would be more effective tools for dealing
00:19:22with all these AI generated issues and pull requests.
00:19:25But GitHub is not working on that.
00:19:27It's not part of their strategy, I guess.
00:19:29Now, maybe that will change.
00:19:30That official post by GitHub I mentioned earlier
00:19:35primarily talks about the reliability and uptime issues
00:19:39and that they wanna be more transparent and so on.
00:19:41But they also mentioned that they have a commitment
00:19:44to support developers.
00:19:46We'll see, I'm not too positive
00:19:49because ultimately it's part of Microsoft
00:19:52and they have their very own strategy here.
00:19:55But what does this mean for GitHub then?
00:19:59Is it time to migrate away?
00:20:02I've heard some voices here and there on X
00:20:05that it's now time for a GitHub alternative.
00:20:08I know that some projects have migrated away.
00:20:12SIG is maybe the most prominent one.
00:20:15They migrated from GitHub to Codeberg in November, 2025.
00:20:20But let's be realistic here.
00:20:22For one, as I mentioned before,
00:20:24the amount of traffic that's hitting GitHub
00:20:28would overwhelm any competitor as well.
00:20:31Likely even more than GitHub
00:20:32because they're not part of Microsoft.
00:20:35So we definitely will not see GitHub being replaced.
00:20:40And while some individual projects,
00:20:42especially open source projects may exit GitHub
00:20:45for reasons I can totally understand,
00:20:48all those companies, all those individual developers
00:20:52will likely not migrate away.
00:20:54GitHub has, despite all its issues,
00:20:57a feature-rich platform with features that are integral
00:21:02of many developers' workflows and day-to-day work.
00:21:06Especially for companies, of course,
00:21:08it's not easy for them to just replace GitHub
00:21:11with some other provider.
00:21:13Even though all the reliability issues
00:21:15are obviously huge issues for companies as well,
00:21:18they will be able and willing to endure a lot more pain
00:21:23before they would even consider moving away.
00:21:25I'm certain of that.
00:21:26GitHub is just too important of a platform.
00:21:30It's the platform for putting your Git managed code
00:21:35into the cloud and working on it and collaborating on it.
00:21:39So I'm sure it's not going anywhere,
00:21:43even if the situation would get worse.
00:21:45Of course, eventually people would leave
00:21:47if GitHub were not doing anything,
00:21:49but clearly they are,
00:21:50at least regarding the uptime and reliability issues.
00:21:55When it comes to open source work and the issues there
00:21:58and the AI slop issues, we'll see.
00:22:01Even there, I believe that GitHub is just too important
00:22:07and has too many advantages for open source maintainers
00:22:10to just leave, at least like all of them.
00:22:14But I definitely understand if individual projects
00:22:17move away from GitHub, so that may happen.
00:22:20But yeah, for companies and GitHub in general,
00:22:23it'll stick around.
00:22:24Nonetheless, one can only hope that this situation here
00:22:28is maybe, maybe a wake up call for Microsoft.
00:22:33Maybe they'll put a CEO back in charge for GitHub.
00:22:38They maybe understand its importance.
00:22:41They maybe understand that it's a developer
00:22:45and development platform, not primarily an AI platform.
00:22:49But yeah, one can hope.
00:22:52I don't know if and when that will happen.
00:22:55But yeah, that is the current GitHub situation.
00:23:00It's bad, it's really bad.
00:23:03And it will stay bad for the near future,
00:23:06but at least the reliability will hopefully get better
00:23:11later this year.
00:23:13We'll see, I guess.