2026-07-12T13:01:09Z https://nostr.ae Codeberg Status https://nostr.ae/npub1z0ase7phympjeflh9wqm6j3gyuwcftpu7udummcfnr49s5yf0p5qfmdh9p https://social.anoxinon.de/system/accounts/avatars/115/651/402/898/351/746/original/cee2bc84c955b291.png https://social.anoxinon.de/system/accounts/avatars/115/651/402/898/351/746/original/cee2bc84c955b291.png https://nostr.ae/nevent1qqsv2zsrupmmmkl8klp2rpwm35xz4zk9ljv9djtr3f96qpj2jw44wcgzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxse7emaw In reply to <a href='/nevent1qqsx9mefmhl4nqnqjhmk3yy52slqanelkpvt78mayrlg4w9gpv5jdmguskh2y'>nevent1q…kh2y</a><br/>_________________________<br/><br/>We have recovered services after power returned, leading to a total downtime of about 7 hours.<br/><br/>We apologize for the impact and we are in touch with our datacenter operator to discuss the incident. 2026-06-28T06:30:48Z https://nostr.ae/nevent1qqsx9mefmhl4nqnqjhmk3yy52slqanelkpvt78mayrlg4w9gpv5jdmgzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsvrwydp In reply to <a href='/nevent1qqsr29946yu3cyrqvevgafceckkn0w0let77ag574vxph2e705dlqdsvqgumm'>nevent1q…gumm</a><br/>_________________________<br/><br/>We do not expect the issue to be resolved soon. We are waiting for a fix from the datacenter operator and will monitor the situation in order to restore service as soon as possible after power returns.<br/><br/>Currently, one server is with power again, but the others remain unreachable. Our investigation shows that network equipment is also affected. 2026-06-28T00:03:56Z https://nostr.ae/nevent1qqsr29946yu3cyrqvevgafceckkn0w0let77ag574vxph2e705dlqdszyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxs72q854 Hello everyone,<br/><br/>All the services we offer are down and unreachable. We are currently investigating the cause. For the time being, it appears that all three servers are without power. 2026-06-27T22:32:21Z https://nostr.ae/nevent1qqsvcurglwtcdjxfm8ecst9334yw4jphlrvzqnwqxhh0mlnrf3t3j6qzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxshedayn We are struggling to keep Codebreg.org available for unauthenticated users due to massive abuse of expensive endpoints.<br/><br/>Our current priority is keeping Codeberg.org responsive for authenticated users. 2026-06-17T12:33:35Z https://nostr.ae/nevent1qqszs2f7sf66a6snzm6ngg9swzssff5v7an78ly7emdq6wjepra6v8gzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxswe6fek Good news: We have addressed recent SSH performance degradation by replacing linear parsing of the authorized_keys file with a custom AuthorizedKeys command.<br/><br/>This is what GitHub and GitLab have been doing for years, and we have grown to a size where this has become necessary for us as well.<br/><br/>The cause for the degradation was still abusive patterns, as connections without valid key take more resources (scanning the file to the end) than legitimate users (scanning is stopped after match). 2026-05-27T14:38:35Z https://nostr.ae/nevent1qqsrhwf44mfjpknv5xnxg9nclqjgxhrcv3f5emrcxr0xj5g2e464h3qzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxss9mrr3 In reply to <a href='/nevent1qqs8nsrzd3fqd5pgq8ygeqxcse3js4ycvmd5ecw6j65jpd99s4y7yyspfa74k'>nevent1q…a74k</a><br/>_________________________<br/><br/>The main Forgejo instance is back, SSH also works again. We have no remaining maintenance for this evening, all services should be working again without any hiccups.<br/><br/>If you feel or see any problem, please report it to <a href="https://codeberg.org/Codeberg/Community/issues">https://codeberg.org/Codeberg/Community/issues</a> 2026-05-18T21:17:32Z https://nostr.ae/nevent1qqs8nsrzd3fqd5pgq8ygeqxcse3js4ycvmd5ecw6j65jpd99s4y7yyszyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsqz3ajz In reply to <a href='/nevent1qqsvlydtgesm5kmq6mlct36mv0p0kw2l6kdcxh5zvd8pn7evy8h7qlgs9sz8s'>nevent1q…sz8s</a><br/>_________________________<br/><br/>This reboot has now finalized, BIOS and iDRAC update was successful. We&#39;re now waiting on the Galera node to be fully synced to the cluster and then we&#39;ll move the main Forgejo instance back. 2026-05-18T21:02:00Z https://nostr.ae/nevent1qqsvlydtgesm5kmq6mlct36mv0p0kw2l6kdcxh5zvd8pn7evy8h7qlgzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxs5277cj In reply to <a href='/nevent1qqs9kzsqj90dj52y0h39agea99t3jqjvpef6za4wsvfefkwdpsdugrgksh456'>nevent1q…h456</a><br/>_________________________<br/><br/>We&#39;re starting the last reboot of the evening, the one that has the main Forgejo instance. We&#39;ve moved this to another server temporarily, all data, queues and session were already stored high availability a while ago so this is luckily a trivial task. Unfortunately we forget to sync SSH host keys, that will stay down<br/><br/>This one has not been rebooted for 110(!) days. We&#39;ll also be updating the BIOS and iDRAC with this reboot, so this one will take a bit longer. But no services should be impacted. 2026-05-18T20:25:08Z https://nostr.ae/nevent1qqs9kzsqj90dj52y0h39agea99t3jqjvpef6za4wsvfefkwdpsdugrgzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsrf4a67 In reply to <a href='/nevent1qqstgnsmmpafa26dnptfjtcnxkluwdy538t38y4np54rvmumvjlrx9cr2w4cv'>nevent1q…w4cv</a><br/>_________________________<br/><br/>We noticed that during this reboot, CephFS barely responded to any FS I/O requests. This caused 504 errors to be seen on codeberg.org during this reboot; that was not our intention sorry for that.<br/><br/>The server we rebooted was the same that had &#34;problems&#34; where Ceph didn&#39;t allow it to go into maintenance mode: <a href="https://social.anoxinon.de/@codebergstatus/116523668707929156">https://social.anoxinon.de/@codebergstatus/116523668707929156</a><br/><br/>We now have a feeling the problem is still not completely fixed and there&#39;s still a big dependency on this server for Ceph to work.<blockquote class="border-l-05rem border-l-strongpink border-solid"><div class="-ml-4 bg-gradient-to-r from-gray-100 dark:from-zinc-800 to-transparent mr-0 mt-0 mb-4 pl-4 pr-2 py-2">quoting <br/><span itemprop="mentions" itemscope itemtype="https://schema.org/Article"><a itemprop="url" href="/note1pax4k9urmaqvy04uf4c5zu9nmggyq94v7qvpywmjzd6pzgux22hsqg8rm6" class="bg-lavender dark:prose:text-neutral-50 dark:text-neutral-50 dark:bg-garnet px-1">note1pax…8rm6</a></span><br/> </div> Hi,<br/><br/>It's taking longer than expected. Ceph made a surprise move on us at the last moment by telling us our network configuration is maybe no longer up to date after we replaced servers (not today, but a while ago). This is preventing us from deploying new monitors to keep quorum. We're working on fixing this.<br/><br/>It's _simply_ preventing us from gracefully shutting down, no data is at risk. </blockquote> 2026-05-18T20:00:47Z https://nostr.ae/nevent1qqstgnsmmpafa26dnptfjtcnxkluwdy538t38y4np54rvmumvjlrx9czyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsnwqxw2 In reply to <a href='/nevent1qqsxzndfl7a20d0c9ffgryjtzkkd2qw40hcps9h8utw7g0wzhh08exgwnnr9j'>nevent1q…nr9j</a><br/>_________________________<br/><br/>We&#39;ll be starting the reboot of the second server. This server only has services that are deployed with high availability, except the Woodpecker agent. 2026-05-18T19:44:04Z https://nostr.ae/nevent1qqsxzndfl7a20d0c9ffgryjtzkkd2qw40hcps9h8utw7g0wzhh08exgzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxshqh2cs In reply to <a href='/nevent1qqsv9r0zn7wmat3ug8y0vg3uxga3gejxj59dtpajkqg9xaglelvpqzqs7t096'>nevent1q…t096</a><br/>_________________________<br/><br/>This rebooted completed. We noticed that HAProxy didn&#39;t start on its own because it tried to read from Ceph before its mounted, we forget to add a mount dependency in the systemd service.<br/><br/>Unfortunately all services that uses the Galera cluster takes more time to start. We noticed for a while that when a Galera node joins the cluster it does a full SST instead of a incremental sync. Which is painfully slow. 2026-05-18T19:18:37Z https://nostr.ae/nevent1qqsv9r0zn7wmat3ug8y0vg3uxga3gejxj59dtpajkqg9xaglelvpqzqzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxs2elj30 In reply to <a href='/nevent1qqsvdyapamwq52n8t65ntrlkr98kc6u5wlddv3vc03j5ag4d2pwj7vqfxucxm'>nevent1q…ucxm</a><br/>_________________________<br/><br/>We&#39;ll be starting to reboot the first server. This server hosts Pages, Translate, Forgejo runner, e.V. forum and join.codeberg.org. 2026-05-18T18:59:02Z https://nostr.ae/nevent1qqsvdyapamwq52n8t65ntrlkr98kc6u5wlddv3vc03j5ag4d2pwj7vqzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsnr6pte In reply to <a href='/nevent1qqsqmd742ykamj66png3yptqdkuuldsjuy5c996x97zjmsz2gqqct5se8qqa5'>nevent1q…qqa5</a><br/>_________________________<br/><br/>To be clear, we mean to boot into a patched Linux kernel version. We have no intentions to migrate to a BSD distribution. 2026-05-18T18:48:21Z https://nostr.ae/nevent1qqsqmd742ykamj66png3yptqdkuuldsjuy5c996x97zjmsz2gqqct5szyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsegd9d7 You might&#39;ve heard it by now: the Linux kernel is so insecure that AI can find many LPEs for them, so we&#39;ll be taking this evening to migrate to *BSD.<br/><br/>In all seriousness, although we&#39;ve mitigated the currently known LPEs via blocking modules, seccomp and sysctl, we still want to run with a kernel that has patched it. We will be doing that this evening. That means that each server will be taken down to boot into the new kernel, and services will be temporarily inaccessible. 2026-05-18T18:28:33Z https://nostr.ae/nevent1qqst5z20kx45d07s7xk99q7mqf3ses7hzltdduyrngd3c3ua3c7rlfczyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsjwu0f6 It has been in the works for months, but the recent increase of downtime has accelerated work. We&#39;ve moved from Caddy to HAProxy for codeberg.org traffic.<br/><br/>Starting over with a new reverse proxy has enabled us to make choices that enable us to set up HAProxy to scale with our increasing traffic. For example, traffic is split (when possible, decided via heuristics) over several instances to load balance expensive traffic.<br/><br/>We hope to share more details at a later time about this change. 2026-05-08T17:07:58Z https://nostr.ae/nevent1qqsysuqcxpasrx6yu8dsahppd599yu655c7mzwwhj2h8uvlu5d5wg5qzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsc3xgdr Please ignore. <br/><br/>Hi Matrix chat! 2026-05-08T09:29:16Z https://nostr.ae/nevent1qqsq7n2mz7pa7sxz867y6u2pwzea5yzqz6k0qxqj8depxaq3ywr99tczyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsch7dmm In reply to <a href='/nevent1qqsx7pugcdl93qaclwtmssc6t8hdfepugm4splwm64w8xr0htlrm6xs3wrmyz'>nevent1q…rmyz</a><br/>_________________________<br/><br/>Hi,<br/><br/>It&#39;s taking longer than expected. Ceph made a surprise move on us at the last moment by telling us our network configuration is maybe no longer up to date after we replaced servers (not today, but a while ago). This is preventing us from deploying new monitors to keep quorum. We&#39;re working on fixing this.<br/><br/>It&#39;s _simply_ preventing us from gracefully shutting down, no data is at risk. 2026-05-05T19:41:21Z https://nostr.ae/nevent1qqsx7pugcdl93qaclwtmssc6t8hdfepugm4splwm64w8xr0htlrm6xszyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxssgupk2 We&#39;ll be performing a reboot of all our servers this evening.<br/><br/>This means that our services (CI, pages, translate) will be temporarily offline. We invested a lot of effort in the last few months to be able to move Forgejo between servers easily and this should see minimal downtime. 2026-05-05T17:55:54Z https://nostr.ae/nevent1qqs0f7z2y3dvzmyjfl5x58fk69g867cfg7hct4x2gwyum6cugfhwhkgzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsm5h8k7 In reply to <a href='/nevent1qqsvgslx9twcr9sg9wr37rxnacvsnqlunm4554psqaffdt3pv7w0a2gdz2wjv'>nevent1q…2wjv</a><br/>_________________________<br/><br/>Hi all, sorry for the long wait.<br/><br/>We wanted to preemptively ensure that our servers were sufficiently shielded against <a href="https://copy.fail">https://copy.fail</a>; we found no evidence of a compromise.<br/><br/>We waited for the greater community to react and provide some additional clarity on what is affected and what isn&#39;t (e.g. container engines), we carefully studied the LKML, and waited for upstreams to release official updates and guidance. 2026-04-30T16:51:03Z https://nostr.ae/nevent1qqsvgslx9twcr9sg9wr37rxnacvsnqlunm4554psqaffdt3pv7w0a2gzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxscpzz9m In reply to <a href='/nevent1qqs0hyu0rc93vffstuh4ajamfm3gtlndgaawf4zr8qjjfzccrp5f44ggehr5c'>nevent1q…hr5c</a><br/>_________________________<br/><br/>We estimate that the recovery process of our CI services may potentially take multiple hours. We ask for your understanding as we are trying to reinforce our services&#39; reliability. 2026-04-29T22:18:43Z https://nostr.ae/nevent1qqs0hyu0rc93vffstuh4ajamfm3gtlndgaawf4zr8qjjfzccrp5f44gzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsltprd2 In reply to <a href='/nevent1qqswafxpfrq2jhrxdcgtkaackdvukds9drh0ylq2hrdwgmgvvft0n0s3zsm9c'>nevent1q…sm9c</a><br/>_________________________<br/><br/>We are expanding our maintenance efforts to include our Forgejo Actions CI as well. Again, we apologize for the inconvenience. We&#39;ll try to bring both services back up as soon as possible. 2026-04-29T22:02:40Z https://nostr.ae/nevent1qqswafxpfrq2jhrxdcgtkaackdvukds9drh0ylq2hrdwgmgvvft0n0szyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsl9dtk6 Our hosted Woodpecker CI/CD is going down for maintenance.<br/><br/>* Builds will be delayed.<br/>* Web UI will be inaccessible for some minutes.<br/>* Commits pushed during the downtime will not trigger builds.<br/><br/>We apologize for the inconvenience. 2026-04-28T19:01:36Z https://nostr.ae/nevent1qqsr0yfxe2s7pe2lggqhp6j8ames70mfgu3sxlfej4ag0eml47myfnczyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxscgnsu2 We are aware of reports of Codeberg being down and are investigating. Thank you for your patience. 2026-04-02T07:16:21Z https://nostr.ae/nevent1qqst9rkf3ta2txwf8a3fzzyglccc5uja9wpe0q4n5sqlnlmut24lzgszyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxs2t5450 In reply to <a href='/nevent1qqsvrnklprfhcty73n997k9hky3y24hdxrum4d3u9yq4gzkpvw7ypwc246gxd'>nevent1q…6gxd</a><br/>_________________________<br/><br/>We were able to restore IPv4 and IPv6 connectivity. Performance is still not optimal, and connections are occasionally dropping. We&#39;ve taken measures to try and maintain uptime overnight (with a certain probability).<br/><br/>Routes protected by iocaine (crawler protection) may return 503 errors. Currently, it appears that these packets are also occasionally being lost.<br/><br/>It&#39;s 2:30am, and we require a good night&#39;s sleep before we can look into this further. 2026-03-05T01:30:46Z https://nostr.ae/nevent1qqsvrnklprfhcty73n997k9hky3y24hdxrum4d3u9yq4gzkpvw7ypwczyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsjhd4x6 Hi everyone! We are still fighting with an ongoing DDoS attack against our service on the network level. We do not yet have any ETA on recovery time, as we&#39;re still investigating different mitigation strategies.<br/><br/>As always, we are very grateful for your support and patience. 2026-03-04T22:09:08Z https://nostr.ae/nevent1qqswfplsusqs94rxuf67lh5xa420tdwpewmr7l07r6vjzuqm4hem0nqzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxs0sh83c In reply to <a href='/nevent1qqspwcga6wllkmdzjutveu9e84srktre2wz94lrx69e5ndnlae7ralscr8kgn'>nevent1q…8kgn</a><br/>_________________________<br/><br/>Yesterday we didn&#39;t found the time to deploy, we will start doing another rebase of our patches on top of some last fixes and prepare for a deployment now. We are aiming to deploy at 20:00 CET (19:00 UTC) 2026-01-15T18:20:59Z https://nostr.ae/nevent1qqspwcga6wllkmdzjutveu9e84srktre2wz94lrx69e5ndnlae7ralszyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsraqzfj With Forgejo&#39;s scheduled v14 release for tomorrow, we will do our final tests of v14 tonight on our test instance.<br/><br/>If we can&#39;t find any issues with v14, we might deploy it today. We will let you know ahead of time when this happens, so you can prepare for 10-15 minutes of downtime. 2026-01-14T10:35:49Z https://nostr.ae/nevent1qqsxqas8gxcdpzx2haqu2663tcl7zzxzd5k0e5kz7x882p8fmg7c76czyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsm9tssw We&#39;re investigating an outage of our Ceph storage system. We&#39;ll hit you up once we know more.<br/><br/>Currently, most Codeberg services are not available. 2026-01-04T21:09:04Z https://nostr.ae/nevent1qqsfnkxngsklyrjsqdc6hcrmd5zdat9y0u44ngyfhhy23n0wmzvj7sqzyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsxv3xvm We are investigating performance issues on Codeberg.org. 2025-12-16T16:01:35Z https://nostr.ae/nevent1qqsq2g5nx2etqg537zttufa02tep97rw92jarz70m6wrvhzc0leqrfszyqflkr8cxunvxt987u4cr0229qn3mp9v8nm3hn00pxvw5kzs39uxsxzsp0h Welcome to our new dedicated account for service notifications surounding Codeberg&#39;s systems.<br/><br/>We hope there won&#39;t be much to share in the near future :) 2025-12-04T14:59:50Z