I have a process which copies emojos from the hajkey blahaj.zone site to our lemmy.

Last night (for me) it copied an invalid emojo into the database and it broke lemmy for hours while Ada and I were sleeping.

I’ve found the issue and fixed it now, so everything should be purring along once again, but for those who were left without the instance to use, I sincerely apologise.

blobcat, sorry

  • ‘Leigh 🏳️‍⚧️
    link
    fedilink
    English
    arrow-up
    30
    ·
    1 year ago

    No worries, things happen — and you’re community-building volunteers, not an on-call rotation. 💜 I’m just glad we’re back now! Thanks for your efforts during these past 24 hours. 🏳️‍⚧️😊

  • Paige (she/her)
    link
    fedilink
    English
    arrow-up
    22
    ·
    1 year ago

    I’m going to imagine that :capybara_animated: was chewing on the server racks

    animated, capybara

    • Kaity AOPMA
      link
      fedilink
      English
      arrow-up
      28
      ·
      1 year ago

      It wasn’t an actual emojo. The script processed the SQL header column names as an emojo and tried to add them. Unfortunately publicUrl is not a valid URL, so lemmy’s /api/v3/site metadata endpoint started returning an error relative URL without a base instead of the JSON that the website was expecting and so the site just stopped working for everyone the next time it tried to load that url.

      • bamboo
        link
        fedilink
        English
        arrow-up
        11
        ·
        1 year ago

        This coincided with me resetting my password, and I thought I brought down the site somehow

  • Noodles
    link
    fedilink
    English
    arrow-up
    16
    ·
    1 year ago

    happy the best instance is back up!! crabrave

  • Krrygon@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    8
    ·
    1 year ago

    Excellent news! The site loads perfectly for me. I am still unable to log in for some reason, but that could just be a me thing.

    • Noodles
      link
      fedilink
      English
      arrow-up
      8
      ·
      1 year ago

      clear your cookies/cache for this site, it should work after that

      • Krrygon
        link
        fedilink
        English
        arrow-up
        9
        ·
        1 year ago

        It works! I am back in the blahaj zone. Feels good.

  • Wirlocke
    link
    fedilink
    English
    arrow-up
    5
    ·
    1 year ago

    I was worried that it got hacked again, since that just happened. Glad that Blahaj lives again!

  • stercar
    link
    fedilink
    English
    arrow-up
    5
    ·
    1 year ago

    So glad it‘s up again. I was thinking it was something like that since it kept giving a parsing error when I was trying to log in. Good that it‘s fixed now.

  • djstini
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    {“data”:{“msg”:“Couln’t upload file, Error in DB, IO error: Os { code: 28, kind: StorageFull, message: "No space left on device" }”,“files”:null},“state”:“success”}

    Hmm, still getting errors, for example when trying to upload profile images

  • essi
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    It happens! Thanks for the writeup, and good work bringing things back online again :)

  • Rozaŭtuno
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    Mistakes happen, I’m just happy the instance is back up.

  • alx
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    Welcome back to the Friends From Sweden and thank you to the admins

  • Lotus Eater
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    Appreciate the fix, but I am having trouble uploading images, wonder if anyone else is having them?