Hi, I just spun up Lemmy 0.17.3 in my kubernetes cluster but I’m having trouble getting it to federate with anything.

I can curl the API endpoints for local posts which all looks good, but all searches fail. In the logs for the backend the stack trace looks like it’s failing at trying to resolve the object.

My instance is https://campfyre.nickwebster.dev (which is funny because I briefly ran a hand-made social network called Campfyre from ~2014-2016)

Edit: I am now running 0.18.0 and still have the problem with search.

Edit 2: I added a RUN update-ca-certificates step to my docker container for lemmy_server and now I can do a direct connection (i.e. https://campfyre.nickwebster.dev/c/memes@reddthat.com) although search still fails.

  • Freeman@lemmy.pub
    link
    fedilink
    English
    arrow-up
    3
    ·
    edit-2
    1 year ago
    1. Make sure you don’t have private instance checked. No idea what it does but I noticed in the logs you can’t have both private instance and federation checked. I’m guess private is just that, no federation, just a walled garden, possibly for non-prod and test.

    2. Leave allowed instances blank. Only use blocked to defederate ones that are undesirable.

    3. Start searching instances and communities in the search feature. This will reach out to them and start syncing posts and content.

    Also, may want to look into 0.18.0. It doesn’t use web sockets for federation anymore. Which is a lot more efficient and I’m seeing less syncing issues with things like comments.

    Also don’t know why numbers don’t show in my replied in mobile

    • sp00kedOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Thanks. I’m not private and that is blank. All searches fail, although it might be federating in the background.

      I haven’t upgraded to 0.18.0 yet because the Dockerfile I wrote uses the crates.io release to build Lemmy from source but they haven’t published 0.18.0 on crates.io yet. I can change that to git though. I’ll probably do that tomorrow and see what happens.

      • Freeman@lemmy.pub
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        How are you searching? The best way I found to load up my instance was to head to this site

        https://lemmyverse.net/

        Click the house in the top right and input your instance domain. This will reformat everything to the right search term.

        Then click the little copy link (it’s the one under the primary instance name). And paste that into searc and mash search a few time. The instance will appear in search after a few tries (heavier trafficked instances are slower to respond). Then just open the instance and subscribe. That will start syncing the community. Your federated instances will slowly grow from there.

        • sp00kedOP
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          Yes, I’ve tried searching with the correct format. I also tried directly navigating to a community from mine via a URL, which threw this interesting error:

          LemmyError { message: Some("couldnt_find_community"), inner: Other errors which are not explicitly handled
          Caused by:
              0: error sending request for url (https://lemmy.ml/.well-known/webfinger?resource=acct:cryptography@lemmy.ml): error trying to connect: error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889: (unable to get local issuer certificate)
              1: error trying to connect: error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889: (unable to get local issuer certificate)
              2: error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889: (unable to get local issuer certificate)
              3: error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889:, context: SpanTrace [{ target: "lemmy_apub::fetcher", name: "resolve_actor_identifier", file: "crates/apub/src/fetcher/mod.rs", line: 21 }, { target: "lemmy_apub::api::read_community", name: "perform", fields: "\u{1b}[3mself\u{1b}[0m\u{1b}[2m=\u{1b}[0mGetCommunity { id: None, name: Some(\"cryptography@lemmy.ml\"), auth: Some(Sensitive) }", file: "crates/apub/src/api/read_community.rs", line: 25 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "\u{1b}[3mhttp.method\u{1b}[0m\u{1b}[2m=\u{1b}[0mGET \u{1b}[3mhttp.scheme\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"http\" \u{1b}[3mhttp.host\u{1b}[0m\u{1b}[2m=\u{1b}[0mcampfyre.nickwebster.dev \u{1b}[3mhttp.target\u{1b}[0m\u{1b}[2m=\u{1b}[0m/api/v3/community \u{1b}[3motel.kind\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"server\" \u{1b}[3mrequest_id\u{1b}[0m\u{1b}[2m=\u{1b}[0mfff82c47-e8f0-4a4b-a475-28a09499a8c3 \u{1b}[3mhttp.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m400 \u{1b}[3motel.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"OK\"", file: "src/root_span_builder.rs", line: 16 }] }
          
          • Freeman@lemmy.pub
            link
            fedilink
            English
            arrow-up
            1
            ·
            edit-2
            1 year ago

            That is because the community hasn’t been synced. In 0.17.4 it was just a generic 404 community not found.

            When you are searching are you leaving the filter to all or setting it to community?

            Can you post a screenshot of your filter/search query?

            Id look at the docker log stream as well when you hit search and see if any errors are thrown.

            Looking at your instances page there are a ton on there.

  • terribleplan@lemmy.nrd.li
    link
    fedilink
    English
    arrow-up
    3
    ·
    1 year ago

    I would start by kubectl exec’ing into your pod and using nsllookup/dig/whatever you have available to check whether DNS resolution is working inside there at all. I don’t think there is an easy way to break DNS resolution via config without it being pretty clear that you’re doing so, but you could try messing with the pod spec’s dnsPolicy (I often end up using ClusterFirst) and dnsConfig.

    I may be able to help more if you post your pod spec.

    • sp00kedOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Thanks. I shelled into my container and I was able to successfully do DNS requests (and full HTTP with the outside world).

      0: error sending request for url (https://lemmy.ml/.well-known/webfinger?resource=acct:cryptography@lemmy.ml): error trying to connect: error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme
      

      (and so on)

      My podspec is as follows:

      volumes:
              - name: lemmy-config
                secret:
                  secretName: lemmy-config
            imagePullSecrets:
              - name: ocirsecret
            automountServiceAccountToken: false
            containers:
              - name: lemmy
                image: [...]
                imagePullPolicy: Always
                env:
                  - name: LEMMY_CONFIG_LOCATION
                    value: /etc/lemmy-config/lemmy.hjson
                ports:
                  - containerPort: 8536
                resources: {}
                securityContext:
                  capabilities:
                    drop:
                    - CAP_MKNOD
                    - CAP_NET_RAW
                    - CAP_AUDIT_WRITE
                volumeMounts:
                - mountPath: /etc/lemmy-config
                  name: lemmy-config
                  readOnly: true
              - name: lemmy-ui
                image: [...]
                imagePullPolicy: Always
                ports:
                  - containerPort: 1234
                resources: {}
                securityContext:
                  capabilities:
                    drop:
                    - CAP_MKNOD
                    - CAP_NET_RAW
                    - CAP_AUDIT_WRITE
            enableServiceLinks: true
            hostname: lemmy
            restartPolicy: Always
      
      • terribleplan@lemmy.nrd.li
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        Yeah, I think you were spot on in your diagnosis of that particular error in your updated OP of needing to update-ca-certificates. As far as I can tell based on that podspec you aren’t really doing anything particularly odd that I would expect to break DNS or something at the network layer.

        Is the issue you are seeing in your logs any different now, or still the same as before the ca certs?

        • nick@campfyre.nickwebster.dev
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          I’m not getting the openssl error although search is still a bit wonky. Given that I can directly subscribe to a community I think it’s okay though.