Updated! Updates are shown in quote text like this. Some scores are updated following app updates.

An Apps Experiment

Cross-posted from https://lemmy.world/post/18159531

Introduction

This is an experiment I performed out of curiosity, and I have a few big disclaimers at the bottom. Basically, I’ve seen a lot of comments recently about one app or another not displaying something right. Lemmy has been around for a while now and can no longer be considered an experimental platform.

Lemmy and the apps that people use to access the platform have become an important part of people’s lives. Whether you are checking the app weekly or daily, and whether you use it to stay up on the news or to stay connected to your hobby, it’s important that it works. I hope that this helps people to see the extent of the challenge, and encourages developers to improve their apps, too.

How I did it

I wanted to investigate objectively how accurately each app displays text of posts and comments using the standard Lemmy markdown. Markdown is a standard part of the Lemmy platform, but not all apps handle it the same. It is basically what gives text useful formatting.

I used the latest release of each app, but did not include pre-releases. I only included apps that have released an update in the last 6 months, which should include most apps in active development. I was unable to test iOS-exclusive apps, so they are not included either. In all, 16 apps met the inclusion criteria.

I also added Eternity, which is in active development, although it has not had a recent update. I was able to include several iOS apps thanks to testing from @jordanlund@lemmy.world – Thanks, Jordan! This made for 20 apps that were tested.

Each app was rated in 5 categories: Text, Format, Spoilers, Links, and Images. I chose these mostly based on the wonderful Markdown Guide from @marvin@sffa.community, which was posted about a year ago in !meta@sffa.community (here).

I checked whether each app correctly displayed each category, then took the overall average. Each category was weighted equally. Text includes italic, bold, strong, strikethrough, superscript, and subscript. Format includes block quotes, lists, code (block and inline), tables, and dividers. Spoilers includes display of hidden, expandable spoilers. Links includes external links, username links, and community links. Images included embedded images, image references, and inline images.

Thanks to input from others, I also added a test to see if lemmy hyperlinks opened in-app. There was a problem with using the SFFA Community Guide that caused some apps to be essentially penalized twice because there was formatting inside formatting, so I created this TEST POST to more clearly and fairly measure each app.

In each case, I checked whether the display was correct based on the rules for Lemmy Markdown, and consistent with the author’s intent. In cases where the app recognized the tag correctly but did not display it accurately, that was treated as a fail.

Results

Out of a possible perfect 10, 6 apps displayed all markdown correctly:

Alexandrite - 10.0

Connect - 10.0

Jerboa (Official Android client) - 10.0

Photon - 10.0

Summit - 10.0

Voyager - 10.0

Quiblr - 9.5

Arctic - 9.3

Interstellar - 9.1

Lemmuy-UI - 9.0

Thunder - 8.9

Tesseract - 8.6

mlmym - 8.0

Racoon - 7.6

Boost - 7.3

Eternity - 7.0

Lemmios - 6.9

Sync - 6.9

Lemmynade - 6.1

Avelon - 5.7

More details of testing here

Disclaimers

Disclaimers

I Love Lemmy Apps (and their devs)

Lemmy apps devs work very hard, and invest a lot in the platform. Lemmy is better because they are doing the work that they do. Like, a LOT better. Everyone who uses the platform has to access it through one app or another. Apps are the face of the entire platform. Whether an app is a FOSS passion project, underwritten by a grant, or generating income through sales or ads, no one is getting rich by making their app. It is for the benefit of the community.

This is not meant to be a rating of the quality or functionality of any app. An app may have a high rating here but be missing other features that users want, or users may love an app that has a lower rating. This is just about how well apps handle markdown.

This is pretty unscientific

You’ll see my methodology above. I’m not a scientist. There is probably a much better way to do this, and I probably have biases in terms of how I went about it. I think it’s interesting and probably has some valuable information. If you think it’s interesting, let me know. If you think of a better way, PM me and I’d be happy to share what I have so you don’t have to start from scratch.

My only goal is to help the community

I do think that accurately displaying markdown should be a standard expectation of a finished app. I hope that devs use this as an opportunity to shore up the areas that are lagging, and that they have a set of standards to aim for.

I don’t have any Apple things

Sorry. This is just Android and Web review. If someone would like to see how iOS apps are doing, please reach out and I’ll share how we can work together to include them.

  • Xylight (Photon Dev)@lemm.eeEnglish
    19·
    1 month ago

    Hey, I’m the Photon dev. I’d like to know which parts Photon incorrectly displayed, so far I only see tables rendering incorrectly. I’ll have this fixed soon.

    Update: fixed table displays, pushed to main

    Could this be updated now? 🥺 (you can test here)

    • randint@lemmy.frozeninferno.xyzEnglish
      6·
      1 month ago

      Holy shit, Photon has gotten this good now? When I tried it a few months back it felt like just yet another Lemmy client. Now it feels so smooth and polished. Works great on mobile even. Thanks for making this!

    • XNX@slrpnk.netEnglish
      3·
      1 month ago

      Hey, the admin of slrpnk.net has been thinking of making Photon the default frontend but updates to it sometimes cause breaking issues? Any chance you could get into contact with them so it can become the default in a way that updates wont break it?

    • XNX@slrpnk.netEnglish
      2·
      1 month ago

      Photon is so great i honestly feel like it should replace the default

      • Blisterexe@lemmy.zipEnglish
        2·
        1 month ago

        Agreed, translating it to french made me discover so many little features, did you knoe it can show the political bias of a linked article?

    • NorthWestWind@lemmy.worldEnglish
      2·
      1 month ago

      Unrelated, but photon keeps randomly redirecting pages to what is previously viewed. It has screwed me over by making me post to the wrong community.

      • Xylight (Photon Dev)@lemm.eeEnglish
        1·
        1 month ago

        I don’t understand what this means. There are no redirect calls at all in photon other than for /comment urls, and certain layouts.

        • NorthWestWind@lemmy.worldEnglish
          1·
          1 month ago

          Not exactly redirect, but sometimes it just goes back to the previous page. Maybe something to do with window.history?

          It was a while ago. I haven’t used it after the incident (July 12)

  • nocturne@sopuli.xyzEnglish
    8·
    1 month ago

    Is there a list of what each app failed? It would be nice for the devs to be able to see. I use Mlem, and there is about to be a new release rebuilding it from the ground up. Hopefully it will rate higher once that happens.

      • nocturne@sopuli.xyzEnglish
        4·
        1 month ago

        Thanks. Interesting how the apps, even those that have lower scores, perform better than a web browser. Using Safari and Firefox (on a laptop) and both open your links in Lemmy.world instead of that thread on my instance. Neither recognize the user as anything other than text.

  • Jimbo@yiffit.netEnglish
    6·
    1 month ago

    One note on Jerboa, at least for me gifs don’t seem to play when embedded in comments. Otherwise 10/10 for me.

  • Aurelius@lemmy.worldEnglish
    4·
    1 month ago

    Left this comment in the other thread too, but posting here for visibility:

    Quiblr should now have each of the markdown criteria fixed. Huge thanks for the feedback and for all this analysis. Consistent markdown is important for a great and consistent user experience across the lemmy ecosystem

    • gedaliyah@lemmy.worldOPEnglish
      1·
      1 month ago

      Great! Thank you for the gorgeous app. I really love the style, and I think the personalized feed is brilliant.

      On my device, the lemmy hyperlink in the test post is still opening outside the app. I’m not sure how other web apps handle this but it would be the only additional change that would make it a perfect score.

      As an aside, I would love to see it as a PWA or standalone app. I don’t know if that’s on your roadmap but I think it would be neat.

      • Aurelius@lemmy.worldEnglish
        1·
        1 month ago

        Thanks for the Quiblr compliments! And Post and Comment links should now open in-app. I think that covers everything

        And Quiblr is a PWA. Native apps are in the works

  • treeofnik@discuss.onlineEnglish
    3·
    1 month ago

    This is awesome, thank you. I switched from memmy (iOS only) to voyager because it doesn’t display code blocks properly (usually doesn’t even show what’s in them) so reading certain posts or comments about computers or programming was a disaster.

    • nocturne@sopuli.xyzEnglish
      1·
      1 month ago

      Memmy was the first app I used, but it is abandoned now, sadly. But Mlem is actively being developed. I have not tried Voyager yet.

  • randint@lemmy.frozeninferno.xyzEnglish
    3·
    1 month ago

    I don’t understand why there isn’t a “markdown library” of some sort that software developers can just use in their app. I haven’t looked too deep into this, but it has always seemed to me that every app must individually implement markdown display. Why?

    • Echo Dot@feddit.ukEnglish
      7·
      1 month ago

      Because markdown has committed the worst of old programming sins. It has no standard.

      However I’m pretty sure that Lemmy has a standard so there’s not really much excuse.

    • micahmo@programming.devEnglish
      2·
      1 month ago

      As one of the Thunder devs, I can say there are markdown libraries. Thunder is written in Dart/Flutter and there is a great library that we use.

      https://pub.dev/packages/flutter_markdown

      That said, and as others have mentioned, markdown is not as well standardized and it seems like just about every site renders it differently, so there are a lot of edge cases to handle. Lemmy also has several unique implementations of things, such as spoilers, superscript/subscript, and the ability to tag users/communities without a hyperlink.

      In fact, one of the things Thunder failed on (table alignment) is a known bug in the markdown library we use. :-)

      https://github.com/flutter/flutter/issues/109487

    • Huschke@lemmy.worldEnglish
      1·
      1 month ago

      The problem isn’t that there are no libraries out there that parse Markdown. There are, in fact, plenty for all different languages. The issue is that every site has its own flavor of it. Lemmy does it one way, GitHub another, and something else does it completely differently yet again.

      It is, unfortunately, kind of a mess.

        • threelonmusketeers@sh.itjust.worksEnglish
          0·
          1 month ago

          What happens if you press tab or click on the suggested item at the point in your screenshot?

          For me, it inserts the link at the cursor position, but doesn’t replace the bit you’ve already typed, resulting in @gedal[@gedaliyah@lemmy.world](URL).

          Anyone else have this issue?

          • Nothing4You@programming.devEnglish
            2·
            1 month ago

            on firefox, if i type @gedal and click or press tab once it replaces the text with [@gedaliyah@lemmy.world](https://lemmy.world/u/gedaliyah) . the behavior is the same whether i hit tab, enter or click the text.

            • threelonmusketeers@sh.itjust.worksEnglish
              2·
              1 month ago

              if i type @gedal and click or press tab once it replaces the text with [@gedaliyah@lemmy.world](https://lemmy.world/u/gedaliyah)

              Ah, you are correct. It turns out that the issue I was encountering was a little more subtle.

              If I type all the way to @gedaliyah@ and click or press tab once it replaces only the second ‘@’, resulting in @gedaliyah[@gedaliyah@lemmy.world](https://lemmy.world/u/gedaliyah).

        • gedaliyah@lemmy.worldOPEnglish
          0·
          1 month ago

          Yes, I’m not sure if that is meant to be a placeholder or a substitute for native user links. What it actually does is generate markup that converts the username into a web link, which is fine for most circumstances, but not ideal. A plaintext username should automatically link to the user. This creates an inconsistent behavior between posts depending on where (and when) they were typed.

          In other words, it’s a very helpful feature, but it is not recognizing and linking usernames.

          • BentiGorlich@thebrainbin.org
            2·
            1 month ago

            Actually that behaviour is very annoying to other platforms. Mbin for example can only link to the lemmy server this user is on and no longer the local profile of that user. Example: @ user @ lemmy.instance gets converted to [@ user @ lemmy.instance](https:// lemmy.instance/u/user so on mbin this does not open the profile of the user on the local server, but instead links the lemmy instance, so you leave your instance to view the profile.

            (spaces included so this won’t get converted to mentions, etc)

              • BentiGorlich@thebrainbin.org
                2·
                1 month ago

                Yes they are, but you have my profile on your server and you do not need to leave the server to view my profile… @ user @ lemmy.instance should link to https:// mbin.instance/u/@user@lemmy.instance and not to https:// lemmy.instance/u/user

      • gedaliyah@lemmy.worldOPEnglish
        1·
        1 month ago

        That’s really interesting. AFAIK Lemmy devs do not have a comprehensive markdown documentation. I thought it was CommonMark plus spoilers and Lemmy links, but it seems like they have other changes as well.

        • JackbyDev@programming.devEnglish
          0·
          1 month ago

          I filed a bug with Jerboa a long time ago about something related to this (I don’t remember exactly). I guess right now the philosophy is that every front end/app can render how they see fit.

          • gedaliyah@lemmy.worldOPEnglish
            1·
            1 month ago

            Ultimately, this is just my opinion about what apps should prioritize in terms of markdown. I don’t think it’s too much of an ask that these be consistent across apps. I’m not sure that there has ever really been an effort by the devs to standardize things in this kind of way. As I said in the post itself. Lemmy is no longer a baby platform. people have been sharing these best practices for markdown for over a year at least.

            I think that when someone posts, they should have a reasonable expectation that people will see what they intend.

  • SomeGuy69@lemmy.worldEnglish
    2·
    1 month ago

    I’m on Connect and haven’t noticed anything displayed wrong. I must be lucky. Table work, spoiler work, embedded images work, emoji work.

    • brianorca@lemmy.worldEnglish
      0·
      1 month ago

      Spoilers in Connect are not readable when I click them. (White on white) Unless I first select the post so the background in grey.

      • SomeGuy69@lemmy.worldEnglish
        3·
        1 month ago

        I use OLED mode, then it’s a bug with the other theme colors, as it works for me without that workaround.

  • jordanlund@lemmy.worldEnglish
    2·
    1 month ago

    @gedaliyah@lemmy.world iOS testing, not sure how you score these so I just listed out the broken stuff.

    Arctic - Link opens in App. Headings fail, images fail, everything else looks fine.

    Avelon - Link opens in browser, not app. Manually went to test post. Bold+Italic fails (Italic works, not Bold). Table fails. Horizontal Rule fails. Spoiler fails. Everything else looks good.

    Bean - Last updated 7 months ago, comments on the app say it’s abandoned. Link opens in browser, not app. Manually went to test post. Text formatting block fails so hard, it’s not even visible(!) Heading fails. Code Block fails, Inline Code fails. Links and Image work, but not inline, only at the bottom of the post. Table fails. Horizontal rule fails.

    CheeseBot - Did not test. $2.99, no free version.

    Lemmios - Link opens in app. Everything looks and works great EXCEPT Spoilers.

    Mlem - Link opens in browser, not app. Manually went to test post. As with Lemmios, everything looks and works great EXCEPT spoilers.

    Remmel - Instant fail. No development in 2 years, unable to even add an instance or an account. Non-starter.

    Thunder - Hard to test. Lots of lag for some reason. Link opens in browser, not app. Manually went to test post. That being said, EVERYTHING worked. The lag may have been because I had just linked my account. Testing everything above, then coming back to Thunder, I found it fast and responsive.

    Voyager - Link opens in app. EVERYTHING worked. No notes.

    So, ranking them:

    Voyager - EVERYTHING worked. No notes.

    Thunder - Everything worked, but laggy to start with when using a year old account with lots of data. Once it caught up, everything was fine. Would probably be great with a new account.

    Lemmios - Link opens in app by default. Spoilers don’t work.

    Mlem - Link opens in browser by default but is user configurable. Spoilers don’t work.

    Arctic - A few minor failures.

    Avelon - A few more failures than Arctic.

    Bean - Hey, it works better than Remmel. Probably abandoned.

    Remmel - Instant fail.

    CheeseBot - Did not test. $2.99, no free version.