I was just looking at obsidian web-clipper's source code because I've been quite impressed at its markdown conversion results and came across Defuddle in there. I'll be using for my bespoke read-it-later/ knowledge-base app, so thank you in advance :D
Been using the obsidian clipper since it was out and this is a really neat. The per website profile based extraction is awesome.
Even if you are not a obsidian user, the markdown extraction quality is the most reliable Ive seen.
Obsidian Web Clipper is a great tool to turn chatGPT conversations in markdown, or to just print it (believe me, it is a user case)
Really nice work. I appreciate the example with JSDOM as that’s exactly how I use readability, and this looks like a nice drop-in replacement.
Question: How did you validate this? You say it works better than readability but I don’t see any tests or datasets in the repo to evaluate accuracy or coverage. Would it be possible to share that as well?
I've been super happy with Obsidian Web Clipper! It's worked really well for me with the one exception of importing publish dates (which is more than forgivable !)
Is Mozilla's Readability really abandoned? The latest release (v0.6.0) is just 2 months ago, and its maintainer (Gijs) is pretty active on responding issues.
The Python analogues seem to be well maintained. I did my own implementation of the Readability algorithm years ago and dropped it in favor them, and I have a few scrapers going strong with regular updates.
I built a similar project called Substance [^0]. Unlike most readability tools that try to solve the problem once and for all, it takes a different approach. It provides a framework to define how each website should be handled, ensuring better results for each website covered.
Serious question - who and why would be using this tool? What is the use case? In other comments I have only seen exporting ChatGPT conversations to md
Does anyone know why readers don’t work for some websites where it looks like they should - ie normal article with lots of text.
You just get a completely white page (on the iPhone reader). Usually it’s a news website.
Is this the website intentionally obscuring the content to ensure they can serve their ads? If so how do they go about it?
This is very cool! Given how messy and busy many websites have become, we really need a robust markdown converter that lets readers focus on reading the content. Nice to see something stepping up where Readability left off.
Thank you for picking up this work :-)
In the playground, after I enter a url, I can't seem to figure out how to submit it to fetch the url? I tried pressing the return key on iOS keyboard but it didn't do anything. Am I missing something?
I’ll give it a try. I’m not happy with my current setup for markdown to HTML on the wysiwyg editor I’m using, this may provide better results if I go with my own tool bar and editor.
I have build something similar:https://devkram.de/markydown but with php. Easy for self hosting
can confirm that readability seems to be on life support. I used it slurp, an obsidian plugin which serves the same basic purpose as web clipper, and always had a hard time getting PRs reviewed and merged.
i started working on my own alternative but life (and web clipper) derailed the work.
it's funny. somehow slurp keeps gaining new users even though web clipper exists. so i might have to refactor it to use your library sometime soon even though I don't use slurp myself anymore.
Are you using ai models behind the scenes? I saw Gemini and others in the code. I am asking mainly to understand the cost of using yours vs. readability. Thank!
Cool. Im looking for something similar but for stripping signatures and boilerplate disclaimers from html email. Could this work for that?
Looks good, I'm gonna try to swap readability in my RSS reader with this.
And with Pocket going away I might have to add save it later to it...
Neat. With ~3 more lines of code, you could get a URL and render it in simpler HTML and be a full fledged replacement.
Since it's written in javascript is there any chance it could be packaged as a bookmarklet?
seems pretty much perfect including obsidian clipper. Thanks!
Interesting that Markdown does not support form element.
Excellent work. A modern alternative to readability was much needed. This is especially useful for building clean web context for LLMs. Thanks for open-sourcing this!
Tried it on some webpages, doesnt work well.
A bit off-topic, but I'm very excited to see the launch of Bases! I've obsessively followed the roadmap for like a year awaiting this day and have been frequently disappointed to still see it stuck somewhere under "planned".
Not that I didn't already implement a read-it-later solution with Obsidian+Dataview, but this definitely makes things simpler!
For those not in the know: [Readability](https://github.com/mozilla/readability)
Interesting as I was researching this recently and certainly not impressed with the quality of the Readability implementations in various languages. Although Readability.js was clearly the best, it being Javascript didn't suit my project.
In the end I found the python trifatura library to extract the best quality content with accurate meta data.
You might want to compare your implementation to trifatura to see if there is room for improvement.