I recently made the tool smry.ai, which bypasses paywalls and instantly gets the summary. In the process, I learned a lot about what works and what doesn't when trying to get past paywalls.
Some general information you need is that there are two types of paywalls: hard paywalls and soft paywalls. Hard paywalls are usually not possible to bypass with traditional methods, as the content is not exposed to the client until you subscribe. In other words, the only way to get this content is if someone who has access individually submits it to something like archive.is.
Now, most sites have instead soft paywalls, which means that the content is accessible, but blocked to users either by popups or only exposed to certain user agents like Googlebot. In this case, here are the best methods for bypassing, that I learned by reading the source code for https://github.com/iamadamdev/bypass-paywalls-chrome (a great tool in its own right, that does everything below).
- Googlebot User Agent: Many sites allow unrestricted access to Googlebot to ensure their SEO ranking. You can emulate Googlebot by changing the User-Agent of the browser to
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)on desktop - Clear cache: This works for an alarming number of sites.
- Bingbot User Agent: Similar to the Googlebot method, some sites allow unrestricted access to Bingbot for SEO purposes. The script can also emulate Bingbot for certain sites.
- Remove Cookies: Some sites use cookies to track how many articles you've read in a month and limit access after a certain number. For many sites, you can read the content if you clear your browser cache/remove cookies. This is probably the easiest method to implement without external tools. Incognito also works for many of these sites.
- Referer Override: For some sites, you want to emulate your referer to 'https://www.google.com/' or 'https://www.facebook.com/' or 'https://t.co/x?amp=1' depending on the site. This can bypass paywalls that allow users coming from search engines or social media unrestricted access.
Now, above are the methods typically used by extensions, or if you want to scrape a paywalled site by using a virtual browser.
However, for most of us, this is far too much work. For one, clearing your cookies can be annoying (instantly logs you out of things) although fantastic for digital hygiene. Also, setting your user agent to Googlebot for all sites is also not a great solution, as it isn't trivial to do and can also mess up some pages, so it's definitely a good idea to use extensions. They are very powerful, and Bypass Paywalls Chrome actually does some more cool stuff I didn't get into.
The most robust solutions are the caches and web archives. They scrape the whole internet, and then archive websites. Here are the best ones, and they are heavily used by the tools below as they can scrape sites most other providers can't without help:
- Archive.is: By far the slowest, but the most robust. If you have been scratching your head for 20 minutes and no other tool works, give this a try. (cool trick is archive.is/latest/<url>) as a shortcut for the latest archive.
- Internet Web Archive (archive.org): This tool is excellent, and is a bit less robust than archive.is, but a bit faster. Best for everyday use. Shortcut is https://web.archive.org/web/2/<url>
- Google Cache: Unreliable. High rate limits. Difficult to scrape. Blazingly fast. You get similar results to just using Googlebot, but in my experience is far more consistent. That said, there are capchas and it works for fewer sites than those above. Shorcut is https://webcache.googleusercontent.com/search?q=cache:<url>
Still, most of us just want to be able to go to a site and be able to read it easily. For that, here is an intro to my favorite bypass sites, how I believe they work, and some background on them.
- 12ft.io. This is currently the most commonly used tool, with tens of millions of visitors per month. It claims that it only fetches without javascript (it uses a proxy so it fetches for you, the request isn't made from your browser), but I'm pretty sure it uses Googlebot, and maybe some other methods as well, although not directly stated. Got banned from its hosting provider recently, but is back up.
- removepaywall.com. This site does many things: it first tries to fetch from Wayback Machine (archive.org) and then with Google cache. Then it tries a direct fetch with Googlebot user agent. It claims it also tries archive.is, but redirects users to archive.is when it fails. In general, this might be the most robust solution I've seen.
- smry.ai. Shameless self-plug (mods were made aware). Does everything removepaywall.com does, is completely open-source, and also generates free summaries of each article until I run out of money. Also, tells you where the content was fetched from and lets you try different options.
- 1ft.io. This one is new and has blown up quickly because it is fast. From what I can guess, it just uses Googlebot. which is why it is so fast (fetching from Wayback Machine or Google cache would be slower). But it also fails a lot. Good quick solution to try before moving on to other more robust methods.
- darkread.com. Read in dark mode. Nuff said.
- https://leiaisso.net. Very popular in Brazil. Pretty buggy for me.
Really curious what other tools/techniques you guys use, and what you think of the tools above.
*Any doesn't include hard paywalls
Edit: I made this post a couple of months ago, and I continue getting comments asking if 'x' is a hard paywall. Here are some tools to figure out if something is under a hard paywall (and therefore is not bypassable without a subscription)
- Does this tool need to show its content to search engines?
If a tool does not need to show content to search engines, it very well be using a hard paywall. This goes for tools like Patreon, Onlyfans, and other subscription services that only cater to subscribed customers. - Is this a downloadable file?
If you need to sign in to download a file, it probably is under a hard paywall. That doesn't necessarily mean that it is secure though, but you likely won't be able to bypass it with one of the tools above. - Is there a visible obstruction of the content?
If some content is visible, and the rest of the article is not accessible or obstructed in some way, it is often a soft paywall. However, if no content at all is visible, it's more likely to be a hard paywall. - Do the tools above work?
If the tools above do not work, that's a strong sign that it's a hard paywall.
Note, don't read the following if you are a hardcore pirate: Also, I want to point out that if paying is an option for you, you should do so. There are several reasons for this, one being it is good to support the creator of the content, but more importantly (in the context of this sub) that bypassing hard paywalls often takes a lot of time and effort, and if you value your time, it can often be cheaper just to pay. Take something like Chegg. You can definitely join some shady Discord server and pay a fraction of the cost to access a document, but this will slow you down, possibly scam you, and you won't have a good time.
