Since the whole point of this package is to avoid paywalls, it only 
makes sense to set the content slot of your elfeed entries to the new 
HTML that you've fetched. However, you'll soon run into this issue (if 
you haven't already): https://github.com/skeeto/elfeed/issues/376.

To summarize: elfeed resets the entry's slots with the information 
provided by the feed whenever it fetches the feeds. To get around this, 
you can put your custom content in a metadata slot, since metadata slots 
are not reset. The problem now is that elfeed regularly deletes any 
elfeed-refs that aren't in the content slot of an entry. I posted a 
possible solution in the issue I linked, and just made a pull request to 
implement it: https://github.com/skeeto/elfeed/pull/457.

You also have a function to replace youtube links with Invidious links, 
which will (I think) encounter the same problem of being reset when 
elfeed fetches that feed again. In this case, the fix is to move the 
replacement out of elfeed-paywall entirely. The URL replacement would 
fit better either as a customization of the `browse-url' package, or 
even as a browser extension. I personally use 
https://github.com/SimonBrazell/privacy-redirect, but there are plenty 
of others. If you want to keep the customization in Emacs, you can try 
the alist `browse-url-browser-function' in Emacs 27 and below, and 
`browse-url-handlers' in Emacs 28. You can have a regex to match youtube 
URLs, and then in the associated function, call `browse-url' on the 
replaced URL, with another alist entry to match everything and call 
`browse-url-default-browser', or whatever your default function is.

Both of these solutions also address the issue of specific youtube 
frontends being slow, unreliable, or taken down. In this case, you only 
have to change the frontend URL in one place, as opposed to changing 
each entry link in elfeed to point to the new website. I experienced 
this problem firsthand when the original Invidious site shut down and my 
firefox bookmarks still pointed there.

Thanks, Zabe (they/them)

Hi Zabe,

Thanks for this detailed write-up - very helpful!

> To summarize: elfeed resets the entry's slots with the information
> provided by the feed whenever it fetches the feeds. To get around
> this, you can put your custom content in a metadata slot, since
> metadata slots are not reset. The problem now is that elfeed regularly
> deletes any elfeed-refs that aren't in the content slot of an entry. I
> posted a possible solution in the issue I linked, and just made a pull
> request to implement it: <https://github.com/skeeto/elfeed/pull/457>.

Yes indeed - I hadn't come across this bug yet, so thank you for
making me aware of it! I'll take a look at the PR you submitted as
well for my own learning.

> If you want to keep the customization in Emacs

Yes! Where at all possible. My plan for elfeed-paywall is to
eventually try to port the readability code to emacs-lisp. The elisp
readability code would become a dependency of this package, making the
`rdrview' dependency optional.

> The URL replacement would fit better either as a customization of
> the `browse-url' package, or even as a browser extension.

Hmm, that's a really interesting idea! My only thought was that
presumably modifying how `browse-url' works could lead to knock-on
side-effects elsewhere in a user's configuration - is this true? Could
such side-effects be mitigated if so?

> I experienced this problem firsthand when the original Invidious site
> shut down and my firefox bookmarks still pointed there.

Yes, I have too: for me it was because I used the original invidious'
RSS feeds in Elfeed. It was such a pain to revert back to YouTube
again, and I wanted to avoid doing that again in the future: this was
part of the motivation for this tool!

Best,
JH

Zabe <zabe@disroot.org> writes:

> Since the whole point of this package is to avoid paywalls, it only
> makes sense to set the content slot of your elfeed entries to the new
> HTML that you've fetched. However, you'll soon run into this issue (if
> you haven't already): <https://github.com/skeeto/elfeed/issues/376>.
>
> To summarize: elfeed resets the entry's slots with the information
> provided by the feed whenever it fetches the feeds. To get around
> this, you can put your custom content in a metadata slot, since
> metadata slots are not reset. The problem now is that elfeed regularly
> deletes any elfeed-refs that aren't in the content slot of an entry. I
> posted a possible solution in the issue I linked, and just made a pull
> request to implement it: <https://github.com/skeeto/elfeed/pull/457>.
>
> You also have a function to replace youtube links with Invidious
> links, which will (I think) encounter the same problem of being reset
> when elfeed fetches that feed again. In this case, the fix is to move
> the replacement out of elfeed-paywall entirely. The URL replacement
> would fit better either as a customization of the `browse-url'
> package, or even as a browser extension. I personally use
> <https://github.com/SimonBrazell/privacy-redirect>, but there are plenty
> of others. If you want to keep the customization in Emacs, you can try
> the alist `browse-url-browser-function' in Emacs 27 and below, and
> `browse-url-handlers' in Emacs 28. You can have a regex to match
> youtube URLs, and then in the associated function, call `browse-url'
> on the replaced URL, with another alist entry to match everything and
> call `browse-url-default-browser', or whatever your default function
> is.
>
> Both of these solutions also address the issue of specific youtube
> frontends being slow, unreliable, or taken down. In this case, you
> only have to change the frontend URL in one place, as opposed to
> changing each entry link in elfeed to point to the new website. I
> experienced this problem firsthand when the original Invidious site
> shut down and my firefox bookmarks still pointed there.
>
> Thanks, Zabe (they/them)

BTW I created a ticket for https://github.com/skeeto/elfeed/pull/457 to
preempt any reports from users on the issue.

https://todo.sr.ht/~johnhamelink/elfeed-paywall/1

Best,
JH

> Yes! Where at all possible. My plan for elfeed-paywall is to
> eventually try to port the readability code to emacs-lisp. The elisp
> readability code would become a dependency of this package, making the
> `rdrview' dependency optional.

I also tackled this problem in my own package. I ended up adapting the 
`eww-readable' function 
(https://github.com/zabe40/elfeed-time/blob/4704265c7c0ca69c602af4a1d075739e7dae2895/elfeed-time.el#L248), 
but I think it would really benefit if the readability algorithm could 
be customized by the user for different websites. A dedicated 
readability library would be a great idea!

>> The URL replacement would fit better either as a customization of
>> the `browse-url' package, or even as a browser extension.
> 
> Hmm, that's a really interesting idea! My only thought was that
> presumably modifying how `browse-url' works could lead to knock-on
> side-effects elsewhere in a user's configuration - is this true? Could
> such side-effects be mitigated if so?

Yep, you're right that unconditionally setting those alists I mentioned 
could mess up a user's custom configuration. One way to get around this 
is to point the user to the proper variables so they can set it up and 
integrate it with any customizations themself. This would redirect 
youtube URLs opened from anywhere in Emacs to the desired Invidious 
instance, and might look something like this:

(defun redirect-url-to-invidious (url &rest args)
   (apply #'browse-url
	 (replace-regexp-in-string "youtube\\.com"
				   "invidious.whatever"
				   url t t)
	 args))

(setf browse-url-browser-function
       '(("youtube\\.com" . redirect-url-to-invidious)
	("." . browse-url-default-browser)))

Another option is to have a function that wraps 
`elfeed-search-browse-url' and/or `elfeed-show-visit'. It would 
temporarily set the proper variable, and since we're adding the front of 
the list using a `let', the variable gets reset to whatever it was 
before after we're done with it. It also has the benefit of still going 
through the user's custom browse-url function if there is one. This 
approach wouldn't affect youtube URLs opened elsewhere in Emacs (for 
better or worse):

(defun elfeed-show-visit-replace (&optional use-generic-p)
   (interactive "P")
   (let ((browse-url-browser-function
	 (cons '("youtube\\.com" . redirect-url-to-invidious)
	       (if (listp browse-url-browser-function)
		   browse-url-browser-function
		 (list (cons (rx anychar)
			     browse-url-browser-function))))))
     (elfeed-show-visit use-generic-p)))

The only drawback I can see with both of these approaches is that the 
elfeed-browse/visit functions can call `browse-url-generic' with a 
prefix argument, but that function doesn't check the alist variable at 
all. I'm not sure how big of a problem this is in practice, since it 
looks like all the major browsers are covered by browse-url already, but 
its still worth keeping in mind.

I think both of these approaches have merit, and which one you pick just 
depends on whether or not you want to redirect ALL youtube URLs opened 
in Emacs, or just those opened in elfeed. One idea is to provide the 
`redirect-url-to-invidious' and `elfeed-show-visit-replace' functions 
with the package, and present the user the option to either set their 
browse-url-browser-function globally, or give 
`elfeed-show-visit-replace' a binding of their choice.

Thanks, Zabe