fead: Add option to skip errors v2 NEEDS REVISION

Ngô Ngọc Đức Huy: 1
 Add option to skip errors

 1 files changed, 18 insertions(+), 3 deletions(-)

[PATCH fead v2] Add option to skip errors Export this patch

Previous one would generate an error because I forgot to return

---
 src/fead.py | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/src/fead.py b/src/fead.py
index 370b623..15bca4c 100755
--- a/src/fead.py
+++ b/src/fead.py
@@ -185,9 +185,19 @@ async def fetch(raw_url):
                         response.getheaders(), response)
 
 
-async def fetch_all(urls):
+async def fetch_skip_error(url):
+    try:
+        return await fetch(url)
+    except Exception as e:
+        warn(f'fail to fetch {url}: {e}', type('ErrorWarning', (Warning,), {}))

Nguyễn Gia Phong <cnx@loang.net>

1 year, 4 months ago

Please return None explicitly.  Would be nice if thou canst come up
with something less ironic than ErrorWarning but I can rename it
before application.

Next

+
+
+async def fetch_all(urls, skip_error):
     """Fetch all given URLs asynchronously and return them parsed."""
-    tasks = gather(*map(fetch, urls))
+    if skip_error:
+        tasks = gather(*map(fetch_skip_error, urls))
+    else:
+        tasks = gather(*map(fetch, urls))
     try:
         return await tasks

Nguyễn Gia Phong <cnx@loang.net>

1 year, 4 months ago

Please filter the None results here.

     except:
@@ -234,11 +244,16 @@ def main():
     parser.add_argument('-o', '--output', metavar='PATH',
                         type=FileType('w'), default=stdout,
                         help='output file (default to stdout)')
+    parser.add_argument('-s', '--skip-error', action='store_true',
+                        default=False,
+                        help="errors not causing failure but logged")
     args = parser.parse_args()
 
     template = args.template.read()
     args.template.close()
-    for ad in select(args.count, (ad for feed in run(fetch_all(args.feeds))
+    for ad in select(args.count, (ad for feed in run(fetch_all(args.feeds,
+                                                               args.skip_error))
+                                  if feed is not None
                                   for ad in select(args.per_feed, feed))):
         args.output.write(template.format(**truncate(ad, args.len)._asdict()))
     args.output.close()
-- 
2.40.1