~reedwade/wee-matter-devel

wee-matter: Hydrate rooms when first switching to buffer, to avoid rate limiting v1 APPLIED

This is my attempt to fix
https://todo.sr.ht/~reedwade/wee-matter/20,
which is a critical bug for me.

Midgard (2):
  Correct some typos
  Hydrate rooms when first switching to buffer

 wee_matter/file.py      |  2 +-
 wee_matter/http.py      |  6 ++---
 wee_matter/post.py      |  2 +-
 wee_matter/room.py      | 35 ++++++++++++++++----------
 wee_matter/server.py    | 16 ++++++------
 wee_matter/websocket.py | 56 +++++++++++++++++++++++++++++++----------
 6 files changed, 78 insertions(+), 39 deletions(-)

-- 
2.26.2
Eddie Barraco
It seems pretty cool ! But what happen if we receive a highlight in a
channel that isnt hidrated ?
I'll probably merge the typo fix after a little checkup. Thanks a lot
for this !


Quoting Eddie Barraco (2020-11-16 09:39:26)
Next
Eddie Barraco
I took time to think about this issue. Here what I got in mind:

Managing hidrated states is a pain as we have to add checks everywhere.
What if I reveice a reaction, what if I get highlither, blablabla.

I think we must avoid this and have all buffer working and the app in a
"sane" state. We canno't tells the Mattermost server "do not notify me
on those channels", the websocket will give us all events.

So what could we do ?

I think we should delay the hidratation process as this is the only
real moment we could have rate limit issue. Instead of sending all
request and let the flow full open, we should send calls one by one in a
"async" loop, room by room.
This should not be so difficult and I think I got some ideas to do that.

I'll take time to work on this soon.
Quoting Eddie Barraco (2020-11-16 10:27:52)
Next
Quoting Eddie Barraco (2020-11-16 10:27:52)
> I think we should delay the hidratation process as this is the only
> real moment we could have rate limit issue.
I saw in the docs that Mattermost should add X-Ratelimit headers:
https://api.mattermost.com/#tag/rate-limiting

To get the most out of the quotum you could take this into account.
Export patchset (mbox)
How do I use this?

Copy & paste the following snippet into your terminal to import this patchset into git:

curl -s https://lists.sr.ht/~reedwade/wee-matter-devel/patches/15024/mbox | git am -3
Learn more about email & git
View this thread in the archives

[PATCH wee-matter 1/2] Correct some typos Export this patch

From: Midgard <2885-Midgard@users.noreply.framagit.org>

---
 wee_matter/file.py      |  2 +-
 wee_matter/http.py      |  6 +++---
 wee_matter/room.py      | 24 ++++++++++++------------
 wee_matter/server.py    | 16 ++++++++--------
 wee_matter/websocket.py | 18 +++++++++---------
 5 files changed, 33 insertions(+), 33 deletions(-)

diff --git a/wee_matter/file.py b/wee_matter/file.py
index 8c47fb7..511d903 100644
--- a/wee_matter/file.py
+++ b/wee_matter/file.py
@@ -45,7 +45,7 @@ def open_file(file_path):

def file_get_cb(file_path, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when downloading file")
        weechat.prnt("", "An error occurred while downloading file")
        return weechat.WEECHAT_RC_ERROR

    open_file(file_path)
diff --git a/wee_matter/http.py b/wee_matter/http.py
index 1b5b9c8..1d65e11 100644
--- a/wee_matter/http.py
+++ b/wee_matter/http.py
@@ -5,8 +5,8 @@ import json
import time
import re

from wee_matter.room import (hidrate_room_read_posts_cb, hidrate_room_posts_cb,
                             hidrate_room_users_cb, hidrate_room_user_cb)
from wee_matter.room import (hydrate_room_read_posts_cb, hydrate_room_posts_cb,
                             hydrate_room_users_cb, hydrate_room_user_cb)

from wee_matter.post import post_post_cb

@@ -24,7 +24,7 @@ def build_file_url(file_id, server):

def singularity_cb(data, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when performing a request")
        weechat.prnt("", "An error occurred while performing a request")
        return weechat.WEECHAT_RC_ERROR

    return weechat.WEECHAT_RC_OK
diff --git a/wee_matter/room.py b/wee_matter/room.py
index b36eb72..777d37a 100644
--- a/wee_matter/room.py
+++ b/wee_matter/room.py
@@ -61,9 +61,9 @@ def room_input_cb(data, buffer, input_data):

    return weechat.WEECHAT_RC_OK

def hidrate_room_posts_cb(buffer, command, rc, out, err):
def hydrate_room_posts_cb(buffer, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when hidrating room")
        weechat.prnt("", "An error occurred while hydrating room")
        return weechat.WEECHAT_RC_ERROR

    server = wee_matter.server.get_server_from_buffer(buffer)
@@ -76,13 +76,13 @@ def hidrate_room_posts_cb(buffer, command, rc, out, err):
        wee_matter.post.write_post(builded_post)

    if "" != response["next_post_id"]:
        wee_matter.http.run_get_channel_posts_after(builded_post.id, builded_post.channel_id, server, "hidrate_room_posts_cb", buffer)
        wee_matter.http.run_get_channel_posts_after(builded_post.id, builded_post.channel_id, server, "hydrate_room_posts_cb", buffer)

    return weechat.WEECHAT_RC_OK

def hidrate_room_read_posts_cb(buffer, command, rc, out, err):
def hydrate_room_read_posts_cb(buffer, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when hidrating room")
        weechat.prnt("", "An error occurred while hydrating room")
        return weechat.WEECHAT_RC_ERROR

    server = wee_matter.server.get_server_from_buffer(buffer)
@@ -102,7 +102,7 @@ def hidrate_room_read_posts_cb(buffer, command, rc, out, err):
    weechat.buffer_set(buffer, "hotlist", "-1")

    if "" != response["next_post_id"]:
        wee_matter.http.run_get_channel_posts_after(post.id, post.channel_id, server, "hidrate_room_posts_cb", buffer)
        wee_matter.http.run_get_channel_posts_after(post.id, post.channel_id, server, "hydrate_room_posts_cb", buffer)

    return weechat.WEECHAT_RC_OK

@@ -118,9 +118,9 @@ def create_room_user_from_user_data(user_data, buffer, server):

    weechat.nicklist_add_nick(buffer, "", user.username, user.color, "@", user.color, 1)

def hidrate_room_users_cb(buffer, command, rc, out, err):
def hydrate_room_users_cb(buffer, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when hidrating room users")
        weechat.prnt("", "An error occurred while hydrating room users")
        return weechat.WEECHAT_RC_ERROR

    response = json.loads(out)
@@ -132,9 +132,9 @@ def hidrate_room_users_cb(buffer, command, rc, out, err):

    return weechat.WEECHAT_RC_OK

def hidrate_room_user_cb(buffer, command, rc, out, err):
def hydrate_room_user_cb(buffer, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when hidrating room user")
        weechat.prnt("", "An error occurred while hydrating room user")
        return weechat.WEECHAT_RC_ERROR

    user_data = json.loads(out)
@@ -205,8 +205,8 @@ def create_room_from_channel_data(channel_data, server):

    weechat.buffer_set(buffer, "number", str(number))

    wee_matter.http.run_get_read_channel_posts(server.user.id, channel_data["id"], server, "hidrate_room_read_posts_cb", buffer)
    wee_matter.http.run_get_channel_members(channel_data["id"], server, "hidrate_room_users_cb", buffer)
    wee_matter.http.run_get_read_channel_posts(server.user.id, channel_data["id"], server, "hydrate_room_read_posts_cb", buffer)
    wee_matter.http.run_get_channel_members(channel_data["id"], server, "hydrate_room_users_cb", buffer)

def buffer_switch_cb(data, signal, buffer):
    if buffer not in channel_buffers.values():
diff --git a/wee_matter/server.py b/wee_matter/server.py
index ce29e01..bcdfc91 100644
--- a/wee_matter/server.py
+++ b/wee_matter/server.py
@@ -175,7 +175,7 @@ def unload_server(server_name):

def connect_server_team_channel_cb(server_name, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when connecting team channel")
        weechat.prnt("", "An error occurred while connecting team channel")
        return weechat.WEECHAT_RC_ERROR

    server = get_server(server_name)
@@ -187,7 +187,7 @@ def connect_server_team_channel_cb(server_name, command, rc, out, err):

def connect_server_team_channels_cb(server_name, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when connecting team channels")
        weechat.prnt("", "An error occurred while connecting team channels")
        return weechat.WEECHAT_RC_ERROR

    server = get_server(server_name)
@@ -200,7 +200,7 @@ def connect_server_team_channels_cb(server_name, command, rc, out, err):

def connect_server_users_cb(data, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when connecting users")
        weechat.prnt("", "An error occurred while connecting users")
        return weechat.WEECHAT_RC_ERROR

    server_name, page = data.split("|")
@@ -223,7 +223,7 @@ def connect_server_users_cb(data, command, rc, out, err):

def connect_server_teams_cb(server_name, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when connecting teams")
        weechat.prnt("", "An error occurred while connecting teams")
        return weechat.WEECHAT_RC_ERROR

    server = get_server(server_name)
@@ -240,7 +240,7 @@ def connect_server_teams_cb(server_name, command, rc, out, err):

def connect_server_team_cb(server_name, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when connecting team")
        weechat.prnt("", "An error occurred while connecting team")
        return weechat.WEECHAT_RC_ERROR

    server = get_server(server_name)
@@ -254,7 +254,7 @@ def connect_server_team_cb(server_name, command, rc, out, err):

def new_user_cb(server_name, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when adding a new user")
        weechat.prnt("", "An error occurred while adding a new user")
        return weechat.WEECHAT_RC_ERROR

    server = get_server(server_name)
@@ -266,7 +266,7 @@ def new_user_cb(server_name, command, rc, out, err):

def connect_server_cb(server_name, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when connecting")
        weechat.prnt("", "An error occurred while connecting")
        return weechat.WEECHAT_RC_ERROR

    token_search = re.search('[tT]oken: (\w*)', out)
@@ -341,7 +341,7 @@ def connect_server(server_name):

def disconnect_server_cb(server_name, command, rc, out, err):
    if rc != 0:
        weechat.prnt("", "An error occured when disconnecting")
        weechat.prnt("", "An error occurred while disconnecting")
        return weechat.WEECHAT_RC_ERROR

    unload_server(server_name)
diff --git a/wee_matter/websocket.py b/wee_matter/websocket.py
index 11bf9b8..9cd5b84 100644
--- a/wee_matter/websocket.py
+++ b/wee_matter/websocket.py
@@ -60,12 +60,12 @@ def create_worker(server):
        last_pong_time= 0,
    )

def rehidrate_server_buffers(server):
def rehydrate_server_buffers(server):
    weechat.prnt("", "Syncing...")
    for buffer in server.buffers:
        last_post_id = weechat.buffer_get_string(buffer, "localvar_last_post_id")
        channel_id = weechat.buffer_get_string(buffer, "localvar_channel_id")
        wee_matter.http.run_get_channel_posts_after(last_post_id, channel_id, server, "hidrate_room_posts_cb", buffer)
        wee_matter.http.run_get_channel_posts_after(last_post_id, channel_id, server, "hydrate_room_posts_cb", buffer)

def reconnection_loop_cb(server_name, remaining_calls):
    server = wee_matter.server.get_server(server_name)
@@ -77,11 +77,11 @@ def reconnection_loop_cb(server_name, remaining_calls):
    new_worker = create_worker(server)
    if new_worker:
        wee_matter.server.update_server_worker(server, new_worker)
        weechat.prnt("", "Connected back.")
        rehidrate_server_buffers(server)
        weechat.prnt("", "Reconnected.")
        rehydrate_server_buffers(server)
        return weechat.WEECHAT_RC_OK

    weechat.prnt("", "Reconnection issue. Trying again in some seconds...")
    weechat.prnt("", "Reconnection issue. Trying again in a few seconds...")
    return weechat.WEECHAT_RC_ERROR

def close_worker(worker):
@@ -89,8 +89,8 @@ def close_worker(worker):
    weechat.unhook(worker.hook_ping)
    worker.ws.close()

def handle_loosed_connection(server):
    weechat.prnt("", "Loosed connection.")
def handle_lost_connection(server):
    weechat.prnt("", "Connection lost.")
    close_worker(server.worker)
    wee_matter.server.update_server_worker(server, None)

@@ -99,7 +99,7 @@ def ws_ping_cb(server_name, remaining_calls):
    worker = server.worker

    if worker.last_pong_time < worker.last_ping_time:
        handle_loosed_connection(server)
        handle_lost_connection(server)
        return weechat.WEECHAT_RC_OK

    try:
@@ -107,7 +107,7 @@ def ws_ping_cb(server_name, remaining_calls):
        worker = worker._replace(last_ping_time=time.time())
        wee_matter.server.update_server_worker(server, worker)
    except (WebSocketConnectionClosedException, socket.error) as e:
        handle_loosed_connection(server)
        handle_lost_connection(server)

    return weechat.WEECHAT_RC_OK

-- 
2.26.2

[PATCH wee-matter 2/2] Hydrate rooms when first switching to buffer Export this patch

From: Midgard <2885-Midgard@users.noreply.framagit.org>

This prevents rate limiting problems when hydrating all rooms at once
when connecting to the server.

This commit introduces a localvar "hydrated" on each buffer and some
defensive code in websocket.py, all to avoid exceptions when things
happen in channels that are not hydrated yet.
---
 wee_matter/post.py      |  2 +-
 wee_matter/room.py      | 15 ++++++++++++---
 wee_matter/websocket.py | 38 ++++++++++++++++++++++++++++++++++----
 3 files changed, 47 insertions(+), 8 deletions(-)

diff --git a/wee_matter/post.py b/wee_matter/post.py
index e798f12..e02681e 100644
--- a/wee_matter/post.py
+++ b/wee_matter/post.py
@@ -32,7 +32,7 @@ Reaction = NamedTuple(
post_buffers = {}

def get_buffer_from_post_id(post_id):
    return post_buffers[post_id]
    return post_buffers.get(post_id)

def post_post_cb(buffer, command, rc, out, err):
    if rc != 0:
diff --git a/wee_matter/room.py b/wee_matter/room.py
index 777d37a..c42d5fe 100644
--- a/wee_matter/room.py
+++ b/wee_matter/room.py
@@ -81,6 +81,9 @@ def hydrate_room_posts_cb(buffer, command, rc, out, err):
    return weechat.WEECHAT_RC_OK

def hydrate_room_read_posts_cb(buffer, command, rc, out, err):
    if weechat.buffer_get_string(buffer, "localvar_hydrated"):
        return weechat.WEECHAT_RC_OK

    if rc != 0:
        weechat.prnt("", "An error occurred while hydrating room")
        return weechat.WEECHAT_RC_ERROR
@@ -89,6 +92,8 @@ def hydrate_room_read_posts_cb(buffer, command, rc, out, err):

    response = json.loads(out)

    weechat.buffer_set(buffer, "localvar_set_hydrated", "1")

    if not response["order"]:
        return weechat.WEECHAT_RC_OK

@@ -205,13 +210,17 @@ def create_room_from_channel_data(channel_data, server):

    weechat.buffer_set(buffer, "number", str(number))

    wee_matter.http.run_get_read_channel_posts(server.user.id, channel_data["id"], server, "hydrate_room_read_posts_cb", buffer)
    wee_matter.http.run_get_channel_members(channel_data["id"], server, "hydrate_room_users_cb", buffer)

def buffer_switch_cb(data, signal, buffer):
    if buffer not in channel_buffers.values():
        return weechat.WEECHAT_RC_OK

    if not weechat.buffer_get_string(buffer, "localvar_hydrated"):
        server = wee_matter.server.get_server_from_buffer(buffer)
        channel_id = weechat.buffer_get_string(buffer, "localvar_channel_id")

        wee_matter.http.run_get_read_channel_posts(server.user.id, channel_id, server, "hydrate_room_read_posts_cb", buffer)
        wee_matter.http.run_get_channel_members(channel_id, server, "hydrate_room_users_cb", buffer)

    mark_channel_as_read(buffer)

    return weechat.WEECHAT_RC_OK
diff --git a/wee_matter/websocket.py b/wee_matter/websocket.py
index 9cd5b84..5f93533 100644
--- a/wee_matter/websocket.py
+++ b/wee_matter/websocket.py
@@ -118,11 +118,13 @@ def handle_posted_message(server, message):
    if data["team_id"] and data["team_id"] not in server.teams:
        return

    buffer = wee_matter.room.get_buffer_from_channel_id(post["channel_id"])
    if not weechat.buffer_get_integer(buffer, "localvar_hydrated"):
        return

    post = wee_matter.post.build_post_from_post_data(post)
    wee_matter.post.write_post(post)

    buffer = wee_matter.room.get_buffer_from_channel_id(post.channel_id)

    if buffer == weechat.current_buffer():
        wee_matter.room.mark_channel_as_read(buffer)

@@ -131,8 +133,12 @@ def handle_reaction_added_message(server, message):

    reaction_data = json.loads(data["reaction"])

    reaction = wee_matter.post.get_reaction_from_reaction_data(reaction_data, server)
    # ignore if post not yet loaded
    buffer = wee_matter.post.get_buffer_from_post_id(reaction_data["post_id"])
    if not buffer:
        return

    reaction = wee_matter.post.get_reaction_from_reaction_data(reaction_data, server)

    wee_matter.post.add_reaction_to_post(buffer, reaction)

@@ -141,8 +147,12 @@ def handle_reaction_removed_message(server, message):

    reaction_data = json.loads(data["reaction"])

    reaction = wee_matter.post.get_reaction_from_reaction_data(reaction_data, server)
    # ignore if post not yet loaded
    buffer = wee_matter.post.get_buffer_from_post_id(reaction_data["post_id"])
    if not buffer:
        return

    reaction = wee_matter.post.get_reaction_from_reaction_data(reaction_data, server)

    wee_matter.post.remove_reaction_from_post(buffer, reaction)

@@ -150,6 +160,12 @@ def handle_post_edited_message(server, message):
    data = message["data"]

    post_data = json.loads(data["post"])

    # ignore if post not yet loaded
    buffer = wee_matter.post.get_buffer_from_post_id(post_data["id"])
    if not buffer:
        return

    post = wee_matter.post.build_post_from_post_data(post_data)
    wee_matter.post.write_post(post)

@@ -157,6 +173,12 @@ def handle_post_deleted_message(server, message):
    data = message["data"]

    post_data = json.loads(data["post"])

    # ignore if post not yet loaded
    buffer = wee_matter.post.get_buffer_from_post_id(post_data["id"])
    if not buffer:
        return

    post = wee_matter.post.build_post_from_post_data(post_data)
    post = post._replace(deleted=True)
    wee_matter.post.write_post(post)
@@ -170,6 +192,10 @@ def handle_user_added_message(server, message):
    data = message["data"]
    broadcast = message["broadcast"]
    buffer = wee_matter.room.get_buffer_from_channel_id(broadcast["channel_id"])

    if not weechat.buffer_get_integer(buffer, "localvar_hydrated"):
        return

    wee_matter.room.create_room_user_from_user_data(data, buffer, server)

def handle_direct_added_message(server, message):
@@ -187,6 +213,10 @@ def handle_user_removed_message(server, message):
    if broadcast["channel_id"]:
        user = server.users[data["user_id"]]
        buffer = wee_matter.room.get_buffer_from_channel_id(broadcast["channel_id"])

        if not weechat.buffer_get_integer(buffer, "localvar_hydrated"):
            return

        wee_matter.room.remove_room_user(buffer, user)

def handle_added_to_team_message(server, message):
-- 
2.26.2
Eddie Barraco
It seems pretty cool ! But what happen if we receive a highlight in a
channel that isnt hidrated ?

I'll probably merge the typo fix after a little checkup. Thanks a lot
for this !