Committed, thanks so much for the changes! I have a few ideas:

1. We should probably update everything that uses depot keys (i.e. get_depot_keys etc.) to use the No-Intro binary format
2. Error messages, in addition to being in red, should probably be sent to stderr. Ideally we can make a function to reuse for this purpose
3. How does one go about getting a No-Intro login to view the datfiles?

(by the way, I joined the discord, my username is @benlowry)

Thanks,
-Benjamin

I've pushed a new commit that should make steamarchiver save depot manifests as exact copies from the CDN: https://git.sr.ht/~blowry/steamarchiver/commit/6c533b685446662c76f7f918d3bde0deec907b98

Let me know if you have any further concerns, or anything else I can do to help make steamarchiver more suitable for No-Intro's uses! I hope to eventually be able to address the issues with upstream being largely unmaintained.

-benjamin

On 2 Jul 2024, at 23:45, Benjamin Lowry wrote:

>> What I'm failing at doing is figuring out how to migrate over to that other Steam library that was suggested, and figure out how to directly save the manifest file before it gets processed without making too many edits to your existing python script nor to the current Steam Python library, so that it can be stored in a more direct way.
>
> Migrating to a maintained upstream library (or just ripping code out of the existing library so it can be maintained as part of steamarchiver) is the long-term goal. In the short-term I don't have time to make extensive changes to the project, but it wouldn't be too difficult to modify try_load_manifest in depot_archiver.py to download the manifest directly from the server rather than calling SteamClient.get_manifest.
>
> I'll commit to the repository soon (if not tonight then hopefully tomorrow) to change this behavior, but for reference: it looks like the issue you're seeing is that get_manifest deserializes the manifest into a DepotManifestClass before returning it, and for some reason the serialization of this class results in a differing output. My plan is to bypass this by simply downloading the manifest directly, saving it as an unmodified download from the CDN, and then loading it from disk.
>

On 29 Jun 2024, at 23:19, Andrew Vineyard wrote:

> What I'm failing at doing is figuring out how to migrate over to that other Steam library that was suggested, and figure out how to directly save the manifest file before it gets processed without making too many edits to your existing python script nor to the current Steam Python library, so that it can be stored in a more direct way.

Migrating to a maintained upstream library (or just ripping code out of the existing library so it can be maintained as part of steamarchiver) is the long-term goal. In the short-term I don't have time to make extensive changes to the project, but it wouldn't be too difficult to modify try_load_manifest in depot_archiver.py to download the manifest directly from the server rather than calling SteamClient.get_manifest.

I'll commit to the repository soon (if not tonight then hopefully tomorrow) to change this behavior, but for reference: it looks like the issue you're seeing is that get_manifest deserializes the manifest into a DepotManifestClass before returning it, and for some reason the serialization of this class results in a differing output. My plan is to bypass this by simply downloading the manifest directly, saving it as an unmodified download from the CDN, and then loading it from disk.

PS: if anyone has been having issues with appinfo downloads, looks like Valve updated the format and there's an outstanding PR (https://github.com/ValvePython/steam/pull/464) to fix it upstream.

> I've been using this project to download steam CDN chunks, manifests, and depot keys for checksum scanning to submit to No-Intro's Dat-O-Matic.
>
> https://wiki.no-intro.org/index.php?title=Steam_preservation_proposal

On 14 Feb 2024, at 3:38, Aussie Gamer wrote:

> all i could think of doing is transplanting in the new login system or
> switch codebases to base it off of
> https://github.com/Gobot1234/steam.py/
Sorry for the delayed response, and thanks for the link. I'm definitely
interested in rewriting to work better with the new login system (and especially
increasing the use of asyncio in the codebase), and if that means moving to a
different upstream library and significantly rewriting the code in the process
then I'd be willing to do it, but I'm really busy with school (hence not
checking my mailing lists for a month) and likely won't have time to do
significant work on this until June at the earliest.

> whatever way its done would be helpful

On 21 Aug 2023, at 9:10, steamarchiver@elbert.anonaddy.com wrote:

> Hey, in the README it's stated that you could download the depots of
> an app even if you don't own the app.m However, whenever I try this,
> I get stucked at "Getting a manifest for app x depot y gid z". It
> works just fine on apps I own though. What am I doing wrong?
Sorry for the extremely late response, I've been having trouble with my email lately.

You weren't doing anything wrong, and what the other user said is correct; while you can still technically download _chunks_ (pieces of game content) from SteamPipe servers without ownership, you have no way of knowing which chunks you need to download without a _content manifest_, which used to be accessible without authentication but now does require authentication. I forgot to update the README when Valve made that change, so thanks for reminding me to do that! (I'll also look into why the program just freezes on "Getting a manifest"; it's supposed to throw an error when the manifest download fails due to lack of authentication.)

If you want to download a non-anonymous (i.e. not a dedicated server, etc.) app using steamarchiver, even if the app is free, you'll need to log in with an account like so:

    python3 depot_archiver.py -a [appid] -u [username]

Use canonical_name to redirect to the tracker page with
a tilde before the username.

Signed-off-by: Benjamin Lowry <ben@ben.gmbh>
---
 todosrht/blueprints/tracker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/todosrht/blueprints/tracker.py b/todosrht/blueprints/tracker.py
index 68cde2c..4d3a5ae 100644
--- a/todosrht/blueprints/tracker.py
+++ b/todosrht/blueprints/tracker.py
@@ -53,7 +53,7 @@ def create_POST():
 
[message trimmed]

This makes the behavior consistent with other sr.ht services

Signed-off-by: Benjamin Lowry <ben@ben.gmbh>
---
 todosrht/access.py | 36 ++++++++++++++++++------------------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/todosrht/access.py b/todosrht/access.py
index 3d5eadd..06e3e86 100644
--- a/todosrht/access.py
+++ b/todosrht/access.py
@@ -39,26 +39,26 @@ def get_tracker(owner, name, with_for_update=False, user=None):
     if not owner:
         return None, None
[message trimmed]

This makes todo.sr.ht's behavior consistent with other sr.ht services
by returning a 404 if you try to omit the tilde in a tracker URL. The
second commit also fixes the "Create & configure tracker" button
redirecting you to https://todo.sr.ht/username/tracker without a tilde.
Fixes https://todo.sr.ht/~sircmpwn/todo.sr.ht/198

Benjamin Lowry (2):
  get_tracker: return None if tilde is ommitted
  redirect to correct page when creating & configuring tracker

 todosrht/access.py             | 36 +++++++++++++++++-----------------
 todosrht/blueprints/tracker.py |  2 +-
 2 files changed, 19 insertions(+), 19 deletions(-)

I'm using the latest versions of everything from the git repositories,
on Alpine Linux. All the oauth clients are preauthorized. All of the
services are using the same config.ini in /etc/sr.ht/.

On Mon, 2020-05-04 at 11:31 -0400, Drew DeVault wrote:
> Can you also share the relevant lines from git.sr.ht's web logs?
127.0.0.1 - - [04/May/2020 10:32:43] "GET /api/repos?start=-1 HTTP/1.0" 200 -
127.0.0.1 - - [04/May/2020 10:32:48] "GET /api/repos/test-repository HTTP/1.0" 200 -
127.0.0.1 - - [04/May/2020 10:32:48] "GET /api/user/webhooks?start=-1 HTTP/1.0" 200 -
127.0.0.1 - - [04/May/2020 10:32:48] "GET /api/~blowry/repos/test-repository/webhooks?start=-1 HTTP/1.0" 200 -
127.0.0.1 - - [04/May/2020 10:32:48] "GET /api/~blowry/repos/test-repository/webhooks HTTP/1.0" 200 -