It seems that upstream is no longer getting updates which the program
over on that Owner being steamctl is broken and still is stuck on old
login system.
all i could think of doing is transplanting in the new login system or
switch codebases to base it off of
https://github.com/Gobot1234/steam.py/
whatever way its done would be helpful
also why can't we just save the manifest outside of a .zip file? and
just save it like as {appid}_{depotid}_{manifestid}
On 14 Feb 2024, at 3:38, Aussie Gamer wrote:
> all i could think of doing is transplanting in the new login system or> switch codebases to base it off of> https://github.com/Gobot1234/steam.py/
Sorry for the delayed response, and thanks for the link. I'm definitely
interested in rewriting to work better with the new login system (and especially
increasing the use of asyncio in the codebase), and if that means moving to a
different upstream library and significantly rewriting the code in the process
then I'd be willing to do it, but I'm really busy with school (hence not
checking my mailing lists for a month) and likely won't have time to do
significant work on this until June at the earliest.
> whatever way its done would be helpful> also why can't we just save the manifest outside of a .zip file? and> just save it like as {appid}_{depotid}_{manifestid}
The .zip is actually the format in which Steam stores the manifest on the
server; it would take extra effort to extract it after download. The zip
extension is added by steamarchiver but it's already a zip when downloaded.
-benjamin
On 19 March 2024, at 8:50, Benjamin Lowry wrote:
> Sorry for the delayed response, and thanks for the link. I'm definitely> interested in rewriting to work better with the new login system (and
especially
> increasing the use of asyncio in the codebase), and if that means moving
to a
> different upstream library and significantly rewriting the code in the
process
> then I'd be willing to do it, but I'm really busy with school (hence not> checking my mailing lists for a month) and likely won't have time to do> significant work on this until June at the earliest.
Sure go ahead take your time but at least that would make everything more
useful in terms of it being up to date as well as working on the newer login
code
> The .zip is actually the format in which Steam stores the manifest on the> server; it would take extra effort to extract it after download. The zip> extension is added by steamarchiver but it's already a zip when
downloaded.
No steam does not store their manifests as .zip nor does it store it as a
zip format
for steamarchiver it seems to me its downloading the original manifest then
compressing it in a .zip file and that is not what is downloaded
-Auzzie
-----Original Message-----
From: Benjamin Lowry <ben@ben.gmbh>
Sent: Tuesday, March 19, 2024 8:50 AM
To: Aussie Gamer <minecraftitsover90@gmail.com>
Cc: ~blowry/steamarchiver@lists.sr.ht
Subject: Re: upstream repo - PythonSteam
On 14 Feb 2024, at 3:38, Aussie Gamer wrote:
> all i could think of doing is transplanting in the new login system or> switch codebases to base it off of> https://github.com/Gobot1234/steam.py/
Sorry for the delayed response, and thanks for the link. I'm definitely
interested in rewriting to work better with the new login system (and
especially
increasing the use of asyncio in the codebase), and if that means moving to
a
different upstream library and significantly rewriting the code in the
process
then I'd be willing to do it, but I'm really busy with school (hence not
checking my mailing lists for a month) and likely won't have time to do
significant work on this until June at the earliest.
> whatever way its done would be helpful> also why can't we just save the manifest outside of a .zip file? and> just save it like as {appid}_{depotid}_{manifestid}
The .zip is actually the format in which Steam stores the manifest on the
server; it would take extra effort to extract it after download. The zip
extension is added by steamarchiver but it's already a zip when downloaded.
-benjamin
I've been going over the steam python library that is currently used,
and I realized that the manifests are downloaded, then processed a
little bit before they are returned to be saved to the drive. I spent
the past few weeks trying to familiarize myself with Python, but so far
I've only gotten far enough to recognize and understand the logic and
syntax a bit to make minor adjustments, like changing the error messages
color so that they stand out.
What I'm failing at doing is figuring out how to migrate over to that
other Steam library that was suggested, and figure out how to directly
save the manifest file before it gets processed without making too many
edits to your existing python script nor to the current Steam Python
library, so that it can be stored in a more direct way. I've been using
this project to download steam CDN chunks, manifests, and depot keys for
checksum scanning to submit to No-Intro's Dat-O-Matic.
https://wiki.no-intro.org/index.php?title=Steam_preservation_proposal
On 29 Jun 2024, at 23:19, Andrew Vineyard wrote:
> What I'm failing at doing is figuring out how to migrate over to that other Steam library that was suggested, and figure out how to directly save the manifest file before it gets processed without making too many edits to your existing python script nor to the current Steam Python library, so that it can be stored in a more direct way.
Migrating to a maintained upstream library (or just ripping code out of the existing library so it can be maintained as part of steamarchiver) is the long-term goal. In the short-term I don't have time to make extensive changes to the project, but it wouldn't be too difficult to modify try_load_manifest in depot_archiver.py to download the manifest directly from the server rather than calling SteamClient.get_manifest.
I'll commit to the repository soon (if not tonight then hopefully tomorrow) to change this behavior, but for reference: it looks like the issue you're seeing is that get_manifest deserializes the manifest into a DepotManifestClass before returning it, and for some reason the serialization of this class results in a differing output. My plan is to bypass this by simply downloading the manifest directly, saving it as an unmodified download from the CDN, and then loading it from disk.
PS: if anyone has been having issues with appinfo downloads, looks like Valve updated the format and there's an outstanding PR (https://github.com/ValvePython/steam/pull/464) to fix it upstream.
> I've been using this project to download steam CDN chunks, manifests, and depot keys for checksum scanning to submit to No-Intro's Dat-O-Matic.>> https://wiki.no-intro.org/index.php?title=Steam_preservation_proposal
Is this tool being used by No-Intro in any official capacity or is it just you? I would be happy to get in contact with anybody from No-Intro interested in making use of steamarchiver to give me feedback on how I can make the tool work well with their preservation workflow (e.g. adding support for auto-generating XML datfiles from a manifest?) Patches are welcome!
Thanks,
-benjamin
I've pushed a new commit that should make steamarchiver save depot manifests as exact copies from the CDN: https://git.sr.ht/~blowry/steamarchiver/commit/6c533b685446662c76f7f918d3bde0deec907b98
Let me know if you have any further concerns, or anything else I can do to help make steamarchiver more suitable for No-Intro's uses! I hope to eventually be able to address the issues with upstream being largely unmaintained.
-benjamin
On 2 Jul 2024, at 23:45, Benjamin Lowry wrote:
>> What I'm failing at doing is figuring out how to migrate over to that other Steam library that was suggested, and figure out how to directly save the manifest file before it gets processed without making too many edits to your existing python script nor to the current Steam Python library, so that it can be stored in a more direct way.>> Migrating to a maintained upstream library (or just ripping code out of the existing library so it can be maintained as part of steamarchiver) is the long-term goal. In the short-term I don't have time to make extensive changes to the project, but it wouldn't be too difficult to modify try_load_manifest in depot_archiver.py to download the manifest directly from the server rather than calling SteamClient.get_manifest.>> I'll commit to the repository soon (if not tonight then hopefully tomorrow) to change this behavior, but for reference: it looks like the issue you're seeing is that get_manifest deserializes the manifest into a DepotManifestClass before returning it, and for some reason the serialization of this class results in a differing output. My plan is to bypass this by simply downloading the manifest directly, saving it as an unmodified download from the CDN, and then loading it from disk.>> PS: if anyone has been having issues with appinfo downloads, looks like Valve updated the format and there's an outstanding PR (https://github.com/ValvePython/steam/pull/464) to fix it upstream.>>> I've been using this project to download steam CDN chunks, manifests, and depot keys for checksum scanning to submit to No-Intro's Dat-O-Matic.>>>> https://wiki.no-intro.org/index.php?title=Steam_preservation_proposal>> Is this tool being used by No-Intro in any official capacity or is it just you? I would be happy to get in contact with anybody from No-Intro interested in making use of steamarchiver to give me feedback on how I can make the tool work well with their preservation workflow (e.g. adding support for auto-generating XML datfiles from a manifest?) Patches are welcome!
>I've pushed a new commit that should make steamarchiver save depot
manifests as exact copies from the CDN:
https://git.sr.ht/~blowry/steamarchiver/commit/6c533b685446662c76f7f918d3bde0deec907b98
>
>Let me know if you have any further concerns, or anything else I can
do to help make steamarchiver more suitable for No-Intro's uses! I hope
to eventually be able to address the issues with upstream being largely
unmaintained.
>
>-benjamin
>
This is perfect. That will help bring us closer to having the raw CDN files.
>On 2 Jul 2024, at 23:45, Benjamin Lowry wrote:
>
>>> What I'm failing at doing is figuring out how to migrate over to
that other Steam library that was suggested, and figure out how to
directly save the manifest file before it gets processed without making
too many edits to your existing python script nor to the current Steam
Python library, so that it can be stored in a more direct way.
>>
>> Migrating to a maintained upstream library (or just ripping code out
of the existing library so it can be maintained as part of
steamarchiver) is the long-term goal. In the short-term I don't have
time to make extensive changes to the project, but it wouldn't be too
difficult to modify try_load_manifest in depot_archiver.py to download
the manifest directly from the server rather than calling
SteamClient.get_manifest.
>>
>> I'll commit to the repository soon (if not tonight then hopefully
tomorrow) to change this behavior, but for reference: it looks like the
issue you're seeing is that get_manifest deserializes the manifest into
a DepotManifestClass before returning it, and for some reason the
serialization of this class results in a differing output. My plan is to
bypass this by simply downloading the manifest directly, saving it as an
unmodified download from the CDN, and then loading it from disk.
>>
>> PS: if anyone has been having issues with appinfo downloads, looks
like Valve updated the format and there's an outstanding PR
(https://github.com/ValvePython/steam/pull/464) to fix it upstream.
>>
>>> I've been using this project to download steam CDN chunks,
manifests, and depot keys for checksum scanning to submit to No-Intro's
Dat-O-Matic.
>>>
>>> https://wiki.no-intro.org/index.php?title=Steam_preservation_proposal
>>
>> Is this tool being used by No-Intro in any official capacity or is
it just you? I would be happy to get in contact with anybody from
No-Intro interested in making use of steamarchiver to give me feedback
on how I can make the tool work well with their preservation workflow
(e.g. adding support for auto-generating XML datfiles from a manifest?)
Patches are welcome!
I have gotten in touch with the team with No-Intro, when I tried
building my own process using nodejs, but one of the things that was
holding me back was the sheer volume of files. When they suggested
combining the chunks into csd/csm files, I quickly stumbled upon your
work, which is lightyears ahead of mine.
They are aware of your project, and I have been testing it daily. Not
counting a few things that prevent stuff like getting depot keys from
working, your work is far better than mine, so I've been updating them
when I finish my tests, which have been a little slow for some games
(CP2077, Ark, etc). They have a Discord Server that you can join. Link
to it can be found here:
https://wiki.no-intro.org/index.php?title=Social_media_accounts
You can discuss with them various things they are looking for, like
downloading every manifest version of a game, branches, etc. Their To-Do
page, https://wiki.no-intro.org/index.php?title=To-do#Steam, originally
brought up the desire to find a way to download Steam CDN Chunks,
Manifests, and Depot Keys to be checksummed and DATed directly, instead
of the outputted/installed files, so this project of yours sounds
perfect for that desire.