Hello list,
I have a repository [1] with my personal website, in which I used
git-lfs for storing fonts, pictures, etc. I tried to utilize
artifactory [2] since LFS is not (yet) supported by sourcehut,
but after about 14 days without activity (read: me pushing anything new
to the LFS repo), they gave me the boot and dropped the LFS repo.
[1]: https://git.sr.ht/~khardix/khardix.cz
[2]: https://jfrog.com/artifactory/
So now I'm looking for a new place for my binary files,
before throwing in the towel and just commiting them in directly.
Anyone has any recommendations, or even a space they could provide?
Any suggestions appreciated!
Thanks in advance!
--
Jan Staněk – Khardix
Hello Jan / Khardix,
I've just registered a few days ago and am looking at solutions (inline).
On 2022-09-20 10:36 a.m., Jan “Khardix” Staněk wrote:
> Hello list,> I have a repository [1] with my personal website, in which I used> git-lfs for storing fonts, pictures, etc. I tried to utilize
I use Hugo and have been using images in 'page bundles', using Hugo's
image processing, and using caches and Git-LFS, but was never too happy
about that solution, partly because of Git-LFS itself (Git-LFS has more
than a few issues, which you can search for if you need to know what
they are), partly because the caching always felt like a bit of kludge
that was needed avoid actually processing the images on every build.
In short I have a similar issue, and have been thinking about how to
solve it.
In my case I have plenty of local storage and plenty of storage on my
web hosting (self-hosted), so that is not the issue for me.
My plan is replace the use of image processing (and page bundles) with:
1. A local directory with my images
2. Using ImageMagick or similar tools to process the images via a script
(initially one big process, and then as I add images).
3. In the script generating a JSON file that contains a mapping of some
short(ish) name to the final URL.
4. Modifying my image processing layouts in Hugo to use the JSON to
convert form the short name to the final URL (and to detect if I have
'dangling' image 'src' attributes).
5. The processed images I intended to upload separate from main site (so
Hugo won't see them, and builds.sr.ht won't have to deal with them) into
a directory (e.g. '/assets-processed/images') or some such, on the hosting.
6. Excluding that directory when I do my sync of the the files under
/public (the Hugo output directory) to the hosting (so that it doesn't
get wiped out when deploying).
With other binary assets, it will be the same except the assets won't
need processing.
If you have some time before you need a solution, I can ping you when I
have got some automation of the above working to my satisfaction.
> artifactory [2] since LFS is not (yet) supported by sourcehut,> but after about 14 days without activity (read: me pushing anything new> to the LFS repo), they gave me the boot and dropped the LFS repo.>> [1]: https://git.sr.ht/~khardix/khardix.cz> [2]: https://jfrog.com/artifactory/>> So now I'm looking for a new place for my binary files,> before throwing in the towel and just commiting them in directly.
I would avoid doing that. Git's not very good at binary files on it's
own. (And could make your repository / usage too large for sr.ht)
> Anyone has any recommendations, or even a space they could provide?> Any suggestions appreciated!>> Thanks in advance!
Hope this is of interest, and can help.
Regards,
Daniel
--
https://wildtechgarden.ca Professional and technical website
https://princesandmadmen.ca Personal and political blog
"Daniel F. Dickinson" <dfdpublic@wildtechgarden.ca> writes:
> In short I have a similar issue, and have been thinking about how to> solve it.>> In my case I have plenty of local storage and plenty of storage on my> web hosting (self-hosted), so that is not the issue for me.>> My plan is replace the use of image processing (and page bundles) with:>> ...>> If you have some time before you need a solution, I can ping you when I> have got some automation of the above working to my satisfaction.
That sounds complicated. I do not foresee having too much images to
process, and I would probably not rely on hugo there and just do this
myself manually as I add them.
The goal of using LFS is essentialy to allow me the assets where I need
them, without having to invent a solution to binary-in-git problem myself.
I was using it for the previous implementation hosted on GH
and did not encounter any problems.
To summarize, I would prefer LFS over custom scripting/automation.
Nevertheless, if you manage to throw something usable together,
I'll probably take a look at least from curiosity :-)
> > So now I'm looking for a new place for my binary files,> > before throwing in the towel and just commiting them in directly.>> I would avoid doing that. Git's not very good at binary files on it's> own. (And could make your repository / usage too large for sr.ht)
I'm aware; that's why I'm asking here :-)
--
Jan Staněk – Khardix
On Wed Sep 21, 2022 at 6:57 AM EDT, Jan “Khardix” Staněk wrote:
> The goal of using LFS is essentialy to allow me the assets where I need> them, without having to invent a solution to binary-in-git problem myself.> I was using it for the previous implementation hosted on GH> and did not encounter any problems.>> To summarize, I would prefer LFS over custom scripting/automation.> Nevertheless, if you manage to throw something usable together,> I'll probably take a look at least from curiosity :-)
You could get an account on a pubnix to host your files and set up LFS
(see tildeverse.org).
Cheers,
--
DJ Chase
They, Them, Theirs
{gemini,https,ipns}://dj-chase.com
DJ Chase <u9000@posteo.mx> writes:
> You could get an account on a pubnix to host your files and set up LFS> (see tildeverse.org).
Thanks, also a possibility. I already have a VPN to host the files,
but that is not running any LFS server; I know about self-hosting
solution(s). I was interested mainly if anyone else already did
the hosting and perhaps provides that as a service.
--
Jan Staněk – Khardix
Jan “Khardix” Staněk <khardix@gmail.com> writes:
> Thanks, also a possibility. I already have a VPN to host the files,
I meant a VPS (server)… d'oh!
I guess the paid youtube promotions of various VPNs are finally
getting to me :-D
--
Jan Staněk – Khardix
On 2022-09-21 6:57 a.m., Jan “Khardix” Staněk wrote:
> "Daniel F. Dickinson" <dfdpublic@wildtechgarden.ca> writes:>> In short I have a similar issue, and have been thinking about how to>> solve it.>>>> In my case I have plenty of local storage and plenty of storage on my>> web hosting (self-hosted), so that is not the issue for me.>>>> My plan is replace the use of image processing (and page bundles) with:>>>> ...>>>> If you have some time before you need a solution, I can ping you when I>> have got some automation of the above working to my satisfaction.> That sounds complicated. I do not foresee having too much images to
While not as complicated as it looks from the list, it is a lot more
work than I really want to do, so I have also been investigating
options. I see that there is something similar to git-lfs called
git-annex (http://git-annex.branchable.com) that is available in Debian
and other distros. It looks promising in not needing support on the git
server, while allowing to use many types of storage (including rsync,
S3, and Drive, etc). (Although I have to verify the 'not needing support
on the git server part).
If this works, it would be a lot less work and likely what I end up
going with.
> process, and I would probably not rely on hugo there and just do this> myself manually as I add them.>> The goal of using LFS is essentialy to allow me the assets where I need> them, without having to invent a solution to binary-in-git problem myself.> I was using it for the previous implementation hosted on GH> and did not encounter any problems.>> To summarize, I would prefer LFS over custom scripting/automation.> Nevertheless, if you manage to throw something usable together,> I'll probably take a look at least from curiosity :-)
I'll let you know if I change my mind (or my experience with git-annex,
if that works).
>>> So now I'm looking for a new place for my binary files,>>> before throwing in the towel and just commiting them in directly.>> I would avoid doing that. Git's not very good at binary files on it's>> own. (And could make your repository / usage too large for sr.ht)> I'm aware; that's why I'm asking here :-)> --> Jan Staněk – Khardix
--
https://wildtechgarden.ca Technical and professional website
https://princesandmadmen.ca Personal and political blog
On Sat Sep 24, 2022 at 2:19 PM PDT, Daniel F. Dickinson wrote:
> ... I have also been investigating options. I see that there is> something similar to git-lfs called git-annex> (http://git-annex.branchable.com)
i can wholeheartedly vouch for git-annex. it is an exceptional piece of
well-documented software that gives power users all the tools they need
to have on-demand access to their files that don't make sense to commit
directly to git.
> It looks promising in not needing support on the git> server, while allowing to use many types of storage (including rsync,> S3, and Drive, etc).
yup.
you can hypothetically use any remote supported by rclone, though ymmv.
i'd consider using a built-in backend over a third-party backend unless
you have a lot of storage space available via a backend that is
supported by a 3rd party implementation such as rclone.
> (Although I have to verify the 'not needing support on the git server part).
correct. a hosted git repo without git-annex support will still contain
the relevant metadata of where files managed by git-annex are hosted.
a git repo with git-annex support as will also be able to store files
managed by git-annex.
> If this works, it would be a lot less work and likely what I end up> going with.
explore this option and report back with your impressions. and feel
free to ask any follow-up questions!
i do wish that sourcehut's team will leave room to consider providing
built-in git-annex support in the future to bridge the large file
support feature gap when compared to other code forges.
On Sat Sep 24, 2022 at 5:19 PM EDT, Daniel F. Dickinson wrote:
> On 2022-09-21 6:57 a.m., Jan “Khardix” Staněk wrote:> > "Daniel F. Dickinson" <dfdpublic@wildtechgarden.ca> writes:> >> In short I have a similar issue, and have been thinking about how to> >> solve it.> >>> >> In my case I have plenty of local storage and plenty of storage on my> >> web hosting (self-hosted), so that is not the issue for me.> >>> >> My plan is replace the use of image processing (and page bundles) with:> >>> >> ...> >>> >> If you have some time before you need a solution, I can ping you when I> >> have got some automation of the above working to my satisfaction.
You could host images over IPFS and then, because IPFS is mountable,
just have symlinks to /ipfs/<cid> in your repo. (Also, I could not find
your original message, so apologies if this is what you suggested.)
Cheers,
--
DJ Chase
They, Them, Theirs
{gemini,https,ipns}://dj-chase.com
akspecs <akspecs@gmail.com> writes:
> On Sat Sep 24, 2022 at 2:19 PM PDT, Daniel F. Dickinson wrote:> > ... I have also been investigating options. I see that there is> > something similar to git-lfs called git-annex> > (http://git-annex.branchable.com)>> i can wholeheartedly vouch for git-annex. it is an exceptional piece of> well-documented software that gives power users all the tools they need> to have on-demand access to their files that don't make sense to commit> directly to git.
I remember trying git-annex back in the day, and giving up on it because
it would keep making weird merge commits (`git annex sync`,
and I hate when tools mess with my commit history),
and the symlinks were too visible*. From that point of view,
the LFS was very nice in that when it worked, it was pretty much
invisible.
On the other hand, I just spent most of last saturday playing with
rudolfs [1], which does not support any authentication. The main idea
was to stick it behind nginx and let that handle the HTTP Basic Auth.
Consequently, I have found out that you cannot just do that
– unless the server asks the client to send the auth header explicitly,
only the initial one will be authenticated. That particular design
decision left a bitter taste in my mouth, so I'm now reconsidering
pretty much everything :-/
* As for the visible simlinks, my `ls` (resp. `exa`) shows the target
next to the symlink, so my directory listings were suddenly full of
hexadecimal hashes. Technically not a problem, but I just could not
bear it :-D.
Anyway, thanks for all the suggestions – hopefully I will be able to
pick from them :)
--
Jan Staněk – Khardix