~radicle-link/dev

radicle-link: linkoxide v1 PROPOSED

gitoxide serves us well for some very low-level git operations, as it presents
itself more modular than libgit2. It is lacking, however, when it comes to
high-level abstractions and even get some datastructures outright wrong.
Additionally, we saw repeated breakage due to "semantic" versioning mistakes.

We are only using a fraction of the functionality of git itself and gitoxide
specifically, and have specific requirements for abstractions built on top of
it. Thus, consolidate the core functionality in a separate crate, `link-git`,
re-exporting only the subset of gitoxide crates which are needed. The crate also
exports the link-specific protocol code, as well as threadsafe, shareable
reference and object databases.

The surface is driven by the needs of the `link-replication` crate, which is
still under active development. The repl3 topic has been rebased on top of this
series.

Published-As: https://git.sr.ht/~kim/radicle-link/refs/patches/linkoxide/v1

Kim Altintop (3):
  link-git-protocol: move to link-git crate
  link-git: shareable odb with pack cache
  link-git: shareable refdb with packed-refs cache

 Cargo.toml                                    |   2 +-
 git-ext/Cargo.toml                            |   4 +-
 git-ext/src/oid.rs                            |  12 +-
 librad/Cargo.toml                             |   4 +-
 librad/src/net/protocol/io/recv/git.rs        |   2 +-
 link-git-protocol/Cargo.toml                  |  38 ---
 link-git/Cargo.toml                           |  62 ++++
 link-git/src/lib.rs                           |  20 ++
 link-git/src/odb.rs                           |  51 ++++
 link-git/src/odb/backend.rs                   |  36 +++
 link-git/src/odb/index.rs                     | 280 ++++++++++++++++++
 link-git/src/odb/pack.rs                      | 129 ++++++++
 link-git/src/odb/window.rs                    | 183 ++++++++++++
 link-git/src/odb/window/metrics.rs            |  86 ++++++
 .../src/lib.rs => link-git/src/protocol.rs    |   5 +-
 .../src => link-git/src/protocol}/fetch.rs    |  23 +-
 .../src => link-git/src/protocol}/ls.rs       |  22 +-
 .../src/protocol}/packwriter.rs               |  22 +-
 .../src => link-git/src/protocol}/take.rs     |   0
 .../src/protocol}/transport.rs                |   2 +-
 .../src/protocol}/upload_pack.rs              |  10 +-
 .../src/protocol}/upload_pack/legacy.rs       |   2 +-
 link-git/src/refs.rs                          |   7 +
 link-git/src/refs/db.rs                       | 266 +++++++++++++++++
 test/Cargo.toml                               |   4 +-
 test/src/test/integration.rs                  |   2 +-
 test/src/test/integration/link_git.rs         |   6 +
 .../protocol.rs}                              |   2 +-
 test/src/test/unit.rs                         |   2 +-
 test/src/test/unit/link_git.rs                |   6 +
 .../protocol.rs}                              |   0
 .../protocol}/take.rs                         |   2 +-
 .../protocol}/upload_pack.rs                  |   2 +-
 33 files changed, 1187 insertions(+), 107 deletions(-)
 delete mode 100644 link-git-protocol/Cargo.toml
 create mode 100644 link-git/Cargo.toml
 create mode 100644 link-git/src/lib.rs
 create mode 100644 link-git/src/odb.rs
 create mode 100644 link-git/src/odb/backend.rs
 create mode 100644 link-git/src/odb/index.rs
 create mode 100644 link-git/src/odb/pack.rs
 create mode 100644 link-git/src/odb/window.rs
 create mode 100644 link-git/src/odb/window/metrics.rs
 rename link-git-protocol/src/lib.rs => link-git/src/protocol.rs (89%)
 rename {link-git-protocol/src => link-git/src/protocol}/fetch.rs (94%)
 rename {link-git-protocol/src => link-git/src/protocol}/ls.rs (90%)
 rename {link-git-protocol/src => link-git/src/protocol}/packwriter.rs (94%)
 rename {link-git-protocol/src => link-git/src/protocol}/take.rs (100%)
 rename {link-git-protocol/src => link-git/src/protocol}/transport.rs (97%)
 rename {link-git-protocol/src => link-git/src/protocol}/upload_pack.rs (96%)
 rename {link-git-protocol/src => link-git/src/protocol}/upload_pack/legacy.rs (99%)
 create mode 100644 link-git/src/refs.rs
 create mode 100644 link-git/src/refs/db.rs
 create mode 100644 test/src/test/integration/link_git.rs
 rename test/src/test/integration/{link_git_protocol.rs => link_git/protocol.rs} (99%)
 create mode 100644 test/src/test/unit/link_git.rs
 rename test/src/test/unit/{link_git_protocol.rs => link_git/protocol.rs} (100%)
 rename test/src/test/unit/{link_git_protocol => link_git/protocol}/take.rs (97%)
 rename test/src/test/unit/{link_git_protocol => link_git/protocol}/upload_pack.rs (98%)

--
2.34.0
#630900 linux-x86_64.yml success
radicle-link/patches/linux-x86_64.yml: SUCCESS in 31m22s

[linkoxide][0] from [Kim Altintop][1]

[0]: https://lists.sr.ht/~radicle-link/dev/patches/26683
[1]: mailto:kim@eagain.st

✓ #630900 SUCCESS radicle-link/patches/linux-x86_64.yml https://builds.sr.ht/~radicle-link/job/630900
Published-As: https://git.sr.ht/~kim/radicle-link/refs/patches/linkoxide/v2
Again, looks good from my side!

On Mon Nov 22, 2021 at 7:25 AM GMT, Kim Altintop wrote:
Next
Export patchset (mbox)
How do I use this?

Copy & paste the following snippet into your terminal to import this patchset into git:

curl -s https://lists.sr.ht/~radicle-link/dev/patches/26683/mbox | git am -3
Learn more about email & git

[PATCH radicle-link 1/3] link-git-protocol: move to link-git crate Export this patch

We make use of only a fraction of the gitoxide suite of crates, and
where we do, the high-level `git-repository` crate doesn't meet our
needs. Thus, establish a `link-git` crate which re-exports just the
things from gitoxide crates deemed useful, and implements the custom
functionality we need.

This is part of a series: as a first step, only move `link-git-protocol`
to `link-git`.

Signed-off-by: Kim Altintop <kim@eagain.st>
---
 Cargo.toml                                    |  2 +-
 git-ext/Cargo.toml                            |  4 +-
 git-ext/src/oid.rs                            | 12 ++--
 librad/Cargo.toml                             |  4 +-
 librad/src/net/protocol/io/recv/git.rs        |  2 +-
 link-git-protocol/Cargo.toml                  | 38 ------------
 link-git/Cargo.toml                           | 62 +++++++++++++++++++
 link-git/src/lib.rs                           | 18 ++++++
 .../src/lib.rs => link-git/src/protocol.rs    |  5 +-
 .../src => link-git/src/protocol}/fetch.rs    | 23 +++----
 .../src => link-git/src/protocol}/ls.rs       | 22 +++----
 .../src/protocol}/packwriter.rs               | 22 +++----
 .../src => link-git/src/protocol}/take.rs     |  0
 .../src/protocol}/transport.rs                |  2 +-
 .../src/protocol}/upload_pack.rs              | 10 ++-
 .../src/protocol}/upload_pack/legacy.rs       |  2 +-
 test/Cargo.toml                               |  4 +-
 test/src/test/integration.rs                  |  2 +-
 test/src/test/integration/link_git.rs         |  6 ++
 .../protocol.rs}                              |  2 +-
 test/src/test/unit.rs                         |  2 +-
 test/src/test/unit/link_git.rs                |  6 ++
 .../protocol.rs}                              |  0
 .../protocol}/take.rs                         |  2 +-
 .../protocol}/upload_pack.rs                  |  2 +-
 25 files changed, 147 insertions(+), 107 deletions(-)
 delete mode 100644 link-git-protocol/Cargo.toml
 create mode 100644 link-git/Cargo.toml
 create mode 100644 link-git/src/lib.rs
 rename link-git-protocol/src/lib.rs => link-git/src/protocol.rs (89%)
 rename {link-git-protocol/src => link-git/src/protocol}/fetch.rs (94%)
 rename {link-git-protocol/src => link-git/src/protocol}/ls.rs (90%)
 rename {link-git-protocol/src => link-git/src/protocol}/packwriter.rs (94%)
 rename {link-git-protocol/src => link-git/src/protocol}/take.rs (100%)
 rename {link-git-protocol/src => link-git/src/protocol}/transport.rs (97%)
 rename {link-git-protocol/src => link-git/src/protocol}/upload_pack.rs (96%)
 rename {link-git-protocol/src => link-git/src/protocol}/upload_pack/legacy.rs (99%)
 create mode 100644 test/src/test/integration/link_git.rs
 rename test/src/test/integration/{link_git_protocol.rs => link_git/protocol.rs} (99%)
 create mode 100644 test/src/test/unit/link_git.rs
 rename test/src/test/unit/{link_git_protocol.rs => link_git/protocol.rs} (100%)
 rename test/src/test/unit/{link_git_protocol => link_git/protocol}/take.rs (97%)
 rename test/src/test/unit/{link_git_protocol => link_git/protocol}/upload_pack.rs (98%)

diff --git a/Cargo.toml b/Cargo.toml
index 31a24f47..2f2b73f0 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -12,7 +12,7 @@ members = [
  "link-canonical",
  "link-canonical-derive",
  "link-crypto",
  "link-git-protocol",
  "link-git",
  "link-identities",
  "macros",
  "node-lib",
diff --git a/git-ext/Cargo.toml b/git-ext/Cargo.toml
index d110549c..79947073 100644
--- a/git-ext/Cargo.toml
+++ b/git-ext/Cargo.toml
@@ -20,8 +20,8 @@ version = ">= 0.13.12, 0.13"
default-features = false
features = []

[dependencies.git-repository]
version = "0.11.0"
[dependencies.link-git]
path = "../link-git"
optional = true

[dependencies.minicbor]
diff --git a/git-ext/src/oid.rs b/git-ext/src/oid.rs
index 0ab5865f..907af143 100644
--- a/git-ext/src/oid.rs
+++ b/git-ext/src/oid.rs
@@ -13,8 +13,8 @@ use std::{
use multihash::{Multihash, MultihashRef};
use thiserror::Error;

#[cfg(feature = "git-repository")]
use git_repository::hash as git_hash;
#[cfg(feature = "link-git")]
use link_git::hash as git_hash;

/// Serializable [`git2::Oid`]
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
@@ -116,7 +116,7 @@ impl AsRef<[u8]> for Oid {
    }
}

#[cfg(feature = "git-repository")]
#[cfg(feature = "link-git")]
impl AsRef<git_hash::oid> for Oid {
    fn as_ref(&self) -> &git_hash::oid {
        // SAFETY: checks the length of the slice, which we know is correct
@@ -136,7 +136,7 @@ impl From<Oid> for git2::Oid {
    }
}

#[cfg(feature = "git-repository")]
#[cfg(feature = "link-git")]
impl From<git_hash::ObjectId> for Oid {
    fn from(git_hash::ObjectId::Sha1(bs): git_hash::ObjectId) -> Self {
        // SAFETY: checks the length of the slice, which we statically know
@@ -144,14 +144,14 @@ impl From<git_hash::ObjectId> for Oid {
    }
}

#[cfg(feature = "git-repository")]
#[cfg(feature = "link-git")]
impl From<Oid> for git_hash::ObjectId {
    fn from(oid: Oid) -> Self {
        Self::from_20_bytes(oid.as_ref())
    }
}

#[cfg(feature = "git-repository")]
#[cfg(feature = "link-git")]
impl<'a> From<&'a Oid> for &'a git_hash::oid {
    fn from(oid: &'a Oid) -> Self {
        oid.as_ref()
diff --git a/librad/Cargo.toml b/librad/Cargo.toml
index c145bcce..b9825dbf 100644
--- a/librad/Cargo.toml
+++ b/librad/Cargo.toml
@@ -95,8 +95,8 @@ path = "../link-canonical"
[dependencies.link-crypto]
path = "../link-crypto"

[dependencies.link-git-protocol]
path = "../link-git-protocol"
[dependencies.link-git]
path = "../link-git"
features = ["git2"]

[dependencies.link-identities]
diff --git a/librad/src/net/protocol/io/recv/git.rs b/librad/src/net/protocol/io/recv/git.rs
index 3666c51a..c7b50b8d 100644
--- a/librad/src/net/protocol/io/recv/git.rs
+++ b/librad/src/net/protocol/io/recv/git.rs
@@ -7,7 +7,7 @@
use std::{io, process::ExitStatus};

use futures::io::{AsyncRead, AsyncWrite};
use link_git_protocol::upload_pack::{upload_pack, Header};
use link_git::protocol::upload_pack::{upload_pack, Header};
use thiserror::Error;
use tracing::{error, info};

diff --git a/link-git-protocol/Cargo.toml b/link-git-protocol/Cargo.toml
deleted file mode 100644
index 76bdc3b3..00000000
--- a/link-git-protocol/Cargo.toml
@@ -1,38 +0,0 @@
[package]
name = "link-git-protocol"
version = "0.1.0"
authors = ["Kim Altintop <kim@eagain.st>"]
edition = "2018"
license = "GPL-3.0-or-later"

description = "radicle-link-flavoured git protocol-v2 client and server"

[lib]
doctest = false
test = false

[dependencies]
async-process = "1.1.0"
async-trait = "0.1"
blocking = "1.0.2"
bstr = "0.2.16"
futures-lite = "1.12.0"
futures-util = "0.3.15"
once_cell = "1.8.0"
pin-project = "1.0.7"
tempfile = "3.2.0"
versions = "3.0.2"

[dependencies.git-repository]
version = "0.11.0"
features = [ "async-network-client", "unstable" ]

[dependencies.git-packetline]
version = "0.12.0"
features = ["async-io"]

[dependencies.git2]
version = "=0.13.20"
default-features = false
features = []
optional = true
diff --git a/link-git/Cargo.toml b/link-git/Cargo.toml
new file mode 100644
index 00000000..8214f2a2
--- /dev/null
+++ b/link-git/Cargo.toml
@@ -0,0 +1,62 @@
[package]
name = "link-git"
version = "0.1.0"
authors = ["Kim Altintop <kim@eagain.st>"]
edition = "2018"
license = "GPL-3.0-or-later"

description = "Core git types and functionality"

[lib]
doctest = false
test = false

[dependencies]
arc-swap = "1.4.0"
async-process = "1.1.0"
async-trait = "0.1"
blocking = "1.0.2"
bstr = "0.2.16"
futures-lite = "1.12.0"
futures-util = "0.3.15"
fxhash = "0.2.1"
im = "15.0.0"
once_cell = "1.8.0"
parking_lot = "0.11.2"
pin-project = "1.0.7"
tempfile = "3.2.0"
thiserror = "1.0.30"
tracing = "0.1"
versions = "3.0.2"

# gitoxide
git-actor = "^0.6.0"
git-hash = "^0.8.0"
git-lock = "^1.0.1"
git-object = "^0.15.1"
git-odb = "^0.23.0"
git-ref = "^0.9.0"
git-traverse = "^0.10.0"

[dependencies.git-features]
version = "^0.17.0"
features = ["progress", "parallel", "zlib-ng-compat"]

[dependencies.git-pack]
version = "^0.13.0"
features = ["object-cache-dynamic", "pack-cache-lru-static", "pack-cache-lru-dynamic"]

[dependencies.git-packetline]
version = "^0.12.0"
features = ["async-io"]

[dependencies.git-protocol]
version = "^0.12.0"
features = ["async-client"]

# compat
[dependencies.git2]
version = "=0.13.20"
default-features = false
features = []
optional = true
diff --git a/link-git/src/lib.rs b/link-git/src/lib.rs
new file mode 100644
index 00000000..7e0cf900
--- /dev/null
+++ b/link-git/src/lib.rs
@@ -0,0 +1,18 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

#![feature(array_map, never_type)]

#[macro_use]
extern crate async_trait;

pub mod protocol;

pub use git_actor as actor;
pub use git_hash as hash;
pub use git_lock as lock;
pub use git_object as object;
pub use git_ref as refs;
pub use git_traverse as traverse;
diff --git a/link-git-protocol/src/lib.rs b/link-git/src/protocol.rs
similarity index 89%
rename from link-git-protocol/src/lib.rs
rename to link-git/src/protocol.rs
index 912e111d..740fc420 100644
--- a/link-git-protocol/src/lib.rs
+++ b/link-git/src/protocol.rs
@@ -3,11 +3,8 @@
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

#[macro_use]
extern crate async_trait;

use bstr::ByteSlice as _;
use git_repository::protocol::transport::client;
use git_protocol::transport::client;
use versions::Version;

pub mod fetch;
diff --git a/link-git-protocol/src/fetch.rs b/link-git/src/protocol/fetch.rs
similarity index 94%
rename from link-git-protocol/src/fetch.rs
rename to link-git/src/protocol/fetch.rs
index ffb34fd6..a5a73065 100644
--- a/link-git-protocol/src/fetch.rs
+++ b/link-git/src/protocol/fetch.rs
@@ -20,22 +20,19 @@ use futures_lite::{
    future,
    io::{AsyncBufRead, AsyncRead, AsyncWrite},
};
use git_repository::{
    progress,
    protocol::{
        self,
        fetch::{response, Action, Arguments, Delegate, DelegateBlocking, LsRefsAction, Response},
        transport::client,
    },
    Progress,
use git_features::progress::{self, Progress};
use git_protocol::{
    fetch::{response, Action, Arguments, Delegate, DelegateBlocking, LsRefsAction, Response},
    transport::client,
};
use once_cell::sync::Lazy;
use pin_project::{pin_project, pinned_drop};
use versions::Version;

pub use git_repository::{hash::ObjectId, protocol::fetch::Ref};
pub use git_hash::ObjectId;
pub use git_protocol::fetch::Ref;

use crate::{packwriter::PackWriter, remote_git_version, transport};
use super::{packwriter::PackWriter, remote_git_version, transport};

// Work around `git-upload-pack` not handling namespaces properly,
//
@@ -136,7 +133,7 @@ impl<P: PackWriter> DelegateBlocking for Fetch<P, P::Output> {

    fn prepare_fetch(
        &mut self,
        _: protocol::transport::Protocol,
        _: git_protocol::transport::Protocol,
        caps: &client::Capabilities,
        _: &mut Vec<(&str, Option<&str>)>,
        _: &[Ref],
@@ -267,12 +264,12 @@ where

        move || {
            let mut delegate = Fetch::new(opt, pack_writer);
            future::block_on(protocol::fetch(
            future::block_on(git_protocol::fetch(
                &mut conn,
                &mut delegate,
                |_| unreachable!("credentials helper requested"),
                progress::Discard,
                protocol::FetchConnection::AllowReuse,
                git_protocol::FetchConnection::AllowReuse,
            ))
            .map_err(|e| io::Error::new(io::ErrorKind::Other, e))?;

diff --git a/link-git-protocol/src/ls.rs b/link-git/src/protocol/ls.rs
similarity index 90%
rename from link-git-protocol/src/ls.rs
rename to link-git/src/protocol/ls.rs
index eb14e1e2..b3516454 100644
--- a/link-git-protocol/src/ls.rs
+++ b/link-git/src/protocol/ls.rs
@@ -7,21 +7,17 @@ use std::io;

use bstr::{BString, ByteVec as _};
use futures_lite::io::{AsyncBufRead, AsyncRead, AsyncWrite};
use git_repository::{
    progress,
    protocol::{
        self,
        fetch::{Action, Arguments, Delegate, DelegateBlocking, LsRefsAction, Response},
        transport::client,
    },
    Progress,
use git_features::progress::{self, Progress};
use git_protocol::{
    fetch::{Action, Arguments, Delegate, DelegateBlocking, LsRefsAction, Response},
    transport::client,
};
use once_cell::sync::Lazy;
use versions::Version;

pub use git_repository::protocol::fetch::Ref;
pub use git_protocol::fetch::Ref;

use crate::{remote_git_version, transport};
use super::{remote_git_version, transport};

// Work around `git-upload-pack` not handling namespaces properly
//
@@ -99,7 +95,7 @@ impl DelegateBlocking for LsRefs {

    fn prepare_fetch(
        &mut self,
        _: protocol::transport::Protocol,
        _: git_protocol::transport::Protocol,
        _: &client::Capabilities,
        _: &mut Vec<(&str, Option<&str>)>,
        refs: &[Ref],
@@ -138,12 +134,12 @@ where
{
    let mut conn = transport::Stateless::new(opt.repo.clone(), recv, send);
    let mut delegate = LsRefs::new(opt);
    git_repository::protocol::fetch(
    git_protocol::fetch(
        &mut conn,
        &mut delegate,
        |_| unreachable!("credentials helper requested"),
        progress::Discard,
        protocol::FetchConnection::AllowReuse,
        git_protocol::FetchConnection::AllowReuse,
    )
    .await
    .map_err(|e| io::Error::new(io::ErrorKind::Other, e))?;
diff --git a/link-git-protocol/src/packwriter.rs b/link-git/src/protocol/packwriter.rs
similarity index 94%
rename from link-git-protocol/src/packwriter.rs
rename to link-git/src/protocol/packwriter.rs
index 5aa4ecba..477feb99 100644
--- a/link-git-protocol/src/packwriter.rs
+++ b/link-git/src/protocol/packwriter.rs
@@ -13,24 +13,21 @@ use std::{
};

use futures_lite::io::{AsyncBufRead, BlockOn};
use git_repository::{
    hash::ObjectId,
    odb::{self, pack},
    Progress,
};
use git_features::progress::Progress;
use git_hash::ObjectId;
use git_odb::{self as odb, pack};

use crate::take::TryTake;
use super::take::TryTake;

#[cfg(feature = "git2")]
pub use libgit::Libgit;

/// What to do with the `packfile` response.
///
/// _This is mostly the same as [`git_repository::protocol::fetch::Delegate`],
/// but without incurring the
/// [`git_repository::protocol::fetch::DelegateBlocking`] super-trait
/// constraint. We can simply make [`crate::fetch::Fetch`] parametric over the
/// packfile sink._
/// _This is mostly the same as [`git_protocol::fetch::Delegate`], but without
/// incurring the [`git_protocol::fetch::DelegateBlocking`] super-trait
/// constraint. We can simply make [`crate::protocol::fetch::Fetch`] parametric
/// over the packfile sink._
pub trait PackWriter {
    type Output;

@@ -159,7 +156,8 @@ impl Thickener for odb::linked::Store {
        id: ObjectId,
        buf: &'a mut Vec<u8>,
    ) -> Option<pack::data::Object<'a>> {
        use git_repository::prelude::FindExt as _;
        use git_odb::FindExt as _;

        self.find(id, buf, &mut pack::cache::Never).ok()
    }
}
diff --git a/link-git-protocol/src/take.rs b/link-git/src/protocol/take.rs
similarity index 100%
rename from link-git-protocol/src/take.rs
rename to link-git/src/protocol/take.rs
diff --git a/link-git-protocol/src/transport.rs b/link-git/src/protocol/transport.rs
similarity index 97%
rename from link-git-protocol/src/transport.rs
rename to link-git/src/protocol/transport.rs
index daa1b771..473f2e5a 100644
--- a/link-git-protocol/src/transport.rs
+++ b/link-git/src/protocol/transport.rs
@@ -5,7 +5,7 @@

use bstr::BString;
use futures_lite::io::{AsyncRead, AsyncWrite};
use git_repository::protocol::transport::{
use git_protocol::transport::{
    client::{
        self,
        git::{ConnectMode, Connection},
diff --git a/link-git-protocol/src/upload_pack.rs b/link-git/src/protocol/upload_pack.rs
similarity index 96%
rename from link-git-protocol/src/upload_pack.rs
rename to link-git/src/protocol/upload_pack.rs
index b0a4e596..64b335cd 100644
--- a/link-git-protocol/src/upload_pack.rs
+++ b/link-git/src/protocol/upload_pack.rs
@@ -8,7 +8,7 @@ use std::{future::Future, io, path::Path, process::ExitStatus, str::FromStr};
use async_process::{Command, Stdio};
use futures_lite::io::{copy, AsyncBufReadExt as _, AsyncRead, AsyncWrite, BufReader};
use futures_util::try_join;
use git_packetline::PacketLineRef;
use git_packetline::{self as packetline, PacketLineRef};
use once_cell::sync::Lazy;
use versions::Version;

@@ -80,7 +80,7 @@ where
            buf.parse().map_err(invalid_data)?
        },
        Some(_) => {
            let mut pktline = git_packetline::StreamingPeekableIter::new(recv, &[]);
            let mut pktline = packetline::StreamingPeekableIter::new(recv, &[]);
            let pkt = pktline
                .read_line()
                .await
@@ -187,8 +187,6 @@ async fn advertise_capabilities<W>(mut send: W) -> io::Result<()>
where
    W: AsyncWrite + Unpin,
{
    use git_packetline::encode;

    // Thou shallt not upgrade your `git` installation while a link instance is
    // running!
    static GIT_VERSION: Lazy<Version> = Lazy::new(|| git_version().unwrap());
@@ -203,9 +201,9 @@ where
    });

    for cap in *CAPABILITIES {
        encode::text_to_write(cap, &mut send).await?;
        packetline::encode::text_to_write(cap, &mut send).await?;
    }
    encode::flush_to_write(&mut send).await?;
    packetline::encode::flush_to_write(&mut send).await?;

    Ok(())
}
diff --git a/link-git-protocol/src/upload_pack/legacy.rs b/link-git/src/protocol/upload_pack/legacy.rs
similarity index 99%
rename from link-git-protocol/src/upload_pack/legacy.rs
rename to link-git/src/protocol/upload_pack/legacy.rs
index 7e1e8de4..30507514 100644
--- a/link-git-protocol/src/upload_pack/legacy.rs
+++ b/link-git/src/protocol/upload_pack/legacy.rs
@@ -9,7 +9,7 @@ use std::{io, path::Path, process::ExitStatus};
use async_process::{Command, Stdio};
use futures_lite::io::{copy, AsyncRead, AsyncReadExt as _, AsyncWrite, AsyncWriteExt as _};
use futures_util::try_join;
use git_repository::refs::{
use git_ref::{
    file::{Store as Refdb, WriteReflog},
    FullName,
    Reference,
diff --git a/test/Cargo.toml b/test/Cargo.toml
index 172f2554..ffb56a93 100644
--- a/test/Cargo.toml
+++ b/test/Cargo.toml
@@ -80,8 +80,8 @@ features = [ "derive" ]
[dependencies.link-identities]
path = "../link-identities"

[dependencies.link-git-protocol]
path = "../link-git-protocol"
[dependencies.link-git]
path = "../link-git"
features = ["git2"]

[dependencies.node-lib]
diff --git a/test/src/test/integration.rs b/test/src/test/integration.rs
index bf6adf8a..cc5dfa9d 100644
--- a/test/src/test/integration.rs
+++ b/test/src/test/integration.rs
@@ -6,7 +6,7 @@
mod daemon;
mod git_helpers;
mod librad;
mod link_git_protocol;
mod link_git;
mod node_lib;
#[cfg(unix)]
mod rad_cli;
diff --git a/test/src/test/integration/link_git.rs b/test/src/test/integration/link_git.rs
new file mode 100644
index 00000000..e388191c
--- /dev/null
+++ b/test/src/test/integration/link_git.rs
@@ -0,0 +1,6 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

mod protocol;
diff --git a/test/src/test/integration/link_git_protocol.rs b/test/src/test/integration/link_git/protocol.rs
similarity index 99%
rename from test/src/test/integration/link_git_protocol.rs
rename to test/src/test/integration/link_git/protocol.rs
index 112e691a..d1df8f1a 100644
--- a/test/src/test/integration/link_git_protocol.rs
+++ b/test/src/test/integration/link_git/protocol.rs
@@ -17,7 +17,7 @@ use git_repository::{
    prelude::*,
    refs::transaction::{Change, PreviousValue, RefEdit},
};
use link_git_protocol::{fetch, ls, packwriter, upload_pack, ObjectId, PackWriter, Ref};
use link_git::protocol::{fetch, ls, packwriter, upload_pack, ObjectId, PackWriter, Ref};
use tempfile::{tempdir, TempDir};

fn upstream() -> TempDir {
diff --git a/test/src/test/unit.rs b/test/src/test/unit.rs
index 0a12952e..d76f819d 100644
--- a/test/src/test/unit.rs
+++ b/test/src/test/unit.rs
@@ -8,7 +8,7 @@ mod git_ext;
mod git_trailers;
mod librad;
mod link_async;
mod link_git_protocol;
mod link_git;
mod node_lib;
mod rad_clib;
mod rad_exe;
diff --git a/test/src/test/unit/link_git.rs b/test/src/test/unit/link_git.rs
new file mode 100644
index 00000000..e388191c
--- /dev/null
+++ b/test/src/test/unit/link_git.rs
@@ -0,0 +1,6 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

mod protocol;
diff --git a/test/src/test/unit/link_git_protocol.rs b/test/src/test/unit/link_git/protocol.rs
similarity index 100%
rename from test/src/test/unit/link_git_protocol.rs
rename to test/src/test/unit/link_git/protocol.rs
diff --git a/test/src/test/unit/link_git_protocol/take.rs b/test/src/test/unit/link_git/protocol/take.rs
similarity index 97%
rename from test/src/test/unit/link_git_protocol/take.rs
rename to test/src/test/unit/link_git/protocol/take.rs
index 47edbe0d..79938013 100644
--- a/test/src/test/unit/link_git_protocol/take.rs
+++ b/test/src/test/unit/link_git/protocol/take.rs
@@ -4,7 +4,7 @@
// Linking Exception. For full terms see the included LICENSE file.

use futures::{executor::block_on, io::Cursor, AsyncReadExt as _};
use link_git_protocol::take::TryTake;
use link_git::protocol::take::TryTake;
use std::io;

#[test]
diff --git a/test/src/test/unit/link_git_protocol/upload_pack.rs b/test/src/test/unit/link_git/protocol/upload_pack.rs
similarity index 98%
rename from test/src/test/unit/link_git_protocol/upload_pack.rs
rename to test/src/test/unit/link_git/protocol/upload_pack.rs
index bc805d3a..d3dae61d 100644
--- a/test/src/test/unit/link_git_protocol/upload_pack.rs
+++ b/test/src/test/unit/link_git/protocol/upload_pack.rs
@@ -3,7 +3,7 @@
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use link_git_protocol::upload_pack;
use link_git::protocol::upload_pack;

mod header {
    use super::*;
-- 
2.34.0

[PATCH radicle-link v2 1/3] link-git-protocol: move to link-git crate Export this patch

We make use of only a fraction of the gitoxide suite of crates, and
where we do, the high-level `git-repository` crate doesn't meet our
needs. Thus, establish a `link-git` crate which re-exports just the
things from gitoxide crates deemed useful, and implements the custom
functionality we need.

This is part of a series: as a first step, only move `link-git-protocol`
to `link-git`.

Signed-off-by: Kim Altintop <kim@eagain.st>
---
 Cargo.toml                                    |  2 +-
 git-ext/Cargo.toml                            |  4 +-
 git-ext/src/oid.rs                            | 12 ++--
 librad/Cargo.toml                             |  4 +-
 librad/src/net/protocol/io/recv/git.rs        |  2 +-
 link-git-protocol/Cargo.toml                  | 38 ------------
 link-git/Cargo.toml                           | 62 +++++++++++++++++++
 link-git/src/lib.rs                           | 18 ++++++
 .../src/lib.rs => link-git/src/protocol.rs    |  5 +-
 .../src => link-git/src/protocol}/fetch.rs    | 23 +++----
 .../src => link-git/src/protocol}/ls.rs       | 22 +++----
 .../src/protocol}/packwriter.rs               | 22 +++----
 .../src => link-git/src/protocol}/take.rs     |  0
 .../src/protocol}/transport.rs                |  2 +-
 .../src/protocol}/upload_pack.rs              | 10 ++-
 .../src/protocol}/upload_pack/legacy.rs       |  2 +-
 test/Cargo.toml                               |  4 +-
 test/src/test/integration.rs                  |  2 +-
 test/src/test/integration/link_git.rs         |  6 ++
 .../protocol.rs}                              |  2 +-
 test/src/test/unit.rs                         |  2 +-
 test/src/test/unit/link_git.rs                |  6 ++
 .../protocol.rs}                              |  0
 .../protocol}/take.rs                         |  2 +-
 .../protocol}/upload_pack.rs                  |  2 +-
 25 files changed, 147 insertions(+), 107 deletions(-)
 delete mode 100644 link-git-protocol/Cargo.toml
 create mode 100644 link-git/Cargo.toml
 create mode 100644 link-git/src/lib.rs
 rename link-git-protocol/src/lib.rs => link-git/src/protocol.rs (89%)
 rename {link-git-protocol/src => link-git/src/protocol}/fetch.rs (94%)
 rename {link-git-protocol/src => link-git/src/protocol}/ls.rs (90%)
 rename {link-git-protocol/src => link-git/src/protocol}/packwriter.rs (94%)
 rename {link-git-protocol/src => link-git/src/protocol}/take.rs (100%)
 rename {link-git-protocol/src => link-git/src/protocol}/transport.rs (97%)
 rename {link-git-protocol/src => link-git/src/protocol}/upload_pack.rs (96%)
 rename {link-git-protocol/src => link-git/src/protocol}/upload_pack/legacy.rs (99%)
 create mode 100644 test/src/test/integration/link_git.rs
 rename test/src/test/integration/{link_git_protocol.rs => link_git/protocol.rs} (99%)
 create mode 100644 test/src/test/unit/link_git.rs
 rename test/src/test/unit/{link_git_protocol.rs => link_git/protocol.rs} (100%)
 rename test/src/test/unit/{link_git_protocol => link_git/protocol}/take.rs (97%)
 rename test/src/test/unit/{link_git_protocol => link_git/protocol}/upload_pack.rs (98%)

diff --git a/Cargo.toml b/Cargo.toml
index 31a24f47..2f2b73f0 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -12,7 +12,7 @@ members = [
  "link-canonical",
  "link-canonical-derive",
  "link-crypto",
  "link-git-protocol",
  "link-git",
  "link-identities",
  "macros",
  "node-lib",
diff --git a/git-ext/Cargo.toml b/git-ext/Cargo.toml
index d110549c..79947073 100644
--- a/git-ext/Cargo.toml
+++ b/git-ext/Cargo.toml
@@ -20,8 +20,8 @@ version = ">= 0.13.12, 0.13"
default-features = false
features = []

[dependencies.git-repository]
version = "0.11.0"
[dependencies.link-git]
path = "../link-git"
optional = true

[dependencies.minicbor]
diff --git a/git-ext/src/oid.rs b/git-ext/src/oid.rs
index 0ab5865f..907af143 100644
--- a/git-ext/src/oid.rs
+++ b/git-ext/src/oid.rs
@@ -13,8 +13,8 @@ use std::{
use multihash::{Multihash, MultihashRef};
use thiserror::Error;

#[cfg(feature = "git-repository")]
use git_repository::hash as git_hash;
#[cfg(feature = "link-git")]
use link_git::hash as git_hash;

/// Serializable [`git2::Oid`]
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
@@ -116,7 +116,7 @@ impl AsRef<[u8]> for Oid {
    }
}

#[cfg(feature = "git-repository")]
#[cfg(feature = "link-git")]
impl AsRef<git_hash::oid> for Oid {
    fn as_ref(&self) -> &git_hash::oid {
        // SAFETY: checks the length of the slice, which we know is correct
@@ -136,7 +136,7 @@ impl From<Oid> for git2::Oid {
    }
}

#[cfg(feature = "git-repository")]
#[cfg(feature = "link-git")]
impl From<git_hash::ObjectId> for Oid {
    fn from(git_hash::ObjectId::Sha1(bs): git_hash::ObjectId) -> Self {
        // SAFETY: checks the length of the slice, which we statically know
@@ -144,14 +144,14 @@ impl From<git_hash::ObjectId> for Oid {
    }
}

#[cfg(feature = "git-repository")]
#[cfg(feature = "link-git")]
impl From<Oid> for git_hash::ObjectId {
    fn from(oid: Oid) -> Self {
        Self::from_20_bytes(oid.as_ref())
    }
}

#[cfg(feature = "git-repository")]
#[cfg(feature = "link-git")]
impl<'a> From<&'a Oid> for &'a git_hash::oid {
    fn from(oid: &'a Oid) -> Self {
        oid.as_ref()
diff --git a/librad/Cargo.toml b/librad/Cargo.toml
index c145bcce..b9825dbf 100644
--- a/librad/Cargo.toml
+++ b/librad/Cargo.toml
@@ -95,8 +95,8 @@ path = "../link-canonical"
[dependencies.link-crypto]
path = "../link-crypto"

[dependencies.link-git-protocol]
path = "../link-git-protocol"
[dependencies.link-git]
path = "../link-git"
features = ["git2"]

[dependencies.link-identities]
diff --git a/librad/src/net/protocol/io/recv/git.rs b/librad/src/net/protocol/io/recv/git.rs
index 3666c51a..c7b50b8d 100644
--- a/librad/src/net/protocol/io/recv/git.rs
+++ b/librad/src/net/protocol/io/recv/git.rs
@@ -7,7 +7,7 @@
use std::{io, process::ExitStatus};

use futures::io::{AsyncRead, AsyncWrite};
use link_git_protocol::upload_pack::{upload_pack, Header};
use link_git::protocol::upload_pack::{upload_pack, Header};
use thiserror::Error;
use tracing::{error, info};

diff --git a/link-git-protocol/Cargo.toml b/link-git-protocol/Cargo.toml
deleted file mode 100644
index 76bdc3b3..00000000
--- a/link-git-protocol/Cargo.toml
@@ -1,38 +0,0 @@
[package]
name = "link-git-protocol"
version = "0.1.0"
authors = ["Kim Altintop <kim@eagain.st>"]
edition = "2018"
license = "GPL-3.0-or-later"

description = "radicle-link-flavoured git protocol-v2 client and server"

[lib]
doctest = false
test = false

[dependencies]
async-process = "1.1.0"
async-trait = "0.1"
blocking = "1.0.2"
bstr = "0.2.16"
futures-lite = "1.12.0"
futures-util = "0.3.15"
once_cell = "1.8.0"
pin-project = "1.0.7"
tempfile = "3.2.0"
versions = "3.0.2"

[dependencies.git-repository]
version = "0.11.0"
features = [ "async-network-client", "unstable" ]

[dependencies.git-packetline]
version = "0.12.0"
features = ["async-io"]

[dependencies.git2]
version = "=0.13.20"
default-features = false
features = []
optional = true
diff --git a/link-git/Cargo.toml b/link-git/Cargo.toml
new file mode 100644
index 00000000..7ffe68f8
--- /dev/null
+++ b/link-git/Cargo.toml
@@ -0,0 +1,62 @@
[package]
name = "link-git"
version = "0.1.0"
authors = ["Kim Altintop <kim@eagain.st>"]
edition = "2018"
license = "GPL-3.0-or-later"

description = "Core git types and functionality"

[lib]
doctest = false
test = false

[dependencies]
arc-swap = "1.4.0"
async-process = "1.1.0"
async-trait = "0.1"
blocking = "1.0.2"
bstr = "0.2.16"
futures-lite = "1.12.0"
futures-util = "0.3.15"
im = "15.0.0"
once_cell = "1.8.0"
parking_lot = "0.11.2"
pin-project = "1.0.7"
rustc-hash = "1.1.0"
tempfile = "3.2.0"
thiserror = "1.0.30"
tracing = "0.1"
versions = "3.0.2"

# gitoxide
git-actor = "^0.6.0"
git-hash = "^0.8.0"
git-lock = "^1.0.1"
git-object = "^0.15.1"
git-odb = "^0.23.0"
git-ref = "^0.9.0"
git-traverse = "^0.10.0"

[dependencies.git-features]
version = "^0.17.0"
features = ["progress", "parallel", "zlib-ng-compat"]

[dependencies.git-pack]
version = "^0.13.0"
features = ["object-cache-dynamic", "pack-cache-lru-static", "pack-cache-lru-dynamic"]

[dependencies.git-packetline]
version = "^0.12.0"
features = ["async-io"]

[dependencies.git-protocol]
version = "^0.12.0"
features = ["async-client"]

# compat
[dependencies.git2]
version = "=0.13.20"
default-features = false
features = []
optional = true
diff --git a/link-git/src/lib.rs b/link-git/src/lib.rs
new file mode 100644
index 00000000..7e0cf900
--- /dev/null
+++ b/link-git/src/lib.rs
@@ -0,0 +1,18 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

#![feature(array_map, never_type)]

#[macro_use]
extern crate async_trait;

pub mod protocol;

pub use git_actor as actor;
pub use git_hash as hash;
pub use git_lock as lock;
pub use git_object as object;
pub use git_ref as refs;
pub use git_traverse as traverse;
diff --git a/link-git-protocol/src/lib.rs b/link-git/src/protocol.rs
similarity index 89%
rename from link-git-protocol/src/lib.rs
rename to link-git/src/protocol.rs
index 912e111d..740fc420 100644
--- a/link-git-protocol/src/lib.rs
+++ b/link-git/src/protocol.rs
@@ -3,11 +3,8 @@
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

#[macro_use]
extern crate async_trait;

use bstr::ByteSlice as _;
use git_repository::protocol::transport::client;
use git_protocol::transport::client;
use versions::Version;

pub mod fetch;
diff --git a/link-git-protocol/src/fetch.rs b/link-git/src/protocol/fetch.rs
similarity index 94%
rename from link-git-protocol/src/fetch.rs
rename to link-git/src/protocol/fetch.rs
index ffb34fd6..a5a73065 100644
--- a/link-git-protocol/src/fetch.rs
+++ b/link-git/src/protocol/fetch.rs
@@ -20,22 +20,19 @@ use futures_lite::{
    future,
    io::{AsyncBufRead, AsyncRead, AsyncWrite},
};
use git_repository::{
    progress,
    protocol::{
        self,
        fetch::{response, Action, Arguments, Delegate, DelegateBlocking, LsRefsAction, Response},
        transport::client,
    },
    Progress,
use git_features::progress::{self, Progress};
use git_protocol::{
    fetch::{response, Action, Arguments, Delegate, DelegateBlocking, LsRefsAction, Response},
    transport::client,
};
use once_cell::sync::Lazy;
use pin_project::{pin_project, pinned_drop};
use versions::Version;

pub use git_repository::{hash::ObjectId, protocol::fetch::Ref};
pub use git_hash::ObjectId;
pub use git_protocol::fetch::Ref;

use crate::{packwriter::PackWriter, remote_git_version, transport};
use super::{packwriter::PackWriter, remote_git_version, transport};

// Work around `git-upload-pack` not handling namespaces properly,
//
@@ -136,7 +133,7 @@ impl<P: PackWriter> DelegateBlocking for Fetch<P, P::Output> {

    fn prepare_fetch(
        &mut self,
        _: protocol::transport::Protocol,
        _: git_protocol::transport::Protocol,
        caps: &client::Capabilities,
        _: &mut Vec<(&str, Option<&str>)>,
        _: &[Ref],
@@ -267,12 +264,12 @@ where

        move || {
            let mut delegate = Fetch::new(opt, pack_writer);
            future::block_on(protocol::fetch(
            future::block_on(git_protocol::fetch(
                &mut conn,
                &mut delegate,
                |_| unreachable!("credentials helper requested"),
                progress::Discard,
                protocol::FetchConnection::AllowReuse,
                git_protocol::FetchConnection::AllowReuse,
            ))
            .map_err(|e| io::Error::new(io::ErrorKind::Other, e))?;

diff --git a/link-git-protocol/src/ls.rs b/link-git/src/protocol/ls.rs
similarity index 90%
rename from link-git-protocol/src/ls.rs
rename to link-git/src/protocol/ls.rs
index eb14e1e2..b3516454 100644
--- a/link-git-protocol/src/ls.rs
+++ b/link-git/src/protocol/ls.rs
@@ -7,21 +7,17 @@ use std::io;

use bstr::{BString, ByteVec as _};
use futures_lite::io::{AsyncBufRead, AsyncRead, AsyncWrite};
use git_repository::{
    progress,
    protocol::{
        self,
        fetch::{Action, Arguments, Delegate, DelegateBlocking, LsRefsAction, Response},
        transport::client,
    },
    Progress,
use git_features::progress::{self, Progress};
use git_protocol::{
    fetch::{Action, Arguments, Delegate, DelegateBlocking, LsRefsAction, Response},
    transport::client,
};
use once_cell::sync::Lazy;
use versions::Version;

pub use git_repository::protocol::fetch::Ref;
pub use git_protocol::fetch::Ref;

use crate::{remote_git_version, transport};
use super::{remote_git_version, transport};

// Work around `git-upload-pack` not handling namespaces properly
//
@@ -99,7 +95,7 @@ impl DelegateBlocking for LsRefs {

    fn prepare_fetch(
        &mut self,
        _: protocol::transport::Protocol,
        _: git_protocol::transport::Protocol,
        _: &client::Capabilities,
        _: &mut Vec<(&str, Option<&str>)>,
        refs: &[Ref],
@@ -138,12 +134,12 @@ where
{
    let mut conn = transport::Stateless::new(opt.repo.clone(), recv, send);
    let mut delegate = LsRefs::new(opt);
    git_repository::protocol::fetch(
    git_protocol::fetch(
        &mut conn,
        &mut delegate,
        |_| unreachable!("credentials helper requested"),
        progress::Discard,
        protocol::FetchConnection::AllowReuse,
        git_protocol::FetchConnection::AllowReuse,
    )
    .await
    .map_err(|e| io::Error::new(io::ErrorKind::Other, e))?;
diff --git a/link-git-protocol/src/packwriter.rs b/link-git/src/protocol/packwriter.rs
similarity index 94%
rename from link-git-protocol/src/packwriter.rs
rename to link-git/src/protocol/packwriter.rs
index 5aa4ecba..477feb99 100644
--- a/link-git-protocol/src/packwriter.rs
+++ b/link-git/src/protocol/packwriter.rs
@@ -13,24 +13,21 @@ use std::{
};

use futures_lite::io::{AsyncBufRead, BlockOn};
use git_repository::{
    hash::ObjectId,
    odb::{self, pack},
    Progress,
};
use git_features::progress::Progress;
use git_hash::ObjectId;
use git_odb::{self as odb, pack};

use crate::take::TryTake;
use super::take::TryTake;

#[cfg(feature = "git2")]
pub use libgit::Libgit;

/// What to do with the `packfile` response.
///
/// _This is mostly the same as [`git_repository::protocol::fetch::Delegate`],
/// but without incurring the
/// [`git_repository::protocol::fetch::DelegateBlocking`] super-trait
/// constraint. We can simply make [`crate::fetch::Fetch`] parametric over the
/// packfile sink._
/// _This is mostly the same as [`git_protocol::fetch::Delegate`], but without
/// incurring the [`git_protocol::fetch::DelegateBlocking`] super-trait
/// constraint. We can simply make [`crate::protocol::fetch::Fetch`] parametric
/// over the packfile sink._
pub trait PackWriter {
    type Output;

@@ -159,7 +156,8 @@ impl Thickener for odb::linked::Store {
        id: ObjectId,
        buf: &'a mut Vec<u8>,
    ) -> Option<pack::data::Object<'a>> {
        use git_repository::prelude::FindExt as _;
        use git_odb::FindExt as _;

        self.find(id, buf, &mut pack::cache::Never).ok()
    }
}
diff --git a/link-git-protocol/src/take.rs b/link-git/src/protocol/take.rs
similarity index 100%
rename from link-git-protocol/src/take.rs
rename to link-git/src/protocol/take.rs
diff --git a/link-git-protocol/src/transport.rs b/link-git/src/protocol/transport.rs
similarity index 97%
rename from link-git-protocol/src/transport.rs
rename to link-git/src/protocol/transport.rs
index daa1b771..473f2e5a 100644
--- a/link-git-protocol/src/transport.rs
+++ b/link-git/src/protocol/transport.rs
@@ -5,7 +5,7 @@

use bstr::BString;
use futures_lite::io::{AsyncRead, AsyncWrite};
use git_repository::protocol::transport::{
use git_protocol::transport::{
    client::{
        self,
        git::{ConnectMode, Connection},
diff --git a/link-git-protocol/src/upload_pack.rs b/link-git/src/protocol/upload_pack.rs
similarity index 96%
rename from link-git-protocol/src/upload_pack.rs
rename to link-git/src/protocol/upload_pack.rs
index b0a4e596..64b335cd 100644
--- a/link-git-protocol/src/upload_pack.rs
+++ b/link-git/src/protocol/upload_pack.rs
@@ -8,7 +8,7 @@ use std::{future::Future, io, path::Path, process::ExitStatus, str::FromStr};
use async_process::{Command, Stdio};
use futures_lite::io::{copy, AsyncBufReadExt as _, AsyncRead, AsyncWrite, BufReader};
use futures_util::try_join;
use git_packetline::PacketLineRef;
use git_packetline::{self as packetline, PacketLineRef};
use once_cell::sync::Lazy;
use versions::Version;

@@ -80,7 +80,7 @@ where
            buf.parse().map_err(invalid_data)?
        },
        Some(_) => {
            let mut pktline = git_packetline::StreamingPeekableIter::new(recv, &[]);
            let mut pktline = packetline::StreamingPeekableIter::new(recv, &[]);
            let pkt = pktline
                .read_line()
                .await
@@ -187,8 +187,6 @@ async fn advertise_capabilities<W>(mut send: W) -> io::Result<()>
where
    W: AsyncWrite + Unpin,
{
    use git_packetline::encode;

    // Thou shallt not upgrade your `git` installation while a link instance is
    // running!
    static GIT_VERSION: Lazy<Version> = Lazy::new(|| git_version().unwrap());
@@ -203,9 +201,9 @@ where
    });

    for cap in *CAPABILITIES {
        encode::text_to_write(cap, &mut send).await?;
        packetline::encode::text_to_write(cap, &mut send).await?;
    }
    encode::flush_to_write(&mut send).await?;
    packetline::encode::flush_to_write(&mut send).await?;

    Ok(())
}
diff --git a/link-git-protocol/src/upload_pack/legacy.rs b/link-git/src/protocol/upload_pack/legacy.rs
similarity index 99%
rename from link-git-protocol/src/upload_pack/legacy.rs
rename to link-git/src/protocol/upload_pack/legacy.rs
index 7e1e8de4..30507514 100644
--- a/link-git-protocol/src/upload_pack/legacy.rs
+++ b/link-git/src/protocol/upload_pack/legacy.rs
@@ -9,7 +9,7 @@ use std::{io, path::Path, process::ExitStatus};
use async_process::{Command, Stdio};
use futures_lite::io::{copy, AsyncRead, AsyncReadExt as _, AsyncWrite, AsyncWriteExt as _};
use futures_util::try_join;
use git_repository::refs::{
use git_ref::{
    file::{Store as Refdb, WriteReflog},
    FullName,
    Reference,
diff --git a/test/Cargo.toml b/test/Cargo.toml
index 172f2554..ffb56a93 100644
--- a/test/Cargo.toml
+++ b/test/Cargo.toml
@@ -80,8 +80,8 @@ features = [ "derive" ]
[dependencies.link-identities]
path = "../link-identities"

[dependencies.link-git-protocol]
path = "../link-git-protocol"
[dependencies.link-git]
path = "../link-git"
features = ["git2"]

[dependencies.node-lib]
diff --git a/test/src/test/integration.rs b/test/src/test/integration.rs
index bf6adf8a..cc5dfa9d 100644
--- a/test/src/test/integration.rs
+++ b/test/src/test/integration.rs
@@ -6,7 +6,7 @@
mod daemon;
mod git_helpers;
mod librad;
mod link_git_protocol;
mod link_git;
mod node_lib;
#[cfg(unix)]
mod rad_cli;
diff --git a/test/src/test/integration/link_git.rs b/test/src/test/integration/link_git.rs
new file mode 100644
index 00000000..e388191c
--- /dev/null
+++ b/test/src/test/integration/link_git.rs
@@ -0,0 +1,6 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

mod protocol;
diff --git a/test/src/test/integration/link_git_protocol.rs b/test/src/test/integration/link_git/protocol.rs
similarity index 99%
rename from test/src/test/integration/link_git_protocol.rs
rename to test/src/test/integration/link_git/protocol.rs
index 112e691a..d1df8f1a 100644
--- a/test/src/test/integration/link_git_protocol.rs
+++ b/test/src/test/integration/link_git/protocol.rs
@@ -17,7 +17,7 @@ use git_repository::{
    prelude::*,
    refs::transaction::{Change, PreviousValue, RefEdit},
};
use link_git_protocol::{fetch, ls, packwriter, upload_pack, ObjectId, PackWriter, Ref};
use link_git::protocol::{fetch, ls, packwriter, upload_pack, ObjectId, PackWriter, Ref};
use tempfile::{tempdir, TempDir};

fn upstream() -> TempDir {
diff --git a/test/src/test/unit.rs b/test/src/test/unit.rs
index 0a12952e..d76f819d 100644
--- a/test/src/test/unit.rs
+++ b/test/src/test/unit.rs
@@ -8,7 +8,7 @@ mod git_ext;
mod git_trailers;
mod librad;
mod link_async;
mod link_git_protocol;
mod link_git;
mod node_lib;
mod rad_clib;
mod rad_exe;
diff --git a/test/src/test/unit/link_git.rs b/test/src/test/unit/link_git.rs
new file mode 100644
index 00000000..e388191c
--- /dev/null
+++ b/test/src/test/unit/link_git.rs
@@ -0,0 +1,6 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

mod protocol;
diff --git a/test/src/test/unit/link_git_protocol.rs b/test/src/test/unit/link_git/protocol.rs
similarity index 100%
rename from test/src/test/unit/link_git_protocol.rs
rename to test/src/test/unit/link_git/protocol.rs
diff --git a/test/src/test/unit/link_git_protocol/take.rs b/test/src/test/unit/link_git/protocol/take.rs
similarity index 97%
rename from test/src/test/unit/link_git_protocol/take.rs
rename to test/src/test/unit/link_git/protocol/take.rs
index 47edbe0d..79938013 100644
--- a/test/src/test/unit/link_git_protocol/take.rs
+++ b/test/src/test/unit/link_git/protocol/take.rs
@@ -4,7 +4,7 @@
// Linking Exception. For full terms see the included LICENSE file.

use futures::{executor::block_on, io::Cursor, AsyncReadExt as _};
use link_git_protocol::take::TryTake;
use link_git::protocol::take::TryTake;
use std::io;

#[test]
diff --git a/test/src/test/unit/link_git_protocol/upload_pack.rs b/test/src/test/unit/link_git/protocol/upload_pack.rs
similarity index 98%
rename from test/src/test/unit/link_git_protocol/upload_pack.rs
rename to test/src/test/unit/link_git/protocol/upload_pack.rs
index bc805d3a..d3dae61d 100644
--- a/test/src/test/unit/link_git_protocol/upload_pack.rs
+++ b/test/src/test/unit/link_git/protocol/upload_pack.rs
@@ -3,7 +3,7 @@
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use link_git_protocol::upload_pack;
use link_git::protocol::upload_pack;

mod header {
    use super::*;
-- 
2.34.0

[PATCH radicle-link 2/3] link-git: shareable odb with pack cache Export this patch

gitoxide requires to load all packfiles of a repository upfront, and
does not offer a meaningful way to share them across threads.

Major git implementations agree on the strategy employed here: load
index (.idx) files eagerly, and data (.pack) files only when needed (as
determined by a match in the corresponding index). The number of loaded
pack files is bounded, so as to limit resource consumption.

The implementation proposed here follows JGit, but with major
simplifications. The advantage over eventually fair locking (eg.
`parking_lot::RwLock`) is that readers never block. This is deemed to be
crucial for the use case, because our readers are typically independent,
ie. tend to access disjoint sets of .pack files in an unoptimised repo.
Of course, real-worlds measurements will need to be taken in order to
back this claim up.

Signed-off-by: Kim Altintop <kim@eagain.st>
---

It is still a bit unclear how to best handle compaction (ie. `git repack`).
During replication, we know when a new pack file is added (because we fetched
it), but removal will need rescanning or filesystem events in order to not leak
resources.

 link-git/src/lib.rs                |   1 +
 link-git/src/odb.rs                |  51 ++++++
 link-git/src/odb/backend.rs        |  36 ++++
 link-git/src/odb/index.rs          | 280 +++++++++++++++++++++++++++++
 link-git/src/odb/pack.rs           | 129 +++++++++++++
 link-git/src/odb/window.rs         | 183 +++++++++++++++++++
 link-git/src/odb/window/metrics.rs |  86 +++++++++
 7 files changed, 766 insertions(+)
 create mode 100644 link-git/src/odb.rs
 create mode 100644 link-git/src/odb/backend.rs
 create mode 100644 link-git/src/odb/index.rs
 create mode 100644 link-git/src/odb/pack.rs
 create mode 100644 link-git/src/odb/window.rs
 create mode 100644 link-git/src/odb/window/metrics.rs

diff --git a/link-git/src/lib.rs b/link-git/src/lib.rs
index 7e0cf900..2ed159a0 100644
--- a/link-git/src/lib.rs
+++ b/link-git/src/lib.rs
@@ -8,6 +8,7 @@
#[macro_use]
extern crate async_trait;

pub mod odb;
pub mod protocol;

pub use git_actor as actor;
diff --git a/link-git/src/odb.rs b/link-git/src/odb.rs
new file mode 100644
index 00000000..04ee5067
--- /dev/null
+++ b/link-git/src/odb.rs
@@ -0,0 +1,51 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use git_hash::oid;
use thiserror::Error;

pub mod backend;
pub mod index;
pub mod pack;
pub mod window;

pub use git_pack::{cache, data::Object};

#[derive(Debug, Error)]
pub enum Error {
    #[error(transparent)]
    Packed(#[from] index::error::Lookup<pack::error::Data>),

    #[error(transparent)]
    Loose(#[from] git_odb::loose::find::Error),
}

pub struct Odb<I, D> {
    pub loose: backend::Loose,
    pub packed: backend::Packed<I, D>,
}

impl<I, D> Odb<I, D>
where
    I: index::Index,
    D: window::Cache,
{
    pub fn contains(&self, id: impl AsRef<oid>) -> bool {
        self.packed.contains(id.as_ref()) || self.loose.contains(id)
    }

    pub fn find<'a>(
        &self,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl cache::DecodeEntry,
    ) -> Result<Option<Object<'a>>, Error> {
        let id = id.as_ref();
        if self.packed.contains(id) {
            return self.packed.find(id, buf, cache).map_err(Into::into);
        }
        self.loose.try_find(id, buf).map_err(Into::into)
    }
}
diff --git a/link-git/src/odb/backend.rs b/link-git/src/odb/backend.rs
new file mode 100644
index 00000000..61aefbd3
--- /dev/null
+++ b/link-git/src/odb/backend.rs
@@ -0,0 +1,36 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use git_hash::oid;
use git_pack::{cache::DecodeEntry, data::Object};

use super::{index, pack, window};

pub type Loose = git_odb::loose::Store;

pub struct Packed<I, D> {
    pub index: I,
    pub data: D,
}

impl<I, D> Packed<I, D>
where
    I: index::Index,
    D: window::Cache,
{
    pub fn contains(&self, id: impl AsRef<oid>) -> bool {
        self.index.contains(id)
    }

    pub fn find<'a>(
        &self,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, index::error::Lookup<pack::error::Data>> {
        self.index
            .lookup(|info| self.data.get(info), id, buf, cache)
    }
}
diff --git a/link-git/src/odb/index.rs b/link-git/src/odb/index.rs
new file mode 100644
index 00000000..b733fba7
--- /dev/null
+++ b/link-git/src/odb/index.rs
@@ -0,0 +1,280 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use std::{collections::VecDeque, fs, io, iter::FromIterator, path::Path, sync::Arc};

use arc_swap::ArcSwap;
use git_hash::oid;
use git_pack::{
    cache::DecodeEntry,
    data::{Object, ResolvedBase},
};
use parking_lot::Mutex;
use tracing::trace;

use super::pack;

pub use git_pack::index::File as IndexFile;

pub mod error {
    use super::*;
    use thiserror::Error;

    #[derive(Debug, Error)]
    pub enum Discover {
        #[error(transparent)]
        Index(#[from] pack::error::Index),

        #[error(transparent)]
        Io(#[from] io::Error),
    }

    #[derive(Debug, Error)]
    pub enum Lookup<E> {
        #[error(transparent)]
        Lookup(E),

        #[error(transparent)]
        Decode(#[from] git_pack::data::decode_entry::Error),
    }
}

pub trait Index {
    fn contains(&self, id: impl AsRef<oid>) -> bool;

    fn lookup<'a, F, E>(
        &self,
        pack_cache: F,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, error::Lookup<E>>
    where
        F: FnOnce(&pack::Info) -> Result<Arc<pack::Data>, E>;
}

/// Attempt to load all pack index files from the provided `GIT_DIR`.
///
/// The returned [`Vec`] is sorted by the file modification time (earlier
/// first).
pub fn discover(git_dir: impl AsRef<Path>) -> Result<Vec<pack::Index>, error::Discover> {
    let pack_dir = git_dir.as_ref().join("objects").join("pack");

    let mut paths = Vec::new();
    trace!("discovering packs at {}", pack_dir.display());
    for entry in fs::read_dir(&pack_dir)? {
        let entry = entry?;
        let path = entry.path();
        trace!("{}", path.display());
        let meta = entry.metadata()?;
        if meta.file_type().is_file() && path.extension().unwrap_or_default() == "idx" {
            let mtime = meta.modified()?;
            paths.push((path, mtime));
        }
    }
    paths.sort_by(|(_, mtime_a), (_, mtime_b)| mtime_a.cmp(mtime_b));

    let indices = paths
        .into_iter()
        .map(|(path, _)| Ok(pack::Index::open(path)?))
        .collect::<Result<_, error::Discover>>()?;

    Ok(indices)
}

/// An [`Index`] which can be shared between threads.
///
/// Writes are guarded by a [`Mutex`], while reads are lock-free (and mostly
/// wait-free). [`Shared`] does not automatically detect changes on the
/// filesystem.
///
/// Lookup methods traverse the set of indices in reverse order, so the iterator
/// to construct a [`Shared`] from via its [`FromIterator`] impl should yield
/// elements an appropriate order. Usually, more recently created indices are
/// more likely to be accessed than older ones.
pub struct Shared {
    indices: ArcSwap<im::Vector<Arc<pack::Index>>>,
    write: Mutex<()>,
}

impl FromIterator<pack::Index> for Shared {
    fn from_iter<T>(iter: T) -> Self
    where
        T: IntoIterator<Item = pack::Index>,
    {
        Self {
            indices: ArcSwap::new(Arc::new(iter.into_iter().map(Arc::new).collect())),
            write: Mutex::new(()),
        }
    }
}

impl Shared {
    /// Add a newly discovered [`pack::Index`].
    ///
    /// This index will be considered first by subsequent lookups.
    pub fn push(&self, idx: pack::Index) {
        let lock = self.write.lock();
        let mut new = self.indices.load_full();
        Arc::make_mut(&mut new).push_back(Arc::new(idx));
        self.indices.store(new);
        drop(lock)
    }

    pub fn remove(&self, info: &pack::Info) {
        let lock = self.write.lock();
        let mut new = self.indices.load_full();
        Arc::make_mut(&mut new).retain(|idx| &idx.info != info);
        self.indices.store(new);
        drop(lock)
    }

    pub fn is_empty(&self) -> bool {
        self.indices.load().is_empty()
    }

    pub fn len(&self) -> usize {
        self.indices.load().len()
    }

    pub fn contains(&self, id: impl AsRef<oid>) -> bool {
        for idx in self.indices.load().iter().rev() {
            if idx.contains(&id) {
                return true;
            }
        }

        false
    }

    pub fn lookup<'a, F, E>(
        &self,
        pack_cache: F,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, error::Lookup<E>>
    where
        F: FnOnce(&pack::Info) -> Result<Arc<pack::Data>, E>,
    {
        for idx in self.indices.load().iter().rev() {
            if let Some(ofs) = idx.ofs(&id) {
                let data = pack_cache(&idx.info).map_err(error::Lookup::Lookup)?;
                let pack = data.file();
                let entry = pack.entry(ofs);
                let obj = pack
                    .decode_entry(
                        entry,
                        buf,
                        |id, _| idx.ofs(id).map(|ofs| ResolvedBase::InPack(pack.entry(ofs))),
                        cache,
                    )
                    .map(move |out| Object {
                        kind: out.kind,
                        data: buf.as_slice(),
                        pack_location: None,
                    })?;

                return Ok(Some(obj));
            }
        }

        Ok(None)
    }
}

impl Index for Shared {
    fn contains(&self, id: impl AsRef<oid>) -> bool {
        self.contains(id)
    }

    fn lookup<'a, F, E>(
        &self,
        pack_cache: F,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, error::Lookup<E>>
    where
        F: FnOnce(&pack::Info) -> Result<Arc<pack::Data>, E>,
    {
        self.lookup(pack_cache, id, buf, cache)
    }
}

/// A simple [`Index`] which can not be modified concurrently.
///
/// Lookup functions traverse the inner [`VecDeque`] in reverse order, so
/// indices which are more likely to contain the requested object should be
/// placed at the end of the [`VecDeque`].
pub struct Static {
    pub indices: VecDeque<pack::Index>,
}

impl Static {
    pub fn contains(&self, id: impl AsRef<oid>) -> bool {
        for idx in self.indices.iter().rev() {
            if idx.contains(&id) {
                return true;
            }
        }

        false
    }

    pub fn lookup<'a, F, E>(
        &self,
        pack_cache: F,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, error::Lookup<E>>
    where
        F: FnOnce(&pack::Info) -> Result<Arc<pack::Data>, E>,
    {
        for idx in self.indices.iter().rev() {
            if let Some(ofs) = idx.ofs(&id) {
                let data = pack_cache(&idx.info).map_err(error::Lookup::Lookup)?;
                let pack = data.file();
                let entry = pack.entry(ofs);
                let obj = pack
                    .decode_entry(
                        entry,
                        buf,
                        |id, _| idx.ofs(id).map(|ofs| ResolvedBase::InPack(pack.entry(ofs))),
                        cache,
                    )
                    .map(move |out| Object {
                        kind: out.kind,
                        data: buf.as_slice(),
                        pack_location: None,
                    })?;

                return Ok(Some(obj));
            }
        }

        Ok(None)
    }
}

impl Index for Static {
    fn contains(&self, id: impl AsRef<oid>) -> bool {
        self.contains(id)
    }

    fn lookup<'a, F, E>(
        &self,
        pack_cache: F,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, error::Lookup<E>>
    where
        F: FnOnce(&pack::Info) -> Result<Arc<pack::Data>, E>,
    {
        self.lookup(pack_cache, id, buf, cache)
    }
}
diff --git a/link-git/src/odb/pack.rs b/link-git/src/odb/pack.rs
new file mode 100644
index 00000000..609e5c4c
--- /dev/null
+++ b/link-git/src/odb/pack.rs
@@ -0,0 +1,129 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use std::{
    path::{Path, PathBuf},
    sync::atomic::{AtomicUsize, Ordering},
};

use git_hash::{oid, ObjectId};
use git_pack::{data, index};
use tracing::warn;

pub mod error {
    use super::*;
    use thiserror::Error;

    #[derive(Debug, Error)]
    #[error("failed to load pack data from {path:?}")]
    pub struct Data {
        pub path: PathBuf,
        pub source: data::header::decode::Error,
    }

    #[derive(Debug, Error)]
    #[error("failed to load pack index from {path:?}")]
    pub struct Index {
        pub path: PathBuf,
        pub source: index::init::Error,
    }
}

pub struct Data {
    pub hash: usize,
    hits: AtomicUsize,
    file: data::File,
}

impl Data {
    pub fn hit(&self) {
        self.hits.fetch_add(1, Ordering::Relaxed);
    }

    pub fn hits(&self) -> usize {
        self.hits.load(Ordering::Relaxed)
    }

    pub fn file(&self) -> &data::File {
        &self.file
    }
}

impl AsRef<data::File> for Data {
    fn as_ref(&self) -> &data::File {
        self.file()
    }
}

#[derive(Clone, PartialEq)]
pub struct Info {
    pub(super) hash: usize,
    pub data_path: PathBuf,
}

impl Info {
    pub fn data(&self) -> Result<Data, error::Data> {
        let file = data::File::at(&self.data_path).map_err(|source| error::Data {
            path: self.data_path.clone(),
            source,
        })?;
        Ok(Data {
            hash: self.hash,
            hits: AtomicUsize::new(0),
            file,
        })
    }
}

pub struct Index {
    pub info: Info,
    file: index::File,
}

impl Index {
    pub fn open(path: impl AsRef<Path>) -> Result<Self, error::Index> {
        let path = path.as_ref();
        let file = index::File::at(path).map_err(|source| error::Index {
            path: path.to_path_buf(),
            source,
        })?;
        let data_path = path.with_extension("pack");
        let hash = {
            let file_name = path
                .file_name()
                .expect("must have a file name, we opened it")
                .to_string_lossy();
            // XXX: inexplicably, gitoxide omits the "pack-" prefix
            let sha_hex = file_name.strip_prefix("pack-").unwrap_or(&file_name);
            match ObjectId::from_hex(&sha_hex.as_bytes()[..40]) {
                Err(e) => {
                    warn!(
                        "unconventional pack name {:?}, falling back to fxhash: {}",
                        path, e
                    );
                    fxhash::hash(path)
                },
                Ok(oid) => {
                    let mut buf = [0u8; 8];
                    buf.copy_from_slice(&oid.sha1()[..8]);
                    usize::from_be_bytes(buf)
                },
            }
        };
        let info = Info { hash, data_path };

        Ok(Self { file, info })
    }

    pub fn contains(&self, id: impl AsRef<oid>) -> bool {
        self.file.lookup(id).is_some()
    }

    pub fn ofs(&self, id: impl AsRef<oid>) -> Option<u64> {
        self.file
            .lookup(id)
            .map(|idx| self.file.pack_offset_at_index(idx))
    }
}
diff --git a/link-git/src/odb/window.rs b/link-git/src/odb/window.rs
new file mode 100644
index 00000000..19b9f431
--- /dev/null
+++ b/link-git/src/odb/window.rs
@@ -0,0 +1,183 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use std::{marker::PhantomData, sync::Arc};

use arc_swap::{ArcSwap, Guard};
use parking_lot::Mutex;

use super::pack;

mod metrics;
pub use metrics::{Metrics, Stats, StatsView, Void};

/// A threadsafe, shareable cache of packfiles.
pub trait Cache {
    type Stats;

    fn stats(&self) -> Self::Stats;

    fn get(&self, info: &pack::Info) -> Result<Arc<pack::Data>, pack::error::Data>;
}

impl<M, const B: usize, const S: usize> Cache for Fixed<M, B, S>
where
    M: Metrics,
{
    type Stats = M::Snapshot;

    fn stats(&self) -> Self::Stats {
        self.stats()
    }

    fn get(&self, info: &pack::Info) -> Result<Arc<pack::Data>, pack::error::Data> {
        self.get(info)
    }
}

/// 128 open files
pub type Small<S> = Fixed<S, 16, 8>;
/// 512 open files
pub type Medium<S> = Fixed<S, 32, 16>;
/// 1024 open files
pub type Large<S> = Fixed<S, 64, 16>;
/// 2048 open files
pub type XLarge<S> = Fixed<S, 128, 16>;

/// A fixed-size [`Cache`].
///
/// [`Fixed`] is essentially a very simple, fixed-capacity hashtable. When a
/// pack (data-) file is requested via [`Cache::get`], the file is loaded
/// (typically `mmap`ed) from disk if it is not already in the cache. Otherwise,
/// a pointer to the already loaded file is returned. Old entries are replaced
/// on an approximate LRU basis when the cache becomes full (this means that old
/// entries are **not** evicted when there is still space).
///
/// The implementation is a somewhat dumbed-down version of JGit's
/// `WindowCache`. The main differences are that the table buckets are of fixed
/// size (`SLOTS`), instead of a linked list. This means that the cache does not
/// allow to (temporarily) commit more entries than its nominal capacity.
///
/// Reading cached values is lock-free and mostly wait-free. Modifications are
/// guarded by locks on individual buckets; if a cache miss occurs, multiple
/// threads requesting the same entry will be blocked until one of them
/// succeeded loading the data and updating the cache. Writers will _not_,
/// however, contend with readers (unlike `RwLock`).
///
/// This favours usage patterns where different threads tend to request disjoint
/// sets of packfiles, and of course their hashes colliding relatively
/// infrequently.
pub struct Fixed<M, const BUCKETS: usize, const SLOTS: usize> {
    entries: [ArcSwap<[Option<Arc<pack::Data>>; SLOTS]>; BUCKETS],
    locks: [Mutex<()>; BUCKETS],
    stats: M,
}

trait AssertSendSync: Send + Sync {}
impl<M, const B: usize, const S: usize> AssertSendSync for Fixed<M, B, S> where M: Send + Sync {}

impl<M, const B: usize, const S: usize> AsRef<Fixed<M, B, S>> for Fixed<M, B, S> {
    fn as_ref(&self) -> &Fixed<M, B, S> {
        self
    }
}

impl<const B: usize, const S: usize> Default for Fixed<Void, B, S> {
    fn default() -> Self {
        Self {
            entries: [(); B].map(|_| ArcSwap::new(Arc::new([(); S].map(|_| None)))),
            locks: [(); B].map(|_| Mutex::new(())),
            stats: PhantomData,
        }
    }
}

impl<M, const B: usize, const S: usize> Fixed<M, B, S>
where
    M: Metrics,
{
    pub fn with_stats(self) -> Fixed<Stats, B, S> {
        self.with_metrics(Stats::default())
    }

    pub fn with_metrics<N: Metrics>(self, m: N) -> Fixed<N, B, S> {
        Fixed {
            entries: self.entries,
            locks: self.locks,
            stats: m,
        }
    }

    pub fn stats(&self) -> M::Snapshot {
        let open_files = self
            .entries
            .iter()
            .map(|bucket| bucket.load().iter().flatten().count())
            .sum();
        self.stats.snapshot(open_files)
    }

    pub fn get(&self, info: &pack::Info) -> Result<Arc<pack::Data>, pack::error::Data> {
        let idx = info.hash % self.entries.len();

        let bucket = self.entries[idx].load();
        for entry in bucket.iter().flatten() {
            if entry.hash == info.hash {
                self.stats.record_hit();
                entry.hit();
                return Ok(Arc::clone(entry));
            }
        }
        drop(bucket);

        self.stats.record_miss();

        // Cache miss, try to load the data file
        let lock = self.locks[idx].lock();
        // Did someone else win the race for the lock?
        let bucket = self.entries[idx].load();
        for entry in bucket.iter().flatten() {
            if entry.hash == info.hash {
                self.stats.record_hit();
                entry.hit();
                return Ok(Arc::clone(entry));
            }
        }
        // No, proceed
        self.stats.record_load();
        let data = Arc::new(info.data()?);

        // Find an empty slot, or swap with the least popular
        let mut access = usize::MAX;
        let mut evict = 0;
        for (i, e) in bucket.iter().enumerate() {
            match e {
                Some(entry) => {
                    let hits = entry.hits();
                    if hits < access {
                        access = hits;
                        evict = i;
                    }
                },
                None => {
                    evict = i;
                    break;
                },
            }
        }
        let mut entries = Guard::into_inner(bucket);
        {
            // This costs `SLOTS` refcount increments if the slot is currently
            // borrowed.
            let mutti = Arc::make_mut(&mut entries);
            mutti[evict] = Some(Arc::clone(&data));
        }
        self.entries[idx].store(entries);
        drop(lock);

        data.hit();
        Ok(data)
    }
}
diff --git a/link-git/src/odb/window/metrics.rs b/link-git/src/odb/window/metrics.rs
new file mode 100644
index 00000000..874f2912
--- /dev/null
+++ b/link-git/src/odb/window/metrics.rs
@@ -0,0 +1,86 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use std::{
    marker::PhantomData,
    sync::atomic::{AtomicUsize, Ordering},
};

use tracing::trace;

pub struct StatsView {
    /// Total number of times the requested data was found in the cache.
    pub cache_hits: usize,
    /// Total number of times the requested data was not found in the cache.
    ///
    /// Note that a cache hit can occur after a miss if another thread was
    /// faster to fill in the missing entry. Thus, `cache_hits + cache_misses`
    /// does not necessarily sum up to the number of cache accesses.
    pub cache_misses: usize,
    /// Total number of times a pack file was attempted to be loaded from disk
    /// (incl. failed attempts).
    pub file_loads: usize,
    /// Total number of pack files the cache holds on to.
    pub open_files: usize,
}

#[derive(Default)]
pub struct Stats {
    hits: AtomicUsize,
    miss: AtomicUsize,
    load: AtomicUsize,
}

pub type Void = PhantomData<!>;

pub trait Metrics {
    type Snapshot;

    fn record_hit(&self);
    fn record_miss(&self);
    fn record_load(&self);

    fn snapshot(&self, open_files: usize) -> Self::Snapshot;
}

impl Metrics for Stats {
    type Snapshot = StatsView;

    fn record_hit(&self) {
        trace!("cache hit");
        self.hits.fetch_add(1, Ordering::Relaxed);
    }

    fn record_miss(&self) {
        trace!("cache miss");
        self.miss.fetch_add(1, Ordering::Relaxed);
    }

    fn record_load(&self) {
        trace!("pack load");
        self.load.fetch_add(1, Ordering::Relaxed);
    }

    fn snapshot(&self, open_files: usize) -> Self::Snapshot {
        StatsView {
            cache_hits: self.hits.load(Ordering::Relaxed),
            cache_misses: self.miss.load(Ordering::Relaxed),
            file_loads: self.load.load(Ordering::Relaxed),
            open_files,
        }
    }
}

impl Metrics for Void {
    type Snapshot = usize;

    fn record_hit(&self) {}
    fn record_miss(&self) {}
    fn record_load(&self) {}

    fn snapshot(&self, open_files: usize) -> Self::Snapshot {
        open_files
    }
}
--
2.34.0

[PATCH radicle-link v2 2/3] link-git: shareable odb with pack cache Export this patch

gitoxide requires to load all packfiles of a repository upfront, and
does not offer a meaningful way to share them across threads.

Major git implementations agree on the strategy employed here: load
index (.idx) files eagerly, and data (.pack) files only when needed (as
determined by a match in the corresponding index). The number of loaded
pack files is bounded, so as to limit resource consumption.

The implementation proposed here follows JGit, but with major
simplifications. The advantage over eventually fair locking (eg.
`parking_lot::RwLock`) is that readers never block. This is deemed to be
crucial for the use case, because our readers are typically independent,
ie. tend to access disjoint sets of .pack files in an unoptimised repo.
Of course, real-worlds measurements will need to be taken in order to
back this claim up.

Signed-off-by: Kim Altintop <kim@eagain.st>
---
 link-git/src/lib.rs                |   1 +
 link-git/src/odb.rs                |  51 +++++
 link-git/src/odb/backend.rs        |  36 ++++
 link-git/src/odb/index.rs          | 296 +++++++++++++++++++++++++++++
 link-git/src/odb/pack.rs           | 138 ++++++++++++++
 link-git/src/odb/window.rs         | 183 ++++++++++++++++++
 link-git/src/odb/window/metrics.rs |  86 +++++++++
 7 files changed, 791 insertions(+)
 create mode 100644 link-git/src/odb.rs
 create mode 100644 link-git/src/odb/backend.rs
 create mode 100644 link-git/src/odb/index.rs
 create mode 100644 link-git/src/odb/pack.rs
 create mode 100644 link-git/src/odb/window.rs
 create mode 100644 link-git/src/odb/window/metrics.rs

diff --git a/link-git/src/lib.rs b/link-git/src/lib.rs
index 7e0cf900..2ed159a0 100644
--- a/link-git/src/lib.rs
+++ b/link-git/src/lib.rs
@@ -8,6 +8,7 @@
#[macro_use]
extern crate async_trait;

pub mod odb;
pub mod protocol;

pub use git_actor as actor;
diff --git a/link-git/src/odb.rs b/link-git/src/odb.rs
new file mode 100644
index 00000000..04ee5067
--- /dev/null
+++ b/link-git/src/odb.rs
@@ -0,0 +1,51 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use git_hash::oid;
use thiserror::Error;

pub mod backend;
pub mod index;
pub mod pack;
pub mod window;

pub use git_pack::{cache, data::Object};

#[derive(Debug, Error)]
pub enum Error {
    #[error(transparent)]
    Packed(#[from] index::error::Lookup<pack::error::Data>),

    #[error(transparent)]
    Loose(#[from] git_odb::loose::find::Error),
}

pub struct Odb<I, D> {
    pub loose: backend::Loose,
    pub packed: backend::Packed<I, D>,
}

impl<I, D> Odb<I, D>
where
    I: index::Index,
    D: window::Cache,
{
    pub fn contains(&self, id: impl AsRef<oid>) -> bool {
        self.packed.contains(id.as_ref()) || self.loose.contains(id)
    }

    pub fn find<'a>(
        &self,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl cache::DecodeEntry,
    ) -> Result<Option<Object<'a>>, Error> {
        let id = id.as_ref();
        if self.packed.contains(id) {
            return self.packed.find(id, buf, cache).map_err(Into::into);
        }
        self.loose.try_find(id, buf).map_err(Into::into)
    }
}
diff --git a/link-git/src/odb/backend.rs b/link-git/src/odb/backend.rs
new file mode 100644
index 00000000..61aefbd3
--- /dev/null
+++ b/link-git/src/odb/backend.rs
@@ -0,0 +1,36 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use git_hash::oid;
use git_pack::{cache::DecodeEntry, data::Object};

use super::{index, pack, window};

pub type Loose = git_odb::loose::Store;

pub struct Packed<I, D> {
    pub index: I,
    pub data: D,
}

impl<I, D> Packed<I, D>
where
    I: index::Index,
    D: window::Cache,
{
    pub fn contains(&self, id: impl AsRef<oid>) -> bool {
        self.index.contains(id)
    }

    pub fn find<'a>(
        &self,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, index::error::Lookup<pack::error::Data>> {
        self.index
            .lookup(|info| self.data.get(info), id, buf, cache)
    }
}
diff --git a/link-git/src/odb/index.rs b/link-git/src/odb/index.rs
new file mode 100644
index 00000000..8f193dab
--- /dev/null
+++ b/link-git/src/odb/index.rs
@@ -0,0 +1,296 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use std::{collections::VecDeque, fs, io, iter::FromIterator, path::Path, sync::Arc};

use arc_swap::ArcSwap;
use git_hash::oid;
use git_pack::{
    cache::DecodeEntry,
    data::{Object, ResolvedBase},
};
use parking_lot::Mutex;
use tracing::trace;

use super::pack;

pub use git_pack::index::File as IndexFile;

pub mod error {
    use super::*;
    use thiserror::Error;

    #[derive(Debug, Error)]
    pub enum Discover {
        #[error(transparent)]
        Index(#[from] pack::error::Index),

        #[error(transparent)]
        Io(#[from] io::Error),
    }

    #[derive(Debug, Error)]
    pub enum Lookup<E> {
        #[error(transparent)]
        Lookup(E),

        #[error(transparent)]
        Decode(#[from] git_pack::data::decode_entry::Error),
    }
}

pub trait Index {
    fn contains(&self, id: impl AsRef<oid>) -> bool;

    fn lookup<'a, F, E>(
        &self,
        pack_cache: F,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, error::Lookup<E>>
    where
        F: FnOnce(&pack::Info) -> Result<Arc<pack::Data>, E>;
}

/// Attempt to load all pack index files from the provided `GIT_DIR`.
///
/// The returned [`Vec`] is sorted by the file modification time (earlier
/// first).
pub fn discover(git_dir: impl AsRef<Path>) -> Result<Vec<pack::Index>, error::Discover> {
    let pack_dir = git_dir.as_ref().join("objects").join("pack");

    let mut paths = Vec::new();
    trace!("discovering packs at {}", pack_dir.display());
    for entry in fs::read_dir(&pack_dir)? {
        let entry = entry?;
        let path = entry.path();
        trace!("{}", path.display());
        let meta = entry.metadata()?;
        if meta.file_type().is_file() && path.extension().unwrap_or_default() == "idx" {
            let mtime = meta.modified()?;
            paths.push((path, mtime));
        }
    }
    paths.sort_by(|(_, mtime_a), (_, mtime_b)| mtime_a.cmp(mtime_b));

    let indices = paths
        .into_iter()
        .map(|(path, _)| Ok(pack::Index::open(path)?))
        .collect::<Result<_, error::Discover>>()?;

    Ok(indices)
}

/// An [`Index`] which can be shared between threads.
///
/// Writes are guarded by a [`Mutex`], while reads are lock-free (and mostly
/// wait-free). [`Shared`] does not automatically detect changes on the
/// filesystem.
///
/// Lookup methods traverse the set of indices in reverse order, so the iterator
/// to construct a [`Shared`] from via its [`FromIterator`] impl should yield
/// elements an appropriate order. Usually, more recently created indices are
/// more likely to be accessed than older ones.
pub struct Shared {
    indices: ArcSwap<im::Vector<Arc<pack::Index>>>,
    write: Mutex<()>,
}

impl FromIterator<pack::Index> for Shared {
    fn from_iter<T>(iter: T) -> Self
    where
        T: IntoIterator<Item = pack::Index>,
    {
        Self {
            indices: ArcSwap::new(Arc::new(iter.into_iter().map(Arc::new).collect())),
            write: Mutex::new(()),
        }
    }
}

impl Shared {
    /// Add a newly discovered [`pack::Index`].
    ///
    /// This index will be considered first by subsequent lookups.
    pub fn push(&self, idx: pack::Index) {
        let lock = self.write.lock();
        let mut new = self.indices.load_full();
        Arc::make_mut(&mut new).push_back(Arc::new(idx));
        self.indices.store(new);
        drop(lock)
    }

    pub fn remove(&self, info: &pack::Info) {
        let lock = self.write.lock();
        let mut new = self.indices.load_full();
        Arc::make_mut(&mut new).retain(|idx| &idx.info != info);
        self.indices.store(new);
        drop(lock)
    }

    pub fn clear(&self) {
        let lock = self.write.lock();
        self.indices.store(Arc::new(im::Vector::new()));
        drop(lock)
    }

    pub fn replace<T>(&self, iter: T)
    where
        T: IntoIterator<Item = pack::Index>,
    {
        let lock = self.write.lock();
        self.indices
            .store(Arc::new(iter.into_iter().map(Arc::new).collect()));
        drop(lock)
    }

    pub fn is_empty(&self) -> bool {
        self.indices.load().is_empty()
    }

    pub fn len(&self) -> usize {
        self.indices.load().len()
    }

    pub fn contains(&self, id: impl AsRef<oid>) -> bool {
        for idx in self.indices.load().iter().rev() {
            if idx.contains(&id) {
                return true;
            }
        }

        false
    }

    pub fn lookup<'a, F, E>(
        &self,
        pack_cache: F,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, error::Lookup<E>>
    where
        F: FnOnce(&pack::Info) -> Result<Arc<pack::Data>, E>,
    {
        for idx in self.indices.load().iter().rev() {
            if let Some(ofs) = idx.ofs(&id) {
                let data = pack_cache(&idx.info).map_err(error::Lookup::Lookup)?;
                let pack = data.file();
                let entry = pack.entry(ofs);
                let obj = pack
                    .decode_entry(
                        entry,
                        buf,
                        |id, _| idx.ofs(id).map(|ofs| ResolvedBase::InPack(pack.entry(ofs))),
                        cache,
                    )
                    .map(move |out| Object {
                        kind: out.kind,
                        data: buf.as_slice(),
                        pack_location: None,
                    })?;

                return Ok(Some(obj));
            }
        }

        Ok(None)
    }
}

impl Index for Shared {
    fn contains(&self, id: impl AsRef<oid>) -> bool {
        self.contains(id)
    }

    fn lookup<'a, F, E>(
        &self,
        pack_cache: F,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, error::Lookup<E>>
    where
        F: FnOnce(&pack::Info) -> Result<Arc<pack::Data>, E>,
    {
        self.lookup(pack_cache, id, buf, cache)
    }
}

/// A simple [`Index`] which can not be modified concurrently.
///
/// Lookup functions traverse the inner [`VecDeque`] in reverse order, so
/// indices which are more likely to contain the requested object should be
/// placed at the end of the [`VecDeque`].
pub struct Static {
    pub indices: VecDeque<pack::Index>,
}

impl Static {
    pub fn contains(&self, id: impl AsRef<oid>) -> bool {
        for idx in self.indices.iter().rev() {
            if idx.contains(&id) {
                return true;
            }
        }

        false
    }

    pub fn lookup<'a, F, E>(
        &self,
        pack_cache: F,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, error::Lookup<E>>
    where
        F: FnOnce(&pack::Info) -> Result<Arc<pack::Data>, E>,
    {
        for idx in self.indices.iter().rev() {
            if let Some(ofs) = idx.ofs(&id) {
                let data = pack_cache(&idx.info).map_err(error::Lookup::Lookup)?;
                let pack = data.file();
                let entry = pack.entry(ofs);
                let obj = pack
                    .decode_entry(
                        entry,
                        buf,
                        |id, _| idx.ofs(id).map(|ofs| ResolvedBase::InPack(pack.entry(ofs))),
                        cache,
                    )
                    .map(move |out| Object {
                        kind: out.kind,
                        data: buf.as_slice(),
                        pack_location: None,
                    })?;

                return Ok(Some(obj));
            }
        }

        Ok(None)
    }
}

impl Index for Static {
    fn contains(&self, id: impl AsRef<oid>) -> bool {
        self.contains(id)
    }

    fn lookup<'a, F, E>(
        &self,
        pack_cache: F,
        id: impl AsRef<oid>,
        buf: &'a mut Vec<u8>,
        cache: &mut impl DecodeEntry,
    ) -> Result<Option<Object<'a>>, error::Lookup<E>>
    where
        F: FnOnce(&pack::Info) -> Result<Arc<pack::Data>, E>,
    {
        self.lookup(pack_cache, id, buf, cache)
    }
}
diff --git a/link-git/src/odb/pack.rs b/link-git/src/odb/pack.rs
new file mode 100644
index 00000000..97155a14
--- /dev/null
+++ b/link-git/src/odb/pack.rs
@@ -0,0 +1,138 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use std::{
    path::{Path, PathBuf},
    sync::atomic::{AtomicUsize, Ordering},
};

use git_hash::{oid, ObjectId};
use git_pack::{data, index};
use rustc_hash::FxHasher;
use tracing::warn;

pub mod error {
    use super::*;
    use thiserror::Error;

    #[derive(Debug, Error)]
    #[error("failed to load pack data from {path:?}")]
    pub struct Data {
        pub path: PathBuf,
        pub source: data::header::decode::Error,
    }

    #[derive(Debug, Error)]
    #[error("failed to load pack index from {path:?}")]
    pub struct Index {
        pub path: PathBuf,
        pub source: index::init::Error,
    }
}

pub struct Data {
    pub hash: u64,
    hits: AtomicUsize,
    file: data::File,
}

impl Data {
    pub fn hit(&self) {
        self.hits.fetch_add(1, Ordering::Relaxed);
    }

    pub fn hits(&self) -> usize {
        self.hits.load(Ordering::Relaxed)
    }

    pub fn file(&self) -> &data::File {
        &self.file
    }
}

impl AsRef<data::File> for Data {
    fn as_ref(&self) -> &data::File {
        self.file()
    }
}

#[derive(Clone, PartialEq)]
pub struct Info {
    pub(super) hash: u64,
    pub data_path: PathBuf,
}

impl Info {
    pub fn data(&self) -> Result<Data, error::Data> {
        let file = data::File::at(&self.data_path).map_err(|source| error::Data {
            path: self.data_path.clone(),
            source,
        })?;
        Ok(Data {
            hash: self.hash,
            hits: AtomicUsize::new(0),
            file,
        })
    }
}

pub struct Index {
    pub info: Info,
    file: index::File,
}

impl Index {
    pub fn open(path: impl AsRef<Path>) -> Result<Self, error::Index> {
        let path = path.as_ref();
        let file = index::File::at(path).map_err(|source| error::Index {
            path: path.to_path_buf(),
            source,
        })?;
        let data_path = path.with_extension("pack");
        let hash = {
            let file_name = path
                .file_name()
                .expect("must have a file name, we opened it")
                .to_string_lossy();
            // XXX: inexplicably, gitoxide omits the "pack-" prefix
            let sha_hex = file_name.strip_prefix("pack-").unwrap_or(&file_name);
            match ObjectId::from_hex(&sha_hex.as_bytes()[..40]) {
                Err(e) => {
                    warn!(
                        "unconventional pack name {:?}, falling back to fxhash: {}",
                        path, e
                    );
                    hash(path)
                },
                Ok(oid) => {
                    let mut buf = [0u8; 8];
                    buf.copy_from_slice(&oid.sha1()[..8]);
                    u64::from_be_bytes(buf)
                },
            }
        };
        let info = Info { hash, data_path };

        Ok(Self { file, info })
    }

    pub fn contains(&self, id: impl AsRef<oid>) -> bool {
        self.file.lookup(id).is_some()
    }

    pub fn ofs(&self, id: impl AsRef<oid>) -> Option<u64> {
        self.file
            .lookup(id)
            .map(|idx| self.file.pack_offset_at_index(idx))
    }
}

fn hash(p: &Path) -> u64 {
    use std::hash::{Hash as _, Hasher as _};

    let mut hasher = FxHasher::default();
    p.hash(&mut hasher);
    hasher.finish()
}
diff --git a/link-git/src/odb/window.rs b/link-git/src/odb/window.rs
new file mode 100644
index 00000000..48f768af
--- /dev/null
+++ b/link-git/src/odb/window.rs
@@ -0,0 +1,183 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use std::{marker::PhantomData, sync::Arc};

use arc_swap::{ArcSwap, Guard};
use parking_lot::Mutex;

use super::pack;

mod metrics;
pub use metrics::{Metrics, Stats, StatsView, Void};

/// A threadsafe, shareable cache of packfiles.
pub trait Cache {
    type Stats;

    fn stats(&self) -> Self::Stats;

    fn get(&self, info: &pack::Info) -> Result<Arc<pack::Data>, pack::error::Data>;
}

impl<M, const B: usize, const S: usize> Cache for Fixed<M, B, S>
where
    M: Metrics,
{
    type Stats = M::Snapshot;

    fn stats(&self) -> Self::Stats {
        self.stats()
    }

    fn get(&self, info: &pack::Info) -> Result<Arc<pack::Data>, pack::error::Data> {
        self.get(info)
    }
}

/// 128 open files
pub type Small<S> = Fixed<S, 16, 8>;
/// 512 open files
pub type Medium<S> = Fixed<S, 32, 16>;
/// 1024 open files
pub type Large<S> = Fixed<S, 64, 16>;
/// 2048 open files
pub type XLarge<S> = Fixed<S, 128, 16>;

/// A fixed-size [`Cache`].
///
/// [`Fixed`] is essentially a very simple, fixed-capacity hashtable. When a
/// pack (data-) file is requested via [`Cache::get`], the file is loaded
/// (typically `mmap`ed) from disk if it is not already in the cache. Otherwise,
/// a pointer to the already loaded file is returned. Old entries are replaced
/// on an approximate LRU basis when the cache becomes full (this means that old
/// entries are **not** evicted when there is still space).
///
/// The implementation is a somewhat dumbed-down version of JGit's
/// `WindowCache`. The main differences are that the table buckets are of fixed
/// size (`SLOTS`), instead of a linked list. This means that the cache does not
/// allow to (temporarily) commit more entries than its nominal capacity.
///
/// Reading cached values is lock-free and mostly wait-free. Modifications are
/// guarded by locks on individual buckets; if a cache miss occurs, multiple
/// threads requesting the same entry will be blocked until one of them
/// succeeded loading the data and updating the cache. Writers will _not_,
/// however, contend with readers (unlike `RwLock`).
///
/// This favours usage patterns where different threads tend to request disjoint
/// sets of packfiles, and of course their hashes colliding relatively
/// infrequently.
pub struct Fixed<M, const BUCKETS: usize, const SLOTS: usize> {
    entries: [ArcSwap<[Option<Arc<pack::Data>>; SLOTS]>; BUCKETS],
    locks: [Mutex<()>; BUCKETS],
    stats: M,
}

trait AssertSendSync: Send + Sync {}
impl<M, const B: usize, const S: usize> AssertSendSync for Fixed<M, B, S> where M: Send + Sync {}

impl<M, const B: usize, const S: usize> AsRef<Fixed<M, B, S>> for Fixed<M, B, S> {
    fn as_ref(&self) -> &Fixed<M, B, S> {
        self
    }
}

impl<const B: usize, const S: usize> Default for Fixed<Void, B, S> {
    fn default() -> Self {
        Self {
            entries: [(); B].map(|_| ArcSwap::new(Arc::new([(); S].map(|_| None)))),
            locks: [(); B].map(|_| Mutex::new(())),
            stats: PhantomData,
        }
    }
}

impl<M, const B: usize, const S: usize> Fixed<M, B, S>
where
    M: Metrics,
{
    pub fn with_stats(self) -> Fixed<Stats, B, S> {
        self.with_metrics(Stats::default())
    }

    pub fn with_metrics<N: Metrics>(self, m: N) -> Fixed<N, B, S> {
        Fixed {
            entries: self.entries,
            locks: self.locks,
            stats: m,
        }
    }

    pub fn stats(&self) -> M::Snapshot {
        let open_files = self
            .entries
            .iter()
            .map(|bucket| bucket.load().iter().flatten().count())
            .sum();
        self.stats.snapshot(open_files)
    }

    pub fn get(&self, info: &pack::Info) -> Result<Arc<pack::Data>, pack::error::Data> {
        let idx = info.hash as usize % self.entries.len();

        let bucket = self.entries[idx].load();
        for entry in bucket.iter().flatten() {
            if entry.hash == info.hash {
                self.stats.record_hit();
                entry.hit();
                return Ok(Arc::clone(entry));
            }
        }
        drop(bucket);

        self.stats.record_miss();

        // Cache miss, try to load the data file
        let lock = self.locks[idx].lock();
        // Did someone else win the race for the lock?
        let bucket = self.entries[idx].load();
        for entry in bucket.iter().flatten() {
            if entry.hash == info.hash {
                self.stats.record_hit();
                entry.hit();
                return Ok(Arc::clone(entry));
            }
        }
        // No, proceed
        self.stats.record_load();
        let data = Arc::new(info.data()?);

        // Find an empty slot, or swap with the least popular
        let mut access = usize::MAX;
        let mut evict = 0;
        for (i, e) in bucket.iter().enumerate() {
            match e {
                Some(entry) => {
                    let hits = entry.hits();
                    if hits < access {
                        access = hits;
                        evict = i;
                    }
                },
                None => {
                    evict = i;
                    break;
                },
            }
        }
        let mut entries = Guard::into_inner(bucket);
        {
            // This costs `SLOTS` refcount increments if the slot is currently
            // borrowed.
            let mutti = Arc::make_mut(&mut entries);
            mutti[evict] = Some(Arc::clone(&data));
        }
        self.entries[idx].store(entries);
        drop(lock);

        data.hit();
        Ok(data)
    }
}
diff --git a/link-git/src/odb/window/metrics.rs b/link-git/src/odb/window/metrics.rs
new file mode 100644
index 00000000..874f2912
--- /dev/null
+++ b/link-git/src/odb/window/metrics.rs
@@ -0,0 +1,86 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use std::{
    marker::PhantomData,
    sync::atomic::{AtomicUsize, Ordering},
};

use tracing::trace;

pub struct StatsView {
    /// Total number of times the requested data was found in the cache.
    pub cache_hits: usize,
    /// Total number of times the requested data was not found in the cache.
    ///
    /// Note that a cache hit can occur after a miss if another thread was
    /// faster to fill in the missing entry. Thus, `cache_hits + cache_misses`
    /// does not necessarily sum up to the number of cache accesses.
    pub cache_misses: usize,
    /// Total number of times a pack file was attempted to be loaded from disk
    /// (incl. failed attempts).
    pub file_loads: usize,
    /// Total number of pack files the cache holds on to.
    pub open_files: usize,
}

#[derive(Default)]
pub struct Stats {
    hits: AtomicUsize,
    miss: AtomicUsize,
    load: AtomicUsize,
}

pub type Void = PhantomData<!>;

pub trait Metrics {
    type Snapshot;

    fn record_hit(&self);
    fn record_miss(&self);
    fn record_load(&self);

    fn snapshot(&self, open_files: usize) -> Self::Snapshot;
}

impl Metrics for Stats {
    type Snapshot = StatsView;

    fn record_hit(&self) {
        trace!("cache hit");
        self.hits.fetch_add(1, Ordering::Relaxed);
    }

    fn record_miss(&self) {
        trace!("cache miss");
        self.miss.fetch_add(1, Ordering::Relaxed);
    }

    fn record_load(&self) {
        trace!("pack load");
        self.load.fetch_add(1, Ordering::Relaxed);
    }

    fn snapshot(&self, open_files: usize) -> Self::Snapshot {
        StatsView {
            cache_hits: self.hits.load(Ordering::Relaxed),
            cache_misses: self.miss.load(Ordering::Relaxed),
            file_loads: self.load.load(Ordering::Relaxed),
            open_files,
        }
    }
}

impl Metrics for Void {
    type Snapshot = usize;

    fn record_hit(&self) {}
    fn record_miss(&self) {}
    fn record_load(&self) {}

    fn snapshot(&self, open_files: usize) -> Self::Snapshot {
        open_files
    }
}
-- 
2.34.0

[PATCH radicle-link 3/3] link-git: shareable refdb with packed-refs cache Export this patch

Loaded `packed-refs` can typically be shared, but determining whether
they need to be reloaded is relatively expensive. Thus load them (if
present) when obtaining a `Refdb` handle, but leave it to the caller to
decide how long to operate on the snapshot.

Also fixes a race found in similar functionality provided by
`git-repository`, which doesn't lock the `packed-refs` while determining
the modification time.

Signed-off-by: Kim Altintop <kim@eagain.st>
---
 link-git/src/lib.rs     |   3 +-
 link-git/src/refs.rs    |   7 ++
 link-git/src/refs/db.rs | 266 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 275 insertions(+), 1 deletion(-)
 create mode 100644 link-git/src/refs.rs
 create mode 100644 link-git/src/refs/db.rs

diff --git a/link-git/src/lib.rs b/link-git/src/lib.rs
index 2ed159a0..d190079e 100644
--- a/link-git/src/lib.rs
+++ b/link-git/src/lib.rs
@@ -10,10 +10,11 @@ extern crate async_trait;

pub mod odb;
pub mod protocol;
pub mod refs;
pub use refs::db as refdb;

pub use git_actor as actor;
pub use git_hash as hash;
pub use git_lock as lock;
pub use git_object as object;
pub use git_ref as refs;
pub use git_traverse as traverse;
diff --git a/link-git/src/refs.rs b/link-git/src/refs.rs
new file mode 100644
index 00000000..e3d76f52
--- /dev/null
+++ b/link-git/src/refs.rs
@@ -0,0 +1,7 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

pub mod db;
pub use git_ref::*;
diff --git a/link-git/src/refs/db.rs b/link-git/src/refs/db.rs
new file mode 100644
index 00000000..c2982421
--- /dev/null
+++ b/link-git/src/refs/db.rs
@@ -0,0 +1,266 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use std::{
    collections::BTreeSet,
    convert::TryInto,
    io,
    path::{Path, PathBuf},
    sync::Arc,
    time::SystemTime,
};

use git_ref::{
    file::{self, iter::LooseThenPacked, Transaction, WriteReflog},
    packed,
    FullName,
    PartialNameRef,
    Reference,
    Target,
};
use parking_lot::RwLock;

pub mod error {
    use super::*;
    use thiserror::Error;

    #[derive(Debug, Error)]
    pub enum Open {
        #[error("failed to take a snapshot of packed-refs")]
        Snapshot(#[from] Snapshot),

        #[error(transparent)]
        Io(#[from] io::Error),
    }

    #[derive(Debug, Error)]
    pub enum Snapshot {
        #[error("failed to lock packed-refs")]
        Lock(#[from] git_lock::acquire::Error),

        #[error("failed to open packed-refs")]
        Open(#[from] packed::buffer::open::Error),

        #[error(transparent)]
        Io(#[from] io::Error),
    }

    #[derive(Debug, Error)]
    pub enum Follow {
        #[error("cyclic symref: {0:?}")]
        Cycle(FullName),

        #[error("reference {0:?} not found")]
        NotFound(FullName),

        #[error("max symref depth {0} exceeded")]
        DepthLimitExceeded(usize),

        #[error(transparent)]
        Find(#[from] file::find::Error),
    }
}

/// Threadsafe refdb with shareable `packed-refs` memory buffer.
///
/// Packed refs are a delicate business: they are written by an external
/// process, [`git-pack-refs`], _or_ when a packed ref is deleted. It may also
/// be that no `packed-refs` currently exist.
///
/// The only way we can be certain to operate on a consistent view of what is
/// committed to disk is to check if the `packed-refs` file has changed since we
/// last read it. This would be quite expensive to do for small operations.
/// Thus, the caller is responsible for determining just how much they can
/// afford to see possibly out-of-date data: the [`Refdb::snapshot`] method
/// checks if the previously loaded `packed-refs` appear to be out-of-date, and
/// reloads them if necessary. The resulting [`Snapshot`] contains a pointer to
/// an immutable memory buffer of the packed refs which can be shared between
/// threads, or cloned.
///
/// [`git-pack-refs`]: https://git-scm.com/docs/git-pack-refs
#[derive(Clone)]
pub struct Refdb {
    store: file::Store,
    packed: Arc<RwLock<Option<Packed>>>,
}

impl Refdb {
    pub fn open(git_dir: impl Into<PathBuf>) -> Result<Self, error::Open> {
        let store = file::Store::at(git_dir, WriteReflog::Normal);
        let packed = Arc::new(RwLock::new(Packed::open(store.packed_refs_path())?));
        Ok(Self { store, packed })
    }

    pub fn snapshot(&self) -> Result<Snapshot, error::Snapshot> {
        let read = self.packed.read();
        match &*read {
            None => {
                drop(read);
                // always modified, because it was None and now is Some
                self.reload(|_| true)
            },

            Some(packed) => {
                if packed.is_modified()? {
                    let mtime = packed.mtime;
                    drop(read);
                    // we don't care what the mtime is, only that we have a
                    // different value than before
                    self.reload(|packed1| packed1.mtime != mtime)
                } else {
                    Ok(Snapshot {
                        store: self.store.clone(),
                        packed: Some(packed.buf.clone()),
                    })
                }
            },
        }
    }

    fn reload<F>(&self, modified_while_blocked: F) -> Result<Snapshot, error::Snapshot>
    where
        F: FnOnce(&Packed) -> bool,
    {
        let mut write = self.packed.write();
        if let Some(packed) = &*write {
            if modified_while_blocked(packed) {
                return Ok(Snapshot {
                    store: self.store.clone(),
                    packed: Some(packed.buf.clone()),
                });
            }
        }

        match Packed::open(self.store.packed_refs_path())? {
            Some(packed) => {
                let buf = packed.buf.clone();
                *write = Some(packed);
                Ok(Snapshot {
                    store: self.store.clone(),
                    packed: Some(buf),
                })
            },

            None => {
                *write = None;
                Ok(Snapshot {
                    store: self.store.clone(),
                    packed: None,
                })
            },
        }
    }
}

#[derive(Clone)]
pub struct Snapshot {
    store: file::Store,
    packed: Option<Arc<packed::Buffer>>,
}

impl Snapshot {
    pub fn find<'a, N, E>(&self, name: N) -> Result<Option<Reference>, file::find::Error>
    where
        N: TryInto<PartialNameRef<'a>, Error = E>,
        file::find::Error: From<E>,
    {
        self.store
            .try_find(name, self.packed.as_ref().map(|arc| arc.as_ref()))
    }

    pub fn transaction(&self) -> Transaction {
        self.store.transaction()
    }

    pub fn iter(&self, prefix: Option<impl AsRef<Path>>) -> io::Result<LooseThenPacked> {
        let packed = self.packed.as_ref().map(|arc| arc.as_ref());
        match prefix {
            None => self.store.iter(packed),
            Some(p) => self.store.iter_prefixed(packed, p),
        }
    }

    /// Follow a symbolic reference until a direct reference is found.
    ///
    /// If `symref` is a direct reference, a copy of it is returned. No more
    /// than five symbolic references will be followed, and cyclic
    /// references are detected. Both result in an error to be returned.
    ///
    /// Note that following is not the same as "peeling": no access to the
    /// object database is made, and thus no assumptions about the kind of
    /// object the reference ultimately points to can be made.
    pub fn follow(&self, symref: &Reference) -> Result<Reference, error::Follow> {
        match &symref.target {
            Target::Peeled(_) => Ok(symref.clone()),
            Target::Symbolic(name) => {
                let mut seen = BTreeSet::new();
                seen.insert(symref.name.clone());

                let mut next = self
                    .find(name.to_partial())?
                    .ok_or_else(|| error::Follow::NotFound(name.clone()))?;
                seen.insert(name.clone());

                const MAX_DEPTH: usize = 5;
                loop {
                    match next.target {
                        Target::Peeled(_) => return Ok(next),
                        Target::Symbolic(sym) => {
                            if seen.contains(&sym) {
                                return Err(error::Follow::Cycle(sym));
                            }
                            next = self
                                .find(sym.to_partial())?
                                .ok_or_else(|| error::Follow::NotFound(sym.clone()))?;
                            seen.insert(sym);
                        },
                    }

                    if seen.len() > MAX_DEPTH {
                        return Err(error::Follow::DepthLimitExceeded(MAX_DEPTH));
                    }
                }
            },
        }
    }
}

struct Packed {
    buf: Arc<packed::Buffer>,
    path: PathBuf,
    mtime: SystemTime,
}

impl Packed {
    fn open(path: PathBuf) -> Result<Option<Self>, error::Snapshot> {
        use git_lock::{acquire, Marker};

        let _lock = Marker::acquire_to_hold_resource(&path, acquire::Fail::Immediately, None)?;
        match path.metadata() {
            // `git-lock` will happily lock a non-existent file
            Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(None),
            Err(e) => Err(e.into()),

            Ok(meta) => {
                let mtime = meta.modified()?;
                let buf = Arc::new(packed::Buffer::open(&path, 32 * 1024)?);
                Ok(Some(Self { buf, path, mtime }))
            },
        }
    }

    fn is_modified(&self) -> io::Result<bool> {
        match self.path.metadata() {
            // it existed before, so gone is modified
            Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(true),
            Err(e) => Err(e),

            Ok(meta) => {
                let mtime = meta.modified()?;
                Ok(self.mtime == mtime)
            },
        }
    }
}
-- 
2.34.0
radicle-link/patches/linux-x86_64.yml: SUCCESS in 31m22s

[linkoxide][0] from [Kim Altintop][1]

[0]: https://lists.sr.ht/~radicle-link/dev/patches/26683
[1]: mailto:kim@eagain.st

✓ #630900 SUCCESS radicle-link/patches/linux-x86_64.yml https://builds.sr.ht/~radicle-link/job/630900

[PATCH radicle-link v2 3/3] link-git: shareable refdb with packed-refs cache Export this patch

Loaded `packed-refs` can typically be shared, but determining whether
they need to be reloaded is relatively expensive. Thus load them (if
present) when obtaining a `Refdb` handle, but leave it to the caller to
decide how long to operate on the snapshot.

Also fixes a race found in similar functionality provided by
`git-repository`, which doesn't lock the `packed-refs` while determining
the modification time.

Signed-off-by: Kim Altintop <kim@eagain.st>
---
 link-git/src/lib.rs     |   3 +-
 link-git/src/refs.rs    |   7 ++
 link-git/src/refs/db.rs | 266 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 275 insertions(+), 1 deletion(-)
 create mode 100644 link-git/src/refs.rs
 create mode 100644 link-git/src/refs/db.rs

diff --git a/link-git/src/lib.rs b/link-git/src/lib.rs
index 2ed159a0..d190079e 100644
--- a/link-git/src/lib.rs
+++ b/link-git/src/lib.rs
@@ -10,10 +10,11 @@ extern crate async_trait;

pub mod odb;
pub mod protocol;
pub mod refs;
pub use refs::db as refdb;

pub use git_actor as actor;
pub use git_hash as hash;
pub use git_lock as lock;
pub use git_object as object;
pub use git_ref as refs;
pub use git_traverse as traverse;
diff --git a/link-git/src/refs.rs b/link-git/src/refs.rs
new file mode 100644
index 00000000..e3d76f52
--- /dev/null
+++ b/link-git/src/refs.rs
@@ -0,0 +1,7 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

pub mod db;
pub use git_ref::*;
diff --git a/link-git/src/refs/db.rs b/link-git/src/refs/db.rs
new file mode 100644
index 00000000..30bcd07e
--- /dev/null
+++ b/link-git/src/refs/db.rs
@@ -0,0 +1,266 @@
// Copyright © 2021 The Radicle Link Contributors
//
// This file is part of radicle-link, distributed under the GPLv3 with Radicle
// Linking Exception. For full terms see the included LICENSE file.

use std::{
    collections::BTreeSet,
    convert::TryInto,
    io,
    path::{Path, PathBuf},
    sync::Arc,
    time::SystemTime,
};

use git_ref::{
    file::{self, iter::LooseThenPacked, Transaction, WriteReflog},
    packed,
    FullName,
    PartialNameRef,
    Reference,
    Target,
};
use parking_lot::RwLock;

pub mod error {
    use super::*;
    use thiserror::Error;

    #[derive(Debug, Error)]
    pub enum Open {
        #[error("failed to take a snapshot of packed-refs")]
        Snapshot(#[from] Snapshot),

        #[error(transparent)]
        Io(#[from] io::Error),
    }

    #[derive(Debug, Error)]
    pub enum Snapshot {
        #[error("failed to lock packed-refs")]
        Lock(#[from] git_lock::acquire::Error),

        #[error("failed to open packed-refs")]
        Open(#[from] packed::buffer::open::Error),

        #[error(transparent)]
        Io(#[from] io::Error),
    }

    #[derive(Debug, Error)]
    pub enum Follow {
        #[error("cyclic symref: {0:?}")]
        Cycle(FullName),

        #[error("reference {0:?} not found")]
        NotFound(FullName),

        #[error("max symref depth {0} exceeded")]
        DepthLimitExceeded(usize),

        #[error(transparent)]
        Find(#[from] file::find::Error),
    }
}

/// Threadsafe refdb with shareable `packed-refs` memory buffer.
///
/// Packed refs are a delicate business: they are written by an external
/// process, [`git-pack-refs`], _or_ when a packed ref is deleted. It may also
/// be that no `packed-refs` currently exist.
///
/// The only way we can be certain to operate on a consistent view of what is
/// committed to disk is to check if the `packed-refs` file has changed since we
/// last read it. This would be quite expensive to do for small operations.
/// Thus, the caller is responsible for determining just how much they can
/// afford to see possibly out-of-date data: the [`Refdb::snapshot`] method
/// checks if the previously loaded `packed-refs` appear to be out-of-date, and
/// reloads them if necessary. The resulting [`Snapshot`] contains a pointer to
/// an immutable memory buffer of the packed refs which can be shared between
/// threads, or cloned.
///
/// [`git-pack-refs`]: https://git-scm.com/docs/git-pack-refs
#[derive(Clone)]
pub struct Refdb {
    store: file::Store,
    packed: Arc<RwLock<Option<Packed>>>,
}

impl Refdb {
    pub fn open(git_dir: impl Into<PathBuf>) -> Result<Self, error::Open> {
        let store = file::Store::at(git_dir, WriteReflog::Normal);
        let packed = Arc::new(RwLock::new(Packed::open(store.packed_refs_path())?));
        Ok(Self { store, packed })
    }

    pub fn snapshot(&self) -> Result<Snapshot, error::Snapshot> {
        let read = self.packed.read();
        match &*read {
            None => {
                drop(read);
                // always modified, because it was None and now is Some
                self.reload(|_| true)
            },

            Some(packed) => {
                if packed.is_modified()? {
                    let mtime = packed.mtime;
                    drop(read);
                    // we don't care what the mtime is, only that we have a
                    // different value than before
                    self.reload(|packed1| packed1.mtime != mtime)
                } else {
                    Ok(Snapshot {
                        store: self.store.clone(),
                        packed: Some(packed.buf.clone()),
                    })
                }
            },
        }
    }

    fn reload<F>(&self, modified_while_blocked: F) -> Result<Snapshot, error::Snapshot>
    where
        F: FnOnce(&Packed) -> bool,
    {
        let mut write = self.packed.write();
        if let Some(packed) = &*write {
            if modified_while_blocked(packed) {
                return Ok(Snapshot {
                    store: self.store.clone(),
                    packed: Some(packed.buf.clone()),
                });
            }
        }

        match Packed::open(self.store.packed_refs_path())? {
            Some(packed) => {
                let buf = packed.buf.clone();
                *write = Some(packed);
                Ok(Snapshot {
                    store: self.store.clone(),
                    packed: Some(buf),
                })
            },

            None => {
                *write = None;
                Ok(Snapshot {
                    store: self.store.clone(),
                    packed: None,
                })
            },
        }
    }
}

#[derive(Clone)]
pub struct Snapshot {
    store: file::Store,
    packed: Option<Arc<packed::Buffer>>,
}

impl Snapshot {
    pub fn find<'a, N, E>(&self, name: N) -> Result<Option<Reference>, file::find::Error>
    where
        N: TryInto<PartialNameRef<'a>, Error = E>,
        file::find::Error: From<E>,
    {
        self.store
            .try_find(name, self.packed.as_ref().map(|arc| arc.as_ref()))
    }

    pub fn transaction(&self) -> Transaction {
        self.store.transaction()
    }

    pub fn iter(&self, prefix: Option<impl AsRef<Path>>) -> io::Result<LooseThenPacked> {
        let packed = self.packed.as_ref().map(|arc| arc.as_ref());
        match prefix {
            None => self.store.iter(packed),
            Some(p) => self.store.iter_prefixed(packed, p),
        }
    }

    /// Follow a symbolic reference until a direct reference is found.
    ///
    /// If `symref` is a direct reference, a copy of it is returned. No more
    /// than five symbolic references will be followed, and cyclic
    /// references are detected. Both result in an error to be returned.
    ///
    /// Note that following is not the same as "peeling": no access to the
    /// object database is made, and thus no assumptions about the kind of
    /// object the reference ultimately points to can be made.
    pub fn follow(&self, symref: &Reference) -> Result<Reference, error::Follow> {
        match &symref.target {
            Target::Peeled(_) => Ok(symref.clone()),
            Target::Symbolic(name) => {
                let mut seen = BTreeSet::new();
                seen.insert(symref.name.clone());

                let mut next = self
                    .find(name.to_partial())?
                    .ok_or_else(|| error::Follow::NotFound(name.clone()))?;
                seen.insert(name.clone());

                const MAX_DEPTH: usize = 5;
                loop {
                    match next.target {
                        Target::Peeled(_) => return Ok(next),
                        Target::Symbolic(sym) => {
                            if seen.len() + 1 > MAX_DEPTH {
                                return Err(error::Follow::DepthLimitExceeded(MAX_DEPTH));
                            }

                            if seen.contains(&sym) {
                                return Err(error::Follow::Cycle(sym));
                            }
                            next = self
                                .find(sym.to_partial())?
                                .ok_or_else(|| error::Follow::NotFound(sym.clone()))?;
                            seen.insert(sym);
                        },
                    }
                }
            },
        }
    }
}

struct Packed {
    buf: Arc<packed::Buffer>,
    path: PathBuf,
    mtime: SystemTime,
}

impl Packed {
    fn open(path: PathBuf) -> Result<Option<Self>, error::Snapshot> {
        use git_lock::{acquire, Marker};

        let _lock = Marker::acquire_to_hold_resource(&path, acquire::Fail::Immediately, None)?;
        match path.metadata() {
            // `git-lock` will happily lock a non-existent file
            Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(None),
            Err(e) => Err(e.into()),

            Ok(meta) => {
                let mtime = meta.modified()?;
                let buf = Arc::new(packed::Buffer::open(&path, 32 * 1024)?);
                Ok(Some(Self { buf, path, mtime }))
            },
        }
    }

    fn is_modified(&self) -> io::Result<bool> {
        match self.path.metadata() {
            // it existed before, so gone is modified
            Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(true),
            Err(e) => Err(e),

            Ok(meta) => {
                let mtime = meta.modified()?;
                Ok(self.mtime == mtime)
            },
        }
    }
}
-- 
2.34.0